docs: plan guided local imports

docs/superpowers/plans/2026-09-09-guided-local-imports.md

@@ -0,0 +1,216 @@
+# Guided Local Imports Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add an Imports page that scans the ignored local folder, maps files to accounts, and automatically commits only clean new transactions.
+
+**Architecture:** A new ingestion-facing service discovers safe direct files and derives partial-card date context without persisting source contents. A second orchestration service creates the existing preview, applies only deterministic `CREATE` and exact-source `SKIP` decisions, and commits each ready file independently. Server-rendered views hold signed scan/preflight state and display safe summaries.
+
+**Tech Stack:** Python, Django, PostgreSQL, Django signed values, existing ingestion services, pytest, Docker Compose.
+
+**Spec:** `docs/superpowers/specs/2026-09-09-guided-local-import-design.md`
+
+## Global Constraints
+
+- Scan direct files only under `IMPORT_ROOT`; reject symlinks, traversal, unsupported inputs, and files over 10 MiB.
+- Never display or commit source descriptions, amounts, source rows, raw paths, parser exceptions, or database errors in UI messages, logs, or tests.
+- Support only `BANK_CSV_V1`, `OUROCARD_FULL_V1`, and `OUROCARD_PARTIAL_V1`; OFX and PDF remain unsupported.
+- Use active BRL accounts for every submitted source section; never infer mapping from owner names or filenames.
+- Preserve parsing, normalization, validation, duplicate detection, preview, decision, and persistence boundaries.
+- Clean rows use `CREATE`; exact existing source rows use `SKIP` / `ALREADY_IMPORTED`; all other issues prevent only that file from committing.
+- Partial-card context comes only from a validated creation date at text line 21; dates later than that month use the preceding year.
+- Use only synthetic test data and never commit local financial documents, review files, exports, backups, or credentials.
+
+---
+
+### Task 1: Discover safe files and derive partial-card context
+
+**Files:**
+- Create: `ingestion/guided_imports.py`
+- Modify: `ingestion/parsers.py`
+- Create: `tests/test_guided_import_discovery.py`
+- Modify: `tests/test_import_parsers.py`
+
+**Interfaces:**
+- Produces `DiscoveredFile(name: str, document_hash: str, profile: str | None, sections: tuple[str, ...], code: str | None)`.
+- Produces `discover_imports() -> tuple[DiscoveredFile, ...]` and `partial_context_from_creation_line(data: bytes) -> dict[str, str]`.
+
+- [ ] **Step 1: Write failing pure discovery tests.**
+
+  Use `override_settings(IMPORT_ROOT=tmp_path)` and synthetic bytes. Assert direct supported files receive a profile and section list; nested files, symlinks, oversize files, PDFs, OFX, and invalid layouts return only safe code/status. Assert no result has a source-text field.
+
+  ```python
+  rows = discover_imports()
+  assert [(row.name, row.profile, row.sections, row.code) for row in rows] == [
+      ("bank.csv", "BANK_CSV_V1", ("1",), None),
+      ("unknown.ofx", None, (), "UNSUPPORTED_FILE"),
+  ]
+  ```
+
+  Add parser tests for valid synthetic line 21 creation date, absent/malformed line 21, and a January creation date that resolves `31/12` to the previous year.
+
+- [ ] **Step 2: Verify RED.**
+
+  Run: `docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py`
+
+  Expected: FAIL because the discovery module and context extractor do not exist.
+
+- [ ] **Step 3: Implement constrained discovery and context extraction.**
+
+  `discover_imports` resolves `settings.IMPORT_ROOT`, iterates only `root.iterdir()`, ignores non-files and symlinks, reads no more than `MAX_BYTES + 1`, hashes data with SHA-256, and determines a profile by attempting the supported pure grammars without persisting parse output. It returns sorted filename-only results.
+
+  ```python
+  @dataclass(frozen=True)
+  class DiscoveredFile:
+      name: str
+      document_hash: str
+      profile: str | None
+      sections: tuple[str, ...]
+      code: str | None
+  ```
+
+  Add `partial_context_from_creation_line` beside the pure parser helpers. It decodes through the existing encoding rules, reads exactly line 21, validates the approved creation-date layout, and returns an ISO `start_date`/`end_date` interval no longer than 366 days. Keep the existing explicit `parse_document(..., context=...)` contract unchanged.
+
+- [ ] **Step 4: Verify GREEN and commit.**
+
+  Run:
+
+  ```powershell
+  docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py
+  docker compose run --rm web python manage.py check
+  ```
+
+  Inspect staged content for real data, then commit:
+
+  ```powershell
+  git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: discover local import files"
+  ```
+
+### Task 2: Preflight and commit deterministic imports
+
+**Files:**
+- Modify: `ingestion/guided_imports.py`
+- Create: `tests/test_guided_import_workflow.py`
+
+**Interfaces:**
+- Consumes `DiscoveredFile`, `Actor`, and `dict[str, str]` section mappings.
+- Produces `PreparedImport(name, document_hash, profile, sections, state, batch_id, revision, counts, code)` and `prepare_import(...)` / `commit_prepared_import(...)`.
+
+- [ ] **Step 1: Write failing orchestration tests.**
+
+  Create synthetic active BRL accounts and source bytes in a temporary import root. Test that a clean file preflights to `READY`, an exact second import preflights to `ALREADY_IMPORTED`, an ambiguous duplicate is `NEEDS_ATTENTION`, and one exceptional file does not prevent another ready file from committing.
+
+  ```python
+  prepared = prepare_import(discovered=bank, section_mapping={"1": str(account.id)}, actor=ACTOR)
+  assert prepared.state == "READY"
+  committed = commit_prepared_import(prepared=prepared, actor=ACTOR)
+  assert committed.counts["created"] == 2
+  ```
+
+- [ ] **Step 2: Verify RED.**
+
+  Run: `docker compose run --rm test pytest -q tests/test_guided_import_workflow.py`
+
+  Expected: FAIL because preparation and automatic decisions do not exist.
+
+- [ ] **Step 3: Implement one service that composes existing lifecycle calls.**
+
+  Re-read the file and require its SHA-256 to equal `discovered.document_hash` before acting. Validate section keys exactly match and map to active BRL `Account` records. For partial files obtain validated context from Task 1. Call `create_preview`, inspect its items, and build complete decisions:
+
+  ```python
+  if item.issues == ["ALREADY_IMPORTED"]:
+      decision = {"locator": item.locator, "action": "SKIP", "reason": "ALREADY_IMPORTED"}
+  elif not item.issues and not item.duplicates["existing"] and not item.duplicates["within"]:
+      decision = {"locator": item.locator, "action": "CREATE"}
+  else:
+      cancel_preview(batch_id=batch.pk, revision=batch.revision, actor=actor)
+      return PreparedImport(..., state="NEEDS_ATTENTION", code="REVIEW_REQUIRED")
+  ```
+
+  Apply the complete decision list only for ready/already-imported batches. `commit_prepared_import` requires the stored batch id/revision and delegates to `commit_preview`; it never writes transactions directly. Return counts and safe codes only. File changes, stale batches, domain failures, and bad mappings yield a safe exception result and cancel any unfinished preview.
+
+- [ ] **Step 4: Verify GREEN and commit.**
+
+  Run:
+
+  ```powershell
+  docker compose run --rm test pytest -q tests/test_guided_import_workflow.py tests/test_import_workflow.py tests/test_import_concurrency.py
+  docker compose run --rm web python manage.py check
+  ```
+
+  Inspect staged content for sensitive data, then commit:
+
+  ```powershell
+  git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: prepare guided imports"
+  ```
+
+### Task 3: Add the Imports page and owner workflow coverage
+
+**Files:**
+- Modify: `config/urls.py`
+- Modify: `core/context.py`
+- Modify: `core/forms.py`
+- Modify: `core/views.py`
+- Create: `core/templates/core/imports/index.html`
+- Create: `tests/test_guided_import_views.py`
+- Modify: `tests/test_mvp_acceptance.py`
+- Modify: `README.md`
+- Modify: `docs/local-workflows.md`
+
+**Interfaces:**
+- Route: `imports-index` at `/imports/`.
+- View actions: `scan`, `prepare`, and `commit`.
+- Signed payload contains only filename, hash, profile, section identifiers, prepared batch id, and revision.
+
+- [ ] **Step 1: Write failing web tests.**
+
+  Use a temporary synthetic `IMPORT_ROOT`. Cover navigation, CSRF, scan results with safe filename/profile/status only, active-BRL account selectors, mapping submission, ready-file commit, exact reimport summary, one exception plus one successful file, and altered/stale signed state rejection.
+
+  ```python
+  response = client.post(reverse("imports-index"), {"action": "scan"})
+  assert "bank.csv" in response.content.decode()
+  assert "synthetic description" not in response.content.decode()
+  ```
+
+- [ ] **Step 2: Verify RED.**
+
+  Run: `docker compose run --rm test pytest -q tests/test_guided_import_views.py`
+
+  Expected: FAIL because the page, routes, forms, and signed workflow do not exist.
+
+- [ ] **Step 3: Implement server-rendered scan, mapping, and commit actions.**
+
+  Add Import navigation and `imports-index`. `scan` calls `discover_imports` and signs safe discovery metadata. `prepare` revalidates it, receives one typed active-account field per discovered section, and calls `prepare_import` independently for every mapped file. `commit` accepts only signed ready batch id/revision values and calls `commit_prepared_import` per file.
+
+  ```python
+  token = signing.dumps(payload, salt="guided-imports")
+  payload = signing.loads(request.POST["scan_token"], salt="guided-imports", max_age=900)
+  ```
+
+  Render only filename, profile, section label, mapping selection, state, safe code, and counts. Do not display candidate descriptions, amounts, source paths, raw exceptions, or batch audit details. Use `Actor("HUMAN", "local-owner")` and redirect after each successful POST.
+
+- [ ] **Step 4: Document and verify the owner path.**
+
+  Replace the primary manual-import instructions in `README.md` and `docs/local-workflows.md` with: copy files to `imports/`, open Imports, scan, map accounts, import ready files, and resolve listed exceptions. Retain the management command as the advanced manual workflow.
+
+  Add one synthetic MVP acceptance flow through scan, mapping, automatic import, and exact reimport skip. Run:
+
+  ```powershell
+  docker compose run --rm test pytest -q tests/test_guided_import_views.py tests/test_mvp_acceptance.py
+  docker compose build
+  docker compose run --rm test pytest -q
+  docker compose run --rm web python manage.py migrate --check
+  docker compose run --rm web python manage.py check
+  docker compose run --rm permissions
+  docker compose up -d web
+  Invoke-RestMethod http://127.0.0.1:8000/health/ | ConvertTo-Json -Compress
+  git diff --check
+  ```
+
+- [ ] **Step 5: Commit the UI, documentation, and acceptance slice.**
+
+  Inspect staged paths and content for financial data and credentials, then commit:
+
+  ```powershell
+  git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: add guided import page"
+  ```