diff --git a/docs/superpowers/plans/2026-09-09-guided-local-imports.md b/docs/superpowers/plans/2026-09-09-guided-local-imports.md new file mode 100644 index 0000000..742c05e --- /dev/null +++ b/docs/superpowers/plans/2026-09-09-guided-local-imports.md @@ -0,0 +1,216 @@ +# Guided Local Imports Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add an Imports page that scans the ignored local folder, maps files to accounts, and automatically commits only clean new transactions. + +**Architecture:** A new ingestion-facing service discovers safe direct files and derives partial-card date context without persisting source contents. A second orchestration service creates the existing preview, applies only deterministic `CREATE` and exact-source `SKIP` decisions, and commits each ready file independently. Server-rendered views hold signed scan/preflight state and display safe summaries. + +**Tech Stack:** Python, Django, PostgreSQL, Django signed values, existing ingestion services, pytest, Docker Compose. + +**Spec:** `docs/superpowers/specs/2026-09-09-guided-local-import-design.md` + +## Global Constraints + +- Scan direct files only under `IMPORT_ROOT`; reject symlinks, traversal, unsupported inputs, and files over 10 MiB. +- Never display or commit source descriptions, amounts, source rows, raw paths, parser exceptions, or database errors in UI messages, logs, or tests. +- Support only `BANK_CSV_V1`, `OUROCARD_FULL_V1`, and `OUROCARD_PARTIAL_V1`; OFX and PDF remain unsupported. +- Use active BRL accounts for every submitted source section; never infer mapping from owner names or filenames. +- Preserve parsing, normalization, validation, duplicate detection, preview, decision, and persistence boundaries. +- Clean rows use `CREATE`; exact existing source rows use `SKIP` / `ALREADY_IMPORTED`; all other issues prevent only that file from committing. +- Partial-card context comes only from a validated creation date at text line 21; dates later than that month use the preceding year. +- Use only synthetic test data and never commit local financial documents, review files, exports, backups, or credentials. + +--- + +### Task 1: Discover safe files and derive partial-card context + +**Files:** +- Create: `ingestion/guided_imports.py` +- Modify: `ingestion/parsers.py` +- Create: `tests/test_guided_import_discovery.py` +- Modify: `tests/test_import_parsers.py` + +**Interfaces:** +- Produces `DiscoveredFile(name: str, document_hash: str, profile: str | None, sections: tuple[str, ...], code: str | None)`. +- Produces `discover_imports() -> tuple[DiscoveredFile, ...]` and `partial_context_from_creation_line(data: bytes) -> dict[str, str]`. + +- [ ] **Step 1: Write failing pure discovery tests.** + + Use `override_settings(IMPORT_ROOT=tmp_path)` and synthetic bytes. Assert direct supported files receive a profile and section list; nested files, symlinks, oversize files, PDFs, OFX, and invalid layouts return only safe code/status. Assert no result has a source-text field. + + ```python + rows = discover_imports() + assert [(row.name, row.profile, row.sections, row.code) for row in rows] == [ + ("bank.csv", "BANK_CSV_V1", ("1",), None), + ("unknown.ofx", None, (), "UNSUPPORTED_FILE"), + ] + ``` + + Add parser tests for valid synthetic line 21 creation date, absent/malformed line 21, and a January creation date that resolves `31/12` to the previous year. + +- [ ] **Step 2: Verify RED.** + + Run: `docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py` + + Expected: FAIL because the discovery module and context extractor do not exist. + +- [ ] **Step 3: Implement constrained discovery and context extraction.** + + `discover_imports` resolves `settings.IMPORT_ROOT`, iterates only `root.iterdir()`, ignores non-files and symlinks, reads no more than `MAX_BYTES + 1`, hashes data with SHA-256, and determines a profile by attempting the supported pure grammars without persisting parse output. It returns sorted filename-only results. + + ```python + @dataclass(frozen=True) + class DiscoveredFile: + name: str + document_hash: str + profile: str | None + sections: tuple[str, ...] + code: str | None + ``` + + Add `partial_context_from_creation_line` beside the pure parser helpers. It decodes through the existing encoding rules, reads exactly line 21, validates the approved creation-date layout, and returns an ISO `start_date`/`end_date` interval no longer than 366 days. Keep the existing explicit `parse_document(..., context=...)` contract unchanged. + +- [ ] **Step 4: Verify GREEN and commit.** + + Run: + + ```powershell + docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py + docker compose run --rm web python manage.py check + ``` + + Inspect staged content for real data, then commit: + + ```powershell + git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: discover local import files" + ``` + +### Task 2: Preflight and commit deterministic imports + +**Files:** +- Modify: `ingestion/guided_imports.py` +- Create: `tests/test_guided_import_workflow.py` + +**Interfaces:** +- Consumes `DiscoveredFile`, `Actor`, and `dict[str, str]` section mappings. +- Produces `PreparedImport(name, document_hash, profile, sections, state, batch_id, revision, counts, code)` and `prepare_import(...)` / `commit_prepared_import(...)`. + +- [ ] **Step 1: Write failing orchestration tests.** + + Create synthetic active BRL accounts and source bytes in a temporary import root. Test that a clean file preflights to `READY`, an exact second import preflights to `ALREADY_IMPORTED`, an ambiguous duplicate is `NEEDS_ATTENTION`, and one exceptional file does not prevent another ready file from committing. + + ```python + prepared = prepare_import(discovered=bank, section_mapping={"1": str(account.id)}, actor=ACTOR) + assert prepared.state == "READY" + committed = commit_prepared_import(prepared=prepared, actor=ACTOR) + assert committed.counts["created"] == 2 + ``` + +- [ ] **Step 2: Verify RED.** + + Run: `docker compose run --rm test pytest -q tests/test_guided_import_workflow.py` + + Expected: FAIL because preparation and automatic decisions do not exist. + +- [ ] **Step 3: Implement one service that composes existing lifecycle calls.** + + Re-read the file and require its SHA-256 to equal `discovered.document_hash` before acting. Validate section keys exactly match and map to active BRL `Account` records. For partial files obtain validated context from Task 1. Call `create_preview`, inspect its items, and build complete decisions: + + ```python + if item.issues == ["ALREADY_IMPORTED"]: + decision = {"locator": item.locator, "action": "SKIP", "reason": "ALREADY_IMPORTED"} + elif not item.issues and not item.duplicates["existing"] and not item.duplicates["within"]: + decision = {"locator": item.locator, "action": "CREATE"} + else: + cancel_preview(batch_id=batch.pk, revision=batch.revision, actor=actor) + return PreparedImport(..., state="NEEDS_ATTENTION", code="REVIEW_REQUIRED") + ``` + + Apply the complete decision list only for ready/already-imported batches. `commit_prepared_import` requires the stored batch id/revision and delegates to `commit_preview`; it never writes transactions directly. Return counts and safe codes only. File changes, stale batches, domain failures, and bad mappings yield a safe exception result and cancel any unfinished preview. + +- [ ] **Step 4: Verify GREEN and commit.** + + Run: + + ```powershell + docker compose run --rm test pytest -q tests/test_guided_import_workflow.py tests/test_import_workflow.py tests/test_import_concurrency.py + docker compose run --rm web python manage.py check + ``` + + Inspect staged content for sensitive data, then commit: + + ```powershell + git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: prepare guided imports" + ``` + +### Task 3: Add the Imports page and owner workflow coverage + +**Files:** +- Modify: `config/urls.py` +- Modify: `core/context.py` +- Modify: `core/forms.py` +- Modify: `core/views.py` +- Create: `core/templates/core/imports/index.html` +- Create: `tests/test_guided_import_views.py` +- Modify: `tests/test_mvp_acceptance.py` +- Modify: `README.md` +- Modify: `docs/local-workflows.md` + +**Interfaces:** +- Route: `imports-index` at `/imports/`. +- View actions: `scan`, `prepare`, and `commit`. +- Signed payload contains only filename, hash, profile, section identifiers, prepared batch id, and revision. + +- [ ] **Step 1: Write failing web tests.** + + Use a temporary synthetic `IMPORT_ROOT`. Cover navigation, CSRF, scan results with safe filename/profile/status only, active-BRL account selectors, mapping submission, ready-file commit, exact reimport summary, one exception plus one successful file, and altered/stale signed state rejection. + + ```python + response = client.post(reverse("imports-index"), {"action": "scan"}) + assert "bank.csv" in response.content.decode() + assert "synthetic description" not in response.content.decode() + ``` + +- [ ] **Step 2: Verify RED.** + + Run: `docker compose run --rm test pytest -q tests/test_guided_import_views.py` + + Expected: FAIL because the page, routes, forms, and signed workflow do not exist. + +- [ ] **Step 3: Implement server-rendered scan, mapping, and commit actions.** + + Add Import navigation and `imports-index`. `scan` calls `discover_imports` and signs safe discovery metadata. `prepare` revalidates it, receives one typed active-account field per discovered section, and calls `prepare_import` independently for every mapped file. `commit` accepts only signed ready batch id/revision values and calls `commit_prepared_import` per file. + + ```python + token = signing.dumps(payload, salt="guided-imports") + payload = signing.loads(request.POST["scan_token"], salt="guided-imports", max_age=900) + ``` + + Render only filename, profile, section label, mapping selection, state, safe code, and counts. Do not display candidate descriptions, amounts, source paths, raw exceptions, or batch audit details. Use `Actor("HUMAN", "local-owner")` and redirect after each successful POST. + +- [ ] **Step 4: Document and verify the owner path.** + + Replace the primary manual-import instructions in `README.md` and `docs/local-workflows.md` with: copy files to `imports/`, open Imports, scan, map accounts, import ready files, and resolve listed exceptions. Retain the management command as the advanced manual workflow. + + Add one synthetic MVP acceptance flow through scan, mapping, automatic import, and exact reimport skip. Run: + + ```powershell + docker compose run --rm test pytest -q tests/test_guided_import_views.py tests/test_mvp_acceptance.py + docker compose build + docker compose run --rm test pytest -q + docker compose run --rm web python manage.py migrate --check + docker compose run --rm web python manage.py check + docker compose run --rm permissions + docker compose up -d web + Invoke-RestMethod http://127.0.0.1:8000/health/ | ConvertTo-Json -Compress + git diff --check + ``` + +- [ ] **Step 5: Commit the UI, documentation, and acceptance slice.** + + Inspect staged paths and content for financial data and credentials, then commit: + + ```powershell + git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: add guided import page" + ```