From 20f411e6e69ee35f280847ed5ca544c0f04772e8 Mon Sep 17 00:00:00 2001 From: Codex Date: Sun, 13 Sep 2026 21:56:47 -0300 Subject: [PATCH] docs: plan staged import files --- .../2026-09-13-staged-guided-import-files.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-13-staged-guided-import-files.md diff --git a/docs/superpowers/plans/2026-09-13-staged-guided-import-files.md b/docs/superpowers/plans/2026-09-13-staged-guided-import-files.md new file mode 100644 index 0000000..4ea706f --- /dev/null +++ b/docs/superpowers/plans/2026-09-13-staged-guided-import-files.md @@ -0,0 +1,141 @@ +# Staged Guided Import Files Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the unapproved ten-file scan cap with a simple incoming → staging → processed local file lifecycle. + +**Architecture:** Keep all lifecycle state in ignored local folders, not browser sessions or database tables. Before handling a file, move it from incoming to staging; after an existing atomic commit, move it to processed and overwrite the destination name. Failures return the staged file to incoming; a later scan recovers any leftover staged files first. + +**Tech Stack:** Python, Django, existing ingestion services, pytest, Docker Compose. + +**Spec:** `docs/superpowers/specs/2026-09-09-guided-local-import-design.md` + +## Global Constraints + +- `imports/` is incoming, `imports/staging/` is temporary, and `imports/processed/` holds successfully committed files. +- Scan only direct incoming files; staging and processed directories are never candidates. +- Move to staging before preview, return failures to incoming, and overwrite processed names only after successful commit. +- Recover direct regular leftover staging files to incoming before each scan. +- Preserve the existing parsing, normalization, validation, duplicate, preview, and persistence boundaries. +- Use no database lifecycle tables, no browser file-count cap, and no new speculative recovery framework. +- Add exactly two synthetic regression tests: one successful lifecycle and one failure/recovery lifecycle. +- Never commit real source files, local reports, database data, backups, or credentials. + +--- + +### Task 1: Stage files around the existing guided-import lifecycle + +**Files:** +- Modify: `ingestion/guided_imports.py` +- Modify: `tests/test_guided_import_workflow.py` + +**Interfaces:** +- `discover_imports() -> tuple[DiscoveredFile, ...]` recovers stale staging entries and returns incoming entries only. +- `prepare_import(...)` accepts an incoming `DiscoveredFile`, moves it to staging, then keeps its current deterministic preview/decision behavior. +- `commit_prepared_import(...)` commits through the existing service, then moves its staged source to processed. + +- [ ] **Step 1: Add the two focused synthetic tests.** + + Extend the existing guided workflow test module with exactly these behaviors: + + ```python + def test_guided_import_stages_then_processes_committed_source(source, bank): + incoming = source("bank.csv", BANK_BYTES) + prepared = prepare(incoming, bank) + committed = commit(prepared) + + assert committed.state == "COMMITTED" + assert not (incoming.parent / "bank.csv").exists() + assert (incoming.parent / "processed" / "bank.csv").read_bytes() == BANK_BYTES + + def test_guided_import_returns_failure_and_recovers_leftover_staging(source, bank): + incoming = source("bad.csv", b"unsupported") + result = prepare(incoming, bank) + + assert result.state == "NEEDS_ATTENTION" + assert (incoming.parent / "bad.csv").exists() + (incoming.parent / "staging" / "interrupted.csv").write_bytes(BANK_BYTES) + assert [row.name for row in guided_imports.discover_imports()] == ["bad.csv", "interrupted.csv"] + ``` + +- [ ] **Step 2: Verify RED.** + + Run: `docker compose run --rm test pytest -q tests/test_guided_import_workflow.py` + + Expected: FAIL because current imports remain in the incoming folder and no staging recovery exists. + +- [ ] **Step 3: Implement the minimal local lifecycle helpers.** + + Add private helpers in `ingestion/guided_imports.py` that use the existing safe directory-descriptor boundary to create and access `staging` and `processed` below the import root. They must reject symlinks and only operate on direct regular files. + + ```python + def _recover_staging() -> None: + """Move direct regular staging files back to incoming before discovery.""" + + def _stage(discovered: DiscoveredFile) -> DiscoveredFile: + """Atomically rename one verified incoming file into staging.""" + + def _return_to_incoming(name: str) -> None: + """Move a staged file back when it was not committed.""" + + def _move_to_processed(name: str) -> None: + """Replace the processed filename after a successful commit.""" + ``` + + Call `_recover_staging()` before listing incoming candidates. In `prepare_import`, stage only after validating safe discovery metadata; then read exclusively from staging and retain the staged identity in `PreparedImport`. On every `NEEDS_ATTENTION` path, return the source to incoming after cancelling an unfinished preview. In `commit_prepared_import`, call `services.commit_preview` first, then `_move_to_processed`. If the final move fails, leave the file in staging: the next discovery returns it to incoming and existing provenance prevents duplicate financial rows. + + Remove the `MAX_GUIDED_FILES` behavior and any scan-session capacity logic from the service-facing metadata; no scan count is stored or enforced. + +- [ ] **Step 4: Verify GREEN and commit.** + + Run: + + ```powershell + docker compose run --rm test pytest -q tests/test_guided_import_workflow.py + docker compose run --rm web python manage.py check + ``` + + Inspect staged paths and staged content for financial data or secrets, then commit: + + ```powershell + git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: stage guided import files" + ``` + +### Task 2: Remove the UI cap and update owner instructions + +**Files:** +- Modify: `core/views.py` +- Modify: `core/templates/core/imports/index.html` +- Modify: `tests/test_guided_import_views.py` +- Modify: `README.md` +- Modify: `docs/local-workflows.md` + +**Interfaces:** +- The Imports page continues to call `discover_imports`, `prepare_import`, and `commit_prepared_import`. +- It has no `MAX_GUIDED_FILES` check, no `SCAN_TOO_LARGE` response, and stores no per-file result list in the signed-cookie session. + +- [ ] **Step 1: Remove the count-cap branch and stale copy.** + + Delete the cap constant, `SCAN_TOO_LARGE` branch, count-specific template copy, per-file signed-cookie-session summaries, and the existing cap/cookie-capacity view tests. Render each safe per-file result list directly in the commit POST response instead of redirecting it through the session. Keep current CSRF, signed metadata, account mapping, and disclosure boundaries unchanged. + +- [ ] **Step 2: Update concise owner instructions.** + + State that files begin in `imports/`, move through `imports/staging/` while an import runs, and appear in `imports/processed/` after a successful commit. Explain that exceptions return to incoming and are safe to correct and rescan. Do not include real filenames, balances, or source content. + +- [ ] **Step 3: Verify and commit.** + + Run: + + ```powershell + docker compose run --rm test pytest -q tests/test_guided_import_views.py tests/test_guided_import_workflow.py + docker compose run --rm test pytest -q + docker compose up -d web + Invoke-RestMethod http://127.0.0.1:8000/health/ | ConvertTo-Json -Compress + git diff --check + ``` + + Inspect staged paths and staged content for financial data or secrets, then commit: + + ```powershell + git -c user.name=Codex -c user.email=codex@localhost commit -m "docs: explain staged import files" + ```