docs: plan staged import files

docs/superpowers/plans/2026-09-13-staged-guided-import-files.md

@@ -0,0 +1,141 @@
+# Staged Guided Import Files Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Replace the unapproved ten-file scan cap with a simple incoming → staging → processed local file lifecycle.
+
+**Architecture:** Keep all lifecycle state in ignored local folders, not browser sessions or database tables. Before handling a file, move it from incoming to staging; after an existing atomic commit, move it to processed and overwrite the destination name. Failures return the staged file to incoming; a later scan recovers any leftover staged files first.
+
+**Tech Stack:** Python, Django, existing ingestion services, pytest, Docker Compose.
+
+**Spec:** `docs/superpowers/specs/2026-09-09-guided-local-import-design.md`
+
+## Global Constraints
+
+- `imports/` is incoming, `imports/staging/` is temporary, and `imports/processed/` holds successfully committed files.
+- Scan only direct incoming files; staging and processed directories are never candidates.
+- Move to staging before preview, return failures to incoming, and overwrite processed names only after successful commit.
+- Recover direct regular leftover staging files to incoming before each scan.
+- Preserve the existing parsing, normalization, validation, duplicate, preview, and persistence boundaries.
+- Use no database lifecycle tables, no browser file-count cap, and no new speculative recovery framework.
+- Add exactly two synthetic regression tests: one successful lifecycle and one failure/recovery lifecycle.
+- Never commit real source files, local reports, database data, backups, or credentials.
+
+---
+
+### Task 1: Stage files around the existing guided-import lifecycle
+
+**Files:**
+- Modify: `ingestion/guided_imports.py`
+- Modify: `tests/test_guided_import_workflow.py`
+
+**Interfaces:**
+- `discover_imports() -> tuple[DiscoveredFile, ...]` recovers stale staging entries and returns incoming entries only.
+- `prepare_import(...)` accepts an incoming `DiscoveredFile`, moves it to staging, then keeps its current deterministic preview/decision behavior.
+- `commit_prepared_import(...)` commits through the existing service, then moves its staged source to processed.
+
+- [ ] **Step 1: Add the two focused synthetic tests.**
+
+  Extend the existing guided workflow test module with exactly these behaviors:
+
+  ```python
+  def test_guided_import_stages_then_processes_committed_source(source, bank):
+      incoming = source("bank.csv", BANK_BYTES)
+      prepared = prepare(incoming, bank)
+      committed = commit(prepared)
+
+      assert committed.state == "COMMITTED"
+      assert not (incoming.parent / "bank.csv").exists()
+      assert (incoming.parent / "processed" / "bank.csv").read_bytes() == BANK_BYTES
+
+  def test_guided_import_returns_failure_and_recovers_leftover_staging(source, bank):
+      incoming = source("bad.csv", b"unsupported")
+      result = prepare(incoming, bank)
+
+      assert result.state == "NEEDS_ATTENTION"
+      assert (incoming.parent / "bad.csv").exists()
+      (incoming.parent / "staging" / "interrupted.csv").write_bytes(BANK_BYTES)
+      assert [row.name for row in guided_imports.discover_imports()] == ["bad.csv", "interrupted.csv"]
+  ```
+
+- [ ] **Step 2: Verify RED.**
+
+  Run: `docker compose run --rm test pytest -q tests/test_guided_import_workflow.py`
+
+  Expected: FAIL because current imports remain in the incoming folder and no staging recovery exists.
+
+- [ ] **Step 3: Implement the minimal local lifecycle helpers.**
+
+  Add private helpers in `ingestion/guided_imports.py` that use the existing safe directory-descriptor boundary to create and access `staging` and `processed` below the import root. They must reject symlinks and only operate on direct regular files.
+
+  ```python
+  def _recover_staging() -> None:
+      """Move direct regular staging files back to incoming before discovery."""
+
+  def _stage(discovered: DiscoveredFile) -> DiscoveredFile:
+      """Atomically rename one verified incoming file into staging."""
+
+  def _return_to_incoming(name: str) -> None:
+      """Move a staged file back when it was not committed."""
+
+  def _move_to_processed(name: str) -> None:
+      """Replace the processed filename after a successful commit."""
+  ```
+
+  Call `_recover_staging()` before listing incoming candidates. In `prepare_import`, stage only after validating safe discovery metadata; then read exclusively from staging and retain the staged identity in `PreparedImport`. On every `NEEDS_ATTENTION` path, return the source to incoming after cancelling an unfinished preview. In `commit_prepared_import`, call `services.commit_preview` first, then `_move_to_processed`. If the final move fails, leave the file in staging: the next discovery returns it to incoming and existing provenance prevents duplicate financial rows.
+
+  Remove the `MAX_GUIDED_FILES` behavior and any scan-session capacity logic from the service-facing metadata; no scan count is stored or enforced.
+
+- [ ] **Step 4: Verify GREEN and commit.**
+
+  Run:
+
+  ```powershell
+  docker compose run --rm test pytest -q tests/test_guided_import_workflow.py
+  docker compose run --rm web python manage.py check
+  ```
+
+  Inspect staged paths and staged content for financial data or secrets, then commit:
+
+  ```powershell
+  git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: stage guided import files"
+  ```
+
+### Task 2: Remove the UI cap and update owner instructions
+
+**Files:**
+- Modify: `core/views.py`
+- Modify: `core/templates/core/imports/index.html`
+- Modify: `tests/test_guided_import_views.py`
+- Modify: `README.md`
+- Modify: `docs/local-workflows.md`
+
+**Interfaces:**
+- The Imports page continues to call `discover_imports`, `prepare_import`, and `commit_prepared_import`.
+- It has no `MAX_GUIDED_FILES` check, no `SCAN_TOO_LARGE` response, and stores no per-file result list in the signed-cookie session.
+
+- [ ] **Step 1: Remove the count-cap branch and stale copy.**
+
+  Delete the cap constant, `SCAN_TOO_LARGE` branch, count-specific template copy, per-file signed-cookie-session summaries, and the existing cap/cookie-capacity view tests. Render each safe per-file result list directly in the commit POST response instead of redirecting it through the session. Keep current CSRF, signed metadata, account mapping, and disclosure boundaries unchanged.
+
+- [ ] **Step 2: Update concise owner instructions.**
+
+  State that files begin in `imports/`, move through `imports/staging/` while an import runs, and appear in `imports/processed/` after a successful commit. Explain that exceptions return to incoming and are safe to correct and rescan. Do not include real filenames, balances, or source content.
+
+- [ ] **Step 3: Verify and commit.**
+
+  Run:
+
+  ```powershell
+  docker compose run --rm test pytest -q tests/test_guided_import_views.py tests/test_guided_import_workflow.py
+  docker compose run --rm test pytest -q
+  docker compose up -d web
+  Invoke-RestMethod http://127.0.0.1:8000/health/ | ConvertTo-Json -Compress
+  git diff --check
+  ```
+
+  Inspect staged paths and staged content for financial data or secrets, then commit:
+
+  ```powershell
+  git -c user.name=Codex -c user.email=codex@localhost commit -m "docs: explain staged import files"
+  ```