From dce6e6102d98a433b695619bf1a3360f22a6428b Mon Sep 17 00:00:00 2001 From: Codex Date: Thu, 10 Sep 2026 21:28:24 -0300 Subject: [PATCH] docs: stage guided import files --- .../2026-09-09-guided-local-import-design.md | 61 ++++++++++++++----- 1 file changed, 45 insertions(+), 16 deletions(-) diff --git a/docs/superpowers/specs/2026-09-09-guided-local-import-design.md b/docs/superpowers/specs/2026-09-09-guided-local-import-design.md index bd28dd7..4a68e8c 100644 --- a/docs/superpowers/specs/2026-09-09-guided-local-import-design.md +++ b/docs/superpowers/specs/2026-09-09-guided-local-import-design.md @@ -16,24 +16,34 @@ replace them or expose source contents in browser messages. 1. The owner copies supported files to the ignored `imports/` directory. 2. The owner opens **Imports** and selects **Scan imports**. -3. Gizmoduck lists each safe candidate filename, detected profile, account +3. Gizmoduck returns interrupted staging files to the incoming folder, then + lists each safe candidate filename, detected profile, account mapping requirements, row counts, and a safe status. 4. The owner selects an account for each bank CSV and every detected card statement section that needs mapping. 5. The owner selects **Import ready files**. -6. Each ready file completes its own preview, automatic-decision, and commit - lifecycle. The page reports created, matched, and skipped counts. +6. Each ready file moves through staging, preview, automatic-decision, and + commit. A committed file moves to processed. The page reports created, + matched, and skipped counts. 7. Files needing attention remain uncommitted. The owner corrects the account choice or required context, rescans, and imports when the file becomes ready. ## Scope and file handling -The scanner examines only direct files under `IMPORT_ROOT`, never follows -symlinks, rejects traversal, and applies the existing 10 MiB input limit. It -supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. -OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe -exceptions and never parsed or committed. +`IMPORT_ROOT` has three local, ignored folders: incoming files at `imports/`, +temporary work at `imports/staging/`, and successful source documents at +`imports/processed/`. The scanner examines only direct files in the incoming +folder; it has no fixed file-count limit. It never follows symlinks, rejects +traversal, and applies the existing 10 MiB input limit. It supports only the +existing Banco do Brasil bank CSV and Ourocard TXT formats. OFX, PDFs, unknown +extensions, and unrecognized layouts are reported as safe exceptions and never +parsed or committed. + +Before each scan, Gizmoduck moves direct regular files left in `staging/` back +to incoming. This recovers a run interrupted between staging and its final +move. Neither `staging/` nor `processed/` is included in normal candidate +scanning. The scanner retains no source rows, descriptions, amounts, or review exports in templates, messages, or logs. Those remain local source material governed by @@ -63,6 +73,22 @@ Every file retains the import service's atomic commit boundary; a failed file creates no partial transactions or provenance. All import lifecycle actions use the local human actor and retain existing audits. +## Local file lifecycle + +Before creating a preview, Gizmoduck atomically moves each selected incoming +file to `staging/` on the same local volume. It then reads and verifies the +staged file, prepares it through the existing import lifecycle, and commits it. +After a successful commit, it moves the staged source file to `processed/`. +An existing processed file with the same name is overwritten only at this final +successful step. + +If preparation, validation, duplicate review, commit, or a pre-commit file +operation fails, Gizmoduck moves the staged file back to incoming and reports a +safe exception. If the process stops after a database commit but before the +processed move, the next scan restores that file to incoming. Existing +provenance then prevents duplicate transactions; the automatic exact-source +skip path may move it to processed. + ## Account mapping The scan presents an account choice for every source section needed by the @@ -104,12 +130,12 @@ context before invoking the existing parser contract. ## Error handling and security -All POST actions use CSRF protection and revalidate scan tokens, file identity, -profile, mappings, and source metadata before creating a preview or committing. -The UI returns field errors and safe status codes. It does not reveal source -text, paths outside the allowed folder, raw parser exceptions, or database -errors. Stale scans require a new scan; the server does not reuse a changed -file under an old readiness result. +All POST actions use CSRF protection and revalidate file identity, profile, +mappings, and source metadata before creating a preview or committing. The UI +returns field errors and safe status codes. It does not reveal source text, +paths outside the allowed folder, raw parser exceptions, or database errors. +Staging identity is verified before every lifecycle step; a changed staged file +returns to incoming without commitment. ## Verification @@ -117,12 +143,15 @@ Tests use only synthetic fixtures and cover: - direct-folder scanning and supported-profile detection; - rejected path/symlink/oversize/unsupported input without source disclosure; +- staged success moving to processed, exception return to incoming, interrupted + staging recovery, and processed-name overwrite only after a successful + commit; - bank and multi-section card account mapping with active BRL references; - clean automatic creation through the existing audit and import lifecycle; - exact reimport skipping through source provenance; - an exceptional file not blocking an unrelated ready file; -- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, - and CSRF rejection; +- ambiguous duplicates, malformed candidates, mapping tampering, and CSRF + rejection; - line-21 partial-statement date extraction, invalid context, and New Year rollover; and - focused web tests plus full Django, migration, permission, and integration