Browse Source

docs: stage guided import files

master
Codex 2 months ago
parent
commit
dce6e6102d
  1. 61
      docs/superpowers/specs/2026-09-09-guided-local-import-design.md

61
docs/superpowers/specs/2026-09-09-guided-local-import-design.md

@ -16,24 +16,34 @@ replace them or expose source contents in browser messages.
1. The owner copies supported files to the ignored `imports/` directory. 1. The owner copies supported files to the ignored `imports/` directory.
2. The owner opens **Imports** and selects **Scan imports**. 2. The owner opens **Imports** and selects **Scan imports**.
3. Gizmoduck lists each safe candidate filename, detected profile, account 3. Gizmoduck returns interrupted staging files to the incoming folder, then
lists each safe candidate filename, detected profile, account
mapping requirements, row counts, and a safe status. mapping requirements, row counts, and a safe status.
4. The owner selects an account for each bank CSV and every detected card 4. The owner selects an account for each bank CSV and every detected card
statement section that needs mapping. statement section that needs mapping.
5. The owner selects **Import ready files**. 5. The owner selects **Import ready files**.
6. Each ready file completes its own preview, automatic-decision, and commit 6. Each ready file moves through staging, preview, automatic-decision, and
lifecycle. The page reports created, matched, and skipped counts. commit. A committed file moves to processed. The page reports created,
matched, and skipped counts.
7. Files needing attention remain uncommitted. The owner corrects the account 7. Files needing attention remain uncommitted. The owner corrects the account
choice or required context, rescans, and imports when the file becomes choice or required context, rescans, and imports when the file becomes
ready. ready.
## Scope and file handling ## Scope and file handling
The scanner examines only direct files under `IMPORT_ROOT`, never follows `IMPORT_ROOT` has three local, ignored folders: incoming files at `imports/`,
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It temporary work at `imports/staging/`, and successful source documents at
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. `imports/processed/`. The scanner examines only direct files in the incoming
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe folder; it has no fixed file-count limit. It never follows symlinks, rejects
exceptions and never parsed or committed. traversal, and applies the existing 10 MiB input limit. It supports only the
existing Banco do Brasil bank CSV and Ourocard TXT formats. OFX, PDFs, unknown
extensions, and unrecognized layouts are reported as safe exceptions and never
parsed or committed.
Before each scan, Gizmoduck moves direct regular files left in `staging/` back
to incoming. This recovers a run interrupted between staging and its final
move. Neither `staging/` nor `processed/` is included in normal candidate
scanning.
The scanner retains no source rows, descriptions, amounts, or review exports in The scanner retains no source rows, descriptions, amounts, or review exports in
templates, messages, or logs. Those remain local source material governed by templates, messages, or logs. Those remain local source material governed by
@ -63,6 +73,22 @@ Every file retains the import service's atomic commit boundary; a failed file
creates no partial transactions or provenance. All import lifecycle actions use creates no partial transactions or provenance. All import lifecycle actions use
the local human actor and retain existing audits. the local human actor and retain existing audits.
## Local file lifecycle
Before creating a preview, Gizmoduck atomically moves each selected incoming
file to `staging/` on the same local volume. It then reads and verifies the
staged file, prepares it through the existing import lifecycle, and commits it.
After a successful commit, it moves the staged source file to `processed/`.
An existing processed file with the same name is overwritten only at this final
successful step.
If preparation, validation, duplicate review, commit, or a pre-commit file
operation fails, Gizmoduck moves the staged file back to incoming and reports a
safe exception. If the process stops after a database commit but before the
processed move, the next scan restores that file to incoming. Existing
provenance then prevents duplicate transactions; the automatic exact-source
skip path may move it to processed.
## Account mapping ## Account mapping
The scan presents an account choice for every source section needed by the The scan presents an account choice for every source section needed by the
@ -104,12 +130,12 @@ context before invoking the existing parser contract.
## Error handling and security ## Error handling and security
All POST actions use CSRF protection and revalidate scan tokens, file identity, All POST actions use CSRF protection and revalidate file identity, profile,
profile, mappings, and source metadata before creating a preview or committing. mappings, and source metadata before creating a preview or committing. The UI
The UI returns field errors and safe status codes. It does not reveal source returns field errors and safe status codes. It does not reveal source text,
text, paths outside the allowed folder, raw parser exceptions, or database paths outside the allowed folder, raw parser exceptions, or database errors.
errors. Stale scans require a new scan; the server does not reuse a changed Staging identity is verified before every lifecycle step; a changed staged file
file under an old readiness result. returns to incoming without commitment.
## Verification ## Verification
@ -117,12 +143,15 @@ Tests use only synthetic fixtures and cover:
- direct-folder scanning and supported-profile detection; - direct-folder scanning and supported-profile detection;
- rejected path/symlink/oversize/unsupported input without source disclosure; - rejected path/symlink/oversize/unsupported input without source disclosure;
- staged success moving to processed, exception return to incoming, interrupted
staging recovery, and processed-name overwrite only after a successful
commit;
- bank and multi-section card account mapping with active BRL references; - bank and multi-section card account mapping with active BRL references;
- clean automatic creation through the existing audit and import lifecycle; - clean automatic creation through the existing audit and import lifecycle;
- exact reimport skipping through source provenance; - exact reimport skipping through source provenance;
- an exceptional file not blocking an unrelated ready file; - an exceptional file not blocking an unrelated ready file;
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, - ambiguous duplicates, malformed candidates, mapping tampering, and CSRF
and CSRF rejection; rejection;
- line-21 partial-statement date extraction, invalid context, and New Year - line-21 partial-statement date extraction, invalid context, and New Year
rollover; and rollover; and
- focused web tests plus full Django, migration, permission, and integration - focused web tests plus full Django, migration, permission, and integration

Loading…
Cancel
Save