From 7f6fea07a03fba7999d5de9a1cffcd63abb7fe01 Mon Sep 17 00:00:00 2001 From: Codex Date: Wed, 9 Sep 2026 19:44:03 -0300 Subject: [PATCH] docs: design guided local imports --- .../2026-09-09-guided-local-import-design.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-09-guided-local-import-design.md diff --git a/docs/superpowers/specs/2026-09-09-guided-local-import-design.md b/docs/superpowers/specs/2026-09-09-guided-local-import-design.md new file mode 100644 index 0000000..bd28dd7 --- /dev/null +++ b/docs/superpowers/specs/2026-09-09-guided-local-import-design.md @@ -0,0 +1,129 @@ +# Guided Local Import Design + +## Purpose + +Let the local owner place supported financial documents in the ignored +`imports/` directory and load clean, new transactions through the web app with +minimal interaction. The owner scans the folder, selects destination accounts +where needed, imports all ready files, and receives safe status summaries. + +The workflow preserves Gizmoduck's existing review boundaries: parsing, +normalization, validation, duplicate detection, preview, decisions, and +persistence remain distinct. The UI orchestrates those boundaries; it does not +replace them or expose source contents in browser messages. + +## User workflow + +1. The owner copies supported files to the ignored `imports/` directory. +2. The owner opens **Imports** and selects **Scan imports**. +3. Gizmoduck lists each safe candidate filename, detected profile, account + mapping requirements, row counts, and a safe status. +4. The owner selects an account for each bank CSV and every detected card + statement section that needs mapping. +5. The owner selects **Import ready files**. +6. Each ready file completes its own preview, automatic-decision, and commit + lifecycle. The page reports created, matched, and skipped counts. +7. Files needing attention remain uncommitted. The owner corrects the account + choice or required context, rescans, and imports when the file becomes + ready. + +## Scope and file handling + +The scanner examines only direct files under `IMPORT_ROOT`, never follows +symlinks, rejects traversal, and applies the existing 10 MiB input limit. It +supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. +OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe +exceptions and never parsed or committed. + +The scanner retains no source rows, descriptions, amounts, or review exports in +templates, messages, or logs. Those remain local source material governed by +the data policy. The web UI displays filenames, profile labels, account mapping +state, safe reason codes, and counts only. + +## Readiness and automatic decisions + +Each scanned file has one of these statuses: + +- **Ready:** every candidate can be created without a validation issue or + unresolved duplicate. +- **Already imported:** every source candidate is already attached through + existing provenance; the file is skipped without financial writes. +- **Needs attention:** the file has an unsupported/unrecognized profile, + malformed or unresolved candidate, missing account mapping, ambiguous + potential duplicate, or missing/invalid statement context. + +The import action creates a normal existing preview per ready file. It applies +`CREATE` to clean candidates. Exact source reimports receive +`SKIP` with `ALREADY_IMPORTED`; they produce no transaction and are not an +exception. Any potential duplicate that is not an exact existing source record +requires review and prevents that file from committing automatically. + +An exception in one file does not prevent other ready files from committing. +Every file retains the import service's atomic commit boundary; a failed file +creates no partial transactions or provenance. All import lifecycle actions use +the local human actor and retain existing audits. + +## Account mapping + +The scan presents an account choice for every source section needed by the +detected profile. Bank CSV has one section. Ourocard documents may have +multiple sections and require a choice for each section. The UI uses active +BRL accounts and cards only. It never maps an owner name or filename to an +account automatically. + +The server validates that every submitted mapping corresponds to the current +scan and to an active account. Missing, archived, stale, or altered mappings +are exceptions; they cannot be committed. + +## Ourocard partial-statement dates + +`OUROCARD_PARTIAL_V1` gains a bounded context extractor. It reads and validates +the statement creation date from line 21 of the TXT document. That date supplies +the statement year and an explicit import interval. Candidate month/day values +later than the creation month belong to the preceding calendar year; remaining +values belong to the creation year. This handles statements crossing New Year +without using the computer clock or filesystem metadata. + +If line 21 is absent, malformed, ambiguous, or does not yield an interval that +resolves every candidate date uniquely, the file is **Needs attention** and is +not committed. The existing explicit parser context remains available for the +management command; this UI-only extractor supplies equivalent validated +context before invoking the existing parser contract. + +## Components + +- `ingestion` receives a folder scanner, safe profile detector, and a + UI-orchestration service that constructs existing previews and applies only + the approved automatic decisions. +- `core` receives Import navigation, scan/mapping/import forms and views, and + server-rendered status templates. +- Existing parser, normalization, validation, duplicate, persistence, audit, + and import-batch models remain their own boundaries. +- The management command remains supported for advanced/manual reviewed import + workflows. + +## Error handling and security + +All POST actions use CSRF protection and revalidate scan tokens, file identity, +profile, mappings, and source metadata before creating a preview or committing. +The UI returns field errors and safe status codes. It does not reveal source +text, paths outside the allowed folder, raw parser exceptions, or database +errors. Stale scans require a new scan; the server does not reuse a changed +file under an old readiness result. + +## Verification + +Tests use only synthetic fixtures and cover: + +- direct-folder scanning and supported-profile detection; +- rejected path/symlink/oversize/unsupported input without source disclosure; +- bank and multi-section card account mapping with active BRL references; +- clean automatic creation through the existing audit and import lifecycle; +- exact reimport skipping through source provenance; +- an exceptional file not blocking an unrelated ready file; +- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, + and CSRF rejection; +- line-21 partial-statement date extraction, invalid context, and New Year + rollover; and +- focused web tests plus full Django, migration, permission, and integration + checks.