Browse Source

docs: design guided local imports

master
Codex 2 months ago
parent
commit
7f6fea07a0
  1. 129
      docs/superpowers/specs/2026-09-09-guided-local-import-design.md

129
docs/superpowers/specs/2026-09-09-guided-local-import-design.md

@ -0,0 +1,129 @@
# Guided Local Import Design
## Purpose
Let the local owner place supported financial documents in the ignored
`imports/` directory and load clean, new transactions through the web app with
minimal interaction. The owner scans the folder, selects destination accounts
where needed, imports all ready files, and receives safe status summaries.
The workflow preserves Gizmoduck's existing review boundaries: parsing,
normalization, validation, duplicate detection, preview, decisions, and
persistence remain distinct. The UI orchestrates those boundaries; it does not
replace them or expose source contents in browser messages.
## User workflow
1. The owner copies supported files to the ignored `imports/` directory.
2. The owner opens **Imports** and selects **Scan imports**.
3. Gizmoduck lists each safe candidate filename, detected profile, account
mapping requirements, row counts, and a safe status.
4. The owner selects an account for each bank CSV and every detected card
statement section that needs mapping.
5. The owner selects **Import ready files**.
6. Each ready file completes its own preview, automatic-decision, and commit
lifecycle. The page reports created, matched, and skipped counts.
7. Files needing attention remain uncommitted. The owner corrects the account
choice or required context, rescans, and imports when the file becomes
ready.
## Scope and file handling
The scanner examines only direct files under `IMPORT_ROOT`, never follows
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats.
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe
exceptions and never parsed or committed.
The scanner retains no source rows, descriptions, amounts, or review exports in
templates, messages, or logs. Those remain local source material governed by
the data policy. The web UI displays filenames, profile labels, account mapping
state, safe reason codes, and counts only.
## Readiness and automatic decisions
Each scanned file has one of these statuses:
- **Ready:** every candidate can be created without a validation issue or
unresolved duplicate.
- **Already imported:** every source candidate is already attached through
existing provenance; the file is skipped without financial writes.
- **Needs attention:** the file has an unsupported/unrecognized profile,
malformed or unresolved candidate, missing account mapping, ambiguous
potential duplicate, or missing/invalid statement context.
The import action creates a normal existing preview per ready file. It applies
`CREATE` to clean candidates. Exact source reimports receive
`SKIP` with `ALREADY_IMPORTED`; they produce no transaction and are not an
exception. Any potential duplicate that is not an exact existing source record
requires review and prevents that file from committing automatically.
An exception in one file does not prevent other ready files from committing.
Every file retains the import service's atomic commit boundary; a failed file
creates no partial transactions or provenance. All import lifecycle actions use
the local human actor and retain existing audits.
## Account mapping
The scan presents an account choice for every source section needed by the
detected profile. Bank CSV has one section. Ourocard documents may have
multiple sections and require a choice for each section. The UI uses active
BRL accounts and cards only. It never maps an owner name or filename to an
account automatically.
The server validates that every submitted mapping corresponds to the current
scan and to an active account. Missing, archived, stale, or altered mappings
are exceptions; they cannot be committed.
## Ourocard partial-statement dates
`OUROCARD_PARTIAL_V1` gains a bounded context extractor. It reads and validates
the statement creation date from line 21 of the TXT document. That date supplies
the statement year and an explicit import interval. Candidate month/day values
later than the creation month belong to the preceding calendar year; remaining
values belong to the creation year. This handles statements crossing New Year
without using the computer clock or filesystem metadata.
If line 21 is absent, malformed, ambiguous, or does not yield an interval that
resolves every candidate date uniquely, the file is **Needs attention** and is
not committed. The existing explicit parser context remains available for the
management command; this UI-only extractor supplies equivalent validated
context before invoking the existing parser contract.
## Components
- `ingestion` receives a folder scanner, safe profile detector, and a
UI-orchestration service that constructs existing previews and applies only
the approved automatic decisions.
- `core` receives Import navigation, scan/mapping/import forms and views, and
server-rendered status templates.
- Existing parser, normalization, validation, duplicate, persistence, audit,
and import-batch models remain their own boundaries.
- The management command remains supported for advanced/manual reviewed import
workflows.
## Error handling and security
All POST actions use CSRF protection and revalidate scan tokens, file identity,
profile, mappings, and source metadata before creating a preview or committing.
The UI returns field errors and safe status codes. It does not reveal source
text, paths outside the allowed folder, raw parser exceptions, or database
errors. Stale scans require a new scan; the server does not reuse a changed
file under an old readiness result.
## Verification
Tests use only synthetic fixtures and cover:
- direct-folder scanning and supported-profile detection;
- rejected path/symlink/oversize/unsupported input without source disclosure;
- bank and multi-section card account mapping with active BRL references;
- clean automatic creation through the existing audit and import lifecycle;
- exact reimport skipping through source provenance;
- an exceptional file not blocking an unrelated ready file;
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering,
and CSRF rejection;
- line-21 partial-statement date extraction, invalid context, and New Year
rollover; and
- focused web tests plus full Django, migration, permission, and integration
checks.
Loading…
Cancel
Save