1 changed files with 129 additions and 0 deletions
@ -0,0 +1,129 @@ |
|||||||
|
# Guided Local Import Design |
||||||
|
|
||||||
|
## Purpose |
||||||
|
|
||||||
|
Let the local owner place supported financial documents in the ignored |
||||||
|
`imports/` directory and load clean, new transactions through the web app with |
||||||
|
minimal interaction. The owner scans the folder, selects destination accounts |
||||||
|
where needed, imports all ready files, and receives safe status summaries. |
||||||
|
|
||||||
|
The workflow preserves Gizmoduck's existing review boundaries: parsing, |
||||||
|
normalization, validation, duplicate detection, preview, decisions, and |
||||||
|
persistence remain distinct. The UI orchestrates those boundaries; it does not |
||||||
|
replace them or expose source contents in browser messages. |
||||||
|
|
||||||
|
## User workflow |
||||||
|
|
||||||
|
1. The owner copies supported files to the ignored `imports/` directory. |
||||||
|
2. The owner opens **Imports** and selects **Scan imports**. |
||||||
|
3. Gizmoduck lists each safe candidate filename, detected profile, account |
||||||
|
mapping requirements, row counts, and a safe status. |
||||||
|
4. The owner selects an account for each bank CSV and every detected card |
||||||
|
statement section that needs mapping. |
||||||
|
5. The owner selects **Import ready files**. |
||||||
|
6. Each ready file completes its own preview, automatic-decision, and commit |
||||||
|
lifecycle. The page reports created, matched, and skipped counts. |
||||||
|
7. Files needing attention remain uncommitted. The owner corrects the account |
||||||
|
choice or required context, rescans, and imports when the file becomes |
||||||
|
ready. |
||||||
|
|
||||||
|
## Scope and file handling |
||||||
|
|
||||||
|
The scanner examines only direct files under `IMPORT_ROOT`, never follows |
||||||
|
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It |
||||||
|
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. |
||||||
|
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe |
||||||
|
exceptions and never parsed or committed. |
||||||
|
|
||||||
|
The scanner retains no source rows, descriptions, amounts, or review exports in |
||||||
|
templates, messages, or logs. Those remain local source material governed by |
||||||
|
the data policy. The web UI displays filenames, profile labels, account mapping |
||||||
|
state, safe reason codes, and counts only. |
||||||
|
|
||||||
|
## Readiness and automatic decisions |
||||||
|
|
||||||
|
Each scanned file has one of these statuses: |
||||||
|
|
||||||
|
- **Ready:** every candidate can be created without a validation issue or |
||||||
|
unresolved duplicate. |
||||||
|
- **Already imported:** every source candidate is already attached through |
||||||
|
existing provenance; the file is skipped without financial writes. |
||||||
|
- **Needs attention:** the file has an unsupported/unrecognized profile, |
||||||
|
malformed or unresolved candidate, missing account mapping, ambiguous |
||||||
|
potential duplicate, or missing/invalid statement context. |
||||||
|
|
||||||
|
The import action creates a normal existing preview per ready file. It applies |
||||||
|
`CREATE` to clean candidates. Exact source reimports receive |
||||||
|
`SKIP` with `ALREADY_IMPORTED`; they produce no transaction and are not an |
||||||
|
exception. Any potential duplicate that is not an exact existing source record |
||||||
|
requires review and prevents that file from committing automatically. |
||||||
|
|
||||||
|
An exception in one file does not prevent other ready files from committing. |
||||||
|
Every file retains the import service's atomic commit boundary; a failed file |
||||||
|
creates no partial transactions or provenance. All import lifecycle actions use |
||||||
|
the local human actor and retain existing audits. |
||||||
|
|
||||||
|
## Account mapping |
||||||
|
|
||||||
|
The scan presents an account choice for every source section needed by the |
||||||
|
detected profile. Bank CSV has one section. Ourocard documents may have |
||||||
|
multiple sections and require a choice for each section. The UI uses active |
||||||
|
BRL accounts and cards only. It never maps an owner name or filename to an |
||||||
|
account automatically. |
||||||
|
|
||||||
|
The server validates that every submitted mapping corresponds to the current |
||||||
|
scan and to an active account. Missing, archived, stale, or altered mappings |
||||||
|
are exceptions; they cannot be committed. |
||||||
|
|
||||||
|
## Ourocard partial-statement dates |
||||||
|
|
||||||
|
`OUROCARD_PARTIAL_V1` gains a bounded context extractor. It reads and validates |
||||||
|
the statement creation date from line 21 of the TXT document. That date supplies |
||||||
|
the statement year and an explicit import interval. Candidate month/day values |
||||||
|
later than the creation month belong to the preceding calendar year; remaining |
||||||
|
values belong to the creation year. This handles statements crossing New Year |
||||||
|
without using the computer clock or filesystem metadata. |
||||||
|
|
||||||
|
If line 21 is absent, malformed, ambiguous, or does not yield an interval that |
||||||
|
resolves every candidate date uniquely, the file is **Needs attention** and is |
||||||
|
not committed. The existing explicit parser context remains available for the |
||||||
|
management command; this UI-only extractor supplies equivalent validated |
||||||
|
context before invoking the existing parser contract. |
||||||
|
|
||||||
|
## Components |
||||||
|
|
||||||
|
- `ingestion` receives a folder scanner, safe profile detector, and a |
||||||
|
UI-orchestration service that constructs existing previews and applies only |
||||||
|
the approved automatic decisions. |
||||||
|
- `core` receives Import navigation, scan/mapping/import forms and views, and |
||||||
|
server-rendered status templates. |
||||||
|
- Existing parser, normalization, validation, duplicate, persistence, audit, |
||||||
|
and import-batch models remain their own boundaries. |
||||||
|
- The management command remains supported for advanced/manual reviewed import |
||||||
|
workflows. |
||||||
|
|
||||||
|
## Error handling and security |
||||||
|
|
||||||
|
All POST actions use CSRF protection and revalidate scan tokens, file identity, |
||||||
|
profile, mappings, and source metadata before creating a preview or committing. |
||||||
|
The UI returns field errors and safe status codes. It does not reveal source |
||||||
|
text, paths outside the allowed folder, raw parser exceptions, or database |
||||||
|
errors. Stale scans require a new scan; the server does not reuse a changed |
||||||
|
file under an old readiness result. |
||||||
|
|
||||||
|
## Verification |
||||||
|
|
||||||
|
Tests use only synthetic fixtures and cover: |
||||||
|
|
||||||
|
- direct-folder scanning and supported-profile detection; |
||||||
|
- rejected path/symlink/oversize/unsupported input without source disclosure; |
||||||
|
- bank and multi-section card account mapping with active BRL references; |
||||||
|
- clean automatic creation through the existing audit and import lifecycle; |
||||||
|
- exact reimport skipping through source provenance; |
||||||
|
- an exceptional file not blocking an unrelated ready file; |
||||||
|
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, |
||||||
|
and CSRF rejection; |
||||||
|
- line-21 partial-statement date extraction, invalid context, and New Year |
||||||
|
rollover; and |
||||||
|
- focused web tests plus full Django, migration, permission, and integration |
||||||
|
checks. |
||||||
Loading…
Reference in new issue