|
|
|
|
@ -16,24 +16,34 @@ replace them or expose source contents in browser messages.
|
|
|
|
|
|
|
|
|
|
1. The owner copies supported files to the ignored `imports/` directory. |
|
|
|
|
2. The owner opens **Imports** and selects **Scan imports**. |
|
|
|
|
3. Gizmoduck lists each safe candidate filename, detected profile, account |
|
|
|
|
3. Gizmoduck returns interrupted staging files to the incoming folder, then |
|
|
|
|
lists each safe candidate filename, detected profile, account |
|
|
|
|
mapping requirements, row counts, and a safe status. |
|
|
|
|
4. The owner selects an account for each bank CSV and every detected card |
|
|
|
|
statement section that needs mapping. |
|
|
|
|
5. The owner selects **Import ready files**. |
|
|
|
|
6. Each ready file completes its own preview, automatic-decision, and commit |
|
|
|
|
lifecycle. The page reports created, matched, and skipped counts. |
|
|
|
|
6. Each ready file moves through staging, preview, automatic-decision, and |
|
|
|
|
commit. A committed file moves to processed. The page reports created, |
|
|
|
|
matched, and skipped counts. |
|
|
|
|
7. Files needing attention remain uncommitted. The owner corrects the account |
|
|
|
|
choice or required context, rescans, and imports when the file becomes |
|
|
|
|
ready. |
|
|
|
|
|
|
|
|
|
## Scope and file handling |
|
|
|
|
|
|
|
|
|
The scanner examines only direct files under `IMPORT_ROOT`, never follows |
|
|
|
|
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It |
|
|
|
|
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. |
|
|
|
|
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe |
|
|
|
|
exceptions and never parsed or committed. |
|
|
|
|
`IMPORT_ROOT` has three local, ignored folders: incoming files at `imports/`, |
|
|
|
|
temporary work at `imports/staging/`, and successful source documents at |
|
|
|
|
`imports/processed/`. The scanner examines only direct files in the incoming |
|
|
|
|
folder; it has no fixed file-count limit. It never follows symlinks, rejects |
|
|
|
|
traversal, and applies the existing 10 MiB input limit. It supports only the |
|
|
|
|
existing Banco do Brasil bank CSV and Ourocard TXT formats. OFX, PDFs, unknown |
|
|
|
|
extensions, and unrecognized layouts are reported as safe exceptions and never |
|
|
|
|
parsed or committed. |
|
|
|
|
|
|
|
|
|
Before each scan, Gizmoduck moves direct regular files left in `staging/` back |
|
|
|
|
to incoming. This recovers a run interrupted between staging and its final |
|
|
|
|
move. Neither `staging/` nor `processed/` is included in normal candidate |
|
|
|
|
scanning. |
|
|
|
|
|
|
|
|
|
The scanner retains no source rows, descriptions, amounts, or review exports in |
|
|
|
|
templates, messages, or logs. Those remain local source material governed by |
|
|
|
|
@ -63,6 +73,22 @@ Every file retains the import service's atomic commit boundary; a failed file
|
|
|
|
|
creates no partial transactions or provenance. All import lifecycle actions use |
|
|
|
|
the local human actor and retain existing audits. |
|
|
|
|
|
|
|
|
|
## Local file lifecycle |
|
|
|
|
|
|
|
|
|
Before creating a preview, Gizmoduck atomically moves each selected incoming |
|
|
|
|
file to `staging/` on the same local volume. It then reads and verifies the |
|
|
|
|
staged file, prepares it through the existing import lifecycle, and commits it. |
|
|
|
|
After a successful commit, it moves the staged source file to `processed/`. |
|
|
|
|
An existing processed file with the same name is overwritten only at this final |
|
|
|
|
successful step. |
|
|
|
|
|
|
|
|
|
If preparation, validation, duplicate review, commit, or a pre-commit file |
|
|
|
|
operation fails, Gizmoduck moves the staged file back to incoming and reports a |
|
|
|
|
safe exception. If the process stops after a database commit but before the |
|
|
|
|
processed move, the next scan restores that file to incoming. Existing |
|
|
|
|
provenance then prevents duplicate transactions; the automatic exact-source |
|
|
|
|
skip path may move it to processed. |
|
|
|
|
|
|
|
|
|
## Account mapping |
|
|
|
|
|
|
|
|
|
The scan presents an account choice for every source section needed by the |
|
|
|
|
@ -104,12 +130,12 @@ context before invoking the existing parser contract.
|
|
|
|
|
|
|
|
|
|
## Error handling and security |
|
|
|
|
|
|
|
|
|
All POST actions use CSRF protection and revalidate scan tokens, file identity, |
|
|
|
|
profile, mappings, and source metadata before creating a preview or committing. |
|
|
|
|
The UI returns field errors and safe status codes. It does not reveal source |
|
|
|
|
text, paths outside the allowed folder, raw parser exceptions, or database |
|
|
|
|
errors. Stale scans require a new scan; the server does not reuse a changed |
|
|
|
|
file under an old readiness result. |
|
|
|
|
All POST actions use CSRF protection and revalidate file identity, profile, |
|
|
|
|
mappings, and source metadata before creating a preview or committing. The UI |
|
|
|
|
returns field errors and safe status codes. It does not reveal source text, |
|
|
|
|
paths outside the allowed folder, raw parser exceptions, or database errors. |
|
|
|
|
Staging identity is verified before every lifecycle step; a changed staged file |
|
|
|
|
returns to incoming without commitment. |
|
|
|
|
|
|
|
|
|
## Verification |
|
|
|
|
|
|
|
|
|
@ -117,12 +143,15 @@ Tests use only synthetic fixtures and cover:
|
|
|
|
|
|
|
|
|
|
- direct-folder scanning and supported-profile detection; |
|
|
|
|
- rejected path/symlink/oversize/unsupported input without source disclosure; |
|
|
|
|
- staged success moving to processed, exception return to incoming, interrupted |
|
|
|
|
staging recovery, and processed-name overwrite only after a successful |
|
|
|
|
commit; |
|
|
|
|
- bank and multi-section card account mapping with active BRL references; |
|
|
|
|
- clean automatic creation through the existing audit and import lifecycle; |
|
|
|
|
- exact reimport skipping through source provenance; |
|
|
|
|
- an exceptional file not blocking an unrelated ready file; |
|
|
|
|
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, |
|
|
|
|
and CSRF rejection; |
|
|
|
|
- ambiguous duplicates, malformed candidates, mapping tampering, and CSRF |
|
|
|
|
rejection; |
|
|
|
|
- line-21 partial-statement date extraction, invalid context, and New Year |
|
|
|
|
rollover; and |
|
|
|
|
- focused web tests plus full Django, migration, permission, and integration |
|
|
|
|
|