gizmoduck/docs/superpowers/specs/2026-09-09-guided-local-import-design.md

Guided Local Import Design

Purpose

Let the local owner place supported financial documents in the ignored imports/ directory and load clean, new transactions through the web app with minimal interaction. The owner scans the folder, selects destination accounts where needed, imports all ready files, and receives safe status summaries.

The workflow preserves Gizmoduck's existing review boundaries: parsing, normalization, validation, duplicate detection, preview, decisions, and persistence remain distinct. The UI orchestrates those boundaries; it does not replace them or expose source contents in browser messages.

User workflow

1. The owner copies supported files to the ignored imports/ directory.
2. The owner opens Imports and selects Scan imports.
3. Gizmoduck returns interrupted staging files to the incoming folder, then lists each safe candidate filename, detected profile, account mapping requirements, row counts, and a safe status.
4. The owner selects an account for each bank CSV and every detected card statement section that needs mapping.
5. The owner selects Import ready files.
6. Each ready file moves through staging, preview, automatic-decision, and commit. A committed file moves to processed. The page reports created, matched, and skipped counts.
7. Files needing attention remain uncommitted. The owner corrects the account choice or required context, rescans, and imports when the file becomes ready.

Scope and file handling

IMPORT_ROOT has three local, ignored folders: incoming files at imports/, temporary work at imports/staging/, and successful source documents at imports/processed/. The scanner examines only direct files in the incoming folder; it has no fixed file-count limit. It never follows symlinks, rejects traversal, and applies the existing 10 MiB input limit. It supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe exceptions and never parsed or committed.

Before each scan, Gizmoduck moves direct regular files left in staging/ back to incoming. This recovers a run interrupted between staging and its final move. Neither staging/ nor processed/ is included in normal candidate scanning.

The scanner retains no source rows, descriptions, amounts, or review exports in templates, messages, or logs. Those remain local source material governed by the data policy. The web UI displays filenames, profile labels, account mapping state, safe reason codes, and counts only.

Readiness and automatic decisions

Each scanned file has one of these statuses:

• Ready: every candidate can be created without a validation issue or unresolved duplicate.
• Already imported: every candidate is already attached through existing provenance or exactly matches a persisted transaction signature; the file is skipped without financial writes.
• Needs attention: the file has an unsupported/unrecognized profile, malformed or unresolved candidate, missing account mapping, ambiguous potential duplicate, or missing/invalid statement context.

The import action creates a normal existing preview per ready file. It applies CREATE to clean candidates. Exact source reimports and candidates with an exact persisted transaction signature receive SKIP with ALREADY_IMPORTED; they produce no transaction and are not an exception. Duplicates within the incoming file still require review and prevent that file from committing automatically.

An exception in one file does not prevent other ready files from committing. Every file retains the import service's atomic commit boundary; a failed file creates no partial transactions or provenance. All import lifecycle actions use the local human actor and retain existing audits.

Local file lifecycle

Before creating a preview, Gizmoduck atomically moves each selected incoming file to staging/ on the same local volume. It then reads and verifies the staged file, prepares it through the existing import lifecycle, and commits it. After a successful commit, it moves the staged source file to processed/. An existing processed file with the same name is overwritten only at this final successful step.

If preparation, validation, duplicate review, commit, or a pre-commit file operation fails, Gizmoduck moves the staged file back to incoming and reports a safe exception. If the process stops after a database commit but before the processed move, the next scan restores that file to incoming. Existing provenance then prevents duplicate transactions; the automatic exact-source skip path may move it to processed.

Account mapping

The scan presents an account choice for every source section needed by the detected profile. Bank CSV has one section. Ourocard documents may have multiple sections and require a choice for each section. The UI uses active BRL accounts and cards only. It never maps an owner name or filename to an account automatically.

The server validates that every submitted mapping corresponds to the current scan and to an active account. Missing, archived, stale, or altered mappings are exceptions; they cannot be committed.

Ourocard partial-statement dates

OUROCARD_PARTIAL_V1 gains a bounded context extractor. It reads and validates the statement creation date from line 21 of the TXT document. That date supplies the statement year and an explicit import interval. Candidate month/day values later than the creation month belong to the preceding calendar year; remaining values belong to the creation year. This handles statements crossing New Year without using the computer clock or filesystem metadata.

If line 21 is absent, malformed, ambiguous, or does not yield an interval that resolves every candidate date uniquely, the file is Needs attention and is not committed. The existing explicit parser context remains available for the management command; this UI-only extractor supplies equivalent validated context before invoking the existing parser contract.

Components

• ingestion receives a folder scanner, safe profile detector, and a UI-orchestration service that constructs existing previews and applies only the approved automatic decisions.
• core receives Import navigation, scan/mapping/import forms and views, and server-rendered status templates.
• Existing parser, normalization, validation, duplicate, persistence, audit, and import-batch models remain their own boundaries.
• The management command remains supported for advanced/manual reviewed import workflows.

Error handling and security

All POST actions use CSRF protection and revalidate file identity, profile, mappings, and source metadata before creating a preview or committing. The UI returns field errors and safe status codes. It does not reveal source text, paths outside the allowed folder, raw parser exceptions, or database errors. Staging identity is verified before every lifecycle step; a changed staged file returns to incoming without commitment.

Verification

Tests use only synthetic fixtures and cover:

• direct-folder scanning and supported-profile detection;
• rejected path/symlink/oversize/unsupported input without source disclosure;
• staged success moving to processed, exception return to incoming, interrupted staging recovery, and processed-name overwrite only after a successful commit;
• bank and multi-section card account mapping with active BRL references;
• clean automatic creation through the existing audit and import lifecycle;
• exact reimport skipping through source provenance;
• an exceptional file not blocking an unrelated ready file;
• ambiguous duplicates, malformed candidates, mapping tampering, and CSRF rejection;
• line-21 partial-statement date extraction, invalid context, and New Year rollover; and
• focused web tests plus full Django, migration, permission, and integration checks.