gizmoduck/docs/superpowers/specs/2026-09-06-reviewed-imports-design.md

Reviewed Imports Design

Date: 2026-09-06 Status: Approved Milestone: Roadmap step 4

Goal and delivery surface

Import supported bank-account CSV and Banco do Brasil/Ourocard TXT through separate parsing, normalization, validation, duplicate detection, preview, and atomic commit stages. Use synthetic fixtures and the existing financial services.

Recommend a local management-command workflow for this milestone: persist a preview in PostgreSQL, export its review document into ignored local storage, then commit explicit row decisions. This delivers a usable review boundary without introducing authentication, upload endpoints, and product screens before roadmap step 5. A web wizard is a viable alternative but expands this milestone into UI and session-security design. An automatic import command would not meet the required review-before-commit behavior.

Source evidence and supported profiles

Inspected code cells only in Robopato's account and credit-card ETL notebooks. No notebook output or financial records are copied into this repository. Its parsers establish these initial compatibility profiles:

• Account CSV: Latin-1 text, comma delimiter, a header followed by records with at least six columns. Column 0 is DD/MM/YYYY, column 2 is description, column 5 is a signed decimal with a decimal point. CSV quoting must work correctly. Recognized S A L D O balance rows are excluded with an explicit summary count.
• Full Ourocard TXT: DD.MM.YYYY followed by fixed-width description/location fields (37 characters in the regular profile), two-character country field, and two Brazilian-formatted monetary columns. Installment markers use PARC n/total. The known PGTO DEBITO CONTA payment line has a separate layout.
• Partial Ourocard TXT: DD/MM, 43-character description/location field, two-character country, and two Brazilian-formatted monetary columns.
• Owner/section headers delimit source sections. Store an ordinal section ID for mapping; do not depend on owner-name database lookups.

These are explicit versioned compatibility profiles, not a claim to support all bank exports. Validate expected CSV header width and row shapes rather than guessing column mappings. Exact accepted header aliases and TXT grammar will be documented alongside synthetic fixtures during implementation. The parser must fail closed on date-like or transaction-like lines it cannot classify. Known section headings, blank lines, and recognized totals may be skipped with counts; other unrecognized nonblank text is surfaced for acknowledgement.

Decode UTF-8 (including BOM) first, then Latin-1. Record the selected encoding; never use replacement-character decoding. Reject NUL bytes and oversized input (over 10 MiB or 20,000 candidate rows). Preserve physical line/item locators. No OFX, PDF parsing, network fetches, bank APIs, or pandas dependency.

Context, dates, and monetary interpretation

Caller selects the parser profile and destination account; formats must match account kind. Bank and Ourocard profiles in this milestone post BRL amounts only. Reject other account currencies, because neither profile supplies a reliable conversion contract. The domain still supports other currencies for manual activity.

Full dates are used as written. For yearless lines require an explicit inclusive start/end context of at most 366 days; resolve each DD/MM only if exactly one valid date lies in that interval. Missing/ambiguous dates block commitment. Never use the current year. Source transaction dates become occurrence and posting dates for these initial profiles; statement association does not replace individual dates. Show this convention in preview.

CSV amounts retain their signs. In regular TXT rows the first monetary column is debit and the second credit: normalized amount = credit - debit using exact Decimal operations independent of caller context. Negative debit represents a credit adjustment. Nonzero values in both columns require review as an ambiguous row rather than silently netting. Known payment lines map the payment amount to a positive unmatched TRANSFER leg; no artificial bank leg is created. Regular card credits are proposed REFUND, bank credits INCOME, and debits EXPENSE.

Source labels are proposals: a reviewer may classify permitted rows as unmatched TRANSFER or REFUND, subject to amount sign and domain validation. Do not infer a transfer from a merchant-like keyword. Balance/previous-balance totals never become expenses or opening entries automatically.

Pure parser and normalized candidate contracts

Create an ingestion Django application; avoid naming a source package imports because that name is reserved for ignored local financial files.

Parsers take bytes plus explicitly supplied date context and return immutable parsed items/issues. They do no database or filesystem access. Normalization converts parsed items into candidates containing source locator, section ordinal, posting/occurrence dates, original description, signed decimal string, proposed kind, optional installment number/count, and parser version. Candidate IDs are stable per document/locator. Parser issues have code, severity, and locator, with no raw line in generic diagnostic messages.

Validation resolves account/reference IDs against existing domain rules and returns field-level issues. Duplicate detection consumes validated candidates and database snapshots through a separate boundary. Preview persistence records results without creating Transaction, Transfer, SourceRecord, or InstallmentPlan rows. Preview audit events and staging records are allowed.

Persisted review lifecycle

ImportBatch: UUID, document SHA-256, format/profile/version, context, selected account/section mapping, actor identity, creation timestamp, state (PREVIEW, COMMITTED, CANCELLED), monotonically increasing revision, and commit result. ImportItem: batch, unique physical locator, normalized candidate payload, validation issues, duplicate suggestions, and explicit decision metadata.

Staged descriptions and review documents are financial data: keep them only in local PostgreSQL and ignored local files. Do not store uploaded bytes in the batch or logs. CLI reads source documents from ignored imports/; only the management service mounts that directory, not the web container. Commit uses the immutable stored candidates, not a re-read of a mutable source file.

A review document is JSON with batch UUID/revision and decisions keyed by item locator. It may contain sensitive descriptions for local human inspection. Reject unknown keys, duplicate locators, foreign-batch items, and stale revisions. Review decisions may specify kind, category, payee, statement, refund reference, and installment plan mapping; they cannot rewrite amount, date, or source text. An incorrect amount/date requires a corrected source/new preview or later manual correction workflow, not an untraceable edit to imported evidence.

Updating decisions increments revision and recomputes preview validation. Every importable row requires an explicit decision. Cancelled batches cannot be committed. Committing an already committed batch returns the recorded result, without new transactions or provenance; retry arguments must refer to that batch.

Duplicate handling and review decisions

Exact physical source identity remains (document hash, item locator). An already attached source item is shown as ALREADY_IMPORTED with its existing transaction; it cannot be created or attached again, even through a new batch or parser version. If it belongs to another account, show a blocking account conflict.

Possible duplicates use the same account, currency, posting date, exact amount, normalized description (Unicode NFC, casefold, collapsed whitespace), kind, and installment number/count when present. Compare both existing records and items within the batch. Do not impose a semantic uniqueness constraint: two legitimate purchases can share this signature.

Allowed decisions:

• CREATE: record a new transaction. Possible duplicates require an explicit acknowledgement flag; default preview never silently deduplicates them.
• MATCH: attach this source item to a specifically selected existing transaction, only when its immutable financial fields and duplicate signature still match. No new transaction is created. Do not mutate the existing record's annotations.
• SKIP: explicitly omit a row. Require a reason code, not a free-form sensitive diagnostic. A skipped source remains unclaimed and can be reviewed in a future batch. Already-imported rows are skipped with an explicit existing-source reason.

Malformed transaction rows block commit until explicitly skipped with a reason; file-level corruption or unsupported profile cannot be overridden. Unrecognized nontransaction lines require acknowledgement. Do not suppress skipped/error counts from the review result.

Installments, refunds, and multiple sections

Do not guess installment identity from merchant and amount. Rows with installment metadata require an explicit existing plan or a named new-plan decision scoped to this batch, with account and total count validated. Create new plans only in the commit transaction, reuse the chosen plan across selected rows, and never create unobserved installments. Do not use bank periodic-transfer text to infer an expense installment automatically; show metadata as a review hint.

Refunds may remain unlinked or explicitly reference a compatible expense. Statements may be selected from existing records but are not auto-created from incomplete header data. A source with multiple owner sections must map each section explicitly to an account or mark it skipped. The same card account can receive multiple sections. Never use a person's name as an automatic account key.

Commit integrity and concurrency

Use one outermost transaction for all accepted rows, provenance, chosen new plans, batch status and success audit. Reuse internal domain validation/posting helpers, not nested public service transactions. A rejected commit rolls back all financial writes and records a sanitized failure audit after rollback; PREVIEW remains available for correction.

Lock the batch, then all affected accounts in UUID order; lock reference rows consistently before target transactions. Revalidate archive state, relationships, source identity and duplicate suggestions under locks immediately before writes. Lock existing candidate-match transactions before comparing annotations. If live state differs materially from the reviewed snapshot, return STALE_PREVIEW and require a refreshed preview/review rather than silently changing a decision. Account locks serialize commits and ordinary postings on the same account. Database source uniqueness resolves identical-file races across accounts.

Ensure the multi-row commit uses one consistent lock order across the entire batch, not each row independently. Plan creation, source attachment and auditing must leave no partial records if a later row fails. Only transaction UUIDs, counts, operation and error codes enter generic completion messages.

Commands and acceptance

Provide management commands to create/export a preview, apply a decision file, refresh a preview, commit a specific batch/revision, and cancel a batch. Commands require explicit actor kind/identifier and account/section mapping. Document PowerShell examples using the management container and ignored local import files. No real financial records are included in the examples.

Test with synthetic files representing quoting, encodings, invalid dates/money, full/partial card layouts, year boundaries, payments, refunds, balance lines, multiple sections, installments, and unknown lines. Verify preview creates no financial rows, cancellation is inert, skipping is explicit, exact reimport is idempotent, semantic duplicates remain reviewable, and MATCH preserves history.

PostgreSQL tests must cover a failing later row rolling back the entire batch, concurrent identical imports, stale review after other writes, archive changes, repeated commit, source conflicts, and failure audits. Verify new table grants, fresh/existing migrations, all financial-domain regressions, and app health.