4.5 KiB
Reviewed Imports Implementation Plan
Spec: Approved design Goal: Ship pure format parsing, persisted previews, explicit decisions, and atomic audited imports through local commands. Architecture: ingestion/parsers.py + normalization.py -> validation.py + duplicates.py -> services.py lifecycle -> management command. Finance internal posting helpers are reused inside one outer transaction.
Interfaces
Profiles BANK_CSV_V1, OUROCARD_FULL_V1, OUROCARD_PARTIAL_V1. parse_document(data: bytes, profile: str, context: dict | None) returns immutable ParseResult(encoding,profile,items,skipped) or raises ParseError(code). normalize(result) returns tuple of immutable Candidate with to_dict(): locator, section, posting_date ISO or null, occurrence_date ISO or null, description, amount decimal string or null, kind, installment_number, installment_total, errors list of safe codes.
Public services require actor=finance.services.Actor: create_preview(data,profile,section_mapping,context=None) -> ImportBatch; export_review(batch_id) -> dict; apply_decisions(batch_id,revision,decisions) -> ImportBatch; refresh_preview(batch_id,revision) -> ImportBatch; commit_preview(batch_id,revision) -> ImportBatch; cancel_preview(batch_id,revision) -> ImportBatch. DomainError.code for failures. Decisions list entries: locator + action CREATE/MATCH/SKIP, optional acknowledge_duplicate bool, transaction_id for MATCH, reason for SKIP (USER_EXCLUDED, MALFORMED, UNRECOGNIZED, ALREADY_IMPORTED, SECTION_EXCLUDED), kind/category_id/payee_id/statement_id/refund_of_id/installment_plan_id/new_plan for CREATE. new_plan is a batch-scoped string name; count/account derived from candidate. section_mapping dict keys numeric ordinal strings (default section '1'), values account UUID strings or null (skip section explicitly). Multi-section no implicit mapping. Export JSON: batch_id, revision, state, profile, encoding, summary, items (locator,candidate,issues,duplicates,decision), decisions list template. Command decision file accepts ONLY batch_id/revision/decisions, or entire exported review with read-only metadata verified against current export.
- Task 1: Pure parser/normalization, synthetic fixtures and edge tests. Owned parser files and tests/test_import_parsers.py.
- Task 2: Batch/item models, duplicate/validation snapshots and lifecycle services. Persist no finance objects until commit. Revalidate locked snapshots and revision; preserve idempotency and independent failure audits.
- Task 3: Management command and service/concurrency tests; demonstrate explicit review/edit/apply/commit with synthetic inputs.
- Task 4: Docker/settings/migrations/runtime grants, full PostgreSQL verification, fresh/existing DB migration, review and developer docs.
Ruling: creation rejects file-level parser errors before staging; malformed individual rows remain reviewable but require an explicit SKIP. Section mappings may be corrected by creating a new preview; candidate bytes/context are immutable. Ruling: refresh increments revision and clears all decisions, requiring renewed consent after live state changes. Applying decisions replaces the complete decision set, never incrementally changes implicit defaults. Ruling: successful repeated commit returns the recorded batch without a second success audit; latest committed revision must match the request.
Completion evidence
- All 138 tests passed against PostgreSQL. Coverage includes the three parser profiles, malformed and unknown lines, immutable preview evidence, explicit decisions, duplicate matching, rollback, stale reviews, and concurrent commits.
- Parser reviewability regressions failed before unknown/malformed source labels were retained in local candidates, then passed after the fix. Generic issue codes remain free of source text.
- The ingestion migration applied to both the existing database and a fresh
gizmoduck-import-verifydatabase. Django system checks, migration drift, and pending-migration checks passed. - Runtime DML and sequence grants passed for both finance and ingestion tables; DDL and role escalation remained denied.
- The documented command flow ran end to end on the isolated database: preview, export, edit, apply, commit, and repeated idempotent commit. Exactly one transaction and one provenance record were created from synthetic input.
- The rehearsal files were removed afterward. The isolated verification database container is stopped and its synthetic volume is retained.