Codex
2 months ago
23 changed files with 2014 additions and 0 deletions
@ -0,0 +1,165 @@ |
|||||||
|
# Reviewed local imports |
||||||
|
|
||||||
|
Imports require a saved preview and explicit row decisions before financial |
||||||
|
records are written. The `import_review` management command supports `preview`, |
||||||
|
`export`, `apply`, `refresh`, `commit`, and `cancel`. Every operation requires |
||||||
|
`--actor-kind HUMAN|APPLICATION|AGENT` and `--actor-identifier`. |
||||||
|
|
||||||
|
Keep source documents, mappings and review JSON in the ignored repository-root |
||||||
|
`imports/` directory. The management container mounts it at `/app/imports`. |
||||||
|
Command paths may be absolute under that root or relative to it; traversal and |
||||||
|
symlink escapes are rejected. Output files must not already exist: choose a new |
||||||
|
filename for every export. Input documents and JSON files are limited to 10 MiB. |
||||||
|
Review files contain financial descriptions and must never enter Git, logs or |
||||||
|
external services. See [data policy](data-policy.md). |
||||||
|
|
||||||
|
## PowerShell workflow |
||||||
|
|
||||||
|
Run the setup and migrations in [development](development.md) first. Create the |
||||||
|
destination bank account or card through the existing financial services; use |
||||||
|
its UUID below. These names and UUIDs are illustrative, synthetic examples. |
||||||
|
|
||||||
|
```powershell |
||||||
|
New-Item -ItemType Directory -Force imports | Out-Null |
||||||
|
git check-ignore imports/source.csv imports/mapping.json imports/review-1.json |
||||||
|
@' |
||||||
|
{"1":"11111111-1111-4111-8111-111111111111"} |
||||||
|
'@ | Set-Content -Encoding utf8 imports/mapping.json |
||||||
|
$actor = @('--actor-kind', 'HUMAN', '--actor-identifier', 'local-owner') |
||||||
|
docker compose run --rm manage import_review preview ` |
||||||
|
--input source.csv --profile BANK_CSV_V1 --mapping mapping.json @actor |
||||||
|
``` |
||||||
|
|
||||||
|
Place the source CSV in `imports/source.csv` before previewing it. Substitute |
||||||
|
the batch UUID printed by `preview`, then export its review: |
||||||
|
|
||||||
|
```powershell |
||||||
|
$batch = '22222222-2222-4222-8222-222222222222' |
||||||
|
docker compose run --rm manage import_review export ` |
||||||
|
--batch $batch --output review-1.json @actor |
||||||
|
notepad imports/review-1.json |
||||||
|
``` |
||||||
|
|
||||||
|
Read `items`, including candidate amounts/dates, validation issues and duplicate |
||||||
|
suggestions. Edit only the top-level `decisions` array. Preserve every other |
||||||
|
field, including each item's stored `decision`. Alternatively save a minimal |
||||||
|
object with exactly `batch_id`, integer `revision`, and `decisions`. Unknown |
||||||
|
keys, repeated JSON keys, repeated locators, foreign batch items, stale |
||||||
|
revisions and changes to source evidence are rejected. |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"batch_id": "22222222-2222-4222-8222-222222222222", |
||||||
|
"revision": 1, |
||||||
|
"decisions": [ |
||||||
|
{"locator": "line:2", "action": "CREATE"}, |
||||||
|
{"locator": "line:3", "action": "SKIP", "reason": "USER_EXCLUDED"} |
||||||
|
] |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
Use the actual locators in the exported file. Apply the complete decision set, |
||||||
|
then export again to inspect the resulting issues and latest revision: |
||||||
|
|
||||||
|
```powershell |
||||||
|
docker compose run --rm manage import_review apply ` |
||||||
|
--batch $batch --input review-1.json @actor |
||||||
|
docker compose run --rm manage import_review export ` |
||||||
|
--batch $batch --output review-2.json @actor |
||||||
|
notepad imports/review-2.json |
||||||
|
$revision = 2 # Replace with the revision printed by apply/export. |
||||||
|
docker compose run --rm manage import_review commit ` |
||||||
|
--batch $batch --revision $revision @actor |
||||||
|
``` |
||||||
|
|
||||||
|
Only `commit` writes financial transactions, source provenance and new |
||||||
|
installment plans. It posts all accepted rows atomically. Repeating a successful |
||||||
|
commit with its latest revision returns the recorded result. If live account, |
||||||
|
reference or duplicate information has materially changed, refresh and review |
||||||
|
again. Refresh increments the revision and clears all decisions: |
||||||
|
|
||||||
|
```powershell |
||||||
|
docker compose run --rm manage import_review refresh ` |
||||||
|
--batch $batch --revision $revision @actor |
||||||
|
docker compose run --rm manage import_review export ` |
||||||
|
--batch $batch --output review-refreshed.json @actor |
||||||
|
``` |
||||||
|
|
||||||
|
To abandon a preview, use its current revision: |
||||||
|
|
||||||
|
```powershell |
||||||
|
docker compose run --rm manage import_review cancel ` |
||||||
|
--batch $batch --revision $revision @actor |
||||||
|
``` |
||||||
|
|
||||||
|
Cancellation creates no financial entries and prevents later commitment. |
||||||
|
Terminal output contains batch identifiers, state, revision and counts; inspect |
||||||
|
the local review file for financial details. Errors expose only safe codes. |
||||||
|
|
||||||
|
## Decisions and duplicate handling |
||||||
|
|
||||||
|
Every candidate needs one explicit decision. Applying a file replaces the |
||||||
|
complete decision set. The supported actions are: |
||||||
|
|
||||||
|
- `CREATE`: create a transaction. Optional annotations are `kind`, `category_id`, |
||||||
|
`payee_id`, `statement_id`, `refund_of_id`, `installment_plan_id`, and `new_plan`. |
||||||
|
Possible duplicates require `"acknowledge_duplicate": true` after review. |
||||||
|
- `MATCH`: attach the source to the existing `transaction_id` selected from |
||||||
|
compatible duplicate suggestions. The immutable financial fields and signature |
||||||
|
must still agree. Existing annotations are preserved. |
||||||
|
- `SKIP`: omit the item with a `reason` code: `USER_EXCLUDED`, `MALFORMED`, |
||||||
|
`UNRECOGNIZED`, `ALREADY_IMPORTED`, or `SECTION_EXCLUDED`. Malformed rows and |
||||||
|
unknown lines require explicit skipping; file-level corruption cannot be |
||||||
|
overridden. Exact already-imported source items cannot be created or matched |
||||||
|
again; acknowledge them with `SKIP` / `ALREADY_IMPORTED`. |
||||||
|
|
||||||
|
For example, an installment decision can include |
||||||
|
`"new_plan": "synthetic-plan-a"`. This is a batch-scoped plan name, not an |
||||||
|
existing UUID; the account and installment count come from the candidate. |
||||||
|
Reuse the same name only for rows belonging to the same plan, or select an |
||||||
|
existing `installment_plan_id`. Plan identity is never inferred from merchant |
||||||
|
names, and missing installments are never generated. Refunds can remain unlinked |
||||||
|
or use an explicit compatible `refund_of_id`. No missing statements or transfer |
||||||
|
counterparts are created automatically. |
||||||
|
|
||||||
|
Section mapping JSON maps ordinal string keys such as `"1"` and `"2"` to account |
||||||
|
UUIDs, or `null` to exclude a section. Map every section explicitly; several |
||||||
|
sections may share one account. Owner names are never database lookup keys. |
||||||
|
Correct mappings or immutable dates/amounts by creating a new preview from a |
||||||
|
corrected input. They cannot be rewritten in decisions. |
||||||
|
|
||||||
|
## Supported compatibility profiles |
||||||
|
|
||||||
|
All profiles decode UTF-8 (including BOM), then Latin-1 without replacement |
||||||
|
characters. NUL bytes, files over 10 MiB and more than 20,000 candidates are |
||||||
|
rejected. Initial profiles require BRL destination accounts. Source dates become |
||||||
|
both occurrence and posting dates. OFX and PDF are unsupported. |
||||||
|
|
||||||
|
| Profile | Accepted layout | |
||||||
|
| --- | --- | |
||||||
|
| `BANK_CSV_V1` | Bank account CSV, comma-delimited with CSV quoting, at least six header columns; date at column 0 as `DD/MM/YYYY`, description at 2, signed decimal-point amount at 5. Each record must match the header width. | |
||||||
|
| `OUROCARD_FULL_V1` | Card TXT, `DD.MM.YYYY`, 37-character description/location field, two-character country field, debit and credit in Brazilian decimal notation. A separate `PGTO DEBITO CONTA` layout is recognized. | |
||||||
|
| `OUROCARD_PARTIAL_V1` | Card TXT, `DD/MM`, 43-character description/location field, country, debit and credit; requires an explicit date interval. | |
||||||
|
|
||||||
|
CSV header aliases are case/accent insensitive: date accepts `Data`, |
||||||
|
`Data movimento`, `Data do movimento`, or `Data lancamento`; description accepts |
||||||
|
`Historico`, `Descricao`, or `Descricao do lancamento`; amount accepts `Valor`, |
||||||
|
`Valor lancamento`, or `Valor do lancamento`. Other columns are retained only as |
||||||
|
layout positions. Recognized balance rows are excluded and counted. |
||||||
|
|
||||||
|
Card amounts use credit minus debit. Nonzero amounts in both columns block |
||||||
|
creation pending explicit review; known account-debit payment lines propose an |
||||||
|
unmatched positive transfer. Card credits propose refunds, bank credits income, |
||||||
|
and debits expenses. `PARC n/total` supplies installment metadata. Section |
||||||
|
headings establish ordinal mappings; unknown nonblank text remains visible for |
||||||
|
acknowledgement, and transaction-like malformed lines are never silently lost. |
||||||
|
|
||||||
|
Yearless dates require both `--context-start` and `--context-end` as ISO dates |
||||||
|
in an inclusive interval of at most 366 days. A date must resolve uniquely; |
||||||
|
the importer never guesses the current year. For example: |
||||||
|
|
||||||
|
```powershell |
||||||
|
docker compose run --rm manage import_review preview ` |
||||||
|
--input card.txt --profile OUROCARD_PARTIAL_V1 --mapping card-mapping.json ` |
||||||
|
--context-start 2025-12-01 --context-end 2026-01-31 @actor |
||||||
|
``` |
||||||
@ -0,0 +1,48 @@ |
|||||||
|
# Reviewed Imports Implementation Plan |
||||||
|
|
||||||
|
**Spec:** [Approved design](../specs/2026-09-06-reviewed-imports-design.md) |
||||||
|
**Goal:** Ship pure format parsing, persisted previews, explicit decisions, and atomic audited imports through local commands. |
||||||
|
**Architecture:** ingestion/parsers.py + normalization.py -> validation.py + duplicates.py -> services.py lifecycle -> management command. Finance internal posting helpers are reused inside one outer transaction. |
||||||
|
|
||||||
|
## Interfaces |
||||||
|
|
||||||
|
Profiles BANK_CSV_V1, OUROCARD_FULL_V1, OUROCARD_PARTIAL_V1. |
||||||
|
parse_document(data: bytes, profile: str, context: dict | None) returns immutable ParseResult(encoding,profile,items,skipped) or raises ParseError(code). |
||||||
|
normalize(result) returns tuple of immutable Candidate with to_dict(): locator, section, posting_date ISO or null, occurrence_date ISO or null, description, amount decimal string or null, kind, installment_number, installment_total, errors list of safe codes. |
||||||
|
|
||||||
|
Public services require actor=finance.services.Actor: |
||||||
|
create_preview(data,profile,section_mapping,context=None) -> ImportBatch; |
||||||
|
export_review(batch_id) -> dict; apply_decisions(batch_id,revision,decisions) -> ImportBatch; |
||||||
|
refresh_preview(batch_id,revision) -> ImportBatch; commit_preview(batch_id,revision) -> ImportBatch; |
||||||
|
cancel_preview(batch_id,revision) -> ImportBatch. DomainError.code for failures. |
||||||
|
Decisions list entries: locator + action CREATE/MATCH/SKIP, optional acknowledge_duplicate bool, transaction_id for MATCH, reason for SKIP (USER_EXCLUDED, MALFORMED, UNRECOGNIZED, ALREADY_IMPORTED, SECTION_EXCLUDED), kind/category_id/payee_id/statement_id/refund_of_id/installment_plan_id/new_plan for CREATE. new_plan is a batch-scoped string name; count/account derived from candidate. |
||||||
|
section_mapping dict keys numeric ordinal strings (default section '1'), values account UUID strings or null (skip section explicitly). Multi-section no implicit mapping. |
||||||
|
Export JSON: batch_id, revision, state, profile, encoding, summary, items (locator,candidate,issues,duplicates,decision), decisions list template. Command decision file accepts ONLY batch_id/revision/decisions, or entire exported review with read-only metadata verified against current export. |
||||||
|
|
||||||
|
- [x] Task 1: Pure parser/normalization, synthetic fixtures and edge tests. Owned parser files and tests/test_import_parsers.py. |
||||||
|
- [x] Task 2: Batch/item models, duplicate/validation snapshots and lifecycle services. Persist no finance objects until commit. Revalidate locked snapshots and revision; preserve idempotency and independent failure audits. |
||||||
|
- [x] Task 3: Management command and service/concurrency tests; demonstrate explicit review/edit/apply/commit with synthetic inputs. |
||||||
|
- [x] Task 4: Docker/settings/migrations/runtime grants, full PostgreSQL verification, fresh/existing DB migration, review and developer docs. |
||||||
|
|
||||||
|
Ruling: creation rejects file-level parser errors before staging; malformed individual rows remain reviewable but require an explicit SKIP. Section mappings may be corrected by creating a new preview; candidate bytes/context are immutable. |
||||||
|
Ruling: refresh increments revision and clears all decisions, requiring renewed consent after live state changes. Applying decisions replaces the complete decision set, never incrementally changes implicit defaults. |
||||||
|
Ruling: successful repeated commit returns the recorded batch without a second success audit; latest committed revision must match the request. |
||||||
|
|
||||||
|
## Completion evidence |
||||||
|
|
||||||
|
- All 138 tests passed against PostgreSQL. Coverage includes the three parser |
||||||
|
profiles, malformed and unknown lines, immutable preview evidence, explicit |
||||||
|
decisions, duplicate matching, rollback, stale reviews, and concurrent commits. |
||||||
|
- Parser reviewability regressions failed before unknown/malformed source labels |
||||||
|
were retained in local candidates, then passed after the fix. Generic issue |
||||||
|
codes remain free of source text. |
||||||
|
- The ingestion migration applied to both the existing database and a fresh |
||||||
|
`gizmoduck-import-verify` database. Django system checks, migration drift, and |
||||||
|
pending-migration checks passed. |
||||||
|
- Runtime DML and sequence grants passed for both finance and ingestion tables; |
||||||
|
DDL and role escalation remained denied. |
||||||
|
- The documented command flow ran end to end on the isolated database: preview, |
||||||
|
export, edit, apply, commit, and repeated idempotent commit. Exactly one |
||||||
|
transaction and one provenance record were created from synthetic input. |
||||||
|
- The rehearsal files were removed afterward. The isolated verification |
||||||
|
database container is stopped and its synthetic volume is retained. |
||||||
@ -0,0 +1,210 @@ |
|||||||
|
# Reviewed Imports Design |
||||||
|
|
||||||
|
**Date:** 2026-09-06 |
||||||
|
**Status:** Approved |
||||||
|
**Milestone:** Roadmap step 4 |
||||||
|
|
||||||
|
## Goal and delivery surface |
||||||
|
|
||||||
|
Import supported bank-account CSV and Banco do Brasil/Ourocard TXT through |
||||||
|
separate parsing, normalization, validation, duplicate detection, preview, and |
||||||
|
atomic commit stages. Use synthetic fixtures and the existing financial services. |
||||||
|
|
||||||
|
Recommend a local management-command workflow for this milestone: persist a |
||||||
|
preview in PostgreSQL, export its review document into ignored local storage, |
||||||
|
then commit explicit row decisions. This delivers a usable review boundary |
||||||
|
without introducing authentication, upload endpoints, and product screens before |
||||||
|
roadmap step 5. A web wizard is a viable alternative but expands this milestone |
||||||
|
into UI and session-security design. An automatic import command would not meet |
||||||
|
the required review-before-commit behavior. |
||||||
|
|
||||||
|
## Source evidence and supported profiles |
||||||
|
|
||||||
|
Inspected code cells only in Robopato's account and credit-card ETL notebooks. |
||||||
|
No notebook output or financial records are copied into this repository. Its |
||||||
|
parsers establish these initial compatibility profiles: |
||||||
|
|
||||||
|
- Account CSV: Latin-1 text, comma delimiter, a header followed by records with |
||||||
|
at least six columns. Column 0 is DD/MM/YYYY, column 2 is description, column 5 |
||||||
|
is a signed decimal with a decimal point. CSV quoting must work correctly. |
||||||
|
Recognized `S A L D O` balance rows are excluded with an explicit summary count. |
||||||
|
- Full Ourocard TXT: DD.MM.YYYY followed by fixed-width description/location |
||||||
|
fields (37 characters in the regular profile), two-character country field, |
||||||
|
and two Brazilian-formatted monetary columns. Installment markers use |
||||||
|
`PARC n/total`. The known `PGTO DEBITO CONTA` payment line has a separate layout. |
||||||
|
- Partial Ourocard TXT: DD/MM, 43-character description/location field, |
||||||
|
two-character country, and two Brazilian-formatted monetary columns. |
||||||
|
- Owner/section headers delimit source sections. Store an ordinal section ID |
||||||
|
for mapping; do not depend on owner-name database lookups. |
||||||
|
|
||||||
|
These are explicit versioned compatibility profiles, not a claim to support |
||||||
|
all bank exports. Validate expected CSV header width and row shapes rather than |
||||||
|
guessing column mappings. Exact accepted header aliases and TXT grammar will |
||||||
|
be documented alongside synthetic fixtures during implementation. The parser |
||||||
|
must fail closed on date-like or transaction-like lines it cannot classify. |
||||||
|
Known section headings, blank lines, and recognized totals may be skipped with |
||||||
|
counts; other unrecognized nonblank text is surfaced for acknowledgement. |
||||||
|
|
||||||
|
Decode UTF-8 (including BOM) first, then Latin-1. Record the selected encoding; |
||||||
|
never use replacement-character decoding. Reject NUL bytes and oversized input |
||||||
|
(over 10 MiB or 20,000 candidate rows). Preserve physical line/item locators. |
||||||
|
No OFX, PDF parsing, network fetches, bank APIs, or pandas dependency. |
||||||
|
|
||||||
|
## Context, dates, and monetary interpretation |
||||||
|
|
||||||
|
Caller selects the parser profile and destination account; formats must match |
||||||
|
account kind. Bank and Ourocard profiles in this milestone post BRL amounts |
||||||
|
only. Reject other account currencies, because neither profile supplies a |
||||||
|
reliable conversion contract. The domain still supports other currencies for |
||||||
|
manual activity. |
||||||
|
|
||||||
|
Full dates are used as written. For yearless lines require an explicit inclusive |
||||||
|
start/end context of at most 366 days; resolve each DD/MM only if exactly one |
||||||
|
valid date lies in that interval. Missing/ambiguous dates block commitment. |
||||||
|
Never use the current year. Source transaction dates become occurrence and |
||||||
|
posting dates for these initial profiles; statement association does not replace |
||||||
|
individual dates. Show this convention in preview. |
||||||
|
|
||||||
|
CSV amounts retain their signs. In regular TXT rows the first monetary column |
||||||
|
is debit and the second credit: normalized amount = credit - debit using exact |
||||||
|
Decimal operations independent of caller context. Negative debit represents a |
||||||
|
credit adjustment. Nonzero values in both columns require review as an ambiguous |
||||||
|
row rather than silently netting. Known payment lines map the payment amount to |
||||||
|
a positive unmatched TRANSFER leg; no artificial bank leg is created. Regular |
||||||
|
card credits are proposed REFUND, bank credits INCOME, and debits EXPENSE. |
||||||
|
|
||||||
|
Source labels are proposals: a reviewer may classify permitted rows as unmatched |
||||||
|
TRANSFER or REFUND, subject to amount sign and domain validation. Do not infer a |
||||||
|
transfer from a merchant-like keyword. Balance/previous-balance totals never |
||||||
|
become expenses or opening entries automatically. |
||||||
|
|
||||||
|
## Pure parser and normalized candidate contracts |
||||||
|
|
||||||
|
Create an `ingestion` Django application; avoid naming a source package `imports` |
||||||
|
because that name is reserved for ignored local financial files. |
||||||
|
|
||||||
|
Parsers take bytes plus explicitly supplied date context and return immutable |
||||||
|
parsed items/issues. They do no database or filesystem access. Normalization |
||||||
|
converts parsed items into candidates containing source locator, section ordinal, |
||||||
|
posting/occurrence dates, original description, signed decimal string, proposed |
||||||
|
kind, optional installment number/count, and parser version. Candidate IDs are |
||||||
|
stable per document/locator. Parser issues have code, severity, and locator, with |
||||||
|
no raw line in generic diagnostic messages. |
||||||
|
|
||||||
|
Validation resolves account/reference IDs against existing domain rules and |
||||||
|
returns field-level issues. Duplicate detection consumes validated candidates |
||||||
|
and database snapshots through a separate boundary. Preview persistence records |
||||||
|
results without creating Transaction, Transfer, SourceRecord, or InstallmentPlan |
||||||
|
rows. Preview audit events and staging records are allowed. |
||||||
|
|
||||||
|
## Persisted review lifecycle |
||||||
|
|
||||||
|
ImportBatch: UUID, document SHA-256, format/profile/version, context, selected |
||||||
|
account/section mapping, actor identity, creation timestamp, state (PREVIEW, |
||||||
|
COMMITTED, CANCELLED), monotonically increasing revision, and commit result. |
||||||
|
ImportItem: batch, unique physical locator, normalized candidate payload, |
||||||
|
validation issues, duplicate suggestions, and explicit decision metadata. |
||||||
|
|
||||||
|
Staged descriptions and review documents are financial data: keep them only in |
||||||
|
local PostgreSQL and ignored local files. Do not store uploaded bytes in the |
||||||
|
batch or logs. CLI reads source documents from ignored `imports/`; only the |
||||||
|
management service mounts that directory, not the web container. Commit uses |
||||||
|
the immutable stored candidates, not a re-read of a mutable source file. |
||||||
|
|
||||||
|
A review document is JSON with batch UUID/revision and decisions keyed by item |
||||||
|
locator. It may contain sensitive descriptions for local human inspection. |
||||||
|
Reject unknown keys, duplicate locators, foreign-batch items, and stale revisions. |
||||||
|
Review decisions may specify kind, category, payee, statement, refund reference, |
||||||
|
and installment plan mapping; they cannot rewrite amount, date, or source text. |
||||||
|
An incorrect amount/date requires a corrected source/new preview or later manual |
||||||
|
correction workflow, not an untraceable edit to imported evidence. |
||||||
|
|
||||||
|
Updating decisions increments revision and recomputes preview validation. |
||||||
|
Every importable row requires an explicit decision. Cancelled batches cannot be |
||||||
|
committed. Committing an already committed batch returns the recorded result, |
||||||
|
without new transactions or provenance; retry arguments must refer to that batch. |
||||||
|
|
||||||
|
## Duplicate handling and review decisions |
||||||
|
|
||||||
|
Exact physical source identity remains (document hash, item locator). An already |
||||||
|
attached source item is shown as ALREADY_IMPORTED with its existing transaction; |
||||||
|
it cannot be created or attached again, even through a new batch or parser version. |
||||||
|
If it belongs to another account, show a blocking account conflict. |
||||||
|
|
||||||
|
Possible duplicates use the same account, currency, posting date, exact amount, |
||||||
|
normalized description (Unicode NFC, casefold, collapsed whitespace), kind, |
||||||
|
and installment number/count when present. Compare both existing records and |
||||||
|
items within the batch. Do not impose a semantic uniqueness constraint: two |
||||||
|
legitimate purchases can share this signature. |
||||||
|
|
||||||
|
Allowed decisions: |
||||||
|
|
||||||
|
- CREATE: record a new transaction. Possible duplicates require an explicit |
||||||
|
acknowledgement flag; default preview never silently deduplicates them. |
||||||
|
- MATCH: attach this source item to a specifically selected existing transaction, |
||||||
|
only when its immutable financial fields and duplicate signature still match. |
||||||
|
No new transaction is created. Do not mutate the existing record's annotations. |
||||||
|
- SKIP: explicitly omit a row. Require a reason code, not a free-form sensitive |
||||||
|
diagnostic. A skipped source remains unclaimed and can be reviewed in a future |
||||||
|
batch. Already-imported rows are skipped with an explicit existing-source reason. |
||||||
|
|
||||||
|
Malformed transaction rows block commit until explicitly skipped with a reason; |
||||||
|
file-level corruption or unsupported profile cannot be overridden. Unrecognized |
||||||
|
nontransaction lines require acknowledgement. Do not suppress skipped/error |
||||||
|
counts from the review result. |
||||||
|
|
||||||
|
## Installments, refunds, and multiple sections |
||||||
|
|
||||||
|
Do not guess installment identity from merchant and amount. Rows with installment |
||||||
|
metadata require an explicit existing plan or a named new-plan decision scoped |
||||||
|
to this batch, with account and total count validated. Create new plans only in |
||||||
|
the commit transaction, reuse the chosen plan across selected rows, and never |
||||||
|
create unobserved installments. Do not use bank periodic-transfer text to infer |
||||||
|
an expense installment automatically; show metadata as a review hint. |
||||||
|
|
||||||
|
Refunds may remain unlinked or explicitly reference a compatible expense. |
||||||
|
Statements may be selected from existing records but are not auto-created from |
||||||
|
incomplete header data. A source with multiple owner sections must map each |
||||||
|
section explicitly to an account or mark it skipped. The same card account can |
||||||
|
receive multiple sections. Never use a person's name as an automatic account key. |
||||||
|
|
||||||
|
## Commit integrity and concurrency |
||||||
|
|
||||||
|
Use one outermost transaction for all accepted rows, provenance, chosen new |
||||||
|
plans, batch status and success audit. Reuse internal domain validation/posting |
||||||
|
helpers, not nested public service transactions. A rejected commit rolls back |
||||||
|
all financial writes and records a sanitized failure audit after rollback; |
||||||
|
PREVIEW remains available for correction. |
||||||
|
|
||||||
|
Lock the batch, then all affected accounts in UUID order; lock reference rows |
||||||
|
consistently before target transactions. Revalidate archive state, relationships, |
||||||
|
source identity and duplicate suggestions under locks immediately before writes. |
||||||
|
Lock existing candidate-match transactions before comparing annotations. If live |
||||||
|
state differs materially from the reviewed snapshot, return STALE_PREVIEW and |
||||||
|
require a refreshed preview/review rather than silently changing a decision. |
||||||
|
Account locks serialize commits and ordinary postings on the same account. |
||||||
|
Database source uniqueness resolves identical-file races across accounts. |
||||||
|
|
||||||
|
Ensure the multi-row commit uses one consistent lock order across the entire |
||||||
|
batch, not each row independently. Plan creation, source attachment and auditing |
||||||
|
must leave no partial records if a later row fails. Only transaction UUIDs, |
||||||
|
counts, operation and error codes enter generic completion messages. |
||||||
|
|
||||||
|
## Commands and acceptance |
||||||
|
|
||||||
|
Provide management commands to create/export a preview, apply a decision file, |
||||||
|
refresh a preview, commit a specific batch/revision, and cancel a batch. Commands |
||||||
|
require explicit actor kind/identifier and account/section mapping. Document |
||||||
|
PowerShell examples using the management container and ignored local import |
||||||
|
files. No real financial records are included in the examples. |
||||||
|
|
||||||
|
Test with synthetic files representing quoting, encodings, invalid dates/money, |
||||||
|
full/partial card layouts, year boundaries, payments, refunds, balance lines, |
||||||
|
multiple sections, installments, and unknown lines. Verify preview creates no |
||||||
|
financial rows, cancellation is inert, skipping is explicit, exact reimport is |
||||||
|
idempotent, semantic duplicates remain reviewable, and MATCH preserves history. |
||||||
|
|
||||||
|
PostgreSQL tests must cover a failing later row rolling back the entire batch, |
||||||
|
concurrent identical imports, stale review after other writes, archive changes, |
||||||
|
repeated commit, source conflicts, and failure audits. Verify new table grants, |
||||||
|
fresh/existing migrations, all financial-domain regressions, and app health. |
||||||
@ -0,0 +1,25 @@ |
|||||||
|
"""Compare snapshots without writing or discarding candidate movements.""" |
||||||
|
from decimal import Decimal |
||||||
|
import unicodedata |
||||||
|
|
||||||
|
|
||||||
|
def signature(*, account_id, currency, posting_date, amount, description, kind, |
||||||
|
installment_number=None, installment_total=None): |
||||||
|
return (str(account_id), currency, str(posting_date), Decimal(amount), |
||||||
|
" ".join(unicodedata.normalize("NFC", description).casefold().split()), |
||||||
|
kind, installment_number, installment_total) |
||||||
|
|
||||||
|
|
||||||
|
def candidate_signature(candidate, account, decision): |
||||||
|
return signature(account_id=account.pk, currency=account.currency, |
||||||
|
posting_date=candidate["posting_date"], amount=candidate["amount"], |
||||||
|
description=candidate["description"], kind=decision.get("kind", candidate["kind"]), |
||||||
|
installment_number=candidate.get("installment_number"), |
||||||
|
installment_total=candidate.get("installment_total")) |
||||||
|
|
||||||
|
|
||||||
|
def transaction_signature(row): |
||||||
|
return signature(account_id=row.account_id, currency=row.account.currency, |
||||||
|
posting_date=row.posting_date, amount=row.amount, description=row.description, |
||||||
|
kind=row.kind, installment_number=row.installment_number, |
||||||
|
installment_total=row.installment_plan.total_installments if row.installment_plan_id else None) |
||||||
@ -0,0 +1,208 @@ |
|||||||
|
"""Local, explicit review workflow. Financial details never go to stdout.""" |
||||||
|
import json |
||||||
|
import re |
||||||
|
from datetime import date |
||||||
|
from pathlib import Path |
||||||
|
from uuid import UUID |
||||||
|
|
||||||
|
from django.conf import settings |
||||||
|
from django.core.management.base import BaseCommand, CommandError |
||||||
|
|
||||||
|
from finance.services import Actor, DomainError |
||||||
|
from ingestion import services |
||||||
|
|
||||||
|
|
||||||
|
MAX_BYTES = 10 * 1024 * 1024 |
||||||
|
PROFILES = ("BANK_CSV_V1", "OUROCARD_FULL_V1", "OUROCARD_PARTIAL_V1") |
||||||
|
|
||||||
|
|
||||||
|
def local_path(value): |
||||||
|
root = Path(getattr(settings, "IMPORT_ROOT", "/app/imports")).resolve() |
||||||
|
requested = Path(value) |
||||||
|
if ".." in requested.parts: |
||||||
|
raise CommandError("INVALID_PATH") |
||||||
|
resolved = (root / requested).resolve() |
||||||
|
if not resolved.is_relative_to(root) or resolved == root: |
||||||
|
raise CommandError("INVALID_PATH") |
||||||
|
return resolved |
||||||
|
|
||||||
|
|
||||||
|
def read_bytes(value): |
||||||
|
path = local_path(value) |
||||||
|
if not path.is_file(): |
||||||
|
raise CommandError("INVALID_INPUT") |
||||||
|
with path.open("rb") as stream: |
||||||
|
data = stream.read(MAX_BYTES + 1) |
||||||
|
if len(data) > MAX_BYTES: |
||||||
|
raise CommandError("INPUT_TOO_LARGE") |
||||||
|
return data |
||||||
|
|
||||||
|
|
||||||
|
def unique_object(pairs): |
||||||
|
result = {} |
||||||
|
for key, value in pairs: |
||||||
|
if key in result: |
||||||
|
raise ValueError("duplicate key") |
||||||
|
result[key] = value |
||||||
|
return result |
||||||
|
|
||||||
|
|
||||||
|
def reject_constant(value): |
||||||
|
raise ValueError("nonfinite JSON number") |
||||||
|
|
||||||
|
|
||||||
|
def read_json(value): |
||||||
|
data = read_bytes(value) |
||||||
|
try: |
||||||
|
return json.loads(data.decode("utf-8-sig"), object_pairs_hook=unique_object, |
||||||
|
parse_constant=reject_constant) |
||||||
|
except (ValueError, RecursionError): |
||||||
|
raise CommandError("INVALID_JSON") from None |
||||||
|
|
||||||
|
|
||||||
|
def identifier(value): |
||||||
|
try: |
||||||
|
return str(UUID(str(value))) |
||||||
|
except (ValueError, AttributeError, TypeError): |
||||||
|
raise CommandError("INVALID_IDENTIFIER") from None |
||||||
|
|
||||||
|
|
||||||
|
def revision_number(value): |
||||||
|
if type(value) is not int or value < 1: |
||||||
|
raise CommandError("INVALID_REVISION") |
||||||
|
return value |
||||||
|
|
||||||
|
|
||||||
|
def checked_decisions(payload, batch_id, actor): |
||||||
|
minimum = {"batch_id", "revision", "decisions"} |
||||||
|
if not isinstance(payload, dict) or not minimum.issubset(payload): |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
if identifier(payload["batch_id"]) != batch_id: |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
revision_number(payload["revision"]) |
||||||
|
if not isinstance(payload["decisions"], list): |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
locators = set() |
||||||
|
for decision in payload["decisions"]: |
||||||
|
if not isinstance(decision, dict) or not isinstance(decision.get("locator"), str): |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
locator = decision["locator"] |
||||||
|
if locator in locators: |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
locators.add(locator) |
||||||
|
if set(payload) != minimum: |
||||||
|
current = services.export_review(batch_id=batch_id) |
||||||
|
if set(payload) != set(current): |
||||||
|
raise CommandError("INVALID_REVIEW") |
||||||
|
# All exported fields except the dedicated decisions array are evidence. |
||||||
|
# JSON encoding also distinguishes true/1 and numeric/string edits. |
||||||
|
for key in current: |
||||||
|
if key != "decisions" and json.dumps(payload[key], sort_keys=True) != json.dumps(current[key], sort_keys=True): |
||||||
|
raise CommandError("READ_ONLY_REVIEW_CHANGED") |
||||||
|
return payload["revision"], payload["decisions"] |
||||||
|
|
||||||
|
|
||||||
|
class Command(BaseCommand): |
||||||
|
help = "Preview, export, apply, refresh, commit or cancel a local reviewed import." |
||||||
|
requires_system_checks = [] |
||||||
|
|
||||||
|
def add_arguments(self, parser): |
||||||
|
commands = parser.add_subparsers(dest="operation", required=True) |
||||||
|
for operation in ("preview", "export", "apply", "refresh", "commit", "cancel"): |
||||||
|
sub = commands.add_parser(operation) |
||||||
|
sub.add_argument("--actor-kind", required=True, choices=("HUMAN", "APPLICATION", "AGENT")) |
||||||
|
sub.add_argument("--actor-identifier", required=True) |
||||||
|
if operation == "preview": |
||||||
|
sub.add_argument("--input", required=True) |
||||||
|
sub.add_argument("--mapping", required=True) |
||||||
|
sub.add_argument("--profile", required=True, choices=PROFILES) |
||||||
|
sub.add_argument("--context-start") |
||||||
|
sub.add_argument("--context-end") |
||||||
|
else: |
||||||
|
sub.add_argument("--batch", required=True) |
||||||
|
if operation == "export": |
||||||
|
sub.add_argument("--output", required=True) |
||||||
|
if operation == "apply": |
||||||
|
sub.add_argument("--input", required=True) |
||||||
|
if operation in {"refresh", "commit", "cancel"}: |
||||||
|
sub.add_argument("--revision", required=True, type=int) |
||||||
|
|
||||||
|
def handle(self, *args, **options): |
||||||
|
try: |
||||||
|
self.perform(options) |
||||||
|
except CommandError: |
||||||
|
raise |
||||||
|
except DomainError as exc: |
||||||
|
code = exc.code |
||||||
|
if not isinstance(code, str) or re.fullmatch(r"[A-Za-z][A-Za-z0-9_]{0,79}", code) is None: |
||||||
|
code = "IMPORT_FAILED" |
||||||
|
raise CommandError(code) from None |
||||||
|
except Exception: |
||||||
|
# Source paths, source text and database errors are local financial data. |
||||||
|
raise CommandError("IMPORT_FAILED") from None |
||||||
|
|
||||||
|
def perform(self, options): |
||||||
|
actor = Actor(options["actor_kind"], options["actor_identifier"]) |
||||||
|
if actor.kind not in {"HUMAN", "APPLICATION", "AGENT"} or not actor.identifier.strip() or len(actor.identifier) > 200: |
||||||
|
raise CommandError("INVALID_ACTOR") |
||||||
|
operation = options["operation"] |
||||||
|
if operation == "preview": |
||||||
|
mapping = read_json(options["mapping"]) |
||||||
|
if not isinstance(mapping, dict) or not mapping: |
||||||
|
raise CommandError("INVALID_MAPPING") |
||||||
|
for section, account in mapping.items(): |
||||||
|
if re.fullmatch(r"[1-9][0-9]*", section) is None: |
||||||
|
raise CommandError("INVALID_MAPPING") |
||||||
|
if account is not None: |
||||||
|
if not isinstance(account, str): |
||||||
|
raise CommandError("INVALID_MAPPING") |
||||||
|
identifier(account) |
||||||
|
start, end = options.get("context_start"), options.get("context_end") |
||||||
|
context = None |
||||||
|
if start or end: |
||||||
|
try: |
||||||
|
first, last = date.fromisoformat(start), date.fromisoformat(end) |
||||||
|
if not 0 <= (last - first).days <= 365: |
||||||
|
raise ValueError |
||||||
|
except (TypeError, ValueError): |
||||||
|
raise CommandError("INVALID_CONTEXT") from None |
||||||
|
context = {"start_date": first.isoformat(), "end_date": last.isoformat()} |
||||||
|
batch = services.create_preview( |
||||||
|
data=read_bytes(options["input"]), |
||||||
|
profile=options["profile"], |
||||||
|
section_mapping=mapping, |
||||||
|
context=context, |
||||||
|
actor=actor, |
||||||
|
) |
||||||
|
else: |
||||||
|
batch_id = identifier(options["batch"]) |
||||||
|
if operation == "export": |
||||||
|
path = local_path(options["output"]) |
||||||
|
review = services.export_review(batch_id=batch_id) |
||||||
|
encoded = json.dumps(review, ensure_ascii=False, indent=2, allow_nan=False) + "\n" |
||||||
|
# Exclusive creation protects source documents and previous reviews. |
||||||
|
with path.open("x", encoding="utf-8") as stream: |
||||||
|
stream.write(encoded) |
||||||
|
self.stdout.write(json.dumps({"batch_id": batch_id, "state": review["state"], |
||||||
|
"revision": review["revision"], "item_count": len(review["items"])})) |
||||||
|
return |
||||||
|
if operation == "apply": |
||||||
|
revision, decisions = checked_decisions(read_json(options["input"]), batch_id, actor) |
||||||
|
batch = services.apply_decisions( |
||||||
|
batch_id=batch_id, |
||||||
|
revision=revision, |
||||||
|
decisions=decisions, |
||||||
|
actor=actor, |
||||||
|
) |
||||||
|
else: |
||||||
|
revision = revision_number(options["revision"]) |
||||||
|
action = {"refresh": services.refresh_preview, "commit": services.commit_preview, |
||||||
|
"cancel": services.cancel_preview}[operation] |
||||||
|
batch = action(batch_id=batch_id, revision=revision, actor=actor) |
||||||
|
counts = getattr(batch, "result", {}) if batch.state == "COMMITTED" else getattr(batch, "summary", {}) |
||||||
|
safe_counts = { |
||||||
|
key: value for key, value in counts.items() |
||||||
|
if key in {"rows", "error_rows", "decided_rows", "created", "matched", "skipped"} |
||||||
|
} |
||||||
|
self.stdout.write(json.dumps({"batch_id": str(batch.pk), "state": batch.state, |
||||||
|
"revision": batch.revision, "counts": safe_counts})) |
||||||
@ -0,0 +1,58 @@ |
|||||||
|
# Generated by Django 5.2.17 on 2026-09-07 20:58 |
||||||
|
|
||||||
|
import django.db.models.deletion |
||||||
|
import uuid |
||||||
|
from django.db import migrations, models |
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration): |
||||||
|
|
||||||
|
initial = True |
||||||
|
|
||||||
|
dependencies = [ |
||||||
|
] |
||||||
|
|
||||||
|
operations = [ |
||||||
|
migrations.CreateModel( |
||||||
|
name='ImportBatch', |
||||||
|
fields=[ |
||||||
|
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)), |
||||||
|
('created_at', models.DateTimeField(auto_now_add=True)), |
||||||
|
('updated_at', models.DateTimeField(auto_now=True)), |
||||||
|
('document_hash', models.CharField(max_length=64)), |
||||||
|
('source_format', models.CharField(max_length=16)), |
||||||
|
('profile', models.CharField(max_length=32)), |
||||||
|
('parser_version', models.CharField(default='1', max_length=32)), |
||||||
|
('encoding', models.CharField(max_length=32)), |
||||||
|
('context', models.JSONField(default=dict)), |
||||||
|
('section_mapping', models.JSONField(default=dict)), |
||||||
|
('summary', models.JSONField(default=dict)), |
||||||
|
('snapshot', models.JSONField(default=dict)), |
||||||
|
('actor_kind', models.CharField(max_length=16)), |
||||||
|
('actor_identifier', models.CharField(max_length=200)), |
||||||
|
('state', models.CharField(default='PREVIEW', max_length=16)), |
||||||
|
('revision', models.PositiveIntegerField(default=1)), |
||||||
|
('result', models.JSONField(default=dict)), |
||||||
|
], |
||||||
|
options={ |
||||||
|
'constraints': [models.CheckConstraint(condition=models.Q(('state__in', ['PREVIEW', 'COMMITTED', 'CANCELLED'])), name='import_batch_state'), models.CheckConstraint(condition=models.Q(('revision__gte', 1)), name='import_batch_revision')], |
||||||
|
}, |
||||||
|
), |
||||||
|
migrations.CreateModel( |
||||||
|
name='ImportItem', |
||||||
|
fields=[ |
||||||
|
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)), |
||||||
|
('created_at', models.DateTimeField(auto_now_add=True)), |
||||||
|
('updated_at', models.DateTimeField(auto_now=True)), |
||||||
|
('locator', models.CharField(max_length=255)), |
||||||
|
('candidate', models.JSONField()), |
||||||
|
('issues', models.JSONField(default=list)), |
||||||
|
('duplicates', models.JSONField(default=dict)), |
||||||
|
('decision', models.JSONField(default=dict)), |
||||||
|
('batch', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='items', to='ingestion.importbatch')), |
||||||
|
], |
||||||
|
options={ |
||||||
|
'constraints': [models.UniqueConstraint(fields=('batch', 'locator'), name='import_item_locator_unique')], |
||||||
|
}, |
||||||
|
), |
||||||
|
] |
||||||
@ -0,0 +1,38 @@ |
|||||||
|
from django.db import models |
||||||
|
from django.db.models import Q |
||||||
|
from finance.models.base import TimestampedModel |
||||||
|
|
||||||
|
|
||||||
|
class ImportBatch(TimestampedModel): |
||||||
|
document_hash = models.CharField(max_length=64) |
||||||
|
source_format = models.CharField(max_length=16) |
||||||
|
profile = models.CharField(max_length=32) |
||||||
|
parser_version = models.CharField(max_length=32, default="1") |
||||||
|
encoding = models.CharField(max_length=32) |
||||||
|
context = models.JSONField(default=dict) |
||||||
|
section_mapping = models.JSONField(default=dict) |
||||||
|
summary = models.JSONField(default=dict) |
||||||
|
snapshot = models.JSONField(default=dict) |
||||||
|
actor_kind = models.CharField(max_length=16) |
||||||
|
actor_identifier = models.CharField(max_length=200) |
||||||
|
state = models.CharField(max_length=16, default="PREVIEW") |
||||||
|
revision = models.PositiveIntegerField(default=1) |
||||||
|
result = models.JSONField(default=dict) |
||||||
|
|
||||||
|
class Meta: |
||||||
|
constraints = [ |
||||||
|
models.CheckConstraint(condition=Q(state__in=["PREVIEW", "COMMITTED", "CANCELLED"]), name="import_batch_state"), |
||||||
|
models.CheckConstraint(condition=Q(revision__gte=1), name="import_batch_revision"), |
||||||
|
] |
||||||
|
|
||||||
|
|
||||||
|
class ImportItem(TimestampedModel): |
||||||
|
batch = models.ForeignKey(ImportBatch, on_delete=models.PROTECT, related_name="items") |
||||||
|
locator = models.CharField(max_length=255) |
||||||
|
candidate = models.JSONField() |
||||||
|
issues = models.JSONField(default=list) |
||||||
|
duplicates = models.JSONField(default=dict) |
||||||
|
decision = models.JSONField(default=dict) |
||||||
|
|
||||||
|
class Meta: |
||||||
|
constraints = [models.UniqueConstraint(fields=["batch", "locator"], name="import_item_locator_unique")] |
||||||
@ -0,0 +1,117 @@ |
|||||||
|
"""Convert parser evidence into immutable, serializable review candidates.""" |
||||||
|
|
||||||
|
import re |
||||||
|
from dataclasses import dataclass, asdict |
||||||
|
from datetime import date, datetime |
||||||
|
from decimal import Decimal |
||||||
|
|
||||||
|
from .parsers import ParseResult |
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True) |
||||||
|
class Candidate: |
||||||
|
locator: str |
||||||
|
section: str |
||||||
|
posting_date: str | None |
||||||
|
occurrence_date: str | None |
||||||
|
description: str |
||||||
|
amount: str | None |
||||||
|
kind: str |
||||||
|
installment_number: int | None |
||||||
|
installment_total: int | None |
||||||
|
errors: tuple[str, ...] |
||||||
|
|
||||||
|
def to_dict(self): |
||||||
|
result = asdict(self) |
||||||
|
result['errors'] = list(self.errors) |
||||||
|
return result |
||||||
|
|
||||||
|
|
||||||
|
def _date(text, profile, context): |
||||||
|
if profile != 'OUROCARD_PARTIAL_V1': |
||||||
|
pattern = r'\d{2}/\d{2}/\d{4}' if profile == 'BANK_CSV_V1' else r'\d{2}\.\d{2}\.\d{4}' |
||||||
|
if not re.fullmatch(pattern, text): |
||||||
|
raise ValueError('INVALID_DATE') |
||||||
|
try: |
||||||
|
return datetime.strptime(text, '%d/%m/%Y' if profile == 'BANK_CSV_V1' else '%d.%m.%Y').date().isoformat() |
||||||
|
except ValueError: |
||||||
|
raise ValueError('INVALID_DATE') from None |
||||||
|
if not context.get('start_date') or not context.get('end_date'): |
||||||
|
raise ValueError('MISSING_DATE_CONTEXT') |
||||||
|
try: |
||||||
|
start, end = date.fromisoformat(context['start_date']), date.fromisoformat(context['end_date']) |
||||||
|
if not 0 <= (end - start).days <= 365: |
||||||
|
raise ValueError |
||||||
|
except (ValueError, TypeError): |
||||||
|
raise ValueError('INVALID_DATE_CONTEXT') from None |
||||||
|
if not re.fullmatch(r'\d{2}/\d{2}', text): |
||||||
|
raise ValueError('INVALID_DATE') |
||||||
|
day, month = map(int, text.split('/')) |
||||||
|
possible = [] |
||||||
|
for year in range(start.year, end.year + 1): |
||||||
|
try: |
||||||
|
candidate = date(year, month, day) |
||||||
|
except ValueError: |
||||||
|
continue |
||||||
|
if start <= candidate <= end: |
||||||
|
possible.append(candidate) |
||||||
|
if len(possible) != 1: |
||||||
|
raise ValueError('AMBIGUOUS_DATE' if possible else 'INVALID_DATE') |
||||||
|
return possible[0].isoformat() |
||||||
|
|
||||||
|
|
||||||
|
def _cents(text, localized=False): |
||||||
|
pattern = r'[+-]?(?:\d+|\d{1,3}(?:\.\d{3})+),\d{2}' if localized else r'[+-]?\d+(?:\.\d{1,2})?' |
||||||
|
if len(text) > 40 or not re.fullmatch(pattern, text): |
||||||
|
raise ValueError('INVALID_AMOUNT') |
||||||
|
clean = text.replace('.', '').replace(',', '.') if localized else text |
||||||
|
# Integer minor units keep all arithmetic independent of Decimal context. |
||||||
|
negative = clean.startswith('-') |
||||||
|
clean = clean.lstrip('+-') |
||||||
|
whole, _, fraction = clean.partition('.') |
||||||
|
return (-1 if negative else 1) * (int(whole) * 100 + int(fraction.ljust(2, '0'))) |
||||||
|
|
||||||
|
|
||||||
|
def _format_cents(value): |
||||||
|
return ('-' if value < 0 else '') + f'{abs(value) // 100}.{abs(value) % 100:02d}' |
||||||
|
|
||||||
|
|
||||||
|
def normalize(result: ParseResult) -> tuple[Candidate, ...]: |
||||||
|
candidates = [] |
||||||
|
for item in result.items: |
||||||
|
errors = list(item.errors) |
||||||
|
posting_date = amount = installment_number = installment_total = None |
||||||
|
kind = 'EXPENSE' |
||||||
|
if not item.errors: |
||||||
|
try: |
||||||
|
posting_date = _date(item.date_text, result.profile, result.context) |
||||||
|
except ValueError as exc: |
||||||
|
errors.append(str(exc)) |
||||||
|
try: |
||||||
|
if result.profile == 'BANK_CSV_V1': |
||||||
|
cents = _cents(item.debit) |
||||||
|
kind = 'INCOME' if cents > 0 else 'EXPENSE' |
||||||
|
else: |
||||||
|
debit, credit = _cents(item.debit, True), _cents(item.credit, True) |
||||||
|
if item.payment: |
||||||
|
if debit and credit: |
||||||
|
errors.append('AMBIGUOUS_AMOUNT') |
||||||
|
# copy_abs is exact even when caller Decimal context traps rounding. |
||||||
|
cents = int(Decimal(debit).copy_abs()) |
||||||
|
kind = 'TRANSFER' |
||||||
|
else: |
||||||
|
if debit and credit: |
||||||
|
errors.append('AMBIGUOUS_AMOUNT') |
||||||
|
cents = credit - debit |
||||||
|
kind = 'REFUND' if cents > 0 else 'EXPENSE' |
||||||
|
amount = _format_cents(cents) |
||||||
|
except ValueError as exc: |
||||||
|
errors.append(str(exc)) |
||||||
|
if result.profile != 'BANK_CSV_V1' and re.search(r'\bPARC\b', item.description): |
||||||
|
match = re.search(r'\bPARC\s+(\d{1,4})/(\d{1,4})\b', item.description) |
||||||
|
if match and 1 <= int(match[1]) <= int(match[2]): |
||||||
|
installment_number, installment_total = map(int, match.groups()) |
||||||
|
else: |
||||||
|
errors.append('INVALID_INSTALLMENT') |
||||||
|
candidates.append(Candidate(item.locator, item.section, posting_date, posting_date, item.description, amount, kind, installment_number, installment_total, tuple(errors))) |
||||||
|
return tuple(candidates) |
||||||
@ -0,0 +1,136 @@ |
|||||||
|
"""Pure, versioned source grammars. No filesystem or database access.""" |
||||||
|
|
||||||
|
import csv |
||||||
|
import io |
||||||
|
import re |
||||||
|
import unicodedata |
||||||
|
from collections import Counter |
||||||
|
from dataclasses import dataclass |
||||||
|
from types import MappingProxyType |
||||||
|
from typing import Mapping |
||||||
|
|
||||||
|
|
||||||
|
PARSER_VERSION = '1' |
||||||
|
PROFILES = frozenset({'BANK_CSV_V1', 'OUROCARD_FULL_V1', 'OUROCARD_PARTIAL_V1'}) |
||||||
|
MAX_BYTES = 10 * 1024 * 1024 |
||||||
|
MAX_ITEMS = 20000 |
||||||
|
|
||||||
|
|
||||||
|
class ParseError(ValueError): |
||||||
|
def __init__(self, code): |
||||||
|
self.code = code |
||||||
|
super().__init__(code) |
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True) |
||||||
|
class ParsedItem: |
||||||
|
locator: str |
||||||
|
section: str |
||||||
|
date_text: str = '' |
||||||
|
description: str = '' |
||||||
|
debit: str = '' |
||||||
|
credit: str = '' |
||||||
|
payment: bool = False |
||||||
|
errors: tuple[str, ...] = () |
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True) |
||||||
|
class ParseResult: |
||||||
|
encoding: str |
||||||
|
profile: str |
||||||
|
items: tuple[ParsedItem, ...] |
||||||
|
skipped: Mapping[str, int] |
||||||
|
context: Mapping[str, str] |
||||||
|
|
||||||
|
|
||||||
|
def _header_key(value): |
||||||
|
return ''.join(c for c in unicodedata.normalize('NFD', value.strip().casefold()) if not unicodedata.combining(c)) |
||||||
|
|
||||||
|
|
||||||
|
def _balance(value): |
||||||
|
return bool(re.fullmatch(r'(?:S\s*A\s*L\s*D\s*O|SALDO ANTERIOR)(?:\s+[-\d.,]+)?', value.strip(), re.I)) |
||||||
|
|
||||||
|
|
||||||
|
def _csv_items(text, skipped): |
||||||
|
reader = csv.reader(io.StringIO(text, newline=''), strict=True) |
||||||
|
try: |
||||||
|
header = next(reader) |
||||||
|
aliases = ({'data', 'data movimento', 'data do movimento', 'data lancamento'}, {'historico', 'descricao', 'descricao do lancamento'}, {'valor', 'valor lancamento', 'valor do lancamento'}) |
||||||
|
if len(header) < 6 or any(_header_key(header[i]) not in allowed for i, allowed in zip((0, 2, 5), aliases)): |
||||||
|
raise ParseError('INVALID_HEADER') |
||||||
|
while True: |
||||||
|
line = reader.line_num + 1 |
||||||
|
try: |
||||||
|
row = next(reader) |
||||||
|
except StopIteration: |
||||||
|
return |
||||||
|
if not row or all(not cell.strip() for cell in row): |
||||||
|
skipped['blank'] += 1 |
||||||
|
continue |
||||||
|
if len(row) != len(header): |
||||||
|
description = row[2] if len(row) > 2 else '' |
||||||
|
yield ParsedItem(f'line:{line}', '1', description=description, |
||||||
|
errors=('INVALID_ROW',)) |
||||||
|
elif _balance(row[2]): |
||||||
|
skipped['balance'] += 1 |
||||||
|
else: |
||||||
|
yield ParsedItem(f'line:{line}', '1', row[0].strip(), row[2], row[5].strip()) |
||||||
|
except StopIteration: |
||||||
|
raise ParseError('INVALID_HEADER') from None |
||||||
|
except csv.Error: |
||||||
|
raise ParseError('INVALID_CSV') from None |
||||||
|
|
||||||
|
|
||||||
|
def _txt_items(text, profile, skipped): |
||||||
|
partial = profile == 'OUROCARD_PARTIAL_V1' |
||||||
|
date_size, body_size = (5, 43) if partial else (10, 37) |
||||||
|
date_re = r'\d{2}/\d{2}' if partial else r'\d{2}\.\d{2}\.\d{4}' |
||||||
|
regular = re.compile(rf'^({date_re})(.{{{body_size}}})([A-Z ]{{2}})\s+(\S+)\s+(\S+)\s*$') |
||||||
|
payment = re.compile(rf'^({date_re})\s*(PGTO DEBITO CONTA.*?)200211\s+(\S+)\s+(\S+)\s*$') |
||||||
|
section, section_count = '1', 0 |
||||||
|
for number, line in enumerate(text.splitlines(), 1): |
||||||
|
locator = f'line:{number}' |
||||||
|
if not line.strip(): |
||||||
|
skipped['blank'] += 1 |
||||||
|
elif re.match(r'^\s*\d\s*-\s*[A-Z]', line): |
||||||
|
section_count += 1 |
||||||
|
section = str(section_count) |
||||||
|
skipped['section'] += 1 |
||||||
|
elif _balance(line): |
||||||
|
skipped['balance'] += 1 |
||||||
|
elif match := payment.fullmatch(line): |
||||||
|
yield ParsedItem(locator, section, match[1], match[2].strip(), match[3], match[4], True) |
||||||
|
elif match := regular.fullmatch(line): |
||||||
|
if _balance(match[2]): |
||||||
|
skipped['balance'] += 1 |
||||||
|
else: |
||||||
|
yield ParsedItem(locator, section, match[1], match[2].rstrip(), match[4], match[5]) |
||||||
|
else: |
||||||
|
code = 'INVALID_ROW' if re.match(r'^\s*\d{1,2}[./]', line) else 'UNRECOGNIZED_LINE' |
||||||
|
yield ParsedItem(locator, section, description=line, errors=(code,)) |
||||||
|
|
||||||
|
|
||||||
|
def parse_document(data: bytes, profile: str, context: dict | None = None) -> ParseResult: |
||||||
|
if profile not in PROFILES: |
||||||
|
raise ParseError('UNSUPPORTED_PROFILE') |
||||||
|
if not isinstance(data, bytes): |
||||||
|
raise ParseError('INVALID_INPUT') |
||||||
|
if len(data) > MAX_BYTES: |
||||||
|
raise ParseError('INPUT_TOO_LARGE') |
||||||
|
if b'\x00' in data: |
||||||
|
raise ParseError('NUL_BYTE') |
||||||
|
if context is not None and (not isinstance(context, dict) or any(not isinstance(v, str) for v in context.values())): |
||||||
|
raise ParseError('INVALID_DATE_CONTEXT') |
||||||
|
try: |
||||||
|
encoding = 'utf-8-sig' if data.startswith(b'\xef\xbb\xbf') else 'utf-8' |
||||||
|
text = data.decode(encoding) |
||||||
|
except UnicodeDecodeError: |
||||||
|
encoding, text = 'latin-1', data.decode('latin-1') |
||||||
|
skipped = Counter() |
||||||
|
source = _csv_items(text, skipped) if profile == 'BANK_CSV_V1' else _txt_items(text, profile, skipped) |
||||||
|
items = [] |
||||||
|
for item in source: |
||||||
|
items.append(item) |
||||||
|
if len(items) > MAX_ITEMS: |
||||||
|
raise ParseError('TOO_MANY_ITEMS') |
||||||
|
return ParseResult(encoding, profile, tuple(items), MappingProxyType(dict(skipped)), MappingProxyType(dict(context or {}))) |
||||||
@ -0,0 +1,335 @@ |
|||||||
|
"""Audited review lifecycle. No financial rows exist until an explicit commit.""" |
||||||
|
from datetime import date |
||||||
|
from functools import wraps |
||||||
|
import hashlib |
||||||
|
import re |
||||||
|
from uuid import UUID, uuid5, NAMESPACE_URL |
||||||
|
|
||||||
|
from django.core.exceptions import ValidationError |
||||||
|
from django.db import IntegrityError, connection, transaction |
||||||
|
from django.db.models import Q |
||||||
|
|
||||||
|
from finance.models import Account, Payee, Category, CardStatement, InstallmentPlan, Transaction, SourceRecord |
||||||
|
from finance.services import Actor, DomainError |
||||||
|
from finance.services.base import audit, identity, save |
||||||
|
from finance.services.posting import record |
||||||
|
from ingestion.models import ImportBatch, ImportItem |
||||||
|
from ingestion.parsers import parse_document, ParseError |
||||||
|
from ingestion.normalization import normalize |
||||||
|
from ingestion.duplicates import candidate_signature, transaction_signature |
||||||
|
from ingestion.validation import validate |
||||||
|
|
||||||
|
|
||||||
|
REASONS = {"USER_EXCLUDED", "MALFORMED", "UNRECOGNIZED", "ALREADY_IMPORTED", "SECTION_EXCLUDED"} |
||||||
|
REFERENCE_FIELDS = {"payee_id": ("payees", Payee), "category_id": ("categories", Category), |
||||||
|
"statement_id": ("statements", CardStatement), "installment_plan_id": ("plans", InstallmentPlan)} |
||||||
|
|
||||||
|
|
||||||
|
def operation(name): |
||||||
|
def decorate(function): |
||||||
|
@wraps(function) |
||||||
|
def wrapped(*args, actor, **kwargs): |
||||||
|
if not isinstance(actor, Actor) or actor.kind not in {"HUMAN", "APPLICATION", "AGENT"} or not isinstance(actor.identifier, str) or not actor.identifier.strip() or len(actor.identifier) > 200: |
||||||
|
raise DomainError("invalid_actor") |
||||||
|
if connection.in_atomic_block: |
||||||
|
raise DomainError("transaction_boundary") |
||||||
|
try: |
||||||
|
with transaction.atomic(durable=True): |
||||||
|
batch, fields, emit = function(*args, actor=actor, **kwargs) |
||||||
|
if emit: |
||||||
|
audit(actor, name, "SUCCESS", batch.pk, fields) |
||||||
|
return batch |
||||||
|
except (DomainError, ParseError, ValidationError, IntegrityError) as error: |
||||||
|
code = getattr(error, "code", None) |
||||||
|
if not isinstance(code, str): |
||||||
|
code = "CONFLICT" if isinstance(error, IntegrityError) else "VALIDATION_ERROR" |
||||||
|
with transaction.atomic(durable=True): |
||||||
|
audit(actor, name, "FAILURE", error_code=code) |
||||||
|
raise DomainError(code) from None |
||||||
|
return wrapped |
||||||
|
return decorate |
||||||
|
|
||||||
|
|
||||||
|
def _batch(batch_id, revision=None, *, preview=True): |
||||||
|
try: |
||||||
|
batch = ImportBatch.objects.select_for_update().get(pk=identity(batch_id)) |
||||||
|
except ImportBatch.DoesNotExist: |
||||||
|
raise DomainError("BATCH_NOT_FOUND") from None |
||||||
|
if revision is not None and (type(revision) is not int or revision != batch.revision): |
||||||
|
raise DomainError("STALE_REVISION") |
||||||
|
if preview and batch.state != "PREVIEW": |
||||||
|
raise DomainError("BATCH_NOT_PREVIEW") |
||||||
|
return batch |
||||||
|
|
||||||
|
|
||||||
|
def _rows(query, locking): |
||||||
|
if locking: |
||||||
|
query = query.select_for_update(of=("self",)) |
||||||
|
return {str(obj.pk): obj for obj in query.order_by("pk")} |
||||||
|
|
||||||
|
|
||||||
|
def _load(batch, items, locking): |
||||||
|
account_ids = [value for value in batch.section_mapping.values() if value] |
||||||
|
accounts = _rows(Account.objects.filter(pk__in=account_ids), locking) |
||||||
|
refs = {} |
||||||
|
for field, (key, model) in REFERENCE_FIELDS.items(): |
||||||
|
ids = [item.decision[field] for item in items if item.decision.get(field)] |
||||||
|
refs[key] = _rows(model.objects.filter(pk__in=ids), locking) |
||||||
|
dates = set() |
||||||
|
for item in items: |
||||||
|
try: |
||||||
|
dates.add(date.fromisoformat(item.candidate.get("posting_date"))) |
||||||
|
except (TypeError, ValueError): |
||||||
|
pass |
||||||
|
target_ids = [item.decision[key] for item in items for key in ("transaction_id", "refund_of_id") if item.decision.get(key)] |
||||||
|
query = Transaction.objects.filter(Q(account_id__in=account_ids, posting_date__in=dates) | Q(pk__in=target_ids)) |
||||||
|
refs["transactions"] = _rows(query.select_related("account", "installment_plan"), locking) |
||||||
|
sources = {row.item_locator: row for row in SourceRecord.objects.filter(document_hash=batch.document_hash).select_related("transaction")} |
||||||
|
return accounts, refs, sources |
||||||
|
|
||||||
|
|
||||||
|
def _fingerprint(obj): |
||||||
|
# IDs and modification timestamps detect reference annotations/archive changes. |
||||||
|
return [str(obj.pk), obj.updated_at.isoformat()] |
||||||
|
|
||||||
|
|
||||||
|
def _evaluate(batch, items, *, locking=False): |
||||||
|
accounts, refs, sources = _load(batch, items, locking) |
||||||
|
keys = {} |
||||||
|
existing = {} |
||||||
|
for row in refs["transactions"].values(): |
||||||
|
existing.setdefault(transaction_signature(row), []).append(str(row.pk)) |
||||||
|
for item in items: |
||||||
|
account = accounts.get(batch.section_mapping.get(item.candidate["section"])) |
||||||
|
if account and not item.candidate.get("errors"): |
||||||
|
try: |
||||||
|
keys[item.locator] = candidate_signature(item.candidate, account, item.decision) |
||||||
|
except (KeyError, TypeError, ValueError): |
||||||
|
pass |
||||||
|
results = {} |
||||||
|
plan_definitions = {} |
||||||
|
plan_memberships = set() |
||||||
|
for item in items: |
||||||
|
candidate, decision = item.candidate, item.decision |
||||||
|
mapped = batch.section_mapping.get(candidate["section"]) |
||||||
|
account = accounts.get(mapped) |
||||||
|
issues = validate(candidate, decision, account, refs, batch.profile) if mapped else ["SECTION_EXCLUDED"] |
||||||
|
key = keys.get(item.locator) |
||||||
|
source = sources.get(item.locator) |
||||||
|
matches = sorted(existing.get(key, [])) if key is not None else [] |
||||||
|
within = sorted(locator for locator, other in keys.items() if locator != item.locator and other == key) if key is not None else [] |
||||||
|
duplicate = {"existing": matches, "within": within, |
||||||
|
"source": str(source.transaction_id) if source else None, |
||||||
|
"source_account": str(source.transaction.account_id) if source else None, |
||||||
|
"status": "ALREADY_IMPORTED" if source else None} |
||||||
|
if source: |
||||||
|
issues.append("ALREADY_IMPORTED" if str(source.transaction.account_id) == mapped else "SOURCE_ACCOUNT_CONFLICT") |
||||||
|
if decision.get("new_plan") and decision.get("action") == "CREATE": |
||||||
|
name = decision["new_plan"] |
||||||
|
definition = (mapped, candidate.get("installment_total")) |
||||||
|
if name in plan_definitions and plan_definitions[name] != definition: |
||||||
|
issues.append("NEW_PLAN_CONFLICT") |
||||||
|
plan_definitions[name] = definition |
||||||
|
if decision.get("action") == "CREATE" and candidate.get("installment_number"): |
||||||
|
plan_key = decision.get("installment_plan_id") or decision.get("new_plan") |
||||||
|
membership = (plan_key, candidate["installment_number"]) |
||||||
|
if plan_key and membership in plan_memberships: |
||||||
|
issues.append("DUPLICATE_INSTALLMENT") |
||||||
|
plan_memberships.add(membership) |
||||||
|
results[item.locator] = {"issues": sorted(set(issues)), "duplicates": duplicate} |
||||||
|
reference_snapshot = {name: [_fingerprint(obj) for obj in table.values()] for name, table in refs.items() if name != "transactions"} |
||||||
|
relevant_ids = {tid for result in results.values() for tid in result["duplicates"]["existing"]} |
||||||
|
relevant_ids.update(item.decision[key] for item in items for key in ("transaction_id", "refund_of_id") if item.decision.get(key)) |
||||||
|
reference_snapshot["transactions"] = [_fingerprint(obj) for key, obj in refs["transactions"].items() if key in relevant_ids] |
||||||
|
return {"accounts": [_fingerprint(obj) for obj in accounts.values()], "references": reference_snapshot, "items": results} |
||||||
|
|
||||||
|
|
||||||
|
def _persist(batch, items, snapshot): |
||||||
|
for item in items: |
||||||
|
current = snapshot["items"][item.locator] |
||||||
|
item.issues, item.duplicates = current["issues"], current["duplicates"] |
||||||
|
ImportItem.objects.bulk_update(items, ["issues", "duplicates", "decision"]) |
||||||
|
batch.snapshot = snapshot |
||||||
|
batch.summary = {**batch.summary, "rows": len(items), "error_rows": sum(bool(item.issues) for item in items), |
||||||
|
"decided_rows": sum(bool(item.decision) for item in items)} |
||||||
|
batch.save() |
||||||
|
|
||||||
|
|
||||||
|
def _current(batch, items): |
||||||
|
snapshot = _evaluate(batch, items, locking=True) |
||||||
|
if snapshot != batch.snapshot: |
||||||
|
raise DomainError("STALE_PREVIEW") |
||||||
|
return snapshot |
||||||
|
|
||||||
|
|
||||||
|
@operation("import_preview") |
||||||
|
def create_preview(*, data, profile, section_mapping, context=None, actor): |
||||||
|
parsed = parse_document(data, profile, context or {}) |
||||||
|
candidates = [candidate.to_dict() for candidate in normalize(parsed)] |
||||||
|
if not isinstance(section_mapping, dict) or any(not isinstance(key, str) for key in section_mapping): |
||||||
|
raise DomainError("INVALID_SECTION_MAPPING") |
||||||
|
sections = {item["section"] for item in candidates} |
||||||
|
if set(section_mapping) != sections: |
||||||
|
raise DomainError("SECTION_MAPPING_REQUIRED") |
||||||
|
mapping = {key: str(identity(value)) if value is not None else None for key, value in section_mapping.items()} |
||||||
|
batch = ImportBatch.objects.create(document_hash=hashlib.sha256(data).hexdigest(), |
||||||
|
source_format="BANK_CSV" if profile == "BANK_CSV_V1" else "OUROCARD_TXT", |
||||||
|
profile=profile, encoding=parsed.encoding, context=context or {}, section_mapping=mapping, |
||||||
|
summary={"parser_skipped": dict(parsed.skipped)}, actor_kind=actor.kind, actor_identifier=actor.identifier) |
||||||
|
items = [ImportItem(batch=batch, locator=candidate["locator"], candidate=candidate) for candidate in candidates] |
||||||
|
ImportItem.objects.bulk_create(items) |
||||||
|
_persist(batch, items, _evaluate(batch, items, locking=True)) |
||||||
|
return batch, ["preview"], True |
||||||
|
|
||||||
|
|
||||||
|
def export_review(*, batch_id): |
||||||
|
# Keep exported revision and item decisions consistent with concurrent editing. |
||||||
|
with transaction.atomic(): |
||||||
|
batch = _batch(batch_id, preview=False) |
||||||
|
items = list(batch.items.order_by("created_at", "locator")) |
||||||
|
return {"batch_id": str(batch.pk), "revision": batch.revision, "state": batch.state, |
||||||
|
"profile": batch.profile, "encoding": batch.encoding, "summary": batch.summary, |
||||||
|
"items": [{"locator": row.locator, "candidate_id": str(uuid5(NAMESPACE_URL, batch.document_hash + ":" + row.locator)), |
||||||
|
"candidate": row.candidate, "issues": row.issues, "duplicates": row.duplicates, |
||||||
|
"decision": row.decision} for row in items], |
||||||
|
"decisions": [{"locator": row.locator, **row.decision} for row in items]} |
||||||
|
|
||||||
|
|
||||||
|
def _decisions(value, locators): |
||||||
|
if not isinstance(value, list): |
||||||
|
raise DomainError("INVALID_DECISIONS") |
||||||
|
result = {} |
||||||
|
for supplied in value: |
||||||
|
if not isinstance(supplied, dict): |
||||||
|
raise DomainError("INVALID_DECISION") |
||||||
|
row = dict(supplied) |
||||||
|
locator = row.pop("locator", None) |
||||||
|
if not isinstance(locator, str) or locator not in locators or locator in result: |
||||||
|
raise DomainError("INVALID_LOCATOR") |
||||||
|
action = row.get("action") |
||||||
|
allowed = {"action"} |
||||||
|
if action == "CREATE": |
||||||
|
allowed |= {"acknowledge_duplicate", "kind", "payee_id", "category_id", "statement_id", "refund_of_id", "installment_plan_id", "new_plan"} |
||||||
|
elif action == "MATCH": |
||||||
|
allowed.add("transaction_id") |
||||||
|
elif action == "SKIP": |
||||||
|
allowed.add("reason") |
||||||
|
else: |
||||||
|
raise DomainError("INVALID_ACTION") |
||||||
|
if set(row) - allowed: |
||||||
|
raise DomainError("UNKNOWN_DECISION_FIELD") |
||||||
|
if "acknowledge_duplicate" in row and type(row["acknowledge_duplicate"]) is not bool: |
||||||
|
raise DomainError("INVALID_ACKNOWLEDGEMENT") |
||||||
|
if action == "SKIP" and row.get("reason") not in REASONS: |
||||||
|
raise DomainError("SKIP_REASON_REQUIRED") |
||||||
|
if action == "MATCH" and not row.get("transaction_id"): |
||||||
|
raise DomainError("MATCH_TARGET_REQUIRED") |
||||||
|
for field in set(row) & {"transaction_id", "payee_id", "category_id", "statement_id", "refund_of_id", "installment_plan_id"}: |
||||||
|
if row[field] is not None: |
||||||
|
row[field] = str(identity(row[field])) |
||||||
|
if "new_plan" in row and (not isinstance(row["new_plan"], str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", row["new_plan"])): |
||||||
|
raise DomainError("INVALID_NEW_PLAN") |
||||||
|
if "kind" in row and row["kind"] not in {"INCOME", "EXPENSE", "REFUND", "TRANSFER"}: |
||||||
|
raise DomainError("INVALID_KIND") |
||||||
|
result[locator] = row |
||||||
|
return result |
||||||
|
|
||||||
|
|
||||||
|
@operation("import_decisions") |
||||||
|
def apply_decisions(*, batch_id, revision, decisions, actor): |
||||||
|
batch = _batch(batch_id, revision) |
||||||
|
items = list(batch.items.order_by("locator")) |
||||||
|
_current(batch, items) |
||||||
|
values = _decisions(decisions, {item.locator for item in items}) |
||||||
|
for item in items: |
||||||
|
item.decision = values.get(item.locator, {}) |
||||||
|
batch.revision += 1 |
||||||
|
_persist(batch, items, _evaluate(batch, items, locking=True)) |
||||||
|
return batch, ["decisions", "revision"], True |
||||||
|
|
||||||
|
|
||||||
|
@operation("import_refresh") |
||||||
|
def refresh_preview(*, batch_id, revision, actor): |
||||||
|
batch = _batch(batch_id, revision) |
||||||
|
items = list(batch.items.order_by("locator")) |
||||||
|
for item in items: |
||||||
|
item.decision = {} |
||||||
|
batch.revision += 1 |
||||||
|
_persist(batch, items, _evaluate(batch, items, locking=True)) |
||||||
|
return batch, ["preview", "decisions", "revision"], True |
||||||
|
|
||||||
|
|
||||||
|
@operation("import_cancel") |
||||||
|
def cancel_preview(*, batch_id, revision, actor): |
||||||
|
batch = _batch(batch_id, revision) |
||||||
|
batch.state = "CANCELLED" |
||||||
|
batch.save() |
||||||
|
return batch, ["state"], True |
||||||
|
|
||||||
|
|
||||||
|
def _check_decision(item): |
||||||
|
decision = item.decision |
||||||
|
if not decision: |
||||||
|
raise DomainError("DECISION_REQUIRED") |
||||||
|
if "SOURCE_ACCOUNT_CONFLICT" in item.issues: |
||||||
|
raise DomainError("SOURCE_ACCOUNT_CONFLICT") |
||||||
|
if decision["action"] == "SKIP": |
||||||
|
if item.duplicates.get("source") and decision["reason"] != "ALREADY_IMPORTED": |
||||||
|
raise DomainError("ALREADY_IMPORTED") |
||||||
|
if "SECTION_EXCLUDED" in item.issues and decision["reason"] != "SECTION_EXCLUDED": |
||||||
|
raise DomainError("SECTION_EXCLUDED") |
||||||
|
return |
||||||
|
if item.duplicates.get("source"): |
||||||
|
raise DomainError("ALREADY_IMPORTED") |
||||||
|
# MATCH validates the signature rather than requiring a new plan assignment. |
||||||
|
issues = [issue for issue in item.issues if not (decision["action"] == "MATCH" and issue == "INSTALLMENT_PLAN_REQUIRED")] |
||||||
|
if issues: |
||||||
|
raise DomainError("UNRESOLVED_ISSUES") |
||||||
|
if decision["action"] == "CREATE" and (item.duplicates["existing"] or item.duplicates["within"]) and not decision.get("acknowledge_duplicate"): |
||||||
|
raise DomainError("DUPLICATE_ACKNOWLEDGEMENT_REQUIRED") |
||||||
|
if decision["action"] == "MATCH" and decision["transaction_id"] not in item.duplicates["existing"]: |
||||||
|
raise DomainError("MATCH_SIGNATURE_MISMATCH") |
||||||
|
|
||||||
|
|
||||||
|
@operation("import_commit") |
||||||
|
def commit_preview(*, batch_id, revision, actor): |
||||||
|
batch = _batch(batch_id, revision, preview=False) |
||||||
|
if batch.state == "COMMITTED": |
||||||
|
return batch, [], False |
||||||
|
if batch.state != "PREVIEW": |
||||||
|
raise DomainError("BATCH_NOT_PREVIEW") |
||||||
|
items = list(batch.items.order_by("locator")) |
||||||
|
_current(batch, items) |
||||||
|
for item in items: |
||||||
|
_check_decision(item) |
||||||
|
plans = {} |
||||||
|
result = {"created": 0, "matched": 0, "skipped": 0, "transaction_ids": [], "parser_skipped": batch.summary.get("parser_skipped", {}), "error_rows": batch.summary.get("error_rows", 0)} |
||||||
|
for item in items: |
||||||
|
candidate, decision = item.candidate, item.decision |
||||||
|
if decision["action"] == "SKIP": |
||||||
|
result["skipped"] += 1 |
||||||
|
continue |
||||||
|
if decision["action"] == "MATCH": |
||||||
|
row = Transaction.objects.get(pk=decision["transaction_id"]) |
||||||
|
result["matched"] += 1 |
||||||
|
else: |
||||||
|
account_id = batch.section_mapping[candidate["section"]] |
||||||
|
plan_id = decision.get("installment_plan_id") |
||||||
|
if decision.get("new_plan"): |
||||||
|
name = decision["new_plan"] |
||||||
|
if name not in plans: |
||||||
|
plans[name] = save(InstallmentPlan(account_id=account_id, description=name, total_installments=candidate["installment_total"])) |
||||||
|
plan_id = plans[name].pk |
||||||
|
fields = {key: decision.get(key) for key in ("payee_id", "category_id", "statement_id", "refund_of_id")} |
||||||
|
row = record(account_id=account_id, amount=candidate["amount"], posting_date=candidate["posting_date"], |
||||||
|
occurrence_date=candidate["occurrence_date"], kind=decision.get("kind", candidate["kind"]), |
||||||
|
description=candidate["description"], installment_plan_id=plan_id, |
||||||
|
installment_number=candidate.get("installment_number"), **fields) |
||||||
|
result["created"] += 1 |
||||||
|
save(SourceRecord(transaction=row, document_hash=batch.document_hash, source_format=batch.source_format, |
||||||
|
parser_version=batch.parser_version, item_locator=item.locator)) |
||||||
|
result["transaction_ids"].append(str(row.pk)) |
||||||
|
batch.state = "COMMITTED" |
||||||
|
batch.result = result |
||||||
|
batch.save() |
||||||
|
return batch, ["state", "transactions", "source_records", "plans"], True |
||||||
@ -0,0 +1,57 @@ |
|||||||
|
"""Read-only domain validation for review; commit also uses finance posting rules.""" |
||||||
|
from datetime import date |
||||||
|
from finance.services.base import DomainError, money |
||||||
|
from finance.models import Transaction |
||||||
|
|
||||||
|
|
||||||
|
def validate(candidate, decision, account, refs, profile): |
||||||
|
issues = list(candidate.get("errors", [])) |
||||||
|
if account is None: |
||||||
|
return issues + ["ACCOUNT_NOT_FOUND"] |
||||||
|
if account.archived: |
||||||
|
issues.append("ACCOUNT_ARCHIVED") |
||||||
|
if account.currency != "BRL": |
||||||
|
issues.append("UNSUPPORTED_CURRENCY") |
||||||
|
expected = "BANK" if profile == "BANK_CSV_V1" else "CREDIT_CARD" |
||||||
|
if account.kind != expected: |
||||||
|
issues.append("ACCOUNT_KIND_MISMATCH") |
||||||
|
if issues: |
||||||
|
return issues |
||||||
|
try: |
||||||
|
amount = money(candidate["amount"]) |
||||||
|
date.fromisoformat(candidate["posting_date"]) |
||||||
|
except (DomainError, ValueError, TypeError, KeyError): |
||||||
|
return ["INVALID_MOVEMENT"] |
||||||
|
kind = decision.get("kind", candidate["kind"]) |
||||||
|
if kind not in {"INCOME", "EXPENSE", "REFUND", "TRANSFER"} or (kind in {"INCOME", "REFUND"} and amount <= 0) or (kind == "EXPENSE" and amount >= 0): |
||||||
|
issues.append("INVALID_KIND_OR_SIGN") |
||||||
|
for field, table in (("payee_id", "payees"), ("category_id", "categories")): |
||||||
|
if decision.get(field): |
||||||
|
ref = refs[table].get(decision[field]) |
||||||
|
if ref is None or ref.archived: |
||||||
|
issues.append("INVALID_" + field.upper()) |
||||||
|
category = refs["categories"].get(decision.get("category_id")) |
||||||
|
if category and category.kind != {"INCOME": "INCOME", "EXPENSE": "EXPENSE", "REFUND": "EXPENSE"}.get(kind): |
||||||
|
issues.append("CATEGORY_KIND_MISMATCH") |
||||||
|
statement = refs["statements"].get(decision.get("statement_id")) |
||||||
|
if decision.get("statement_id") and (statement is None or statement.account_id != account.pk or account.kind != "CREDIT_CARD"): |
||||||
|
issues.append("STATEMENT_ACCOUNT_MISMATCH") |
||||||
|
original = refs["transactions"].get(decision.get("refund_of_id")) |
||||||
|
if decision.get("refund_of_id") and (original is None or kind != "REFUND" or original.kind != "EXPENSE" or original.account.currency != account.currency): |
||||||
|
issues.append("INVALID_REFUND_REFERENCE") |
||||||
|
number, total = candidate.get("installment_number"), candidate.get("installment_total") |
||||||
|
if number is not None or total is not None: |
||||||
|
if not isinstance(number, int) or not isinstance(total, int) or not 1 <= number <= total or kind != "EXPENSE": |
||||||
|
issues.append("INVALID_INSTALLMENT") |
||||||
|
plan_id, new_plan = decision.get("installment_plan_id"), decision.get("new_plan") |
||||||
|
if bool(plan_id) == bool(new_plan): |
||||||
|
issues.append("INSTALLMENT_PLAN_REQUIRED") |
||||||
|
if plan_id: |
||||||
|
plan = refs["plans"].get(plan_id) |
||||||
|
if plan is None or plan.account_id != account.pk or plan.total_installments != total: |
||||||
|
issues.append("INSTALLMENT_PLAN_MISMATCH") |
||||||
|
elif decision.get("action") == "CREATE" and Transaction.objects.filter(installment_plan=plan, installment_number=number).exists(): |
||||||
|
issues.append("INSTALLMENT_ALREADY_POSTED") |
||||||
|
elif decision.get("installment_plan_id") or decision.get("new_plan"): |
||||||
|
issues.append("UNEXPECTED_INSTALLMENT_PLAN") |
||||||
|
return issues |
||||||
@ -0,0 +1,12 @@ |
|||||||
|
# Synthetic import fixtures |
||||||
|
|
||||||
|
Every name, amount, date, and identifier in this directory is invented for |
||||||
|
tests. The files model the three versioned import profiles and contain no data |
||||||
|
copied from Robopato or a financial institution. |
||||||
|
|
||||||
|
- `bank.csv`: UTF-8, comma-separated columns with date at index 0, |
||||||
|
description at index 2, and signed decimal amount at index 5. |
||||||
|
- `full.txt`: full `DD.MM.YYYY` Ourocard-style fixed-width rows, including a |
||||||
|
credit adjustment and account-payment line. |
||||||
|
- `partial.txt`: yearless `DD/MM` fixed-width rows crossing a year boundary; |
||||||
|
parsing requires explicit start and end dates. |
||||||
|
@ -0,0 +1,4 @@ |
|||||||
|
1 - SYNTHETIC OWNER |
||||||
|
15.02.2026SYNTHETIC SHOP BR 12,34 0,00 |
||||||
|
16.02.2026SYNTHETIC REFUND BR -8,11 0,00 |
||||||
|
17.02.2026PGTO DEBITO CONTA 200211 33,00 0,00 |
||||||
@ -0,0 +1,3 @@ |
|||||||
|
1 - SYNTHETIC OWNER |
||||||
|
31/12SYNTHETIC PARTIAL BR 12,34 0,00 |
||||||
|
01/01SYNTHETIC NEW YEAR BR 12,34 0,00 |
||||||
@ -0,0 +1,132 @@ |
|||||||
|
"""Synthetic command-boundary tests; lifecycle persistence has separate coverage.""" |
||||||
|
import copy |
||||||
|
import importlib |
||||||
|
import json |
||||||
|
from io import StringIO |
||||||
|
from types import SimpleNamespace |
||||||
|
from uuid import UUID |
||||||
|
|
||||||
|
import pytest |
||||||
|
from django.core.management import call_command |
||||||
|
from django.core.management.base import CommandError |
||||||
|
|
||||||
|
|
||||||
|
BATCH = "11111111-1111-4111-8111-111111111111" |
||||||
|
ACTOR = ["--actor-kind", "HUMAN", "--actor-identifier", "synthetic-owner"] |
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture |
||||||
|
def command(tmp_path, settings, monkeypatch): |
||||||
|
settings.IMPORT_ROOT = tmp_path |
||||||
|
module = importlib.import_module("ingestion.management.commands.import_review") |
||||||
|
review = { |
||||||
|
"batch_id": BATCH, "revision": 1, "state": "PREVIEW", |
||||||
|
"profile": "BANK_CSV_V1", "encoding": "utf-8", |
||||||
|
"summary": {"items": 1}, |
||||||
|
"items": [{"locator": "line:2", "candidate": {"description": "SYNTHETIC PRIVATE LABEL"}, |
||||||
|
"issues": [], "duplicates": [], "decision": None}], |
||||||
|
"decisions": [{"locator": "line:2", "action": "CREATE"}], |
||||||
|
} |
||||||
|
batch = SimpleNamespace(pk=UUID(BATCH), state="PREVIEW", revision=1) |
||||||
|
monkeypatch.setattr(module.services, "export_review", lambda *a, **kw: copy.deepcopy(review)) |
||||||
|
monkeypatch.setattr(module.services, "apply_decisions", lambda *a, **kw: batch) |
||||||
|
return module, review |
||||||
|
|
||||||
|
|
||||||
|
def run(command, operation, *args): |
||||||
|
output = StringIO() |
||||||
|
call_command(command[0].Command(), operation, *args, *ACTOR, stdout=output) |
||||||
|
return output.getvalue() |
||||||
|
|
||||||
|
|
||||||
|
def test_export_keeps_candidate_text_in_local_file_only(command, tmp_path): |
||||||
|
output = run(command, "export", "--batch", BATCH, "--output", "review.json") |
||||||
|
assert "SYNTHETIC PRIVATE LABEL" not in output |
||||||
|
assert json.loads((tmp_path / "review.json").read_text())["items"][0]["candidate"]["description"] == "SYNTHETIC PRIVATE LABEL" |
||||||
|
assert json.loads(output)["batch_id"] == BATCH |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("path", ["../outside.json", "nested/../../outside.json"]) |
||||||
|
def test_output_rejects_traversal(command, path): |
||||||
|
with pytest.raises(CommandError, match="INVALID_PATH"): |
||||||
|
run(command, "export", "--batch", BATCH, "--output", path) |
||||||
|
|
||||||
|
|
||||||
|
def test_output_never_overwrites_input_or_existing_review(command, tmp_path): |
||||||
|
destination = tmp_path / "source.json" |
||||||
|
destination.write_text("synthetic original") |
||||||
|
with pytest.raises(CommandError): |
||||||
|
run(command, "export", "--batch", BATCH, "--output", "source.json") |
||||||
|
assert destination.read_text() == "synthetic original" |
||||||
|
|
||||||
|
|
||||||
|
def test_output_rejects_symlink_escape(command, tmp_path): |
||||||
|
link = tmp_path / "escape" |
||||||
|
try: |
||||||
|
link.symlink_to(tmp_path.parent, target_is_directory=True) |
||||||
|
except OSError: |
||||||
|
pytest.skip("Symlink creation unavailable") |
||||||
|
with pytest.raises(CommandError, match="INVALID_PATH"): |
||||||
|
run(command, "export", "--batch", BATCH, "--output", "escape/review.json") |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", [ |
||||||
|
'{"batch_id":"' + BATCH + '","revision":1,"revision":2,"decisions":[]}', |
||||||
|
'{"batch_id":"' + BATCH + '","revision":1,"decisions":[{"locator":"2","action":"CREATE","action":"SKIP"}]}', |
||||||
|
'{"batch_id":"' + BATCH + '","revision":NaN,"decisions":[]}', |
||||||
|
]) |
||||||
|
def test_apply_rejects_ambiguous_json(command, tmp_path, payload): |
||||||
|
(tmp_path / "decisions.json").write_text(payload) |
||||||
|
with pytest.raises(CommandError, match="INVALID_JSON"): |
||||||
|
run(command, "apply", "--batch", BATCH, "--input", "decisions.json") |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("change", ["candidate", "metadata", "unknown", "revision", "batch"]) |
||||||
|
def test_apply_rejects_modified_evidence_or_wrong_envelope(command, tmp_path, change): |
||||||
|
payload = copy.deepcopy(command[1]) |
||||||
|
if change == "candidate": |
||||||
|
payload["items"][0]["candidate"]["description"] = "tampered" |
||||||
|
elif change == "metadata": |
||||||
|
payload["encoding"] = "latin-1" |
||||||
|
elif change == "unknown": |
||||||
|
payload["extra"] = True |
||||||
|
elif change == "revision": |
||||||
|
payload["revision"] = True |
||||||
|
else: |
||||||
|
payload["batch_id"] = "22222222-2222-4222-8222-222222222222" |
||||||
|
(tmp_path / "decisions.json").write_text(json.dumps(payload)) |
||||||
|
with pytest.raises(CommandError): |
||||||
|
run(command, "apply", "--batch", BATCH, "--input", "decisions.json") |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("full", [True, False]) |
||||||
|
def test_apply_accepts_decisions_only_or_unchanged_full_export(command, tmp_path, full): |
||||||
|
payload = copy.deepcopy(command[1]) |
||||||
|
payload["decisions"] = [{"locator": "line:2", "action": "SKIP", "reason": "USER_EXCLUDED"}] |
||||||
|
if not full: |
||||||
|
payload = {key: payload[key] for key in ("batch_id", "revision", "decisions")} |
||||||
|
(tmp_path / "decisions.json").write_text(json.dumps(payload)) |
||||||
|
output = run(command, "apply", "--batch", BATCH, "--input", "decisions.json") |
||||||
|
assert "SYNTHETIC PRIVATE LABEL" not in output |
||||||
|
assert json.loads(output)["state"] == "PREVIEW" |
||||||
|
|
||||||
|
|
||||||
|
def test_file_read_is_bounded(command, tmp_path): |
||||||
|
with (tmp_path / "large.json").open("wb") as stream: |
||||||
|
stream.truncate(10 * 1024 * 1024 + 1) |
||||||
|
with pytest.raises(CommandError, match="INPUT_TOO_LARGE"): |
||||||
|
run(command, "apply", "--batch", BATCH, "--input", "large.json") |
||||||
|
|
||||||
|
|
||||||
|
def test_service_exception_does_not_disclose_source(command, monkeypatch): |
||||||
|
def fail(*args, **kwargs): |
||||||
|
raise RuntimeError("SYNTHETIC PRIVATE LABEL") |
||||||
|
monkeypatch.setattr(command[0].services, "export_review", fail) |
||||||
|
with pytest.raises(CommandError) as caught: |
||||||
|
run(command, "export", "--batch", BATCH, "--output", "review.json") |
||||||
|
assert "SYNTHETIC" not in str(caught.value) |
||||||
|
|
||||||
|
|
||||||
|
def test_all_operations_require_explicit_actor(command): |
||||||
|
with pytest.raises(CommandError): |
||||||
|
call_command(command[0].Command(), "export", "--batch", BATCH, "--output", "review.json") |
||||||
@ -0,0 +1,72 @@ |
|||||||
|
"""Competing import commits use independent PostgreSQL connections.""" |
||||||
|
|
||||||
|
from concurrent.futures import ThreadPoolExecutor |
||||||
|
from threading import Barrier |
||||||
|
|
||||||
|
import pytest |
||||||
|
from django.db import close_old_connections, connection, connections |
||||||
|
|
||||||
|
from finance.models import AuditEvent, SourceRecord, Transaction |
||||||
|
from finance.services import Actor, DomainError, create_account |
||||||
|
from ingestion import services |
||||||
|
|
||||||
|
pytestmark = pytest.mark.django_db(transaction=True) |
||||||
|
ACTOR = Actor(kind="AGENT", identifier="synthetic-import-concurrency") |
||||||
|
DATA = b"Data,Documento,Historico,Agencia,Lote,Valor\n15/01/2026,1,Synthetic purchase,0,0,-12.50\n" |
||||||
|
|
||||||
|
|
||||||
|
def reviewed(bank): |
||||||
|
batch = services.create_preview(data=DATA, profile="BANK_CSV_V1", section_mapping={"1": str(bank.pk)}, actor=ACTOR) |
||||||
|
return services.apply_decisions(batch_id=batch.pk, revision=batch.revision, actor=ACTOR, |
||||||
|
decisions=[{"locator": batch.items.get().locator, "action": "CREATE"}]) |
||||||
|
|
||||||
|
|
||||||
|
def race(batches): |
||||||
|
assert connection.vendor == "postgresql" |
||||||
|
barrier = Barrier(2) |
||||||
|
|
||||||
|
def attempt(batch): |
||||||
|
close_old_connections() |
||||||
|
try: |
||||||
|
with connection.cursor() as cursor: |
||||||
|
cursor.execute("SET statement_timeout = '20s'") |
||||||
|
cursor.execute("SELECT pg_backend_pid()") |
||||||
|
backend_pid = cursor.fetchone()[0] |
||||||
|
barrier.wait(timeout=15) |
||||||
|
try: |
||||||
|
result = services.commit_preview(batch_id=batch.pk, revision=batch.revision, actor=ACTOR) |
||||||
|
return backend_pid, "success", result.result |
||||||
|
except DomainError as error: |
||||||
|
return backend_pid, "failure", error.code |
||||||
|
finally: |
||||||
|
connections.close_all() |
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=2) as pool: |
||||||
|
results = list(pool.map(attempt, batches)) |
||||||
|
assert len({item[0] for item in results}) == 2 |
||||||
|
return results |
||||||
|
|
||||||
|
|
||||||
|
def test_two_commits_of_same_batch_return_same_result_without_duplicate_audit(): |
||||||
|
bank = create_account(name="Synthetic concurrent bank", kind="BANK", actor=ACTOR) |
||||||
|
batch = reviewed(bank) |
||||||
|
previous_audits = set(AuditEvent.objects.values_list("pk", flat=True)) |
||||||
|
results = race([batch, batch]) |
||||||
|
assert [item[1] for item in results] == ["success", "success"] |
||||||
|
assert results[0][2] == results[1][2] |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
|
assert list(AuditEvent.objects.exclude(pk__in=previous_audits).values_list("outcome", flat=True)) == ["SUCCESS"] |
||||||
|
|
||||||
|
|
||||||
|
def test_identical_batches_have_one_success_and_one_stale_review(): |
||||||
|
bank = create_account(name="Synthetic concurrent bank", kind="BANK", actor=ACTOR) |
||||||
|
batches = [reviewed(bank), reviewed(bank)] |
||||||
|
previous_audits = set(AuditEvent.objects.values_list("pk", flat=True)) |
||||||
|
results = race(batches) |
||||||
|
assert sorted(item[1] for item in results) == ["failure", "success"] |
||||||
|
assert next(item[2] for item in results if item[1] == "failure") == "STALE_PREVIEW" |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
|
for batch in batches: |
||||||
|
batch.refresh_from_db() |
||||||
|
assert sorted(batch.state for batch in batches) == ["COMMITTED", "PREVIEW"] |
||||||
|
assert sorted(AuditEvent.objects.exclude(pk__in=previous_audits).values_list("outcome", flat=True)) == ["FAILURE", "SUCCESS"] |
||||||
@ -0,0 +1,127 @@ |
|||||||
|
"""All examples in this module are invented synthetic financial records.""" |
||||||
|
|
||||||
|
from decimal import localcontext, Inexact |
||||||
|
from pathlib import Path |
||||||
|
|
||||||
|
import pytest |
||||||
|
|
||||||
|
from ingestion.parsers import ParseError, parse_document |
||||||
|
from ingestion.normalization import normalize |
||||||
|
|
||||||
|
|
||||||
|
HEADER = 'Data,Dependencia,Historico,Documento,Tipo,Valor\n' |
||||||
|
|
||||||
|
|
||||||
|
def bank(rows, encoding='utf-8'): |
||||||
|
return parse_document((HEADER + rows).encode(encoding), 'BANK_CSV_V1') |
||||||
|
|
||||||
|
|
||||||
|
def card_line(day='15.02.2026', body='LOJA INVENTADA', debit='12,34', credit='0,00', partial=False): |
||||||
|
return day + body.ljust(43 if partial else 37) + 'BR' + f' {debit} {credit}' |
||||||
|
|
||||||
|
|
||||||
|
def card(text, partial=False, context=None): |
||||||
|
return normalize(parse_document(text.encode(), 'OUROCARD_PARTIAL_V1' if partial else 'OUROCARD_FULL_V1', context)) |
||||||
|
|
||||||
|
|
||||||
|
def test_csv_quotes_signs_balances_encoding_and_physical_locators(): |
||||||
|
result = bank('01/02/2026,,"Café, inventado",,,-12.34\n02/02/2026,,S A L D O,,,22.00\n\n03/02/2026,,RECEITA,,,23.00\n', 'latin-1') |
||||||
|
rows = normalize(result) |
||||||
|
assert result.encoding == 'latin-1' |
||||||
|
assert sum(result.skipped.values()) == 2 |
||||||
|
assert [r.to_dict()['locator'] for r in rows] == ['line:2', 'line:5'] |
||||||
|
assert rows[0].to_dict() == dict(locator='line:2', section='1', posting_date='2026-02-01', occurrence_date='2026-02-01', description='Café, inventado', amount='-12.34', kind='EXPENSE', installment_number=None, installment_total=None, errors=[]) |
||||||
|
assert rows[1].kind == 'INCOME' |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('data,profile,code', [(b'\x00','BANK_CSV_V1','NUL_BYTE'), (b'x','OFX','UNSUPPORTED_PROFILE'), (b'a,b,c\n','BANK_CSV_V1','INVALID_HEADER'), (b'x' * (10*1024*1024+1),'BANK_CSV_V1','INPUT_TOO_LARGE')], ids=['nul','unsupported','header','size']) |
||||||
|
def test_file_errors(data, profile, code): |
||||||
|
with pytest.raises(ParseError) as exc: |
||||||
|
parse_document(data, profile) |
||||||
|
assert exc.value.code == code |
||||||
|
|
||||||
|
|
||||||
|
def test_bad_csv_rows_are_reviewable_without_raw_diagnostics(): |
||||||
|
rows = normalize(bank('31/02/2026,,SECRET,,,NaN\nnot,a,row\n')) |
||||||
|
assert set(rows[0].errors) == {'INVALID_DATE', 'INVALID_AMOUNT'} |
||||||
|
assert rows[0].description == 'SECRET' |
||||||
|
assert rows[1].errors == ('INVALID_ROW',) |
||||||
|
assert rows[1].description == 'row' |
||||||
|
assert 'SECRET' not in str(rows[0].errors) |
||||||
|
|
||||||
|
|
||||||
|
def test_full_card_sections_installments_credit_and_payment(): |
||||||
|
rows = card('1 - SYNTHETIC OWNER\n' + card_line(body='LOJA PARC 2/3') + '\n2 - SECOND INVENTED OWNER\n' + card_line(debit='-8,11') + '\n15.02.2026PGTO DEBITO CONTA 200211 33,00 0,00\nSaldo Anterior 20,00\nunknown text') |
||||||
|
assert rows[0].section == '1' |
||||||
|
assert rows[0].installment_number == 2 and rows[0].installment_total == 3 |
||||||
|
assert rows[1].section == '2' and rows[1].amount == '8.11' and rows[1].kind == 'REFUND' |
||||||
|
assert rows[2].amount == '33.00' and rows[2].kind == 'TRANSFER' |
||||||
|
assert rows[3].errors == ('UNRECOGNIZED_LINE',) |
||||||
|
assert rows[3].description == 'unknown text' |
||||||
|
|
||||||
|
|
||||||
|
def test_card_ambiguity_bad_shapes_and_installments_block(): |
||||||
|
assert 'AMBIGUOUS_AMOUNT' in card(card_line(debit='2,00', credit='1,00'))[0].errors |
||||||
|
assert 'INVALID_ROW' in card('15.02.2026 MALFORMED')[0].errors |
||||||
|
assert 'INVALID_INSTALLMENT' in card(card_line(body='LOJA PARC 3/2'))[0].errors |
||||||
|
assert 'INVALID_AMOUNT' in card(card_line(debit='1.23,00'))[0].errors |
||||||
|
|
||||||
|
|
||||||
|
def test_partial_dates_require_explicit_unique_context_and_retain_snapshot(): |
||||||
|
text = card_line(day='31/12', partial=True) + '\n' + card_line(day='01/01', partial=True) |
||||||
|
context = {'start_date':'2025-12-01','end_date':'2026-01-31'} |
||||||
|
parsed = parse_document(text.encode(), 'OUROCARD_PARTIAL_V1', context) |
||||||
|
context['start_date'] = '2000-01-01' |
||||||
|
assert [r.posting_date for r in normalize(parsed)] == ['2025-12-31','2026-01-01'] |
||||||
|
assert 'MISSING_DATE_CONTEXT' in card(text, True)[0].errors |
||||||
|
assert 'AMBIGUOUS_DATE' in card(card_line(day='01/01',partial=True), True, {'start_date':'2025-01-01','end_date':'2026-01-01'})[0].errors |
||||||
|
assert 'INVALID_DATE_CONTEXT' in card(text, True, {'start_date':'2024-01-01','end_date':'2026-01-01'})[0].errors |
||||||
|
|
||||||
|
|
||||||
|
def test_decimal_context_does_not_change_amounts(): |
||||||
|
with localcontext() as ctx: |
||||||
|
ctx.prec = 2 |
||||||
|
ctx.traps[Inexact] = True |
||||||
|
assert card(card_line(debit='12.345,67'))[0].amount == '-12345.67' |
||||||
|
assert card('15.02.2026PGTO DEBITO CONTA 200211 -12.345,67 0,00')[0].amount == '12345.67' |
||||||
|
|
||||||
|
|
||||||
|
def test_csv_multiline_locator_and_bom(): |
||||||
|
parsed = parse_document(('\ufeff' + HEADER + '01/02/2026,,"two\nlines",,,1.00\n02/02/2026,,next,,,2.00\n').encode(), 'BANK_CSV_V1') |
||||||
|
assert parsed.encoding == 'utf-8-sig' |
||||||
|
assert [r.locator for r in normalize(parsed)] == ['line:2','line:4'] |
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_row_limit(): |
||||||
|
with pytest.raises(ParseError, match='TOO_MANY_ITEMS'): |
||||||
|
bank('01/02/2026,,INVENTED,,,1.00\n' * 20001) |
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_quoting_is_file_error_and_header_positions_are_fixed(): |
||||||
|
with pytest.raises(ParseError, match='INVALID_CSV'): |
||||||
|
bank('01/02/2026,,"unterminated,,,1.00') |
||||||
|
with pytest.raises(ParseError, match='INVALID_HEADER'): |
||||||
|
parse_document(b'Valor,Unused,Historico,Unused,Unused,Data\n', 'BANK_CSV_V1') |
||||||
|
|
||||||
|
|
||||||
|
def test_partial_leap_date_and_invalid_ranges(): |
||||||
|
text = card_line(day='29/02', partial=True) |
||||||
|
assert card(text, True, {'start_date':'2024-01-01','end_date':'2024-12-31'})[0].posting_date == '2024-02-29' |
||||||
|
assert card(text, True, {'start_date':'2025-01-01','end_date':'2025-12-31'})[0].errors == ('INVALID_DATE',) |
||||||
|
assert card(text, True, {'start_date':'2026-01-01','end_date':'2025-12-31'})[0].errors == ('INVALID_DATE_CONTEXT',) |
||||||
|
|
||||||
|
|
||||||
|
def test_nonzero_payment_secondary_column_is_ambiguous(): |
||||||
|
assert card('15.02.2026PGTO DEBITO CONTA 200211 33,00 2,00')[0].errors == ('AMBIGUOUS_AMOUNT',) |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('filename,profile,amounts', [ |
||||||
|
('bank.csv', 'BANK_CSV_V1', ['-12.34','100.00']), |
||||||
|
('full.txt', 'OUROCARD_FULL_V1', ['-12.34','8.11','33.00']), |
||||||
|
('partial.txt', 'OUROCARD_PARTIAL_V1', ['-12.34','-12.34']), |
||||||
|
]) |
||||||
|
def test_synthetic_fixture_profiles(filename, profile, amounts): |
||||||
|
data = (Path(__file__).parent / 'fixtures' / 'synthetic' / 'ingestion' / filename).read_bytes() |
||||||
|
rows = normalize(parse_document(data, profile, {'start_date':'2025-12-01','end_date':'2026-01-31'})) |
||||||
|
assert [row.amount for row in rows] == amounts |
||||||
|
assert all(not row.errors for row in rows) |
||||||
@ -0,0 +1,264 @@ |
|||||||
|
"""Synthetic acceptance cases for explicit review and atomic import persistence.""" |
||||||
|
|
||||||
|
from datetime import date |
||||||
|
from decimal import Decimal |
||||||
|
from hashlib import sha256 |
||||||
|
|
||||||
|
import pytest |
||||||
|
|
||||||
|
from finance.models import AuditEvent, InstallmentPlan, SourceRecord, Transaction, Transfer |
||||||
|
from finance.services import Actor, DomainError, archive, create_account, create_category, create_payee, record_transaction |
||||||
|
from ingestion import services |
||||||
|
|
||||||
|
pytestmark = pytest.mark.django_db(transaction=True) |
||||||
|
ACTOR = Actor(kind="AGENT", identifier="synthetic-import-test") |
||||||
|
HEADER = "Data,Documento,Historico,Agencia,Lote,Valor\n" |
||||||
|
ROW = "15/01/2026,1,Synthetic purchase,0,0,-12.50\n" |
||||||
|
DATA = (HEADER + ROW).encode() |
||||||
|
|
||||||
|
|
||||||
|
def account(name="Synthetic import bank"): |
||||||
|
return create_account(name=name, kind="BANK", actor=ACTOR) |
||||||
|
|
||||||
|
|
||||||
|
def preview(bank, data=DATA): |
||||||
|
return services.create_preview(data=data, profile="BANK_CSV_V1", section_mapping={"1": str(bank.pk)}, actor=ACTOR) |
||||||
|
|
||||||
|
|
||||||
|
def decide(batch, **fields): |
||||||
|
return services.apply_decisions(batch_id=batch.pk, revision=batch.revision, actor=ACTOR, |
||||||
|
decisions=[{"locator": item.locator, "action": "CREATE", **fields} |
||||||
|
for item in batch.items.all()]) |
||||||
|
|
||||||
|
|
||||||
|
def commit(batch): |
||||||
|
return services.commit_preview(batch_id=batch.pk, revision=batch.revision, actor=ACTOR) |
||||||
|
|
||||||
|
|
||||||
|
def assert_no_financial_rows(): |
||||||
|
for model in (Transaction, SourceRecord, InstallmentPlan, Transfer): |
||||||
|
assert model.objects.count() == 0 |
||||||
|
|
||||||
|
|
||||||
|
def test_preview_and_decisions_are_inert_then_commit_preserves_source_and_audit(): |
||||||
|
batch = preview(account()) |
||||||
|
assert batch.state == "PREVIEW" |
||||||
|
assert batch.revision == 1 |
||||||
|
assert_no_financial_rows() |
||||||
|
review = services.export_review(batch_id=batch.pk) |
||||||
|
assert review["batch_id"] == str(batch.pk) |
||||||
|
assert review["revision"] == 1 |
||||||
|
assert len(review["items"]) == 1 |
||||||
|
assert review["items"][0]["candidate"]["amount"] == "-12.50" |
||||||
|
batch = decide(batch) |
||||||
|
assert batch.revision == 2 |
||||||
|
assert_no_financial_rows() |
||||||
|
previous_audits = set(AuditEvent.objects.values_list("pk", flat=True)) |
||||||
|
result = commit(batch) |
||||||
|
assert result.state == "COMMITTED" |
||||||
|
assert result.revision == batch.revision |
||||||
|
row = Transaction.objects.get() |
||||||
|
assert row.amount == Decimal("-12.50") |
||||||
|
assert row.posting_date == row.occurrence_date == date(2026, 1, 15) |
||||||
|
source = SourceRecord.objects.get() |
||||||
|
assert source.transaction_id == row.pk |
||||||
|
assert source.document_hash == sha256(DATA).hexdigest() |
||||||
|
assert source.item_locator == batch.items.get().locator |
||||||
|
events = list(AuditEvent.objects.exclude(pk__in=previous_audits)) |
||||||
|
assert len(events) == 1 |
||||||
|
assert events[0].outcome == "SUCCESS" |
||||||
|
assert events[0].actor_identifier == ACTOR.identifier |
||||||
|
audit_count = AuditEvent.objects.count() |
||||||
|
repeated = commit(batch) |
||||||
|
assert repeated.result == result.result |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
|
assert AuditEvent.objects.count() == audit_count |
||||||
|
|
||||||
|
|
||||||
|
def test_every_row_requires_explicit_decision_and_rejected_commit_is_audited(): |
||||||
|
batch = preview(account()) |
||||||
|
previous_audits = set(AuditEvent.objects.values_list("pk", flat=True)) |
||||||
|
with pytest.raises(DomainError) as caught: |
||||||
|
commit(batch) |
||||||
|
assert caught.value.code |
||||||
|
assert_no_financial_rows() |
||||||
|
batch.refresh_from_db() |
||||||
|
assert batch.state == "PREVIEW" |
||||||
|
event = AuditEvent.objects.exclude(pk__in=previous_audits).get() |
||||||
|
assert event.outcome == "FAILURE" |
||||||
|
assert event.error_code == caught.value.code |
||||||
|
assert event.actor_identifier == ACTOR.identifier |
||||||
|
|
||||||
|
|
||||||
|
def test_malformed_row_can_only_be_omitted_with_explicit_skip(): |
||||||
|
batch = preview(account(), (HEADER + "31/02/2026,1,Synthetic invalid date,0,0,-12.50\n").encode()) |
||||||
|
assert batch.items.get().issues |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
batch = decide(batch, action="SKIP", reason="MALFORMED") |
||||||
|
assert commit(batch).state == "COMMITTED" |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_exact_reimport_requires_already_imported_skip(): |
||||||
|
bank = account() |
||||||
|
commit(decide(preview(bank))) |
||||||
|
batch = preview(bank) |
||||||
|
assert "ALREADY_IMPORTED" in str(batch.items.get().duplicates) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
batch = decide(batch, action="SKIP", reason="ALREADY_IMPORTED") |
||||||
|
assert commit(batch).state == "COMMITTED" |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
|
|
||||||
|
|
||||||
|
def test_semantic_duplicates_require_acknowledgement_but_can_be_legitimate(): |
||||||
|
bank = account() |
||||||
|
commit(decide(preview(bank))) |
||||||
|
# The document ID changes the physical source without changing financial fields. |
||||||
|
batch = preview(bank, DATA.replace(b",1,", b",2,")) |
||||||
|
assert batch.items.get().duplicates |
||||||
|
batch = decide(batch) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
batch = decide(batch, acknowledge_duplicate=True) |
||||||
|
commit(batch) |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 2 |
||||||
|
|
||||||
|
|
||||||
|
def test_match_attaches_provenance_without_changing_existing_annotations(): |
||||||
|
bank = account() |
||||||
|
category = create_category(name="Synthetic category", kind="EXPENSE", actor=ACTOR) |
||||||
|
payee = create_payee(name="Synthetic payee", actor=ACTOR) |
||||||
|
row = record_transaction(account_id=bank.pk, amount=Decimal("-12.50"), kind="EXPENSE", |
||||||
|
posting_date=date(2026, 1, 15), occurrence_date=date(2026, 1, 15), |
||||||
|
description="Synthetic purchase", category_id=category.pk, |
||||||
|
payee_id=payee.pk, actor=ACTOR) |
||||||
|
before = (row.description, row.category_id, row.payee_id, row.updated_at) |
||||||
|
batch = decide(preview(bank), action="MATCH", transaction_id=str(row.pk)) |
||||||
|
commit(batch) |
||||||
|
row.refresh_from_db() |
||||||
|
assert (row.description, row.category_id, row.payee_id, row.updated_at) == before |
||||||
|
assert Transaction.objects.count() == 1 |
||||||
|
assert SourceRecord.objects.get().transaction_id == row.pk |
||||||
|
|
||||||
|
|
||||||
|
def test_cancellation_and_stale_revision_cannot_commit(): |
||||||
|
batch = preview(account()) |
||||||
|
old_revision = batch.revision |
||||||
|
batch = decide(batch) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
services.commit_preview(batch_id=batch.pk, revision=old_revision, actor=ACTOR) |
||||||
|
batch = services.cancel_preview(batch_id=batch.pk, revision=batch.revision, actor=ACTOR) |
||||||
|
assert batch.state == "CANCELLED" |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_new_duplicate_makes_review_stale_and_refresh_requires_new_consent(): |
||||||
|
bank = account() |
||||||
|
batch = decide(preview(bank)) |
||||||
|
record_transaction(account_id=bank.pk, amount=Decimal("-12.50"), kind="EXPENSE", |
||||||
|
posting_date=date(2026, 1, 15), occurrence_date=date(2026, 1, 15), |
||||||
|
description="Synthetic purchase", actor=ACTOR) |
||||||
|
with pytest.raises(DomainError) as caught: |
||||||
|
commit(batch) |
||||||
|
assert caught.value.code == "STALE_PREVIEW" |
||||||
|
assert SourceRecord.objects.count() == 0 |
||||||
|
refreshed = services.refresh_preview(batch_id=batch.pk, revision=batch.revision, actor=ACTOR) |
||||||
|
assert refreshed.revision == batch.revision + 1 |
||||||
|
assert all(not item.decision for item in refreshed.items.all()) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(refreshed) |
||||||
|
commit(decide(refreshed, acknowledge_duplicate=True)) |
||||||
|
assert Transaction.objects.count() == 2 |
||||||
|
assert SourceRecord.objects.count() == 1 |
||||||
|
|
||||||
|
|
||||||
|
def test_archiving_destination_invalidates_review(): |
||||||
|
bank = account() |
||||||
|
batch = decide(preview(bank)) |
||||||
|
archive(bank, actor=ACTOR) |
||||||
|
with pytest.raises(DomainError) as caught: |
||||||
|
commit(batch) |
||||||
|
assert caught.value.code == "STALE_PREVIEW" |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_later_row_failure_rolls_back_posting_sources_and_success_audit(monkeypatch): |
||||||
|
batch = decide(preview(account(), (HEADER + ROW + "16/01/2026,2,Synthetic second purchase,0,0,-7.25\n").encode())) |
||||||
|
original = services.record |
||||||
|
calls = [] |
||||||
|
|
||||||
|
def fail_second(**kwargs): |
||||||
|
calls.append(kwargs) |
||||||
|
if len(calls) == 2: |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
|
raise DomainError("synthetic_later_row_failure") |
||||||
|
return original(**kwargs) |
||||||
|
|
||||||
|
monkeypatch.setattr(services, "record", fail_second) |
||||||
|
previous_audits = set(AuditEvent.objects.values_list("pk", flat=True)) |
||||||
|
with pytest.raises(DomainError) as caught: |
||||||
|
commit(batch) |
||||||
|
assert len(calls) == 2 |
||||||
|
assert caught.value.code == "synthetic_later_row_failure" |
||||||
|
assert_no_financial_rows() |
||||||
|
batch.refresh_from_db() |
||||||
|
assert batch.state == "PREVIEW" |
||||||
|
event = AuditEvent.objects.exclude(pk__in=previous_audits).get() |
||||||
|
assert event.outcome == "FAILURE" |
||||||
|
assert event.error_code == caught.value.code |
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("extra", [{"amount": "-1.00"}, {"description": "Replacement"}, {"unexpected": True}]) |
||||||
|
def test_decisions_cannot_change_source_evidence_or_accept_unknown_fields(extra): |
||||||
|
batch = preview(account()) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
decide(batch, **extra) |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_duplicate_locators_are_rejected(): |
||||||
|
batch = preview(account()) |
||||||
|
decision = {"locator": batch.items.get().locator, "action": "CREATE"} |
||||||
|
with pytest.raises(DomainError): |
||||||
|
services.apply_decisions(batch_id=batch.pk, revision=batch.revision, decisions=[decision, decision], actor=ACTOR) |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_installments_share_explicit_new_plan_without_creating_unobserved_rows(): |
||||||
|
card = create_account(name="Synthetic installment card", kind="CREDIT_CARD", actor=ACTOR) |
||||||
|
data = (f"15.01.2026{'Synthetic purchase PARC 2/6':37}BR 12,50 0,00\n" |
||||||
|
f"15.02.2026{'Synthetic purchase PARC 3/6':37}BR 12,50 0,00\n").encode() |
||||||
|
batch = services.create_preview(data=data, profile="OUROCARD_FULL_V1", |
||||||
|
section_mapping={"1": str(card.pk)}, actor=ACTOR) |
||||||
|
batch = decide(batch, new_plan="synthetic-purchase") |
||||||
|
assert_no_financial_rows() |
||||||
|
commit(batch) |
||||||
|
plan = InstallmentPlan.objects.get() |
||||||
|
assert plan.account_id == card.pk |
||||||
|
assert plan.total_installments == 6 |
||||||
|
assert set(Transaction.objects.values_list("installment_number", flat=True)) == {2, 3} |
||||||
|
assert set(Transaction.objects.values_list("installment_plan_id", flat=True)) == {plan.pk} |
||||||
|
assert SourceRecord.objects.count() == 2 |
||||||
|
|
||||||
|
|
||||||
|
def test_applying_decisions_replaces_complete_set_and_missing_rows_block_commit(): |
||||||
|
batch = decide(preview(account(), (HEADER + ROW + "16/01/2026,2,Synthetic second purchase,0,0,-7.25\n").encode())) |
||||||
|
first = batch.items.order_by("locator").first() |
||||||
|
batch = services.apply_decisions(batch_id=batch.pk, revision=batch.revision, actor=ACTOR, |
||||||
|
decisions=[{"locator": first.locator, "action": "CREATE"}]) |
||||||
|
assert not batch.items.exclude(pk=first.pk).get().decision |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
assert_no_financial_rows() |
||||||
|
|
||||||
|
|
||||||
|
def test_source_already_attached_to_another_account_cannot_be_claimed(): |
||||||
|
commit(decide(preview(account()))) |
||||||
|
batch = decide(preview(account("Synthetic second import bank"))) |
||||||
|
with pytest.raises(DomainError): |
||||||
|
commit(batch) |
||||||
|
assert Transaction.objects.count() == SourceRecord.objects.count() == 1 |
||||||
Loading…
Reference in new issue