11 KiB

Raw Blame History

Guided Local Imports Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add an Imports page that scans the ignored local folder, maps files to accounts, and automatically commits only clean new transactions.

Architecture: A new ingestion-facing service discovers safe direct files and derives partial-card date context without persisting source contents. A second orchestration service creates the existing preview, applies only deterministic CREATE and exact-source SKIP decisions, and commits each ready file independently. Server-rendered views hold signed scan/preflight state and display safe summaries.

Tech Stack: Python, Django, PostgreSQL, Django signed values, existing ingestion services, pytest, Docker Compose.

Spec: docs/superpowers/specs/2026-09-09-guided-local-import-design.md

Global Constraints

Scan direct files only under IMPORT_ROOT; reject symlinks, traversal, unsupported inputs, and files over 10 MiB.
Never display or commit source descriptions, amounts, source rows, raw paths, parser exceptions, or database errors in UI messages, logs, or tests.
Support only BANK_CSV_V1, OUROCARD_FULL_V1, and OUROCARD_PARTIAL_V1; OFX and PDF remain unsupported.
Use active BRL accounts for every submitted source section; never infer mapping from owner names or filenames.
Preserve parsing, normalization, validation, duplicate detection, preview, decision, and persistence boundaries.
Clean rows use CREATE; exact existing source rows use SKIP / ALREADY_IMPORTED; all other issues prevent only that file from committing.
Partial-card context comes only from a validated creation date at text line 21; dates later than that month use the preceding year.
Use only synthetic test data and never commit local financial documents, review files, exports, backups, or credentials.

Task 1: Discover safe files and derive partial-card context

Files:

Create: ingestion/guided_imports.py
Modify: ingestion/parsers.py
Create: tests/test_guided_import_discovery.py
Modify: tests/test_import_parsers.py

Interfaces:

Produces DiscoveredFile(name: str, document_hash: str, profile: str | None, sections: tuple[str, ...], code: str | None).
Produces discover_imports() -> tuple[DiscoveredFile, ...] and partial_context_from_creation_line(data: bytes) -> dict[str, str].
Step 1: Write failing pure discovery tests.

Use override_settings(IMPORT_ROOT=tmp_path) and synthetic bytes. Assert direct supported files receive a profile and section list; nested files, symlinks, oversize files, PDFs, OFX, and invalid layouts return only safe code/status. Assert no result has a source-text field.
```
rows = discover_imports()
assert [(row.name, row.profile, row.sections, row.code) for row in rows] == [
    ("bank.csv", "BANK_CSV_V1", ("1",), None),
    ("unknown.ofx", None, (), "UNSUPPORTED_FILE"),
]
```
Add parser tests for valid synthetic line 21 creation date, absent/malformed line 21, and a January creation date that resolves 31/12 to the previous year.
Step 2: Verify RED.

Run: docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py

Expected: FAIL because the discovery module and context extractor do not exist.
Step 3: Implement constrained discovery and context extraction.

discover_imports resolves settings.IMPORT_ROOT, iterates only root.iterdir(), ignores non-files and symlinks, reads no more than MAX_BYTES + 1, hashes data with SHA-256, and determines a profile by attempting the supported pure grammars without persisting parse output. It returns sorted filename-only results.
```
@dataclass(frozen=True)
class DiscoveredFile:
    name: str
    document_hash: str
    profile: str | None
    sections: tuple[str, ...]
    code: str | None
```
Add partial_context_from_creation_line beside the pure parser helpers. It decodes through the existing encoding rules, reads exactly line 21, validates the approved creation-date layout, and returns an ISO start_date/end_date interval no longer than 366 days. Keep the existing explicit parse_document(..., context=...) contract unchanged.

Step 4: Verify GREEN and commit.

Run:

docker compose run --rm test pytest -q tests/test_guided_import_discovery.py tests/test_import_parsers.py
docker compose run --rm web python manage.py check

Inspect staged content for real data, then commit:

git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: discover local import files"

Task 2: Preflight and commit deterministic imports

Files:

Modify: ingestion/guided_imports.py
Create: tests/test_guided_import_workflow.py

Interfaces:

Consumes DiscoveredFile, Actor, and dict[str, str] section mappings.
Produces PreparedImport(name, document_hash, profile, sections, state, batch_id, revision, counts, code) and prepare_import(...) / commit_prepared_import(...).
Step 1: Write failing orchestration tests.

Create synthetic active BRL accounts and source bytes in a temporary import root. Test that a clean file preflights to READY, an exact second import preflights to ALREADY_IMPORTED, an ambiguous duplicate is NEEDS_ATTENTION, and one exceptional file does not prevent another ready file from committing.
```
prepared = prepare_import(discovered=bank, section_mapping={"1": str(account.id)}, actor=ACTOR)
assert prepared.state == "READY"
committed = commit_prepared_import(prepared=prepared, actor=ACTOR)
assert committed.counts["created"] == 2
```
Step 2: Verify RED.

Run: docker compose run --rm test pytest -q tests/test_guided_import_workflow.py

Expected: FAIL because preparation and automatic decisions do not exist.
Step 3: Implement one service that composes existing lifecycle calls.

Re-read the file and require its SHA-256 to equal discovered.document_hash before acting. Validate section keys exactly match and map to active BRL Account records. For partial files obtain validated context from Task 1. Call create_preview, inspect its items, and build complete decisions:
```
if item.issues == ["ALREADY_IMPORTED"]:
    decision = {"locator": item.locator, "action": "SKIP", "reason": "ALREADY_IMPORTED"}
elif not item.issues and not item.duplicates["existing"] and not item.duplicates["within"]:
    decision = {"locator": item.locator, "action": "CREATE"}
else:
    cancel_preview(batch_id=batch.pk, revision=batch.revision, actor=actor)
    return PreparedImport(..., state="NEEDS_ATTENTION", code="REVIEW_REQUIRED")
```
Apply the complete decision list only for ready/already-imported batches. commit_prepared_import requires the stored batch id/revision and delegates to commit_preview; it never writes transactions directly. Return counts and safe codes only. File changes, stale batches, domain failures, and bad mappings yield a safe exception result and cancel any unfinished preview.

Step 4: Verify GREEN and commit.

Run:

docker compose run --rm test pytest -q tests/test_guided_import_workflow.py tests/test_import_workflow.py tests/test_import_concurrency.py
docker compose run --rm web python manage.py check

Inspect staged content for sensitive data, then commit:

git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: prepare guided imports"

Task 3: Add the Imports page and owner workflow coverage

Files:

Modify: config/urls.py
Modify: core/context.py
Modify: core/forms.py
Modify: core/views.py
Create: core/templates/core/imports/index.html
Create: tests/test_guided_import_views.py
Modify: tests/test_mvp_acceptance.py
Modify: README.md
Modify: docs/local-workflows.md

Interfaces:

Route: imports-index at /imports/.
View actions: scan, prepare, and commit.
Signed payload contains only filename, hash, profile, section identifiers, prepared batch id, and revision.
Step 1: Write failing web tests.

Use a temporary synthetic IMPORT_ROOT. Cover navigation, CSRF, scan results with safe filename/profile/status only, active-BRL account selectors, mapping submission, ready-file commit, exact reimport summary, one exception plus one successful file, and altered/stale signed state rejection.
```
response = client.post(reverse("imports-index"), {"action": "scan"})
assert "bank.csv" in response.content.decode()
assert "synthetic description" not in response.content.decode()
```
Step 2: Verify RED.

Run: docker compose run --rm test pytest -q tests/test_guided_import_views.py

Expected: FAIL because the page, routes, forms, and signed workflow do not exist.
Step 3: Implement server-rendered scan, mapping, and commit actions.

Add Import navigation and imports-index. scan calls discover_imports and signs safe discovery metadata. prepare revalidates it, receives one typed active-account field per discovered section, and calls prepare_import independently for every mapped file. commit accepts only signed ready batch id/revision values and calls commit_prepared_import per file.
```
token = signing.dumps(payload, salt="guided-imports")
payload = signing.loads(request.POST["scan_token"], salt="guided-imports", max_age=900)
```
Render only filename, profile, section label, mapping selection, state, safe code, and counts. Do not display candidate descriptions, amounts, source paths, raw exceptions, or batch audit details. Use Actor("HUMAN", "local-owner") and redirect after each successful POST.

Step 4: Document and verify the owner path.

Replace the primary manual-import instructions in README.md and docs/local-workflows.md with: copy files to imports/, open Imports, scan, map accounts, import ready files, and resolve listed exceptions. Retain the management command as the advanced manual workflow.

Add one synthetic MVP acceptance flow through scan, mapping, automatic import, and exact reimport skip. Run:

docker compose run --rm test pytest -q tests/test_guided_import_views.py tests/test_mvp_acceptance.py
docker compose build
docker compose run --rm test pytest -q
docker compose run --rm web python manage.py migrate --check
docker compose run --rm web python manage.py check
docker compose run --rm permissions
docker compose up -d web
Invoke-RestMethod http://127.0.0.1:8000/health/ | ConvertTo-Json -Compress
git diff --check

Step 5: Commit the UI, documentation, and acceptance slice.

Inspect staged paths and content for financial data and credentials, then commit:
```
git -c user.name=Codex -c user.email=codex@localhost commit -m "feat: add guided import page"
```

11 KiB Raw Blame History

Guided Local Imports Implementation Plan

Global Constraints

Task 1: Discover safe files and derive partial-card context

Task 2: Preflight and commit deterministic imports

Task 3: Add the Imports page and owner workflow coverage

11 KiB

Raw Blame History