diff --git a/docs/superpowers/plans/2026-09-07-local-mvp-workflows.md b/docs/superpowers/plans/2026-09-07-local-mvp-workflows.md new file mode 100644 index 0000000..7211466 --- /dev/null +++ b/docs/superpowers/plans/2026-09-07-local-mvp-workflows.md @@ -0,0 +1,453 @@ +# Local MVP Workflows Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Deliver a local server-rendered workflow for managing and categorizing transactions, viewing monthly reports, exporting user-owned data, and creating and verifying PostgreSQL backups. + +**Architecture:** `finance.queries` and `finance.reports` own read behavior, existing audited finance services own every mutation, and `core` owns HTTP forms, views, URLs, and templates. A versioned export service is shared by browser and command entry points; backup commands wrap PostgreSQL 17 client tools and verify restores only against a separate explicitly marked database. + +**Tech Stack:** Python 3.13, Django 5.2, PostgreSQL 17, psycopg 3, Django templates/forms, pytest and pytest-django, Docker Compose. + +**Spec:** `docs/superpowers/specs/2026-09-07-local-mvp-workflows-design.md` + +## Global Constraints + +- All monthly reports use `posting_date`, an inclusive calendar month, and exactly one currency. +- `INCOME` is inflow; `EXPENSE` is spending; `REFUND` reduces spending in its posting month; `TRANSFER` and `OPENING` are excluded from cash flow and spending. +- Account balances include every entry through month end, including openings and transfer legs, and are never summed across currencies. +- Browser writes must call audited `finance.services` with `Actor(kind="HUMAN", identifier="local-owner")`; views never save financial models directly. +- Transactions and source evidence cannot be deleted or have immutable financial fields rewritten. +- All mutation forms use POST and Django CSRF protection. +- Export and backup artifacts must remain under ignored `exports/` and `backups/`, never overwrite existing files, and never publish partial files. +- Terminal output, HTTP errors, tests, and committed fixtures must contain only synthetic data and safe error codes. +- PostgreSQL backup clients must use major version 17 and receive passwords through environment variables, never command arguments. +- OFX, bank APIs, investments, forecasting, receipt recognition, mobile clients, hosted multi-user operation, and automatic or AI categorization remain excluded. + +--- + +### Task 1: Query, indexing, and report services + +**Files:** +- Create: `finance/queries.py` +- Create: `finance/reports.py` +- Create: `finance/migrations/0002_transaction_report_indexes.py` +- Test: `tests/finance/test_queries.py` +- Test: `tests/finance/test_reports.py` + +**Interfaces:** +- Consumes: `finance.models.Account`, `Category`, `Payee`, and `Transaction`. +- Produces: `TransactionFilters.from_mapping(mapping)`, `filtered_transactions(filters)`, `available_currencies()`, `monthly_summary(month, currency)`, `spending_by_category(month, currency)`, and `account_balances(month, currency)`. +- `month` is a `datetime.date` whose day must be `1`; currency is three uppercase ASCII letters. +- Report return types are frozen dataclasses containing `Decimal` values. + +- [ ] **Step 1: Write failing query tests** + +```python +def test_filtered_transactions_combines_filters_and_orders_stably(): + filters = TransactionFilters.from_mapping({ + "date_from": "2026-01-01", "date_to": "2026-01-31", + "currency": "BRL", "uncategorized": "yes", + }) + rows = list(filtered_transactions(filters)) + assert [row.pk for row in rows] == [newer.pk, older.pk] + +def test_invalid_filter_is_explicit(): + with pytest.raises(QueryInputError) as caught: + TransactionFilters.from_mapping({"date_from": "31/01/2026"}) + assert caught.value.errors == {"date_from": "invalid_date"} +``` + +- [ ] **Step 2: Run the query tests and confirm imports fail because `finance.queries` does not exist** + +Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py` + +- [ ] **Step 3: Implement validated filters and deterministic query ordering** + +Implement a frozen `TransactionFilters` dataclass with optional dates and UUIDs, kind, currency, and tri-state uncategorized value. Parse ISO dates and UUIDs once, return field error codes through `QueryInputError.errors`, use `select_related`, and order by `-posting_date`, `-id`. + +- [ ] **Step 4: Run query tests and confirm they pass** + +Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py` + +- [ ] **Step 5: Write failing report-semantic tests** + +```python +def test_monthly_summary_counts_card_purchase_once_and_refund_in_posting_month(): + result = monthly_summary(date(2026, 1, 1), "BRL") + assert result.inflow == Decimal("1000.00") + assert result.expense == Decimal("300.00") + assert result.refund == Decimal("50.00") + assert result.net_spending == Decimal("250.00") + assert result.net_activity == Decimal("750.00") + +def test_spending_has_uncategorized_bucket_and_never_combines_currency(): + rows = spending_by_category(date(2026, 1, 1), "BRL") + assert [(row.category_name, row.net_spending) for row in rows] == [ + ("Synthetic Food", Decimal("80.00")), + ("Uncategorized", Decimal("20.00")), + ] +``` + +- [ ] **Step 6: Run report tests and confirm failure because report functions are absent** + +Run: `docker compose run --rm test pytest -q tests/finance/test_reports.py` + +- [ ] **Step 7: Implement monthly summaries, category spending, balances, and currency choices** + +Use conditional PostgreSQL aggregates with explicit `Decimal("0.00")` defaults. Validate month and currency. Group nullable categories into an explicit Uncategorized result without manufacturing a Category row. Sort category results by descending net spending then category name; sort balances by account name then UUID. + +- [ ] **Step 8: Add report indexes through a migration** + +Add named indexes on `Transaction(posting_date)` and `Transaction(account, posting_date)`. Do not alter financial constraints. + +- [ ] **Step 9: Run focused tests and migration drift check** + +Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py tests/finance/test_reports.py` + +Run: `docker compose run --rm manage makemigrations --check --dry-run` + +- [ ] **Step 10: Commit the query and reporting slice** + +```powershell +git add finance/queries.py finance/reports.py finance/migrations/0002_transaction_report_indexes.py tests/finance/test_queries.py tests/finance/test_reports.py +git diff --cached --name-only +git diff --cached +git commit -m "feat: add financial queries and monthly reports" +``` + +### Task 2: Local web shell and audited management workflows + +**Files:** +- Create: `core/forms.py` +- Create: `core/context.py` +- Create: `core/templates/core/base.html` +- Create: `core/templates/core/transactions/list.html` +- Create: `core/templates/core/transactions/detail.html` +- Create: `core/templates/core/transactions/form.html` +- Create: `core/templates/core/references/list.html` +- Create: `core/templates/core/references/form.html` +- Create: `core/static/core/app.css` +- Modify: `core/views.py` +- Modify: `config/urls.py` +- Modify: `config/settings.py` +- Test: `tests/test_web_workflows.py` + +**Interfaces:** +- Consumes: Task 1 query APIs and existing `finance.services` mutation APIs. +- Produces named URLs `home`, `transaction-list`, `transaction-detail`, `transaction-create`, `transaction-annotate`, `account-list`, `account-create`, `account-archive`, `category-list`, `category-create`, `category-archive`, `payee-list`, `payee-create`, and `payee-archive`. +- Produces forms that return validated scalar values and UUIDs but never save models. + +- [ ] **Step 1: Write failing request tests for navigation, list filters, detail, and CSRF** + +```python +def test_root_and_transaction_pages_are_discoverable(client): + assert client.get(reverse("home")).status_code == 200 + page = client.get(reverse("transaction-list"), {"currency": "BRL"}) + assert page.status_code == 200 + assert "Synthetic purchase" in page.content.decode() + +def test_mutation_requires_csrf(): + guarded = Client(enforce_csrf_checks=True) + response = guarded.post(reverse("category-create"), {"name": "Synthetic Food", "kind": "EXPENSE"}) + assert response.status_code == 403 +``` + +- [ ] **Step 2: Run web workflow tests and confirm missing routes fail** + +Run: `docker compose run --rm test pytest -q tests/test_web_workflows.py` + +- [ ] **Step 3: Enable template, static, CSRF, session, and message infrastructure** + +Add Django sessions and messages apps/middleware, template backend with request and message context processors, `STATIC_URL`, and signed-cookie sessions. Run the sessions migration; no financial data is stored in the session. + +- [ ] **Step 4: Implement forms and request-to-domain error mapping** + +Build plain `forms.Form` classes for manual transactions, annotations, accounts, categories, and payees. Query active references for choices. Convert `DomainError.code` into a fixed safe message dictionary; unknown codes map to `Unable to save this change.` + +- [ ] **Step 5: Implement transaction list/detail/create/annotate views** + +Use `finance.queries` for reads, Django `Paginator` with 50 rows per page, and existing finance services for writes. Derive `LOCAL_ACTOR = Actor(kind="HUMAN", identifier="local-owner")` server-side. Redirect after successful POST. + +- [ ] **Step 6: Implement reference list/create/archive views** + +Use services only for mutations. Archive endpoints accept POST only and render archived values as historical/inactive on list pages. + +- [ ] **Step 7: Add accessible templates and minimal responsive CSS** + +Provide one `h1`, associated labels, error summaries, visible focus styles, tabular headings, empty states, and navigation on every page. Keep all core workflows functional without JavaScript. + +- [ ] **Step 8: Run request and existing domain tests** + +Run: `docker compose run --rm test pytest -q tests/test_web_workflows.py tests/finance/test_services.py` + +- [ ] **Step 9: Commit the audited web workflow slice after staged privacy inspection** + +```powershell +git add config/settings.py config/urls.py core tests/test_web_workflows.py +git diff --cached --name-only +git diff --cached +git commit -m "feat: add local transaction workflows" +``` + +### Task 3: Dashboard and monthly report screens + +**Files:** +- Create: `core/templates/core/dashboard.html` +- Create: `core/templates/core/reports/monthly.html` +- Modify: `core/forms.py` +- Modify: `core/views.py` +- Modify: `config/urls.py` +- Test: `tests/test_report_views.py` + +**Interfaces:** +- Consumes: `monthly_summary`, `spending_by_category`, `account_balances`, and `available_currencies` from Task 1. +- Produces named URL `monthly-report`; `home` renders the current selected month and currency rather than transaction-management content. + +- [ ] **Step 1: Write failing report-view tests** + +```python +def test_monthly_report_renders_exact_totals_and_category_rows(client): + response = client.get(reverse("monthly-report"), {"month": "2026-01", "currency": "BRL"}) + assert response.status_code == 200 + body = response.content.decode() + assert "750.00" in body + assert "Synthetic Food" in body + +def test_invalid_month_is_visible_and_not_silently_replaced(client): + response = client.get(reverse("monthly-report"), {"month": "01/2026", "currency": "BRL"}) + assert response.status_code == 400 + assert "invalid_month" in response.content.decode() +``` + +- [ ] **Step 2: Run the focused tests and confirm the route/template behavior is absent** + +Run: `docker compose run --rm test pytest -q tests/test_report_views.py` + +- [ ] **Step 3: Implement month/currency selection and report views** + +Parse `YYYY-MM` into the first day of the month. Default to the current local calendar month and the first available currency, falling back to `BRL` for an empty database. Invalid supplied values return status 400 with field errors. + +- [ ] **Step 4: Render dashboard and detailed report templates** + +Show inflow, expense, refund, net spending, net activity, categorized/uncategorized rows, and per-account balances. Label negative net spending and balances without changing their sign. + +- [ ] **Step 5: Run report service and view tests** + +Run: `docker compose run --rm test pytest -q tests/finance/test_reports.py tests/test_report_views.py` + +- [ ] **Step 6: Commit the report UI slice after staged privacy inspection** + +```powershell +git add core/forms.py core/views.py core/templates/core/dashboard.html core/templates/core/reports/monthly.html config/urls.py tests/test_report_views.py +git diff --cached --name-only +git diff --cached +git commit -m "feat: add monthly finance dashboard" +``` + +### Task 4: Versioned portable exports + +**Files:** +- Create: `finance/exports.py` +- Create: `finance/management/__init__.py` +- Create: `finance/management/commands/__init__.py` +- Create: `finance/management/commands/export_finances.py` +- Create: `core/templates/core/exports.html` +- Modify: `core/views.py` +- Modify: `config/urls.py` +- Modify: `config/settings.py` +- Modify: `compose.yaml` +- Test: `tests/finance/test_exports.py` +- Test: `tests/test_export_command.py` +- Test: `tests/test_export_view.py` + +**Interfaces:** +- Consumes: finance models through read-only ordered querysets. +- Produces `build_export() -> ExportArtifact` for an in-memory snapshot and `write_export(name: str | None = None) -> ExportResult` under `settings.EXPORT_ROOT`. +- Produces `export_finances [--output NAME]` and named URLs `export-index`, `export-download`. + +- [ ] **Step 1: Write failing package-contract tests** + +```python +def test_export_has_versioned_manifest_and_stable_csv_contract(): + artifact = build_export() + with ZipFile(BytesIO(artifact.content)) as package: + manifest = json.loads(package.read("manifest.json")) + assert manifest["format"] == "gizmoduck-export" + assert manifest["version"] == 1 + assert package.read("transactions.csv").decode("utf-8").splitlines()[0] == EXPECTED_TRANSACTION_HEADER + assert manifest["files"]["transactions.csv"]["sha256"] == sha256(package.read("transactions.csv")).hexdigest() +``` + +- [ ] **Step 2: Run export tests and confirm the service is missing** + +Run: `docker compose run --rm test pytest -q tests/finance/test_exports.py` + +- [ ] **Step 3: Implement the consistent export snapshot** + +Use `transaction.atomic()` and `SET TRANSACTION ISOLATION LEVEL REPEATABLE READ` before reads. Serialize all specified tables in UUID order, exact decimals, ISO dates/timestamps, lowercase booleans, and blank nullable fields. Write deterministic UTF-8 CSV content and checksums into a ZIP with `manifest.json` last. + +- [ ] **Step 4: Write failing safe-file and command tests** + +Cover generated names, traversal, absolute paths, symlinks, existing targets, safe stdout, temporary cleanup, and database failure before publication. + +- [ ] **Step 5: Implement exclusive atomic publication and command entry point** + +Resolve the real export root, accept plain `.zip` names only, create the destination exclusively, write to a same-directory private temporary file, flush and `os.fsync`, then `os.replace`. Reject an existing destination before building the financial snapshot and remove temporary files on every failure. + +- [ ] **Step 6: Write failing browser download tests and implement export pages** + +The GET index explains export contents. CSRF-protected POST builds the same in-memory artifact and returns it with `Content-Disposition: attachment`; the request cannot supply a server path. + +- [ ] **Step 7: Mount ignored export storage and add settings** + +Set `EXPORT_ROOT` from environment with `/app/exports` default. Mount `./exports:/app/exports` into `manage`; the browser download remains in memory and does not require a writable host mount. + +- [ ] **Step 8: Run all export tests** + +Run: `docker compose run --rm test pytest -q tests/finance/test_exports.py tests/test_export_command.py tests/test_export_view.py` + +- [ ] **Step 9: Commit the export slice after checking that no generated ZIP is staged** + +```powershell +git add finance/exports.py finance/management core config/settings.py config/urls.py compose.yaml tests/finance/test_exports.py tests/test_export_command.py tests/test_export_view.py +git diff --cached --name-only +git diff --cached +git commit -m "feat: add versioned financial exports" +``` + +### Task 5: PostgreSQL backup creation and isolated restore verification + +**Files:** +- Create: `core/backup.py` +- Create: `core/management/__init__.py` +- Create: `core/management/commands/__init__.py` +- Create: `core/management/commands/backup_database.py` +- Create: `core/management/commands/verify_backup.py` +- Modify: `Dockerfile` +- Modify: `config/settings.py` +- Modify: `compose.yaml` +- Modify: `.env.example` +- Test: `tests/test_backup.py` +- Test: `tests/test_backup_commands.py` +- Test: `tests/test_backup_restore.py` + +**Interfaces:** +- Produces `create_backup(name: str | None = None) -> BackupResult` and `verify_backup(name: str, target: DatabaseTarget) -> VerificationResult`. +- Produces commands `backup_database [--output NAME]` and `verify_backup --input NAME --target-name NAME --target-host HOST --target-user USER`. +- Verification requires `BACKUP_VERIFY_ALLOWED=1` and rejects the live `(DB_HOST, DB_NAME)` pair. + +- [ ] **Step 1: Write failing unit tests for safe subprocess construction and manifests** + +```python +def test_pg_dump_receives_password_only_in_environment(runner, settings): + create_backup("synthetic.dump") + argv, env = runner.calls[0] + assert argv[:3] == ["pg_dump", "--format=custom", "--no-owner"] + assert settings.DATABASES["default"]["PASSWORD"] not in argv + assert env["PGPASSWORD"] == settings.DATABASES["default"]["PASSWORD"] + +def test_verification_rejects_live_database(settings): + with pytest.raises(BackupError, match="live_target"): + verify_backup("synthetic.dump", DatabaseTarget.from_default(settings.DATABASES["default"])) +``` + +- [ ] **Step 2: Run backup tests and confirm the module is absent** + +Run: `docker compose run --rm test pytest -q tests/test_backup.py` + +- [ ] **Step 3: Install PostgreSQL 17 clients and implement safe backup creation** + +Add the PostgreSQL apt repository and pin/install `postgresql-client-17` in the application image. Build subprocess argument arrays without a shell. Use the default database settings and `PGPASSWORD`, custom format, `--no-owner`, `--no-acl`, private temporary files, digest manifest, `fsync`, exclusive destination reservation, and atomic rename. + +- [ ] **Step 4: Implement manifest validation and isolated restore verification** + +Require exact manifest format/version, matching size/digest, explicit allow flag, a target distinct from live host/name, and a target containing no non-system tables. Invoke `pg_restore --exit-on-error --single-transaction --no-owner --no-acl` without a shell. Validate `django_migrations` equality and manifest row counts through a separate psycopg connection to the target. + +- [ ] **Step 5: Write failing command tests and implement safe command adapters** + +Commands print artifact name, format version, and table counts only. Convert `BackupError.code` into `CommandError(code)` without paths, SQL, subprocess stderr, or credentials. + +- [ ] **Step 6: Add backup and verification Compose configuration** + +Set `BACKUP_ROOT=/app/backups`, mount `./backups:/app/backups` into `manage`, and add a profile-gated `restore-db` PostgreSQL 17 service on tmpfs. Add a `verify-backup` application service whose default database settings point to `restore-db`, with `BACKUP_VERIFY_ALLOWED=1`; keep live credentials out of that service. + +- [ ] **Step 7: Write and run the synthetic backup/restore integration test** + +Populate only synthetic finance, provenance, audit, and import-review rows; create a backup; restore it into the empty `restore-db`; compare migrations and per-table counts; then query the original database to prove it remains unchanged. + +Run: `docker compose build` + +Run: `docker compose run --rm test pytest -q tests/test_backup.py tests/test_backup_commands.py` + +Run: `docker compose --profile backup run --rm verify-backup python manage.py test_backup_restore` + +- [ ] **Step 8: Commit the backup slice after verifying no dumps, manifests, secrets, or generated reports are staged** + +```powershell +git add Dockerfile config/settings.py compose.yaml .env.example core/backup.py core/management tests/test_backup.py tests/test_backup_commands.py tests/test_backup_restore.py +git diff --cached --name-only +git diff --cached +git commit -m "feat: add verified PostgreSQL backups" +``` + +### Task 6: Owner documentation and complete synthetic acceptance workflow + +**Files:** +- Create: `docs/local-workflows.md` +- Create: `tests/test_mvp_acceptance.py` +- Modify: `README.md` +- Modify: `docs/development.md` +- Modify: `docs/roadmap.md` +- Modify: `.gitignore` + +**Interfaces:** +- Consumes: every public workflow created by Tasks 1–5 and the existing `import_review` command. +- Produces a copy-paste local workflow from synthetic import through categorization, reports, export, backup, and disposable restore verification. + +- [ ] **Step 1: Write the failing acceptance test** + +The test must create an account, preview/apply/commit `tests/fixtures/synthetic/ingestion/bank.csv`, POST a category assignment through the CSRF-enabled browser flow, assert the monthly report totals, inspect a generated export manifest, create a logical backup through the injectable runner boundary, and verify that the restore-verification boundary checks migrations and table counts. + +- [ ] **Step 2: Run the acceptance test and confirm any missing integration behavior** + +Run: `docker compose run --rm test pytest -q tests/test_mvp_acceptance.py` + +- [ ] **Step 3: Fix only integration gaps exposed by the failing acceptance test** + +Keep changes within existing Task 1–5 boundaries. Add no new product behavior. Re-run the focused tests for every amended module. + +- [ ] **Step 4: Document the owner workflow and recovery procedure** + +Document local URLs, supported edits, report semantics, command output safety, export contents, backup creation, disposable restore verification, and deliberate live recovery. The live recovery sequence must stop `web`, create a separate safety backup, name the target database explicitly, restore with PostgreSQL 17 tools, run migrations/checks, and restart only after validation. + +- [ ] **Step 5: Reconcile configuration and project status documentation** + +Document the loopback PostgreSQL port currently present in Compose, link the workflow guide from README, and mark roadmap step 5 complete only after full verification. Confirm `.gitignore` covers export and backup contents plus temporary suffixes while retaining synthetic fixtures. + +- [ ] **Step 6: Run the complete verification matrix** + +```powershell +docker compose config --quiet +docker compose build +docker compose run --rm manage migrate +docker compose run --rm manage check +docker compose run --rm manage makemigrations --check --dry-run +docker compose run --rm manage migrate --check +docker compose run --rm test +docker compose run --rm permissions +docker compose --profile backup run --rm verify-backup python manage.py test_backup_restore +Invoke-RestMethod http://127.0.0.1:8000/health/ +``` + +- [ ] **Step 7: Inspect all staged content for financial data and secrets, then commit** + +```powershell +git add README.md docs/development.md docs/local-workflows.md docs/roadmap.md .gitignore tests/test_mvp_acceptance.py +git diff --cached --name-only +git diff --cached +git commit -m "docs: complete local MVP workflow" +``` + +- [ ] **Step 8: Run a final whole-branch review against this plan and resolve all critical or important findings** + +Use the review package required by `superpowers:subagent-driven-development`, include all deferred-minor ledger entries, and re-run the complete verification matrix after any fix wave.