docs: plan local MVP workflows

2 months ago · ed008bee09
1 changed files with 453 additions and 0 deletions
--- a/docs/superpowers/plans/2026-09-07-local-mvp-workflows.md
+++ b/docs/superpowers/plans/2026-09-07-local-mvp-workflows.md
@ -0,0 +1,453 @@
+# Local MVP Workflows Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Deliver a local server-rendered workflow for managing and categorizing transactions, viewing monthly reports, exporting user-owned data, and creating and verifying PostgreSQL backups.
+
+**Architecture:** `finance.queries` and `finance.reports` own read behavior, existing audited finance services own every mutation, and `core` owns HTTP forms, views, URLs, and templates. A versioned export service is shared by browser and command entry points; backup commands wrap PostgreSQL 17 client tools and verify restores only against a separate explicitly marked database.
+
+**Tech Stack:** Python 3.13, Django 5.2, PostgreSQL 17, psycopg 3, Django templates/forms, pytest and pytest-django, Docker Compose.
+
+**Spec:** `docs/superpowers/specs/2026-09-07-local-mvp-workflows-design.md`
+
+## Global Constraints
+
+- All monthly reports use `posting_date`, an inclusive calendar month, and exactly one currency.
+- `INCOME` is inflow; `EXPENSE` is spending; `REFUND` reduces spending in its posting month; `TRANSFER` and `OPENING` are excluded from cash flow and spending.
+- Account balances include every entry through month end, including openings and transfer legs, and are never summed across currencies.
+- Browser writes must call audited `finance.services` with `Actor(kind="HUMAN", identifier="local-owner")`; views never save financial models directly.
+- Transactions and source evidence cannot be deleted or have immutable financial fields rewritten.
+- All mutation forms use POST and Django CSRF protection.
+- Export and backup artifacts must remain under ignored `exports/` and `backups/`, never overwrite existing files, and never publish partial files.
+- Terminal output, HTTP errors, tests, and committed fixtures must contain only synthetic data and safe error codes.
+- PostgreSQL backup clients must use major version 17 and receive passwords through environment variables, never command arguments.
+- OFX, bank APIs, investments, forecasting, receipt recognition, mobile clients, hosted multi-user operation, and automatic or AI categorization remain excluded.
+
+---
+
+### Task 1: Query, indexing, and report services
+
+**Files:**
+- Create: `finance/queries.py`
+- Create: `finance/reports.py`
+- Create: `finance/migrations/0002_transaction_report_indexes.py`
+- Test: `tests/finance/test_queries.py`
+- Test: `tests/finance/test_reports.py`
+
+**Interfaces:**
+- Consumes: `finance.models.Account`, `Category`, `Payee`, and `Transaction`.
+- Produces: `TransactionFilters.from_mapping(mapping)`, `filtered_transactions(filters)`, `available_currencies()`, `monthly_summary(month, currency)`, `spending_by_category(month, currency)`, and `account_balances(month, currency)`.
+- `month` is a `datetime.date` whose day must be `1`; currency is three uppercase ASCII letters.
+- Report return types are frozen dataclasses containing `Decimal` values.
+
+- [ ] **Step 1: Write failing query tests**
+
+```python
+def test_filtered_transactions_combines_filters_and_orders_stably():
+    filters = TransactionFilters.from_mapping({
+        "date_from": "2026-01-01", "date_to": "2026-01-31",
+        "currency": "BRL", "uncategorized": "yes",
+    })
+    rows = list(filtered_transactions(filters))
+    assert [row.pk for row in rows] == [newer.pk, older.pk]
+
+def test_invalid_filter_is_explicit():
+    with pytest.raises(QueryInputError) as caught:
+        TransactionFilters.from_mapping({"date_from": "31/01/2026"})
+    assert caught.value.errors == {"date_from": "invalid_date"}
+```
+
+- [ ] **Step 2: Run the query tests and confirm imports fail because `finance.queries` does not exist**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py`
+
+- [ ] **Step 3: Implement validated filters and deterministic query ordering**
+
+Implement a frozen `TransactionFilters` dataclass with optional dates and UUIDs, kind, currency, and tri-state uncategorized value. Parse ISO dates and UUIDs once, return field error codes through `QueryInputError.errors`, use `select_related`, and order by `-posting_date`, `-id`.
+
+- [ ] **Step 4: Run query tests and confirm they pass**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py`
+
+- [ ] **Step 5: Write failing report-semantic tests**
+
+```python
+def test_monthly_summary_counts_card_purchase_once_and_refund_in_posting_month():
+    result = monthly_summary(date(2026, 1, 1), "BRL")
+    assert result.inflow == Decimal("1000.00")
+    assert result.expense == Decimal("300.00")
+    assert result.refund == Decimal("50.00")
+    assert result.net_spending == Decimal("250.00")
+    assert result.net_activity == Decimal("750.00")
+
+def test_spending_has_uncategorized_bucket_and_never_combines_currency():
+    rows = spending_by_category(date(2026, 1, 1), "BRL")
+    assert [(row.category_name, row.net_spending) for row in rows] == [
+        ("Synthetic Food", Decimal("80.00")),
+        ("Uncategorized", Decimal("20.00")),
+    ]
+```
+
+- [ ] **Step 6: Run report tests and confirm failure because report functions are absent**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_reports.py`
+
+- [ ] **Step 7: Implement monthly summaries, category spending, balances, and currency choices**
+
+Use conditional PostgreSQL aggregates with explicit `Decimal("0.00")` defaults. Validate month and currency. Group nullable categories into an explicit Uncategorized result without manufacturing a Category row. Sort category results by descending net spending then category name; sort balances by account name then UUID.
+
+- [ ] **Step 8: Add report indexes through a migration**
+
+Add named indexes on `Transaction(posting_date)` and `Transaction(account, posting_date)`. Do not alter financial constraints.
+
+- [ ] **Step 9: Run focused tests and migration drift check**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_queries.py tests/finance/test_reports.py`
+
+Run: `docker compose run --rm manage makemigrations --check --dry-run`
+
+- [ ] **Step 10: Commit the query and reporting slice**
+
+```powershell
+git add finance/queries.py finance/reports.py finance/migrations/0002_transaction_report_indexes.py tests/finance/test_queries.py tests/finance/test_reports.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "feat: add financial queries and monthly reports"
+```
+
+### Task 2: Local web shell and audited management workflows
+
+**Files:**
+- Create: `core/forms.py`
+- Create: `core/context.py`
+- Create: `core/templates/core/base.html`
+- Create: `core/templates/core/transactions/list.html`
+- Create: `core/templates/core/transactions/detail.html`
+- Create: `core/templates/core/transactions/form.html`
+- Create: `core/templates/core/references/list.html`
+- Create: `core/templates/core/references/form.html`
+- Create: `core/static/core/app.css`
+- Modify: `core/views.py`
+- Modify: `config/urls.py`
+- Modify: `config/settings.py`
+- Test: `tests/test_web_workflows.py`
+
+**Interfaces:**
+- Consumes: Task 1 query APIs and existing `finance.services` mutation APIs.
+- Produces named URLs `home`, `transaction-list`, `transaction-detail`, `transaction-create`, `transaction-annotate`, `account-list`, `account-create`, `account-archive`, `category-list`, `category-create`, `category-archive`, `payee-list`, `payee-create`, and `payee-archive`.
+- Produces forms that return validated scalar values and UUIDs but never save models.
+
+- [ ] **Step 1: Write failing request tests for navigation, list filters, detail, and CSRF**
+
+```python
+def test_root_and_transaction_pages_are_discoverable(client):
+    assert client.get(reverse("home")).status_code == 200
+    page = client.get(reverse("transaction-list"), {"currency": "BRL"})
+    assert page.status_code == 200
+    assert "Synthetic purchase" in page.content.decode()
+
+def test_mutation_requires_csrf():
+    guarded = Client(enforce_csrf_checks=True)
+    response = guarded.post(reverse("category-create"), {"name": "Synthetic Food", "kind": "EXPENSE"})
+    assert response.status_code == 403
+```
+
+- [ ] **Step 2: Run web workflow tests and confirm missing routes fail**
+
+Run: `docker compose run --rm test pytest -q tests/test_web_workflows.py`
+
+- [ ] **Step 3: Enable template, static, CSRF, session, and message infrastructure**
+
+Add Django sessions and messages apps/middleware, template backend with request and message context processors, `STATIC_URL`, and signed-cookie sessions. Run the sessions migration; no financial data is stored in the session.
+
+- [ ] **Step 4: Implement forms and request-to-domain error mapping**
+
+Build plain `forms.Form` classes for manual transactions, annotations, accounts, categories, and payees. Query active references for choices. Convert `DomainError.code` into a fixed safe message dictionary; unknown codes map to `Unable to save this change.`
+
+- [ ] **Step 5: Implement transaction list/detail/create/annotate views**
+
+Use `finance.queries` for reads, Django `Paginator` with 50 rows per page, and existing finance services for writes. Derive `LOCAL_ACTOR = Actor(kind="HUMAN", identifier="local-owner")` server-side. Redirect after successful POST.
+
+- [ ] **Step 6: Implement reference list/create/archive views**
+
+Use services only for mutations. Archive endpoints accept POST only and render archived values as historical/inactive on list pages.
+
+- [ ] **Step 7: Add accessible templates and minimal responsive CSS**
+
+Provide one `h1`, associated labels, error summaries, visible focus styles, tabular headings, empty states, and navigation on every page. Keep all core workflows functional without JavaScript.
+
+- [ ] **Step 8: Run request and existing domain tests**
+
+Run: `docker compose run --rm test pytest -q tests/test_web_workflows.py tests/finance/test_services.py`
+
+- [ ] **Step 9: Commit the audited web workflow slice after staged privacy inspection**
+
+```powershell
+git add config/settings.py config/urls.py core tests/test_web_workflows.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "feat: add local transaction workflows"
+```
+
+### Task 3: Dashboard and monthly report screens
+
+**Files:**
+- Create: `core/templates/core/dashboard.html`
+- Create: `core/templates/core/reports/monthly.html`
+- Modify: `core/forms.py`
+- Modify: `core/views.py`
+- Modify: `config/urls.py`
+- Test: `tests/test_report_views.py`
+
+**Interfaces:**
+- Consumes: `monthly_summary`, `spending_by_category`, `account_balances`, and `available_currencies` from Task 1.
+- Produces named URL `monthly-report`; `home` renders the current selected month and currency rather than transaction-management content.
+
+- [ ] **Step 1: Write failing report-view tests**
+
+```python
+def test_monthly_report_renders_exact_totals_and_category_rows(client):
+    response = client.get(reverse("monthly-report"), {"month": "2026-01", "currency": "BRL"})
+    assert response.status_code == 200
+    body = response.content.decode()
+    assert "750.00" in body
+    assert "Synthetic Food" in body
+
+def test_invalid_month_is_visible_and_not_silently_replaced(client):
+    response = client.get(reverse("monthly-report"), {"month": "01/2026", "currency": "BRL"})
+    assert response.status_code == 400
+    assert "invalid_month" in response.content.decode()
+```
+
+- [ ] **Step 2: Run the focused tests and confirm the route/template behavior is absent**
+
+Run: `docker compose run --rm test pytest -q tests/test_report_views.py`
+
+- [ ] **Step 3: Implement month/currency selection and report views**
+
+Parse `YYYY-MM` into the first day of the month. Default to the current local calendar month and the first available currency, falling back to `BRL` for an empty database. Invalid supplied values return status 400 with field errors.
+
+- [ ] **Step 4: Render dashboard and detailed report templates**
+
+Show inflow, expense, refund, net spending, net activity, categorized/uncategorized rows, and per-account balances. Label negative net spending and balances without changing their sign.
+
+- [ ] **Step 5: Run report service and view tests**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_reports.py tests/test_report_views.py`
+
+- [ ] **Step 6: Commit the report UI slice after staged privacy inspection**
+
+```powershell
+git add core/forms.py core/views.py core/templates/core/dashboard.html core/templates/core/reports/monthly.html config/urls.py tests/test_report_views.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "feat: add monthly finance dashboard"
+```
+
+### Task 4: Versioned portable exports
+
+**Files:**
+- Create: `finance/exports.py`
+- Create: `finance/management/__init__.py`
+- Create: `finance/management/commands/__init__.py`
+- Create: `finance/management/commands/export_finances.py`
+- Create: `core/templates/core/exports.html`
+- Modify: `core/views.py`
+- Modify: `config/urls.py`
+- Modify: `config/settings.py`
+- Modify: `compose.yaml`
+- Test: `tests/finance/test_exports.py`
+- Test: `tests/test_export_command.py`
+- Test: `tests/test_export_view.py`
+
+**Interfaces:**
+- Consumes: finance models through read-only ordered querysets.
+- Produces `build_export() -> ExportArtifact` for an in-memory snapshot and `write_export(name: str | None = None) -> ExportResult` under `settings.EXPORT_ROOT`.
+- Produces `export_finances [--output NAME]` and named URLs `export-index`, `export-download`.
+
+- [ ] **Step 1: Write failing package-contract tests**
+
+```python
+def test_export_has_versioned_manifest_and_stable_csv_contract():
+    artifact = build_export()
+    with ZipFile(BytesIO(artifact.content)) as package:
+        manifest = json.loads(package.read("manifest.json"))
+        assert manifest["format"] == "gizmoduck-export"
+        assert manifest["version"] == 1
+        assert package.read("transactions.csv").decode("utf-8").splitlines()[0] == EXPECTED_TRANSACTION_HEADER
+        assert manifest["files"]["transactions.csv"]["sha256"] == sha256(package.read("transactions.csv")).hexdigest()
+```
+
+- [ ] **Step 2: Run export tests and confirm the service is missing**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_exports.py`
+
+- [ ] **Step 3: Implement the consistent export snapshot**
+
+Use `transaction.atomic()` and `SET TRANSACTION ISOLATION LEVEL REPEATABLE READ` before reads. Serialize all specified tables in UUID order, exact decimals, ISO dates/timestamps, lowercase booleans, and blank nullable fields. Write deterministic UTF-8 CSV content and checksums into a ZIP with `manifest.json` last.
+
+- [ ] **Step 4: Write failing safe-file and command tests**
+
+Cover generated names, traversal, absolute paths, symlinks, existing targets, safe stdout, temporary cleanup, and database failure before publication.
+
+- [ ] **Step 5: Implement exclusive atomic publication and command entry point**
+
+Resolve the real export root, accept plain `.zip` names only, create the destination exclusively, write to a same-directory private temporary file, flush and `os.fsync`, then `os.replace`. Reject an existing destination before building the financial snapshot and remove temporary files on every failure.
+
+- [ ] **Step 6: Write failing browser download tests and implement export pages**
+
+The GET index explains export contents. CSRF-protected POST builds the same in-memory artifact and returns it with `Content-Disposition: attachment`; the request cannot supply a server path.
+
+- [ ] **Step 7: Mount ignored export storage and add settings**
+
+Set `EXPORT_ROOT` from environment with `/app/exports` default. Mount `./exports:/app/exports` into `manage`; the browser download remains in memory and does not require a writable host mount.
+
+- [ ] **Step 8: Run all export tests**
+
+Run: `docker compose run --rm test pytest -q tests/finance/test_exports.py tests/test_export_command.py tests/test_export_view.py`
+
+- [ ] **Step 9: Commit the export slice after checking that no generated ZIP is staged**
+
+```powershell
+git add finance/exports.py finance/management core config/settings.py config/urls.py compose.yaml tests/finance/test_exports.py tests/test_export_command.py tests/test_export_view.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "feat: add versioned financial exports"
+```
+
+### Task 5: PostgreSQL backup creation and isolated restore verification
+
+**Files:**
+- Create: `core/backup.py`
+- Create: `core/management/__init__.py`
+- Create: `core/management/commands/__init__.py`
+- Create: `core/management/commands/backup_database.py`
+- Create: `core/management/commands/verify_backup.py`
+- Modify: `Dockerfile`
+- Modify: `config/settings.py`
+- Modify: `compose.yaml`
+- Modify: `.env.example`
+- Test: `tests/test_backup.py`
+- Test: `tests/test_backup_commands.py`
+- Test: `tests/test_backup_restore.py`
+
+**Interfaces:**
+- Produces `create_backup(name: str | None = None) -> BackupResult` and `verify_backup(name: str, target: DatabaseTarget) -> VerificationResult`.
+- Produces commands `backup_database [--output NAME]` and `verify_backup --input NAME --target-name NAME --target-host HOST --target-user USER`.
+- Verification requires `BACKUP_VERIFY_ALLOWED=1` and rejects the live `(DB_HOST, DB_NAME)` pair.
+
+- [ ] **Step 1: Write failing unit tests for safe subprocess construction and manifests**
+
+```python
+def test_pg_dump_receives_password_only_in_environment(runner, settings):
+    create_backup("synthetic.dump")
+    argv, env = runner.calls[0]
+    assert argv[:3] == ["pg_dump", "--format=custom", "--no-owner"]
+    assert settings.DATABASES["default"]["PASSWORD"] not in argv
+    assert env["PGPASSWORD"] == settings.DATABASES["default"]["PASSWORD"]
+
+def test_verification_rejects_live_database(settings):
+    with pytest.raises(BackupError, match="live_target"):
+        verify_backup("synthetic.dump", DatabaseTarget.from_default(settings.DATABASES["default"]))
+```
+
+- [ ] **Step 2: Run backup tests and confirm the module is absent**
+
+Run: `docker compose run --rm test pytest -q tests/test_backup.py`
+
+- [ ] **Step 3: Install PostgreSQL 17 clients and implement safe backup creation**
+
+Add the PostgreSQL apt repository and pin/install `postgresql-client-17` in the application image. Build subprocess argument arrays without a shell. Use the default database settings and `PGPASSWORD`, custom format, `--no-owner`, `--no-acl`, private temporary files, digest manifest, `fsync`, exclusive destination reservation, and atomic rename.
+
+- [ ] **Step 4: Implement manifest validation and isolated restore verification**
+
+Require exact manifest format/version, matching size/digest, explicit allow flag, a target distinct from live host/name, and a target containing no non-system tables. Invoke `pg_restore --exit-on-error --single-transaction --no-owner --no-acl` without a shell. Validate `django_migrations` equality and manifest row counts through a separate psycopg connection to the target.
+
+- [ ] **Step 5: Write failing command tests and implement safe command adapters**
+
+Commands print artifact name, format version, and table counts only. Convert `BackupError.code` into `CommandError(code)` without paths, SQL, subprocess stderr, or credentials.
+
+- [ ] **Step 6: Add backup and verification Compose configuration**
+
+Set `BACKUP_ROOT=/app/backups`, mount `./backups:/app/backups` into `manage`, and add a profile-gated `restore-db` PostgreSQL 17 service on tmpfs. Add a `verify-backup` application service whose default database settings point to `restore-db`, with `BACKUP_VERIFY_ALLOWED=1`; keep live credentials out of that service.
+
+- [ ] **Step 7: Write and run the synthetic backup/restore integration test**
+
+Populate only synthetic finance, provenance, audit, and import-review rows; create a backup; restore it into the empty `restore-db`; compare migrations and per-table counts; then query the original database to prove it remains unchanged.
+
+Run: `docker compose build`
+
+Run: `docker compose run --rm test pytest -q tests/test_backup.py tests/test_backup_commands.py`
+
+Run: `docker compose --profile backup run --rm verify-backup python manage.py test_backup_restore`
+
+- [ ] **Step 8: Commit the backup slice after verifying no dumps, manifests, secrets, or generated reports are staged**
+
+```powershell
+git add Dockerfile config/settings.py compose.yaml .env.example core/backup.py core/management tests/test_backup.py tests/test_backup_commands.py tests/test_backup_restore.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "feat: add verified PostgreSQL backups"
+```
+
+### Task 6: Owner documentation and complete synthetic acceptance workflow
+
+**Files:**
+- Create: `docs/local-workflows.md`
+- Create: `tests/test_mvp_acceptance.py`
+- Modify: `README.md`
+- Modify: `docs/development.md`
+- Modify: `docs/roadmap.md`
+- Modify: `.gitignore`
+
+**Interfaces:**
+- Consumes: every public workflow created by Tasks 1–5 and the existing `import_review` command.
+- Produces a copy-paste local workflow from synthetic import through categorization, reports, export, backup, and disposable restore verification.
+
+- [ ] **Step 1: Write the failing acceptance test**
+
+The test must create an account, preview/apply/commit `tests/fixtures/synthetic/ingestion/bank.csv`, POST a category assignment through the CSRF-enabled browser flow, assert the monthly report totals, inspect a generated export manifest, create a logical backup through the injectable runner boundary, and verify that the restore-verification boundary checks migrations and table counts.
+
+- [ ] **Step 2: Run the acceptance test and confirm any missing integration behavior**
+
+Run: `docker compose run --rm test pytest -q tests/test_mvp_acceptance.py`
+
+- [ ] **Step 3: Fix only integration gaps exposed by the failing acceptance test**
+
+Keep changes within existing Task 1–5 boundaries. Add no new product behavior. Re-run the focused tests for every amended module.
+
+- [ ] **Step 4: Document the owner workflow and recovery procedure**
+
+Document local URLs, supported edits, report semantics, command output safety, export contents, backup creation, disposable restore verification, and deliberate live recovery. The live recovery sequence must stop `web`, create a separate safety backup, name the target database explicitly, restore with PostgreSQL 17 tools, run migrations/checks, and restart only after validation.
+
+- [ ] **Step 5: Reconcile configuration and project status documentation**
+
+Document the loopback PostgreSQL port currently present in Compose, link the workflow guide from README, and mark roadmap step 5 complete only after full verification. Confirm `.gitignore` covers export and backup contents plus temporary suffixes while retaining synthetic fixtures.
+
+- [ ] **Step 6: Run the complete verification matrix**
+
+```powershell
+docker compose config --quiet
+docker compose build
+docker compose run --rm manage migrate
+docker compose run --rm manage check
+docker compose run --rm manage makemigrations --check --dry-run
+docker compose run --rm manage migrate --check
+docker compose run --rm test
+docker compose run --rm permissions
+docker compose --profile backup run --rm verify-backup python manage.py test_backup_restore
+Invoke-RestMethod http://127.0.0.1:8000/health/
+```
+
+- [ ] **Step 7: Inspect all staged content for financial data and secrets, then commit**
+
+```powershell
+git add README.md docs/development.md docs/local-workflows.md docs/roadmap.md .gitignore tests/test_mvp_acceptance.py
+git diff --cached --name-only
+git diff --cached
+git commit -m "docs: complete local MVP workflow"
+```
+
+- [ ] **Step 8: Run a final whole-branch review against this plan and resolve all critical or important findings**
+
+Use the review package required by `superpowers:subagent-driven-development`, include all deferred-minor ledger entries, and re-run the complete verification matrix after any fix wave.