docs: check in the fin skill library, ignore local Claude settings

Twelve skills that had been sitting uncommitted in the working tree: the
architecture contract, accounting reference, change-control rulebook, failure
archaeology, debugging playbook, diagnostics scripts, ledger-analysis toolkit,
research frontier, run/operate runbook, build-and-env, test-coverage campaign,
and validation/QA guide.

Updates three frontier statuses to match what actually shipped today rather
than checking in a roadmap that describes closed work as open:

- A1 double-entry hardening -> partly-done. The create half is enforced; what
  remains is running the existing audit_linked_legs.py against a prod backup.
- B3 safety rails -> partly-done. Idempotency shipped, including the note that
  the index is not unique-when-not-null and why. dry_run is still open.
- D3 CSV import -> unblocked but still gated. Its stated blocker was B3's
  idempotency keys, which now exist.

Also gitignores .claude/settings.local.json, which is per-machine and was
showing up as untracked on every status.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3qYLuu9jgXsCRXmJ18YGY
This commit is contained in:
EislM0203andClaude Opus 5 committed 2026-08-22 22:09:22 +00:00
1 parent c9a1375fb9
commit cb97d97d81
19 files changed
+4647 -1

No files matched your search

+36
View File
@@ -0,0 +1,36 @@
# fin skill library — index
13 skills, authored 2026-07-05 against commit `082b608`. Each SKILL.md is self-contained, ends with a "Provenance and maintenance" section (re-verification one-liners), and states when NOT to use it. Code beats skills; skills beat memory.
## Task → skill map
| You are about to… | Load |
|---|---|
| Query/record finances via the `fin` CLI | **fin** |
| Change any code, schema, or doc — find the gate | **fin-change-control** |
| Debug "a number looks wrong" / triage any failure | **fin-debugging-playbook** |
| Fix something that looks like an obvious bug | **fin-failure-archaeology** first (is it settled?) |
| Design/refactor; add endpoint, table, account type | **fin-architecture-contract** |
| Reason about money: types, spending math, FX, RSU | **fin-accounting-reference** |
| Set up an environment, run tests, build images | **fin-build-and-env** |
| Deploy, back up, configure env vars, operate prod | **fin-run-and-operate** |
| Measure data integrity; before/after evidence | **fin-diagnostics-and-tooling** (scripts/) |
| Claim a change works; write/extend e2e tests | **fin-validation-and-qa** |
| Expand test coverage / add CI (the campaign) | **fin-test-coverage-campaign** |
| Prove an accounting property; predict a delta | **fin-ledger-analysis-toolkit** |
| Choose the next project / roadmap | **fin-research-frontier** |
## Ownership (one home per fact)
- Backup/restore runbook → fin-run-and-operate §3 (others cross-reference).
- Accounting formulas/definitions → fin-accounting-reference; proof methods on top → fin-ledger-analysis-toolkit.
- Invariant catalog incl. unenforced gaps (U1–U8) → fin-ledger-analysis-toolkit.
- Test inventory (certified/not) → fin-validation-and-qa §2; coverage expansion → fin-test-coverage-campaign.
- Settled design questions & incident chronicle → fin-failure-archaeology.
- Config catalog → fin-run-and-operate §2.
## Standing rules baked into every skill
1. The prod SQLite DB (volume `sqlite_data`, `/data/fin.db`) is sacred: never test against it, back up before migrations.
2. Behavior changes route through fin-change-control's gates; accounting claims need level-(a)/(b) evidence (fin-validation-and-qa).
3. Anything date-stamped "as of 2026-07-05" may have drifted — run the skill's provenance one-liners before relying on it.
@@ -0,0 +1,178 @@
---
name: fin-accounting-reference
description: The complete semantic model of money in the fin app. Load whenever reasoning about amounts, balances, spending or income math, currencies and FX conversion, transaction types, transfers/settlements, Splitwise shares, receivables/payables, stock holdings, RSU grants and vesting, or net worth — and BEFORE writing or reviewing any accounting logic in api/app/crud.py, api/app/services/net_worth.py, routers, the CLI, or QuickAdd flows.
---
# fin Accounting Reference (domain theory)
Every formula below is derived from the code as of 2026-07-05, with `file:function` citations. Code is the truth; if this document and the code disagree, trust the code and update this file (see "Provenance and maintenance").
## 1. Units, signs, and columns
- **`amount`**: positive = money into the account, negative = money out. Always in the transaction's `currency` (model comment in `api/app/models.py:Transaction`, PLAN.md data model).
- **`fee`**: a **positive** number representing an extra cost, stored separately from `amount`. Default 0.
- **Balance formula** (bank / splitwise / receivable / payable accounts): `balance = SUM(amount − fee)` over all of the account's transactions. Source: `api/app/services/net_worth.py:calculate_net_worth` (the `balance_rows` query: `func.sum(Transaction.amount - Transaction.fee)`). Never sum `amount` alone (CLAUDE.md invariant 5).
- Worked example: expense of €50 with a €2 fee → `amount = -50`, `fee = 2` → balance impact = `-50 − 2 = -52`.
- On transfers, the fee sits on the **source leg only**; the destination leg is always written with `fee = 0` (`crud.py:create_transfer`). So a €500 transfer with €1 fee debits `-501` net and credits `+500`.
- **Currency**: each `Transaction` carries its own `currency` (default `"EUR"`), and each `Account` has a declared `currency`. Balances are raw sums with **no per-transaction FX conversion**; net worth converts the summed native balance using the **account's** currency (`net_worth.py`). Consequence: a transaction's `currency` must match its account's currency or the balance is silently wrong — cross-currency transfers handle this via `to_amount`/`to_currency` (§3).
- **`exchange_rate_used`** (nullable, on `transactions` and `stock_transactions`): an audit column recording the EUR/USD rate in effect. It is only *used* computationally in `crud.py:create_funded_buy`, which converts the cash leg with `bank_amount = cash_native / used_rate` when the funding account's currency differs from the stock's currency (`used_rate = req.exchange_rate_used or eur_usd_rate`). Everywhere else it is informational.
- **Opening balance**: `crud.py:create_account` with `initial_balance > 0` writes an `income` transaction with `description = "Opening balance"`. That exact description string is what excludes it from income totals (§4) — do not rename it.
## 2. Transaction type semantics
All six types from `models.py:Transaction` / `schemas.py:TransactionBase`. "Two-leg" means two rows linked via `linked_account_id` + `linked_transaction_id`.
| Type | Meaning | Sign pattern | In spending? | In income? | Two-leg? | Created by |
|---|---|---|---|---|---|---|
| `expense` | Consumption (incl. debt to a person via a payable account) | negative | Yes (raw amount) | No | No — except as part of a split-expense triple | QuickAdd ExpenseFlow, IOweFlow; `fin tx add --type expense`; split-expense endpoint (its expense row) |
| `income` | Earnings; also opening balances | positive | No | Yes (except `description = 'Opening balance'`) | No | QuickAdd IncomeFlow; `fin tx add --type income`; `crud.py:create_account` |
| `transfer` | Asset movement between own accounts (incl. lending money = bank→receivable, and funded-buy cash legs) | source −, destination + | Never | Never | Yes | QuickAdd TransferFlow, IAmOwedFlow; `fin transfer`; `POST /api/transactions/transfer`; `crud.py:create_funded_buy` cash leg; split-expense partner legs |
| `splitwise_pay` | I paid a shared bill; partner owes their share | bank leg − (full bill), splitwise leg + (partner share) | Yes — bank leg only, at net share `amount + splitwise_partner_share` | No | Yes | QuickAdd SplitwiseIPaidFlow; `fin splitwise-paid`; `POST /api/transactions/splitwise-i-paid` |
| `splitwise_receive` | Partner paid; I owe my share | single negative row on the splitwise account | Yes (raw amount) | No | No | QuickAdd SplitwiseShePayedFlow; `fin tx add --type splitwise_receive` |
| `settlement` | Clearing a splitwise/receivable/payable balance against a bank account | one leg +, one leg − (opposite accounts) | Never | Never | Yes | QuickAdd SplitwiseSettleFlow; `fin settle`; `fin reconcile`; `POST /api/transactions/settlement` and `/reconcile` |
History note: funded-buy cash legs used to be typed `expense` and reconcile legs `income`/`expense`, which inflated the metrics. Migration `api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py` retyped legacy rows to `transfer`/`settlement`; new rows are written correctly (`crud.py:create_funded_buy` docstring, commit e36d719). Never reintroduce income/expense typing for internal cash movements.
`fin tx add` deliberately only accepts `expense`, `income`, `splitwise_pay`, `splitwise_receive` (`cli/fin_cli/tx.py:TX_TYPES`) — single-row transfers/settlements corrupt balances; use the paired commands.
## 3. Two-leg (double-entry) mechanics
All constructs below are written atomically: `db.add_all([...])`, `db.flush()` to get ids, cross-set `linked_transaction_id`, single `db.commit()` (CLAUDE.md invariant 1).
| Construct | Function | Legs (account: amount, type) | Invariant preserved |
|---|---|---|---|
| Transfer | `crud.py:create_transfer` | from: `−amount` (fee=fee); to: `+to_amount ?? +amount` (fee=0) — both `transfer` | Same-currency: net worth unchanged minus fee. Cross-currency: caller supplies `to_amount`/`to_currency` (see `schemas.py:TransferRequest`); the two legs need not sum to zero in nominal terms |
| Splitwise settlement | `crud.py:create_settlement` | direction `splitwise_to_bank`: bank `+amount`, splitwise `−amount`; `bank_to_splitwise`: reversed — both `settlement` | Cash ↔ receivable swap: net worth delta = 0 |
| Split expense (3 rows) | `crud.py:create_split_expense` | bank: `−(total − partner_share)` `expense`; bank: `−partner_share` `transfer`; receivable: `+partner_share` `transfer`. Expense is linked into the group via `expense.linked_transaction_id = from_leg.id` | Bank debited the full total; only your share is spending; partner share becomes a receivable asset. Net worth delta = −your share |
| Splitwise "I paid" | `crud.py:create_splitwise_i_paid` | bank: `−total` `splitwise_pay` with `splitwise_partner_share = partner_share`; splitwise: `+partner_share` `splitwise_pay` | Net worth delta = −your share (PLAN.md Flow A) |
| Reconcile | `routers/transactions.py:reconcile_accounts` | receivable: bank `+amount`, receivable `−amount`; payable: bank `−amount`, payable `+amount` — both `settlement` | Net worth delta = 0; the consumption was already booked when the payable/receivable was created |
| Funded buy/sell | `crud.py:create_funded_buy` | StockTransaction + one bank row typed `transfer` (buy: `−total_cost`; sell: `+shares×price − fee`), FX-converted via `used_rate` if currencies differ | Cash ↔ securities swap never touches spending/income |
**Group delete**: `crud.py:delete_transaction` walks `linked_transaction_id` in both directions to collect the whole group (both legs, plus a split-expense's expense row) and deletes it atomically; a legacy fallback matches on `linked_account_id` + date + type + closest id for pre-`linked_transaction_id` rows (migration `e5f6a7b8c9d0`). Never delete a single leg.
**Edit lock**: `PUT /api/transactions/{id}` returns 422 for any row with `linked_account_id` set — delete both legs and re-enter (`routers/transactions.py:update_transaction`).
## 4. Spending & income math
Source: `crud.py:get_monthly_spending` (per-month) and `crud.py:get_spending_summary` (yearly/custom = per-month loop, then aggregate).
**Spending** for month `YYYY-MM` = sum of two query sets, grouped by category, reported as positive numbers:
1. Rows with `type IN ('expense', 'splitwise_receive')`, `amount < 0`, joined to accounts with `Account.type NOT IN ('receivable')` — counted at **raw amount**.
2. Rows with `type = 'splitwise_pay'`, `amount < 0`, same account filter — counted at the **user's net share**: `amount + splitwise_partner_share`.
Worked €100-dinner example (50/50, I paid, via `splitwise-i-paid`):
- Rows written: bank `amount = −100`, `splitwise_partner_share = 50`, type `splitwise_pay`; splitwise account `amount = +50`, type `splitwise_pay`.
- Spending contribution: `−100 + 50 = −50` → **€50** (the splitwise `+50` leg fails the `amount < 0` filter, so it never double-counts).
- Net worth: bank −100, splitwise +50 → **−50**. Consistent.
If instead the partner paid an €80 bill split 50/50: one `splitwise_receive` row, `amount = −40`, on the splitwise account → spending €40, no cash movement.
**Income** for the month = `SUM(amount)` where `type = 'income'`, account type `NOT IN ('payable', 'receivable')`, and `description != 'Opening balance'`.
**Summary** (`get_spending_summary`): `grand_total` = Σ monthly spending, `income_total` = Σ monthly income, `net_total = income_total − grand_total`. Categories with a net non-negative month total are skipped; percentages are share of the month's overall total.
Two intentional surprises — do not "fix" them:
1. **I-Owe counts as spending immediately.** IOweFlow records an `expense` on a **payable** account; commit e61b254 changed the account filter from `NOT IN ('payable','receivable')` to `NOT IN ('receivable')` precisely so payable expenses count when the debt arises. Cash leaves the bank later via reconcile, which is neutral. (Receivable accounts stay excluded — a9f045f — because lending is a transfer, not spending; see `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` §3.)
2. **Reconcile/settlement is never income or spending**, even though cash arrives in/leaves the bank. Both legs are `settlement`, which appears in neither query (h8c9d0e1f2a3 migration rationale).
Also: `transfer` never appears in either aggregate, and category `type` (`expense`/`income`, `models.py:Category`) is a **UI-only** picker filter — the spending queries never join on it (spec doc §1).
## 5. Net worth
Source: `api/app/services/net_worth.py:calculate_net_worth`. Only `is_active` accounts are included. Result: `{total_eur, eur_usd_rate, accounts: [{id, name, type, currency, balance_native, balance_eur, is_override}]}`.
Per account type:
- **bank / splitwise / receivable / payable**: `balance_native = manual_override_balance ?? SUM(amount − fee)`; `balance_eur = _to_eur(balance_native, account.currency, price_map)`.
- **investment / rsu**: if `manual_override_balance` is set, it is treated as being in the account's declared currency and converted with `_to_eur` (fix in commit 37bc101). Otherwise `Σ over holdings: shares × price_cache[ticker]`, each holding converted using the **price row's** currency (`ccy_map`), not the account's.
- Unknown account types are skipped with a warning.
`_to_eur(amount, ccy, price_map)`: EUR passes through; otherwise divide by `price_map["EUR/<CCY>"]` (rates are quoted as *units of CCY per 1 EUR*, so EUR value = native / rate). **If the rate is missing, the value contributes `0.0` and a warning is logged** — never a wrong-currency conversion (see §8). Two residual 1.0 fallbacks exist when `EUR/USD` is uncached: the top-level `eur_usd_rate` response field, and `net_worth.py:_get_eur_usd_rate`, which `routers/stock_transactions.py` feeds into funded-buy conversion — a documented gap (U8 in fin-ledger-analysis-toolkit), not a pattern to copy.
**Unvested RSUs are NOT in net worth.** `calculate_net_worth` only reads `holdings`, which only change on vest. The dashboard shows unvested value greyed-out, client-side only (commit babea72, `web/src/pages/Dashboard.tsx`): `Σ gross_shares × (1 − withholding_pct) × current_price`, converted to EUR via the EUR/USD rate (any non-EUR price currency uses EUR/USD there — a dashboard-only approximation).
**Snapshot**: `services/scheduler.py:fetch_prices_and_snapshot` runs at midnight — fetches prices, computes net worth, upserts one `net_worth_snapshots` row per calendar day with `total_eur` and `breakdown_json` = the JSON-serialized `accounts` list. A separate 5-minute job refreshes prices only (no snapshot).
## 6. Stock & holdings math
Source: `crud.py:recompute_holding_from_stock_transactions` — the single source of truth. Holdings are never edited directly; every create/delete of a `StockTransaction` flushes then recomputes from the full per-`(account_id, ticker)` history in date-then-id order (CLAUDE.md invariant 2). Note: as of 2026-07-05, insert also does a full recompute (`crud.py:create_stock_transaction`), not an incremental update.
Running-total algorithm:
- `buy` / `rsu_vest`: `shares += tx.shares`; `total_cost += tx.shares × tx.price_per_share + tx.fee` — **fees are capitalized into cost basis**.
- `sell`: reject if `tx.shares > shares` at that point in history (ValueError → HTTP 422). Otherwise reduce `total_cost` by `sell_shares × (total_cost / shares)` (average-cost method), then `shares −= sell_shares`; `total_cost` floored at 0.
- Final `avg_cost_basis_per_share = total_cost / shares` (0 if no shares). Zero remaining transactions → the holding row is deleted.
Example: buy 10 @ $100 fee $5 → shares 10, cost 1005, avg 100.50. Buy 5 @ $120 → shares 15, cost 1605, avg 107. Sell 6 → cost 1605 − 6×107 = 963, shares 9, avg still 107.
**Fractional shares / total-first buy**: shares are floats. In QuickAdd StockTradeFlow, a buy into an `investment` account is entered as total invested; `shares = totalInvested / price` (fractional), `total_cost = totalInvested + fee` (`web/src/components/QuickAdd/flows/StockTradeFlow.tsx`, `isTotalFirstBuy`).
**Funded buy cash leg** (`crud.py:create_funded_buy`, commit e36d719): the bank row is typed **`transfer`** — asset conversion, never spending/income. Buy: `−total_cost`; sell: `+(shares × price − fee)`; converted `cash_native / used_rate` when the funding account's currency differs from the stock currency. CLI: `fin stock funded-buy`; plain `fin stock add` writes no cash leg.
**P&L** (`routers/holdings.py:list_holdings`, feeds Portfolio page and `fin holdings`): per holding, `current_value_eur = _to_eur(shares × live_price, price_ccy)`, `cost_basis_eur = _to_eur(shares × avg_cost_basis_per_share, price_ccy)`, `pnl_eur = current − cost`, `pnl_pct = pnl_eur / cost_basis_eur × 100` (0 when basis is 0). Caveat: cost basis is converted at **today's** FX rate, not the historical one.
## 7. RSU model
`models.py:RSUGrant`: `account_id, ticker, vest_date, gross_shares, withholding_pct (default 0.5), notes, vested_at, vested_stock_transaction_id`. One row = one tranche. "One pot per account": grants attach to an `rsu` (or `investment`) account (`crud.py:create_rsu_grant` enforces this); all tranches of a ticker share that account's single holding. The UI "pot form" (`web/src/pages/Portfolio.tsx`) defaults withholding to **51%** (0.51, commit c835118); the schema/CLI default is **0.5** (`schemas.py:RSUGrantBase`, `cli/fin_cli/rsu.py:add`). Withholding is stored as a 0..1 fraction.
**Vest** (`crud.py:vest_rsu_grant`, `POST /api/rsu-grants/{id}/vest`, `fin rsu vest`):
- `net_shares = net_shares_override ?? floor(gross_shares × (1 − withholding_pct))` — **floored to whole shares**, must be ≥ 1 (422 otherwise). E.g. gross 30, withholding 0.51 → floor(14.7) = **14** net shares. (UI preview floors the same way — commit 225e69e.)
- Price and **currency come from `price_cache[ticker]`** (422 if no cached price > 0 — refresh prices first). Writes an `rsu_vest` StockTransaction: `shares = net_shares`, `price_per_share = cached price`, `fee = 0`, `total_cost = net_shares × price`, `date = grant.vest_date` (not today), then recomputes the holding and stamps `vested_at` + `vested_stock_transaction_id`.
- Cost basis consequence: vests enter the holding at market price, so the "cost basis" of RSU shares is their vest-day value.
- Deleting the linked stock transaction reverts the grant to pending (`crud.py:delete_stock_transaction` clears `vested_at`/`vested_stock_transaction_id` first). A vested grant itself cannot be deleted (`crud.py:delete_rsu_grant`).
**Schedule generation** (`crud.py:_generate_rsu_schedule`; endpoints `/api/rsu-grants/preview-schedule` and `/generate-schedule`; CLI `fin rsu schedule preview|generate`): one cliff tranche at `grant_date + cliff_months`, then a tranche every `frequency_months` until `grant_date + total_months`. Defaults: cliff 12, total 36, frequency 3 (`schemas.py:RSUScheduleRequest`). Each tranche gets `floor(total / n_tranches)` whole shares; the remainder is added to the **last** tranche so the sum equals the (floored) total.
Pending grants keep their ticker in the price-fetch set (`price_fetcher.py:get_tracked_tickers` unions holdings with unvested grants), which is what makes the dashboard's unvested estimate (§5) and vesting possible.
## 8. Multi-currency
- **Base currency is EUR.** Everything reported EUR-first; native balances are shown alongside.
- **FX pairs in `price_cache`** (as of 2026-07-05): `EUR/USD` and `EUR/GBP`, both from Frankfurter (`https://api.frankfurter.dev/v1/latest?from=EUR&to=...`), fetched in `price_fetcher.py:fetch_eur_usd` / `fetch_eur_gbp`. Quotation: CCY per 1 EUR → convert to EUR by **dividing**.
- **Ticker prices** are cached in whatever currency yfinance reports — not always USD. European suffixes (`.DE`, `.L`, `.AS`, …) are auto-tried; LSE prices in pence (`GBp`/`GBX`) are divided by 100 and stored as GBP (`price_fetcher.py:_try_fetch_candidate`). The price row's `currency` drives conversion, never an assumption of USD.
- **Staleness policy**: prices refresh every 5 minutes and at midnight (scheduler jobs); on fetch failure the last cached value is served and a warning logged — the API never crashes on Yahoo/Frankfurter outages.
- **No wrong-currency fallbacks.** Commit 81091fc removed a `1.0` FX fallback: if a rate can't be fetched *and* no cache exists, the fetcher returns `None` and skips the upsert, and `net_worth._to_eur` contributes `0.0` with a warning for missing rates. Rationale: a silently-wrong rate (e.g. valuing GBP at par with EUR) corrupts net worth history via snapshots; an obviously-missing value plus a log line is recoverable. Never reintroduce a numeric FX fallback. Known residual exception (as of 2026-07-05): `_get_eur_usd_rate` still falls back to `1.0` and feeds funded-buy conversion — tracked as gap U8 (fin-ledger-analysis-toolkit) / candidate fix (fin-research-frontier), not license to add more.
## 9. Glossary
| Term | Meaning in fin |
|---|---|
| Receivable | Account type: money others owe you (an asset). Created by lending (bank→receivable `transfer`) or a split expense's partner share. Excluded from spending queries. |
| Payable | Account type: money you owe others (a liability, negative balance). An `expense` on it counts as spending immediately; paying it off is a neutral `settlement`. |
| Settlement | Two-leg `settlement`-typed pair clearing a splitwise/receivable/payable balance against a bank account. Never income or spending. |
| Reconcile | The receivable/payable variant of settlement — `POST /api/transactions/reconcile`, `fin reconcile`. |
| Splitwise account | Single account holding the running net balance with the partner: positive = they owe you, negative = you owe them. |
| Partner share | `splitwise_partner_share` column: the partner's portion of a bill you paid; subtracted (via `amount + partner_share`) so spending shows only your net share. |
| Pot | An `rsu` account treated as one pool: all grants/tranches of a ticker vest into the same holding; the UI pot form seeds grants with a 51% withholding default. |
| Vest | Converting a pending RSU grant into an `rsu_vest` StockTransaction for `floor(gross × (1 − withholding))` net shares at the cached market price. |
| Withholding | Fraction (0..1) of gross shares withheld at vest (tax). Schema default 0.5; UI default 0.51. |
| Opening balance | Auto-generated `income` transaction (`description = 'Opening balance'`) from `initial_balance` at account creation; excluded from income totals. |
| Manual override | `accounts.manual_override_balance`: replaces the calculated balance in net worth (interpreted in the account's currency); calculation resumes when cleared. |
| Snapshot | One `net_worth_snapshots` row per day (midnight job): `total_eur` + per-account `breakdown_json`. Feeds the History chart. |
| Holding | Derived position row (unique per account+ticker): `shares` and `avg_cost_basis_per_share`, always recomputed from the stock-transaction log. |
| Funded buy | Stock trade plus an atomic bank cash leg typed `transfer`. |
## When NOT to use this skill
- Running CLI commands against the live API → **fin** skill (`.claude/skills/fin/SKILL.md`).
- Architecture/layering rules, what code goes where → **fin-architecture-contract**.
- Making changes safely (migrations, review gates) → **fin-change-control**; building/running → **fin-build-and-env**, **fin-run-and-operate**.
- Debugging a wrong number in the running system → **fin-debugging-playbook**, **fin-diagnostics-and-tooling**; past incidents → **fin-failure-archaeology**.
- Querying/analyzing the ledger data itself → **fin-ledger-analysis-toolkit**; tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**; open questions → **fin-research-frontier**.
## Provenance and maintenance
All facts verified 2026-07-05 against the working tree at commit 082b608. Re-verify before trusting:
- Balance formula: `grep -n "amount - models.Transaction.fee" api/app/services/net_worth.py`
- Transaction types: `grep -n "type = Column" api/app/models.py` and `grep -n "Literal\[" api/app/schemas.py`
- Two-leg writers: `grep -n "def create_\(transfer\|settlement\|split_expense\|splitwise_i_paid\|funded_buy\)" api/app/crud.py`; reconcile: `grep -n "def reconcile_accounts" api/app/routers/transactions.py`
- Spending filters: `grep -n "notin_\|splitwise_partner_share\|Opening balance" api/app/crud.py` (expect `notin_(["receivable"])` for spending, `notin_(["payable", "receivable"])` for income)
- Net worth & FX-missing→0: `grep -n "_to_eur\|return 0.0" api/app/services/net_worth.py`
- Holding recompute (avg cost, sell guard): `sed -n '453,520p' api/app/crud.py`
- Vest floor + price-cache currency: `grep -n "math.floor\|price_row" api/app/crud.py` (in `vest_rsu_grant`)
- Schedule tranche math: `grep -n "_generate_rsu_schedule" api/app/crud.py api/app/routers/rsu_grants.py`
- FX pairs & pence handling: `grep -n "frankfurter.dev\|GBp\|GBX" api/app/services/price_fetcher.py`
- Retype migration: `cat api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py`
- UI withholding default: `grep -n "withholding_pct: '51'" web/src/pages/Portfolio.tsx`; unvested dashboard calc: `grep -n "unvestedEurByAccount" web/src/pages/Dashboard.tsx`
- CLI command surface: `grep -n "add_typer\|app.command" cli/fin_cli/main.py`
- Key commits: `git log --oneline` — e36d719 (funded-buy transfer typing), h8c9d0e1f2a3 in bc256f6-era (retype migration), e61b254 (payable in spending), a9f045f (receivable/payable exclusion + summary), babea72 (unvested display), 81091fc (no FX fallback), 225e69e (floor net shares), c835118 (51% UI default)
@@ -0,0 +1,201 @@
---
name: fin-architecture-contract
description: >-
Load-bearing design decisions, invariants, and known weak points of the fin
app. Load BEFORE any design or refactor decision, before adding endpoints,
tables, services, account types, or transaction types, before changing data
flow between layers (router/crud/services/frontend), or whenever you ask
"why is it built this way?" or "what breaks if I change this?". Also load
when reviewing a diff that touches crud.py, models.py, scheduler.py, auth.py,
or web/src/api/client.ts.
---
# fin — Architecture Contract
The design decisions that hold this app together, WHY they exist, and where it
is honestly weak. Every claim below was verified by reading the code on
2026-07-05 (commit 082b608). Trust this file over memory; trust the code over
this file.
Jargon used once, defined once:
- **Leg** — one of the two (or three) `Transaction` rows written together for a
transfer/settlement/splitwise event, linked via `linked_account_id` and
`linked_transaction_id`.
- **Derived state** — a table whose contents can be rebuilt from another table
(holdings from stock_transactions). Never edit derived state directly.
- **Opt-in auth** — bearer-token check that is a no-op unless the API's
`FIN_API_TOKEN` env var is set.
## 1. System map
Three write paths reach the same SQLite file. Know which one you are on.
```
Browser (React SPA) fin CLI (typer)
web/src/api/client.ts cli/fin_cli/client.py (httpx)
fetch('/api/...') NO auth header Bearer token if FIN_API_TOKEN set
│ │
▼ │
nginx :80 (web container) │ direct HTTP to API base URL
/api/* → proxy_pass http://${API_UPSTREAM} │ (FIN_API_URL, no default)
/* → SPA static files + index fallback │
│ │
▼ ▼
uvicorn :8000 (api container, non-root uid 1000)
FastAPI app (main.py) ── Depends(require_api_token) on every router
│
routers/* ──► crud.py / services/* ──► SQLite /data/fin.db (WAL)
▲
APScheduler BackgroundScheduler (same process, │
background thread, started in lifespan): │
• price_refresh — every 5 min │
• daily_snapshot — cron 00:00 │
Each job run builds its OWN engine + session ────┘
(services/scheduler.py:_make_engine), never the request-scoped session.
```
Why the scheduler has its own engine (`services/scheduler.py`): jobs run in
APScheduler's thread pool, outside any request. Sharing `SessionLocal`
request sessions across threads is unsafe; instead each job run calls
`create_engine(...)`, applies the same `PRAGMA journal_mode=WAL; PRAGMA
foreign_keys=ON` connect listener as `database.py`, and disposes the engine in
`finally`. WAL is what lets this second writer coexist with request traffic.
Job definitions persist in the same DB via `SQLAlchemyJobStore`
(table `apscheduler_jobs`).
Container startup order (`api/Dockerfile` CMD): `python -m app.preflight`
(verifies the DB dir is writable) → `alembic upgrade head` → uvicorn. The
`web` container waits on the api healthcheck (`/api/health`) before starting.
Images are pushed to `harbor.cloud.traunseenet.com/library/fin-{api,web}`.
## 2. Layering contract
| Layer | Owns | Must NOT do |
|---|---|---|
| `routers/*` | Input validation, HTTP status mapping, `Depends(get_db)` | Business logic, DB mutations (two known violations — see §5) |
| `crud.py` | ALL DB mutations, multi-row atomic writes | HTTP concerns (it does import HTTPException in places — tolerated, don't extend) |
| `services/` | Logic spanning tables or external APIs (net worth, prices, scheduler) | Being called synchronously from a request handler if it sleeps (`fetch_all_prices` retries with `time.sleep`) |
| `web/src/api/client.ts` | Every frontend HTTP call | — nothing else in `web/src` may call `fetch()` directly |
| `cli/fin_cli/client.py` | Every CLI HTTP call; zero typer/rich imports (MCP-ready) | Importing CLI presentation deps |
Frontend refresh contract (verified in `web/src`): after a successful write,
`Layout.tsx`, `Dashboard.tsx`, and `Settings.tsx` dispatch
`window.dispatchEvent(new CustomEvent('fin:transaction-added'))`; the pages
Dashboard, Transactions, Spending, and Portfolio re-fetch on that event in a
`useEffect`. This is the ONLY cross-component refresh mechanism — there is no
global state manager. Breaking it makes pages silently stale.
## 3. Load-bearing decisions
| Decision | Why | What breaks if changed casually |
|---|---|---|
| **Sync SQLAlchemy, sync endpoints** (`database.py`: `create_engine`, `sessionmaker`) | aiosqlite just wraps a thread anyway; FastAPI already runs sync handlers in a threadpool. Simpler, no async-session lifetime bugs. | Introducing `async def` endpoints with the sync engine blocks the event loop; introducing `AsyncSession` breaks every function in crud.py and the scheduler. |
| **SQLite + WAL + `foreign_keys=ON` on every connection** (both `database.py` and `scheduler.py:_make_engine` register a connect listener) | Two writers (requests + scheduler thread) need concurrent readers; FK pragma is per-connection in SQLite, so it must be re-applied on connect. | Dropping the listener on either engine silently disables FK enforcement / reintroduces writer starvation for that engine only — hard to spot. |
| **Scheduler builds a throwaway engine per job run** | Thread safety; also guarantees fresh pragmas and no pooled connections leaking across jobs. `engine.dispose()` in `finally`. | Reusing the request `SessionLocal` in a job = cross-thread session use; corrupted sessions under load. |
| **Holdings are derived state** — `recompute_holding_from_stock_transactions` (crud.py) is a full chronological fold of `stock_transactions`, called on BOTH insert (`create_stock_transaction`, `create_funded_buy`, `vest_rsu_grant`) and delete (`delete_stock_transaction`), before `db.commit()` | Single source of truth. The fold also validates history (selling more shares than held at that point raises `ValueError` → 422 and rolls back). Note: CLAUDE.md says "incrementally updated on insert" — the code as of 2026-07-05 does a full recompute on insert too. | Writing to `holdings` directly, or skipping the recompute call, desyncs positions from the log with no error until someone notices wrong P&L. |
| **Two-leg atomic linked writes** — `create_transfer`, `create_settlement`, `create_splitwise_i_paid` (2 legs), `create_split_expense` (3 rows: expense + 2 transfer legs), and the `/reconcile` route: `db.add_all([...]); db.flush()`; then set `linked_transaction_id` on each leg pointing at its partner; single `db.commit()` | Balances are `SUM(amount − fee)` per account, so an orphaned leg is a permanent balance error. `linked_transaction_id` lets `delete_transaction` walk the whole group (both directions) and delete it atomically. | Deleting one leg by raw SQL, or committing between legs, corrupts two account balances at once. Editing a linked leg is blocked at the router (`PUT /api/transactions/{id}` → 422). |
| **`price_cache` = one row per symbol, upserted** (`_upsert_price_cache` in price_fetcher.py; `symbol` is UNIQUE; columns include `name` for display) | No price history table by design — history lives in daily snapshots. FX convention: rows `EUR/USD` and `EUR/GBP` (both fetched from frankfurter.app, as of 2026-07-05). Ticker prices are stored in their native currency (`currency` column); yfinance GBp/GBX pence quotes are divided by 100 and stored as GBP. `net_worth._to_eur` converts via `EUR/<ccy>` and EXCLUDES the amount (0.0 + warning) if no rate is cached. | Adding multiple rows per symbol breaks every `filter_by(symbol=...)` `.first()` read. Changing FX symbol naming breaks `_to_eur`'s `f"EUR/{ccy}"` lookup. |
| **`manual_override_balance`** (Account, nullable float) | When not NULL it REPLACES the computed balance in net worth (`is_override` in the breakdown). For investment/rsu accounts the override is assumed to be in the account's declared currency and skips holdings entirely. | Treating it as an adjustment (additive) double-counts. Note the separate `/accounts/{id}/adjust-balance` endpoint which instead writes a real income/expense delta transaction — two different mechanisms, don't conflate them. |
| **Opt-in bearer auth** (`auth.py:require_api_token`) | Single-user homelab: zero-config by default. Token read from env AT REQUEST TIME (rotatable without restart), compared with `secrets.compare_digest`. `/api/health` is unauthenticated (Docker healthcheck needs it). Docs/redoc/openapi are disabled on the app; the OpenAPI spec is served from an explicit `GET /api/openapi.json` route with the auth dependency, because FastAPI's built-in openapi route bypasses DI (see comment in main.py). Every router is included with `dependencies=[Depends(require_api_token)]` in main.py. | Known tradeoff: setting `FIN_API_TOKEN` on the API locks out the React frontend — `client.ts` sends no Authorization header. Mitigation options (nginx header injection, frontend token) are NOT implemented as of 2026-07-05. Also: a new router added without the `_auth` dependency list is silently unauthenticated. |
| **Soft-delete accounts** (`is_active=False` via `DELETE /api/accounts/{id}` → `crud.deactivate_account`) | Transactions keep their FK; history and old snapshots stay interpretable. Net worth and account lists filter `is_active.is_(True)`. | Hard-deleting an account orphans transactions and breaks FK integrity (FKs are ON). There is no undelete endpoint — flip `is_active` via `PUT`. |
| **Non-root Docker user** (api/Dockerfile: `useradd -m -u 1000 app`, `/data` chowned, `USER app`) | Container hardening; uid 1000 matches typical host volume ownership. | Reverting to root breaks the security posture and can flip `/data` file ownership so the non-root image can no longer write the volume. |
| **Bank leg of a funded stock trade is typed `transfer`** (`create_funded_buy`), and receivable/payable settles are typed `settlement` (`/reconcile`) | Asset conversion and debt repayment are not consumption/income; spending and income metrics filter by type, so these must not appear there (migration h8c9d0e1f2a3 retyped old rows). | Typing them `expense`/`income` inflates the Spending page — this was a real fixed bug (commit e36d719). |
## 4. Data-model contract
Eight tables in `models.py` as of 2026-07-05 (plus `apscheduler_jobs` owned by
APScheduler). Account `type` now includes `payable` — six values, enforced by
`Literal` in schemas.py, not by the DB (column is Text):
`bank | investment | rsu | splitwise | receivable | payable`.
(CLAUDE.md still lists five; schemas.py is the truth.)
| Table | Role | Source of truth or derived? |
|---|---|---|
| `accounts` | Top-level entity; `manual_override_balance`, `is_active`, `currency` | Source |
| `categories` | Spending/income buckets; `type` ∈ expense\|income; deleting one nulls `category_id` on its transactions | Source |
| `transactions` | Cash ledger for bank/splitwise/receivable/payable accounts | Source. **Balance = `SUM(amount − fee)`** — never `SUM(amount)`. Sign convention: negative = money out. Fee lives only on the from-leg of a transfer (to-leg fee=0). |
| `stock_transactions` | Append-only equity event log (buy\|sell\|rsu_vest) | Source for holdings |
| `holdings` | Current position per (account_id, ticker) — UNIQUE constraint | DERIVED: exact fold of stock_transactions. `avg_cost_basis_per_share = total_cost/shares` with average-cost sell accounting. Never write directly. |
| `rsu_grants` | Pending vest schedule; `vested_at` + `vested_stock_transaction_id` FK link a vested grant to the `rsu_vest` stock transaction it produced | Source. Deleting that stock transaction reverts the grant to pending first (crud.delete_stock_transaction), else the FK 500s. Vested grants cannot be deleted directly. |
| `price_cache` | One row per symbol (UNIQUE), columns: symbol, price, currency, source (yfinance\|frankfurter), fetched_at, `name` | Cache — safe to lose, refetched within 5 min |
| `net_worth_snapshots` | One row per day (UNIQUE date): total_eur + `breakdown_json` (per-account values frozen at snapshot time) | FROZEN valuation. Cannot be recomputed later — old prices are gone (price_cache keeps no history). The midnight job upserts today's row. |
Cross-table invariants:
1. Linked legs form a group closed under `linked_transaction_id` (both
directions); `delete_transaction` deletes the whole group in one commit.
Same-currency transfer legs net to `−amount` / `+amount` (fee reduces the
source side); cross-currency transfers (`to_amount`/`to_currency`) do NOT
sum to zero in either currency — that is expected.
2. Splitwise accounting: `splitwise_pay` bank leg has `amount = −total`,
`splitwise_partner_share = partner's part`; the user's true expense is
`amount + splitwise_partner_share`. `get_monthly_spending` (crud.py) counts
it exactly that way in a separate per-row pass. Do not "simplify" it into
the grouped SUM query.
3. Spending excludes transactions on `receivable` accounts; income excludes
`payable`/`receivable` accounts AND rows described `'Opening balance'`
(magic string — see §5).
4. Migration chain (8 revisions): `a1b2c3d4e5f6 → b2c3d4e5f6a7 → c3d4e5f6a7b8
→ d4e5f6a7b8c9 → e5f6a7b8c9d0 → f6a7b8c9d0e1 → g7b8c9d0e1f2 →
h8c9d0e1f2a3`. `Base.metadata.create_all` in lifespan is first-boot
convenience only, never a migration strategy.
## 5. Known weak points (honest, as of 2026-07-05)
| # | Weak point | Risk | Blast radius | Current mitigation |
|---|---|---|---|---|
| 1 | **crud.py is a 995-line monolith with no direct tests.** No `api/tests/` exists; no `.github/` → no CI at all. The only automated coverage is `cli/tests/test_cli.py` (25 tests, e2e: FinClient → TestClient → SQLite), which exercises the API only through CLI-shaped paths. | High — this is the owner's stated hardest problem | Every balance/spending/holding number | The e2e suite; the invariants in this file. See sibling `fin-test-coverage-campaign`. |
| 2 | **Layering violations in routers**: `POST /api/transactions/reconcile` (routers/transactions.py) and `POST /api/accounts/{id}/adjust-balance` (routers/accounts.py) construct and commit `models.Transaction` rows directly in the router, bypassing crud.py. | Medium — atomicity is currently correct, but these paths are invisible to anyone auditing crud.py for mutation logic | Balances of bank + payable/receivable accounts | None. If you touch linked-write logic, remember these two live outside crud.py. |
| 3 | **Legacy linked-leg deletion heuristic**: rows written before `linked_transaction_id` existed (pre-e5f6a7b8c9d0) are matched by (accounts swapped, same date, same type, closest id) in `delete_transaction`. | Low-probability wrong-partner delete on old data | Two account balances | Closest-id preference; new writes always set `linked_transaction_id`. |
| 4 | **Auth lockout tradeoff**: enabling `FIN_API_TOKEN` on the API breaks the React frontend entirely (client.ts sends no auth header; nginx injects nothing). | Certain, not probabilistic — it WILL lock out the UI | Whole web UI | Documented in CLAUDE.md; token currently meant for CLI-only deployments. |
| 5 | **yfinance has no SLA, and `fetch_ticker_price` sleeps in-process**: retry schedule 1s/2s/4s per candidate × up to 9 candidates (bare + 8 EU exchange suffixes). `POST /api/prices/fetch-ticker` calls it SYNCHRONOUSLY in the request handler — worst case blocks that worker for minutes. `POST /api/prices/refresh` correctly defers to BackgroundTasks (202). | Medium | One threadpool worker per stuck request; stale prices on Yahoo outage | Stale-cache fallback everywhere (`_to_eur` warnings, cached EUR/USD, `EUR/USD→1.0` fallback — which itself silently skews net worth when cache is empty). |
| 6 | **No DB backup automation anywhere in the repo** — the prod SQLite volume (`sqlite_data:/data/fin.db`) is sacred and unprotected. Snapshots are frozen JSON, so data loss is unrecoverable, not just inconvenient. | High impact, unmitigated | All financial history | None in repo. Treat any operation touching `/data` as irreversible; see siblings `fin-run-and-operate` / `fin-change-control`. |
| 7 | **Money as Float**: every amount/fee/price column is `Column(Float)`; balance math is IEEE-754. The adjust-balance endpoint already needs an `abs(delta) < 0.001` epsilon. | Low per-transaction, accumulates | Cosmetic cent drift in balances | Epsilon comparisons; single-user scale keeps it tolerable. Do not "fix" ad hoc — a Decimal migration is a project. |
| 8 | **Magic-string semantics**: monthly income excludes rows where `description = 'Opening balance'` (crud.get_monthly_spending). Renaming that description in `create_account` breaks income stats silently. | Low | Spending page income numbers | The string is set in exactly one place (`crud.create_account`). |
| 9 | **Unauthenticated-by-omission routers**: auth is applied per-router in main.py (`dependencies=_auth`); a newly added router that skips the list is open even when the token is set. | Medium when auth is on | New endpoints | Convention only — check main.py in review. |
## 6. Extension seams — exact touch order
**New account type** (no migration needed — `accounts.type` is Text):
1. `api/app/schemas.py` — add to both `Literal[...]` lists (AccountCreate, AccountUpdate).
2. `api/app/services/net_worth.py` — decide: cash-like branch (`SUM(amount−fee)`) or holdings branch; unknown types are skipped with a warning.
3. `api/app/routers/accounts.py` — extend the adjust-balance allowlist if cash-like.
4. `api/app/crud.py` — review the `Account.type.notin_([...])` exclusion lists in `get_monthly_spending` (spending AND income queries).
5. `web/src` — Settings account form + Dashboard grouping; API types in `client.ts`.
6. Update the comment in `models.py` and CLAUDE.md's data-model paragraph.
**New transaction type**:
1. `schemas.py` `TransactionCreate.type` Literal → 2. `crud.py` creator (multi-leg? follow the `add_all → flush → set linked_transaction_id → commit` pattern exactly) → 3. spending/income filters in `get_monthly_spending` (does it count as consumption, income, or neither?) → 4. `delete_transaction` group semantics (linked legs are handled automatically if you set `linked_transaction_id`) → 5. router endpoint → 6. `client.ts` function → 7. `web/src/components/QuickAdd/flows/` component that calls `onSuccess()` → 8. CLI: `cli/fin_cli/client.py` method + subcommand module.
**New endpoint on an existing resource**:
router function (validation + HTTP mapping only) → crud/service function (all DB work) → schemas → confirm the router is registered in `main.py` WITH `dependencies=_auth` → `client.ts` → CLI client + command → e2e test in `cli/tests/test_cli.py`.
**New table**: `models.py` → `alembic revision --autogenerate -m "..."` inside the api container (verified statically — no venv in this sandbox), review the generated file, chain it after `h8c9d0e1f2a3` → schemas → crud → router (+ `_auth`) → clients. Never `create_all` as migration, never raw DDL.
**New page**: `web/src/pages/` → route in `App.tsx` → data via `client.ts` only → listen for `fin:transaction-added` if it displays transaction-derived data.
## When NOT to use this skill
- Recording expenses / querying the live app → sibling `fin` (CLI usage).
- Deploy, restart, volume, or prod-DB operations → `fin-run-and-operate`, and `fin-change-control` before anything irreversible.
- Debugging a wrong number or a 500 → `fin-debugging-playbook`; past incidents → `fin-failure-archaeology`.
- What the accounting SHOULD be (splitwise math, RSU semantics) → `fin-accounting-reference`; bulk ledger queries → `fin-ledger-analysis-toolkit`.
- Local build/toolchain problems → `fin-build-and-env`; ad-hoc inspection tools → `fin-diagnostics-and-tooling`.
- Writing or expanding tests → `fin-test-coverage-campaign` and `fin-validation-and-qa`.
- Speculative redesigns (Postgres, async, multi-user) → `fin-research-frontier` — but re-read §3 first; most "obvious" upgrades break a listed invariant.
## Provenance and maintenance
All facts verified by reading the repo at commit 082b608 on 2026-07-05. Re-verify before relying on volatile items:
- Account/transaction type lists: `grep -n "Literal" api/app/schemas.py`
- Table set + columns: read `api/app/models.py` (30 s)
- Migration head: `ls api/alembic/versions/` and check `down_revision` chain
- Scheduler jobs + intervals: read `api/app/services/scheduler.py:create_scheduler`
- Auth wiring: `grep -n "dependencies" api/app/main.py` (every router must carry `_auth`)
- FX symbols cached: `grep -n "EUR/" api/app/services/price_fetcher.py`
- Frontend auth header still absent: `grep -n "Authorization" web/src/api/client.ts` (expect no hits)
- Event bus intact: `grep -rn "fin:transaction-added" web/src`
- Test/CI status: `ls api/tests .github cli/tests` (as of 2026-07-05: no api/tests, no .github, 25 tests in cli/tests/test_cli.py)
- Router-layer mutation exceptions: `grep -n "db.add" api/app/routers/*.py`
+201
View File
@@ -0,0 +1,201 @@
---
name: fin-build-and-env
description: Recreate the fin dev/build environment from scratch and avoid known toolchain traps. Load when setting up a dev environment on a fresh machine or sandbox, running the pytest suite locally, building the Docker images (api or web), installing the CLI, or hitting build/toolchain errors (pip/venv missing, npm ci failures, alembic path errors, yfinance import weight, SQLite WAL files).
---
# fin — build and environment runbook
Read `/CLAUDE.md` at the repo root first if you have not. This skill covers only
"how do I get a working environment"; siblings cover everything else.
All facts below were **verified statically** against the named files as of
2026-07-05. Nothing in this file was executed in the authoring session (the
authoring sandbox had no pip, venv, docker, or node).
## Prerequisites by task
| Task | Needs | Version source (verified statically) |
|---|---|---|
| (a) Full-stack run | Docker + docker compose. Nothing else — both images build everything internally. | `docker-compose.yml`, `api/Dockerfile`, `web/Dockerfile` |
| (b) Backend tests/dev without Docker | Python ≥ 3.12, pip, venv. No node, no docker. | `api/Dockerfile` (`FROM python:3.12-slim`), `cli/pyproject.toml` (`requires-python = ">=3.12"`) |
| (c) Frontend dev/build | Node 20 (Docker uses `node:20-alpine`), npm; the image upgrades to npm 11 before `npm ci` (see traps). `package.json` declares no `engines` field. | `web/Dockerfile`, `web/package.json` |
| (d) CLI install | Python ≥ 3.12 + pip. Deps: `typer>=0.12`, `httpx>=0.27`, `rich>=13`. Build backend: hatchling. | `cli/pyproject.toml` |
Python version note (as of 2026-07-05): Docker runs Python 3.12
(`python:3.12-slim`), but the test suite was last executed under **CPython 3.13
with pytest 9.1.1** — evidenced by `cli/tests/__pycache__/test_cli.cpython-313-pytest-9.1.1.pyc`.
Both work; `requires-python = ">=3.12"` is the floor.
`api/requirements.txt` (mostly unpinned — resolves to latest at build time):
`fastapi`, `uvicorn[standard]`, `sqlalchemy`, `alembic`, `pydantic>=2.0`,
`yfinance`, `apscheduler>=3.10`, `httpx`.
## (a) Full-stack Docker run
```bash
docker compose up --build
```
Anatomy (verified statically against `docker-compose.yml`, `api/Dockerfile`,
`web/Dockerfile`, `web/nginx.conf`):
- **api** service: builds `./api`, image tag `harbor.cloud.traunseenet.com/library/fin-api:latest`.
Dockerfile CMD, verbatim:
```
CMD sh -c "python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000"
```
`app.preflight` validates the SQLite directory exists and is writable
(`app/db_config.ensure_sqlite_database_dir`), then Alembic migrates, then
uvicorn starts. Runs as non-root user `app` (uid 1000) — do not revert.
`DATABASE_URL=sqlite:////data/fin.db`; named volume `sqlite_data` → `/data`.
- **api healthcheck**: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')"`
every 30s, timeout 10s, 3 retries.
- **web** service: builds `./web`, publishes port 80, and `depends_on: api:
condition: service_healthy` — nginx does not start until the API healthcheck
passes. Env `API_UPSTREAM=api:8000`.
- **web image**: two-stage. Builder: `node:20-alpine`, `npm install -g npm@11 && npm ci`,
`npm run build` (= `tsc && vite build`). Serve stage: `nginx:alpine`,
dist → `/usr/share/nginx/html`, and `nginx.conf` copied to
`/etc/nginx/templates/default.conf.template` — the stock nginx:alpine
entrypoint runs envsubst on `/etc/nginx/templates/*.template`, substituting
`${API_UPSTREAM}` in the `proxy_pass http://${API_UPSTREAM};` line. nginx
proxies `/api/*` to the API, serves the SPA with `try_files` fallback, and
sends `Cache-Control: no-cache` on `index.html`.
DB location: named volume `sqlite_data` mounted at `/data`, file `/data/fin.db`
(plus `fin.db-wal` / `fin.db-shm`, see traps).
## (b) Backend test environment WITHOUT Docker — the path you will actually use
```bash
cd /path/to/fin
python -m venv .venv
. .venv/bin/activate
pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0"
pip install -e cli/ # pulls typer, httpx, rich; installs the `fin` entry point
python -m pytest cli/tests/ -q
```
Minimal-dependency rationale (verified statically against
`cli/tests/conftest.py` and module-scope imports in `api/app/`):
- `conftest.py` imports `fastapi.testclient.TestClient`, `app.database`,
`app.main`, `fin_cli.client`. `app.main` imports the routers and
`app.services.scheduler`, which imports **apscheduler** at module scope — so
apscheduler is required even though the scheduler never runs.
- **yfinance is deliberately NOT needed**: conftest stubs it before the app
import chain runs — `sys.modules.setdefault("yfinance", types.ModuleType("yfinance"))`.
It is a heavy package (pandas etc.); do not install it just to run tests.
- **alembic is NOT needed for tests**: the fixture uses
`Base.metadata.drop_all` / `create_all` per test, not migrations.
- `uvicorn` is NOT needed for tests (only to actually serve the API).
conftest mechanics (read `cli/tests/conftest.py`, 59 lines):
1. Computes `REPO_ROOT = Path(__file__).resolve().parents[2]` and inserts
`REPO_ROOT/api` and `REPO_ROOT/cli` into `sys.path` — **absolute paths, so
the working directory does not matter**; `python -m pytest cli/tests/ -q`
from the repo root and `python -m pytest tests/ -q` from `cli/` both work.
2. Sets a throwaway `DATABASE_URL` (`sqlite:///<mkdtemp>/test.db`) **before**
importing `app.database`, which reads it at import time. Your real DB is
never touched.
3. Autouse `clean_db` fixture: deletes `FIN_API_TOKEN` from the env, then
`drop_all` + `create_all` per test — every test starts on an empty schema.
4. The APScheduler job never starts: tests use `TestClient(fastapi_app)`
without running the app's lifespan.
5. `make_client` / `client` fixtures wrap the in-process app in a `FinClient`
(`cli/fin_cli/client.py`, plain httpx, no typer/rich imports) — the tests
are true e2e: FinClient → real FastAPI app → real crud → SQLite.
Expected result: **25 tests** (counted statically: 25 `def test_` functions in
`cli/tests/test_cli.py`, no parametrize), all passing as of 2026-07-05.
To run the API itself without Docker, additionally `pip install uvicorn alembic yfinance`,
then from `api/`: `alembic upgrade head && uvicorn app.main:app --port 8000`
(see the alembic trap below; running the server is fin-run-and-operate territory).
## (c) Frontend build
```bash
cd web
npm ci # NOT npm install — lockfile is the contract (commit bfe7590)
npm run build # = tsc && vite build → dist/
npm run dev # vite dev server (script exists in package.json)
```
- **Lockfile discipline**: `bfe7590` switched the Dockerfile from
`COPY package.json` + `npm install` to `COPY package.json package-lock.json`
+ `npm ci` for reproducible builds. Keep `package-lock.json` committed and
in sync.
- **Dev-server honesty**: CLAUDE.md says "No separate dev server setup exists"
— meaning there is no documented/wired dev workflow (no proxy config for
`/api` in `vite.config.ts` is guaranteed; check before relying on it). But
`package.json` does define `dev: vite` and `preview: vite preview` scripts.
The supported path is `docker compose up --build`.
- Tailwind 3.4 + shadcn/ui structure: `tailwind.config.js`,
`postcss.config.js`, components under `src/components/ui/`. New Radix
primitives are scaffolded per `web/components.json` (style `default`,
baseColor `slate`, aliases `@/components`, `@/lib/utils`). Do not add new UI
libraries.
- `package.json` also carries `allowScripts: { "esbuild@0.21.5": true }`
(added alongside the registry migration, see traps).
## (d) CLI install
```bash
pip install -e cli/ # from repo root; entry point: fin = fin_cli.main:app
export FIN_API_URL=http://<host> # required, no default
export FIN_API_TOKEN=<token> # only if the API enforces auth
fin health
```
Agent-facing usage docs: `.claude/skills/fin/SKILL.md`.
## Known traps
| Trap | Story / fix |
|---|---|
| Debian/Ubuntu: `python -m venv` fails (`ensurepip` missing) | Minimal Debian images and some sandboxes ship python3 without `python3-venv`/`python3-pip`. Fix: `apt install python3-venv python3-pip`, or use `python -m venv --without-pip` + a pip bootstrap, or `uv venv` if uv exists. The authoring sandbox for this skill had neither pip nor venv — that is why nothing here was executed. |
| npm on Alpine: silent/failed `npm ci` exit-handler bug | Commit `89d241a` (2026-07-01): `web/Dockerfile` prepends `npm install -g npm@11` before `npm ci` because the npm bundled with `node:20-alpine` hits an Alpine exit-handler bug. If you touch the web Dockerfile, keep the npm 11 upgrade line. |
| Lockfile pointed at internal Dynatrace npm mirror | Commit `3c2766e` (2026-07-01) rewrote `package-lock.json` `resolved` URLs from the Dynatrace mirror to `https://registry.npmjs.org`. If `npm ci` fails resolving packages off-VPN, check `grep -c 'registry.npmjs.org' web/package-lock.json` — every `resolved` must be the public registry. Never regenerate the lockfile while an internal `.npmrc` registry override is active. |
| Installing yfinance to run tests | Unnecessary. `cli/tests/conftest.py` stubs `sys.modules["yfinance"]` precisely because yfinance is a heavy module-scope import in `app/services/price_fetcher.py`. Only install it to run the real server/price fetcher. |
| Stray `fin.db-wal` / `fin.db-shm` files | WAL mode is applied on every connection (`PRAGMA journal_mode=WAL`, invariant #3 in CLAUDE.md), so the DB is three files. Never copy/back up `fin.db` alone while the app runs; never delete the `-wal` file — it holds unmerged committed data. |
| `alembic upgrade head` fails with "No 'script_location'…" or can't import app | Alembic must run **from `api/`**: `api/alembic.ini` has relative `script_location = alembic` and `prepend_sys_path = .`; the Docker CMD runs it with WORKDIR `/app` (= the api dir). `alembic/env.py` also inserts the api dir into `sys.path` and reads `DATABASE_URL` via `app.db_config` (default `sqlite:///./fin.db` per `db_config.py`; the `sqlalchemy.url` in alembic.ini is a fallback). So: `cd api && alembic upgrade head`, with `DATABASE_URL` exported if you don't want `./fin.db`. |
## Quick sanity checklist after setup
```bash
# 1. Backend modules compile (mirrors the allowlisted command in .claude/settings.local.json; run from api/)
cd api && python -m py_compile app/crud.py app/schemas.py app/routers/transactions.py app/services/net_worth.py app/services/scheduler.py app/services/price_fetcher.py
# 2. Test suite green — expect 25 passed (count as of 2026-07-05)
cd .. && python -m pytest cli/tests/ -q
# 3. CLI reaches a running stack
FIN_API_URL=http://localhost fin health # or :8000 if hitting uvicorn directly
```
## When NOT to use this skill
- Operating/deploying an already-built stack, backups, price refresh, scheduler behavior → **fin-run-and-operate**
- Using the `fin` CLI against a live API (commands, flows) → **fin** skill
- Diagnosing runtime bugs, log reading → **fin-debugging-playbook**, **fin-diagnostics-and-tooling**
- Making/reviewing code changes, migrations policy → **fin-change-control**, **fin-architecture-contract**
- Writing or expanding tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**
- Accounting semantics (balances, spending math) → **fin-accounting-reference**, **fin-ledger-analysis-toolkit**
- Historical incidents → **fin-failure-archaeology**; open research → **fin-research-frontier**
## Provenance and maintenance
Everything above was verified **statically** on 2026-07-05; nothing was executed. Re-verify with:
- Base images / CMD: `grep -E 'FROM|CMD' api/Dockerfile web/Dockerfile`
- Compose wiring (healthcheck, depends_on, volume): `cat docker-compose.yml`
- Python deps: `cat api/requirements.txt cli/pyproject.toml`
- Test count: `grep -c 'def test_' cli/tests/test_cli.py` (25 as of 2026-07-05)
- Last-run interpreter: `ls cli/tests/__pycache__/` (cpython-313, pytest 9.1.1 as of 2026-07-05)
- conftest mechanics: `sed -n '1,40p' cli/tests/conftest.py`
- npm traps: `git show 89d241a bfe7590 3c2766e --stat` and `grep -vc 'registry.npmjs.org' <(grep '"resolved"' web/package-lock.json)` (should be 0)
- Alembic cwd requirement: `grep -E 'script_location|prepend_sys_path' api/alembic.ini`
+266
View File
@@ -0,0 +1,266 @@
---
name: fin-change-control
description: Change-control rulebook for the fin repo — how to classify a change (trivial / behavior / schema / accounting-logic / deploy-affecting), which gate each class requires, the non-negotiable invariants with the historical incident behind each, the prod-DB-is-sacred rule, Alembic migration discipline, the spec→plan workflow in docs/superpowers, docs-of-record house style, and the done-checklist. Load BEFORE making any code, schema, or accounting change to fin; before writing or reviewing an Alembic migration; before touching crud.py transfer/settlement/holding/spending logic; before anything that alters Dockerfiles, docker-compose.yml, or env vars; and when writing specs, plans, commit messages, or updating CLAUDE.md/README.md.
---
# fin change control
How changes to `fin` are classified, gated, reviewed, and documented. `fin` is a single-user personal-finance app whose production SQLite database holds the owner's **real financial history** — the entire process below exists to protect that ledger. Read `/CLAUDE.md` (rulebook) and `/PLAN.md` (original design) alongside this skill; nothing here overrides CLAUDE.md.
All file paths are repo-relative. Line numbers and the migration head are correct **as of 2026-07-05** — re-verify with the commands in "Provenance and maintenance" if time has passed.
## When NOT to use this skill
| You actually want to… | Use instead |
|---|---|
| Query/record finances via the `fin` CLI | `fin` (`.claude/skills/fin/SKILL.md`) |
| Debug a live bug | `fin-debugging-playbook` |
| Read the full story behind past incidents | `fin-failure-archaeology` |
| Understand layering/architecture rules in depth | `fin-architecture-contract` |
| Understand the double-entry/Splitwise accounting model | `fin-accounting-reference` |
| Build images / set up an environment | `fin-build-and-env` |
| Deploy, operate, back up on the homelab | `fin-run-and-operate` |
| Run tests / QA a change hands-on | `fin-validation-and-qa` |
| Write new API tests (the coverage campaign) | `fin-test-coverage-campaign` |
| Analyze ledger data | `fin-ledger-analysis-toolkit` |
This skill is the **gatekeeper**: it tells you what a change must satisfy before it counts as done.
## 1. Change classification
Classify every change before writing code. A change belongs to the **highest** matching row (a schema change that alters spending math is an accounting-logic change).
| Class | Examples | Required gate |
|---|---|---|
| **Trivial fix** | Typo, label text, log message, comment, dead-code removal | Compile/build check (§7 items 1 & 3). Confirm no invariant file (`crud.py`, `net_worth.py`, migrations) was touched. |
| **Behavior change** | New endpoint, new QuickAdd flow, new CLI command, changed validation | Trivial gate **+** `pytest cli/tests` green **+** a test covering the new behavior **+** README/CLAUDE.md update if a convention or config var changed. |
| **Schema change** | New column/table/index, retyped column | Behavior gate **+** Alembic migration per §4 (autogenerated, reviewed, backward-safe) **+** scratch-DB `alembic upgrade head` dry-run **+** spec/plan (§5) if it also spans the frontend. |
| **Accounting-logic change** | Anything touching balances, spending/income math, holdings recompute, net-worth calc, transfer/settlement legs (`api/app/crud.py`, `api/app/services/net_worth.py`) | Schema gate (if schema involved) **+** written spec in `docs/superpowers/specs/` **+** re-verify invariants 1, 2, 5, 6 in §3 **+** if the *meaning* of existing rows changes, a data migration retyping history (precedent: `h8c9d0e1f2a3`, §4.4). |
| **Deploy-affecting change** | `api/Dockerfile`, `web/Dockerfile`, `docker-compose.yml`, `web/nginx.conf`, any migration, new env var | Relevant gate above **+** prod DB backup before rollout (§2.3) **+** README "Environment / Config" table updated for new vars **+** remember: the api container runs `alembic upgrade head` automatically on start (§2.4). |
"Gate" means: do not declare the change done, and do not deploy, until every item passes.
## 2. Non-negotiable zero: the production database is SACRED
The prod DB is the SQLite file `/data/fin.db` inside the Docker named volume `sqlite_data` on the homelab (see `docker-compose.yml`; `DATABASE_URL=sqlite:////data/fin.db`). It contains real, irreplaceable financial history. There is no upstream to re-sync from.
### 2.1 Never test against it
- Tests use throwaway databases only. `cli/tests/conftest.py` creates a temp-dir SQLite file and sets `DATABASE_URL` before importing the app — keep that pattern for all future tests.
- Never point a local checkout, a REPL, `alembic`, or the `fin` CLI's write commands at the prod volume "just to try something".
- Analytical *reads* against a **copy** of the DB are fine (see `fin-ledger-analysis-toolkit`); take the copy with the backup command below.
### 2.2 Backward-safe migrations only
- New columns must be **nullable or carry `server_default`** so existing rows stay valid (precedent: `f6a7b8c9d0e1` adds `categories.type` with `server_default='expense'`; design rationale in `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md`).
- No destructive column/table drops without a copy-forward plan (write the data somewhere first, verify, then drop in a later migration).
- Data migrations that rewrite rows must be surgical and documented in the migration docstring; if the downgrade cannot restore the data, say so explicitly in `downgrade()` (precedent: `h8c9d0e1f2a3`).
- SQLite cannot `ALTER COLUMN`/`DROP COLUMN` freely — use Alembic `batch_alter_table` when a migration needs more than `ADD COLUMN`.
### 2.3 Back up before any risky operation
Before deploying a migration or running any manual DB surgery, snapshot the prod DB. (Canonical backup/restore runbook: **fin-run-and-operate §3** — it uses the equivalent Python `sqlite3.backup()` API; keep the two in sync.) `VACUUM INTO` produces a consistent single-file copy even under WAL, and the `python:3.12-slim` image bundles a new-enough SQLite (≥ 3.27). Run on the homelab host (verified statically against `docker-compose.yml` and `api/Dockerfile` — no docker in the authoring sandbox):
```bash
# 1. Snapshot inside the container (writes into the same volume)
docker compose exec api python -c "import sqlite3; sqlite3.connect('/data/fin.db').execute(\"VACUUM INTO '/data/fin-backup-$(date +%F).db'\")"
# 2. Copy the snapshot off the container
docker compose cp api:/data/fin-backup-$(date +%F).db ./
```
Cold-backup alternative (stack stopped): `docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/out alpine tar czf /out/fin-db-backup.tgz -C /data .` — note the volume's real name carries the compose project prefix; find it with `docker volume ls | grep sqlite_data`.
### 2.4 Migrations auto-run on container start
`api/Dockerfile` CMD: `python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000`. Pushing an image with a bad migration to the Harbor registry means it **executes against prod on the next restart**. That is why migration review (§4) is a hard gate, not a courtesy.
## 3. The non-negotiables — invariant, rationale, incident, blast radius
Every invariant from CLAUDE.md, each with the commit history that motivated it. `git show <hash>` for the full story; `fin-failure-archaeology` for narratives.
### 3.1 Atomic two-leg writes
- **Rule**: `create_transfer` and `create_settlement` in `api/app/crud.py` write both legs with `db.add_all([...]); db.commit()` in one transaction (lines 233/269; the split-expense triple at line 350). Legs are cross-linked via `linked_transaction_id`. Never create, update, or delete a single leg.
- **Rationale**: each leg is one side of a double-entry pair; a lone leg silently corrupts two account balances at once.
- **Incidents**: `a06f921` — deletes left orphaned paired rows "corrupting balances"; `a1f109b` — the account-pair lookup used for paired deletes was ambiguous, so `linked_transaction_id` was added for precise pairing; `a1289f6` / `2bd4730` — API and UI had to *block* edits on linked legs; `9d06e1b` — UI still showed orphan legs after delete; `bc256f6` — split expense was two sequential HTTP requests that could partially fail, replaced by the atomic `POST /api/transactions/split-expense`; `e36d719` — `delete_transaction` now follows `linked_transaction_id` in both directions and deletes the whole group atomically. The CLI enforces this too: `fin tx add` refuses `transfer`/`settlement` types (`082b608`).
- **If violated**: balances on two accounts diverge from the transaction log; net worth is wrong; no error is raised anywhere.
### 3.2 Holding recompute on delete
- **Rule**: `delete_stock_transaction` (crud.py:522) calls `recompute_holding_from_stock_transactions` after the delete and **before** `db.commit()`. Holdings are derived state — never edit them directly.
- **Rationale**: `Holding` rows are a cache of the `stock_transactions` log; any log mutation that skips the recompute desynchronizes portfolio value and cost basis.
- **Incidents**: `1480088` — stock-transaction delete and funded-buy bugs plus an over-sell guard; `bc256f6` — deleting a vest transaction hit an FK 500 and made vested grants undeletable (deadlock: 422 on grant delete, 500 on transaction delete); fixed by reverting the grant to pending inside the same atomic delete.
- **If violated**: portfolio shows shares that were never bought or misses shares that were; avg cost basis and P&L are fiction.
### 3.3 WAL mode on every connection
- **Rule**: `PRAGMA journal_mode=WAL` is applied per-connection in `api/app/database.py:22` **and** in the scheduler's private engine `api/app/services/scheduler.py:25`. The scheduler thread must never share a request-scoped session.
- **Rationale**: founding design decision (PLAN.md: "Multiple concurrent reads (API + scheduler) without locking; mandatory for this pattern"). Preventive — no incident on record, keep it that way.
- **If violated**: `database is locked` errors when the midnight snapshot job overlaps API traffic.
### 3.4 Non-root Docker user
- **Rule**: `api/Dockerfile` creates user `app` (uid 1000) and runs as it. Do not revert to root.
- **Incident**: `ffa28bb` — "security hardening and startup performance — non-root user, disable docs, async prices, remove blocking startup fetch".
- **If violated**: a container escape or dependency compromise gets root on the homelab host's kernel; also `/data` ownership breaks (`chown app:app /data` is baked into the image).
### 3.5 Balance = SUM(amount − fee)
- **Rule**: for bank/splitwise/receivable accounts, balance is `func.sum(Transaction.amount - Transaction.fee)` (`api/app/services/net_worth.py:88`). Never sum `amount` alone.
- **Incident**: `1480088` — "fee balance" was one of the bug-hunt findings: fees were paid in reality but missing from computed balances.
- **If violated**: every account with fee-bearing transactions drifts upward from its true bank statement.
### 3.6 Spending counts the user's net share, and only real consumption
- **Rule**: in `get_monthly_spending` (crud.py:624), `splitwise_pay` expenses count at `amount + splitwise_partner_share` (the user's net share, not the full amount). Internal asset movements (transfers, settlements, funded stock-buy cash legs) are typed `transfer`/`settlement` precisely so spending/income queries exclude them.
- **Incidents**: `a9f045f` — payable/receivable excluded from spending; `e61b254` — then payable *expenses* had to be re-included (both directions of this mistake exist in history); `e36d719` + data migration `h8c9d0e1f2a3` — funded stock buys were typed `expense` and receivable/payable settles typed `income`/`expense`, inflating spending and income; new rows retyped and **historical rows migrated**.
- **If violated**: monthly spending/income analytics are wrong, which silently poisons every future forecasting/analytics ambition.
### 3.7 Never fabricate an FX rate
- **Rule**: if the EUR/USD (or EUR/GBP) rate is missing, fail loudly or return 0 with a warning — never fall back to `1.0` or the wrong currency. (Not a numbered CLAUDE.md invariant, but hard-won; treat as one.)
- **Incidents**: `6b7d19b` — `_to_eur` silently fell back to treating USD as EUR; `81091fc` — removed a `1.0` FX fallback in the price fetcher, skipping the upsert instead; `9b78d0e` — GBP fetch error handling aligned with USD; `424b926` / `bc256f6` — frontend hardcoded EUR instead of deriving currency from the selected account; `bc256f6` also made the remaining `1.0` fallback in `calculate_net_worth` log a warning.
- **If violated**: net worth is off by the full FX spread and nobody notices until a snapshot chart looks weird.
### 3.8 The remaining "What NOT to do" rules and their grounding
| Rule (CLAUDE.md) | Grounding |
|---|---|
| No `async` SQLAlchemy sessions | Design decision, PLAN.md: aiosqlite adds a threading layer anyway; FastAPI threadpools sync handlers. |
| No raw DDL / no hand-edits to the SQLite file; Alembic only | §2 + §4. `Base.metadata.create_all` will NOT fix drift (§4.3). |
| No new model columns without a migration | Same — `create_all` never alters existing tables, so prod silently lacks the column while tests (which create fresh) pass. |
| Never bypass `web/src/api/client.ts` | Single API layer; the auth work (`082b608`) shows why: headers/auth must be changeable in one place. |
| No auth/multi-user without understanding single-user assumptions | `082b608` added opt-in bearer auth and CLAUDE.md documents the trap: setting `FIN_API_TOKEN` on the API locks out the React frontend, which sends no `Authorization` header. |
| Never call `fetch_all_prices` synchronously on the hot path | `ffa28bb` removed a blocking startup fetch; `POST /api/prices/refresh` uses FastAPI `BackgroundTasks` (`api/app/routers/prices.py:20-22`). It does blocking HTTP with retry sleeps. |
| Use local dates, not UTC `toISOString()` | `bc256f6`: UTC dates booked trades/vests/settles to *yesterday* before ~1–2 AM local; frontend uses a `todayLocalISO()` helper. |
## 4. Alembic migration discipline
### 4.1 Workflow (exact)
```bash
cd api
# Point alembic at a scratch DB, never at anything real (env.py reads DATABASE_URL first):
export DATABASE_URL=sqlite:////tmp/mig-scratch.db
alembic upgrade head # bring scratch DB to current head
# 1. Edit app/models.py
alembic revision --autogenerate -m "add X to Y"
# 2. READ the generated file. Autogenerate misses server_default intent, index
# names, and SQLite batch requirements. Fix by hand.
alembic upgrade head # 3. Prove it applies
alembic downgrade -1 && alembic upgrade head # 4. Prove downgrade at least runs
# 5. Commit the migration together with the model change.
```
(Verified statically: `api/alembic/env.py` resolves the URL as `DATABASE_URL` env → `-x db_url` → `app.db_config.get_database_url()`; the `alembic.ini` fallback is `sqlite:///./fin.db`, which is why you must export `DATABASE_URL` — otherwise you create a stray `api/fin.db`. This sandbox cannot execute alembic.)
### 4.2 The chain (as of 2026-07-05, head = `h8c9d0e1f2a3`)
| Revision | down_revision | What |
|---|---|---|
| `a1b2c3d4e5f6` | *(None)* | initial schema |
| `b2c3d4e5f6a7` | a1b2c3d4e5f6 | transaction indexes |
| `c3d4e5f6a7b8` | b2c3d4e5f6a7 | missing indexes |
| `d4e5f6a7b8c9` | c3d4e5f6a7b8 | rsu_grants table |
| `e5f6a7b8c9d0` | d4e5f6a7b8c9 | `linked_transaction_id` (invariant 3.1) |
| `f6a7b8c9d0e1` | e5f6a7b8c9d0 | `categories.type`, `server_default='expense'` |
| `g7b8c9d0e1f2` | f6a7b8c9d0e1 | `price_cache.name` |
| `h8c9d0e1f2a3` | g7b8c9d0e1f2 | data migration: retype internal cash flows (invariant 3.6) |
Single linear chain, no branches. A new revision's `down_revision` must be the current head — run `alembic heads` (or read the files) before generating.
### 4.3 The `create_all` gotcha
`api/app/main.py:19` calls `Base.metadata.create_all(bind=engine)` in the lifespan. This is **first-boot convenience only** (fresh dev DBs, tests) — it creates missing *tables* but never alters existing ones. It is NOT a migration path. In prod the Dockerfile CMD runs `alembic upgrade head` before uvicorn ever starts, so migrations are the real mechanism. Consequence: a model change without a migration works in tests (fresh DB via `create_all`) and fails in prod — the most dangerous drift in this repo.
### 4.4 Data-migration template
`h8c9d0e1f2a3_retype_internal_cash_flows.py` is the house pattern: docstring explains the semantic bug and the fix; `op.get_bind()` + `sa.text()` for surgical `UPDATE`s; a compiled regex to avoid retyping user-entered rows that merely look similar (`Buy groceries` vs generated `Buy 5 VWCE`); an honest no-op `downgrade()` with a comment stating the original typing is not restored.
## 5. Spec → plan → implement workflow
Big changes get a written spec and plan **before** implementation, stored in-repo:
- Spec: `docs/superpowers/specs/YYYY-MM-DD-slug.md`
- Plan: `docs/superpowers/plans/YYYY-MM-DD-slug-plan.md` (same slug + `-plan`)
**When required** (any one triggers it): a schema change coupled to frontend work; any accounting-logic change (§1); any change touching ≥3 files across both `api/` and `web/`; any redesign of an existing flow's semantics. Precedent trail: spec `e86df27` → plan `196bef0` → implementation commits `07ad335`…`062ba52`.
**Spec format** (mirror `2026-07-01-category-types-receivable-redesign.md`): title `# Design: …`, `Date:` line, `## Overview` with numbered goals, one `##` section per goal covering data model, migration notes (including `server_default` decisions and what the migration must NOT touch), backend schema/model deltas, and UI behavior.
**Plan format** (mirror `2026-07-01-category-types-receivable-redesign-plan.md`): goal/architecture/tech-stack header; a `## Global Constraints` section that *restates the relevant invariants from §3*; a `## File Map` table (file → change); then `## Task N` sections with `- [ ]` checkboxes, each listing files, interfaces produced, and verification steps.
Specs and plans are **immutable once implemented** — they are the historical record; corrections go in a new dated spec.
## 6. Docs of record — what lives where
| Document | Role | Must be updated when |
|---|---|---|
| `CLAUDE.md` | Agent rulebook: conventions, invariants, structure | A new invariant is learned (add to "Key invariants" with rationale); a convention changes; a top-level directory is added (e.g. `cli/` in `082b608`) |
| `README.md` | Human quickstart + config reference | New env var → row in the "Environment / Config" table; new user-facing feature → Features list |
| `PLAN.md` | Original design doc — **frozen historical reference** (only ever touched at repo birth, `a9de3c8`) | Never retrofit it. Where reality diverged, CLAUDE.md wins. |
| `docs/superpowers/specs/`, `plans/` | Per-change design docs, dated, immutable | Every change that meets the §5 threshold |
| `.claude/skills/fin/SKILL.md` | Agent-facing `fin` CLI usage | CLI commands or flags change |
| `.claude/skills/fin-*/SKILL.md` | Process/knowledge skills (this library) | The process itself changes |
**Commit message house style** (mined from `git log`, as of 2026-07-05 — 41 `fix:`, 33 `feat:`, 5 `chore:`, 2 `docs:`, plus stray `perf:`/`polish:`):
- Subject: `type: imperative summary`, lower-case after the prefix, no trailing period; em-dash sub-clauses for multi-part fixes (`fix: bug hunt findings — currency derivation, local dates, …`).
- Body for anything non-trivial: bulleted, grouped under `Backend:` / `Frontend:` (see `e36d719`, `bc256f6`); each bullet names the file/function and the user-visible effect.
- Migration-bearing commits mention the revision id in the body (`Migration h8c9d0e1f2a3 retypes historical rows the same way`).
- Agent-authored commits carry `Co-Authored-By:` and `Claude-Session:` trailers (observed on `082b608`, `e36d719`).
- Doc commits: `docs: spec for …` then `docs: implementation plan for …` as separate commits before the feature commits.
## 7. Done-checklist — run before declaring any change complete
Copy-paste block (verified statically; this sandbox has no pip/docker — expect them to work in a normal dev env with `pip install -e cli/` + api requirements installed):
```bash
# 1. Syntax check (stdlib only, no deps needed)
python -m compileall -q api/app cli/fin_cli
# 2. E2E suite: FinClient → real app via TestClient → throwaway SQLite
python -m pytest cli/tests -q
# 3. Frontend type-check + build (package.json: "build": "tsc && vite build")
cd web && npm run build && cd ..
# 4. If a migration exists: dry-run on a scratch DB
cd api && DATABASE_URL=sqlite:////tmp/mig-check.db alembic upgrade head && cd ..
```
Then the manual gate:
- [ ] Change classified per §1 and its full gate satisfied
- [ ] No single-leg transfer/settlement create/update/delete anywhere in the diff (§3.1)
- [ ] Any `stock_transactions` mutation recomputes holdings before commit (§3.2)
- [ ] No new balance query summing `amount` without `- fee` (§3.5)
- [ ] Spending/income queries still exclude `transfer`/`settlement` rows; splitwise net-share math untouched or spec'd (§3.6)
- [ ] No `1.0`/wrong-currency FX fallback introduced (§3.7)
- [ ] No `async` session, no raw DDL, no `fetch()` outside `web/src/api/client.ts`
- [ ] Migration reviewed by hand: backward-safe (nullable/`server_default`), correct `down_revision` = current head, honest `downgrade()` (§4)
- [ ] New behavior has a test in `cli/tests/` (the API test-coverage gap is the project's known weak point — never widen it; see `fin-test-coverage-campaign`)
- [ ] CLAUDE.md updated if an invariant/convention changed; README env table updated for new config vars; spec/plan written if §5 applies
- [ ] If deploy-affecting: prod DB backed up (§2.3) before the image restarts and auto-migrates (§2.4)
- [ ] Commit message follows §6 style
## Provenance and maintenance
All claims verified against the repo on **2026-07-05** (commit `082b608` at HEAD). Commands were verified statically where noted (no pip/venv/docker in the authoring sandbox). One-line re-verification for anything that may drift:
```bash
git -C /path/to/fin log --oneline -5 # HEAD moved? re-check volatile facts
ls api/alembic/versions/ && grep -h "down_revision" api/alembic/versions/*.py # §4.2 chain + head
grep -n "create_all" api/app/main.py # §4.3 gotcha still at lifespan
grep -n "CMD" api/Dockerfile # §2.4 auto-migrate on start
grep -n "useradd" api/Dockerfile # §3.4 non-root user
grep -n "sqlite_data\|DATABASE_URL" docker-compose.yml # §2 volume + URL
grep -n "add_all" api/app/crud.py # §3.1 atomic legs
grep -n "def delete_stock_transaction" -A 25 api/app/crud.py | grep recompute # §3.2
grep -rn "journal_mode=WAL" api/app # §3.3 both engines
grep -n "Transaction.amount - " api/app/services/net_worth.py # §3.5 fee formula
grep -n "def get_monthly_spending" api/app/crud.py # §3.6
grep -n '"build"' web/package.json # §7 build command
git -C /path/to/fin log --format='%s' | grep -oE '^[a-z]+:' | sort | uniq -c # §6 prefix stats
ls docs/superpowers/specs docs/superpowers/plans # §5 naming convention
git show a06f921 a1f109b bc256f6 e36d719 1480088 ffa28bb 6b7d19b 81091fc --stat # incident evidence
```
If any command's output contradicts this file, update this file — CLAUDE.md and the repo are the ground truth, this skill is the map.
@@ -0,0 +1,232 @@
---
name: fin-debugging-playbook
description: Symptom-to-cause triage playbook for the fin app. Load when anything looks wrong and the cause is unknown — balances look wrong, spending or income totals too high/inflated, net worth off or in the wrong currency, prices stale or missing, holdings don't match trades, dates off by one day, 401s or frontend can't reach the API, container won't start or is unhealthy — or for any bug triage/debugging session in fin.
---
# fin Debugging Playbook
Symptom → likely cause → discriminating check, for this codebase's *actual* historical failure modes. Every claim below is grounded in code or a fix commit (SHAs cited). Facts date-stamped 2026-07-05; re-verify per "Provenance" at the bottom.
**Jargon (defined once):**
- **Leg**: one of the two (or three) `transactions` rows written together for a transfer/settlement/splitwise-pay/split-expense. Linked via `linked_transaction_id` (both directions) and `linked_account_id`.
- **Orphan leg**: a leg whose partner row was deleted — corrupts both account balances.
- **price_cache**: one row per symbol (tickers + `EUR/USD`, `EUR/GBP`), upserted; `fetched_at` is UTC.
- **Recompute**: `recompute_holding_from_stock_transactions` in `api/app/crud.py` — holdings are derived state, rebuilt from the full `stock_transactions` history for that account+ticker.
## When NOT to use this skill
- Recording/querying finances via CLI → sibling `fin` skill (`.claude/skills/fin/SKILL.md`).
- Making a code change safely → `fin-change-control`; architecture questions → `fin-architecture-contract`.
- Accounting semantics (what *should* the numbers be) → `fin-accounting-reference`; ad-hoc ledger SQL → `fin-ledger-analysis-toolkit`.
- Build/deploy/env setup → `fin-build-and-env`; day-2 ops (not active bug hunt) → `fin-run-and-operate`.
- Deep history of past incidents → `fin-failure-archaeology`; tooling setup → `fin-diagnostics-and-tooling`.
- Writing regression tests after the fix → `fin-test-coverage-campaign`, `fin-validation-and-qa`.
## Rule zero: reproduce on a copy — never debug against prod SQLite
The production DB (Docker volume `sqlite_data`, file `/data/fin.db` inside the `api` container — see `docker-compose.yml`) is sacred. Copy it out first; all SQL below runs against the copy. The DB runs in WAL mode, so a naive file copy can miss un-checkpointed writes in `fin.db-wal` — use `VACUUM INTO`, which produces a consistent single-file snapshot. Verified statically against `docker-compose.yml` and `api/Dockerfile` (python:3.12-slim has stdlib `sqlite3`; `/data` is writable by user `app`); run on the homelab host:
```bash
# 1. Consistent snapshot inside the container (read-only w.r.t. live data)
docker compose exec api python -c "import sqlite3; sqlite3.connect('/data/fin.db').execute(\"VACUUM INTO '/data/fin-debug-copy.db'\")"
# 2. Copy it out and clean up
docker compose cp api:/data/fin-debug-copy.db ./fin-debug-copy.db
docker compose exec api rm /data/fin-debug-copy.db
# 3. Query the copy locally
sqlite3 ./fin-debug-copy.db "SELECT count(*) FROM transactions;"
# 4. Optionally run a whole local API against the copy (from api/):
DATABASE_URL=sqlite:///$PWD/../fin-debug-copy.db uvicorn app.main:app --port 8001
```
Caveat for step 4: the app starts APScheduler and a 5-min price job that *writes* — fine against the copy, never point it at the real file. Destructive experiments (deletes, migrations) go against the copy only.
## First 15 minutes: "something is wrong and I don't know what"
Run in order; stop when a step discriminates.
1. **Is the API up?** `curl -fsS http://<host>/api/health` — always unauthenticated (`api/app/main.py`). If this fails, jump to the *Container* family.
2. **Container state + logs** (on homelab host): `docker compose ps` then `docker compose logs --tail 100 api`. Look for: alembic errors at start, `No EUR/... rate in price cache` warnings, yfinance retry storms, `database is locked`.
3. **Price freshness** (via API, no DB needed): `fin prices fx --json` and check `fetched_at` in the copy:
`sqlite3 fin-debug-copy.db "SELECT symbol, price, currency, fetched_at FROM price_cache ORDER BY fetched_at;"` — anything older than ~10 min means the 5-min scheduler job is not running (times are UTC).
4. **Orphan legs** (corrupt balances):
```sql
SELECT t.id, t.date, t.type, t.amount, t.account_id FROM transactions t
WHERE t.linked_transaction_id IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM transactions p WHERE p.id = t.linked_transaction_id);
```
Also count legacy unpaired legs: `SELECT count(*) FROM transactions WHERE type IN ('transfer','settlement') AND linked_transaction_id IS NULL;`
5. **What changed recently?** `git log --oneline -10` and `git diff HEAD~3 -- api/app/crud.py api/app/services/` — most fin bugs shipped in the last few commits.
6. **Cross-check a number two ways**: `fin networth --json | jq '.total_eur'` vs summing the copy yourself (see Balance family) — tells you whether data or calculation is wrong.
## Master symptom table
| Symptom | Likely cause (most→least) | Discriminating check |
|---|---|---|
| Spending/income totals inflated | Internal cash flow typed `expense`/`income` (stock buy leg, receivable settle) — e36d719; opening-balance income; missing `splitwise_partner_share` | SQL S1/S2 below |
| Spending "off by partner's share" | Not a bug: `splitwise_pay` counts `amount + splitwise_partner_share`; I-Owe counts as spending by design (e61b254) | Read invariant 6 in CLAUDE.md before "fixing" |
| One account balance wrong | Orphan leg; `fee` ignored (balance = SUM(amount−fee)); `manual_override_balance` set; forgotten "Opening balance" row | SQL B1/B2/B3 below |
| Net worth total wrong / a currency ignored | Missing `EUR/<CCY>` row in price_cache → asset contributes **0** (6b7d19b); EUR/USD missing → rate falls back 1.0; override in wrong currency (37bc101) | `grep 'No EUR/' logs`; SQL N1 |
| Prices stale/missing | Frankfurter URL rot (582e39d); yfinance down (stale cache served); Euro ETF needs exchange suffix (bc8c8d5); scheduler dead | `fetched_at` query; P1 below |
| Transactions dated yesterday | UTC vs local date: `toISOString()` before ~1–2 AM CET (bc256f6, `todayLocalISO` in `web/src/lib/utils.ts`) | Grep check D1 |
| Holdings ≠ trade history | Recompute skipped/failed; vested-grant FK deadlock (bc256f6) | SQL H1 |
| 401s / frontend broken but API "works" | `FIN_API_TOKEN` set on API env — locks out React frontend (sends no Authorization header) | `curl /api/accounts` with vs without Bearer |
| Container won't start / unhealthy | Alembic migration failure in CMD; healthcheck failing; `web` blocked on `api` health; WAL lock from external reader | `docker compose logs api` first 30 lines |
## Family runbooks
### 1. Spending/income totals look inflated or wrong
Ground truth (`get_monthly_spending`, `api/app/crud.py` ~line 624):
- Spending = rows with `amount < 0` and type `expense`/`splitwise_receive` (raw amount) + `splitwise_pay` (counted as `amount + splitwise_partner_share`, the user's net share). `transfer` and `settlement` are excluded.
- Accounts of type `receivable` are excluded; **`payable` is included on purpose** (I-Owe is real consumption — e61b254 deliberately reverted part of a9f045f).
- Income = type `income`, excluding `payable`/`receivable` accounts and rows with description `Opening balance`.
Discriminating checks (against the copy):
```sql
-- S1: internal flows mistyped as expense/income (the e36d719 bug class).
-- Any hit = data predating migration h8c9d0e1f2a3 or a new regression.
SELECT id, date, type, amount, description FROM transactions
WHERE type IN ('expense','income') AND linked_account_id IS NOT NULL;
-- S2: splitwise_pay rows missing partner share (counted at full amount)
SELECT id, date, amount, splitwise_partner_share FROM transactions
WHERE type='splitwise_pay' AND amount < 0
AND (splitwise_partner_share IS NULL OR splitwise_partner_share = 0);
```
API cross-check: `fin spending month 2026-06 --json | jq '.total, .income_total'` and eyeball big categories with `fin tx list --month 2026-06 --json | jq '[.[] | select(.type=="expense")] | sort_by(.amount) | .[0:5]'`.
### 2. Balance looks wrong
Ground truth: for bank/splitwise/receivable/payable accounts, balance = `SUM(amount - fee)` over all its transactions (`calculate_net_worth` in `api/app/services/net_worth.py`). `manual_override_balance` on the account *silently replaces* the computed balance.
```sql
-- B1: recompute the balance yourself and compare with the UI
SELECT a.id, a.name, a.type, a.manual_override_balance,
ROUND(COALESCE(SUM(t.amount - t.fee),0), 2) AS computed
FROM accounts a LEFT JOIN transactions t ON t.account_id = a.id
WHERE a.is_active = 1 GROUP BY a.id;
-- B2: orphan legs touching a specific account (see First-15-min step 4 for global)
SELECT * FROM transactions t
WHERE t.account_id = :acct AND t.linked_transaction_id IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM transactions p WHERE p.id = t.linked_transaction_id);
-- B3: overrides and opening balances that explain "mystery" amounts
SELECT id, name, manual_override_balance FROM accounts WHERE manual_override_balance IS NOT NULL;
SELECT id, account_id, date, amount FROM transactions WHERE description = 'Opening balance';
```
Decision: B1 computed ≠ UI → override or FX (family 3). B1 computed ≠ your expectation → B2 orphans, then fee column (`SELECT id, amount, fee FROM transactions WHERE account_id=:acct AND fee != 0;`).
### 3. Net worth wrong / currency wrong
Ground truth (`api/app/services/net_worth.py`):
- `_to_eur` converts via `price_cache` row `EUR/<CCY>`. **If the rate row is missing, the asset contributes 0.0 to net worth** with a log warning `No EUR/<CCY> rate in price cache` (6b7d19b — deliberately no wrong-currency fallback).
- If `EUR/USD` itself is missing, the *reported rate* falls back to 1.0 with a warning — and the same 1.0 fallback (`net_worth._get_eur_usd_rate`) is fed into funded-buy conversions at write time (gap U8, fin-ledger-analysis-toolkit), so a cross-currency funded buy booked while the cache was cold may carry a wrong amount permanently.
- Investment/RSU `manual_override_balance` is converted assuming the account's declared currency (37bc101).
- LSE pence: fetcher converts `GBp`/`GBX` → price/100, currency `GBP` (`price_fetcher.py`).
```sql
-- N1: which FX rates exist, and how old
SELECT symbol, price, currency, fetched_at FROM price_cache WHERE symbol LIKE 'EUR/%';
-- N2: currencies in use that need a rate
SELECT DISTINCT currency FROM accounts WHERE is_active=1
UNION SELECT DISTINCT currency FROM price_cache WHERE symbol NOT LIKE 'EUR/%';
```
If an account/holding currency appears in N2 but has no `EUR/<CCY>` row in N1, that asset is being valued at €0. Check `docker compose logs api | grep 'No EUR/'`. Per-account drill-down: `fin networth --json | jq '.accounts[] | {name, currency, balance_native, balance_eur, is_override}'` — `balance_native` nonzero with `balance_eur` 0 is the smoking gun.
### 4. Prices stale or missing
Ground truth (`api/app/services/price_fetcher.py`, `scheduler.py`, as of 2026-07-05):
- FX from `https://api.frankfurter.dev/v1/latest?from=EUR&to=USD` (and `to=GBP`). The old `api.frankfurter.app` host was retired — 582e39d. If FX dies again, suspect URL rot first.
- Tickers via yfinance: tries bare ticker, then suffixes `.DE .L .AS .PA .MI .SW .F .VI` (bc8c8d5); price cached under the *original* symbol. 3 retries/candidate with 1/2/4 s sleeps.
- On total failure: last cached price is served; `0.0` only when no cache row exists at all.
- Scheduler (`0b5ba7c`): `price_refresh` interval job every 5 min + `daily_snapshot` cron at midnight, persisted in table `apscheduler_jobs`, running on a *private* engine.
- Tracked = holdings with shares > 0 plus tickers of pending RSU grants.
Checks:
```bash
# P1: is anything updating? (fetched_at is UTC; should be < ~10 min old)
sqlite3 fin-debug-copy.db "SELECT symbol, fetched_at FROM price_cache ORDER BY fetched_at DESC;"
docker compose logs --tail 200 api | grep -E "Price refresh|Failed to fetch|All candidates|retrying"
# P2: force one ticker end-to-end (writes to price_cache — prod-safe by design)
fin prices fetch VWCE --json
# P3: full refresh runs as a 202 background task (POST /api/prices/refresh)
fin prices refresh
```
Never call `fetch_all_prices` synchronously in a request handler — it blocks and sleeps (CLAUDE.md).
### 5. Dates off by one day
Story: `new Date().toISOString().split('T')[0]` returns the **UTC** date; in UTC+1/+2 anything entered before ~1–2 AM was booked to *yesterday*. Fixed by `todayLocalISO()` in `web/src/lib/utils.ts` (bc256f6). Backend stores plain dates; only frontend default-date derivation was affected.
```bash
# D1: any regressions? Should return only the explanatory comment in utils.ts.
grep -rn "toISOString" /home/markuseisl/Documents/git/fin/web/src
```
If a user reports one specific wrong-dated row, fix the data with `fin tx update <id> --date YYYY-MM-DD` (allowed — it's not a linked leg... verify first: updates on legs with `linked_transaction_id` are rejected by the API, a1289f6).
### 6. Holdings don't match trades
Ground truth (`recompute_holding_from_stock_transactions`, `api/app/crud.py`): replay all `stock_transactions` for account+ticker in date,id order; buy/rsu_vest add shares and cost (incl. fee); sell removes shares at average cost; oversell raises → 422 and rollback. Delete triggers full recompute *before* commit; deleting a vest transaction also reverts the linked RSU grant to pending (bc256f6 — previously a FK deadlock made vested grants undeletable).
```sql
-- H1: expected shares from the log vs the holdings row
SELECT st.account_id, st.ticker,
ROUND(SUM(CASE WHEN lower(st.type) IN ('buy','rsu_vest') THEN st.shares ELSE -st.shares END), 6) AS expected,
h.shares AS stored
FROM stock_transactions st
LEFT JOIN holdings h ON h.account_id = st.account_id AND h.ticker = st.ticker
GROUP BY st.account_id, st.ticker
HAVING ABS(COALESCE(stored,0) - expected) > 1e-6;
```
Mismatch → trigger a recompute by the supported path: delete + re-add any one stock transaction for that ticker via `fin stock rm` / `fin stock add` (on a copy first). Never edit `holdings` rows directly.
### 7. 401s / frontend can't reach API
Ground truth: `api/app/auth.py` + `main.py`. Auth is **opt-in**: enforced only when env `FIN_API_TOKEN` is set on the API container, checked per-request (rotatable without restart... of code; the env var itself needs a container recreate). `/api/health` is always open; everything else, including `/api/openapi.json`, requires `Authorization: Bearer <token>`. The React frontend sends **no** Authorization header (`web/src/api/client.ts`) — setting the token locks the UI out until nginx injects the header. Nginx (`web/nginx.conf`) proxies `/api/` to `${API_UPSTREAM}` (default `api:8000`, set in `web/Dockerfile` and `docker-compose.yml`).
```bash
curl -fsS http://<host>/api/health # up at all?
curl -s -o /dev/null -w '%{http_code}\n' http://<host>/api/accounts # 401 → token enforced
curl -s -H "Authorization: Bearer $FIN_API_TOKEN" http://<host>/api/accounts | head -c 200
docker compose exec api printenv FIN_API_TOKEN # is it set on the container?
```
502 from nginx instead of 401 → `API_UPSTREAM` wrong or api container down.
### 8. Container won't start / unhealthy
Ground truth: `api/Dockerfile` CMD is `python -m app.preflight && alembic upgrade head && uvicorn ...` — a failing migration crash-loops the container *before* the app exists. Healthcheck (docker-compose.yml) curls `/api/health` every 30 s, 3 retries; `web` has `depends_on: api: condition: service_healthy`, so a sick api also keeps web down. Runs as non-root user `app` (uid 1000). Migration chain head as of 2026-07-05: `h8c9d0e1f2a3` (`api/alembic/versions/`).
```bash
docker compose ps
docker compose logs --tail 50 api # alembic tracebacks appear here first
docker compose exec api alembic current # applied revision vs files in api/alembic/versions/
docker compose exec api ls -la /data # fin.db + -wal + -shm owned by app (uid 1000)?
```
`database is locked` in logs → something outside the two engines is holding the file (e.g. a host `sqlite3` shell on the live DB — that's exactly why you debug on a copy). Both the request engine and the scheduler's private engine set `PRAGMA journal_mode=WAL` and `foreign_keys=ON` per connection (`database.py`, `scheduler.py`).
## Traps that cost real time
- **"Spending is too high" that wasn't a spending bug** (e36d719): stock buys' bank legs were typed `expense` and receivable/payable reconciles typed `income`/`expense`, inflating both metrics for weeks. The fix retyped them `transfer`/`settlement` and shipped migration `h8c9d0e1f2a3` to rewrite history. Lesson: when a total looks wrong, first ask *what types feed this aggregate* (run S1), not *is the arithmetic wrong*.
- **The payable flip-flop** (a9f045f → e61b254): payable accounts were first excluded from spending together with receivables, then deliberately re-included one commit later — money you owe and spent via I-Owe *is* consumption. Sonnet-class agents keep wanting to "fix" this. Don't; it's invariant 6 territory.
- **Orphan legs from single-leg deletes** (a06f921, a1f109b, 9d06e1b): deleting one transfer leg silently skewed two balances. Fixes added paired delete, then `linked_transaction_id` for precise pairing (the old account-pair heuristic could grab a leg of a *different* transfer between the same accounts), then frontend refetch-after-delete. Legacy rows without `linked_transaction_id` still exist — the fallback matcher in `delete_transaction` handles them.
- **Silent wrong-currency conversion** (6b7d19b): `_to_eur` used to fall back to the EUR/USD rate for *any* currency, quietly mis-valuing GBP assets. Now a missing rate excludes the asset (0 €) with a warning — so "asset disappeared from net worth" is usually a *missing FX row*, not lost data.
- **FX provider URL rot** (582e39d): frankfurter moved to `api.frankfurter.dev/v1`; until fixed, the app served an ever-staler cached rate with no user-visible error. Stale-graceful design hides upstream breakage — check `fetched_at`, not just whether numbers exist.
- **European ETFs invisible to yfinance** (bc8c8d5): bare ISIN-ish tickers (e.g. VWCE) return no data on Yahoo without an exchange suffix; the fetcher now walks `.DE .L .AS ...`. A ticker priced at 0 with "All candidates exhausted" in logs may simply trade under a suffix not in the list.
- **Midnight-minus-two hours dates** (bc256f6): every QuickAdd flow defaulted its date via `toISOString()`, so late-evening entries landed on the previous day for UTC+ users. Grep D1 whenever a new date input is added.
- **Undeletable vested grants** (bc256f6): deleting a vest's stock transaction hit the `vested_stock_transaction_id` FK (500), while deleting the grant was blocked (422) — a deadlock. `delete_stock_transaction` now reverts the grant to pending first.
- **Auth that "worked in curl"** (082b608): enabling `FIN_API_TOKEN` broke only the browser UI, because the frontend sends no Authorization header. Test both paths after touching auth.
## Provenance and maintenance
Verified against the working tree at commit 082b608 (2026-07-05). Docker/CLI commands verified statically (no docker/pip in authoring sandbox). Re-verify one-liners:
- Spending rules: `grep -n "notin_\|splitwise_partner_share\|Opening balance" api/app/crud.py`
- Balance formula & _to_eur: `grep -n "amount - \|return 0.0" api/app/services/net_worth.py`
- Frankfurter URL & suffixes: `grep -n "frankfurter\|_EXCHANGE_SUFFIXES" api/app/services/price_fetcher.py`
- Scheduler cadence: `grep -n "minutes=5\|hour=0" api/app/services/scheduler.py`
- Auth wiring: `grep -n "FIN_API_TOKEN" api/app/auth.py; grep -n "require_api_token" api/app/main.py`
- Container CMD/healthcheck/volume: `grep -n "CMD" api/Dockerfile; grep -n "healthcheck\|sqlite_data\|API_UPSTREAM" -A2 docker-compose.yml`
- Migration head: `ls api/alembic/versions/`
- Date helper: `grep -rn "todayLocalISO" web/src/lib/utils.ts`
- CLI commands: `grep -n "add_typer\|@app.command" cli/fin_cli/main.py`
@@ -0,0 +1,295 @@
---
name: fin-diagnostics-and-tooling
description: Read-only diagnostic scripts and API probes for the fin app. Load when you need to verify data integrity, quantify a suspected bug with real numbers, audit the ledger (linked transfer/settlement legs, holdings vs stock transactions, account balances), check price_cache or net-worth-snapshot freshness, independently recompute a month's spending, or gather before/after evidence around a change. The measurement layer — measure, don't eyeball.
---
# fin — diagnostics and tooling
Five tested, stdlib-only, READ-ONLY Python scripts plus API probes that turn
"the numbers look wrong" into a precise, diffable anomaly list. Every script:
- takes the SQLite DB path as `argv[1]` and opens it with
`sqlite3.connect("file:...?mode=ro", uri=True)` — it **cannot** write
- exits **0 = clean, 1 = anomalies found** (exception: `spending_recompute.py`
is a recompute tool, exit 0 on success)
- supports `--json` for machine-diffable output
- needs only `/usr/bin/python3` (3.11+), no pip, no venv, no docker
The prod DB is sacred. These scripts are safe to point at a live copy, but
prefer a copy anyway (below). Never run anything that writes against prod.
## Get a safe DB copy first
```bash
# from the homelab host running the stack (full runbook: fin-run-and-operate)
docker compose exec api python3 -c "import sqlite3; s=sqlite3.connect('file:/data/fin.db?mode=ro', uri=True); d=sqlite3.connect('/tmp/fin-copy.db'); s.backup(d); d.close()"
docker compose cp api:/tmp/fin-copy.db ./fin-copy.db
```
`sqlite3 .backup` snapshots WAL databases consistently. Note: a `mode=ro`
open of a *live* WAL DB can fail if the `-wal`/`-shm` files need recovery
and aren't writable — if that happens, use the backup copy, never loosen the
mode. Backup/restore discipline and volume paths: see **fin-run-and-operate**.
## Diagnostic decision table
| Question | Tool | Healthy looks like |
|---|---|---|
| Is the API up / authed? | `curl -s $FIN_API_URL/api/health` (always open) | `{"status":"ok"...}` |
| Is net worth plausible right now? | `fin networth --json \| jq '.total_eur, .eur_usd_rate'` | total matches expectation; rate ≈ market EUR/USD, not `1.0` |
| Does a transfer/settlement have both legs? | `scripts/audit_linked_legs.py` | exit 0 |
| Do holdings match the trade log? | `scripts/audit_holdings.py` | exit 0 |
| What is each account's computed balance? Any masking overrides? | `scripts/audit_balances.py` | exit 0; computed values match the UI |
| Are prices/FX fresh? Snapshot gaps? | `scripts/check_freshness.py` | exit 0; ticker ages are minutes (5-min scheduler job) |
| Is the spending total right for month X? | `scripts/spending_recompute.py` vs `fin spending month X --json` | identical totals per category |
| Prices stale in prod, why? | `check_freshness.py` on a copy + api logs (see fin-run-and-operate) | — |
| Is the cause a known bug pattern? | not this skill — **fin-debugging-playbook** / **fin-failure-archaeology** | — |
DB scripts answer "is the stored data consistent with the code's rules?".
API probes answer "what is the running system reporting?". A bug report needs
both: if the DB audits are clean but the API number is wrong, the bug is in a
service/router; if an audit fails, the data itself is damaged.
## Scripts
All paths below relative to this skill's `scripts/` directory. All five were
tested (as of 2026-07-05) against a synthetic SQLite DB built from the
`api/app/models.py` schema, seeded with healthy rows *and* one deliberately
broken case per check — each script flagged exactly the broken cases and
exited 0 on the clean DB (30/30 assertions passed).
### audit_linked_legs.py — two-leg integrity
```bash
python3 scripts/audit_linked_legs.py fin-copy.db [--allow-legacy] [--json]
```
Checks every transaction against the pairing rules in `crud.py`
(`create_transfer`, `create_settlement`, `create_split_expense`,
`create_splitwise_i_paid`, `create_funded_buy`, `delete_transaction`).
Interpretation:
- **CLEAN + counts line** — healthy. The counts tell you how many
same-currency pairs were amount-checked, how many cross-currency pairs and
funded-buy single legs were (correctly) skipped.
- **`orphan-link`** — `linked_transaction_id` points at a deleted row. One leg
of a pair was removed without the other: the surviving leg inflates/deflates
one account's balance. This is invariant #1 damage (see CLAUDE.md).
- **`self-link` / `self-account-link`** — a row references itself; corrupt
write, never produced by crud.py.
- **`amounts-dont-cancel`** — a *same-currency* transfer/settlement pair whose
amounts don't sum to 0 (tolerance 0.005; fees live in `fee` and are
excluded). One side was edited after creation. Cross-currency pairs are
exempt: `create_transfer` writes `to_amount` on the destination leg.
- **`non-reciprocal`** — A links to B but B doesn't link back. Deleting B via
the API would still find A (delete follows links both ways), but the pair is
malformed; expect an edit went through `PATCH /api/transactions`.
- **`unlinked-leg`** — transfer/settlement with NULL `linked_transaction_id`.
Rows written before that column existed are "legacy"; rerun with
`--allow-legacy` to accept legacy rows that have a crud-style fallback match
(reversed accounts + same date + same type). A leg that's still flagged with
`--allow-legacy` is a genuine orphan.
- **`missing-account` / `missing-linked-account`** — leg points at a deleted
account id (accounts are normally deactivated, never deleted).
- **Never flagged, by design**: funded-buy cash legs (single `transfer` linked
to an investment/rsu account — `create_funded_buy` writes no partner row);
`splitwise_pay` pair amounts (bank leg = −total, splitwise leg =
+partner_share; they must NOT cancel); the split-expense `expense` row's
one-way link into its transfer pair.
### audit_holdings.py — holdings vs stock_transactions
```bash
python3 scripts/audit_holdings.py fin-copy.db [--json]
```
Faithful reimplementation of `crud.recompute_holding_from_stock_transactions`
(date-ASC/id-ASC replay; buy/rsu_vest add `shares*price + fee` to cost; sell
removes at running average cost, clamped at 0; fractional shares supported),
diffed against the `holdings` table.
Interpretation:
- **`shares-mismatch` / `cost-basis-mismatch`** (tolerance 1e-6 shares / 0.005
cost) — the holding was mutated outside the recompute path, or a recompute
was skipped (invariant #2). Fix by re-running the recompute via the API
(delete+recreate a stock txn) — never by editing the holding row.
- **`missing-holding`** — stock transactions exist but no holding row; the
position is invisible to net worth.
- **`stale-holding`** — holding row with zero stock transactions; recompute
would delete it; it inflates net worth.
- **`invalid-history`** — a sell exceeds shares held at that point in the
replay. crud rejects this on write (422), so its presence means the history
was edited/imported around validation.
- **`unknown-stock-txn-type`** — type outside buy/sell/rsu_vest; crud silently
ignores such rows, so the holding may still "match" while the ledger carries
a dead row.
- **WARNING `currency-drift`** (not exit-affecting) — holding currency differs
from the first transaction's; crud never updates currency on existing
holdings, so this can drift legitimately after deleting the first txn.
### audit_balances.py — computed balance per account
```bash
python3 scripts/audit_balances.py fin-copy.db [--allow-overrides] [--json]
```
Prints `SUM(amount - fee)` per account (invariant #5 — never `SUM(amount)`),
with type, currency, txn count, and any `manual_override_balance`.
Interpretation:
- The table itself is the evidence: capture it before/after any change that
touches transactions.
- **`manual-override`** — the override *silently replaces* the computed value
in `services/net_worth.py`. The printed `delta` is how much net worth
differs from what the ledger says. Overrides may be intentional (e.g. a
hand-set Splitwise balance) — use `--allow-overrides` once confirmed, but
a large drifting delta means the ledger under the override has rotted.
- **`orphan-transaction`** — txn whose `account_id` no longer exists; it is
invisible to every balance and net worth calculation.
- Investment/rsu accounts show computed ≈ 0 with 0 txns — normal; their value
comes from holdings × price_cache, not transactions.
### check_freshness.py — prices, FX, snapshot gaps
```bash
python3 scripts/check_freshness.py fin-copy.db [--hours-stale 26] [--days 30] [--json]
```
Interpretation (thresholds as of 2026-07-05):
- **Age column** — the scheduler refreshes *every 5 minutes* plus a midnight
job, so healthy ages are minutes while the api container is up. Hours old =
container was down or yfinance/frankfurter kept failing (both fall back to
the stale cache silently — check api logs, see fin-run-and-operate). The
default 26 h threshold only catches hard outages; tighten with
`--hours-stale 1` when investigating.
- **`missing-fx` EUR/USD** — worst case: `calculate_net_worth` falls back to
rate **1.0** and produces confidently wrong totals.
- **`missing-fx` EUR/GBP** — flagged only when GBP data exists; then `_to_eur`
returns 0 and GBP amounts vanish from net worth (deliberate — see
fin-failure-archaeology before "fixing").
- **`missing-ticker-price`** — a tracked ticker (holding with shares > 0, or a
pending RSU grant's ticker) has no cache row: valued at €0. Fix:
`fin prices fetch TICKER` or `POST /api/prices/refresh`.
- **`zero-price`** — cached price ≤ 0; same €0 effect.
- **`missing-snapshot`** — gap in the daily net-worth series (window ends
today, container-local date). Today missing usually means the api container
was down at midnight. Gaps make History charts lie by interpolation; the
midnight job upserts, so a restart today backfills only today.
### spending_recompute.py — independent spending recompute
```bash
python3 scripts/spending_recompute.py fin-copy.db 2026-06 [--account N] [--json]
```
Line-by-line replication of `crud.get_monthly_spending`: expense +
`splitwise_receive` at raw amount; `splitwise_pay` at
`amount + splitwise_partner_share` (user's net share — intentional, do not
"fix"); receivable accounts excluded from spending; income excludes
receivable/payable accounts, `Opening balance` rows, **and rows with NULL
description** (SQL `<>` semantics — replicated deliberately). No
`is_active` filter — inactive accounts count, same as the API.
Interpretation — diff against the API:
```bash
python3 scripts/spending_recompute.py fin-copy.db 2026-06 --json | jq -S '{total, income_total, categories: [.categories[] | {category_id, total}] | sort_by(.category_id)}' > /tmp/recomputed.json
fin spending month 2026-06 --json | jq -S '{total, income_total, categories: [.categories[] | {category_id, total}] | sort_by(.category_id)}' > /tmp/api.json
diff /tmp/recomputed.json /tmp/api.json
```
- **Identical** — spending math is consistent between code and this spec; if
the user still disputes the number, the disagreement is about *semantics*
(load **fin-accounting-reference**), not a bug.
- **Differs** — either the DB copy is older than the live DB (recopy!), or
`get_monthly_spending` changed since 2026-07-05 (see Provenance), or you
found a real bug — capture both JSONs as evidence.
- Category *order* may differ (script sorts by total DESC; API preserves dict
order) — always diff sorted by `category_id` as above.
## API-level probes
Full CLI surface: the **fin** skill. Always `--json | jq`. Curl equivalents
(add `-H "Authorization: Bearer $FIN_API_TOKEN"` only if the API enforces it):
```bash
fin networth --json | jq '{total: .total_eur, rate: .eur_usd_rate}' # rate==1.0 => EUR/USD cache row missing
curl -s $FIN_API_URL/api/net-worth/current | jq '.total_eur'
fin networth history --range 1M --json | jq 'length' # ~30 => no snapshot gaps
curl -s "$FIN_API_URL/api/net-worth/history?range=1M" | jq 'length'
fin spending month 2026-06 --json | jq '{total, income_total}'
curl -s "$FIN_API_URL/api/spending/monthly?month=2026-06" | jq .
fin prices fx --json # cached EUR/USD
curl -s $FIN_API_URL/api/prices/fx-rate | jq .
curl -s -X POST $FIN_API_URL/api/prices/refresh # on-demand refresh (202)
fin accounts list --json | jq '[.[] | {id, name, type, balance}]' # cross-check vs audit_balances.py
```
Cross-check pattern: API balance ≠ `audit_balances.py` computed balance for
the same account → look for a `manual_override_balance` before suspecting
crud.
## Evidence workflow — before/after a change
The project discipline (see **fin-validation-and-qa** for the evidence
hierarchy): predict the numbers, then measure them.
```bash
EV=/tmp/evidence-$(date +%Y%m%d-%H%M%S); mkdir -p $EV
S=.claude/skills/fin-diagnostics-and-tooling/scripts
for a in audit_linked_legs audit_holdings audit_balances check_freshness; do
python3 $S/$a.py fin-copy.db --json > $EV/before-$a.json; echo "$a: exit $?"
done
python3 $S/spending_recompute.py fin-copy.db 2026-06 --json > $EV/before-spending.json
fin networth --json > $EV/before-networth.json
# ... apply the change, re-copy the DB ...
for a in audit_linked_legs audit_holdings audit_balances check_freshness; do
python3 $S/$a.py fin-copy.db --json > $EV/after-$a.json
done
python3 $S/spending_recompute.py fin-copy.db 2026-06 --json > $EV/after-spending.json
fin networth --json > $EV/after-networth.json
for f in $EV/before-*.json; do diff "$f" "${f/before-/after-}" && echo "unchanged: $f"; done
```
Rules: state *which* numbers the change should move and by how much **before**
running the after-pass; every diff line must be either predicted or explained;
audits that were clean before must be clean after. Attach `$EV` contents to
the change record (**fin-change-control**).
## When NOT to use this skill
- Recording transactions / querying finances for the user → **fin**
- What a formula *should* be (spending semantics, FX, RSU math) → **fin-accounting-reference**; proving ledger properties → **fin-ledger-analysis-toolkit**
- Symptom→cause triage when you don't yet know what to measure → **fin-debugging-playbook**; historical bug context → **fin-failure-archaeology**
- Backup/restore, env vars, container ops, prod logs → **fin-run-and-operate**
- Deciding what evidence a change needs / writing tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**
- Design/invariant questions before changing code → **fin-architecture-contract**, **fin-change-control**
## Provenance and maintenance
- Ground truth (read, not guessed): `api/app/models.py` (all tables/columns),
`api/app/crud.py` (pairing rules, holding recompute, spending math),
`api/app/services/price_fetcher.py` (EUR/USD + EUR/GBP + yfinance, tracked
tickers), `api/app/services/net_worth.py` (balance formula, fallbacks),
`api/app/services/scheduler.py` (5-min price job, midnight snapshot job).
Written against repo HEAD `082b608`, 2026-07-05.
- All five scripts tested 2026-07-05 on Python 3.13 against a synthetic DB
(schema derived from models.py; healthy rows + one seeded defect per check;
each script flagged exactly the seeded defects, exit codes verified).
- Schema-drift check — run before trusting the scripts; any output means the
ground truth moved and scripts/SKILL need re-verification:
```bash
git -C /home/markuseisl/Documents/git/fin diff --stat 082b608 -- api/app/models.py api/app/crud.py api/app/services/price_fetcher.py api/app/services/net_worth.py api/app/services/scheduler.py
```
- When drift is found: re-read the changed function, update the affected
script *and* its interpretation section, re-run the synthetic-DB test
(build a tiny DB with one broken case per check), and bump the commit hash
and dates in this section.
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""Compute per-account balances (SUM(amount - fee)) and flag masking overrides.
READ-ONLY. Usage: python3 audit_balances.py /path/to/fin.db [--allow-overrides] [--json]
Ground truth: api/app/services/net_worth.py — for bank/splitwise/receivable/
payable accounts, balance = SUM(amount - fee) over all transactions, UNLESS
manual_override_balance is set, in which case the override silently replaces
the computed value in net worth. Investment/rsu balances come from holdings x
price_cache, not from transactions (any computed value shown for them here is
just leftover cash legs and is not used by net worth).
Anomalies:
- manual_override_balance set (masks the computed balance; suppress with
--allow-overrides if the owner set it deliberately)
- transactions whose account_id points at a nonexistent account (invisible
to every balance and net-worth calculation)
Exit 0 = clean, 1 = anomalies found.
"""
import argparse
import json
import sqlite3
import sys
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("--allow-overrides", action="store_true",
help="do not count manual_override_balance as an anomaly")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
con.row_factory = sqlite3.Row
rows = con.execute(
"SELECT a.id, a.name, a.type, a.currency, a.is_active, "
" a.manual_override_balance AS override, "
" COUNT(t.id) AS txn_count, "
" COALESCE(SUM(t.amount - t.fee), 0.0) AS computed "
"FROM accounts a LEFT JOIN transactions t ON t.account_id = a.id "
"GROUP BY a.id ORDER BY a.type, a.id").fetchall()
orphans = con.execute(
"SELECT id, account_id, amount, date FROM transactions "
"WHERE account_id NOT IN (SELECT id FROM accounts)").fetchall()
anomalies: list[dict] = []
accounts = []
for r in rows:
d = dict(r)
d["computed"] = round(d["computed"], 2)
if d["override"] is not None:
d["override_delta"] = round(d["override"] - d["computed"], 2)
if not args.allow_overrides:
anomalies.append({
"check": "manual-override", "account_id": d["id"],
"detail": (f"{d['name']} ({d['type']}): override "
f"{d['override']} masks computed {d['computed']} "
f"(delta {d['override_delta']:+})")})
accounts.append(d)
for o in orphans:
anomalies.append({"check": "orphan-transaction", "account_id": o["account_id"],
"detail": (f"txn {o['id']} ({o['date']}, {o['amount']}) points "
f"at nonexistent account {o['account_id']}")})
if args.json:
print(json.dumps({"accounts": accounts, "anomalies": anomalies}, indent=2))
else:
print(f"{'id':>4} {'type':<12} {'active':<6} {'ccy':<4} "
f"{'txns':>5} {'computed':>12} {'override':>10} name")
for d in accounts:
ov = "" if d["override"] is None else f"{d['override']:.2f}"
print(f"{d['id']:>4} {d['type']:<12} {str(bool(d['is_active'])):<6} "
f"{d['currency']:<4} {d['txn_count']:>5} {d['computed']:>12.2f} "
f"{ov:>10} {d['name']}")
for a in anomalies:
print(f"ANOMALY [{a['check']}]: {a['detail']}")
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
return 1 if anomalies else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,127 @@
#!/usr/bin/env python3
"""Recompute every holding from stock_transactions and diff against holdings rows.
READ-ONLY. Usage: python3 audit_holdings.py /path/to/fin.db [--json]
Faithful replication of crud.recompute_holding_from_stock_transactions
(api/app/crud.py): iterate stock_transactions for each (account_id, ticker)
ordered by date ASC, id ASC;
buy / rsu_vest : shares += tx.shares; total_cost += tx.shares*price_per_share + fee
sell : error if tx.shares > shares held at that point;
total_cost -= tx.shares * (total_cost/shares); shares -= tx.shares;
total_cost = max(0, total_cost)
avg_cost_basis = total_cost/shares if shares > 0 else 0
currency = first transaction's currency
If no stock transactions remain, crud deletes the holding — so a holding row
with zero stock transactions is stale.
Tolerances: shares 1e-6, avg cost basis 0.005 (float accumulation).
Currency mismatch is a WARNING only (crud never updates currency on an
existing holding, so drift is possible without a bug).
Exit 0 = clean, 1 = anomalies found.
"""
import argparse
import json
import sqlite3
import sys
SHARES_EPS = 1e-6
COST_EPS = 0.005
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
con.row_factory = sqlite3.Row
txns_by_key: dict[tuple, list] = {}
for r in con.execute(
"SELECT id, account_id, ticker, type, shares, price_per_share, fee, "
"date, currency FROM stock_transactions ORDER BY date ASC, id ASC"):
txns_by_key.setdefault((r["account_id"], r["ticker"]), []).append(dict(r))
holdings = {(r["account_id"], r["ticker"]): dict(r) for r in con.execute(
"SELECT id, account_id, ticker, shares, avg_cost_basis_per_share, currency "
"FROM holdings")}
anomalies: list[dict] = []
warnings: list[str] = []
def flag(check, key, detail):
anomalies.append({"check": check, "account_id": key[0],
"ticker": key[1], "detail": detail})
for key, txns in txns_by_key.items():
shares = 0.0
total_cost = 0.0
currency = txns[0]["currency"]
invalid = None
for tx in txns:
t = (tx["type"] or "").lower()
if t in ("buy", "rsu_vest"):
shares += tx["shares"]
total_cost += tx["shares"] * tx["price_per_share"] + tx["fee"]
elif t == "sell":
if tx["shares"] > shares:
invalid = (f"stock_txn {tx['id']} sells {tx['shares']} but only "
f"{round(shares, 6)} held at that point in history")
break
if shares > 0:
total_cost -= tx["shares"] * (total_cost / shares)
shares -= tx["shares"]
total_cost = max(0.0, total_cost)
else:
flag("unknown-stock-txn-type", key,
f"stock_txn {tx['id']} has type {tx['type']!r} "
f"(crud silently ignores it)")
if invalid:
flag("invalid-history", key, invalid)
continue
avg = (total_cost / shares) if shares > 0 else 0.0
h = holdings.get(key)
if h is None:
flag("missing-holding", key,
f"{len(txns)} stock txns imply shares={round(shares, 6)}, "
f"avg={round(avg, 4)}, but no holdings row exists")
continue
if abs(h["shares"] - shares) > SHARES_EPS:
flag("shares-mismatch", key,
f"holding {h['id']}: stored {h['shares']}, recomputed {round(shares, 6)}")
if abs(h["avg_cost_basis_per_share"] - avg) > COST_EPS:
flag("cost-basis-mismatch", key,
f"holding {h['id']}: stored {h['avg_cost_basis_per_share']}, "
f"recomputed {round(avg, 6)}")
if h["currency"] != currency:
warnings.append(f"WARNING [currency-drift] account {key[0]} {key[1]}: "
f"holding says {h['currency']}, first txn says {currency}")
for key, h in holdings.items():
if key not in txns_by_key:
flag("stale-holding", key,
f"holding {h['id']} ({h['shares']} shares) has no stock "
f"transactions — recompute would delete it")
if args.json:
print(json.dumps({"holdings": len(holdings),
"keys_with_txns": len(txns_by_key),
"warnings": warnings, "anomalies": anomalies}, indent=2))
else:
print(f"audit_holdings: {len(holdings)} holdings, "
f"{len(txns_by_key)} (account,ticker) keys with stock txns")
for w in warnings:
print(w)
for a in anomalies:
print(f"ANOMALY [{a['check']}] account {a['account_id']} "
f"{a['ticker']}: {a['detail']}")
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
return 1 if anomalies else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Audit linked transaction legs (transfers, settlements, splitwise pairs).
READ-ONLY. Usage: python3 audit_linked_legs.py /path/to/fin.db [--allow-legacy] [--json]
Ground truth: api/app/crud.py (create_transfer, create_settlement,
create_split_expense, create_splitwise_i_paid, create_funded_buy,
delete_transaction legacy fallback) and api/app/models.py.
Rules encoded:
- Every 'transfer'/'settlement' row must have a partner via linked_transaction_id,
EXCEPT the funded-buy cash leg: a single 'transfer' row whose linked_account_id
points to an investment/rsu account (create_funded_buy writes no partner row).
- Rows written before linked_transaction_id existed are "legacy": NULL link but
linked_account_id set. delete_transaction pairs them by reversed accounts +
same date + same type. --allow-legacy accepts legacy rows that have such a match.
- linked_transaction_id must resolve, must not be self, and for
transfer/settlement/splitwise_pay the partner must link back (reciprocal).
The split-expense 'expense' row links one-way into its transfer group by
design and is exempt from reciprocity.
- Same-currency transfer/settlement pairs must cancel: amount_a + amount_b == 0
(fees live in the fee column and are excluded). Cross-currency transfer legs
legitimately differ (to_amount conversion) and are skipped.
- splitwise_pay pairs never cancel by design (bank leg = -total, splitwise leg
= +partner_share) and are never amount-checked.
Exit 0 = clean, 1 = anomalies found.
"""
import argparse
import json
import sqlite3
import sys
EPS = 0.005
PARTNER_REQUIRED = ("transfer", "settlement")
RECIPROCAL = ("transfer", "settlement", "splitwise_pay")
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("--allow-legacy", action="store_true",
help="accept legacy NULL-link legs that have a crud-style fallback match")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
con.row_factory = sqlite3.Row
accounts = {r["id"]: dict(r) for r in con.execute(
"SELECT id, name, type FROM accounts")}
txns = {r["id"]: dict(r) for r in con.execute(
"SELECT id, date, account_id, amount, currency, type, fee, "
"linked_account_id, linked_transaction_id, description "
"FROM transactions")}
anomalies: list[dict] = []
info = {"transactions": len(txns), "checked_pairs": 0,
"cross_currency_pairs": 0, "funded_buy_legs": 0, "legacy_matched": 0}
def flag(check, txn_id, detail):
anomalies.append({"check": check, "txn_id": txn_id, "detail": detail})
for t in txns.values():
tid, ttype = t["id"], t["type"]
if t["account_id"] not in accounts:
flag("missing-account", tid,
f"account_id={t['account_id']} does not exist")
la = t["linked_account_id"]
if la is not None and la not in accounts:
flag("missing-linked-account", tid,
f"linked_account_id={la} does not exist")
if la is not None and la == t["account_id"]:
flag("self-account-link", tid, "linked_account_id == account_id")
lid = t["linked_transaction_id"]
if lid is not None:
if lid == tid:
flag("self-link", tid, "linked_transaction_id == id")
continue
partner = txns.get(lid)
if partner is None:
flag("orphan-link", tid,
f"linked_transaction_id={lid} does not exist")
continue
if ttype in RECIPROCAL and partner["linked_transaction_id"] != tid:
flag("non-reciprocal", tid,
f"partner {lid} links to {partner['linked_transaction_id']!r}, not back")
if la is not None and partner["account_id"] != la:
flag("partner-account-mismatch", tid,
f"partner {lid} lives on account {partner['account_id']}, "
f"but linked_account_id={la}")
# amount cancellation — once per pair, same type, same currency only
if (ttype in ("transfer", "settlement") and partner["type"] == ttype
and tid < lid):
if t["currency"] == partner["currency"]:
info["checked_pairs"] += 1
resid = t["amount"] + partner["amount"]
if abs(resid) > EPS:
flag("amounts-dont-cancel", tid,
f"pair ({tid},{lid}): {t['amount']} + {partner['amount']} "
f"= {resid:+.2f} {t['currency']}")
else:
info["cross_currency_pairs"] += 1
elif ttype in PARTNER_REQUIRED:
la_type = accounts.get(la, {}).get("type") if la is not None else None
if ttype == "transfer" and la_type in ("investment", "rsu"):
info["funded_buy_legs"] += 1 # single-leg by design
continue
legacy_ok = False
if args.allow_legacy and la is not None:
# replicate delete_transaction's legacy fallback match
for o in txns.values():
if (o["id"] != tid
and o["account_id"] == la
and o["linked_account_id"] == t["account_id"]
and o["linked_transaction_id"] is None
and o["date"] == t["date"]
and o["type"] == ttype):
legacy_ok = True
break
if legacy_ok:
info["legacy_matched"] += 1
else:
flag("unlinked-leg", tid,
f"{ttype} with NULL linked_transaction_id "
f"(linked_account_id={la}, date={t['date']}, amount={t['amount']})")
if args.json:
print(json.dumps({"info": info, "anomalies": anomalies}, indent=2))
else:
print(f"audit_linked_legs: {info['transactions']} transactions, "
f"{info['checked_pairs']} same-currency pairs amount-checked, "
f"{info['cross_currency_pairs']} cross-currency pairs skipped, "
f"{info['funded_buy_legs']} funded-buy single legs, "
f"{info['legacy_matched']} legacy pairs accepted")
for a in anomalies:
print(f"ANOMALY [{a['check']}] txn {a['txn_id']}: {a['detail']}")
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
return 1 if anomalies else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Check price_cache freshness, required FX rows, and net_worth_snapshot gaps.
READ-ONLY. Usage:
python3 check_freshness.py /path/to/fin.db [--hours-stale 26] [--days 30] [--json]
Ground truth (as of 2026-07-05):
- api/app/services/price_fetcher.py fetches EUR/USD and EUR/GBP from
frankfurter.app plus one yfinance price per tracked ticker. Tracked =
holdings with shares > 0 UNION tickers of pending (unvested) rsu_grants.
- api/app/services/scheduler.py refreshes prices every 5 minutes and writes
one net_worth_snapshot per calendar day at midnight (date.today() of the
api container). fetched_at is datetime.utcnow().
- api/app/services/net_worth.py falls back to EUR/USD = 1.0 when the row is
missing (silently wrong totals) and values a ticker without a price row
at 0.0. Missing EUR/GBP excludes GBP amounts entirely (_to_eur returns 0).
Anomalies:
- price_cache row older than --hours-stale (default 26h; healthy is minutes
while the api container is up, thanks to the 5-minute job)
- price_cache row with price <= 0
- EUR/USD row missing (always required)
- EUR/GBP row missing while anything GBP exists (account, holding, or cached
price in GBP)
- tracked ticker with no price_cache row at all
- missing snapshot days in the last --days calendar days (window ends today,
local date)
Exit 0 = clean, 1 = anomalies found.
"""
import argparse
import json
import sqlite3
import sys
from datetime import date, datetime, timedelta, timezone
def parse_dt(s: str) -> datetime | None:
try:
return datetime.fromisoformat(s.replace("T", " "))
except (ValueError, TypeError):
return None
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("--hours-stale", type=float, default=26.0)
ap.add_argument("--days", type=int, default=30)
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
con.row_factory = sqlite3.Row
now = datetime.now(timezone.utc).replace(tzinfo=None) # fetched_at is utcnow()
anomalies: list[dict] = []
def flag(check, detail):
anomalies.append({"check": check, "detail": detail})
prices = {r["symbol"]: dict(r) for r in con.execute(
"SELECT symbol, price, currency, source, fetched_at FROM price_cache")}
report = []
for sym, p in sorted(prices.items()):
dt = parse_dt(p["fetched_at"])
age_h = round((now - dt).total_seconds() / 3600, 1) if dt else None
report.append({"symbol": sym, "price": p["price"], "currency": p["currency"],
"source": p["source"], "age_hours": age_h})
if age_h is None:
flag("unparseable-fetched-at", f"{sym}: fetched_at={p['fetched_at']!r}")
elif age_h > args.hours_stale:
flag("stale-price", f"{sym}: {age_h}h old (> {args.hours_stale}h)")
if p["price"] is None or p["price"] <= 0:
flag("zero-price", f"{sym}: cached price {p['price']} — net worth "
f"values it at 0")
# required FX rows
if "EUR/USD" not in prices:
flag("missing-fx", "EUR/USD not in price_cache — net worth silently "
"falls back to rate 1.0")
gbp_seen = con.execute(
"SELECT EXISTS(SELECT 1 FROM accounts WHERE currency='GBP') "
"OR EXISTS(SELECT 1 FROM holdings WHERE currency='GBP') "
"OR EXISTS(SELECT 1 FROM price_cache WHERE currency='GBP' "
" AND symbol NOT LIKE 'EUR/%')").fetchone()[0]
if gbp_seen and "EUR/GBP" not in prices:
flag("missing-fx", "EUR/GBP not in price_cache but GBP data exists — "
"GBP amounts are excluded from net worth (_to_eur -> 0)")
# tracked tickers must have a price row
tracked = [r[0] for r in con.execute(
"SELECT DISTINCT ticker FROM holdings WHERE shares > 0 "
"UNION SELECT DISTINCT ticker FROM rsu_grants WHERE vested_at IS NULL")]
for t in tracked:
if t not in prices:
flag("missing-ticker-price", f"{t} is tracked (holding or pending "
f"grant) but has no price_cache row")
# snapshot gaps
today = date.today()
start = today - timedelta(days=args.days - 1)
have = {r[0] for r in con.execute(
"SELECT snapshot_date FROM net_worth_snapshots WHERE snapshot_date >= ?",
(start.isoformat(),))}
missing = [(start + timedelta(days=i)).isoformat()
for i in range(args.days)
if (start + timedelta(days=i)).isoformat() not in have]
for m in missing:
flag("missing-snapshot", f"no net_worth_snapshot for {m}")
if args.json:
print(json.dumps({"prices": report, "tracked_tickers": tracked,
"snapshot_window": [start.isoformat(), today.isoformat()],
"missing_snapshot_days": missing,
"anomalies": anomalies}, indent=2))
else:
print(f"{'symbol':<12} {'price':>12} {'ccy':<4} {'source':<11} {'age_h':>7}")
for p in report:
print(f"{p['symbol']:<12} {p['price']:>12.4f} {p['currency']:<4} "
f"{p['source']:<11} {p['age_hours'] if p['age_hours'] is not None else '?':>7}")
print(f"tracked tickers: {', '.join(tracked) or '(none)'}")
print(f"snapshot window {start} .. {today}: "
f"{args.days - len(missing)}/{args.days} days present")
for a in anomalies:
print(f"ANOMALY [{a['check']}]: {a['detail']}")
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
return 1 if anomalies else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Independently recompute one month's spending per category from raw transactions.
READ-ONLY. Usage:
python3 spending_recompute.py /path/to/fin.db YYYY-MM [--account N] [--json]
Line-by-line replication of crud.get_monthly_spending (api/app/crud.py, as of
2026-07-05), so the output can be diffed against `fin spending month YYYY-MM --json`:
- Query 1: type IN ('expense','splitwise_receive'), amount < 0, account type
NOT IN ('receivable'), date in [month-start, next-month-start); SUM(amount)
grouped by category_id. NOTE: no is_active filter — inactive accounts count.
- Query 2: type = 'splitwise_pay', amount < 0, same account exclusion; each
row contributes amount + COALESCE(splitwise_partner_share, 0) — the user's
net share (e.g. -100 paid, partner owes 50 -> -50).
- Categories whose merged total >= 0 are dropped. total = sum of abs values.
- income_total: type = 'income', account type NOT IN ('payable','receivable'),
description <> 'Opening balance'. SQL NULL semantics apply: rows with NULL
description are ALSO excluded (matches SQLAlchemy's `!=`). Replicated as-is.
- percentage = round(total/overall*100, 1).
This is a recompute tool, not an audit: exit 0 on success, 2 on usage error.
Category order here is sorted by total DESC for readability; the API preserves
dict insertion order — diff by category, not by position.
"""
import argparse
import json
import sqlite3
import sys
from datetime import date
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("month", help="YYYY-MM")
ap.add_argument("--account", type=int, default=None)
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
try:
year, mon = int(args.month[:4]), int(args.month[5:7])
start = date(year, mon, 1).isoformat()
end = (date(year + 1, 1, 1) if mon == 12 else date(year, mon + 1, 1)).isoformat()
except ValueError:
print("month must be YYYY-MM", file=sys.stderr)
return 2
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
con.row_factory = sqlite3.Row
acct_filter = "" if args.account is None else " AND t.account_id = :acct"
params = {"start": start, "end": end, "acct": args.account}
# Query 1: expense + splitwise_receive at raw amount
merged: dict = {}
for r in con.execute(
"SELECT t.category_id AS cid, SUM(t.amount) AS total "
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
"WHERE t.date >= :start AND t.date < :end "
"AND t.type IN ('expense','splitwise_receive') AND t.amount < 0 "
"AND a.type NOT IN ('receivable')" + acct_filter +
" GROUP BY t.category_id", params):
merged[r["cid"]] = merged.get(r["cid"], 0.0) + r["total"]
# Query 2: splitwise_pay at user's net share
for r in con.execute(
"SELECT t.category_id AS cid, t.amount AS amount, "
"t.splitwise_partner_share AS share "
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
"WHERE t.date >= :start AND t.date < :end "
"AND t.type = 'splitwise_pay' AND t.amount < 0 "
"AND a.type NOT IN ('receivable')" + acct_filter, params):
user_share = r["amount"] + (r["share"] or 0.0)
merged[r["cid"]] = merged.get(r["cid"], 0.0) + user_share
cats = {r["id"]: dict(r) for r in con.execute(
"SELECT id, name, color, icon FROM categories")}
categories = []
overall = 0.0
for cid, total in merged.items():
if total >= 0:
continue # no net expense for this category
positive = abs(total)
overall += positive
c = cats.get(cid) if cid is not None else None
categories.append({
"category_id": cid,
"category_name": c["name"] if c else None,
"color": c["color"] if c else None,
"icon": c["icon"] if c else None,
"total": positive,
"percentage": 0.0,
})
for item in categories:
item["percentage"] = round(item["total"] / overall * 100, 1) if overall > 0 else 0.0
categories.sort(key=lambda x: x["total"], reverse=True)
income = con.execute(
"SELECT COALESCE(SUM(t.amount), 0.0) "
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
"WHERE t.date >= :start AND t.date < :end AND t.type = 'income' "
"AND a.type NOT IN ('payable','receivable') "
"AND t.description <> 'Opening balance'" + acct_filter, params).fetchone()[0]
result = {"month": args.month, "total": overall,
"income_total": income, "categories": categories}
if args.json:
print(json.dumps(result, indent=2))
else:
print(f"spending_recompute {args.month}"
+ (f" (account {args.account})" if args.account is not None else ""))
for c in categories:
print(f" {c['category_name'] or '(uncategorized)':<24} "
f"{c['total']:>10.2f} {c['percentage']:>5.1f}%")
print(f" {'TOTAL spending':<24} {overall:>10.2f}")
print(f" {'income_total':<24} {income:>10.2f}")
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,202 @@
---
name: fin-failure-archaeology
description: The chronicle of every significant bug, dead end, and design correction in fin's git history. Load BEFORE fixing anything that looks like a bug, whenever a fix idea feels obvious ("just add a fallback", "just count that as an expense", "just delete the row"), when investigating why code is written a strange way (why _to_eur returns 0, why delete follows links both ways, why RSU is one pot), or before re-opening any settled design question (splitwise net-share math, transfer restrictions, FX fallbacks, RSU lots). If a change would touch spending semantics, linked transaction legs, currency conversion, or the RSU model — read this first.
---
# fin — Failure Archaeology
Chronicle of every investigation, bug, dead end, and design retreat in this repo, mined from the full git history. Purpose: nobody re-fights a settled battle, and nobody "fixes" code back into a bug that was already fixed once.
**As of 2026-07-05.** History at time of writing: 107 commits, single `main` branch, **no reverts, no abandoned branches, no dead ends left in the tree** — every correction was made by a forward fix commit. The whole app was built in an intense burst: 2026-06-30 through 2026-07-03. That means fix commits are dense and each one is a real lesson, not noise.
Jargon used below:
- **Leg** — one of the two `Transaction` rows a transfer/settlement writes (debit + credit), linked via `linked_account_id` and (since a1f109b) `linked_transaction_id`.
- **Funded buy** — a stock buy that also debits a bank account for the cash.
- **Net-share splitwise math** — counting only the user's share (`amount + splitwise_partner_share`) as spending, not the full bank outflow.
- **RSU pot** — the aggregate of pending gross shares per RSU account (no per-grant schedule tracking).
## Settled questions — DO NOT re-open
These are the battles most likely to be re-fought by a well-meaning future engineer. Every one is intentional and evidence-backed.
| Settled question | Verdict | Evidence |
|---|---|---|
| `splitwise_pay` spending counts `amount + splitwise_partner_share` (user's net share), not the full bank outflow | INTENTIONAL — full amount double-counted the partner's half | 3e7f8ec |
| Stock-buy cash legs and receivable/payable settles are typed `transfer`/`settlement`, never `expense`/`income` | INTENTIONAL — asset movement is not consumption/earnings | e36d719 + migration h8c9d0e1f2a3 |
| No silent FX fallback in conversion paths; missing rate ⇒ skip/exclude with warning | REMOVED deliberately, twice. One residual 1.0 fallback survives in `net_worth._get_eur_usd_rate` (feeds funded buys) — a documented gap (U8, fin-ledger-analysis-toolkit), not precedent for adding more | 6b7d19b, 81091fc |
| Transfers restricted to cash-type accounts (`bank`, `splitwise`, `receivable`, `payable`) with same currency (same-currency later relaxed via explicit cross-currency support) | INTENTIONAL | 225e69e (restriction), b2a7c7a/9a49be2 (explicit cross-currency path) |
| RSU = one pot of pending gross shares per account, NOT per-grant vesting-schedule lots | DESIGN RETREAT — schedule generator was built (6385c06) and abandoned | c134801, c8dd938 |
| Vest net shares are entered manually from the brokerage statement (the UI retreat in c134801); the API retains `floor(gross × (1 − withholding))` as the fallback when no explicit net shares are given (`crud.vest_rsu_grant`) | INTENTIONAL — brokerage statement is the source of truth; the computed path is a preview/fallback, not the authority | c134801, 225e69e |
| Lending money ("I am owed") = bank→receivable **transfer**, not an income transaction | REDESIGNED — income version left cash untracked | e86df27 (spec), 16a19af (impl) |
| `payable`-account expenses COUNT in spending; only `receivable` is excluded | INTENTIONAL — a9f045f excluded both, e61b254 deliberately re-included payable (expenses you owe her for are real consumption) | a9f045f → e61b254 |
| Paired legs cannot be edited or deleted individually — delete cascades over the whole linked group, edit is blocked | INTENTIONAL | a06f921, a1f109b, a1289f6, e36d719 |
| Holdings are always recomputed from full stock-transaction history, not incrementally patched | UNIFIED deliberately after the two paths diverged | c558999 |
| Frankfurter API base URL is `https://api.frankfurter.dev/v1` (NOT `api.frankfurter.app`) | The old domain broke; do not "restore" it | 582e39d |
| `npm install -g npm@11` before `npm ci` in web Dockerfile | Works around a real Alpine npm exit-handler bug — not cruft | 89d241a |
| Category `type` (expense/income) is purely a UI picker concern; spending queries never filter transactions by category type | INTENTIONAL | e86df27 spec, 07ad335 |
| Dates use local-time `todayLocalISO()`, never `toISOString().split('T')[0]` | UTC version booked entries to yesterday before ~1–2 AM CET | bc256f6, 00409f0, 9d06e1b |
## When NOT to use this skill
- Operating/deploying the app → `fin-run-and-operate`, `fin-build-and-env`
- Using the `fin` CLI against the live API → `fin` skill
- Making a new change safely (process, migrations, review gates) → `fin-change-control`
- Live debugging of a current failure → `fin-debugging-playbook`, `fin-diagnostics-and-tooling`
- Understanding the current architecture (not its history) → `fin-architecture-contract`
- Accounting semantics reference (current rules, not how we got them) → `fin-accounting-reference`
- Verifying data / analyzing the ledger → `fin-ledger-analysis-toolkit`, `fin-validation-and-qa`
- Test writing → `fin-test-coverage-campaign`; open research → `fin-research-frontier`
Use THIS skill when the question is "why is it like this?", "is this a bug or a decision?", or "has this been tried before?".
---
## Chapter 1 — Ledger integrity and linked legs
The longest-running battle in the repo: keeping two-leg transactions atomic through create, delete, and edit. Fought in five escalating rounds.
| Round | Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|---|
| 1 | Transfer could half-fail: one leg written, one lost | Frontend made two sequential `createTransaction` calls | c67d34f — atomic `POST /transactions/transfer` and `/settlement` writing both legs in one DB transaction | Settled |
| 2 | Deleting one leg of a transfer orphaned the other and corrupted balances | `delete_transaction` deleted only the requested row | a06f921 — delete partner leg via account-pair query | Superseded by round 3 |
| 3 | Account-pair partner lookup was ambiguous: could delete a leg of a *different* transfer between the same two accounts | No direct link between legs | a1f109b — `linked_transaction_id` column (migration e5f6a7b8c9d0); all paired writers `flush()` then cross-link both legs before commit. Legacy account-pair fallback retained | Settled; fallback further constrained by date/type/closest-id in bc256f6 |
| 4 | Editing one leg desynced the pair | Edit endpoint didn't know about pairs | a1289f6 (backend blocks update on paired legs), 2bd4730 (frontend blocks edit UI on linked transactions) | Settled |
| 5 | Split expense = 3 rows (expense + 2 transfer legs); deleting one left the others | Expense row wasn't linked to the transfer legs | bc256f6 (atomic `POST /transactions/split-expense`), e36d719 (`create_split_expense` links all rows; `delete_transaction` follows `linked_transaction_id` in BOTH directions and deletes the full group) | Settled |
Supporting fixes: 9d06e1b (frontend refetches after delete so orphan legs never linger visually), 082b608 (`fin tx add` in the CLI *refuses* transfer/settlement types to prevent single-leg writes over HTTP).
**Do not redo:** never add a "delete just this row" path, never let any client write a transfer as two separate transaction POSTs, and never remove the flush-then-cross-link dance in the paired writers — the IDs don't exist until flush.
## Chapter 2 — Spending and income semantics
The definition of "what counts as spending" was corrected four separate times. Current state is the sum of all four lessons.
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| Splitwise "I paid" double-counted: full €100 bank outflow shown as user's expense even on a 50/50 split | Spending summed raw `amount` for `splitwise_pay` | 3e7f8ec — sum `amount + splitwise_partner_share` (user's net share); also added the offsetting +half to the splitwise account so net worth reflects the receivable | Settled — invariant #6 in CLAUDE.md |
| Receivable/payable bookkeeping rows polluted spending | Spending query didn't filter account types | a9f045f — exclude `payable` AND `receivable` account types | Partially reverted — see next row |
| Real expenses vanished from spending after a9f045f | Expenses recorded on `payable` accounts (IOweFlow: partner paid, user owes their share) ARE real consumption | e61b254 — re-include `payable`; only `receivable` stays excluded | Settled. Do NOT "clean up" the asymmetry — it is the point |
| Stock buys inflated spending; collecting a receivable inflated income | Funded-buy bank leg was typed `expense`; reconcile legs were typed `income`/`expense` | e36d719 — retyped to `transfer`/`settlement` (which spending queries already exclude); migration h8c9d0e1f2a3 retypes historical rows, matching funded-buy rows by the auto-generated `Buy <n> <TICKER>` description regex so user expenses like "Buy groceries" are untouched | Settled |
| Lending money showed up as income | IAmOwedFlow wrote a single `income` transaction on the receivable and never touched the bank | e86df27 (spec) + 16a19af — redesigned as bank→receivable transfer; zero spending/income impact | Settled |
Also: 1e8e826 (original monthly-spending endpoint), 0a97ad0 (Earned/Spent/Net strip), b2a7c7a (income totals). Design docs for the pivots live in `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` and `docs/superpowers/plans/`.
**Do not redo:** if spending numbers look "too low", check the net-share math and the retyped internal flows before assuming a bug. If they look "too high", suspect a new flow writing `expense`-typed rows for asset movements — that exact bug shipped twice.
## Chapter 3 — Currency and FX
Theme: every implicit currency assumption ("it's probably USD", "EUR/USD is close enough", "1.0 is a safe default") turned out to be a bug.
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| GBP holdings valued as if USD | Only EUR/USD existed; ticker prices assumed USD | 61fc838 — store actual price currency in `price_cache`, add EUR/GBP rate | Settled |
| Missing FX rate silently mis-converted using a *different* currency's rate | `_to_eur` fell back to EUR/USD for any unknown currency | 6b7d19b — return 0.0 with a logged warning; the amount is *excluded* from net worth rather than wrong | Settled. A dropped account value + warning log = missing rate, not a math bug |
| Fresh DB cached a fake 1.0 FX rate forever | `fetch_eur_usd`/`fetch_eur_gbp` returned 1.0 when fetch failed and no cache existed | 81091fc — return `None`, skip the upsert entirely | Settled. Never reintroduce a numeric FX default |
| GBP/other cash balances converted with hardcoded EUR/USD division | `net_worth.py` special-cased USD only | a06f921 — route all non-EUR cash through `_to_eur()` | Settled |
| Manual override balances on investment/rsu accounts mis-converted the same way | Same hardcoded division survived in the override branch | 37bc101 — override branch also uses `_to_eur()` | Settled |
| EUR/GBP fetch crashed instead of degrading | Error handling only existed for EUR/USD | 9b78d0e — mirrored the try/cache-fallback pattern | Settled |
| Transfers/expenses hardcoded EUR regardless of account | QuickAdd flows had `currency: 'EUR'` literals | 424b926 (transfer uses source-account currency), 0496a58, bc256f6 (all splitwise/IOwe flows derive currency from selected account + mismatch guards) | Settled |
| Funded buy double-converted for USD bank accounts | Bank debit always converted to EUR | b3062b7 — debit uses funding account's actual currency | Settled |
| RSU vest transactions hardcoded USD | Literal `'USD'` in vest path | e36d719 — currency taken from `PriceCache` for the ticker | Settled |
| LSE prices 100× too big | yfinance reports LSE in pence as `GBp`/`GBX` | Handled in fetcher (visible in bc8c8d5 diff): divide by 100, store GBP | Settled — keep the pence branch |
Cross-currency transfers were then added *explicitly* (b2a7c7a backend, 9a49be2 UI with FX pre-fill) rather than by loosening the same-currency guard silently.
**Do not redo:** any patch that makes a missing rate "just work" (defaults, nearest-rate fallbacks) reintroduces the class of bug this chapter killed four times.
## Chapter 4 — Dates and time
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| Trades/vests/settles entered before ~1–2 AM local time were booked to *yesterday* | `new Date().toISOString().split('T')[0]` is UTC; user is CET/CEST | 00409f0, 9d06e1b (IOwe/IAmOwed), bc256f6 (swept Dashboard, Portfolio, Spending, all QuickAdd flows to a local-time `todayLocalISO()` helper) | Settled — watch for regression in any NEW flow that defaults a date field |
| Malformed `month` param → 500 | No validation on `YYYY-MM` parsing | a1b5763 — 400 on malformed month, validate custom range order | Settled |
| Date-range queries subtly wrong | String date comparisons | dde3ee8 — proper date-range query handling (with indexes) | Settled |
## Chapter 5 — RSU model evolution (a design retreat)
The one place where the repo *built the sophisticated thing first and retreated to the simple thing*. Read this before proposing per-grant vesting schedules — it existed and was removed on purpose, same day it was built.
Timeline (all 2026-06-30):
1. 747e239 — unvested RSU grants: add, vest with withholding, delete.
2. 6385c06 — **schedule generator**: cliff + quarterly tranches with preview table.
3. 79d8220, 43a11fe — a cascade of fixes the schedule UI required: zero-price guards, blocked vested-delete, section gating, date parsing, per-row submitting state, server-side validation.
4. a3487f9 — whole-share rounding, grant ticker price fetch, estimated values, configurable splitwise split.
5. 225e69e — floor (not round) RSU preview net shares.
6. c134801 — **retreat step 1**: schedule generator form replaced by a plain Add Grant form; vest takes *explicit net shares from the brokerage statement* instead of computing from a withholding % override. Rationale in commit: the brokerage is the source of truth for net shares.
7. c8dd938 — **retreat step 2**: per-grant row UI replaced by one aggregated pot per RSU account (total pending gross shares by ticker) with inline Add-to-Pot and Vest actions; per-grant rows demoted to a collapsible Manage view.
8. 51744a1 — post-vest guidance, per-holding error scoping.
9. c835118 — withholding % had been hardcoded to 0 in the pot form (bug introduced by the simplification); now a UI field defaulting to 51%.
10. babea72 — unvested value (after withholding) shown greyed-out on dashboard.
11. bc256f6 — the **undeletable vested-grant deadlock**: deleting a vested grant returned 422, deleting its vest stock-transaction hit an FK 500 — a perfect deadlock. Fix: `delete_stock_transaction` reverts the grant to `pending` when its vest transaction is deleted.
**Do not redo:** rebuilding tranche schedules, computed withholding, or per-grant lot accounting. The complexity was tried; the maintenance cost (rounds 3–5 above) is why it was removed. The 51% withholding default is the user's real marginal rate — not a placeholder.
## Chapter 6 — Holdings and stock transactions
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| Cost basis drifted between create and delete paths | Create used incremental weighted-average update; delete used full recompute — two divergent implementations | c558999 — create ALSO uses `recompute_holding_from_stock_transactions`; deleted `_update_holding_buy`/`_update_holding_sell` | Settled. Do not reintroduce incremental holding math |
| Over-sell silently clamped to zero shares | `sell_shares = min(tx.shares, shares)` hid data errors | c558999 — recompute raises `ValueError` (→ 422) if history over-sells at any point | Settled |
| Stock tx delete / funded-buy bugs, fee ignored in balances | Assorted v1 backend bugs | 1480088 — over-sell guard, `SUM(amount - fee)` balance formula (now invariant #5), schema hardening, indexes | Settled |
| Selling gave no way to receive proceeds | Funded path only supported buy | e36d719 — funded sell credits `shares × price − fee` to a bank account, typed `transfer` | Settled |
## Chapter 7 — Price fetching
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| All FX fetches suddenly failing | Frankfurter moved to `api.frankfurter.dev/v1`; old `api.frankfurter.app` URL dead | 582e39d | Settled — don't "fix" the URL back |
| European ETF tickers (no exchange suffix) returned no data | yfinance needs `.DE`, `.L`, `.AS`, … suffixes | bc8c8d5 — auto-discovery: try bare ticker, then suffix list ordered by European-retail likelihood; cache under the *original* symbol; no-data ≠ transient error (only transient errors retry) | Settled |
| Startup blocked / request handlers blocked on price fetches | Synchronous `fetch_all_prices` in startup and refresh endpoint (it sleeps 1s/2s/4s between retries) | ffa28bb — removed blocking startup fetch, made refresh async/queued | Settled — CLAUDE.md "What NOT to do" |
| Refresh button lied ("updated: 0") | Queued refresh returned a fake count | a06f921 — honest "Refresh queued" message; bc256f6 — count only real updates | Settled |
| Prices stale during the day | Only midnight scheduler | 0b5ba7c — 5-minute refresh interval | Settled |
## Chapter 8 — Build, packaging, environment
| Symptom | Root cause | Fix (SHA) | Status |
|---|---|---|---|
| `npm ci` crashed on node:20-alpine | npm exit-handler bug on Alpine | 89d241a — `npm install -g npm@11` before `npm ci`. Looks like cruft; is load-bearing | Settled |
| Non-reproducible web builds | `npm install` from bare package.json | bfe7590 — `npm ci` + committed lockfile | Settled |
| Builds failed outside corporate network | Lockfile pinned to Dynatrace's npm mirror | 3c2766e — re-resolved against the public registry | Settled |
| Container crashed cryptically on unwritable volume / bad DATABASE_URL | No preflight; env-var handling scattered | d6001dc (unlabeled "fixes" — see Thin history below) — `app/db_config.py` (URL parsing, sqlite dir creation, writability check), `app/preflight.py` run before alembic in the Dockerfile CMD, `DATABASE_URL` default baked into the image, alembic env fallback chain. Error message explicitly mentions "volume mount permissions or Kubernetes fsGroup" — this was a real deploy failure | Settled |
| Healthcheck failed although API was fine | `curl` doesn't exist in python:3.12-slim | 3e7f8ec — stdlib Python one-liner healthcheck | Settled |
| Migrations not applied on deploy | Nothing ran alembic | 3e7f8ec — `alembic upgrade head` in Dockerfile CMD before uvicorn | Settled |
| Alembic ignored DATABASE_URL | env.py read only alembic.ini | a8c94cc, extended by d6001dc | Settled |
| Deploys served stale frontend | index.html browser-cached | bc256f6 — nginx `Cache-Control: no-cache` on index.html | Settled |
| Container ran as root | v1 Dockerfile | ffa28bb — user `app` uid 1000 (invariant #4); bc256f6 adds `COPY --chown=app:app` | Settled |
## Chapter 9 — Input validation and UI correctness guards
Smaller but recurring: the frontend initially trusted inputs and hardcoded assumptions.
- 54d64e0 — `Literal` types for transaction/account type fields; readable 422 messages (garbage types previously reached the DB).
- 256c27f — reject negative shares/price in add-trade form; 0a8a43f — block splitwise submit when computed user share is 0; bc256f6/e36d719 — reject 0% split and `partner_share >= total` server-side (422).
- a3d9da7 — accept comma as decimal separator everywhere (European keyboards; "12,50" previously parsed as 12).
- 00409f0 + 37bc101 — post-feature review sweeps: userShare guards, `initialAccountId` plumb-through, dialog state reset on `onOpenChange`, stale preview cleared on mode switch. Pattern: every big feature commit was followed by a same-day review-fix commit.
- 424b926, 7794264, 9d06e1b — error handling on every page fetch; stale state cleared on mode switches.
- dde3ee8 — FK enforcement ON for SQLite, security headers, category delete safety.
## Thin or ambiguous areas of history
- **No reverts, no branches**: `git log --all` shows 107 commits, all on `main`. Nothing was ever reverted; every correction is a forward fix. There is no abandoned-experiment branch to consult — the RSU schedule generator (Chapter 5) is the only "built then removed" feature, and it lives in mainline history.
- **`9aae3e8` ("fix")** — inspected: removes the unused `hasReceivable` prop from Dashboard's `AccountCard`. Pure dead-code cleanup, 1 file, 4 lines. Its sibling 42308a1 finished the same cleanup. No behavior change.
- **`d6001dc` ("fixes")** — inspected: NOT trivial despite the message. It is the deployment-hardening commit described in Chapter 8 (db_config.py, preflight.py, Dockerfile/compose changes). The Kubernetes fsGroup error message and `API_UPSTREAM` addition (0b5ba7c, same day) indicate the app was being adapted for the homelab/k8s deploy and hit a volume-permission failure in the wild. The commit message records none of this — the diff is the only evidence.
- **Motivation gaps**: most one-line fix commits (e.g. 225e69e, 424b926) record *what* changed but not the incident that triggered it. Where this file states a symptom for those, it is inferred from the diff, and worded accordingly.
- **Pre-history**: `PLAN.md` (commit a9de3c8) is the original design. Notable deviations that happened silently: `payable` account type added later (e3c8291), `manual_override_balance` largely replaced by balance-adjustment transactions (bf76e1d), and the 6-type transaction enum gained no new types but two retypings of existing rows (h8c9d0e1f2a3).
## Provenance and maintenance
- Every SHA above was verified against this repo on **2026-07-05** by reading `git show <sha>` (commit message for all; full diff for the load-bearing ones: 6b7d19b, 81091fc, a1f109b, c558999, 225e69e, e61b254, e36d719/h8c9d0e1f2a3, bc8c8d5, d6001dc, 9aae3e8, 89d241a, bfe7590, 582e39d, a06f921, 37bc101, c835118). Design-pivot rationale from `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` and `docs/superpowers/plans/`.
- Head commit at time of writing: `082b608` (fin CLI + opt-in bearer auth).
- **To extend this chronicle**, find behavior-changing commits since the last audited SHA:
```bash
git log --oneline 082b608..HEAD
git log 082b608..HEAD --grep='fix' -i --format='%h %s'
# then, for each candidate:
git show <sha> # read message AND diff before adding an entry
```
Also re-check for unlabeled fixes (messages like "fix"/"fixes"/"polish") — inspect their diffs, don't trust the subject line:
```bash
git log 082b608..HEAD --format='%h %s' | grep -viE 'feat|docs|chore'
```
- Entry format: Symptom → Root cause → Fix (SHA) → Status (settled / partially mitigated / watch for regression) → Do-not-redo note. Add new settled design questions to the top table. Never add an entry without reading the diff; never invent a symptom — mark it "inferred from diff" if the commit message doesn't state one.
@@ -0,0 +1,269 @@
---
name: fin-ledger-analysis-toolkit
description: Proof methods for fin's ledger. Load whenever you must PROVE an accounting property rather than eyeball it — verifying that balances/spending/net-worth are correct, deriving what a number SHOULD be before measuring it, computing the effect of a proposed change on aggregates before implementing it, adjudicating whether observed data is wrong vs expected, auditing a code path for currency correctness, checking a claimed fix actually fixed an aggregate, or stating/classifying ledger invariants (conservation, derivation, definition) as enforced vs unenforced. Also load before accepting any hypothesis about a money bug.
---
# fin — Ledger Analysis Toolkit (proof methods)
House standard: **prove it, don't just eyeball it.** A number on a dashboard is a claim; this skill is the set of methods for deriving what the number *must* be from the transaction rows and the code, and for refusing conclusions that haven't survived an attempted refutation.
All facts below derived from code as of **2026-07-05** (HEAD = 082b608), with `file:function` citations. Worked examples use real commits via `git show`; where production amounts are not in the repo, numbers are **faithful synthetic reconstructions and labeled as such**.
Division of labor: `fin-accounting-reference` owns the *definitions and formulas* (balance formula, type-semantics table, two-leg mechanics). This skill owns the *proof methods built on them*. Don't re-derive definitions — cite that skill; do re-verify enforcement status, because that is this skill's subject.
## When NOT to use this skill
- You need a formula or type-semantics definition → `fin-accounting-reference`.
- Something is broken and you don't yet know where → `fin-debugging-playbook` first; come back here when you have a hypothesis to prove.
- You want the history of a bug or whether a design question is settled → `fin-failure-archaeology`.
- You need runnable audit scripts (the mechanized form of these methods) → `fin-diagnostics-and-tooling`.
- You're deciding what evidence a change needs before "done" → `fin-validation-and-qa` (its evidence hierarchy is the acceptance policy; this skill supplies the proof techniques it ranks).
- You're turning an invariant into a regression/property test → `fin-test-coverage-campaign`.
- Design/refactor risk questions ("what breaks if I change this?") → `fin-architecture-contract`; change gating → `fin-change-control`.
---
## Method 1 — Invariant derivation & classification
**Purpose.** State a ledger invariant precisely enough to test it, then classify it honestly: ENFORCED (code prevents violation — cite where), INTENDED-UNENFORCED (design assumes it, nothing stops a violation — these are exactly where corruption enters), or STATISTICAL (expected to hold most of the time, gaps are normal).
**Inputs.** `api/app/crud.py`, `api/app/routers/transactions.py`, `api/app/schemas.py`, `api/app/models.py`, `api/app/services/net_worth.py`.
**Procedure.**
1. Pick the quantity (a balance, a linked group, a holding) and write the invariant as an equation or a closed predicate over rows — no prose like "should be consistent". Three shapes exist in this codebase:
- **Conservation**: a linked group of rows sums to a known constant (usually 0 or the recognized expense).
- **Derivation**: a stored row is a pure function (fold) of other rows.
- **Definition**: an aggregate is *defined* as a query; it cannot be "wrong", only mis-consumed.
2. For each write path that can touch the rows (router → crud → raw POST), ask: what code would reject a violating write? If none, the invariant is unenforced no matter how much the happy-path code respects it.
3. Check *at-rest* enforcement separately from *at-creation* enforcement: `create_transfer` writing both legs atomically does not stop a later single-leg write from another path.
4. Record the classification with the citation. An invariant without a citation is a wish.
### Current invariant catalog (as of 2026-07-05)
**Conservation (linked groups; sums are over `amount` unless noted):**
| # | Invariant | Status |
|---|---|---|
| C1 | Same-currency transfer pair: `Σ amount = 0`; net-of-fee `Σ(amount−fee) = −fee` (fee sits on source leg only) | ENFORCED at creation (`crud.py:create_transfer` — `add_all` + single commit); **UNENFORCED at rest** (see U1) |
| C2 | Cross-currency transfer: `−amount` in from-ccy, `+to_amount` in to-ccy; conservation holds only at the implied rate `amount/to_amount` | INTENDED-UNENFORCED — `to_amount` is a free field, no rate sanity check (`schemas.py:TransferRequest`, router only checks `> 0`) |
| C3 | Settlement pair (`create_settlement`, `/reconcile`): `Σ amount = 0`, both fees 0 | ENFORCED at creation (`crud.py:create_settlement`, `routers/transactions.py:reconcile_accounts`); UNENFORCED at rest |
| C4 | `splitwise_pay` pair: bank `−total`, splitwise `+partner_share` ⇒ `Σ amount = −(total − partner_share)` = the user's recognized expense | ENFORCED at creation (`crud.py:create_splitwise_i_paid`; router rejects `partner_share ≥ total`) |
| C5 | Split-expense triple: `−user_share − partner_share + partner_share = −user_share` | ENFORCED at creation (`crud.py:create_split_expense`, atomic since bc256f6/e36d719) |
| C6 | Funded buy/sell (cross-table): bank `transfer` leg + stock transaction satisfy `bank_Δ ≈ −cash_native/rate`; the "counter-leg" is the StockTransaction, not a Transaction row | INTENDED-UNENFORCED at rest — the bank leg is a deliberately **one-row** transfer (`crud.py:create_funded_buy`); deleting either side alone is allowed and leaves the other orphaned |
**Derivation:**
| # | Invariant | Status |
|---|---|---|
| D1 | `Holding(account,ticker)` = fold over all its StockTransactions ordered by `(date, id)`: buys/vests add `shares` and `shares×price+fee` to cost; sells remove at average cost; oversell at any historical point is rejected | ENFORCED — every insert/delete recomputes (`crud.py:recompute_holding_from_stock_transactions`, called from `create_stock_transaction`, `delete_stock_transaction`, `create_funded_buy`, `vest_rsu_grant`); oversell → ValueError → 422 |
| D2 | Holding currency = currency of the **first** stock transaction; mixed-currency txns per ticker are not blocked | INTENDED-UNENFORCED (`recompute…`: `currency = txns[0].currency`) |
| D3 | `StockTransaction.total_cost` is informational — the fold uses `shares×price_per_share+fee`, never `total_cost` | Definition/UNENFORCED — stored `total_cost` can diverge from the computed cost with no error |
| D4 | Vested grant ↔ vest transaction: FK `rsu_grants.vested_stock_transaction_id`; deleting the vest txn reverts the grant to pending | ENFORCED (FK in `models.py:RSUGrant`; revert in `crud.py:delete_stock_transaction`, added in bc256f6) |
**Definition (aggregates — see fin-accounting-reference for full semantics):**
| # | Invariant | Where defined |
|---|---|---|
| B1 | balance ≡ `SUM(amount − fee)` per account, all types | `net_worth.py:calculate_net_worth` balance query |
| S1 | monthly spending ≡ raw `expense`/`splitwise_receive` (amount<0, non-receivable accounts) + `splitwise_pay` at `amount + splitwise_partner_share` | `crud.py:get_monthly_spending` |
| I1 | income ≡ `income` rows on non-payable/receivable accounts, excluding `description = 'Opening balance'` | `crud.py:get_monthly_spending` income query — string-match exclusion is fragile (a user-entered "Opening balance" income is silently excluded) |
| N1 | net worth ≡ Σ per-account EUR balances (cash: B1; investment/rsu: `shares × cached_price` → `_to_eur`); missing FX rate ⇒ contribute **0.0** + warning | `net_worth.py:calculate_net_worth`, `_to_eur` |
**Integrity at rest (all INTENDED-UNENFORCED — the gap list):**
- **U1 — single-leg transfer/settlement via raw POST.** `POST /api/transactions` (`routers/transactions.py:create_transaction`) does **zero** validation beyond the Pydantic schema, and `schemas.py:TransactionBase` accepts `type` = any of the six including `transfer`/`settlement`, with `linked_account_id` freely settable and no partner row required. Verified truthfully: yes, a raw POST can create an unlinked `transfer` row. The CLI refuses (`cli/fin_cli/tx.py:TX_TYPES = ["expense","income","splitwise_pay","splitwise_receive"]`) — but that is client-side courtesy, not enforcement. (Small mercy: `linked_transaction_id` is NOT in the schema, so raw POST can't forge group links.)
- **U2 — transaction currency vs account currency**: never checked anywhere; a USD row on a EUR account silently corrupts the native balance (balances are raw sums, no per-row FX).
- **U3 — sign conventions**: raw POST accepts a positive-amount `expense` (drops out of spending via `amount < 0` filter but still credits the balance) and a negative `fee` (inflates balance via `amount − fee`).
- **U4 — `linked_transaction_id` has no FK** (`models.py:Transaction` line 70 — plain `Integer`, not `ForeignKey`): dangling group links possible; `delete_transaction`'s graph walk silently skips missing partners.
- **U5 — edit block covers only `linked_account_id`** (`routers/transactions.py:update_transaction`): the split-expense *expense* row carries only `linked_transaction_id`, so PUT can edit its amount, desyncing it from `total − partner_share` of its transfer legs.
- **U6 — `PUT /api/accounts/{id}` can change `type`** with history attached (`crud.py:update_account` sets any field) — e.g. bank→investment makes its transactions invisible to net worth (investment balance comes from holdings).
- **U7 — `splitwise_partner_share` unvalidated on raw POST**: can exceed `|amount|`, making the "user share" positive; `get_monthly_spending` then silently drops the category (`if total_amount >= 0: continue`).
- **U8 — funded-buy FX fallback**: `routers/stock_transactions.py:create_funded_buy` gets its rate from `net_worth._get_eur_usd_rate`, which **still falls back to 1.0** when EUR/USD is missing — the 6b7d19b fix removed the fallback only in `_to_eur`. A cross-currency funded buy with a cold price cache converts at 1.0.
**Statistical:**
- **ST1 — snapshot continuity**: one `net_worth_snapshots` row per calendar day, written by the midnight cron (`services/scheduler.py:fetch_prices_and_snapshot`, upsert on `snapshot_date = date.today()`). Gaps mean the container was down at midnight — expected, not corruption. Never treat a missing day as a data bug; treat a *discontinuous jump* between adjacent days as a lead (diff `breakdown_json` per account).
**Most dangerous unenforced invariant: U1.** A single-leg `transfer` shifts one account's balance and net worth with no counterparty, and — because spending/income exclude `transfer` by type — it is invisible to the two aggregates a human actually reviews. It is the one corruption mode that evades every dashboard.
### Worked example — e36d719 as a precise classification violation
Real commit (`git show e36d719`): before the fix, `crud.py:create_funded_buy` wrote the bank cash leg as `type="expense"`, `description=f"Buy {shares} {ticker}"`, no `linked_account_id`. State the violated invariant precisely: *internal asset conversions must contribute 0 to spending and 0 to income* (they change asset composition, not net worth or consumption).
Faithful reconstruction (repo has no prod amounts): buy 10 VUAA at $100, fee $2 ⇒ `total_cost = 1002` USD; funding account EUR, `eur_usd_rate = 1.08`, no override ⇒ bank leg `amount = −1002/1.08 = −927.78 EUR`, type `expense`.
- Spending (S1): row matches `type in ('expense', …) AND amount < 0` ⇒ month total **inflated by 927.78** under category `None`. Violation: Δspending = +927.78, required 0.
- Balance (B1): −927.78 either way — **unchanged by the fix**, because B1 sums all types.
- Net worth (N1): unchanged (cash −927.78, holdings +~927.78 at cost).
The fix retyped the leg to `transfer` (excluded from S1 by type) and migration h8c9d0e1f2a3 retyped historical rows (regex `^Buy \d+(\.\d+)? [A-Za-z0-9.\-]{1,12}$` so "Buy groceries" is untouched). The signature of a *classification* bug: exactly one aggregate moves, balances and net worth stay fixed. If your fix for a classification bug changes a balance, you fixed the wrong thing.
**Failure modes of this method.** (1) Classifying by intent instead of code — "create_transfer is atomic" says nothing about at-rest enforcement; always check the raw POST path. (2) Missing a write path (migrations and the scheduler also write rows). (3) Catalog rot: enforcement status changes with every router/schema edit — re-verify citations before relying on a status, don't trust this table's date-stamped snapshot blindly.
---
## Method 2 — Delta analysis of a proposed change
**Purpose.** Compute the exact effect of a change on the three top-level aggregates — per-account balance (B1), monthly spending (S1/I1), net worth (N1) — **before** writing the code. The prediction is the acceptance test: after implementing, measured deltas must equal predicted deltas (fin-validation-and-qa's hypothesis-predicts-numbers discipline).
**Inputs.** The proposed diff (or migration), the aggregate queries in `crud.py:get_monthly_spending` and `net_worth.py:calculate_net_worth`, a small synthetic ledger.
**Procedure.**
1. Identify every query whose row-set or per-row contribution the change touches. Grep for the column/type you're changing across `crud.py` and `net_worth.py` — a type retype touches S1/I1 (type filters) but not B1 (no type filter): read the filters, don't assume.
2. Write each touched aggregate as an explicit formula over rows (copy the actual filter predicates).
3. Build a 5-ish-row synthetic ledger that exercises every predicate branch the change touches (in-filter, out-of-filter, boundary).
4. Hand-compute each aggregate before and after. Record the deltas, **including the aggregates predicted not to move** — those negative predictions are the cheapest bug detectors you own.
5. After implementation, reconcile measured vs predicted (Method 3).
### Worked example — migration h8c9d0e1f2a3 (real file, synthetic ledger)
`api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py` does two UPDATEs: (a) `income`/`expense` rows with `linked_account_id IS NOT NULL` → `settlement` (legacy reconcile legs); (b) `expense` rows whose description matches the funded-buy regex → `transfer`.
Synthetic ledger, month 2026-06 (labeled reconstruction; account 1 = bank EUR, account 3 = receivable):
| id | acct | type (before) | amount | description | linked_acct |
|---|---|---|---|---|---|
| t1 | 1 | expense | −50.00 | groceries | — |
| t2 | 1 | expense | −927.78 | Buy 10 VUAA | — |
| t3 | 1 | income | +3000.00 | salary | — |
| t4 | 1 | income | +200.00 | Received payment | 3 |
| t5 | 3 | expense | −200.00 | Received payment | 1 |
Before: spending = 50 + 927.78 = **977.78** (t5 excluded — receivable account, S1's `Account.type.notin_(["receivable"])`); income = 3000 + 200 = **3200**; bank balance = −50 − 927.78 + 3000 + 200 = **2222.22**.
Migration hits: t2 (regex match) → `transfer`; t4, t5 (linked + income/expense) → `settlement`. t1, t3 untouched (no link, no regex match — "Buy groceries" would also survive, that's what the regex is for).
After: spending = **50.00** (Δ = −927.78); income = **3000.00** (Δ = −200.00); bank balance = **2222.22** (Δ = 0); net worth Δ = 0. The spending/income drop is **by design** — the migration exists to remove exactly those contributions; the zero deltas on balance and net worth are the safety proof that it is a pure re-labeling. If you had predicted these five numbers before running the migration on a DB copy, one `SELECT` per aggregate confirms it.
**Failure modes of this method.** (1) Synthetic ledger too clean — always include a row designed to be *incorrectly* caught (the "Buy groceries" decoy) and one on an excluded account type. (2) Forgetting month boundaries — S1 is per-month; a date-shifting change moves spending *between* months with zero total-across-months delta. (3) Predicting from the schema instead of the query — e.g. assuming income excludes opening balances "somehow" instead of reading the literal `description != 'Opening balance'` filter. (4) Not writing predictions down before measuring; post-hoc "yes that looks right" is eyeballing with extra steps.
---
## Method 3 — Reconciliation proof
**Purpose.** Prove a quantity by computing it three independent ways and showing agreement. **Rule of the house: any claim that a fix corrected an aggregate needs a reconciliation triple** — (1) the API's answer, (2) a raw-SQL recompute from first principles, (3) a hand calculation on the specific rows. Two agreeing numbers that share a code path prove nothing (the API and the frontend agree by construction).
**Inputs.** A **read-only copy** of the DB (prod DB is sacred — copy it out per fin-run-and-operate; never point tests or ad-hoc writes at it, per fin-validation-and-qa), the running API or `fin --json`, `sqlite3`.
**Procedure.**
1. **API answer**: `curl -s http://<host>/api/net-worth/current | jq .` or `fin networth --json` / `fin spending --json` (CLI is a thin HTTP wrapper — same code path as the frontend, counts as ONE source).
2. **Raw-SQL recompute** — re-derive from the definition, not from the API's implementation. Balances:
```sql
SELECT account_id, ROUND(SUM(amount - fee), 2) AS bal
FROM transactions GROUP BY account_id;
```
Spending for a month (mirror S1's two parts *independently*):
```sql
SELECT ROUND(-SUM(t.amount), 2) FROM transactions t
JOIN accounts a ON a.id = t.account_id
WHERE t.date >= '2026-06-01' AND t.date < '2026-07-01'
AND t.type IN ('expense','splitwise_receive') AND t.amount < 0
AND a.type != 'receivable';
SELECT ROUND(-SUM(t.amount + COALESCE(t.splitwise_partner_share,0)), 2)
FROM transactions t JOIN accounts a ON a.id = t.account_id
WHERE t.date >= '2026-06-01' AND t.date < '2026-07-01'
AND t.type = 'splitwise_pay' AND t.amount < 0 AND a.type != 'receivable';
```
Holdings: fold `stock_transactions` per (account, ticker) ordered by `(date, id)` and compare to the `holdings` row (D1).
3. **Hand calculation**: for the rows your change/bug touches (usually < 10), compute the contribution manually. This is the only leg that catches "the query and the API are consistently wrong together".
4. Accept iff all three agree within **|Δ| < 0.01 per account/category** (money is `Float` everywhere — see Method 5; exact equality is not achievable and demanding it produces false alarms).
5. Mechanized form: `.claude/skills/fin-diagnostics-and-tooling/` hosts executable audit scripts that are instances of these recomputes — use them for the SQL leg when they exist; the hand-calculation leg can never be mechanized away.
**Worked micro-example** (synthetic, from Method 2's ledger): claim "after migration, June spending is 50.00". Triple: `fin spending --month 2026-06 --json` → `total: 50.0`; SQL part 1 → 50.00, part 2 → NULL (no splitwise_pay rows); hand: only t1 qualifies, |−50| = 50. Agreement across three derivations ⇒ proven, not eyeballed.
**Failure modes of this method.** (1) Fake independence — recomputing with the ORM query copy-pasted from `crud.py` is source (1) again; write the SQL from the *definition*. (2) Reconciling against a moving target — the 5-minute price-refresh job (`scheduler.py`) mutates `price_cache`, so net-worth triples must pin prices (work on the DB copy, or capture `price_cache` first). (3) Currency blindness: the SQL balance is *native*; the API answer is EUR — convert with the same cached rate (`SELECT price FROM price_cache WHERE symbol='EUR/USD'`) before comparing, or you'll "find" an 8% bug. (4) Proving the aggregate but not the rows — totals can agree while two row-level errors cancel; when adjudicating a specific bug, reconcile the affected rows, not just the total.
---
## Method 4 — Currency-boundary analysis
**Purpose.** Audit any money-touching code path for currency correctness: where is a currency attached, converted, assumed, or silently defaulted?
**The conventions (as of 2026-07-05, from code):**
- Every cached FX symbol is `EUR/X` meaning **X units per 1 EUR** (`price_fetcher.py:fetch_eur_usd` caches frankfurter's `from=EUR&to=USD` rate under `EUR/USD`). Therefore converting X→EUR is always a **division**: `_to_eur` does `amount_native / rate` (`net_worth.py:_to_eur`), and `create_funded_buy` does `cash_native / used_rate` (`crud.py`). Any multiplication by an `EUR/X` rate when going *to* EUR is a bug; multiplication is only correct going EUR→X.
- Only **EUR/USD and EUR/GBP** are fetched (`price_fetcher.py:fetch_all_prices`). Any holding priced in CHF/JPY/SEK/… contributes **0.0** to net worth with a warning (N1). That is by design — see the worked example.
- **GBp/pence handling**: yfinance reports LSE tickers in pence as `GBp`/`GBX`; `price_fetcher.py:_try_fetch_candidate` divides the price by 100 and stores currency `GBP`. If you ever see a UK holding ~100× too large, this normalization got bypassed.
- Ticker prices are cached in **native currency** with the currency stored on the row (`price_cache.currency`); `calculate_net_worth` converts using the *price row's* currency (`ccy_map`), while cash balances convert using the *account's* declared currency. Transactions carry their own `currency` but balances never per-row convert (U2).
**Checklist — run this against any diff that touches money:**
1. For every number, name its currency at every step (native → EUR is the only conversion direction in aggregates).
2. Every conversion: is it a division by an `EUR/X` rate? If it multiplies, prove the direction.
3. What happens when the rate is **missing**? Acceptable answers: exclude-with-warning (`_to_eur` → 0.0) or refuse (vest requires a cached price, `crud.py:vest_rsu_grant` → 422). Unacceptable: any silent numeric fallback. Known residual violation: **U8** — `_get_eur_usd_rate` still falls back to 1.0 and `routers/stock_transactions.py` feeds it to `create_funded_buy`.
4. Could a *wrong currency's* rate be used? (The pre-6b7d19b bug class.)
5. Pence/GBX: does the path consume `price_cache` (already normalized) or raw yfinance data (must normalize)?
6. Does a currency default lurk? (`ccy = (price_ccy or "USD")` in `_to_eur`; `Transaction.currency` defaults `"EUR"`, `StockTransaction.currency` defaults `"USD"` — a caller omitting currency gets these silently.)
### Worked example — commit 6b7d19b (real diff; amounts reconstructed)
`git show 6b7d19b`: before, `_to_eur` with a missing `EUR/{ccy}` rate fell back to `amount_native / EUR_USD_rate` — converting with a **different currency's** rate. Reconstruction: 100 shares NESN.SW at 90 CHF = 9,000 CHF; cache has `EUR/USD = 1.08` but no `EUR/CHF` (true today — CHF is never fetched). Fallback books `9000 / 1.08 = 8333.33` "EUR"; the correct value at EUR/CHF ≈ 0.93 is `9000 / 0.93 = 9677.42`. Silent error: **−1344.09 EUR (−13.9%)**, buried inside a plausible-looking total.
Why returning 0 is *better* than returning the unconverted-or-misconverted number: (a) 8333.33 is plausible — no one investigates plausible; a 9,677 hole in net worth is investigated the same day; (b) the midnight job **persists** net worth into `net_worth_snapshots` — a plausible wrong value corrupts the historical series permanently, while an obvious dip is a visible, dateable, explainable artifact; (c) the warning log names the missing rate, turning a silent data bug into an ops task ("fetch EUR/CHF"). This is the codebase's settled loud-zero-over-plausible-lie policy (fin-failure-archaeology: no-1.0-fallback, removed twice) — U8 is the one place it hasn't reached yet.
**Failure modes of this method.** (1) Auditing only the read path — currency bugs are often planted at *write* time (RSU vest hardcoded `"USD"` until e36d719 fixed it to read `price_cache.currency`). (2) Trusting variable names — `eur_usd_rate` in `create_funded_buy` is used for *any* account-vs-stock currency pair, not just EUR/USD; the name lies when the account is GBP. (3) Checking direction with round-trip algebra on symmetric test values (rate 1.0, amount 100) — always test with an asymmetric rate like 1.08 so a flipped division actually shows.
---
## Method 5 — Boundary/edge catalog for money math
**Purpose.** The edges this codebase has historically gotten wrong, each with the standard probe to run against new money code. Every entry is a real commit.
| Edge | Incident (real commit) | Standard probe |
|---|---|---|
| Zero share / 0% split | 0a8a43f — SplitwiseShePayed with 0% computed user share submitted a 0-amount row; blocked client-side. Backend: split endpoints require `partner_share > 0` and `< total` (`schemas.py` `gt=0`, router checks); raw POST still accepts `amount = 0` (U3-adjacent) | Feed 0 and the value that *computes* to 0 (e.g. total = 2×partner_share rounding to user share 0.00) |
| Negative inputs | 256c27f — Portfolio add-trade accepted negative shares/price (a "negative buy" is a disguised sell that corrupts avg cost). Backend now `gt=0` on `StockTransactionBase`; **`Transaction.amount`/`fee` remain unvalidated on raw POST** | Probe every numeric field with −1; decide reject-vs-meaning explicitly |
| Comma decimals | a3d9da7 — `parseFloat("12,50")` silently returns **12**; German-locale entry lost cents/magnitude. QuickAdd inputs now normalize commas | Enter "12,50" and "1.234,56" in any new amount input; assert 12.50, not 12 |
| UTC vs local midnight | bc256f6 — frontend default dates used `toISOString().split('T')[0]` (UTC); entries between 00:00 and ~01–02 CET booked to **yesterday**, silently moving spending across month boundaries. Fixed with `todayLocalISO()` (`web/src/lib/utils.ts`) | Test any date-defaulting path at simulated 00:30 local; grep new frontend code for `toISOString` |
| Float money | No single incident — a standing condition: **every money column is `Float`** (`models.py`: `amount`, `fee`, `shares`, `price_per_share`, `total_cost`, `total_eur`, …). There is no Decimal and no backend rounding policy; the only rounding in the stack is frontend cents-rounding of `partner_share` (bc256f6) and display rounding (`round(pct,1)` in `get_monthly_spending`). Implications: (a) reconciliation must use tolerance |Δ| < 0.01, never equality (Method 3); (b) `recompute…` uses a **strict** `if tx.shares > shares` compare — selling exactly-all after fractional accumulations (0.1+0.2 ≠ 0.3) can 422 a legitimate sell-all, or conversely permit a dust oversell within epsilon | Sell-all after ≥3 fractional buys; sum 0.1+0.2-style amounts through an aggregate and check the API's raw JSON for 0.30000000000000004-type values |
| RSU whole-share flooring | 225e69e — vest preview must **floor**, not round: `net = floor(gross × (1 − withholding))` (`crud.py:vest_rsu_grant`); schedule generator floors per tranche and pushes the remainder into the last (`crud.py:_generate_rsu_schedule`); `net_shares ≤ 0` → 422 | gross=10, pct=0.51 ⇒ 4 (floor of 4.9 — round would say 5); gross=1, pct=0.5 ⇒ 0 ⇒ must 422 |
**Failure modes of this catalog.** It is a list of *known* edges — passing all probes proves you didn't repeat history, not that no new edge exists. And probes at the frontend prove nothing about the API (comma handling, negative rejection are client-side in several places): probe the layer you changed.
---
## Method 6 — Adversarial self-refutation
**Purpose.** Before accepting any conclusion about ledger data ("the bug is X", "the data is actually correct", "the fix worked"), attack it yourself. Sonnet-and-owner failure mode this method exists to stop: the first mechanism that explains the headline observation gets accepted while a cheaper observation that contradicts it is never checked.
**Procedure.**
1. Write down **every** observation the hypothesis must explain — explicitly including the negatives: *which aggregates did NOT change? which rows were NOT affected? who is NOT seeing the problem?* The negative space discriminates between mechanisms far better than the headline symptom (Method 2's classification signature — "spending moved, balance didn't" — is exactly such a negative).
2. Construct at least **one alternative mechanism** that explains the headline observation. If you can't think of one, you haven't understood the system; check the layer above and below (frontend default vs backend default vs DB default all produce "wrong value in DB").
3. Find a **discriminating check**: an observation the two mechanisms predict differently. Run it.
4. Accept the hypothesis only when the alternative is *falsified by the check* — not when it merely feels less likely.
5. For fixes: the prediction from Method 2 plus the triple from Method 3 is the acceptance; "the number looks right now" is not.
### Worked example — the bc256f6 off-by-one-day bug (real incident; investigation reconstructed faithfully from the commit message and diff)
Headline observation: trades, vests, and settlements entered via the UI sometimes carry **yesterday's** date.
Full observation list, including negatives:
- O1: affected rows are dated one day early;
- O2 (negative): amounts, accounts, balances all correct — only the `date` column;
- O3: only entries made shortly after local midnight (before ~01–02 CET) are affected;
- O4 (negative): rows created with an explicitly typed date, and CLI/API-created rows, are correct;
- O5 (negative): the scheduler's snapshot dates are correct.
Hypothesis A (first instinct): the backend/server clock or timezone shifts dates. Hypothesis B: the **frontend's default date** is computed in UTC.
Discriminating checks: (i) A predicts O4 fails — the backend would shift *stored* dates regardless of who sent them; but explicitly-dated rows are stored verbatim (the `Date` column stores what it's given, `models.py:Transaction.date`), so A is falsified by O4. (ii) A predicts O5 fails too — the scheduler runs server-side (`scheduler.py` uses `date.today()`), and its snapshots were fine. (iii) B predicts the bug lives where defaults are computed client-side: `grep -rn "toISOString" web/src` found `toISOString().split('T')[0]` in Dashboard, Portfolio, Spending and the QuickAdd flows — JS `toISOString()` is UTC, so between local midnight and UTC midnight it yields yesterday. B explains O1–O5 with nothing left over; A is dead on two independent checks. The fix (`todayLocalISO()` helper, bc256f6) touched only the frontend — which is itself a post-hoc confirmation that B was the mechanism.
The refutation discipline is what made this cheap: checking O4 (one CLI insert with an explicit date) cost seconds and killed the expensive hypothesis (timezone archaeology on the container) before anyone spent an hour on it. Also note O3's precision — "before ~01–02 CET" is CET/CEST's UTC offset speaking; a hypothesis that can't derive the cutoff hour from its mechanism doesn't actually explain O3.
**Failure modes of this method.** (1) Strawman alternatives — the alternative must be one you'd genuinely believe if the check went the other way. (2) Confirmation-shaped checks — pick the observation the hypotheses *disagree* on, not another one they both predict. (3) Stopping at one bug: bc256f6 was a bug-*hunt* commit; after falsifying A for one symptom, re-run the loop per symptom — several independent mechanisms (currency hardcoding, non-atomic split expense, vest-unlink deadlock) were live simultaneously, and a single unified explanation for all of them would have been wrong.
---
## Provenance and maintenance
All claims verified against HEAD = 082b608, 2026-07-05. Re-verification one-liners (from repo root):
- Invariant catalog / U1: `sed -n '24,29p' api/app/routers/transactions.py` (create_transaction has no checks) and `grep -n "TX_TYPES" cli/fin_cli/tx.py` (CLI refusal).
- U4: `grep -n "linked_transaction_id" api/app/models.py` — still plain `Integer`?
- U5: `grep -n "linked_account_id is not None" api/app/routers/transactions.py` — edit block still ignores `linked_transaction_id`-only rows?
- U8: `grep -n "_get_eur_usd_rate" api/app/routers/stock_transactions.py api/app/services/net_worth.py` — 1.0 fallback still feeding funded buys?
- B1/S1/I1/N1 formulas: `grep -n "amount - models.Transaction.fee" api/app/services/net_worth.py`; `grep -n "splitwise_partner_share\|Opening balance" api/app/crud.py`.
- FX direction & GBp: `grep -n "amount_native / rate" api/app/services/net_worth.py`; `grep -n "GBp\|GBX" api/app/services/price_fetcher.py`; fetched rates: `grep -n "fetch_eur_" api/app/services/price_fetcher.py`.
- Float money: `grep -c "Float" api/app/models.py` (any Decimal yet?).
- Worked-example commits still resolve: `git show --stat e36d719 6b7d19b bc256f6 225e69e a3d9da7 0a8a43f 256c27f | grep ^commit`.
- Migration example: `sed -n '30,52p' api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py`.
If any check disagrees with this file, the code wins — update the catalog entry and its date stamp in the same commit as the change (per fin-change-control's docs-of-record rule).
@@ -0,0 +1,635 @@
---
name: fin-research-frontier
description: >-
The ranked research-and-development frontier for the fin app — what to build
next and why. Load when choosing the next significant project, when asked
"what should I work on next", "what's next", "what's missing", or "what's
the roadmap", and BEFORE starting any MCP-server, forecasting, budgeting,
recurring-transaction, portfolio-analytics, backup-automation, or
correctness-hardening work. Contains per-item first steps, milestones, and
the anti-backlog of projects deliberately not worth doing.
---
# fin — Research Frontier
The owner's four ambitions for this codebase, broken into concrete, ranked
projects. Every repo claim below was verified by reading the code on
2026-07-05 (commit 082b608, plus uncommitted `.claude/` work). Nothing here is
described as done unless it is; nothing is described as easy unless the
enabling code was actually read.
Ambitions: **(a)** provable ledger correctness, **(b)** agent-native finance,
**(c)** analytics & forecasting, **(d)** practical backlog & polish.
How to use this skill:
1. Pick the highest-ranked `open` item in the theme you care about. Items are
ranked by value÷risk within each theme.
2. Re-verify the "why current state falls short" claim before starting — the
code may have moved. Each item cites the exact file.
3. Any behavior/schema change routes through `fin-change-control` (gates,
migration discipline) and `fin-validation-and-qa` (evidence before "done").
Design constraints live in `fin-architecture-contract` — several items
below map directly onto its §5 "known weak points" table; the mapping is
noted per item.
Status labels: `open` (ready to start), `candidate` (worth doing, but do the
listed prerequisite or read-only probe first), `blocked-on-X` (a named thing
must land first).
---
## Theme A — Provable ledger correctness
### A1. Double-entry hardening: make intended invariants enforced — `partly-done 2026-08-22`
> **Status update 2026-08-22.** The create half is closed: `TransactionCreate`
> now narrows `type` to the single-leg kinds (expense/income/splitwise_pay/
> splitwise_receive) and rejects a caller-supplied `linked_account_id`, so
> `POST /api/transactions` can no longer write an orphan leg. Covered by
> `cli/tests/test_idempotency.py::TestOrphanLegGuard`. Note `splitwise_pay`
> stays allowed — a standalone row carrying `splitwise_partner_share` is a
> legitimate record and is what `fin tx add --type splitwise_pay` writes.
> **Still open:** only the measurement. The audit script itself already
> exists as `fin-diagnostics-and-tooling/scripts/audit_linked_legs.py` and
> already encodes the funded-buy whitelist described below; it has not been
> run against a prod backup, so the "0 violations" half of the result
> criterion is unmeasured.
**Why current state falls short.** The two-leg invariant is enforced only in
the dedicated endpoints. The raw `POST /api/transactions` accepts
`type: "transfer"` and `type: "settlement"` — both are in the
`TransactionCreate` Literal (`api/app/schemas.py:77`) and the schema even
accepts a free-standing `linked_account_id` (`schemas.py:83`) — and
`crud.create_transaction` (`api/app/crud.py:152`) writes
`models.Transaction(**model_dump)` verbatim with no linkage validation.
Verified 2026-07-05: nothing stops a client from creating a single unlinked
transfer leg, which is exactly the balance corruption that invariant #1 in
CLAUDE.md exists to prevent. The router blocks *editing* linked legs
(`routers/transactions.py:36-40`) but not *creating* orphans. Maps to
`fin-architecture-contract` §3 "two-leg atomic linked writes".
**Tractable because.** Every legitimate multi-leg write already has a
dedicated atomic endpoint (`/transfer`, `/settlement`, `/split-expense`,
`/splitwise-i-paid`, `/reconcile`) — rejecting raw multi-leg types loses no
capability. And the whole ledger is one SQLite file, so "are there existing
orphans?" is one query.
**First three steps.**
1. Grep all three clients for raw creates with these types before changing
anything: `grep -n "createTransaction\|create_transaction" web/src cli/fin_cli -r`
and check `web/src/components/QuickAdd/flows/` and `cli/fin_cli/tx.py`
only ever send `expense`/`income` through the raw endpoint.
2. In `api/app/routers/transactions.py::create_transaction`, reject
`type in ("transfer", "settlement")` and any non-null `linked_account_id`
with a 422 pointing at the dedicated endpoints. Router-level validation is
the sanctioned layer for this (`fin-architecture-contract` §2).
3. Add an e2e test in `cli/tests/test_cli.py` asserting the raw create of an
unlinked transfer is rejected, and write the orphan-leg audit query (leg
with `linked_account_id` set, `linked_transaction_id` null, no reciprocal
partner) as a read-only script for `fin-diagnostics-and-tooling` —
**caveat found while verifying**: funded-buy bank legs
(`crud.create_funded_buy`, line 603-612) legitimately have
`linked_account_id` set and `linked_transaction_id` null (their "partner"
is a StockTransaction, not a Transaction). The audit must whitelist legs
whose linked account is type `investment`/`rsu`, or it will false-positive
on every funded buy.
**You have a result when** the new e2e test passes (raw unlinked
transfer/settlement → 422) AND the orphan audit run against a prod backup
reports 0 violations outside the funded-buy whitelist. Measurable: test count
goes from 25 to ≥26 and the audit script prints a violation count.
### A2. Nightly self-audit job — `open`
**Why current state falls short.** No integrity check exists anywhere: no
`api/tests/`, no CI (`fin-architecture-contract` §5 #1), and derived state
(holdings) plus linked-leg groups can silently desync — e.g. the funded-buy
pair gap in D3 below is invisible until a balance looks wrong. Errors are
currently discovered by a human noticing a wrong number.
**Tractable because.** Every invariant is expressible as SQL over one SQLite
file, and `api/app/services/scheduler.py` is a clean extension seam: verified
2026-07-05, `create_scheduler` (line 84) registers jobs via `add_job`, and
both existing jobs show the exact thread-safe pattern to copy — build a
private engine via `_make_engine(database_url)` (WAL + FK pragmas, line
19-28), open a session, `dispose()` in `finally`. A third job is ~10 lines of
registration.
**First three steps.**
1. Create `api/app/services/integrity.py` with pure functions
`(db) -> list[violation dict]`: orphan linked legs (with the funded-buy
whitelist from A1), holdings vs. a re-run of
`recompute_holding_from_stock_transactions` fold, snapshot-date gaps,
`price_cache.fetched_at` staleness, vested RSU grants whose
`vested_stock_transaction_id` dangles. Reuse the query shapes from the
`fin-diagnostics-and-tooling` scripts rather than inventing new ones.
2. Register a `nightly_audit` job in `create_scheduler` (cron 00:30, after
the 00:00 snapshot job), following the `fetch_prices_and_snapshot`
engine-per-run pattern exactly; log findings at WARNING. Log-only first —
no schema, no migration, trivially revertable.
3. Expose `GET /api/integrity` (new router registered in `main.py` **with**
`dependencies=_auth` — see `fin-architecture-contract` §5 #9) + a
`FinClient.get_integrity()` method and `fin` subcommand, so both the owner
and agents can pull the latest result on demand.
**You have a result when** a test fixture DB with one deliberately deleted
transfer leg (raw SQL on a throwaway copy — never prod) yields exactly that
one violation from the audit functions, and a clean fixture yields zero.
Measurable: an e2e test asserts violation count 1 and 0 respectively.
### A3. Property-based ledger testing — `candidate` (execution lives in `fin-test-coverage-campaign`)
**Why current state falls short.** The 25 e2e tests in
`cli/tests/test_cli.py` certify specific scenarios; nothing certifies the
ledger under *arbitrary* operation sequences (create/delete transfers,
funded buys, RSU vests in random order). crud.py is 995 lines with no direct
tests (`fin-architecture-contract` §5 #1).
**Tractable because.** `FinClient` accepts an injected httpx transport /
client (`cli/fin_cli/client.py:31-54`, verified), so a Hypothesis strategy
can drive the full stack (FinClient → TestClient → SQLite) exactly as the
existing suite does, and check invariants after every step: per-account
balance = `SUM(amount − fee)`, transfer groups net to zero in same-currency
cases, holdings equal the fold of stock_transactions, delete is a true
inverse.
**First three steps + milestone.** Deliberately not duplicated here — this is
ambition (a)'s executable flagship and is specified in
`fin-test-coverage-campaign`. Start there; this entry exists only so the
frontier list is complete. Result criterion (shared with that skill): a
Hypothesis test that shrinks a real invariant violation to a minimal
counterexample, or N≥100 random 20-op sequences passing all invariants.
---
## Theme B — Agent-native finance
### B1. MCP server wrapping FinClient — done (2026-07-09, uncommitted): read-only phase + gated writes implemented
**Why current state falls short.** Agents today reach fin only through the
`fin` CLI (via the `fin` skill) — every query is a subprocess with text
parsing, and there is no tool-schema contract. No MCP server exists in the
repo (verified: `cli/` contains only `fin_cli/`, `tests/`, `pyproject.toml`).
**Tractable because.** This was designed for. Verified 2026-07-05:
`cli/fin_cli/client.py` imports only `os`, `typing`, and `httpx` — zero
typer/rich — and already covers every endpoint (net worth, accounts,
transactions incl. all atomic multi-leg creators, holdings, RSU grants,
spending, prices). The atomic endpoints are the safety asset: an agent
calling `create_transfer` can never half-write a ledger group. Errors arrive
as `FinClientError` with status codes — clean tool-error mapping.
**First three steps.**
1. Sketch package: `cli/fin_mcp/server.py` (FastMCP from the `mcp` SDK)
importing `fin_cli.client.FinClient`; add an optional dependency group
`mcp` to `cli/pyproject.toml` and a `fin-mcp` entry in `[project.scripts]`
so the core CLI install stays dependency-light. NOTE: the sandbox has no
pip/venv — implement statically, verify on the homelab or in the api
container (see `fin-build-and-env`).
2. Register **read-only tools only**: `get_net_worth`, `list_accounts`,
`list_transactions`, `get_spending_month`, `get_spending_summary`,
`get_holdings`, `list_rsu_grants`, `get_fx_rate`. Each returns the API
JSON untouched — same payloads `fin --json` already emits, so the `fin`
skill's documented semantics carry over.
3. Test without a network: reuse the existing suite's pattern of building
`FinClient(http=...)` over FastAPI's TestClient and call the tool
functions directly in `cli/tests/` (new `test_mcp.py`).
**You have a result when** an MCP client config pointing at `fin-mcp` lists
the read-only tools and `get_net_worth` returns byte-equal JSON to
`fin networth --json` against the same API. Measurable: one e2e test
asserting tool output == FinClient output for three tools.
Write tools (`create_transaction`, `create_transfer`, …) are `candidate`:
gate them behind an explicit env flag (e.g. `FIN_MCP_ALLOW_WRITES=1`) and do
B3's safety rails first. For any deployment where the API is reachable beyond
the LAN, writes are additionally **blocked-on-B2**.
### B2. Auth without locking out the frontend — `open` (prerequisite for exposed write access)
**Why current state falls short.** Verified and documented everywhere
(CLAUDE.md; `fin-architecture-contract` §5 #4): setting `FIN_API_TOKEN` on
the API locks out the React frontend entirely — `web/src/api/client.ts`
sends no `Authorization` header and nginx injects nothing. So today the
choice is "no auth" or "no web UI". Any agent-write story that leaves the
LAN needs this fixed first.
**Tractable because.** The single-user model means one static bearer token is
sufficient — no user table, no sessions. Nginx already fronts every `/api/*`
call from the browser (`web/nginx.conf`), so injecting
`proxy_set_header Authorization "Bearer <token>"` in that one proxy block
covers the whole SPA without the token ever reaching the browser.
**First three steps.**
1. Read `web/nginx.conf` and `web/Dockerfile` to pick the templating
mechanism (the stock nginx image runs `envsubst` over
`/etc/nginx/templates/*.template`; check whether the current conf is
copied verbatim or templated — `API_UPSTREAM` handling shows the current
approach).
2. Add an optional env (e.g. `WEB_API_BEARER`) that, when set, injects the
header in the `/api/` proxy block only; when unset, config is
byte-identical to today. Wire it in `docker-compose.yml`.
3. Update the configuration catalog in `fin-run-and-operate` §2 and the
CLAUDE.md warning paragraph; add the threat-model caveat that the token
then protects only direct `:8000` access, not the nginx-fronted UI.
**You have a result when**, with `FIN_API_TOKEN` set on the api container and
`WEB_API_BEARER` set on web: the Dashboard loads and a QuickAdd expense
saves; `curl` direct to the API without a token gets 401; and with both
unset, behavior is unchanged. Three checks, all binary. (Runtime verification
happens on the homelab — sandbox has no docker; see `fin-run-and-operate`.)
### B3. Natural-language entry safety rails: dry-run + idempotency — `partly-done 2026-08-22`
> **Status update 2026-08-22.** Idempotency shipped. Migration
> `i9d0e1f2a3b4` adds a nullable `client_ref` to `transactions`, accepted by
> `POST /api/transactions`, `/batch`, `/transfer` and `/settlement`, and
> exposed on the MCP `create_transaction`/`create_transfer`/
> `create_settlement` tools. Every row of one write shares the key and the
> creators check `crud.find_by_client_ref` first, so a retry replays the
> original rows. **Deviation from the plan below:** the index is NOT
> unique-when-not-null — a transfer writes two rows and a batch many under
> one key, so uniqueness is enforced by check-then-write instead, which is
> sound because SQLite serialises writers. Covered by
> `cli/tests/test_idempotency.py`.
> **Still open:** the `dry_run` flag on the multi-leg creators, and the
> duplicate-suspect diagnostic query.
**Why current state falls short.** NL parsing itself already works — the
`fin` skill has Sonnet-class agents booking via CLI today. What the repo
lacks is machinery that makes autonomous booking *safe*: there is no dry-run
mode on any write endpoint (verified: no such param in
`routers/transactions.py` or `schemas.py`) and no idempotency mechanism — an
agent that retries a timed-out `create_transfer` books it twice, and nothing
detects the duplicate.
**Tractable because.** The multi-leg creators are already pure
"request-schema in → legs out" functions in crud.py; a dry-run that builds
legs, returns them, and rolls back instead of committing reuses the exact
code path. Duplicate detection has full history in one SQLite file to check
against.
**First three steps.**
1. Read-only probe first: a duplicate-suspect query (same account pair, same
amount, same date, distinct link groups) as a `fin-diagnostics-and-tooling`
script — measures whether double-booking has ever actually happened.
2. Add `dry_run: bool = False` to `TransferRequest`/`SettlementRequest` in
`schemas.py`; in the crud creators, on dry-run build the legs, serialize,
`db.rollback()`, return with a marker. No schema migration needed.
3. Idempotency needs a nullable `client_ref` column on `transactions` →
Alembic revision chained after `h8c9d0e1f2a3` (this is the migration-gated
half; route through `fin-change-control` first). Unique-when-not-null;
creators return the existing group on a match instead of writing.
**You have a result when** an e2e test submits the same transfer twice with
one `client_ref` and exactly one two-leg group exists afterwards, and a
dry-run call leaves the transaction count unchanged while returning the
would-be legs. Status detail: dry-run is startable now; idempotency is
blocked-on a reviewed migration; both are most useful **after B1**, since
MCP tool descriptions are where the confirm-before-write protocol gets
enforced.
---
## Theme C — Analytics & forecasting
### C1. Category budget targets — `open`
**Why current state falls short.** No budget concept exists anywhere:
`grep -ri budget api/app web/src cli/fin_cli` returns nothing (verified
2026-07-05). The Spending page shows actuals only; there is no target to
compare against, which blocks the most basic "am I on track this month"
question.
**Tractable because.** `get_monthly_spending` (`api/app/crud.py:624`) already
returns per-category totals with names/colors/icons — a budget is one
nullable column and one extra field in that response. The Spending page
already renders the per-category list to attach progress bars to.
**First three steps.**
1. Add `monthly_budget = Column(Float, nullable=True)` to `Category` in
`api/app/models.py`; generate the Alembic revision chained after
`h8c9d0e1f2a3`, review it, and gate via `fin-change-control` (schema
class).
2. Surface it: `CategoryCreate`/`CategoryUpdate` in `schemas.py` (Optional,
`exclude_unset` pattern), and add `budget`/`remaining` to each category
dict in `get_monthly_spending`'s `category_data` loop (crud.py:714-721).
3. Clients: `web/src/api/client.ts` types + Settings category form +
Spending.tsx progress indicator; `fin_cli` category/spending commands.
E2e test asserting budget math for a seeded month.
**You have a result when** `fin spending month --json` shows
`budget`/`remaining` per category matching a hand-computed fixture in an e2e
test, and the Spending page renders over-budget categories distinctly.
### C2. Recurring-transaction detection — `open` (read-only analysis FIRST, no schema)
**Why current state falls short.** Nothing in the repo knows that rent,
salary, or subscriptions repeat — every forecast or "upcoming bills" feature
is blocked on this, and there is no detection code anywhere in `api/app` or
`cli/fin_cli` (verified).
**Tractable because.** The full transaction history lives in one SQLite file
with `date`, `amount`, `description`, `category_id` — recurrence detection is
a grouping problem over data already in hand, and it needs **zero** schema or
API changes to evaluate.
**First three steps.**
1. Write a read-only script (home: `fin-diagnostics-and-tooling` scripts
collection, or the scratchpad) that pulls transactions via
`FinClient.list_transactions` or reads a **backup copy** of the DB, groups
by (normalized description, amount within ±10%), and flags groups with
near-constant intervals (~7/14/30/365 days ± tolerance).
2. Run it on real (backup) data; manually judge the top-20 candidates for
precision. Tune tolerances. This step decides whether the idea graduates.
3. Only if precision is acceptable: promote to
`api/app/services/recurring.py` + `GET /api/spending/recurring` (router
with `_auth`) returning candidates with period and next expected date. No
stored state yet — recompute per request is fine at this scale.
**You have a result when** the script's top-10 candidates on real data are
≥80% genuinely recurring by the owner's judgment (falsifiable: the owner
labels them), with period and next-expected-date printed per candidate.
### C3. Cash-flow forecasting — `candidate` (baseline startable now; quality depends on C2)
**Why current state falls short.** The app answers "what happened" only.
`get_spending_summary` (`api/app/crud.py:751`) produces a per-month series of
`total` / `income_total` / `net_total`, but nothing projects forward, so
"can I afford X in October" is unanswerable in-app.
**Tractable because.** Three assets: the monthly net series from
`/api/spending/summary` (baseline signal); pending RSU grants in `rsu_grants`
with **future `vest_date` and `gross_shares`** — deterministic, dated future
income already in the DB (verified `models.py:115-126`); and
`net_worth_snapshots` for backtesting any model against what actually
happened.
**First three steps.**
1. Baseline script (read-only, via `FinClient.get_spending_summary`):
trailing-6-month median net per calendar month → project bank balances 3
months out. No repo changes.
2. Layer in known events: `list_rsu_grants` pending vests valued at cached
ticker price, and C2's recurring items when available.
3. Backtest: hindcast the last 3 months using only prior data; compare
against actual monthly nets and the naive "last month repeats" baseline.
Only promote to an API endpoint (`services/forecast.py`) if it beats
naive.
**You have a result when** the backtest prints a mean absolute error over ≥3
hindcast months that beats the naive baseline. If it cannot beat naive, the
honest result is "publish the negative result here and stop" — do not ship a
forecast worse than a constant.
### C4. Portfolio returns — money-weighted `open`; time-weighted `blocked-on-valuation-history`
**Why current state falls short.** The Portfolio page shows current value and
P&L vs. cost basis only; no return *rate* exists anywhere. And be honest
about the data: verified in `api/app/services/net_worth.py:104-175`, the
snapshot `breakdown_json` stores **per-account** `balance_native`/
`balance_eur` only — no per-holding values, no prices. `price_cache` keeps
one row per symbol with no history (`fin-architecture-contract` §3). So
per-holding valuation history does not exist and cannot be reconstructed.
**Tractable because (MWR).** `stock_transactions` is a complete dated
cashflow log (buy/sell/rsu_vest with `total_cost`, `fee`, `currency`,
verified `models.py:96-110`), and current value comes from holdings ×
`price_cache`. That is exactly the input XIRR needs. Funded-buy bank legs
(`crud.create_funded_buy`) even mark external cash flows into investment
accounts.
**First three steps.**
1. Read-only script: per investment account, build the cashflow vector from
`stock_transactions` (buys negative, sells positive, rsu_vest as inflow at
vest-day value) + terminal value from current holdings × cached prices;
solve XIRR by bisection (no new dependency needed).
2. Validate on a 3-trade hand-computed fixture to 4 decimal places.
3. Promote to `GET /api/portfolio/returns` in a new
`api/app/services/portfolio.py` + router with `_auth` + `client.ts` +
Portfolio page display; e2e test with the fixture.
**You have a result when** the e2e fixture's MWR matches the hand-computed
XIRR to 4 decimals, and the endpoint returns per-account MWR on real data
without error.
**TWR unblock path (do NOT backfill):** extend the snapshot breakdown for
investment/rsu accounts to include per-holding `{ticker, shares, price,
value_eur}` — an additive JSON change in `calculate_net_worth`
(`net_worth.py:126-153`), no migration, old snapshots simply stay
account-level. TWR becomes computable only for dates after that ships;
account-level TWR from existing daily snapshots + flow dates is a coarse
interim approximation. Anyone claiming full-history TWR is wrong — the price
history is gone.
---
## Theme D — Practical backlog & polish
Ranked by value÷risk. Verified backlog state first: the plan
`docs/superpowers/plans/2026-07-01-investment-income-features-plan.md` is
**fully implemented** — checked 2026-07-05 against the code: total-first buy
and currency selector are in `StockTradeFlow.tsx`, cross-currency transfers
in `crud.create_transfer` (`to_amount`/`to_currency`, lines 237-258),
`IncomeFlow.tsx` exists, income totals are in `get_monthly_spending`/
`get_spending_summary`, and both of the plan's open questions were resolved
(opening balances excluded via the `'Opening balance'` description filter,
crud.py:737; Spending.tsx guards relaxed to `total > 0 || income_total > 0`).
The only remainder is D5's `exchange_rate_used` note. No TODO/FIXME markers
exist anywhere in `api/app`, `web/src`, or `cli/fin_cli` (verified by grep).
### D1. Automated DB backups — `open` (highest value in this theme)
**Why current state falls short.** `fin-architecture-contract` §5 #6: the
prod SQLite volume is sacred and unprotected — no backup automation exists
anywhere in the repo, and snapshots are frozen JSON, so loss is
unrecoverable. `fin-run-and-operate` §3 has a proven **manual** runbook
(Python stdlib `sqlite3.backup()` inside the container — WAL-safe, and the
image has no `sqlite3` binary, so the Python API is the only option); this
item automates it.
**Tractable because.** The hard part (a WAL-safe consistent snapshot command)
is already written and tested in the runbook; automation is scheduling +
rotation + getting the file **off the volume**.
**First three steps.**
1. Commit the runbook as a script: `deploy/backup.sh` wrapping the
`docker compose exec api python -c "...src.backup(dst)..."` one-liner,
then `docker cp`/volume-mount copy to a host directory, then rotation
(keep last N, dated filenames). Repo-side only — runs on the homelab host.
2. Schedule it host-side (cron or systemd timer — document the unit in
`fin-run-and-operate`, since the repo cannot install host timers). An
in-app APScheduler job is NOT sufficient alone: it can only write to the
same volume it is trying to protect.
3. Restore drill, once: restore a backup into a scratch container, run
`alembic current` + `GET /api/health` + `fin networth --json` against it.
Document the drill result in `fin-run-and-operate`.
**You have a result when** ≥2 dated backup files exist outside the
`sqlite_data` volume, produced without manual action on consecutive days, and
one restore drill has produced a working API against the restored copy.
### D2. Funded-buy pair delete consistency — `open` (verified gap, small fix)
**Why current state falls short.** Found while verifying this skill
(2026-07-05): `create_funded_buy` (`crud.py:549-617`) writes a
StockTransaction and a bank `transfer` leg with **no reference between
them** — the bank leg gets `linked_account_id` but no
`linked_transaction_id`, and the stock txn has no back-pointer. Consequence,
confirmed by reading both delete paths: `delete_stock_transaction`
(crud.py:522) deletes the stock txn and recomputes the holding but leaves the
bank leg (cash stays deducted, position gone); `delete_transaction` on the
bank leg falls into the legacy heuristic (crud.py:205-225), finds no partner
on the investment account, and deletes only the bank leg (cash reappears,
shares remain). Either order leaves the ledger inconsistent.
**Tractable because.** The pairing already exists at creation time inside one
function — persisting it is one nullable FK.
**First three steps.**
1. Alembic migration (chained after `h8c9d0e1f2a3`, via `fin-change-control`)
adding nullable `funding_transaction_id` FK to `stock_transactions` (or
the mirror-image column on `transactions` — pick one direction, document
it in `fin-accounting-reference`).
2. Set it in `create_funded_buy`; in `delete_stock_transaction`, delete the
linked bank leg in the same commit; in `delete_transaction`, detect a bank
leg referenced by a stock txn and either cascade (delete both +
holding recompute) or 422 with "delete the stock transaction instead" —
the 422 is simpler and safer; start there.
3. E2e tests for both delete orders; extend the A2 audit with a
"funded-buy halves both present" check for legacy rows (old pairs stay
unlinked — the audit's date+amount matching flags survivors).
**You have a result when** deleting a funded buy through either entry point
leaves both the bank balance and the holding at their pre-buy values (e2e
asserted), and the audit reports 0 half-deleted pairs.
### D3. CSV export — `open`; import — `unblocked 2026-08-22, still gated`
> **Status update 2026-08-22.** B3's idempotency keys landed, so the
> re-import double-booking objection no longer holds. Import is still gated
> on A1's orphan-leg audit and on deciding how imported rows map to linked
> pairs — do not treat this as simply open.
**Why current state falls short.** No CSV/export code exists in any router or
CLI module (verified by grep, 2026-07-05). The data is only extractable as
JSON via API, which is awkward for spreadsheets and for the owner's
own-your-data expectations of a self-hosted app.
**Tractable because.** Export needs zero API changes: `FinClient.
list_transactions` already returns everything, so a CLI-side `fin tx export`
is pure presentation (stdlib `csv`, allowed in `fin_cli` command modules —
only `client.py` must stay presentation-free).
**First three steps.**
1. `fin tx export --output file.csv [--account ID] [--month YYYY-MM]` in
`cli/fin_cli/tx.py`, columns = the Transaction schema fields incl. `type`,
`fee`, `linked_account_id`, `splitwise_partner_share` (so the export is
semantically complete, not just pretty).
2. E2e test: seeded transactions → export → parse → per-account
`SUM(amount − fee)` equals the API's reported balances.
3. Same pattern for `stock_transactions`.
**You have a result when** the exported CSV re-summed per account matches
API balances exactly in an e2e test. **Import stays blocked**: naive
row-inserts through `POST /api/transactions` would create exactly the
unlinked legs A1 forbids, and re-imports double-book without B3's
idempotency keys. Do not build import before those land.
### D4. Fix synchronous `fetch-ticker` blocking — `open`
**Why current state falls short.** `fin-architecture-contract` §5 #5,
re-verified: `POST /api/prices/fetch-ticker` calls `fetch_ticker_price`
synchronously in the request handler, which retries 1s/2s/4s across up to 9
symbol candidates — worst case blocks a threadpool worker for minutes, and
CLAUDE.md explicitly forbids blocking price fetches on the hot path.
`POST /api/prices/refresh` already shows the correct pattern
(BackgroundTasks + 202).
**First three steps.** (1) Mirror the `/refresh` BackgroundTasks pattern in
`api/app/routers/prices.py` for fetch-ticker, returning 202; (2) adjust the
frontend caller in `client.ts`/Portfolio and `cli/fin_cli/prices.py` to poll
the price via the cache after a beat instead of expecting the price in the
response — this is a small behavior change, gate via `fin-change-control`;
(3) e2e test asserting 202 and eventual cache population (fake transport).
**You have a result when** fetch-ticker returns in <100 ms regardless of
yfinance behavior (assertable with a transport that sleeps) and the price
still lands in `price_cache`.
### D5. Small polish items — `open`, batchable
Each is grounded and tiny; batch them into an idle session.
- **`exchange_rate_used` on cross-currency transfers**: the only remainder of
the 2026-07-01 plan — `create_transfer` leaves it null (verified
crud.py:239-258). Prefill from `price_cache` at write time so historical
legs are auditable. Result: new cross-currency legs carry a rate; e2e
asserts it.
- **CLAUDE.md truth sync**: it still lists five account types (schemas.py has
six — `payable`) and says holdings are "incrementally updated on insert"
(code does a full recompute on insert too, crud.py:428-441/582). Result:
CLAUDE.md matches code; `fin-architecture-contract` already carries the
corrections.
- **Router-layer mutation cleanup** (`fin-architecture-contract` §5 #2): move
the `/reconcile` body (`routers/transactions.py:100-154`) and
`adjust-balance` into crud.py functions — pure relocation, no behavior
change, makes crud.py the complete mutation audit surface. Result:
`grep -n "db.add" api/app/routers/*.py` returns nothing and the existing
e2e tests still pass. Status: `candidate` — zero user value, do it when
already touching those files.
---
## Anti-backlog — deliberately not worth doing (as of 2026-07-05)
Re-litigate only with new facts; each reason is structural, not taste.
| Rejected project | Why not |
|---|---|
| **Multi-user / per-user auth** | Single-user assumptions are baked into net worth calculation and account filtering (CLAUDE.md "What NOT to do" says exactly this). Every query in crud.py is implicitly "the one user". B2's single static token is the ceiling. |
| **Async/AsyncSession rewrite** | Explicitly forbidden by CLAUDE.md; PLAN.md records the rationale (aiosqlite wraps a thread anyway; FastAPI threadpools sync handlers). Zero throughput need at single-user scale; guaranteed breakage of crud.py and the scheduler engine pattern. |
| **Real-time / streaming / intraday prices, or a price-history table** | yfinance has no SLA (`fin-architecture-contract` §5 #5); the 5-minute refresh job already exceeds any single-user need; and a history table contradicts the load-bearing "one row per symbol, history lives in snapshots" design (§3). Valuation-history needs are served by the C4 breakdown extension instead. |
| **Decimal/money-type migration** | Money is Float everywhere (§5 #7) and drift is cosmetic at this scale; the migration touches every column, every schema, every client, and every test for no user-visible gain. Revisit only if A2's audit measures drift beyond cents. |
| **Postgres migration** | SQLite+WAL fits the two-writer (requests + scheduler) single-user reality; would invalidate the pragma listeners, the backup runbook, the preflight check, and the volume model, and buy nothing. |
| **True journal/postings double-entry rewrite** | The linked-leg model already delivers the guarantees once A1+A2+D2 land, at a fraction of the risk. A rewrite re-opens every settled invariant in `fin-failure-archaeology` for zero new capability. |
| **Bank auto-sync (PSD2/Plaid-style)** | Heavy external dependency + credential custody on a self-hosted box; providers churn. CSV export (D3) plus, later, a gated import covers the realistic need. |
---
## Provenance and maintenance
All repo claims verified 2026-07-05 against commit 082b608 by reading:
`api/app/crud.py` (create/update/delete transaction, transfers, settlements,
funded buy, delete_stock_transaction, spending/summary), `api/app/schemas.py`
(type Literals, TransactionCreate fields), `api/app/models.py` (all tables),
`api/app/services/{scheduler,net_worth}.py`, `api/app/routers/
transactions.py`, `cli/fin_cli/client.py`, `cli/pyproject.toml`,
`web/src/components/QuickAdd/flows/StockTradeFlow.tsx`, `web/src/pages/
Spending.tsx`, `PLAN.md`, `docs/superpowers/plans/2026-07-01-investment-
income-features-plan.md`, and sibling skills (`fin-architecture-contract`
§5 weak points, `fin-run-and-operate` §3 backup runbook). Sibling skills
`fin-test-coverage-campaign`, `fin-ledger-analysis-toolkit`, and the
`fin-diagnostics-and-tooling` SKILL.md were being authored in parallel on the
same date — their referenced content may still be landing.
Volatile facts to re-verify before acting on an item:
- A1 gap still open: `grep -n "transfer" api/app/routers/transactions.py`
(look for a rejection in `create_transaction`) and schemas.py:77 Literal.
- D2 gap still open: `grep -n "linked" api/app/crud.py | grep -i fund` and
whether `stock_transactions` gained a funding FK (`models.py`).
- Migration head still `h8c9d0e1f2a3`: `ls api/alembic/versions/`.
- Frontend still tokenless: `grep -n "Authorization" web/src/api/client.ts`
(expect no hits) — else B2 is done, unblock B1 writes.
- MCP still absent: `ls cli/` (expect no `fin_mcp/`).
- Budget still absent: `grep -ri budget api/app web/src cli/fin_cli`.
- Test count baseline: `grep -c "def test" cli/tests/test_cli.py` (25 as of
this writing).
Maintenance rule: when an item ships, do not delete it — flip its status to
`done (YYYY-MM-DD, commit)` in one line and move the details to the relevant
sibling skill (`fin-architecture-contract` for new invariants,
`fin-run-and-operate` for new operational surfaces). When an item is
attempted and abandoned, record why here — a documented dead end is this
skill's second-most-valuable content after the ranking itself.
+407
View File
@@ -0,0 +1,407 @@
---
name: fin-run-and-operate
description: >
Runbook for deploying, restarting, backing up, and operating the fin stack on
the homelab. Load this skill when: deploying or rebuilding the Docker images,
pushing to Harbor, restarting containers, configuring or auditing any env var
(DATABASE_URL, FIN_API_TOKEN, FIN_API_URL, API_UPSTREAM), backing up or
restoring the production SQLite DB, investigating the APScheduler jobs or
stale prices in production, reading api logs, checking /api/health, or doing
anything that touches the homelab deployment. Contains the complete
configuration catalog for the project.
---
# fin — Run and Operate
Operational runbook for the fin stack (FastAPI `api` + nginx `web`, Docker
Compose, private Harbor registry). All commands below were **verified
statically** against `docker-compose.yml`, `api/Dockerfile`, `web/Dockerfile`,
and the source tree as of 2026-07-05 — this sandbox has no Docker, so nothing
here was executed live. Homelab-specific facts the repo cannot show are marked
**operator-supplied convention — confirm on the homelab**.
## When NOT to use this skill
| You actually want to… | Use instead |
|---|---|
| Drive the `fin` CLI to query/record finances | `fin` |
| Change code, write migrations, review invariants | `fin-change-control`, `fin-architecture-contract` |
| Debug wrong numbers / app bugs (incl. stale-price root-causing) | `fin-debugging-playbook` |
| Understand past incidents | `fin-failure-archaeology` |
| Set up a local build/dev environment | `fin-build-and-env` |
| Inspect data ad hoc, run queries | `fin-diagnostics-and-tooling`, `fin-ledger-analysis-toolkit` |
| Test/QA a change before deploy | `fin-validation-and-qa`, `fin-test-coverage-campaign` |
| Accounting semantics (balances, spending math) | `fin-accounting-reference` |
## The one iron rule
**The production SQLite DB (`/data/fin.db` in volume `sqlite_data`) holds the
owner's real financial history and is irreplaceable.** Before anything risky on
the homelab — deploy with new migrations, alembic downgrade, manual DB surgery —
take a backup (section below). Never point tests, dev servers, or the CLI's
write commands at production casually. Local/dev work uses a different
`DATABASE_URL`; see `fin-build-and-env`.
---
## 1. Deploy anatomy
### docker-compose.yml walkthrough (repo-verified)
| Aspect | `api` service | `web` service |
|---|---|---|
| Image | `harbor.cloud.traunseenet.com/library/fin-api:latest` | `harbor.cloud.traunseenet.com/library/fin-web:latest` |
| Build context | `./api` | `./web` |
| Ports | none published (internal :8000 only) | `80:80` (host 80 → nginx 80) |
| Volumes | `sqlite_data:/data` (named volume, the prod DB) | none |
| Environment | `DATABASE_URL=sqlite:////data/fin.db` | `API_UPSTREAM=api:8000` |
| Restart | `unless-stopped` | `unless-stopped` |
| Healthcheck | see verbatim below | none |
| depends_on | — | `api: condition: service_healthy` — web will not start until api's healthcheck passes |
Healthcheck, verbatim from `docker-compose.yml`:
```yaml
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')"]
interval: 30s
timeout: 10s
retries: 3
```
Note: the healthcheck hits `/api/health`, which is **always unauthenticated**
(registered in `main.py` outside the token dependency), so enabling
`FIN_API_TOKEN` never breaks the healthcheck or `depends_on`.
### What happens when the api container starts (repo-verified)
`api/Dockerfile` CMD, verbatim:
```dockerfile
CMD sh -c "python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000"
```
Sequence:
1. **Preflight** (`app/preflight.py`): resolves `DATABASE_URL`, creates the
SQLite parent directory, and verifies it is writable (writes a temp file).
Fails fast with "SQLite database directory is not writable" on bad volume
permissions.
2. **`alembic upgrade head`**: migrations apply automatically on every start.
`alembic/env.py` reads `DATABASE_URL` first, so migrations always target the
same DB the app uses. If a migration fails, the container never reaches
uvicorn → healthcheck fails → web never (re)starts.
3. **uvicorn** starts on `0.0.0.0:8000`, single process, no `--workers` flag
→ exactly one worker. **Do not add `--workers N>1`**: APScheduler starts in
the FastAPI lifespan, so multiple workers would run duplicate schedulers.
4. **Lifespan** (`app/main.py`): `Base.metadata.create_all(bind=engine)`
(first-boot convenience only, no-op afterwards — Alembic is the real
migration mechanism), then seeds 10 default categories iff the `categories`
table is empty, then builds and starts the APScheduler
`BackgroundScheduler` with a `SQLAlchemyJobStore` persisted in the same DB
(table `apscheduler_jobs`).
The container runs as non-root user `app` (uid 1000), created in the
Dockerfile with `/data` chowned to it. Do not revert to root.
The web image is a two-stage build (node:20-alpine `npm ci && npm run build`
→ nginx:alpine). `nginx.conf` is copied to
`/etc/nginx/templates/default.conf.template`; the stock nginx:alpine
entrypoint runs envsubst on templates, which is how `${API_UPSTREAM}` in
`proxy_pass http://${API_UPSTREAM};` gets substituted. nginx serves the SPA,
proxies `/api/*` to the api container, never caches `index.html`.
### Standard deploy runbook
On the dev machine (repo checkout):
```bash
# 1. Build both images (repo-verified: compose has build: for both services)
docker compose build
# 2. Push to Harbor (repo-verified: image: names point at the registry, so
# `docker compose push` targets harbor.cloud.traunseenet.com/library/fin-{api,web}:latest.
# Requires prior `docker login harbor.cloud.traunseenet.com` —
# operator-supplied convention: credentials live on the operator's machines.)
docker compose push
```
On the homelab (**operator-supplied convention — confirm on the homelab**:
where the compose file lives there and how it is kept in sync with the repo is
not visible in this repo):
```bash
docker compose pull && docker compose up -d
```
Post-deploy verification (always do this):
```bash
docker compose ps # api should be "healthy", web "running"
docker compose logs --tail=50 api # look for "Database preflight passed",
# alembic output, "APScheduler started"
curl -s http://localhost/api/health # via nginx → {"status":"ok"}
```
**Pre-deploy checklist when the deploy includes new Alembic migrations:**
- [ ] Backup the prod DB first (section 3).
- [ ] Migration is backward-safe (old code can still run against the new
schema), because the old container may restart against the migrated DB
during a botched rollout — and rollback means old image + new schema.
- [ ] Never plan to `alembic downgrade` on prod as a rollback strategy without
a backup in hand.
---
## 2. Configuration catalog
Every configuration axis in the project (as of 2026-07-05). Exhaustive: a grep
for `os.getenv`/`os.environ` across `api/` and `cli/` (excluding tests) finds
only `DATABASE_URL`, `FIN_API_TOKEN`, and `FIN_API_URL`; everything else is a
code constant or Docker-level setting.
| Axis | Where read | Default | Prod value / required? | Effect | Guard / caveat |
|---|---|---|---|---|---|
| `DATABASE_URL` | `api/app/db_config.py:12` (app+scheduler), `api/alembic/env.py` (~42, ~65) | `sqlite:///./fin.db` (code); Dockerfile bakes `sqlite:////data/fin.db` | Compose sets `sqlite:////data/fin.db` | DB location for app, Alembic, and the scheduler's private engine — all three read the same var | THE prod-safety axis. Tests/dev must use a different value. Note four slashes = absolute path |
| `FIN_API_TOKEN` (API side) | `api/app/auth.py:15`, read **per request** | unset → auth disabled (no-op) | Optional | When set, every router requires `Authorization: Bearer <token>`; 401 otherwise. `/api/health` always open; `/api/openapi.json` is token-guarded | **Setting it locks out the React frontend** — `web/src/api/client.ts` sends no Authorization header — until nginx injects the header or the frontend learns the token. Per-request read means rotation needs no restart, only env change + container recreate |
| `FIN_API_URL` (CLI side) | `cli/fin_cli/client.py:41` | none — **required**, deliberate | Required wherever `fin` runs | Base URL the CLI targets | No default so a misconfigured shell can never silently hit the wrong host |
| `FIN_API_TOKEN` (CLI side) | `cli/fin_cli/client.py:47` | empty → no header sent | Only if API enforces auth | Adds `Authorization: Bearer …` to all CLI requests | Must match the API-side token exactly |
| `API_UPSTREAM` | `web/nginx.conf:14` via nginx:alpine envsubst templating; ENV default in `web/Dockerfile:14` | `api:8000` | Compose sets `api:8000` | Where nginx proxies `/api/*` | Only change if the api service name/port changes |
| Healthcheck params | `docker-compose.yml:10-14` | interval 30s, timeout 10s, retries 3 | as-is | Container "healthy" gate; web start depends on it | Hits `/api/health` from inside the container with python urllib — no curl in the image |
| Price refresh interval | `api/app/services/scheduler.py:98-105` | `interval, minutes=5`, job id `price_refresh` | code constant | Fetches all prices every 5 min, no snapshot | Change in code only; jobs use `replace_existing=True` so redeploys update the persisted jobstore |
| Daily snapshot time | `api/app/services/scheduler.py:108-116` | `cron, hour=0, minute=0`, job id `daily_snapshot` | code constant | Midnight: fetch prices + upsert one `net_worth_snapshots` row per calendar day | Container timezone is UTC (python:3.12-slim default) unless TZ is set — "midnight" means UTC midnight |
| APScheduler jobstore | `scheduler.py:91-93` | `SQLAlchemyJobStore`, table `apscheduler_jobs` in the main DB | as-is | Jobs persist across restarts | Backups of fin.db include the jobstore — that is fine and expected |
| Price fetch retries | `api/app/services/price_fetcher.py:153-201` | 3 attempts/candidate, backoff `2**attempt` = 1s/2s/4s; suffix fallbacks `.DE .L .AS .PA .MI .SW .F .VI`; falls back to cached price, then 0.0 | code constant | yfinance fetch resilience | This blocks+sleeps — never call `fetch_all_prices` synchronously in a request handler (POST /refresh correctly uses BackgroundTasks) |
| FX fetch timeout | `price_fetcher.py:52-58, 80-86` | httpx `timeout=10` against `api.frankfurter.dev` (EUR/USD, EUR/GBP) | code constant | FX rates for net worth | On failure returns last cached rate |
| SQLite pragmas | `api/app/database.py:18-24` and `scheduler.py:21-27` | `journal_mode=WAL`, `foreign_keys=ON` on **every** new connection, both engines | invariant | Concurrency + FK integrity | Never remove; the scheduler thread has its own engine on purpose |
| uvicorn bind/workers | `api/Dockerfile:17` CMD | `--host 0.0.0.0 --port 8000`, no `--workers` → 1 | as-is | Single process runs both API and scheduler | Do not add workers (duplicate schedulers) |
| nginx / host port | `web/Dockerfile` EXPOSE 80; compose `"80:80"` | 80 | as-is | Public entry point | — |
| Container user | `api/Dockerfile:9-10` | user `app`, uid 1000, owns `/data` | invariant | Non-root runtime | Do not revert to root; volume files are uid 1000 |
| CLI HTTP timeout | `cli/fin_cli/client.py:52` | 30.0s | code constant | CLI request timeout | — |
| FastAPI docs | `api/app/main.py:51-57` | `docs_url/redoc_url/openapi_url=None` | as-is | No public Swagger; spec only at token-guarded `/api/openapi.json` | Deliberate — do not re-enable public docs |
### How to add a new configuration axis (checklist)
- [ ] Read it via `os.getenv` in exactly one place — `db_config.py` for
DB-related, the owning service module otherwise. Give it a safe default
so existing deployments keep working (follow the `FIN_API_TOKEN` opt-in
pattern).
- [ ] Thread it through `docker-compose.yml` `environment:` (and Dockerfile
`ENV` if a baked default makes sense, mirroring `DATABASE_URL`).
- [ ] If the scheduler needs it, pass it as a job arg like `database_url` —
the scheduler runs in a background thread with its own engine.
- [ ] Add a row to the env-var table in `README.md` (existing table around
line 32).
- [ ] Add a row to the catalog table in **this file** with file:line and
caveats.
- [ ] If it changes CLI behavior, update `.claude/skills/fin/SKILL.md`.
---
## 3. Production DB protection (backup & restore)
Facts (repo-verified): the DB is SQLite in WAL mode at `/data/fin.db` inside
the `api` container, on named volume `sqlite_data`. The image is
`python:3.12-slim` with only pip packages installed — **the `sqlite3`
command-line binary is NOT in the image** (no apt-get in `api/Dockerfile`), but
Python's stdlib `sqlite3` module is. So backups use the Python backup API,
which is WAL-safe (it takes a consistent snapshot without stopping writers).
### Backup runbook (online, WAL-safe — preferred)
Run on the homelab host:
```bash
# 1. Create a consistent snapshot inside the container (stdlib backup API)
docker compose exec api python -c "
import sqlite3
src = sqlite3.connect('/data/fin.db')
dst = sqlite3.connect('/data/fin-backup.db')
with dst:
src.backup(dst)
dst.close(); src.close()
print('backup written')
"
# 2. Copy it out and clean up
docker compose cp api:/data/fin-backup.db ./fin-backup-$(date +%F).db
docker compose exec api rm /data/fin-backup.db
# 3. Verify the copy is a sane database
python3 -c "
import sqlite3
db = sqlite3.connect('fin-backup-$(date +%F).db')
print(db.execute('PRAGMA integrity_check').fetchone())
print('transactions:', db.execute('SELECT COUNT(*) FROM transactions').fetchone()[0])
"
```
Move the verified file off the homelab (**operator-supplied convention —
confirm where offsite backups live**).
### Backup runbook (offline, raw copy — when the stack is stopped anyway)
A raw copy of a live WAL database is only consistent if you copy `fin.db` +
`fin.db-wal` + `fin.db-shm` together **and** nothing is writing. Simplest safe
form — stop the api first:
```bash
docker compose stop api
docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/backup alpine \
tar czf /backup/fin-volume-$(date +%F).tar.gz -C /data .
docker compose start api
```
`<project>_sqlite_data` is the real volume name — compose prefixes the project
name (directory name by default; check with `docker volume ls`).
**Operator-supplied convention — confirm the actual volume name on the
homelab.**
### Restore runbook
```bash
docker compose stop api # 1. stop all writers
docker compose cp ./fin-backup-YYYY-MM-DD.db api:/data/fin.db # 2. replace the DB
# If the container won't run stopped-exec on your compose version, use the
# volume-mount approach instead:
# docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/backup alpine \
# sh -c "cp /backup/fin-backup-YYYY-MM-DD.db /data/fin.db && rm -f /data/fin.db-wal /data/fin.db-shm && chown 1000:1000 /data/fin.db"
# 3. Remove stale WAL sidecars — a leftover -wal from the OLD database would
# corrupt the restored one on first open
docker compose exec api rm -f /data/fin.db-wal /data/fin.db-shm # (before start, via the volume-mount form above if api is stopped)
docker compose start api # 4. start; alembic upgrade head re-runs
docker compose logs --tail=30 api # 5. verify preflight + migrations + scheduler start
curl -s http://localhost/api/health
```
File ownership matters: the api runs as uid 1000, so restored files must be
readable/writable by uid 1000 (the `chown 1000:1000` above).
### Hard rules
1. **Never point tests at prod.** pytest, dev servers, and experiments use
their own `DATABASE_URL`. The prod URL is exactly `sqlite:////data/fin.db`
inside the api container and nowhere else.
2. **Never run `alembic downgrade` on prod without a same-day backup.**
3. **Migrations must be backward-safe** (additive first; destructive changes
only after the code that needed the old shape is gone).
4. **Never edit `/data/fin.db` with raw DDL or by hand** — CLAUDE.md rule; all
schema changes go through Alembic, all data changes through the API/CLI.
---
## 4. Routine operations
| Task | Command | Expected result |
|---|---|---|
| Health check | `curl -s http://localhost/api/health` (via nginx) or from the host running compose: `docker compose exec api python -c "import urllib.request;print(urllib.request.urlopen('http://localhost:8000/api/health').read())"` | `{"status":"ok"}` — always open, even with auth enabled |
| Manual price refresh (API) | `curl -s -X POST http://localhost/api/prices/refresh` (+ `-H "Authorization: Bearer $FIN_API_TOKEN"` if auth on) | `202` `{"status":"accepted","message":"Price refresh queued"}` — runs as a FastAPI BackgroundTask, verified in `api/app/routers/prices.py:19-23` |
| Manual price refresh (CLI) | `fin prices refresh` | same 202 via `FinClient.refresh_prices()` |
| Read api logs | `docker compose logs --tail=100 api` / `docker compose logs -f api` | See patterns below |
| Restart api only | `docker compose restart api` | Preflight → alembic → uvicorn → scheduler again; web keeps running |
| Verify daily snapshots | `fin networth history --json` or `curl -s http://localhost/api/net-worth/history?range=1M` (auth header if enabled) | One row per calendar day; the newest `snapshot_date` should be today or yesterday (UTC midnight job) |
| Check scheduler health | `docker compose logs api \| grep -E "Price refresh|Daily job|APScheduler"` | `Price refresh: updated N price(s)` roughly every 5 min; `Daily job: created/updated snapshot for YYYY-MM-DD` once per day; `APScheduler started` on boot |
Healthy-startup log fingerprint (in order):
`Database preflight passed` → Alembic `Running upgrade …` lines (or nothing if
already at head) → `APScheduler started (daily_snapshot job scheduled at
midnight)` → uvicorn `Application startup complete`.
---
## 5. Data and artifact conventions
Verified: the application writes **no files other than the SQLite database and
its WAL sidecars**. Everything lands in `/data` on volume `sqlite_data`:
- `fin.db` — all tables: accounts, transactions, holdings, stock_transactions,
rsu_grants, categories, `price_cache` (one upserted row per symbol, incl.
`EUR/USD` and `EUR/GBP`), `net_worth_snapshots` (one row per day: total_eur
+ JSON breakdown), and `apscheduler_jobs` (persisted scheduler state).
- `fin.db-wal`, `fin.db-shm` — WAL sidecars, present while the app runs.
No log files on disk (logs go to stdout → `docker compose logs`), no export
files, no uploads. Backing up `/data` backs up everything.
---
## 6. Failure drills
**api container unhealthy / restart-looping**
`docker compose logs --tail=200 api`. The CMD chain fails in order:
1. `SQLite database directory is not writable` → volume permission problem;
`/data` must be owned by uid 1000.
2. Alembic traceback → a migration failed; the app never started, so the DB is
whatever state the failed migration left. Do NOT retry blindly — restore
from the pre-deploy backup or fix-forward with a corrected migration
(see `fin-change-control`).
3. Uvicorn import error → bad image; roll back to the previous image tag
(**operator convention: only `:latest` is pushed per the compose file — if
no older tag exists in Harbor, rebuild from the last good commit**).
**web up but SPA can't reach API**
`depends_on: service_healthy` only gates startup ordering, not runtime. Check
api health directly, then check whether `FIN_API_TOKEN` was recently set on
the api — that 401s the frontend by design (catalog caveat above).
**Prices stale**
Confirm the 5-minute `Price refresh` log lines are appearing. If the job runs
but prices don't move, it is a fetcher/upstream problem (yfinance,
frankfurter) — hand off to `fin-debugging-playbook`. If the job lines are
absent entirely, the scheduler died with the process: restart api and check
startup logs. Quick mitigation either way: `fin prices refresh`.
**Missing daily snapshot rows**
The midnight job upserts by `snapshot_date`; a gap means the container was
down at UTC midnight or the job errored (`Daily job: failed to upsert
snapshot` / `Daily job: unexpected error` in logs). A missed day cannot be
recomputed retroactively (it snapshots *current* net worth) — accept the gap;
`POST /api/prices/refresh` does not write snapshots.
**Disk growth / huge fin.db-wal**
WAL normally auto-checkpoints (SQLite default ~1000 pages). A persistently
large `-wal` means a long-lived reader is blocking checkpoints. Force one:
```bash
docker compose exec api python -c "
import sqlite3
db = sqlite3.connect('/data/fin.db')
print(db.execute('PRAGMA wal_checkpoint(TRUNCATE)').fetchone())
db.close()
"
```
`(0, N, N)` means success. If it stays blocked, `docker compose restart api`
clears lingering connections, then checkpoint again.
---
## Provenance and maintenance
All facts verified statically against the repo at commit `082b608` (2026-07-05).
Re-verify before trusting, with one command each:
- Compose topology, healthcheck, ports, volume: `cat docker-compose.yml`
- api start sequence + uvicorn args + uid: `cat api/Dockerfile` (CMD on line 17)
- Lifespan (create_all, category seed, scheduler start): `sed -n '16,49p' api/app/main.py`
- Scheduler timings (5 min interval, midnight cron, jobstore): `sed -n '84,119p' api/app/services/scheduler.py`
- Env var inventory: `grep -rn "os.getenv\|os.environ" api/ cli/ --include="*.py" | grep -v tests`
- Auth semantics: `cat api/app/auth.py`; open health route: `grep -n "api/health" api/app/main.py`
- Price refresh endpoint (POST, 202, background): `sed -n '19,23p' api/app/routers/prices.py`
- Retry/backoff constants: `sed -n '153,201p' api/app/services/price_fetcher.py`
- nginx templating of API_UPSTREAM: `cat web/nginx.conf web/Dockerfile`
- sqlite3 CLI still absent from image: `grep -n "apt-get\|sqlite3" api/Dockerfile` (expect no hits)
Operator-supplied conventions to re-confirm on the homelab (not derivable from
the repo): where the compose file lives on the homelab and how it is synced;
Harbor login credentials/procedure; the real docker volume name
(`docker volume ls`); offsite backup destination; whether any tag other than
`:latest` is pushed.
Update this file whenever: a new env var is added (catalog table + README),
scheduler timings change, the Dockerfile CMD changes, or the registry/deploy
path changes.
@@ -0,0 +1,831 @@
---
name: fin-test-coverage-campaign
description: The executable campaign to take fin from 25 e2e CLI tests to a real safety net — direct api-level tests for crud.py, regression tests for every historical bug family, property tests for ledger invariants, service tests (net worth, price fetcher, scheduler), Alembic migration tests, and CI. Load when asked to improve test coverage, harden crud.py or services against regressions, add CI or a test gate, write api/tests, measure coverage, or after ANY accounting bug is found or fixed (to lock down its whole family). Also load before refactoring crud.py — characterization tests come first.
---
# fin — Test Coverage Campaign
Mission: the only tests are `cli/tests/` (25 e2e tests). `api/app/crud.py` — 995
lines holding ALL DB mutation logic — has no direct tests. `services/`, router
error mapping, and the 8-revision Alembic chain are untested. There is no CI.
Every accounting bug in this project's history shipped because nothing failed
(chronicle: `fin-failure-archaeology`). This campaign closes that, phase by
phase, with a measurable gate at every step.
All facts and numbers date-stamped **2026-07-05** (HEAD `082b608`). The
authoring sandbox had no pip/venv, so every command below is **verified
statically** unless it appears in the git history or skill siblings as
executed; the gates therefore say "record the number you observe; expected ≈N".
If your observation differs from the expectation, follow the branch instruction
— never shrug and continue.
Read first: `/CLAUDE.md` (root rulebook). Alignment contracts:
- **Promotion**: every phase lands as its own reviewed change through
`fin-change-control` (§1 classification, §7 done-checklist). Tests-only diffs
that touch no invariant file classify as behavior changes whose "test
covering the new behavior" requirement is satisfied by the tests themselves.
- **Evidence**: this campaign produces level-(a) evidence in the
`fin-validation-and-qa` hierarchy. Follow its hypothesis-predicts-numbers
discipline: write the expected number and mechanism as a comment BEFORE the
first run of every new test.
- **Regression targets**: Phase 2 converts the `fin-failure-archaeology`
chronicle into named tests. Read that skill before writing Phase 2.
- **Environment**: `fin-build-and-env` owns setup (venv, minimal deps,
yfinance-stub rationale). Not duplicated here.
- **Oracles**: `fin-diagnostics-and-tooling` and `fin-ledger-analysis-toolkit`
(being written in parallel as of 2026-07-05) may provide audit scripts and
derived invariants; check them before hand-rolling an oracle in Phase 3.
## When NOT to use this skill
| You actually want to… | Use instead |
|---|---|
| Add one e2e test for a change you just made | `fin-validation-and-qa` §4 |
| Decide whether a change is allowed / gates | `fin-change-control` |
| Set up the Python environment | `fin-build-and-env` |
| Understand accounting math for expected values | `fin-accounting-reference` |
| Look up a past bug before pinning it | `fin-failure-archaeology` |
| Debug a live failure | `fin-debugging-playbook` |
Use THIS skill to execute or resume the coverage campaign, and whenever a new
bug is fixed — its family gets a Phase-2-style regression test even after the
campaign is "done".
## Ground truth — what exists (2026-07-05)
### crud.py public surface — the coverage target
35 top-level `def`s in `api/app/crud.py` (verify: `grep -cE '^def ' api/app/crud.py`),
33 public + 2 private:
| Area | Functions (count) |
|---|---|
| Accounts (5) | `get_accounts`, `get_account`, `create_account`, `update_account`, `deactivate_account` |
| Categories (5) | `get_categories`, `get_category`, `create_category`, `update_category`, `delete_category` |
| Transactions (9) | `get_transactions`, `get_transaction`, `create_transaction`, `update_transaction`, `delete_transaction`, `create_transfer`, `create_settlement`, `create_split_expense`, `create_splitwise_i_paid` |
| Holdings (2) | `get_holdings`, `get_holding` |
| Stock transactions (5) | `create_stock_transaction`, `get_stock_transactions`, `recompute_holding_from_stock_transactions`, `delete_stock_transaction`, `create_funded_buy` |
| Spending (2) | `get_monthly_spending`, `get_spending_summary` |
| RSU helpers, private (2) | `_add_months`, `_generate_rsu_schedule` |
| RSU grants (5) | `get_rsu_grants`, `get_rsu_grant`, `create_rsu_grant`, `delete_rsu_grant`, `vest_rsu_grant` |
Exercised by the existing 25 cli tests (13 of 33): `get_accounts`,
`create_account`, `get_categories`, `create_category`, `get_transactions`,
`get_transaction`, `create_transaction`, `delete_transaction`,
`create_transfer`, `create_settlement`, `create_split_expense`,
`create_splitwise_i_paid`, `get_monthly_spending`.
**Never executed by any test (20 public + 2 private)**: everything in the
Accounts area except create/list, all category mutations, `update_transaction`
(the one existing edit test is rejected at the router before crud runs), the
entire Holdings, Stock-transactions, RSU areas, and `get_spending_summary`.
CLAUDE.md invariant 2 (holdings recompute) is completely untested.
Also untested but NOT in crud.py: `reconcile_accounts` — a two-leg settlement
writer that lives in `api/app/routers/transactions.py` (line ~100), the only
paired-leg writer outside crud. It is a Phase 2 target too.
### Existing suite
`cli/tests/test_cli.py`: **25 `def test_` functions** (verify:
`grep -c 'def test_' cli/tests/test_cli.py`), inventoried per-test in
`fin-validation-and-qa` §2. `cli/tests/conftest.py` (59 lines): sets a
throwaway `DATABASE_URL` before the `app.*` import chain, stubs `yfinance` via
`sys.modules.setdefault`, autouse `drop_all`+`create_all` per test.
### Migration chain
8 revisions, linear, head `h8c9d0e1f2a3` (verify: `ls api/alembic/versions/`).
Table in `fin-change-control` §4.2.
### Git remote — checked for Phase 6
```
origin https://gitea.cloud.traunseenet.com/markuseisl/fin.git
```
Self-hosted **Gitea**, not GitHub (observed 2026-07-05 via `git remote -v`).
GitHub Actions is off the menu; see Phase 6.
---
## Phase 0 — Baseline (run before touching anything)
### 0.1 Environment
Follow `fin-build-and-env` §(b), then add the coverage tool:
```bash
cd /home/markuseisl/Documents/git/fin
python -m venv .venv && . .venv/bin/activate
pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0" coverage
pip install -e cli/
```
Do NOT install yfinance (deliberately stubbed) or alembic yet (needed only
from Phase 5).
### 0.2 Run the suite as-found
```bash
python -m pytest cli/tests -q
```
**Record**: test count, pass/fail, wall time.
**Expected**: `25 passed`, well under 60 s (as of 2026-07-05).
**Gate 0a — branch table:**
- `25 passed` → continue.
- More than 25 passed → someone added tests since 2026-07-05; re-read
`cli/tests/`, update the ground-truth section counts, continue.
- Any failure or error **as-found** → **STOP the campaign.** A broken baseline
means either the environment is wrong (missing dep → `fin-build-and-env`
traps table) or a real regression shipped untested. Fix or quarantine
(`@pytest.mark.skip` with a linked reason) as its own change through
`fin-change-control` first, and record the finding as a new entry candidate
for `fin-failure-archaeology`. The campaign starts from green, always.
### 0.3 Baseline coverage
```bash
python -m coverage run -m pytest cli/tests -q
python -m coverage report --include='*/api/app/*' -m
```
**Record** (into your session notes and the final phase commit message):
per-file % for `crud.py`, `services/net_worth.py`, `services/price_fetcher.py`,
`services/scheduler.py`, `routers/*.py`, and the total.
**Expected shape** (predicted statically from the exercised-function map above
— record what you actually see): `crud.py` roughly 45–65% (20 of 33 public
functions never run); `net_worth.py` moderate (cash paths only);
`price_fetcher.py` and `scheduler.py` near import-only (≈15–35%); router
coverage high for `transactions.py`/`accounts.py`, near-zero for
`stock_transactions.py`/`rsu_grants.py`/`holdings.py`/`prices.py`.
**Gate 0b:** if `crud.py` reports > 80% at baseline, your `--include` filter is
wrong (probably matching `fin_cli`) or the suite grew — investigate before
trusting any later delta. If `coverage` reports "no data collected", you ran
pytest without `coverage run` wrapping it.
The recorded table is the campaign's before-photo; every later phase gate is a
delta against it.
---
## Phase 1 — Infrastructure: `api/tests/` for direct crud/service tests
The cli suite tests the deployed HTTP contract; it cannot reach crud/service
functions the routers don't expose (e.g. `recompute_holding_from_stock_transactions`
mid-history, `get_spending_summary` edge modes, `_generate_rsu_schedule`), and
it cannot seed `price_cache` directly. That is why a second, api-level suite is
justified rather than growing `cli/tests` (which stays e2e-only per
`fin-validation-and-qa` §4).
### 1.1 Create `api/tests/conftest.py` — exact content
Modeled on `cli/tests/conftest.py` mechanics: throwaway `DATABASE_URL` before
the import chain, yfinance stub. Verified statically against
`app/database.py` (reads `DATABASE_URL` at import time via `db_config`).
```python
"""Direct api-layer fixtures: crud/services against a throwaway SQLite.
Same bootstrap discipline as cli/tests/conftest.py: DATABASE_URL must point at
a throwaway file BEFORE anything imports app.database (which reads it at
import time), and yfinance is stubbed so the heavy package is not required.
Unconditional overwrite: a real DATABASE_URL in the shell must never leak in.
"""
import os
import sys
import tempfile
import types
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT / "api"))
sys.modules.setdefault("yfinance", types.ModuleType("yfinance"))
_db_dir = tempfile.mkdtemp(prefix="fin-api-tests-")
os.environ["DATABASE_URL"] = f"sqlite:///{_db_dir}/test.db"
from app.database import Base, SessionLocal, engine # noqa: E402
from app import models # noqa: E402
@pytest.fixture(autouse=True)
def _fresh_schema():
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
yield
@pytest.fixture
def db():
session = SessionLocal()
try:
yield session
finally:
session.rollback()
session.close()
@pytest.fixture
def make_account(db):
def _make(name="Giro", type_="bank", currency="EUR"):
acct = models.Account(name=name, type=type_, currency=currency)
db.add(acct)
db.commit()
db.refresh(acct)
return acct
return _make
@pytest.fixture
def seed_price(db):
"""Insert a price_cache row directly — the only sanctioned way to get a
price into tests (yfinance is a stub; never fetch)."""
from datetime import datetime
def _seed(symbol, price, currency="USD", source="test"):
db.add(models.PriceCache(symbol=symbol, price=price,
currency=currency, source=source,
fetched_at=datetime.utcnow()))
db.commit()
return _seed
```
Coexistence note (verified statically): when both suites run in one pytest
invocation, both conftests set throwaway URLs but `app.database` is imported
once — the engine binds to whichever conftest imports first. Both URLs are
throwaway temp dirs, so this is safe; both autouse fixtures drop/create the
same engine's schema per test. If you ever see one suite's data in the other,
that is a missing autouse fixture, not a conftest bug.
Rule inherited from `fin-change-control` §2.1: **tests use throwaway databases
only** — the temp-dir-then-import pattern above is the load-bearing part. Do
not "simplify" it.
### 1.2 One trivial proving test — `api/tests/test_crud_accounts.py`
```python
from app import crud, schemas
def test_create_account_with_opening_balance(db):
acct = crud.create_account(db, schemas.AccountCreate(
name="Giro", type="bank", currency="EUR", initial_balance=250.0))
# create_account writes one opening 'income' transaction of 250
txns = crud.get_transactions(db, account_id=acct.id)
assert len(txns) == 1
assert txns[0].amount == 250.0
assert txns[0].type == "income"
assert txns[0].description == "Opening balance"
```
### 1.3 Gate 1
```bash
python -m pytest api/tests -q # expect: 1 passed
python -m pytest cli/tests -q # expect: still 25 passed
python -m pytest cli/tests api/tests -q # expect: 26 passed — coexistence proven
```
Branches:
- `ModuleNotFoundError: app` in api/tests → the `sys.path.insert` line is
missing or the file moved; do not fix by installing the api package or
editing cli's conftest.
- cli/tests suddenly failing → your api conftest leaked state (most likely you
made the `DATABASE_URL` assignment conditional, or imported `app.*` above the
env assignment). Revert to the exact code above.
- Combined run failing while individual runs pass → fixture-ordering conflict;
check both autouse fixtures still drop/create and that neither conftest
caches Session objects at module scope.
Land Phase 1 as its own change (conftest + one test). Classification:
behavior-change gate per `fin-change-control` §1.
---
## Phase 2 — Regression lockdown: the chronicle becomes tests
Convert `fin-failure-archaeology`'s bug families into named tests. Naming
convention: `test_regress_<sha>_<slug>` so the pinned incident is greppable.
For each family below: the exact scenario, the predicted numbers (write them as
comments per `fin-validation-and-qa` §1), and the function it pins. Put crud-
level tests in `api/tests/test_regressions.py` (split by area when it grows);
anything asserting HTTP status codes belongs in `cli/tests` or an api-level
`TestClient` test.
### Family A — spending/income inflation (`e36d719`, migration `h8c9d0e1f2a3`)
Pins: `create_funded_buy`, `get_monthly_spending`, router `reconcile_accounts`.
1. `test_regress_e36d719_funded_buy_not_spending` — EUR bank account +
investment account; `crud.create_funded_buy` with
`FundedBuyRequest(type="buy", shares=5, price_per_share=100, fee=0,
total_cost=500, currency="USD", funded_by_account_id=bank.id, …)` and
`eur_usd_rate=1.25`. Predict: bank leg `type == "transfer"`,
`amount == -400.0` (−500/1.25), description `"Buy 5 <TICKER>"`;
`get_monthly_spending` for that month → `total == 0.0` and
`income_total == 0.0`.
2. `test_regress_e36d719_funded_sell_not_income` — funded sell 5 @ 100, fee 2:
bank leg `+498/1.25 == 398.4`, type `transfer`; income_total 0.
3. `test_regress_e36d719_reconcile_not_income` — via `TestClient` POST
`/api/transactions/reconcile` (bank + receivable account): two `settlement`
legs, bank `+amount`, receivable `−amount`; spending total 0, income_total 0.
4. `test_regress_a9f045f_e61b254_payable_counts_receivable_excluded` — expense
−30 on a **payable** account counts (total 30); expense −30 on a
**receivable** account is excluded (total 0). The asymmetry is intentional —
do not "fix" it (settled question, archaeology top table).
5. `test_opening_balance_not_income` — `create_account(initial_balance=500)` →
`get_monthly_spending(current month)["income_total"] == 0.0` (the
`description != 'Opening balance'` filter in crud).
### Family B — orphan legs / linked-group deletes (`a06f921`, `a1f109b`, `e36d719`)
Pins: `delete_transaction`, `create_transfer`, `create_settlement`,
`create_split_expense`, `create_splitwise_i_paid`.
1. `test_regress_a06f921_delete_either_transfer_leg_removes_both` — create
transfer, delete leg 1 → 0 rows remain; repeat deleting leg 2 first.
2. `test_regress_e36d719_split_expense_delete_any_of_three` — three
parametrized cases: deleting the expense row, the from-leg, or the to-leg
each removes all 3 rows (the expense links via `linked_transaction_id` to
the from-leg; delete follows links in both directions).
3. `test_regress_a1f109b_legacy_fallback_picks_adjacent_leg` — two transfers
between the same two accounts on the same date; null out
`linked_transaction_id` on all four rows (simulating pre-migration legacy
rows, direct UPDATE via the db fixture); delete one leg → exactly its
closest-id partner dies with it, the other transfer's 2 rows survive.
4. `test_zero_sum_after_any_group_delete` — after each delete above, no
remaining row's `linked_transaction_id` references a missing id.
### Family C — FX missing means excluded, never mis-converted (`6b7d19b`, `81091fc`)
Pins: `services/net_worth._to_eur` via `calculate_net_worth`.
1. `test_regress_6b7d19b_missing_gbp_rate_excludes_not_converts` — GBP bank
account with balance 100 (one income txn); seed only `EUR/USD = 1.10`.
Predict: that account's `balance_eur == 0.0`, `total_eur == 0.0`, and
`caplog` contains `"No EUR/GBP rate"`. Negative control in the same test:
seed `EUR/GBP = 0.85` → `balance_eur == pytest.approx(117.647, rel=1e-4)`
(100 / 0.85).
2. `test_eur_usd_fallback_is_reported_not_used_for_conversion` — no price rows
at all, one EUR account with 100: `eur_usd_rate == 1.0` (reported fallback,
warning logged) but `total_eur == 100.0` untouched. This pins the current
behavior; if the 1.0 report is ever removed, this test is the notification.
### Family D — holdings recompute (`c558999`, `1480088`, `bc256f6`)
Pins: `create_stock_transaction`, `delete_stock_transaction`,
`recompute_holding_from_stock_transactions`, `vest_rsu_grant`. This is
CLAUDE.md invariant 2's first-ever test coverage.
1. `test_regress_c558999_avg_cost_fold` — buy 10 @ 100 fee 5 → holding
`shares == 10`, `avg_cost_basis_per_share == 100.5`; then sell 4 →
`shares == 6`, avg stays `100.5` (weighted-average sell reduces cost
proportionally).
2. `test_regress_c558999_oversell_rejected_422` — sell 20 with 10 held:
`crud.create_stock_transaction` raises `HTTPException` with
`status_code == 422` and the holding is unchanged (rollback happened).
3. `test_regress_1480088_delete_mid_history_recomputes` — buy 10, buy 10, sell
5; delete the second buy → holding recomputed over the remaining history to
`shares == 5` (10 − 5); predict the resulting avg cost by hand in the
comment first. Then: buy 10, sell 8, delete the buy →
`delete_stock_transaction` raises 422 ("Cannot sell 8"), the buy row still
exists, holding unchanged.
4. `test_delete_last_stock_txn_deletes_holding` — one buy, delete it → holding
row gone (`get_holding` returns None).
5. `test_regress_bc256f6_vest_delete_reverts_grant` — RSU account, grant
(gross 100, withholding 0.5), seed price for the ticker, `vest_rsu_grant` →
`net_shares == 50` (`floor(100 × 0.5)`), grant has `vested_at` and
`vested_stock_transaction_id`; delete that stock transaction → grant back to
pending (`vested_at is None`, link cleared), holding gone. This was the
undeletable-grant deadlock.
6. `test_vest_without_cached_price_is_422` — no price row →
`vest_rsu_grant` raises 422 mentioning "No cached price". Never let a vest
invent a price.
### Family E — atomic multi-row writes (`bc256f6`, `c67d34f`)
Pins: `create_split_expense`, `create_splitwise_i_paid`, cross-linking.
1. `test_regress_bc256f6_split_expense_three_rows_linked` — total 100, partner
40: expense `amount == -60.0` typed `expense`; transfer legs −40/+40; the
expense's `linked_transaction_id == from_leg.id`; both transfer legs
cross-linked.
2. `test_regress_3e7f8ec_splitwise_net_share_per_category` — two
`splitwise_i_paid` in one month with different categories (100/45 and 60/30)
→ spending total `55 + 30 == 85.0`, per-category totals 55 and 30. Extends
the existing cli test to the multi-category merge path in
`get_monthly_spending`.
### Family F — funded-buy currency handling (`b3062b7`)
1. `test_regress_b3062b7_usd_funding_account_no_double_conversion` — USD bank
account funds a USD buy, total_cost 500: bank leg `amount == -500.0` exactly
(currency match branch — no division by the rate).
2. `test_funded_buy_zero_rate_guard` — `eur_usd_rate=0` and no
`exchange_rate_used`, EUR funding account: bank amount falls back to
`cash_native` un-divided (the `used_rate > 0` guard). Pins current behavior.
### Gate 2
```bash
python -m coverage run -m pytest cli/tests api/tests -q
python -m coverage report --include='*/api/app/*' -m
```
- **Test count**: record it; expected ≈ 25 + 20–28 new.
- **Coverage floor**: `crud.py` ≥ **75%**, and `coverage report -m` shows no
crud function with an entirely-red body except the RSU-schedule helpers
(`_generate_rsu_schedule` / `_add_months` — Phase 3/4 material) and pure
getters you consciously deferred. List any deferred function by name in the
commit body.
- Branch: a regression test that passes without any product-code change is
correct here (the bugs are already fixed — you are pinning). But if a
regression test FAILS, you have found a live regression: stop the phase,
handle it as a bug through `fin-debugging-playbook` + `fin-change-control`,
and add the incident to `fin-failure-archaeology`.
Land Phase 2 as one change (or one per family if review size demands).
---
## Phase 3 — Invariant/property tests
Encode the ledger's algebraic invariants so *unknown future* bugs in the
families above fail loudly. Derivations for these invariants live in
`fin-accounting-reference` (and, when it lands, `fin-ledger-analysis-toolkit`
— check it before re-deriving).
### Invariants to encode (file: `api/tests/test_properties.py`)
- **I1 — two-leg zero-sum**: every same-currency `transfer`/`settlement` group
written by `create_transfer`/`create_settlement`/`reconcile` sums to 0 across
its legs. Documented exceptions, assert them explicitly instead: cross-
currency transfers (amounts differ by design), `splitwise_pay` pairs
(−total / +partner_share — intentionally non-zero, invariant 6).
- **I2 — balance formula equivalence**: for any bank account,
`calculate_net_worth`'s `balance_native` equals a test-local Python fold of
`sum(t.amount - t.fee)` over its transactions (CLAUDE.md invariant 5,
computed by an independent implementation, not by re-querying the same SQL).
- **I3 — holdings fold**: after any valid random buy/sell sequence,
`recompute_holding_from_stock_transactions` equals a reference weighted-
average fold implemented in the test (~15 lines — keep it independent).
- **I4 — spending invariance under asset movement**: take any month's
`get_monthly_spending` result, then add N random transfers/settlements/
funded buys in that month → `total` and `income_total` unchanged.
- **I5 — delete closure**: after deleting any randomly chosen transaction, no
surviving row's `linked_transaction_id` points at a missing id, and no
account's balance changed except those in the deleted group.
### Solution menu — ranked, with obligations
**Option 1 (preferred): `hypothesis`** — `pip install hypothesis` (dev-only;
do not add to `api/requirements.txt`, which is the Docker image's dependency
list).
Obligations: define an action strategy (`st.sampled_from` over account/txn
constructors with `st.floats(allow_nan=False, allow_infinity=False,
min_value=0.01, max_value=1e6)` rounded to cents); `@settings(deadline=None)`
(SQLite file IO makes per-example deadlines flaky); set
`hypothesis.settings(database=None)` or gitignore `.hypothesis/` so shrink
artifacts never enter the repo; promote every found counterexample to an
explicit `@example(...)` so it reruns forever; `derandomize=True` in the CI
profile (Phase 6) for reproducibility.
**Option 2: hand-rolled random-action generator** — no new dependency.
Obligations you now own manually: seed from `os.environ.get("FIN_TEST_SEED")`
or `random.randrange(2**32)`, ALWAYS print the seed in the failure message
(`assert ok, f"seed={seed}: …"`), document replay
(`FIN_TEST_SEED=<n> python -m pytest …`), and cap runtime (≤200 actions,
≤20 iterations) so the suite stays under a minute. No shrinking — expect ugly
failures.
**Option 3: exhaustive small-case enumeration** — enumerate all sequences of
≤4 actions from a fixed small action set (itertools.product). Deterministic,
dependency-free, but tiny state space; acceptable as a floor for I1/I5 only.
Choose 1 if the environment allows new dev dependencies, else 2. Record the
choice and seed policy in the test module docstring.
### Gate 3
- At least I1–I5 implemented (any option).
- **Prove the oracle bites**: temporarily flip a sign in one reference fold
(e.g. I3's fee handling) and observe the property test fail with a
reproducible seed/example; revert. A property test that never failed during
development is unverified.
- Full suite still green; runtime recorded — if the combined suite now exceeds
~90 s, cut iteration counts, don't delete invariants.
---
## Phase 4 — Services: net worth, price fetcher, scheduler
File: `api/tests/test_services.py` (split later if large). NO live network,
ever: yfinance stays a stub; `httpx.get` gets monkeypatched.
### 4.1 `services/net_worth.py` — synthetic `price_cache` rows
Using the `seed_price` fixture:
- USD cash: bank account (currency USD) with +110 income; seed
`EUR/USD = 1.10` → `balance_eur == 100.0`, `total_eur == 100.0`.
- Investment holding: create via `crud.create_stock_transaction` (buy 10
VWCE @ 100), seed `VWCE = 100 (USD)` and `EUR/USD = 1.25` →
account `balance_native == 1000.0`, `balance_eur == 800.0`.
- GBP-priced holding: seed `TICK = 50` with `currency="GBP"` and
`EUR/GBP = 0.85` → `balance_eur == 10*50/0.85`.
- Missing ticker price → holding contributes 0 (price_map default), no crash.
- `manual_override_balance` on an investment account wins over holdings and is
converted via the account currency.
- Unknown account type: insert an `Account` with `type="weird"` directly via
the db fixture (the schema `Literal` blocks it at the API; crud/services must
still not crash) → account skipped, warning logged.
- Inactive accounts excluded from the total.
### 4.2 `services/price_fetcher.py` — mocked httpx and a rigged yfinance stub
The conftest already made `yfinance` an empty module; give it behavior
per-test: `import yfinance` then `monkeypatch.setattr(yfinance, "Ticker",
FakeTicker, raising=False)` (the module-scope `import yfinance as yf` in
price_fetcher resolves to the same stub object). Build `FakeTicker` with a
`fast_info` namespace (`last_price`, `currency`) and an `info` dict.
**Mandatory obligation: `monkeypatch.setattr(time, "sleep", lambda s: None)`**
(patch `app.services.price_fetcher.time.sleep`) in every retry-path test — the
real schedule sleeps 1+2+4 s per candidate across 9 candidates.
- `fetch_eur_usd` success: monkeypatch `httpx.get` to return a fake response
(`raise_for_status` no-op, `json()` → `{"rates": {"USD": 1.1}}`) → returns
1.1, upserts `EUR/USD` row with source `frankfurter`.
- `fetch_eur_usd` failure with cache: httpx raises; pre-seed `EUR/USD = 1.08` →
returns 1.08, `fetched_at` NOT bumped.
- **`test_regress_81091fc_no_fake_rate_on_cold_failure`**: httpx raises, no
cache → returns `None` and `price_cache` stays EMPTY (the 1.0-fallback bug).
- **`test_regress_bc8c8d5_pence_division`**: FakeTicker reports
`fast_info.currency == "GBp"`, price 12345 → cached price `123.45`, currency
`GBP`.
- Suffix discovery: bare ticker returns no data (`last_price is None`, `info`
without `regularMarketPrice`), `TICK.DE` succeeds → row cached under the
ORIGINAL symbol `TICK`.
- All candidates fail → returns cached value if present, else `0.0`; no row
written.
- `get_tracked_tickers`: union of holdings with shares > 0 and pending grants;
vested grants and zero-share holdings excluded.
### 4.3 `services/scheduler.py` — call the job function directly
Never `create_scheduler(...).start()` in tests. Call
`fetch_prices_and_snapshot(database_url)` with the conftest's throwaway URL
(`os.environ["DATABASE_URL"]`), monkeypatching
`app.services.scheduler.fetch_all_prices` (it is imported into the scheduler
module's namespace — patch it THERE) to `lambda db: 0`:
- One `NetWorthSnapshot` row for `date.today()` with `total_eur` matching
seeded accounts and parseable `breakdown_json`.
- Run twice same day → still exactly one row (update path), values refreshed.
- `create_scheduler` unit check: returns a scheduler with job ids
`{"price_refresh", "daily_snapshot"}` and correct triggers — WITHOUT calling
`.start()`. Note (verified statically): `SQLAlchemyJobStore(url=…)` in
`create_scheduler` touches the DB URL at construction; if this creates an
unwanted `apscheduler_jobs` table in the shared test DB, point it at its own
temp file via a distinct URL argument — the function takes the URL as a
parameter precisely so you can.
### Gate 4
`coverage report --include='*/api/app/services/*' -m` — floors:
`net_worth.py` ≥ **90%**, `price_fetcher.py` ≥ **80%**, `scheduler.py` ≥
**70%** (the `create_scheduler` wiring may stay partially red). No single test
over 2 s (if one is, you forgot the `time.sleep` patch). Full suite green.
---
## Phase 5 — Migration tests: the 8-revision chain
Now `pip install alembic` (first phase that needs it). File:
`api/tests/test_migrations.py`.
Key mechanics (verified statically against `api/alembic/env.py`): env.py
resolves the URL as `DATABASE_URL` env var → `sqlalchemy.url` → app default,
and inserts the api dir into `sys.path` itself. `fileConfig` is skipped when
the Config has no file. So a programmatic Config works from any cwd:
```python
import tempfile
from pathlib import Path
import sqlalchemy as sa
from alembic import command
from alembic.config import Config
REPO_ROOT = Path(__file__).resolve().parents[2]
def _migrated_engine(monkeypatch, tmp_path):
url = f"sqlite:///{tmp_path}/mig.db"
# env.py reads DATABASE_URL FIRST — without this the conftest's shared
# test-DB URL would be migrated instead. Always a fresh file per test.
monkeypatch.setenv("DATABASE_URL", url)
cfg = Config()
cfg.set_main_option("script_location", str(REPO_ROOT / "api" / "alembic"))
command.upgrade(cfg, "head")
return sa.create_engine(url)
```
Tests:
1. `test_chain_is_linear_with_8_revisions` — via
`alembic.script.ScriptDirectory(str(REPO_ROOT / "api" / "alembic"))`:
exactly one head, id `h8c9d0e1f2a3`, walk from base counts **8** revisions,
no branchpoints. (If this fails because a NEW revision exists: update the
expected head/count AND the table in `fin-change-control` §4.2 — that is
maintenance, not a bug.)
2. `test_empty_to_head_matches_metadata` — migrate a fresh temp DB, then
compare against `Base.metadata`: build
`{table: sorted(column names)}` from `sa.inspect(engine)` (excluding
`alembic_version`) and from `Base.metadata.tables`; assert equal. **First
run**: record any diff verbatim. Expected: empty. If NOT empty, you have
found model/migration drift — the exact failure mode `fin-change-control`
§4.3 warns about (`create_all` hides it in every other test). STOP, file it
as a schema-change fix through change control; only allowlist a difference
with an inline comment naming the reason.
3. `test_downgrade_one_runs` — after upgrade, `command.downgrade(cfg, "-1")`
then `command.upgrade(cfg, "head")` completes. This asserts the downgrade
*runs*, not that it restores data (`h8c9d0e1f2a3`'s downgrade is an honest
no-op by design).
4. Stretch (optional, document if skipped):
`alembic.autogenerate.compare_metadata` against the migrated DB expecting
`[]` — catches type/index drift the name-comparison misses, but is noisier
on SQLite; adopt only if the noise is manageable.
Gate 5: all migration tests green; the recorded schema diff is empty or each
entry is allowlisted-with-reason. Full suite still green (migration tests must
not have touched the shared conftest DB — if `cli/tests` now fails with an
`alembic_version` artifact, a test forgot the `monkeypatch.setenv`).
---
## Phase 6 — CI
Checked first, per protocol: `git remote -v` → sole remote is
`https://gitea.cloud.traunseenet.com/markuseisl/fin.git` — **self-hosted
Gitea** (observed 2026-07-05). GitHub Actions does not apply.
Hard constraints for ANY option: no docker required, no live yfinance or
frankfurter calls (the suite is already offline-by-construction — keep it
that way), total runtime target < 3 minutes.
### Menu — ranked
**Option 1: Gitea Actions** (GitHub-Actions-compatible syntax). Preconditions
you must VERIFY before choosing it — do not assume: (a) Actions is enabled on
the instance and repo (repo Settings → Actions, or ask the owner — it is
off by default on older Gitea), (b) an `act_runner` is registered. If either
fails → Option 2. Workflow `.gitea/workflows/tests.yml`:
```yaml
name: tests
on: [push]
jobs:
test:
runs-on: ubuntu-latest
container: python:3.12-slim
steps:
- uses: actions/checkout@v4
- run: pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0" coverage alembic hypothesis
- run: pip install -e cli/
- run: python -m coverage run -m pytest cli/tests api/tests -q
- run: python -m coverage report --include='*/api/app/*' --fail-under=80
```
(Verified statically — could not be executed from the sandbox; the dependency
list mirrors Phase 0/5. Adjust `hypothesis` to your Phase 3 choice. If the
runner lacks internet for pip, bake a runner image — that is
`fin-run-and-operate` territory.)
Obligations: pin `--fail-under` to your actual Gate-6 floor, not aspiration;
add the badge/URL to README only once it has passed on a real push.
**Option 2: committed pre-push hook** (works with zero server support).
`scripts/git-hooks/pre-push`:
```bash
#!/bin/sh
set -e
python -m pytest cli/tests api/tests -q
```
plus a one-time install command documented in README:
`git config core.hooksPath scripts/git-hooks`. Obligations: keep it under a
minute; document that `git push --no-verify` bypasses it (an emergency valve,
not a habit); this protects only machines that ran the config line.
**Option 3: documented gate only** — the `fin-change-control` §7 checklist
already mandates the pytest run; add the combined
`cli/tests api/tests` command there and to CLAUDE.md. Weakest — no automation
— acceptable only as an interim while Option 1's preconditions are pending.
Gate 6: state which option you chose and WHY (what you found when verifying
Option 1's preconditions); demonstrate one real failing run blocking a push
(introduce a deliberate test failure on a scratch branch, observe the block,
revert). Record the evidence in the commit message.
---
## Promotion protocol (routes through fin-change-control)
1. **One phase = one reviewed change** (Phase 2 may split by family). Never
bundle test phases with product-code fixes — if a phase uncovers a product
bug, the fix is a separate change with its own class and gate.
2. Classification per `fin-change-control` §1: Phases 1–5 are tests-only →
behavior-change gate (compile + full suite green; the tests are their own
coverage). Phase 6 adds repo config → same, plus README update for the hook
install line. Nothing here is deploy-affecting: **no phase may touch
`api/requirements.txt`, Dockerfiles, or compose** (dev-only deps like
coverage/hypothesis/alembic-for-tests live in the venv / CI config, not the
image).
3. Every phase commit body records the gate numbers: test count before/after,
coverage table delta, runtime. Commit style per `fin-change-control` §6.
4. **CLAUDE.md** gains a single test-run line
(`python -m pytest cli/tests api/tests -q`) only after Phase 2 has landed
and stayed green across at least one subsequent unrelated change — not
before (stability first, advertisement second). Update
`fin-validation-and-qa` §2's inventory pointers at the same time.
5. Campaign completion floors (measure, don't eyeball): `crud.py` ≥ **90%**,
`api/app` total ≥ **80%**, every CLAUDE.md invariant (1, 2, 5, 6) named in
at least one test docstring, migration chain tested, CI or hook active.
Record the final coverage table in the last phase's commit.
## Fenced-off wrong paths — do not do these
- **Async test rewrite** (pytest-asyncio, AsyncSession, httpx.AsyncClient
against the app): the engine is synchronous by design (CLAUDE.md). Any
"modernization" here is architecture change, not testing.
- **Pointing any test at the prod DB or a copy of it inside the test tree**:
prod `/data/fin.db` is sacred (`fin-change-control` §2). Throwaway
temp-file SQLite only, created by the conftest pattern above.
- **Live yfinance/frankfurter in tests or CI**: the yfinance stub and httpx
monkeypatching are load-bearing. A test that "just fetches one real price"
makes the suite flaky, slow, and network-dependent — the exact failure CI
must never have.
- **Editing `cli/tests/conftest.py`'s sys.path/env bootstrapping casually**:
the set-DATABASE_URL-before-import dance is what keeps real DBs out of
reach. Extend by copying the pattern (Phase 1), never by "cleaning it up".
- **`Base.metadata.create_all` as a migration-test substitute**: it creates a
schema that matches the models by construction and proves nothing about the
migration chain — the drift it hides is the point of Phase 5
(`fin-change-control` §4.3).
- **Refactoring crud.py "for testability" before characterization tests
exist**: no extracting pure functions, no dependency injection, no splitting
the file until Phase 2's floor is met. The current shape is testable as-is
(the `db` fixture proves it); refactors before pinning behavior are how
accounting semantics silently change.
- **Chasing 100%**: the last percent (defensive `except` arms in
`fetch_all_prices`, log-only branches) buys nothing. The floors above are
the contract; spend surplus effort on Phase 3 invariants instead.
- **Skipping/xfailing a red test to make a gate**: with no CI history, a
skipped test is a deleted test (`fin-validation-and-qa` §5: fix or delete,
never skip).
## Provenance and maintenance
Authored 2026-07-05 against HEAD `082b608`; the authoring sandbox had no
pip/venv/docker, so all run-commands are verified statically (files read in
full: `api/app/crud.py`, `cli/tests/conftest.py`, `cli/tests/test_cli.py`,
`api/app/services/{net_worth,price_fetcher,scheduler}.py`,
`api/app/routers/transactions.py`, `api/app/{main,database}.py`,
`api/app/schemas.py`, `api/alembic/env.py`; sibling skills: change-control,
failure-archaeology, validation-and-qa, build-and-env).
Recount the volatile facts before trusting them:
```bash
grep -cE '^def ' api/app/crud.py # 35 defs (33 public) as of 2026-07-05
wc -l api/app/crud.py # 995
grep -c 'def test_' cli/tests/test_cli.py # 25
ls api/tests/ 2>/dev/null # campaign progress: which phases landed
grep -rc 'def test_' api/tests/ 2>/dev/null # api-suite size
ls api/alembic/versions/ # 8 revisions, head h8c9d0e1f2a3
git remote -v # Gitea (Phase 6 premise)
ls .gitea/workflows scripts/git-hooks 2>/dev/null # which Phase-6 option landed
git log --oneline -5 # HEAD moved? re-audit archaeology for new families
```
Maintenance rules: when a phase lands, update the ground-truth counts and mark
the phase done in this file (same change). When a new bug is fixed anywhere in
fin, add its regression test using the Phase 2 naming scheme and register the
incident in `fin-failure-archaeology` — the campaign's steady state is that
those two updates always travel together. If `crud.py`'s function inventory
changes, regenerate the area table with
`grep -nE '^def |^# ---' api/app/crud.py` and re-derive the
exercised/never-executed split from the current test suites, not from memory.
@@ -0,0 +1,280 @@
---
name: fin-validation-and-qa
description: >
Load BEFORE claiming any change to fin works, when writing or extending tests
in cli/tests/, when deciding what evidence a fix or feature needs before it
can be called done, or when reviewing someone else's change. Defines the
evidence hierarchy (test > diagnostics numbers > fin --json > compile > eyeball),
the golden inventory of what the e2e suite actually certifies, how to run and
add tests safely (prod DB never a fixture), and the hypothesis-predicts-numbers
discipline for accepting a fix.
---
# fin — Validation and QA
Steady-state quality discipline for the fin repo. The only test suite is
`cli/tests/` (e2e: `FinClient` → FastAPI `TestClient` → real `crud.py` →
throwaway SQLite). There is NO CI — the green run on your machine IS the gate.
The prod DB is sacred: no test, experiment, or "quick check" ever touches it.
Read `/home/markuseisl/Documents/git/fin/CLAUDE.md` first if you haven't.
## When NOT to use this skill
| You want to… | Go to |
|---|---|
| Use the `fin` CLI against the live API | `fin` |
| Expand test coverage as a campaign (direct crud.py tests, new layers) | `fin-test-coverage-campaign` |
| Set up Python/pytest environment, install deps | `fin-build-and-env` |
| Run the audit/diagnostics scripts themselves | `fin-diagnostics-and-tooling` |
| Decide whether a change is allowed at all / merge gates | `fin-change-control` |
| Debug a live misbehavior (not validate a fix) | `fin-debugging-playbook` |
| Look up correct accounting math for expected values | `fin-accounting-reference` |
| Check whether a bug class has happened before | `fin-failure-archaeology` |
## 1. Evidence hierarchy (strongest → weakest)
When you claim "X is fixed" or "X works", state which level your evidence is.
| Level | Evidence | Counts as proof for |
|---|---|---|
| (a) | Automated test in `cli/tests/` that **fails before** the change and **passes after** | Anything. The gold standard. |
| (b) | Diagnostics-script numbers before/after, run on a **synthetic or copied** DB (see `fin-diagnostics-and-tooling`) | Accounting/balance logic when a test is genuinely impractical |
| (c) | `fin --json` API-level checks against a throwaway/dev instance, with expected values written down first | Wiring, endpoint behavior, CLI plumbing |
| (d) | `python -m py_compile …` / `tsc` via `vite build` passes | Nothing works — only that it parses/typechecks |
| (e) | Manual UI walkthrough | Rendering and UX only |
**Hard rule: accounting-logic changes (balances, spending, transfers,
settlements, splits, holdings, net worth) require level (a) or (b). Never
eyeball a balance.**
### Hypothesis-predicts-numbers discipline
Before running ANY check (test, script, `fin --json` call):
1. Write down the exact number/output you expect and WHY (one sentence of mechanism).
2. Run the check.
3. If the number differs — even in a direction that "looks fixed" — the
hypothesis is wrong or incomplete. Do not accept the fix.
4. A fix is accepted only when **one mechanism explains ALL observations,
including the negatives** (the checks that were expected to stay unchanged
and did). A fix that explains the failing case but not why case Y was
unaffected is a guess, not a fix.
## 2. Golden inventory — what a green run actually certifies
Source of truth: `/home/markuseisl/Documents/git/fin/cli/tests/test_cli.py`.
**25 test functions as of 2026-07-05** (9 auth, 2 accounts, 2 transactions,
3 transfer, 1 settlement, 1 split-expense, 1 splitwise-paid, 2 net-worth,
1 spending, 3 CLI wiring). Re-count before trusting this table (see Provenance).
### Certified (locked by a test)
| Area | Test | Invariant it locks |
|---|---|---|
| Auth | `TestAuth::test_auth_off_allows_requests` | No `FIN_API_TOKEN` env → API open |
| Auth | `test_health_ok`, `test_health_bypasses_auth` | `/api/health` always open, even with auth on |
| Auth | `test_auth_on_rejects_missing_token`, `test_auth_on_rejects_wrong_token` | Auth on → 401 without/with wrong bearer token |
| Auth | `test_auth_on_accepts_correct_token` | Correct token passes |
| Auth | `test_bearer_scheme_is_case_insensitive` | `bearer` == `Bearer` |
| Auth | `test_openapi_gated_when_auth_on`, `test_openapi_open_when_auth_off` | OpenAPI spec follows the auth gate |
| Accounts | `TestAccounts::test_create_and_list` | Create/list roundtrip |
| Accounts | `test_initial_balance_creates_opening_transaction` | `initial_balance` → one opening `income` txn of that amount |
| Transactions | `TestTransactions::test_add_and_list_with_limit` | List is newest-first; `limit` respected |
| Transactions | `test_limit_must_be_positive` | `limit=0` → 422 |
| Transfer | `TestTransfer::test_two_legs_created_and_linked` | Two legs, ±amount, cross-linked via `linked_account_id` (CLAUDE.md invariant 1) |
| Transfer | `test_deleting_one_leg_removes_whole_group` | Deleting either leg deletes both — no orphan legs |
| Transfer | `test_editing_one_leg_is_rejected` | Editing one transfer leg → 422 |
| Settlement | `TestSettlement::test_bank_to_splitwise_two_legs_linked` | bank→splitwise: two linked `settlement` legs, ±amount |
| Split expense | `TestSplitExpense::test_three_rows_and_user_share_spending` | total 100/partner 40 → expense −60 + transfer ±40 (3 rows); spending counts only the 60 |
| Splitwise paid | `TestSplitwisePaid::test_spending_counts_net_share` | bank −100, `splitwise_partner_share`=45, splitwise leg +45; spending = 55 (invariant 6) |
| Net worth | `TestNetWorth::test_current_reflects_bank_balance` | Cash balance flows into `total_eur` |
| Net worth | `test_fee_reduces_balance` | balance = SUM(amount − fee) (invariant 5): 100 − 50 − 2 = 48 |
| Spending | `TestSpending::test_monthly_totals` | Monthly total + per-category breakdown |
| CLI wiring | `TestCliWiring::test_accounts_list_json` | `fin accounts list --json` emits parseable JSON |
| CLI wiring | `test_error_exits_nonzero` | API error → exit code 1 |
| CLI wiring | `test_tx_add_refuses_transfer_type` | `tx add --type transfer` rejected client-side, nothing written |
### NOT certified (a green run says nothing about these)
| Area | Gap (as of 2026-07-05) |
|---|---|
| **Stocks/holdings** | Zero tests. `create_stock_transaction`, `delete_stock_transaction`, `recompute_holding_from_stock_transactions` (CLAUDE.md **invariant 2 is untested**), `create_funded_buy` |
| **RSU** | Zero tests: grants, vest, schedule preview/generate, vest→holding linkage |
| crud.py direct | No unit tests at all (~995 lines). Only paths reachable via the API calls above are exercised — `fin-test-coverage-campaign` owns closing this |
| Mutations | `update_account`, `deactivate_account`, balance adjust/`reconcile`, category update/delete, `update_transaction` on plain txns, `delete_transaction` of a plain txn |
| Settlement | `splitwise_to_bank` direction untested (only `bank_to_splitwise`) |
| Spending | `get_spending_summary` (multi-month), spending with transfers/settlements/stock trades mixed in a month (the e36d719 bug class) |
| Net worth | `services/net_worth.py` with holdings + `price_cache` (only pure-cash paths hit), history/snapshots, non-EUR accounts, FX conversion |
| Prices | `price_fetcher.py`, `/api/prices/*` — yfinance is a stub module in tests; nothing about fetching is certified |
| Scheduler | `services/scheduler.py` never starts in tests; midnight job untested |
| Routers | Error mapping beyond the handful of 422/404 cases above |
| **Frontend** | Zero automated tests of any kind — see §6 |
| CLI | Most subcommands beyond the 3 wiring smoke tests; all human (non-`--json`) output |
Consult this map before trusting "tests pass" as evidence for your specific change.
## 3. Running the tests
Commands are statically verified against `conftest.py`/`pyproject.toml` — this
authoring environment has no pip/pytest, so they were not executed here.
```bash
cd /home/markuseisl/Documents/git/fin
python -m pytest cli/tests -q
```
- Working directory does not matter for imports: `conftest.py` computes
`REPO_ROOT` from its own path and inserts `api/` and `cli/` into `sys.path`.
Repo root is still the conventional place to run from.
- `conftest.py` sets `DATABASE_URL` to `sqlite:///<mkdtemp>/test.db` **before**
importing `app.*`, and stubs `yfinance` as an empty module. Never import
`app.database` in a new conftest/helper before that env var is set.
- Green run as of 2026-07-05: `25 passed` (dots, no warnings-as-errors config).
- Requires `pytest`, `fastapi`, plus the api/ and cli/ dependencies installed.
Environment failures (missing modules, wrong Python) → `fin-build-and-env`.
- The suite never needs `FIN_API_URL`/`FIN_API_TOKEN`; the autouse fixture
deletes `FIN_API_TOKEN` so a token in your shell can't poison auth tests.
## 4. Adding a test — checklist and skeleton
Where: a new `test_*.py` in `/home/markuseisl/Documents/git/fin/cli/tests/`
(or a new class in `test_cli.py` if it fits an existing area). pytest discovers
it automatically; `conftest.py` fixtures apply.
Checklist:
- [ ] Use the `client` fixture (a `FinClient` over in-process `TestClient`) or
`make_client(token=...)` when testing auth behavior.
- [ ] Rely on the autouse `clean_db` fixture: it `drop_all` + `create_all`s the
throwaway DB before **every** test — tests start empty and must not
depend on each other's data or ordering.
- [ ] Exercise multi-step flows through the API surface only: create account →
transact → assert balances via `client.get_net_worth()` /
`client.list_transactions()` / `client.get_spending_month()`.
- [ ] Do NOT import from `app.*` internals when the API surface suffices —
these are e2e tests of the deployed contract. (Direct crud-layer tests
are a different layer with different rules; that belongs to
`fin-test-coverage-campaign`.)
- [ ] No live prices, ever: `yfinance` is a stub. A test that triggers a real
fetch (e.g. `client.refresh_prices()` reaching yfinance) will crash or
hang — if you need a price, insert into `price_cache` via whatever
seeding mechanism the campaign skill establishes, or don't test it here.
- [ ] Write the expected numbers as literals with a comment stating the
mechanism (see `test_fee_reduces_balance` citing invariant 5).
- [ ] Assert on `FinClientError.status_code` for expected failures.
Worked skeleton (multi-step flow):
```python
import pytest
from fin_cli.client import FinClientError
class TestSettlementSplitwiseToBank:
def test_splitwise_to_bank_direction(self, client):
bank = client.create_account(
{"name": "Giro", "type": "bank", "currency": "EUR",
"initial_balance": 100.0}
)
sw = client.create_account(
{"name": "Splitwise", "type": "splitwise", "currency": "EUR"}
)
legs = client.create_settlement(
{"bank_account_id": bank["id"], "splitwise_account_id": sw["id"],
"amount": 30.0, "date": "2026-06-20",
"direction": "splitwise_to_bank"}
)
by_account = {leg["account_id"]: leg for leg in legs}
# money moves toward the bank: bank +30, splitwise -30
assert by_account[bank["id"]]["amount"] == 30.0
assert by_account[sw["id"]]["amount"] == -30.0
# settlements are not spending
assert client.get_spending_month("2026-06")["total"] == pytest.approx(0.0)
# net worth: 100 opening + 30 in - 30 out across own accounts = 100
assert client.get_net_worth()["total_eur"] == pytest.approx(100.0)
```
(Predict the numbers in comments BEFORE running — §1 discipline. Verify the
expected sign convention against the router/crud before committing; the
skeleton's ±30 is the hypothesis, the first run is the experiment.)
## 5. Acceptance thresholds and regression discipline
- **Every fixed bug gains a test that reproduces it** — written to fail on the
pre-fix code. History shows why: the bug-hunt commits (`bc256f6`, `e36d719`)
fixed accounting bugs without adding regression tests, and bugs recurred in
families (spending-inflation class hit twice). Details: `fin-failure-archaeology`.
- Tests must be **deterministic**: no network (yfinance stubbed, no httpx to
the outside), no clock assumptions beyond the app's own local-date helpers —
pass explicit `date` strings, never "today", when asserting on month buckets.
- **Prod DB is never a fixture.** Do not point `DATABASE_URL` or `FIN_API_URL`
at the live instance from a test, and never copy `/data/fin.db` into the test
tree. Experiments on real data happen on a copied DB via
`fin-diagnostics-and-tooling` / `fin-ledger-analysis-toolkit`, read-only.
- A change that touches crud.py accounting paths and ships with only level
(c)–(e) evidence is not done. Flag it in review.
- No flaky-test tolerance: with 25 tests and no CI, a single intermittent
failure erodes the only gate the project has. Fix or delete, never skip.
## 6. Frontend validation — honest current bar
**Zero frontend tests exist (as of 2026-07-05).** The bar is:
1. `vite build` (runs `tsc`) must pass — level (d) evidence only.
2. Manual walkthrough of the critical flows — level (e).
Manual QA script for a frontend-touching change (worked numbers; exact
accounting rules in `fin-accounting-reference`). Do this on a **dev/throwaway
stack**, not prod. Start state: one bank account "Giro" with balance 1,000.00 €.
| # | Action (QuickAdd unless noted) | Expected result |
|---|---|---|
| 1 | Expense −50.00, fee 2.00, category Dining | Giro 948.00 (Δ −52: amount − fee) |
| 2 | Spending page, current month | Total 50.00, Dining 50.00 (fee excluded from spending) |
| 3 | Create second bank account "Savings" (0.00) | Net worth unchanged: 948.00 |
| 4 | Transfer 100.00 Giro → Savings | Giro 848.00, Savings 100.00, net worth still 948.00, two linked rows on Transactions page |
| 5 | Split expense: total 100.00, partner share 40.00, receivable acct | Giro 748.00 (−100), Receivable +40.00, Spending +60.00 → 110.00, net worth 888.00 |
| 6 | Splitwise "I paid": total 100.00, partner share 45.00 | Giro 648.00, Splitwise +45.00, Spending +55.00 → 165.00, net worth 833.00 |
| 7 | Portfolio: buy 2 AAPL @ 100.00 USD on investment acct | Holding row AAPL qty 2, avg cost 100.00 USD; cash leg only if funded-buy used |
| 8 | RSU: vest a grant tranche | Holding appears/increments on the rsu account; grant's remaining unvested decreases |
| 9 | Refresh prices (Settings) then Dashboard | Portfolio EUR value = qty × cached USD price ÷ EUR/USD rate; no NaN/0 flash |
| 10 | Each add above fired the refresh event | Dashboard/Transactions updated without manual reload (`fin:transaction-added` bus intact) |
Any delta mismatch = stop, apply §1: write the mechanism you expected, find
the one mechanism explaining all ten observations.
## 7. Idea lifecycle (methodology-lite)
Every "I think X is broken / X would fix it" follows this pipeline — no
skipping stages:
1. **Hunch** — state it in one falsifiable sentence.
2. **Discriminating check defined** — pick the observation that distinguishes
your hypothesis from the nearest alternative, and write the predicted
numbers down first.
3. **Experiment** — run it on a synthetic DB (test fixtures) or a copied DB
(`fin-diagnostics-and-tooling`). Never prod.
4. **Test written** — the discriminating check becomes a `cli/tests/` test
(fails pre-fix, passes post-fix).
5. **Change lands** — through `fin-change-control` gates.
6. **Archive** — if the investigation settled a question about how fin behaves
(or used to misbehave), record it in `fin-failure-archaeology` so the next
session doesn't re-derive it.
## Provenance and maintenance
Ground truth as of 2026-07-05. Re-verify before relying on volatile facts:
- Test count / inventory: `grep -c "def test_" cli/tests/test_cli.py` (25) and
`ls cli/tests/` (only `conftest.py` + `test_cli.py`) — re-read new files and
update §2 tables when either changes.
- Fixture semantics: re-read `cli/tests/conftest.py` (autouse `clean_db`,
`make_client`, yfinance stub, `DATABASE_URL` set pre-import).
- Run command: confirm against `cli/pyproject.toml` and `fin-build-and-env`
(commands here were statically derived, not executed — no pip in the
authoring sandbox).
- Frontend still testless: `ls web/src/**/*.test.* web/vitest* 2>/dev/null`
should return nothing; if it doesn't, §6 is stale.
- Coverage gaps: cross-check §2 "NOT certified" against
`fin-test-coverage-campaign` progress before repeating its claims.
+2 -1
View File
@@ -2,4 +2,5 @@
**/__pycache__/ **/__pycache__/
api/fin.db api/fin.db
web/dist/ web/dist/
web/node_modules/ web/node_modules/
.claude/settings.local.json