Compare commits
5
Commits
082b6081b7
...
631bdbb13f
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
631bdbb13f | ||
|
|
cb97d97d81 | ||
|
|
c9a1375fb9 | ||
|
|
3eb5fdbe55 | ||
|
|
439a15264b |
No files matched your search
@@ -0,0 +1,36 @@
|
||||
# fin skill library — index
|
||||
|
||||
13 skills, authored 2026-07-05 against commit `082b608`. Each SKILL.md is self-contained, ends with a "Provenance and maintenance" section (re-verification one-liners), and states when NOT to use it. Code beats skills; skills beat memory.
|
||||
|
||||
## Task → skill map
|
||||
|
||||
| You are about to… | Load |
|
||||
|---|---|
|
||||
| Query/record finances via the `fin` CLI | **fin** |
|
||||
| Change any code, schema, or doc — find the gate | **fin-change-control** |
|
||||
| Debug "a number looks wrong" / triage any failure | **fin-debugging-playbook** |
|
||||
| Fix something that looks like an obvious bug | **fin-failure-archaeology** first (is it settled?) |
|
||||
| Design/refactor; add endpoint, table, account type | **fin-architecture-contract** |
|
||||
| Reason about money: types, spending math, FX, RSU | **fin-accounting-reference** |
|
||||
| Set up an environment, run tests, build images | **fin-build-and-env** |
|
||||
| Deploy, back up, configure env vars, operate prod | **fin-run-and-operate** |
|
||||
| Measure data integrity; before/after evidence | **fin-diagnostics-and-tooling** (scripts/) |
|
||||
| Claim a change works; write/extend e2e tests | **fin-validation-and-qa** |
|
||||
| Expand test coverage / add CI (the campaign) | **fin-test-coverage-campaign** |
|
||||
| Prove an accounting property; predict a delta | **fin-ledger-analysis-toolkit** |
|
||||
| Choose the next project / roadmap | **fin-research-frontier** |
|
||||
|
||||
## Ownership (one home per fact)
|
||||
|
||||
- Backup/restore runbook → fin-run-and-operate §3 (others cross-reference).
|
||||
- Accounting formulas/definitions → fin-accounting-reference; proof methods on top → fin-ledger-analysis-toolkit.
|
||||
- Invariant catalog incl. unenforced gaps (U1–U8) → fin-ledger-analysis-toolkit.
|
||||
- Test inventory (certified/not) → fin-validation-and-qa §2; coverage expansion → fin-test-coverage-campaign.
|
||||
- Settled design questions & incident chronicle → fin-failure-archaeology.
|
||||
- Config catalog → fin-run-and-operate §2.
|
||||
|
||||
## Standing rules baked into every skill
|
||||
|
||||
1. The prod SQLite DB (volume `sqlite_data`, `/data/fin.db`) is sacred: never test against it, back up before migrations.
|
||||
2. Behavior changes route through fin-change-control's gates; accounting claims need level-(a)/(b) evidence (fin-validation-and-qa).
|
||||
3. Anything date-stamped "as of 2026-07-05" may have drifted — run the skill's provenance one-liners before relying on it.
|
||||
@@ -0,0 +1,178 @@
|
||||
---
|
||||
name: fin-accounting-reference
|
||||
description: The complete semantic model of money in the fin app. Load whenever reasoning about amounts, balances, spending or income math, currencies and FX conversion, transaction types, transfers/settlements, Splitwise shares, receivables/payables, stock holdings, RSU grants and vesting, or net worth — and BEFORE writing or reviewing any accounting logic in api/app/crud.py, api/app/services/net_worth.py, routers, the CLI, or QuickAdd flows.
|
||||
---
|
||||
|
||||
# fin Accounting Reference (domain theory)
|
||||
|
||||
Every formula below is derived from the code as of 2026-07-05, with `file:function` citations. Code is the truth; if this document and the code disagree, trust the code and update this file (see "Provenance and maintenance").
|
||||
|
||||
## 1. Units, signs, and columns
|
||||
|
||||
- **`amount`**: positive = money into the account, negative = money out. Always in the transaction's `currency` (model comment in `api/app/models.py:Transaction`, PLAN.md data model).
|
||||
- **`fee`**: a **positive** number representing an extra cost, stored separately from `amount`. Default 0.
|
||||
- **Balance formula** (bank / splitwise / receivable / payable accounts): `balance = SUM(amount − fee)` over all of the account's transactions. Source: `api/app/services/net_worth.py:calculate_net_worth` (the `balance_rows` query: `func.sum(Transaction.amount - Transaction.fee)`). Never sum `amount` alone (CLAUDE.md invariant 5).
|
||||
- Worked example: expense of €50 with a €2 fee → `amount = -50`, `fee = 2` → balance impact = `-50 − 2 = -52`.
|
||||
- On transfers, the fee sits on the **source leg only**; the destination leg is always written with `fee = 0` (`crud.py:create_transfer`). So a €500 transfer with €1 fee debits `-501` net and credits `+500`.
|
||||
- **Currency**: each `Transaction` carries its own `currency` (default `"EUR"`), and each `Account` has a declared `currency`. Balances are raw sums with **no per-transaction FX conversion**; net worth converts the summed native balance using the **account's** currency (`net_worth.py`). Consequence: a transaction's `currency` must match its account's currency or the balance is silently wrong — cross-currency transfers handle this via `to_amount`/`to_currency` (§3).
|
||||
- **`exchange_rate_used`** (nullable, on `transactions` and `stock_transactions`): an audit column recording the EUR/USD rate in effect. It is only *used* computationally in `crud.py:create_funded_buy`, which converts the cash leg with `bank_amount = cash_native / used_rate` when the funding account's currency differs from the stock's currency (`used_rate = req.exchange_rate_used or eur_usd_rate`). Everywhere else it is informational.
|
||||
- **Opening balance**: `crud.py:create_account` with `initial_balance > 0` writes an `income` transaction with `description = "Opening balance"`. That exact description string is what excludes it from income totals (§4) — do not rename it.
|
||||
|
||||
## 2. Transaction type semantics
|
||||
|
||||
All six types from `models.py:Transaction` / `schemas.py:TransactionBase`. "Two-leg" means two rows linked via `linked_account_id` + `linked_transaction_id`.
|
||||
|
||||
| Type | Meaning | Sign pattern | In spending? | In income? | Two-leg? | Created by |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `expense` | Consumption (incl. debt to a person via a payable account) | negative | Yes (raw amount) | No | No — except as part of a split-expense triple | QuickAdd ExpenseFlow, IOweFlow; `fin tx add --type expense`; split-expense endpoint (its expense row) |
|
||||
| `income` | Earnings; also opening balances | positive | No | Yes (except `description = 'Opening balance'`) | No | QuickAdd IncomeFlow; `fin tx add --type income`; `crud.py:create_account` |
|
||||
| `transfer` | Asset movement between own accounts (incl. lending money = bank→receivable, and funded-buy cash legs) | source −, destination + | Never | Never | Yes | QuickAdd TransferFlow, IAmOwedFlow; `fin transfer`; `POST /api/transactions/transfer`; `crud.py:create_funded_buy` cash leg; split-expense partner legs |
|
||||
| `splitwise_pay` | I paid a shared bill; partner owes their share | bank leg − (full bill), splitwise leg + (partner share) | Yes — bank leg only, at net share `amount + splitwise_partner_share` | No | Yes | QuickAdd SplitwiseIPaidFlow; `fin splitwise-paid`; `POST /api/transactions/splitwise-i-paid` |
|
||||
| `splitwise_receive` | Partner paid; I owe my share | single negative row on the splitwise account | Yes (raw amount) | No | No | QuickAdd SplitwiseShePayedFlow; `fin tx add --type splitwise_receive` |
|
||||
| `settlement` | Clearing a splitwise/receivable/payable balance against a bank account | one leg +, one leg − (opposite accounts) | Never | Never | Yes | QuickAdd SplitwiseSettleFlow; `fin settle`; `fin reconcile`; `POST /api/transactions/settlement` and `/reconcile` |
|
||||
|
||||
History note: funded-buy cash legs used to be typed `expense` and reconcile legs `income`/`expense`, which inflated the metrics. Migration `api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py` retyped legacy rows to `transfer`/`settlement`; new rows are written correctly (`crud.py:create_funded_buy` docstring, commit e36d719). Never reintroduce income/expense typing for internal cash movements.
|
||||
|
||||
`fin tx add` deliberately only accepts `expense`, `income`, `splitwise_pay`, `splitwise_receive` (`cli/fin_cli/tx.py:TX_TYPES`) — single-row transfers/settlements corrupt balances; use the paired commands.
|
||||
|
||||
## 3. Two-leg (double-entry) mechanics
|
||||
|
||||
All constructs below are written atomically: `db.add_all([...])`, `db.flush()` to get ids, cross-set `linked_transaction_id`, single `db.commit()` (CLAUDE.md invariant 1).
|
||||
|
||||
| Construct | Function | Legs (account: amount, type) | Invariant preserved |
|
||||
|---|---|---|---|
|
||||
| Transfer | `crud.py:create_transfer` | from: `−amount` (fee=fee); to: `+to_amount ?? +amount` (fee=0) — both `transfer` | Same-currency: net worth unchanged minus fee. Cross-currency: caller supplies `to_amount`/`to_currency` (see `schemas.py:TransferRequest`); the two legs need not sum to zero in nominal terms |
|
||||
| Splitwise settlement | `crud.py:create_settlement` | direction `splitwise_to_bank`: bank `+amount`, splitwise `−amount`; `bank_to_splitwise`: reversed — both `settlement` | Cash ↔ receivable swap: net worth delta = 0 |
|
||||
| Split expense (3 rows) | `crud.py:create_split_expense` | bank: `−(total − partner_share)` `expense`; bank: `−partner_share` `transfer`; receivable: `+partner_share` `transfer`. Expense is linked into the group via `expense.linked_transaction_id = from_leg.id` | Bank debited the full total; only your share is spending; partner share becomes a receivable asset. Net worth delta = −your share |
|
||||
| Splitwise "I paid" | `crud.py:create_splitwise_i_paid` | bank: `−total` `splitwise_pay` with `splitwise_partner_share = partner_share`; splitwise: `+partner_share` `splitwise_pay` | Net worth delta = −your share (PLAN.md Flow A) |
|
||||
| Reconcile | `routers/transactions.py:reconcile_accounts` | receivable: bank `+amount`, receivable `−amount`; payable: bank `−amount`, payable `+amount` — both `settlement` | Net worth delta = 0; the consumption was already booked when the payable/receivable was created |
|
||||
| Funded buy/sell | `crud.py:create_funded_buy` | StockTransaction + one bank row typed `transfer` (buy: `−total_cost`; sell: `+shares×price − fee`), FX-converted via `used_rate` if currencies differ | Cash ↔ securities swap never touches spending/income |
|
||||
|
||||
**Group delete**: `crud.py:delete_transaction` walks `linked_transaction_id` in both directions to collect the whole group (both legs, plus a split-expense's expense row) and deletes it atomically; a legacy fallback matches on `linked_account_id` + date + type + closest id for pre-`linked_transaction_id` rows (migration `e5f6a7b8c9d0`). Never delete a single leg.
|
||||
|
||||
**Edit lock**: `PUT /api/transactions/{id}` returns 422 for any row with `linked_account_id` set — delete both legs and re-enter (`routers/transactions.py:update_transaction`).
|
||||
|
||||
## 4. Spending & income math
|
||||
|
||||
Source: `crud.py:get_monthly_spending` (per-month) and `crud.py:get_spending_summary` (yearly/custom = per-month loop, then aggregate).
|
||||
|
||||
**Spending** for month `YYYY-MM` = sum of two query sets, grouped by category, reported as positive numbers:
|
||||
|
||||
1. Rows with `type IN ('expense', 'splitwise_receive')`, `amount < 0`, joined to accounts with `Account.type NOT IN ('receivable')` — counted at **raw amount**.
|
||||
2. Rows with `type = 'splitwise_pay'`, `amount < 0`, same account filter — counted at the **user's net share**: `amount + splitwise_partner_share`.
|
||||
|
||||
Worked €100-dinner example (50/50, I paid, via `splitwise-i-paid`):
|
||||
- Rows written: bank `amount = −100`, `splitwise_partner_share = 50`, type `splitwise_pay`; splitwise account `amount = +50`, type `splitwise_pay`.
|
||||
- Spending contribution: `−100 + 50 = −50` → **€50** (the splitwise `+50` leg fails the `amount < 0` filter, so it never double-counts).
|
||||
- Net worth: bank −100, splitwise +50 → **−50**. Consistent.
|
||||
|
||||
If instead the partner paid an €80 bill split 50/50: one `splitwise_receive` row, `amount = −40`, on the splitwise account → spending €40, no cash movement.
|
||||
|
||||
**Income** for the month = `SUM(amount)` where `type = 'income'`, account type `NOT IN ('payable', 'receivable')`, and `description != 'Opening balance'`.
|
||||
|
||||
**Summary** (`get_spending_summary`): `grand_total` = Σ monthly spending, `income_total` = Σ monthly income, `net_total = income_total − grand_total`. Categories with a net non-negative month total are skipped; percentages are share of the month's overall total.
|
||||
|
||||
Two intentional surprises — do not "fix" them:
|
||||
1. **I-Owe counts as spending immediately.** IOweFlow records an `expense` on a **payable** account; commit e61b254 changed the account filter from `NOT IN ('payable','receivable')` to `NOT IN ('receivable')` precisely so payable expenses count when the debt arises. Cash leaves the bank later via reconcile, which is neutral. (Receivable accounts stay excluded — a9f045f — because lending is a transfer, not spending; see `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` §3.)
|
||||
2. **Reconcile/settlement is never income or spending**, even though cash arrives in/leaves the bank. Both legs are `settlement`, which appears in neither query (h8c9d0e1f2a3 migration rationale).
|
||||
|
||||
Also: `transfer` never appears in either aggregate, and category `type` (`expense`/`income`, `models.py:Category`) is a **UI-only** picker filter — the spending queries never join on it (spec doc §1).
|
||||
|
||||
## 5. Net worth
|
||||
|
||||
Source: `api/app/services/net_worth.py:calculate_net_worth`. Only `is_active` accounts are included. Result: `{total_eur, eur_usd_rate, accounts: [{id, name, type, currency, balance_native, balance_eur, is_override}]}`.
|
||||
|
||||
Per account type:
|
||||
- **bank / splitwise / receivable / payable**: `balance_native = manual_override_balance ?? SUM(amount − fee)`; `balance_eur = _to_eur(balance_native, account.currency, price_map)`.
|
||||
- **investment / rsu**: if `manual_override_balance` is set, it is treated as being in the account's declared currency and converted with `_to_eur` (fix in commit 37bc101). Otherwise `Σ over holdings: shares × price_cache[ticker]`, each holding converted using the **price row's** currency (`ccy_map`), not the account's.
|
||||
- Unknown account types are skipped with a warning.
|
||||
|
||||
`_to_eur(amount, ccy, price_map)`: EUR passes through; otherwise divide by `price_map["EUR/<CCY>"]` (rates are quoted as *units of CCY per 1 EUR*, so EUR value = native / rate). **If the rate is missing, the value contributes `0.0` and a warning is logged** — never a wrong-currency conversion (see §8). Two residual 1.0 fallbacks exist when `EUR/USD` is uncached: the top-level `eur_usd_rate` response field, and `net_worth.py:_get_eur_usd_rate`, which `routers/stock_transactions.py` feeds into funded-buy conversion — a documented gap (U8 in fin-ledger-analysis-toolkit), not a pattern to copy.
|
||||
|
||||
**Unvested RSUs are NOT in net worth.** `calculate_net_worth` only reads `holdings`, which only change on vest. The dashboard shows unvested value greyed-out, client-side only (commit babea72, `web/src/pages/Dashboard.tsx`): `Σ gross_shares × (1 − withholding_pct) × current_price`, converted to EUR via the EUR/USD rate (any non-EUR price currency uses EUR/USD there — a dashboard-only approximation).
|
||||
|
||||
**Snapshot**: `services/scheduler.py:fetch_prices_and_snapshot` runs at midnight — fetches prices, computes net worth, upserts one `net_worth_snapshots` row per calendar day with `total_eur` and `breakdown_json` = the JSON-serialized `accounts` list. A separate 5-minute job refreshes prices only (no snapshot).
|
||||
|
||||
## 6. Stock & holdings math
|
||||
|
||||
Source: `crud.py:recompute_holding_from_stock_transactions` — the single source of truth. Holdings are never edited directly; every create/delete of a `StockTransaction` flushes then recomputes from the full per-`(account_id, ticker)` history in date-then-id order (CLAUDE.md invariant 2). Note: as of 2026-07-05, insert also does a full recompute (`crud.py:create_stock_transaction`), not an incremental update.
|
||||
|
||||
Running-total algorithm:
|
||||
- `buy` / `rsu_vest`: `shares += tx.shares`; `total_cost += tx.shares × tx.price_per_share + tx.fee` — **fees are capitalized into cost basis**.
|
||||
- `sell`: reject if `tx.shares > shares` at that point in history (ValueError → HTTP 422). Otherwise reduce `total_cost` by `sell_shares × (total_cost / shares)` (average-cost method), then `shares −= sell_shares`; `total_cost` floored at 0.
|
||||
- Final `avg_cost_basis_per_share = total_cost / shares` (0 if no shares). Zero remaining transactions → the holding row is deleted.
|
||||
|
||||
Example: buy 10 @ $100 fee $5 → shares 10, cost 1005, avg 100.50. Buy 5 @ $120 → shares 15, cost 1605, avg 107. Sell 6 → cost 1605 − 6×107 = 963, shares 9, avg still 107.
|
||||
|
||||
**Fractional shares / total-first buy**: shares are floats. In QuickAdd StockTradeFlow, a buy into an `investment` account is entered as total invested; `shares = totalInvested / price` (fractional), `total_cost = totalInvested + fee` (`web/src/components/QuickAdd/flows/StockTradeFlow.tsx`, `isTotalFirstBuy`).
|
||||
|
||||
**Funded buy cash leg** (`crud.py:create_funded_buy`, commit e36d719): the bank row is typed **`transfer`** — asset conversion, never spending/income. Buy: `−total_cost`; sell: `+(shares × price − fee)`; converted `cash_native / used_rate` when the funding account's currency differs from the stock currency. CLI: `fin stock funded-buy`; plain `fin stock add` writes no cash leg.
|
||||
|
||||
**P&L** (`routers/holdings.py:list_holdings`, feeds Portfolio page and `fin holdings`): per holding, `current_value_eur = _to_eur(shares × live_price, price_ccy)`, `cost_basis_eur = _to_eur(shares × avg_cost_basis_per_share, price_ccy)`, `pnl_eur = current − cost`, `pnl_pct = pnl_eur / cost_basis_eur × 100` (0 when basis is 0). Caveat: cost basis is converted at **today's** FX rate, not the historical one.
|
||||
|
||||
## 7. RSU model
|
||||
|
||||
`models.py:RSUGrant`: `account_id, ticker, vest_date, gross_shares, withholding_pct (default 0.5), notes, vested_at, vested_stock_transaction_id`. One row = one tranche. "One pot per account": grants attach to an `rsu` (or `investment`) account (`crud.py:create_rsu_grant` enforces this); all tranches of a ticker share that account's single holding. The UI "pot form" (`web/src/pages/Portfolio.tsx`) defaults withholding to **51%** (0.51, commit c835118); the schema/CLI default is **0.5** (`schemas.py:RSUGrantBase`, `cli/fin_cli/rsu.py:add`). Withholding is stored as a 0..1 fraction.
|
||||
|
||||
**Vest** (`crud.py:vest_rsu_grant`, `POST /api/rsu-grants/{id}/vest`, `fin rsu vest`):
|
||||
- `net_shares = net_shares_override ?? floor(gross_shares × (1 − withholding_pct))` — **floored to whole shares**, must be ≥ 1 (422 otherwise). E.g. gross 30, withholding 0.51 → floor(14.7) = **14** net shares. (UI preview floors the same way — commit 225e69e.)
|
||||
- Price and **currency come from `price_cache[ticker]`** (422 if no cached price > 0 — refresh prices first). Writes an `rsu_vest` StockTransaction: `shares = net_shares`, `price_per_share = cached price`, `fee = 0`, `total_cost = net_shares × price`, `date = grant.vest_date` (not today), then recomputes the holding and stamps `vested_at` + `vested_stock_transaction_id`.
|
||||
- Cost basis consequence: vests enter the holding at market price, so the "cost basis" of RSU shares is their vest-day value.
|
||||
- Deleting the linked stock transaction reverts the grant to pending (`crud.py:delete_stock_transaction` clears `vested_at`/`vested_stock_transaction_id` first). A vested grant itself cannot be deleted (`crud.py:delete_rsu_grant`).
|
||||
|
||||
**Schedule generation** (`crud.py:_generate_rsu_schedule`; endpoints `/api/rsu-grants/preview-schedule` and `/generate-schedule`; CLI `fin rsu schedule preview|generate`): one cliff tranche at `grant_date + cliff_months`, then a tranche every `frequency_months` until `grant_date + total_months`. Defaults: cliff 12, total 36, frequency 3 (`schemas.py:RSUScheduleRequest`). Each tranche gets `floor(total / n_tranches)` whole shares; the remainder is added to the **last** tranche so the sum equals the (floored) total.
|
||||
|
||||
Pending grants keep their ticker in the price-fetch set (`price_fetcher.py:get_tracked_tickers` unions holdings with unvested grants), which is what makes the dashboard's unvested estimate (§5) and vesting possible.
|
||||
|
||||
## 8. Multi-currency
|
||||
|
||||
- **Base currency is EUR.** Everything reported EUR-first; native balances are shown alongside.
|
||||
- **FX pairs in `price_cache`** (as of 2026-07-05): `EUR/USD` and `EUR/GBP`, both from Frankfurter (`https://api.frankfurter.dev/v1/latest?from=EUR&to=...`), fetched in `price_fetcher.py:fetch_eur_usd` / `fetch_eur_gbp`. Quotation: CCY per 1 EUR → convert to EUR by **dividing**.
|
||||
- **Ticker prices** are cached in whatever currency yfinance reports — not always USD. European suffixes (`.DE`, `.L`, `.AS`, …) are auto-tried; LSE prices in pence (`GBp`/`GBX`) are divided by 100 and stored as GBP (`price_fetcher.py:_try_fetch_candidate`). The price row's `currency` drives conversion, never an assumption of USD.
|
||||
- **Staleness policy**: prices refresh every 5 minutes and at midnight (scheduler jobs); on fetch failure the last cached value is served and a warning logged — the API never crashes on Yahoo/Frankfurter outages.
|
||||
- **No wrong-currency fallbacks.** Commit 81091fc removed a `1.0` FX fallback: if a rate can't be fetched *and* no cache exists, the fetcher returns `None` and skips the upsert, and `net_worth._to_eur` contributes `0.0` with a warning for missing rates. Rationale: a silently-wrong rate (e.g. valuing GBP at par with EUR) corrupts net worth history via snapshots; an obviously-missing value plus a log line is recoverable. Never reintroduce a numeric FX fallback. Known residual exception (as of 2026-07-05): `_get_eur_usd_rate` still falls back to `1.0` and feeds funded-buy conversion — tracked as gap U8 (fin-ledger-analysis-toolkit) / candidate fix (fin-research-frontier), not license to add more.
|
||||
|
||||
## 9. Glossary
|
||||
|
||||
| Term | Meaning in fin |
|
||||
|---|---|
|
||||
| Receivable | Account type: money others owe you (an asset). Created by lending (bank→receivable `transfer`) or a split expense's partner share. Excluded from spending queries. |
|
||||
| Payable | Account type: money you owe others (a liability, negative balance). An `expense` on it counts as spending immediately; paying it off is a neutral `settlement`. |
|
||||
| Settlement | Two-leg `settlement`-typed pair clearing a splitwise/receivable/payable balance against a bank account. Never income or spending. |
|
||||
| Reconcile | The receivable/payable variant of settlement — `POST /api/transactions/reconcile`, `fin reconcile`. |
|
||||
| Splitwise account | Single account holding the running net balance with the partner: positive = they owe you, negative = you owe them. |
|
||||
| Partner share | `splitwise_partner_share` column: the partner's portion of a bill you paid; subtracted (via `amount + partner_share`) so spending shows only your net share. |
|
||||
| Pot | An `rsu` account treated as one pool: all grants/tranches of a ticker vest into the same holding; the UI pot form seeds grants with a 51% withholding default. |
|
||||
| Vest | Converting a pending RSU grant into an `rsu_vest` StockTransaction for `floor(gross × (1 − withholding))` net shares at the cached market price. |
|
||||
| Withholding | Fraction (0..1) of gross shares withheld at vest (tax). Schema default 0.5; UI default 0.51. |
|
||||
| Opening balance | Auto-generated `income` transaction (`description = 'Opening balance'`) from `initial_balance` at account creation; excluded from income totals. |
|
||||
| Manual override | `accounts.manual_override_balance`: replaces the calculated balance in net worth (interpreted in the account's currency); calculation resumes when cleared. |
|
||||
| Snapshot | One `net_worth_snapshots` row per day (midnight job): `total_eur` + per-account `breakdown_json`. Feeds the History chart. |
|
||||
| Holding | Derived position row (unique per account+ticker): `shares` and `avg_cost_basis_per_share`, always recomputed from the stock-transaction log. |
|
||||
| Funded buy | Stock trade plus an atomic bank cash leg typed `transfer`. |
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Running CLI commands against the live API → **fin** skill (`.claude/skills/fin/SKILL.md`).
|
||||
- Architecture/layering rules, what code goes where → **fin-architecture-contract**.
|
||||
- Making changes safely (migrations, review gates) → **fin-change-control**; building/running → **fin-build-and-env**, **fin-run-and-operate**.
|
||||
- Debugging a wrong number in the running system → **fin-debugging-playbook**, **fin-diagnostics-and-tooling**; past incidents → **fin-failure-archaeology**.
|
||||
- Querying/analyzing the ledger data itself → **fin-ledger-analysis-toolkit**; tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**; open questions → **fin-research-frontier**.
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All facts verified 2026-07-05 against the working tree at commit 082b608. Re-verify before trusting:
|
||||
|
||||
- Balance formula: `grep -n "amount - models.Transaction.fee" api/app/services/net_worth.py`
|
||||
- Transaction types: `grep -n "type = Column" api/app/models.py` and `grep -n "Literal\[" api/app/schemas.py`
|
||||
- Two-leg writers: `grep -n "def create_\(transfer\|settlement\|split_expense\|splitwise_i_paid\|funded_buy\)" api/app/crud.py`; reconcile: `grep -n "def reconcile_accounts" api/app/routers/transactions.py`
|
||||
- Spending filters: `grep -n "notin_\|splitwise_partner_share\|Opening balance" api/app/crud.py` (expect `notin_(["receivable"])` for spending, `notin_(["payable", "receivable"])` for income)
|
||||
- Net worth & FX-missing→0: `grep -n "_to_eur\|return 0.0" api/app/services/net_worth.py`
|
||||
- Holding recompute (avg cost, sell guard): `sed -n '453,520p' api/app/crud.py`
|
||||
- Vest floor + price-cache currency: `grep -n "math.floor\|price_row" api/app/crud.py` (in `vest_rsu_grant`)
|
||||
- Schedule tranche math: `grep -n "_generate_rsu_schedule" api/app/crud.py api/app/routers/rsu_grants.py`
|
||||
- FX pairs & pence handling: `grep -n "frankfurter.dev\|GBp\|GBX" api/app/services/price_fetcher.py`
|
||||
- Retype migration: `cat api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py`
|
||||
- UI withholding default: `grep -n "withholding_pct: '51'" web/src/pages/Portfolio.tsx`; unvested dashboard calc: `grep -n "unvestedEurByAccount" web/src/pages/Dashboard.tsx`
|
||||
- CLI command surface: `grep -n "add_typer\|app.command" cli/fin_cli/main.py`
|
||||
- Key commits: `git log --oneline` — e36d719 (funded-buy transfer typing), h8c9d0e1f2a3 in bc256f6-era (retype migration), e61b254 (payable in spending), a9f045f (receivable/payable exclusion + summary), babea72 (unvested display), 81091fc (no FX fallback), 225e69e (floor net shares), c835118 (51% UI default)
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: fin-architecture-contract
|
||||
description: >-
|
||||
Load-bearing design decisions, invariants, and known weak points of the fin
|
||||
app. Load BEFORE any design or refactor decision, before adding endpoints,
|
||||
tables, services, account types, or transaction types, before changing data
|
||||
flow between layers (router/crud/services/frontend), or whenever you ask
|
||||
"why is it built this way?" or "what breaks if I change this?". Also load
|
||||
when reviewing a diff that touches crud.py, models.py, scheduler.py, auth.py,
|
||||
or web/src/api/client.ts.
|
||||
---
|
||||
|
||||
# fin — Architecture Contract
|
||||
|
||||
The design decisions that hold this app together, WHY they exist, and where it
|
||||
is honestly weak. Every claim below was verified by reading the code on
|
||||
2026-07-05 (commit 082b608). Trust this file over memory; trust the code over
|
||||
this file.
|
||||
|
||||
Jargon used once, defined once:
|
||||
|
||||
- **Leg** — one of the two (or three) `Transaction` rows written together for a
|
||||
transfer/settlement/splitwise event, linked via `linked_account_id` and
|
||||
`linked_transaction_id`.
|
||||
- **Derived state** — a table whose contents can be rebuilt from another table
|
||||
(holdings from stock_transactions). Never edit derived state directly.
|
||||
- **Opt-in auth** — bearer-token check that is a no-op unless the API's
|
||||
`FIN_API_TOKEN` env var is set.
|
||||
|
||||
## 1. System map
|
||||
|
||||
Three write paths reach the same SQLite file. Know which one you are on.
|
||||
|
||||
```
|
||||
Browser (React SPA) fin CLI (typer)
|
||||
web/src/api/client.ts cli/fin_cli/client.py (httpx)
|
||||
fetch('/api/...') NO auth header Bearer token if FIN_API_TOKEN set
|
||||
│ │
|
||||
▼ │
|
||||
nginx :80 (web container) │ direct HTTP to API base URL
|
||||
/api/* → proxy_pass http://${API_UPSTREAM} │ (FIN_API_URL, no default)
|
||||
/* → SPA static files + index fallback │
|
||||
│ │
|
||||
▼ ▼
|
||||
uvicorn :8000 (api container, non-root uid 1000)
|
||||
FastAPI app (main.py) ── Depends(require_api_token) on every router
|
||||
│
|
||||
routers/* ──► crud.py / services/* ──► SQLite /data/fin.db (WAL)
|
||||
▲
|
||||
APScheduler BackgroundScheduler (same process, │
|
||||
background thread, started in lifespan): │
|
||||
• price_refresh — every 5 min │
|
||||
• daily_snapshot — cron 00:00 │
|
||||
Each job run builds its OWN engine + session ────┘
|
||||
(services/scheduler.py:_make_engine), never the request-scoped session.
|
||||
```
|
||||
|
||||
Why the scheduler has its own engine (`services/scheduler.py`): jobs run in
|
||||
APScheduler's thread pool, outside any request. Sharing `SessionLocal`
|
||||
request sessions across threads is unsafe; instead each job run calls
|
||||
`create_engine(...)`, applies the same `PRAGMA journal_mode=WAL; PRAGMA
|
||||
foreign_keys=ON` connect listener as `database.py`, and disposes the engine in
|
||||
`finally`. WAL is what lets this second writer coexist with request traffic.
|
||||
Job definitions persist in the same DB via `SQLAlchemyJobStore`
|
||||
(table `apscheduler_jobs`).
|
||||
|
||||
Container startup order (`api/Dockerfile` CMD): `python -m app.preflight`
|
||||
(verifies the DB dir is writable) → `alembic upgrade head` → uvicorn. The
|
||||
`web` container waits on the api healthcheck (`/api/health`) before starting.
|
||||
Images are pushed to `harbor.cloud.traunseenet.com/library/fin-{api,web}`.
|
||||
|
||||
## 2. Layering contract
|
||||
|
||||
| Layer | Owns | Must NOT do |
|
||||
|---|---|---|
|
||||
| `routers/*` | Input validation, HTTP status mapping, `Depends(get_db)` | Business logic, DB mutations (two known violations — see §5) |
|
||||
| `crud.py` | ALL DB mutations, multi-row atomic writes | HTTP concerns (it does import HTTPException in places — tolerated, don't extend) |
|
||||
| `services/` | Logic spanning tables or external APIs (net worth, prices, scheduler) | Being called synchronously from a request handler if it sleeps (`fetch_all_prices` retries with `time.sleep`) |
|
||||
| `web/src/api/client.ts` | Every frontend HTTP call | — nothing else in `web/src` may call `fetch()` directly |
|
||||
| `cli/fin_cli/client.py` | Every CLI HTTP call; zero typer/rich imports (MCP-ready) | Importing CLI presentation deps |
|
||||
|
||||
Frontend refresh contract (verified in `web/src`): after a successful write,
|
||||
`Layout.tsx`, `Dashboard.tsx`, and `Settings.tsx` dispatch
|
||||
`window.dispatchEvent(new CustomEvent('fin:transaction-added'))`; the pages
|
||||
Dashboard, Transactions, Spending, and Portfolio re-fetch on that event in a
|
||||
`useEffect`. This is the ONLY cross-component refresh mechanism — there is no
|
||||
global state manager. Breaking it makes pages silently stale.
|
||||
|
||||
## 3. Load-bearing decisions
|
||||
|
||||
| Decision | Why | What breaks if changed casually |
|
||||
|---|---|---|
|
||||
| **Sync SQLAlchemy, sync endpoints** (`database.py`: `create_engine`, `sessionmaker`) | aiosqlite just wraps a thread anyway; FastAPI already runs sync handlers in a threadpool. Simpler, no async-session lifetime bugs. | Introducing `async def` endpoints with the sync engine blocks the event loop; introducing `AsyncSession` breaks every function in crud.py and the scheduler. |
|
||||
| **SQLite + WAL + `foreign_keys=ON` on every connection** (both `database.py` and `scheduler.py:_make_engine` register a connect listener) | Two writers (requests + scheduler thread) need concurrent readers; FK pragma is per-connection in SQLite, so it must be re-applied on connect. | Dropping the listener on either engine silently disables FK enforcement / reintroduces writer starvation for that engine only — hard to spot. |
|
||||
| **Scheduler builds a throwaway engine per job run** | Thread safety; also guarantees fresh pragmas and no pooled connections leaking across jobs. `engine.dispose()` in `finally`. | Reusing the request `SessionLocal` in a job = cross-thread session use; corrupted sessions under load. |
|
||||
| **Holdings are derived state** — `recompute_holding_from_stock_transactions` (crud.py) is a full chronological fold of `stock_transactions`, called on BOTH insert (`create_stock_transaction`, `create_funded_buy`, `vest_rsu_grant`) and delete (`delete_stock_transaction`), before `db.commit()` | Single source of truth. The fold also validates history (selling more shares than held at that point raises `ValueError` → 422 and rolls back). Note: CLAUDE.md says "incrementally updated on insert" — the code as of 2026-07-05 does a full recompute on insert too. | Writing to `holdings` directly, or skipping the recompute call, desyncs positions from the log with no error until someone notices wrong P&L. |
|
||||
| **Two-leg atomic linked writes** — `create_transfer`, `create_settlement`, `create_splitwise_i_paid` (2 legs), `create_split_expense` (3 rows: expense + 2 transfer legs), and the `/reconcile` route: `db.add_all([...]); db.flush()`; then set `linked_transaction_id` on each leg pointing at its partner; single `db.commit()` | Balances are `SUM(amount − fee)` per account, so an orphaned leg is a permanent balance error. `linked_transaction_id` lets `delete_transaction` walk the whole group (both directions) and delete it atomically. | Deleting one leg by raw SQL, or committing between legs, corrupts two account balances at once. Editing a linked leg is blocked at the router (`PUT /api/transactions/{id}` → 422). |
|
||||
| **`price_cache` = one row per symbol, upserted** (`_upsert_price_cache` in price_fetcher.py; `symbol` is UNIQUE; columns include `name` for display) | No price history table by design — history lives in daily snapshots. FX convention: rows `EUR/USD` and `EUR/GBP` (both fetched from frankfurter.app, as of 2026-07-05). Ticker prices are stored in their native currency (`currency` column); yfinance GBp/GBX pence quotes are divided by 100 and stored as GBP. `net_worth._to_eur` converts via `EUR/<ccy>` and EXCLUDES the amount (0.0 + warning) if no rate is cached. | Adding multiple rows per symbol breaks every `filter_by(symbol=...)` `.first()` read. Changing FX symbol naming breaks `_to_eur`'s `f"EUR/{ccy}"` lookup. |
|
||||
| **`manual_override_balance`** (Account, nullable float) | When not NULL it REPLACES the computed balance in net worth (`is_override` in the breakdown). For investment/rsu accounts the override is assumed to be in the account's declared currency and skips holdings entirely. | Treating it as an adjustment (additive) double-counts. Note the separate `/accounts/{id}/adjust-balance` endpoint which instead writes a real income/expense delta transaction — two different mechanisms, don't conflate them. |
|
||||
| **Opt-in bearer auth** (`auth.py:require_api_token`) | Single-user homelab: zero-config by default. Token read from env AT REQUEST TIME (rotatable without restart), compared with `secrets.compare_digest`. `/api/health` is unauthenticated (Docker healthcheck needs it). Docs/redoc/openapi are disabled on the app; the OpenAPI spec is served from an explicit `GET /api/openapi.json` route with the auth dependency, because FastAPI's built-in openapi route bypasses DI (see comment in main.py). Every router is included with `dependencies=[Depends(require_api_token)]` in main.py. | Known tradeoff: setting `FIN_API_TOKEN` on the API locks out the React frontend — `client.ts` sends no Authorization header. Mitigation options (nginx header injection, frontend token) are NOT implemented as of 2026-07-05. Also: a new router added without the `_auth` dependency list is silently unauthenticated. |
|
||||
| **Soft-delete accounts** (`is_active=False` via `DELETE /api/accounts/{id}` → `crud.deactivate_account`) | Transactions keep their FK; history and old snapshots stay interpretable. Net worth and account lists filter `is_active.is_(True)`. | Hard-deleting an account orphans transactions and breaks FK integrity (FKs are ON). There is no undelete endpoint — flip `is_active` via `PUT`. |
|
||||
| **Non-root Docker user** (api/Dockerfile: `useradd -m -u 1000 app`, `/data` chowned, `USER app`) | Container hardening; uid 1000 matches typical host volume ownership. | Reverting to root breaks the security posture and can flip `/data` file ownership so the non-root image can no longer write the volume. |
|
||||
| **Bank leg of a funded stock trade is typed `transfer`** (`create_funded_buy`), and receivable/payable settles are typed `settlement` (`/reconcile`) | Asset conversion and debt repayment are not consumption/income; spending and income metrics filter by type, so these must not appear there (migration h8c9d0e1f2a3 retyped old rows). | Typing them `expense`/`income` inflates the Spending page — this was a real fixed bug (commit e36d719). |
|
||||
|
||||
## 4. Data-model contract
|
||||
|
||||
Eight tables in `models.py` as of 2026-07-05 (plus `apscheduler_jobs` owned by
|
||||
APScheduler). Account `type` now includes `payable` — six values, enforced by
|
||||
`Literal` in schemas.py, not by the DB (column is Text):
|
||||
`bank | investment | rsu | splitwise | receivable | payable`.
|
||||
(CLAUDE.md still lists five; schemas.py is the truth.)
|
||||
|
||||
| Table | Role | Source of truth or derived? |
|
||||
|---|---|---|
|
||||
| `accounts` | Top-level entity; `manual_override_balance`, `is_active`, `currency` | Source |
|
||||
| `categories` | Spending/income buckets; `type` ∈ expense\|income; deleting one nulls `category_id` on its transactions | Source |
|
||||
| `transactions` | Cash ledger for bank/splitwise/receivable/payable accounts | Source. **Balance = `SUM(amount − fee)`** — never `SUM(amount)`. Sign convention: negative = money out. Fee lives only on the from-leg of a transfer (to-leg fee=0). |
|
||||
| `stock_transactions` | Append-only equity event log (buy\|sell\|rsu_vest) | Source for holdings |
|
||||
| `holdings` | Current position per (account_id, ticker) — UNIQUE constraint | DERIVED: exact fold of stock_transactions. `avg_cost_basis_per_share = total_cost/shares` with average-cost sell accounting. Never write directly. |
|
||||
| `rsu_grants` | Pending vest schedule; `vested_at` + `vested_stock_transaction_id` FK link a vested grant to the `rsu_vest` stock transaction it produced | Source. Deleting that stock transaction reverts the grant to pending first (crud.delete_stock_transaction), else the FK 500s. Vested grants cannot be deleted directly. |
|
||||
| `price_cache` | One row per symbol (UNIQUE), columns: symbol, price, currency, source (yfinance\|frankfurter), fetched_at, `name` | Cache — safe to lose, refetched within 5 min |
|
||||
| `net_worth_snapshots` | One row per day (UNIQUE date): total_eur + `breakdown_json` (per-account values frozen at snapshot time) | FROZEN valuation. Cannot be recomputed later — old prices are gone (price_cache keeps no history). The midnight job upserts today's row. |
|
||||
|
||||
Cross-table invariants:
|
||||
|
||||
1. Linked legs form a group closed under `linked_transaction_id` (both
|
||||
directions); `delete_transaction` deletes the whole group in one commit.
|
||||
Same-currency transfer legs net to `−amount` / `+amount` (fee reduces the
|
||||
source side); cross-currency transfers (`to_amount`/`to_currency`) do NOT
|
||||
sum to zero in either currency — that is expected.
|
||||
2. Splitwise accounting: `splitwise_pay` bank leg has `amount = −total`,
|
||||
`splitwise_partner_share = partner's part`; the user's true expense is
|
||||
`amount + splitwise_partner_share`. `get_monthly_spending` (crud.py) counts
|
||||
it exactly that way in a separate per-row pass. Do not "simplify" it into
|
||||
the grouped SUM query.
|
||||
3. Spending excludes transactions on `receivable` accounts; income excludes
|
||||
`payable`/`receivable` accounts AND rows described `'Opening balance'`
|
||||
(magic string — see §5).
|
||||
4. Migration chain (8 revisions): `a1b2c3d4e5f6 → b2c3d4e5f6a7 → c3d4e5f6a7b8
|
||||
→ d4e5f6a7b8c9 → e5f6a7b8c9d0 → f6a7b8c9d0e1 → g7b8c9d0e1f2 →
|
||||
h8c9d0e1f2a3`. `Base.metadata.create_all` in lifespan is first-boot
|
||||
convenience only, never a migration strategy.
|
||||
|
||||
## 5. Known weak points (honest, as of 2026-07-05)
|
||||
|
||||
| # | Weak point | Risk | Blast radius | Current mitigation |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **crud.py is a 995-line monolith with no direct tests.** No `api/tests/` exists; no `.github/` → no CI at all. The only automated coverage is `cli/tests/test_cli.py` (25 tests, e2e: FinClient → TestClient → SQLite), which exercises the API only through CLI-shaped paths. | High — this is the owner's stated hardest problem | Every balance/spending/holding number | The e2e suite; the invariants in this file. See sibling `fin-test-coverage-campaign`. |
|
||||
| 2 | **Layering violations in routers**: `POST /api/transactions/reconcile` (routers/transactions.py) and `POST /api/accounts/{id}/adjust-balance` (routers/accounts.py) construct and commit `models.Transaction` rows directly in the router, bypassing crud.py. | Medium — atomicity is currently correct, but these paths are invisible to anyone auditing crud.py for mutation logic | Balances of bank + payable/receivable accounts | None. If you touch linked-write logic, remember these two live outside crud.py. |
|
||||
| 3 | **Legacy linked-leg deletion heuristic**: rows written before `linked_transaction_id` existed (pre-e5f6a7b8c9d0) are matched by (accounts swapped, same date, same type, closest id) in `delete_transaction`. | Low-probability wrong-partner delete on old data | Two account balances | Closest-id preference; new writes always set `linked_transaction_id`. |
|
||||
| 4 | **Auth lockout tradeoff**: enabling `FIN_API_TOKEN` on the API breaks the React frontend entirely (client.ts sends no auth header; nginx injects nothing). | Certain, not probabilistic — it WILL lock out the UI | Whole web UI | Documented in CLAUDE.md; token currently meant for CLI-only deployments. |
|
||||
| 5 | **yfinance has no SLA, and `fetch_ticker_price` sleeps in-process**: retry schedule 1s/2s/4s per candidate × up to 9 candidates (bare + 8 EU exchange suffixes). `POST /api/prices/fetch-ticker` calls it SYNCHRONOUSLY in the request handler — worst case blocks that worker for minutes. `POST /api/prices/refresh` correctly defers to BackgroundTasks (202). | Medium | One threadpool worker per stuck request; stale prices on Yahoo outage | Stale-cache fallback everywhere (`_to_eur` warnings, cached EUR/USD, `EUR/USD→1.0` fallback — which itself silently skews net worth when cache is empty). |
|
||||
| 6 | **No DB backup automation anywhere in the repo** — the prod SQLite volume (`sqlite_data:/data/fin.db`) is sacred and unprotected. Snapshots are frozen JSON, so data loss is unrecoverable, not just inconvenient. | High impact, unmitigated | All financial history | None in repo. Treat any operation touching `/data` as irreversible; see siblings `fin-run-and-operate` / `fin-change-control`. |
|
||||
| 7 | **Money as Float**: every amount/fee/price column is `Column(Float)`; balance math is IEEE-754. The adjust-balance endpoint already needs an `abs(delta) < 0.001` epsilon. | Low per-transaction, accumulates | Cosmetic cent drift in balances | Epsilon comparisons; single-user scale keeps it tolerable. Do not "fix" ad hoc — a Decimal migration is a project. |
|
||||
| 8 | **Magic-string semantics**: monthly income excludes rows where `description = 'Opening balance'` (crud.get_monthly_spending). Renaming that description in `create_account` breaks income stats silently. | Low | Spending page income numbers | The string is set in exactly one place (`crud.create_account`). |
|
||||
| 9 | **Unauthenticated-by-omission routers**: auth is applied per-router in main.py (`dependencies=_auth`); a newly added router that skips the list is open even when the token is set. | Medium when auth is on | New endpoints | Convention only — check main.py in review. |
|
||||
|
||||
## 6. Extension seams — exact touch order
|
||||
|
||||
**New account type** (no migration needed — `accounts.type` is Text):
|
||||
1. `api/app/schemas.py` — add to both `Literal[...]` lists (AccountCreate, AccountUpdate).
|
||||
2. `api/app/services/net_worth.py` — decide: cash-like branch (`SUM(amount−fee)`) or holdings branch; unknown types are skipped with a warning.
|
||||
3. `api/app/routers/accounts.py` — extend the adjust-balance allowlist if cash-like.
|
||||
4. `api/app/crud.py` — review the `Account.type.notin_([...])` exclusion lists in `get_monthly_spending` (spending AND income queries).
|
||||
5. `web/src` — Settings account form + Dashboard grouping; API types in `client.ts`.
|
||||
6. Update the comment in `models.py` and CLAUDE.md's data-model paragraph.
|
||||
|
||||
**New transaction type**:
|
||||
1. `schemas.py` `TransactionCreate.type` Literal → 2. `crud.py` creator (multi-leg? follow the `add_all → flush → set linked_transaction_id → commit` pattern exactly) → 3. spending/income filters in `get_monthly_spending` (does it count as consumption, income, or neither?) → 4. `delete_transaction` group semantics (linked legs are handled automatically if you set `linked_transaction_id`) → 5. router endpoint → 6. `client.ts` function → 7. `web/src/components/QuickAdd/flows/` component that calls `onSuccess()` → 8. CLI: `cli/fin_cli/client.py` method + subcommand module.
|
||||
|
||||
**New endpoint on an existing resource**:
|
||||
router function (validation + HTTP mapping only) → crud/service function (all DB work) → schemas → confirm the router is registered in `main.py` WITH `dependencies=_auth` → `client.ts` → CLI client + command → e2e test in `cli/tests/test_cli.py`.
|
||||
|
||||
**New table**: `models.py` → `alembic revision --autogenerate -m "..."` inside the api container (verified statically — no venv in this sandbox), review the generated file, chain it after `h8c9d0e1f2a3` → schemas → crud → router (+ `_auth`) → clients. Never `create_all` as migration, never raw DDL.
|
||||
|
||||
**New page**: `web/src/pages/` → route in `App.tsx` → data via `client.ts` only → listen for `fin:transaction-added` if it displays transaction-derived data.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Recording expenses / querying the live app → sibling `fin` (CLI usage).
|
||||
- Deploy, restart, volume, or prod-DB operations → `fin-run-and-operate`, and `fin-change-control` before anything irreversible.
|
||||
- Debugging a wrong number or a 500 → `fin-debugging-playbook`; past incidents → `fin-failure-archaeology`.
|
||||
- What the accounting SHOULD be (splitwise math, RSU semantics) → `fin-accounting-reference`; bulk ledger queries → `fin-ledger-analysis-toolkit`.
|
||||
- Local build/toolchain problems → `fin-build-and-env`; ad-hoc inspection tools → `fin-diagnostics-and-tooling`.
|
||||
- Writing or expanding tests → `fin-test-coverage-campaign` and `fin-validation-and-qa`.
|
||||
- Speculative redesigns (Postgres, async, multi-user) → `fin-research-frontier` — but re-read §3 first; most "obvious" upgrades break a listed invariant.
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All facts verified by reading the repo at commit 082b608 on 2026-07-05. Re-verify before relying on volatile items:
|
||||
|
||||
- Account/transaction type lists: `grep -n "Literal" api/app/schemas.py`
|
||||
- Table set + columns: read `api/app/models.py` (30 s)
|
||||
- Migration head: `ls api/alembic/versions/` and check `down_revision` chain
|
||||
- Scheduler jobs + intervals: read `api/app/services/scheduler.py:create_scheduler`
|
||||
- Auth wiring: `grep -n "dependencies" api/app/main.py` (every router must carry `_auth`)
|
||||
- FX symbols cached: `grep -n "EUR/" api/app/services/price_fetcher.py`
|
||||
- Frontend auth header still absent: `grep -n "Authorization" web/src/api/client.ts` (expect no hits)
|
||||
- Event bus intact: `grep -rn "fin:transaction-added" web/src`
|
||||
- Test/CI status: `ls api/tests .github cli/tests` (as of 2026-07-05: no api/tests, no .github, 25 tests in cli/tests/test_cli.py)
|
||||
- Router-layer mutation exceptions: `grep -n "db.add" api/app/routers/*.py`
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: fin-build-and-env
|
||||
description: Recreate the fin dev/build environment from scratch and avoid known toolchain traps. Load when setting up a dev environment on a fresh machine or sandbox, running the pytest suite locally, building the Docker images (api or web), installing the CLI, or hitting build/toolchain errors (pip/venv missing, npm ci failures, alembic path errors, yfinance import weight, SQLite WAL files).
|
||||
---
|
||||
|
||||
# fin — build and environment runbook
|
||||
|
||||
Read `/CLAUDE.md` at the repo root first if you have not. This skill covers only
|
||||
"how do I get a working environment"; siblings cover everything else.
|
||||
|
||||
All facts below were **verified statically** against the named files as of
|
||||
2026-07-05. Nothing in this file was executed in the authoring session (the
|
||||
authoring sandbox had no pip, venv, docker, or node).
|
||||
|
||||
## Prerequisites by task
|
||||
|
||||
| Task | Needs | Version source (verified statically) |
|
||||
|---|---|---|
|
||||
| (a) Full-stack run | Docker + docker compose. Nothing else — both images build everything internally. | `docker-compose.yml`, `api/Dockerfile`, `web/Dockerfile` |
|
||||
| (b) Backend tests/dev without Docker | Python ≥ 3.12, pip, venv. No node, no docker. | `api/Dockerfile` (`FROM python:3.12-slim`), `cli/pyproject.toml` (`requires-python = ">=3.12"`) |
|
||||
| (c) Frontend dev/build | Node 20 (Docker uses `node:20-alpine`), npm; the image upgrades to npm 11 before `npm ci` (see traps). `package.json` declares no `engines` field. | `web/Dockerfile`, `web/package.json` |
|
||||
| (d) CLI install | Python ≥ 3.12 + pip. Deps: `typer>=0.12`, `httpx>=0.27`, `rich>=13`. Build backend: hatchling. | `cli/pyproject.toml` |
|
||||
|
||||
Python version note (as of 2026-07-05): Docker runs Python 3.12
|
||||
(`python:3.12-slim`), but the test suite was last executed under **CPython 3.13
|
||||
with pytest 9.1.1** — evidenced by `cli/tests/__pycache__/test_cli.cpython-313-pytest-9.1.1.pyc`.
|
||||
Both work; `requires-python = ">=3.12"` is the floor.
|
||||
|
||||
`api/requirements.txt` (mostly unpinned — resolves to latest at build time):
|
||||
`fastapi`, `uvicorn[standard]`, `sqlalchemy`, `alembic`, `pydantic>=2.0`,
|
||||
`yfinance`, `apscheduler>=3.10`, `httpx`.
|
||||
|
||||
## (a) Full-stack Docker run
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Anatomy (verified statically against `docker-compose.yml`, `api/Dockerfile`,
|
||||
`web/Dockerfile`, `web/nginx.conf`):
|
||||
|
||||
- **api** service: builds `./api`, image tag `harbor.cloud.traunseenet.com/library/fin-api:latest`.
|
||||
Dockerfile CMD, verbatim:
|
||||
|
||||
```
|
||||
CMD sh -c "python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000"
|
||||
```
|
||||
|
||||
`app.preflight` validates the SQLite directory exists and is writable
|
||||
(`app/db_config.ensure_sqlite_database_dir`), then Alembic migrates, then
|
||||
uvicorn starts. Runs as non-root user `app` (uid 1000) — do not revert.
|
||||
`DATABASE_URL=sqlite:////data/fin.db`; named volume `sqlite_data` → `/data`.
|
||||
- **api healthcheck**: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')"`
|
||||
every 30s, timeout 10s, 3 retries.
|
||||
- **web** service: builds `./web`, publishes port 80, and `depends_on: api:
|
||||
condition: service_healthy` — nginx does not start until the API healthcheck
|
||||
passes. Env `API_UPSTREAM=api:8000`.
|
||||
- **web image**: two-stage. Builder: `node:20-alpine`, `npm install -g npm@11 && npm ci`,
|
||||
`npm run build` (= `tsc && vite build`). Serve stage: `nginx:alpine`,
|
||||
dist → `/usr/share/nginx/html`, and `nginx.conf` copied to
|
||||
`/etc/nginx/templates/default.conf.template` — the stock nginx:alpine
|
||||
entrypoint runs envsubst on `/etc/nginx/templates/*.template`, substituting
|
||||
`${API_UPSTREAM}` in the `proxy_pass http://${API_UPSTREAM};` line. nginx
|
||||
proxies `/api/*` to the API, serves the SPA with `try_files` fallback, and
|
||||
sends `Cache-Control: no-cache` on `index.html`.
|
||||
|
||||
DB location: named volume `sqlite_data` mounted at `/data`, file `/data/fin.db`
|
||||
(plus `fin.db-wal` / `fin.db-shm`, see traps).
|
||||
|
||||
## (b) Backend test environment WITHOUT Docker — the path you will actually use
|
||||
|
||||
```bash
|
||||
cd /path/to/fin
|
||||
python -m venv .venv
|
||||
. .venv/bin/activate
|
||||
pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0"
|
||||
pip install -e cli/ # pulls typer, httpx, rich; installs the `fin` entry point
|
||||
python -m pytest cli/tests/ -q
|
||||
```
|
||||
|
||||
Minimal-dependency rationale (verified statically against
|
||||
`cli/tests/conftest.py` and module-scope imports in `api/app/`):
|
||||
|
||||
- `conftest.py` imports `fastapi.testclient.TestClient`, `app.database`,
|
||||
`app.main`, `fin_cli.client`. `app.main` imports the routers and
|
||||
`app.services.scheduler`, which imports **apscheduler** at module scope — so
|
||||
apscheduler is required even though the scheduler never runs.
|
||||
- **yfinance is deliberately NOT needed**: conftest stubs it before the app
|
||||
import chain runs — `sys.modules.setdefault("yfinance", types.ModuleType("yfinance"))`.
|
||||
It is a heavy package (pandas etc.); do not install it just to run tests.
|
||||
- **alembic is NOT needed for tests**: the fixture uses
|
||||
`Base.metadata.drop_all` / `create_all` per test, not migrations.
|
||||
- `uvicorn` is NOT needed for tests (only to actually serve the API).
|
||||
|
||||
conftest mechanics (read `cli/tests/conftest.py`, 59 lines):
|
||||
|
||||
1. Computes `REPO_ROOT = Path(__file__).resolve().parents[2]` and inserts
|
||||
`REPO_ROOT/api` and `REPO_ROOT/cli` into `sys.path` — **absolute paths, so
|
||||
the working directory does not matter**; `python -m pytest cli/tests/ -q`
|
||||
from the repo root and `python -m pytest tests/ -q` from `cli/` both work.
|
||||
2. Sets a throwaway `DATABASE_URL` (`sqlite:///<mkdtemp>/test.db`) **before**
|
||||
importing `app.database`, which reads it at import time. Your real DB is
|
||||
never touched.
|
||||
3. Autouse `clean_db` fixture: deletes `FIN_API_TOKEN` from the env, then
|
||||
`drop_all` + `create_all` per test — every test starts on an empty schema.
|
||||
4. The APScheduler job never starts: tests use `TestClient(fastapi_app)`
|
||||
without running the app's lifespan.
|
||||
5. `make_client` / `client` fixtures wrap the in-process app in a `FinClient`
|
||||
(`cli/fin_cli/client.py`, plain httpx, no typer/rich imports) — the tests
|
||||
are true e2e: FinClient → real FastAPI app → real crud → SQLite.
|
||||
|
||||
Expected result: **25 tests** (counted statically: 25 `def test_` functions in
|
||||
`cli/tests/test_cli.py`, no parametrize), all passing as of 2026-07-05.
|
||||
|
||||
To run the API itself without Docker, additionally `pip install uvicorn alembic yfinance`,
|
||||
then from `api/`: `alembic upgrade head && uvicorn app.main:app --port 8000`
|
||||
(see the alembic trap below; running the server is fin-run-and-operate territory).
|
||||
|
||||
## (c) Frontend build
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npm ci # NOT npm install — lockfile is the contract (commit bfe7590)
|
||||
npm run build # = tsc && vite build → dist/
|
||||
npm run dev # vite dev server (script exists in package.json)
|
||||
```
|
||||
|
||||
- **Lockfile discipline**: `bfe7590` switched the Dockerfile from
|
||||
`COPY package.json` + `npm install` to `COPY package.json package-lock.json`
|
||||
+ `npm ci` for reproducible builds. Keep `package-lock.json` committed and
|
||||
in sync.
|
||||
- **Dev-server honesty**: CLAUDE.md says "No separate dev server setup exists"
|
||||
— meaning there is no documented/wired dev workflow (no proxy config for
|
||||
`/api` in `vite.config.ts` is guaranteed; check before relying on it). But
|
||||
`package.json` does define `dev: vite` and `preview: vite preview` scripts.
|
||||
The supported path is `docker compose up --build`.
|
||||
- Tailwind 3.4 + shadcn/ui structure: `tailwind.config.js`,
|
||||
`postcss.config.js`, components under `src/components/ui/`. New Radix
|
||||
primitives are scaffolded per `web/components.json` (style `default`,
|
||||
baseColor `slate`, aliases `@/components`, `@/lib/utils`). Do not add new UI
|
||||
libraries.
|
||||
- `package.json` also carries `allowScripts: { "esbuild@0.21.5": true }`
|
||||
(added alongside the registry migration, see traps).
|
||||
|
||||
## (d) CLI install
|
||||
|
||||
```bash
|
||||
pip install -e cli/ # from repo root; entry point: fin = fin_cli.main:app
|
||||
export FIN_API_URL=http://<host> # required, no default
|
||||
export FIN_API_TOKEN=<token> # only if the API enforces auth
|
||||
fin health
|
||||
```
|
||||
|
||||
Agent-facing usage docs: `.claude/skills/fin/SKILL.md`.
|
||||
|
||||
## Known traps
|
||||
|
||||
| Trap | Story / fix |
|
||||
|---|---|
|
||||
| Debian/Ubuntu: `python -m venv` fails (`ensurepip` missing) | Minimal Debian images and some sandboxes ship python3 without `python3-venv`/`python3-pip`. Fix: `apt install python3-venv python3-pip`, or use `python -m venv --without-pip` + a pip bootstrap, or `uv venv` if uv exists. The authoring sandbox for this skill had neither pip nor venv — that is why nothing here was executed. |
|
||||
| npm on Alpine: silent/failed `npm ci` exit-handler bug | Commit `89d241a` (2026-07-01): `web/Dockerfile` prepends `npm install -g npm@11` before `npm ci` because the npm bundled with `node:20-alpine` hits an Alpine exit-handler bug. If you touch the web Dockerfile, keep the npm 11 upgrade line. |
|
||||
| Lockfile pointed at internal Dynatrace npm mirror | Commit `3c2766e` (2026-07-01) rewrote `package-lock.json` `resolved` URLs from the Dynatrace mirror to `https://registry.npmjs.org`. If `npm ci` fails resolving packages off-VPN, check `grep -c 'registry.npmjs.org' web/package-lock.json` — every `resolved` must be the public registry. Never regenerate the lockfile while an internal `.npmrc` registry override is active. |
|
||||
| Installing yfinance to run tests | Unnecessary. `cli/tests/conftest.py` stubs `sys.modules["yfinance"]` precisely because yfinance is a heavy module-scope import in `app/services/price_fetcher.py`. Only install it to run the real server/price fetcher. |
|
||||
| Stray `fin.db-wal` / `fin.db-shm` files | WAL mode is applied on every connection (`PRAGMA journal_mode=WAL`, invariant #3 in CLAUDE.md), so the DB is three files. Never copy/back up `fin.db` alone while the app runs; never delete the `-wal` file — it holds unmerged committed data. |
|
||||
| `alembic upgrade head` fails with "No 'script_location'…" or can't import app | Alembic must run **from `api/`**: `api/alembic.ini` has relative `script_location = alembic` and `prepend_sys_path = .`; the Docker CMD runs it with WORKDIR `/app` (= the api dir). `alembic/env.py` also inserts the api dir into `sys.path` and reads `DATABASE_URL` via `app.db_config` (default `sqlite:///./fin.db` per `db_config.py`; the `sqlalchemy.url` in alembic.ini is a fallback). So: `cd api && alembic upgrade head`, with `DATABASE_URL` exported if you don't want `./fin.db`. |
|
||||
|
||||
## Quick sanity checklist after setup
|
||||
|
||||
```bash
|
||||
# 1. Backend modules compile (mirrors the allowlisted command in .claude/settings.local.json; run from api/)
|
||||
cd api && python -m py_compile app/crud.py app/schemas.py app/routers/transactions.py app/services/net_worth.py app/services/scheduler.py app/services/price_fetcher.py
|
||||
|
||||
# 2. Test suite green — expect 25 passed (count as of 2026-07-05)
|
||||
cd .. && python -m pytest cli/tests/ -q
|
||||
|
||||
# 3. CLI reaches a running stack
|
||||
FIN_API_URL=http://localhost fin health # or :8000 if hitting uvicorn directly
|
||||
```
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Operating/deploying an already-built stack, backups, price refresh, scheduler behavior → **fin-run-and-operate**
|
||||
- Using the `fin` CLI against a live API (commands, flows) → **fin** skill
|
||||
- Diagnosing runtime bugs, log reading → **fin-debugging-playbook**, **fin-diagnostics-and-tooling**
|
||||
- Making/reviewing code changes, migrations policy → **fin-change-control**, **fin-architecture-contract**
|
||||
- Writing or expanding tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**
|
||||
- Accounting semantics (balances, spending math) → **fin-accounting-reference**, **fin-ledger-analysis-toolkit**
|
||||
- Historical incidents → **fin-failure-archaeology**; open research → **fin-research-frontier**
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
Everything above was verified **statically** on 2026-07-05; nothing was executed. Re-verify with:
|
||||
|
||||
- Base images / CMD: `grep -E 'FROM|CMD' api/Dockerfile web/Dockerfile`
|
||||
- Compose wiring (healthcheck, depends_on, volume): `cat docker-compose.yml`
|
||||
- Python deps: `cat api/requirements.txt cli/pyproject.toml`
|
||||
- Test count: `grep -c 'def test_' cli/tests/test_cli.py` (25 as of 2026-07-05)
|
||||
- Last-run interpreter: `ls cli/tests/__pycache__/` (cpython-313, pytest 9.1.1 as of 2026-07-05)
|
||||
- conftest mechanics: `sed -n '1,40p' cli/tests/conftest.py`
|
||||
- npm traps: `git show 89d241a bfe7590 3c2766e --stat` and `grep -vc 'registry.npmjs.org' <(grep '"resolved"' web/package-lock.json)` (should be 0)
|
||||
- Alembic cwd requirement: `grep -E 'script_location|prepend_sys_path' api/alembic.ini`
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
name: fin-change-control
|
||||
description: Change-control rulebook for the fin repo — how to classify a change (trivial / behavior / schema / accounting-logic / deploy-affecting), which gate each class requires, the non-negotiable invariants with the historical incident behind each, the prod-DB-is-sacred rule, Alembic migration discipline, the spec→plan workflow in docs/superpowers, docs-of-record house style, and the done-checklist. Load BEFORE making any code, schema, or accounting change to fin; before writing or reviewing an Alembic migration; before touching crud.py transfer/settlement/holding/spending logic; before anything that alters Dockerfiles, docker-compose.yml, or env vars; and when writing specs, plans, commit messages, or updating CLAUDE.md/README.md.
|
||||
---
|
||||
|
||||
# fin change control
|
||||
|
||||
How changes to `fin` are classified, gated, reviewed, and documented. `fin` is a single-user personal-finance app whose production SQLite database holds the owner's **real financial history** — the entire process below exists to protect that ledger. Read `/CLAUDE.md` (rulebook) and `/PLAN.md` (original design) alongside this skill; nothing here overrides CLAUDE.md.
|
||||
|
||||
All file paths are repo-relative. Line numbers and the migration head are correct **as of 2026-07-05** — re-verify with the commands in "Provenance and maintenance" if time has passed.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
| You actually want to… | Use instead |
|
||||
|---|---|
|
||||
| Query/record finances via the `fin` CLI | `fin` (`.claude/skills/fin/SKILL.md`) |
|
||||
| Debug a live bug | `fin-debugging-playbook` |
|
||||
| Read the full story behind past incidents | `fin-failure-archaeology` |
|
||||
| Understand layering/architecture rules in depth | `fin-architecture-contract` |
|
||||
| Understand the double-entry/Splitwise accounting model | `fin-accounting-reference` |
|
||||
| Build images / set up an environment | `fin-build-and-env` |
|
||||
| Deploy, operate, back up on the homelab | `fin-run-and-operate` |
|
||||
| Run tests / QA a change hands-on | `fin-validation-and-qa` |
|
||||
| Write new API tests (the coverage campaign) | `fin-test-coverage-campaign` |
|
||||
| Analyze ledger data | `fin-ledger-analysis-toolkit` |
|
||||
|
||||
This skill is the **gatekeeper**: it tells you what a change must satisfy before it counts as done.
|
||||
|
||||
## 1. Change classification
|
||||
|
||||
Classify every change before writing code. A change belongs to the **highest** matching row (a schema change that alters spending math is an accounting-logic change).
|
||||
|
||||
| Class | Examples | Required gate |
|
||||
|---|---|---|
|
||||
| **Trivial fix** | Typo, label text, log message, comment, dead-code removal | Compile/build check (§7 items 1 & 3). Confirm no invariant file (`crud.py`, `net_worth.py`, migrations) was touched. |
|
||||
| **Behavior change** | New endpoint, new QuickAdd flow, new CLI command, changed validation | Trivial gate **+** `pytest cli/tests` green **+** a test covering the new behavior **+** README/CLAUDE.md update if a convention or config var changed. |
|
||||
| **Schema change** | New column/table/index, retyped column | Behavior gate **+** Alembic migration per §4 (autogenerated, reviewed, backward-safe) **+** scratch-DB `alembic upgrade head` dry-run **+** spec/plan (§5) if it also spans the frontend. |
|
||||
| **Accounting-logic change** | Anything touching balances, spending/income math, holdings recompute, net-worth calc, transfer/settlement legs (`api/app/crud.py`, `api/app/services/net_worth.py`) | Schema gate (if schema involved) **+** written spec in `docs/superpowers/specs/` **+** re-verify invariants 1, 2, 5, 6 in §3 **+** if the *meaning* of existing rows changes, a data migration retyping history (precedent: `h8c9d0e1f2a3`, §4.4). |
|
||||
| **Deploy-affecting change** | `api/Dockerfile`, `web/Dockerfile`, `docker-compose.yml`, `web/nginx.conf`, any migration, new env var | Relevant gate above **+** prod DB backup before rollout (§2.3) **+** README "Environment / Config" table updated for new vars **+** remember: the api container runs `alembic upgrade head` automatically on start (§2.4). |
|
||||
|
||||
"Gate" means: do not declare the change done, and do not deploy, until every item passes.
|
||||
|
||||
## 2. Non-negotiable zero: the production database is SACRED
|
||||
|
||||
The prod DB is the SQLite file `/data/fin.db` inside the Docker named volume `sqlite_data` on the homelab (see `docker-compose.yml`; `DATABASE_URL=sqlite:////data/fin.db`). It contains real, irreplaceable financial history. There is no upstream to re-sync from.
|
||||
|
||||
### 2.1 Never test against it
|
||||
|
||||
- Tests use throwaway databases only. `cli/tests/conftest.py` creates a temp-dir SQLite file and sets `DATABASE_URL` before importing the app — keep that pattern for all future tests.
|
||||
- Never point a local checkout, a REPL, `alembic`, or the `fin` CLI's write commands at the prod volume "just to try something".
|
||||
- Analytical *reads* against a **copy** of the DB are fine (see `fin-ledger-analysis-toolkit`); take the copy with the backup command below.
|
||||
|
||||
### 2.2 Backward-safe migrations only
|
||||
|
||||
- New columns must be **nullable or carry `server_default`** so existing rows stay valid (precedent: `f6a7b8c9d0e1` adds `categories.type` with `server_default='expense'`; design rationale in `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md`).
|
||||
- No destructive column/table drops without a copy-forward plan (write the data somewhere first, verify, then drop in a later migration).
|
||||
- Data migrations that rewrite rows must be surgical and documented in the migration docstring; if the downgrade cannot restore the data, say so explicitly in `downgrade()` (precedent: `h8c9d0e1f2a3`).
|
||||
- SQLite cannot `ALTER COLUMN`/`DROP COLUMN` freely — use Alembic `batch_alter_table` when a migration needs more than `ADD COLUMN`.
|
||||
|
||||
### 2.3 Back up before any risky operation
|
||||
|
||||
Before deploying a migration or running any manual DB surgery, snapshot the prod DB. (Canonical backup/restore runbook: **fin-run-and-operate §3** — it uses the equivalent Python `sqlite3.backup()` API; keep the two in sync.) `VACUUM INTO` produces a consistent single-file copy even under WAL, and the `python:3.12-slim` image bundles a new-enough SQLite (≥ 3.27). Run on the homelab host (verified statically against `docker-compose.yml` and `api/Dockerfile` — no docker in the authoring sandbox):
|
||||
|
||||
```bash
|
||||
# 1. Snapshot inside the container (writes into the same volume)
|
||||
docker compose exec api python -c "import sqlite3; sqlite3.connect('/data/fin.db').execute(\"VACUUM INTO '/data/fin-backup-$(date +%F).db'\")"
|
||||
# 2. Copy the snapshot off the container
|
||||
docker compose cp api:/data/fin-backup-$(date +%F).db ./
|
||||
```
|
||||
|
||||
Cold-backup alternative (stack stopped): `docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/out alpine tar czf /out/fin-db-backup.tgz -C /data .` — note the volume's real name carries the compose project prefix; find it with `docker volume ls | grep sqlite_data`.
|
||||
|
||||
### 2.4 Migrations auto-run on container start
|
||||
|
||||
`api/Dockerfile` CMD: `python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000`. Pushing an image with a bad migration to the Harbor registry means it **executes against prod on the next restart**. That is why migration review (§4) is a hard gate, not a courtesy.
|
||||
|
||||
## 3. The non-negotiables — invariant, rationale, incident, blast radius
|
||||
|
||||
Every invariant from CLAUDE.md, each with the commit history that motivated it. `git show <hash>` for the full story; `fin-failure-archaeology` for narratives.
|
||||
|
||||
### 3.1 Atomic two-leg writes
|
||||
|
||||
- **Rule**: `create_transfer` and `create_settlement` in `api/app/crud.py` write both legs with `db.add_all([...]); db.commit()` in one transaction (lines 233/269; the split-expense triple at line 350). Legs are cross-linked via `linked_transaction_id`. Never create, update, or delete a single leg.
|
||||
- **Rationale**: each leg is one side of a double-entry pair; a lone leg silently corrupts two account balances at once.
|
||||
- **Incidents**: `a06f921` — deletes left orphaned paired rows "corrupting balances"; `a1f109b` — the account-pair lookup used for paired deletes was ambiguous, so `linked_transaction_id` was added for precise pairing; `a1289f6` / `2bd4730` — API and UI had to *block* edits on linked legs; `9d06e1b` — UI still showed orphan legs after delete; `bc256f6` — split expense was two sequential HTTP requests that could partially fail, replaced by the atomic `POST /api/transactions/split-expense`; `e36d719` — `delete_transaction` now follows `linked_transaction_id` in both directions and deletes the whole group atomically. The CLI enforces this too: `fin tx add` refuses `transfer`/`settlement` types (`082b608`).
|
||||
- **If violated**: balances on two accounts diverge from the transaction log; net worth is wrong; no error is raised anywhere.
|
||||
|
||||
### 3.2 Holding recompute on delete
|
||||
|
||||
- **Rule**: `delete_stock_transaction` (crud.py:522) calls `recompute_holding_from_stock_transactions` after the delete and **before** `db.commit()`. Holdings are derived state — never edit them directly.
|
||||
- **Rationale**: `Holding` rows are a cache of the `stock_transactions` log; any log mutation that skips the recompute desynchronizes portfolio value and cost basis.
|
||||
- **Incidents**: `1480088` — stock-transaction delete and funded-buy bugs plus an over-sell guard; `bc256f6` — deleting a vest transaction hit an FK 500 and made vested grants undeletable (deadlock: 422 on grant delete, 500 on transaction delete); fixed by reverting the grant to pending inside the same atomic delete.
|
||||
- **If violated**: portfolio shows shares that were never bought or misses shares that were; avg cost basis and P&L are fiction.
|
||||
|
||||
### 3.3 WAL mode on every connection
|
||||
|
||||
- **Rule**: `PRAGMA journal_mode=WAL` is applied per-connection in `api/app/database.py:22` **and** in the scheduler's private engine `api/app/services/scheduler.py:25`. The scheduler thread must never share a request-scoped session.
|
||||
- **Rationale**: founding design decision (PLAN.md: "Multiple concurrent reads (API + scheduler) without locking; mandatory for this pattern"). Preventive — no incident on record, keep it that way.
|
||||
- **If violated**: `database is locked` errors when the midnight snapshot job overlaps API traffic.
|
||||
|
||||
### 3.4 Non-root Docker user
|
||||
|
||||
- **Rule**: `api/Dockerfile` creates user `app` (uid 1000) and runs as it. Do not revert to root.
|
||||
- **Incident**: `ffa28bb` — "security hardening and startup performance — non-root user, disable docs, async prices, remove blocking startup fetch".
|
||||
- **If violated**: a container escape or dependency compromise gets root on the homelab host's kernel; also `/data` ownership breaks (`chown app:app /data` is baked into the image).
|
||||
|
||||
### 3.5 Balance = SUM(amount − fee)
|
||||
|
||||
- **Rule**: for bank/splitwise/receivable accounts, balance is `func.sum(Transaction.amount - Transaction.fee)` (`api/app/services/net_worth.py:88`). Never sum `amount` alone.
|
||||
- **Incident**: `1480088` — "fee balance" was one of the bug-hunt findings: fees were paid in reality but missing from computed balances.
|
||||
- **If violated**: every account with fee-bearing transactions drifts upward from its true bank statement.
|
||||
|
||||
### 3.6 Spending counts the user's net share, and only real consumption
|
||||
|
||||
- **Rule**: in `get_monthly_spending` (crud.py:624), `splitwise_pay` expenses count at `amount + splitwise_partner_share` (the user's net share, not the full amount). Internal asset movements (transfers, settlements, funded stock-buy cash legs) are typed `transfer`/`settlement` precisely so spending/income queries exclude them.
|
||||
- **Incidents**: `a9f045f` — payable/receivable excluded from spending; `e61b254` — then payable *expenses* had to be re-included (both directions of this mistake exist in history); `e36d719` + data migration `h8c9d0e1f2a3` — funded stock buys were typed `expense` and receivable/payable settles typed `income`/`expense`, inflating spending and income; new rows retyped and **historical rows migrated**.
|
||||
- **If violated**: monthly spending/income analytics are wrong, which silently poisons every future forecasting/analytics ambition.
|
||||
|
||||
### 3.7 Never fabricate an FX rate
|
||||
|
||||
- **Rule**: if the EUR/USD (or EUR/GBP) rate is missing, fail loudly or return 0 with a warning — never fall back to `1.0` or the wrong currency. (Not a numbered CLAUDE.md invariant, but hard-won; treat as one.)
|
||||
- **Incidents**: `6b7d19b` — `_to_eur` silently fell back to treating USD as EUR; `81091fc` — removed a `1.0` FX fallback in the price fetcher, skipping the upsert instead; `9b78d0e` — GBP fetch error handling aligned with USD; `424b926` / `bc256f6` — frontend hardcoded EUR instead of deriving currency from the selected account; `bc256f6` also made the remaining `1.0` fallback in `calculate_net_worth` log a warning.
|
||||
- **If violated**: net worth is off by the full FX spread and nobody notices until a snapshot chart looks weird.
|
||||
|
||||
### 3.8 The remaining "What NOT to do" rules and their grounding
|
||||
|
||||
| Rule (CLAUDE.md) | Grounding |
|
||||
|---|---|
|
||||
| No `async` SQLAlchemy sessions | Design decision, PLAN.md: aiosqlite adds a threading layer anyway; FastAPI threadpools sync handlers. |
|
||||
| No raw DDL / no hand-edits to the SQLite file; Alembic only | §2 + §4. `Base.metadata.create_all` will NOT fix drift (§4.3). |
|
||||
| No new model columns without a migration | Same — `create_all` never alters existing tables, so prod silently lacks the column while tests (which create fresh) pass. |
|
||||
| Never bypass `web/src/api/client.ts` | Single API layer; the auth work (`082b608`) shows why: headers/auth must be changeable in one place. |
|
||||
| No auth/multi-user without understanding single-user assumptions | `082b608` added opt-in bearer auth and CLAUDE.md documents the trap: setting `FIN_API_TOKEN` on the API locks out the React frontend, which sends no `Authorization` header. |
|
||||
| Never call `fetch_all_prices` synchronously on the hot path | `ffa28bb` removed a blocking startup fetch; `POST /api/prices/refresh` uses FastAPI `BackgroundTasks` (`api/app/routers/prices.py:20-22`). It does blocking HTTP with retry sleeps. |
|
||||
| Use local dates, not UTC `toISOString()` | `bc256f6`: UTC dates booked trades/vests/settles to *yesterday* before ~1–2 AM local; frontend uses a `todayLocalISO()` helper. |
|
||||
|
||||
## 4. Alembic migration discipline
|
||||
|
||||
### 4.1 Workflow (exact)
|
||||
|
||||
```bash
|
||||
cd api
|
||||
# Point alembic at a scratch DB, never at anything real (env.py reads DATABASE_URL first):
|
||||
export DATABASE_URL=sqlite:////tmp/mig-scratch.db
|
||||
alembic upgrade head # bring scratch DB to current head
|
||||
# 1. Edit app/models.py
|
||||
alembic revision --autogenerate -m "add X to Y"
|
||||
# 2. READ the generated file. Autogenerate misses server_default intent, index
|
||||
# names, and SQLite batch requirements. Fix by hand.
|
||||
alembic upgrade head # 3. Prove it applies
|
||||
alembic downgrade -1 && alembic upgrade head # 4. Prove downgrade at least runs
|
||||
# 5. Commit the migration together with the model change.
|
||||
```
|
||||
|
||||
(Verified statically: `api/alembic/env.py` resolves the URL as `DATABASE_URL` env → `-x db_url` → `app.db_config.get_database_url()`; the `alembic.ini` fallback is `sqlite:///./fin.db`, which is why you must export `DATABASE_URL` — otherwise you create a stray `api/fin.db`. This sandbox cannot execute alembic.)
|
||||
|
||||
### 4.2 The chain (as of 2026-07-05, head = `h8c9d0e1f2a3`)
|
||||
|
||||
| Revision | down_revision | What |
|
||||
|---|---|---|
|
||||
| `a1b2c3d4e5f6` | *(None)* | initial schema |
|
||||
| `b2c3d4e5f6a7` | a1b2c3d4e5f6 | transaction indexes |
|
||||
| `c3d4e5f6a7b8` | b2c3d4e5f6a7 | missing indexes |
|
||||
| `d4e5f6a7b8c9` | c3d4e5f6a7b8 | rsu_grants table |
|
||||
| `e5f6a7b8c9d0` | d4e5f6a7b8c9 | `linked_transaction_id` (invariant 3.1) |
|
||||
| `f6a7b8c9d0e1` | e5f6a7b8c9d0 | `categories.type`, `server_default='expense'` |
|
||||
| `g7b8c9d0e1f2` | f6a7b8c9d0e1 | `price_cache.name` |
|
||||
| `h8c9d0e1f2a3` | g7b8c9d0e1f2 | data migration: retype internal cash flows (invariant 3.6) |
|
||||
|
||||
Single linear chain, no branches. A new revision's `down_revision` must be the current head — run `alembic heads` (or read the files) before generating.
|
||||
|
||||
### 4.3 The `create_all` gotcha
|
||||
|
||||
`api/app/main.py:19` calls `Base.metadata.create_all(bind=engine)` in the lifespan. This is **first-boot convenience only** (fresh dev DBs, tests) — it creates missing *tables* but never alters existing ones. It is NOT a migration path. In prod the Dockerfile CMD runs `alembic upgrade head` before uvicorn ever starts, so migrations are the real mechanism. Consequence: a model change without a migration works in tests (fresh DB via `create_all`) and fails in prod — the most dangerous drift in this repo.
|
||||
|
||||
### 4.4 Data-migration template
|
||||
|
||||
`h8c9d0e1f2a3_retype_internal_cash_flows.py` is the house pattern: docstring explains the semantic bug and the fix; `op.get_bind()` + `sa.text()` for surgical `UPDATE`s; a compiled regex to avoid retyping user-entered rows that merely look similar (`Buy groceries` vs generated `Buy 5 VWCE`); an honest no-op `downgrade()` with a comment stating the original typing is not restored.
|
||||
|
||||
## 5. Spec → plan → implement workflow
|
||||
|
||||
Big changes get a written spec and plan **before** implementation, stored in-repo:
|
||||
|
||||
- Spec: `docs/superpowers/specs/YYYY-MM-DD-slug.md`
|
||||
- Plan: `docs/superpowers/plans/YYYY-MM-DD-slug-plan.md` (same slug + `-plan`)
|
||||
|
||||
**When required** (any one triggers it): a schema change coupled to frontend work; any accounting-logic change (§1); any change touching ≥3 files across both `api/` and `web/`; any redesign of an existing flow's semantics. Precedent trail: spec `e86df27` → plan `196bef0` → implementation commits `07ad335`…`062ba52`.
|
||||
|
||||
**Spec format** (mirror `2026-07-01-category-types-receivable-redesign.md`): title `# Design: …`, `Date:` line, `## Overview` with numbered goals, one `##` section per goal covering data model, migration notes (including `server_default` decisions and what the migration must NOT touch), backend schema/model deltas, and UI behavior.
|
||||
|
||||
**Plan format** (mirror `2026-07-01-category-types-receivable-redesign-plan.md`): goal/architecture/tech-stack header; a `## Global Constraints` section that *restates the relevant invariants from §3*; a `## File Map` table (file → change); then `## Task N` sections with `- [ ]` checkboxes, each listing files, interfaces produced, and verification steps.
|
||||
|
||||
Specs and plans are **immutable once implemented** — they are the historical record; corrections go in a new dated spec.
|
||||
|
||||
## 6. Docs of record — what lives where
|
||||
|
||||
| Document | Role | Must be updated when |
|
||||
|---|---|---|
|
||||
| `CLAUDE.md` | Agent rulebook: conventions, invariants, structure | A new invariant is learned (add to "Key invariants" with rationale); a convention changes; a top-level directory is added (e.g. `cli/` in `082b608`) |
|
||||
| `README.md` | Human quickstart + config reference | New env var → row in the "Environment / Config" table; new user-facing feature → Features list |
|
||||
| `PLAN.md` | Original design doc — **frozen historical reference** (only ever touched at repo birth, `a9de3c8`) | Never retrofit it. Where reality diverged, CLAUDE.md wins. |
|
||||
| `docs/superpowers/specs/`, `plans/` | Per-change design docs, dated, immutable | Every change that meets the §5 threshold |
|
||||
| `.claude/skills/fin/SKILL.md` | Agent-facing `fin` CLI usage | CLI commands or flags change |
|
||||
| `.claude/skills/fin-*/SKILL.md` | Process/knowledge skills (this library) | The process itself changes |
|
||||
|
||||
**Commit message house style** (mined from `git log`, as of 2026-07-05 — 41 `fix:`, 33 `feat:`, 5 `chore:`, 2 `docs:`, plus stray `perf:`/`polish:`):
|
||||
|
||||
- Subject: `type: imperative summary`, lower-case after the prefix, no trailing period; em-dash sub-clauses for multi-part fixes (`fix: bug hunt findings — currency derivation, local dates, …`).
|
||||
- Body for anything non-trivial: bulleted, grouped under `Backend:` / `Frontend:` (see `e36d719`, `bc256f6`); each bullet names the file/function and the user-visible effect.
|
||||
- Migration-bearing commits mention the revision id in the body (`Migration h8c9d0e1f2a3 retypes historical rows the same way`).
|
||||
- Agent-authored commits carry `Co-Authored-By:` and `Claude-Session:` trailers (observed on `082b608`, `e36d719`).
|
||||
- Doc commits: `docs: spec for …` then `docs: implementation plan for …` as separate commits before the feature commits.
|
||||
|
||||
## 7. Done-checklist — run before declaring any change complete
|
||||
|
||||
Copy-paste block (verified statically; this sandbox has no pip/docker — expect them to work in a normal dev env with `pip install -e cli/` + api requirements installed):
|
||||
|
||||
```bash
|
||||
# 1. Syntax check (stdlib only, no deps needed)
|
||||
python -m compileall -q api/app cli/fin_cli
|
||||
# 2. E2E suite: FinClient → real app via TestClient → throwaway SQLite
|
||||
python -m pytest cli/tests -q
|
||||
# 3. Frontend type-check + build (package.json: "build": "tsc && vite build")
|
||||
cd web && npm run build && cd ..
|
||||
# 4. If a migration exists: dry-run on a scratch DB
|
||||
cd api && DATABASE_URL=sqlite:////tmp/mig-check.db alembic upgrade head && cd ..
|
||||
```
|
||||
|
||||
Then the manual gate:
|
||||
|
||||
- [ ] Change classified per §1 and its full gate satisfied
|
||||
- [ ] No single-leg transfer/settlement create/update/delete anywhere in the diff (§3.1)
|
||||
- [ ] Any `stock_transactions` mutation recomputes holdings before commit (§3.2)
|
||||
- [ ] No new balance query summing `amount` without `- fee` (§3.5)
|
||||
- [ ] Spending/income queries still exclude `transfer`/`settlement` rows; splitwise net-share math untouched or spec'd (§3.6)
|
||||
- [ ] No `1.0`/wrong-currency FX fallback introduced (§3.7)
|
||||
- [ ] No `async` session, no raw DDL, no `fetch()` outside `web/src/api/client.ts`
|
||||
- [ ] Migration reviewed by hand: backward-safe (nullable/`server_default`), correct `down_revision` = current head, honest `downgrade()` (§4)
|
||||
- [ ] New behavior has a test in `cli/tests/` (the API test-coverage gap is the project's known weak point — never widen it; see `fin-test-coverage-campaign`)
|
||||
- [ ] CLAUDE.md updated if an invariant/convention changed; README env table updated for new config vars; spec/plan written if §5 applies
|
||||
- [ ] If deploy-affecting: prod DB backed up (§2.3) before the image restarts and auto-migrates (§2.4)
|
||||
- [ ] Commit message follows §6 style
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All claims verified against the repo on **2026-07-05** (commit `082b608` at HEAD). Commands were verified statically where noted (no pip/venv/docker in the authoring sandbox). One-line re-verification for anything that may drift:
|
||||
|
||||
```bash
|
||||
git -C /path/to/fin log --oneline -5 # HEAD moved? re-check volatile facts
|
||||
ls api/alembic/versions/ && grep -h "down_revision" api/alembic/versions/*.py # §4.2 chain + head
|
||||
grep -n "create_all" api/app/main.py # §4.3 gotcha still at lifespan
|
||||
grep -n "CMD" api/Dockerfile # §2.4 auto-migrate on start
|
||||
grep -n "useradd" api/Dockerfile # §3.4 non-root user
|
||||
grep -n "sqlite_data\|DATABASE_URL" docker-compose.yml # §2 volume + URL
|
||||
grep -n "add_all" api/app/crud.py # §3.1 atomic legs
|
||||
grep -n "def delete_stock_transaction" -A 25 api/app/crud.py | grep recompute # §3.2
|
||||
grep -rn "journal_mode=WAL" api/app # §3.3 both engines
|
||||
grep -n "Transaction.amount - " api/app/services/net_worth.py # §3.5 fee formula
|
||||
grep -n "def get_monthly_spending" api/app/crud.py # §3.6
|
||||
grep -n '"build"' web/package.json # §7 build command
|
||||
git -C /path/to/fin log --format='%s' | grep -oE '^[a-z]+:' | sort | uniq -c # §6 prefix stats
|
||||
ls docs/superpowers/specs docs/superpowers/plans # §5 naming convention
|
||||
git show a06f921 a1f109b bc256f6 e36d719 1480088 ffa28bb 6b7d19b 81091fc --stat # incident evidence
|
||||
```
|
||||
|
||||
If any command's output contradicts this file, update this file — CLAUDE.md and the repo are the ground truth, this skill is the map.
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
name: fin-debugging-playbook
|
||||
description: Symptom-to-cause triage playbook for the fin app. Load when anything looks wrong and the cause is unknown — balances look wrong, spending or income totals too high/inflated, net worth off or in the wrong currency, prices stale or missing, holdings don't match trades, dates off by one day, 401s or frontend can't reach the API, container won't start or is unhealthy — or for any bug triage/debugging session in fin.
|
||||
---
|
||||
|
||||
# fin Debugging Playbook
|
||||
|
||||
Symptom → likely cause → discriminating check, for this codebase's *actual* historical failure modes. Every claim below is grounded in code or a fix commit (SHAs cited). Facts date-stamped 2026-07-05; re-verify per "Provenance" at the bottom.
|
||||
|
||||
**Jargon (defined once):**
|
||||
- **Leg**: one of the two (or three) `transactions` rows written together for a transfer/settlement/splitwise-pay/split-expense. Linked via `linked_transaction_id` (both directions) and `linked_account_id`.
|
||||
- **Orphan leg**: a leg whose partner row was deleted — corrupts both account balances.
|
||||
- **price_cache**: one row per symbol (tickers + `EUR/USD`, `EUR/GBP`), upserted; `fetched_at` is UTC.
|
||||
- **Recompute**: `recompute_holding_from_stock_transactions` in `api/app/crud.py` — holdings are derived state, rebuilt from the full `stock_transactions` history for that account+ticker.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Recording/querying finances via CLI → sibling `fin` skill (`.claude/skills/fin/SKILL.md`).
|
||||
- Making a code change safely → `fin-change-control`; architecture questions → `fin-architecture-contract`.
|
||||
- Accounting semantics (what *should* the numbers be) → `fin-accounting-reference`; ad-hoc ledger SQL → `fin-ledger-analysis-toolkit`.
|
||||
- Build/deploy/env setup → `fin-build-and-env`; day-2 ops (not active bug hunt) → `fin-run-and-operate`.
|
||||
- Deep history of past incidents → `fin-failure-archaeology`; tooling setup → `fin-diagnostics-and-tooling`.
|
||||
- Writing regression tests after the fix → `fin-test-coverage-campaign`, `fin-validation-and-qa`.
|
||||
|
||||
## Rule zero: reproduce on a copy — never debug against prod SQLite
|
||||
|
||||
The production DB (Docker volume `sqlite_data`, file `/data/fin.db` inside the `api` container — see `docker-compose.yml`) is sacred. Copy it out first; all SQL below runs against the copy. The DB runs in WAL mode, so a naive file copy can miss un-checkpointed writes in `fin.db-wal` — use `VACUUM INTO`, which produces a consistent single-file snapshot. Verified statically against `docker-compose.yml` and `api/Dockerfile` (python:3.12-slim has stdlib `sqlite3`; `/data` is writable by user `app`); run on the homelab host:
|
||||
|
||||
```bash
|
||||
# 1. Consistent snapshot inside the container (read-only w.r.t. live data)
|
||||
docker compose exec api python -c "import sqlite3; sqlite3.connect('/data/fin.db').execute(\"VACUUM INTO '/data/fin-debug-copy.db'\")"
|
||||
# 2. Copy it out and clean up
|
||||
docker compose cp api:/data/fin-debug-copy.db ./fin-debug-copy.db
|
||||
docker compose exec api rm /data/fin-debug-copy.db
|
||||
# 3. Query the copy locally
|
||||
sqlite3 ./fin-debug-copy.db "SELECT count(*) FROM transactions;"
|
||||
# 4. Optionally run a whole local API against the copy (from api/):
|
||||
DATABASE_URL=sqlite:///$PWD/../fin-debug-copy.db uvicorn app.main:app --port 8001
|
||||
```
|
||||
|
||||
Caveat for step 4: the app starts APScheduler and a 5-min price job that *writes* — fine against the copy, never point it at the real file. Destructive experiments (deletes, migrations) go against the copy only.
|
||||
|
||||
## First 15 minutes: "something is wrong and I don't know what"
|
||||
|
||||
Run in order; stop when a step discriminates.
|
||||
|
||||
1. **Is the API up?** `curl -fsS http://<host>/api/health` — always unauthenticated (`api/app/main.py`). If this fails, jump to the *Container* family.
|
||||
2. **Container state + logs** (on homelab host): `docker compose ps` then `docker compose logs --tail 100 api`. Look for: alembic errors at start, `No EUR/... rate in price cache` warnings, yfinance retry storms, `database is locked`.
|
||||
3. **Price freshness** (via API, no DB needed): `fin prices fx --json` and check `fetched_at` in the copy:
|
||||
`sqlite3 fin-debug-copy.db "SELECT symbol, price, currency, fetched_at FROM price_cache ORDER BY fetched_at;"` — anything older than ~10 min means the 5-min scheduler job is not running (times are UTC).
|
||||
4. **Orphan legs** (corrupt balances):
|
||||
```sql
|
||||
SELECT t.id, t.date, t.type, t.amount, t.account_id FROM transactions t
|
||||
WHERE t.linked_transaction_id IS NOT NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM transactions p WHERE p.id = t.linked_transaction_id);
|
||||
```
|
||||
Also count legacy unpaired legs: `SELECT count(*) FROM transactions WHERE type IN ('transfer','settlement') AND linked_transaction_id IS NULL;`
|
||||
5. **What changed recently?** `git log --oneline -10` and `git diff HEAD~3 -- api/app/crud.py api/app/services/` — most fin bugs shipped in the last few commits.
|
||||
6. **Cross-check a number two ways**: `fin networth --json | jq '.total_eur'` vs summing the copy yourself (see Balance family) — tells you whether data or calculation is wrong.
|
||||
|
||||
## Master symptom table
|
||||
|
||||
| Symptom | Likely cause (most→least) | Discriminating check |
|
||||
|---|---|---|
|
||||
| Spending/income totals inflated | Internal cash flow typed `expense`/`income` (stock buy leg, receivable settle) — e36d719; opening-balance income; missing `splitwise_partner_share` | SQL S1/S2 below |
|
||||
| Spending "off by partner's share" | Not a bug: `splitwise_pay` counts `amount + splitwise_partner_share`; I-Owe counts as spending by design (e61b254) | Read invariant 6 in CLAUDE.md before "fixing" |
|
||||
| One account balance wrong | Orphan leg; `fee` ignored (balance = SUM(amount−fee)); `manual_override_balance` set; forgotten "Opening balance" row | SQL B1/B2/B3 below |
|
||||
| Net worth total wrong / a currency ignored | Missing `EUR/<CCY>` row in price_cache → asset contributes **0** (6b7d19b); EUR/USD missing → rate falls back 1.0; override in wrong currency (37bc101) | `grep 'No EUR/' logs`; SQL N1 |
|
||||
| Prices stale/missing | Frankfurter URL rot (582e39d); yfinance down (stale cache served); Euro ETF needs exchange suffix (bc8c8d5); scheduler dead | `fetched_at` query; P1 below |
|
||||
| Transactions dated yesterday | UTC vs local date: `toISOString()` before ~1–2 AM CET (bc256f6, `todayLocalISO` in `web/src/lib/utils.ts`) | Grep check D1 |
|
||||
| Holdings ≠ trade history | Recompute skipped/failed; vested-grant FK deadlock (bc256f6) | SQL H1 |
|
||||
| 401s / frontend broken but API "works" | `FIN_API_TOKEN` set on API env — locks out React frontend (sends no Authorization header) | `curl /api/accounts` with vs without Bearer |
|
||||
| Container won't start / unhealthy | Alembic migration failure in CMD; healthcheck failing; `web` blocked on `api` health; WAL lock from external reader | `docker compose logs api` first 30 lines |
|
||||
|
||||
## Family runbooks
|
||||
|
||||
### 1. Spending/income totals look inflated or wrong
|
||||
|
||||
Ground truth (`get_monthly_spending`, `api/app/crud.py` ~line 624):
|
||||
- Spending = rows with `amount < 0` and type `expense`/`splitwise_receive` (raw amount) + `splitwise_pay` (counted as `amount + splitwise_partner_share`, the user's net share). `transfer` and `settlement` are excluded.
|
||||
- Accounts of type `receivable` are excluded; **`payable` is included on purpose** (I-Owe is real consumption — e61b254 deliberately reverted part of a9f045f).
|
||||
- Income = type `income`, excluding `payable`/`receivable` accounts and rows with description `Opening balance`.
|
||||
|
||||
Discriminating checks (against the copy):
|
||||
```sql
|
||||
-- S1: internal flows mistyped as expense/income (the e36d719 bug class).
|
||||
-- Any hit = data predating migration h8c9d0e1f2a3 or a new regression.
|
||||
SELECT id, date, type, amount, description FROM transactions
|
||||
WHERE type IN ('expense','income') AND linked_account_id IS NOT NULL;
|
||||
|
||||
-- S2: splitwise_pay rows missing partner share (counted at full amount)
|
||||
SELECT id, date, amount, splitwise_partner_share FROM transactions
|
||||
WHERE type='splitwise_pay' AND amount < 0
|
||||
AND (splitwise_partner_share IS NULL OR splitwise_partner_share = 0);
|
||||
```
|
||||
API cross-check: `fin spending month 2026-06 --json | jq '.total, .income_total'` and eyeball big categories with `fin tx list --month 2026-06 --json | jq '[.[] | select(.type=="expense")] | sort_by(.amount) | .[0:5]'`.
|
||||
|
||||
### 2. Balance looks wrong
|
||||
|
||||
Ground truth: for bank/splitwise/receivable/payable accounts, balance = `SUM(amount - fee)` over all its transactions (`calculate_net_worth` in `api/app/services/net_worth.py`). `manual_override_balance` on the account *silently replaces* the computed balance.
|
||||
|
||||
```sql
|
||||
-- B1: recompute the balance yourself and compare with the UI
|
||||
SELECT a.id, a.name, a.type, a.manual_override_balance,
|
||||
ROUND(COALESCE(SUM(t.amount - t.fee),0), 2) AS computed
|
||||
FROM accounts a LEFT JOIN transactions t ON t.account_id = a.id
|
||||
WHERE a.is_active = 1 GROUP BY a.id;
|
||||
|
||||
-- B2: orphan legs touching a specific account (see First-15-min step 4 for global)
|
||||
SELECT * FROM transactions t
|
||||
WHERE t.account_id = :acct AND t.linked_transaction_id IS NOT NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM transactions p WHERE p.id = t.linked_transaction_id);
|
||||
|
||||
-- B3: overrides and opening balances that explain "mystery" amounts
|
||||
SELECT id, name, manual_override_balance FROM accounts WHERE manual_override_balance IS NOT NULL;
|
||||
SELECT id, account_id, date, amount FROM transactions WHERE description = 'Opening balance';
|
||||
```
|
||||
Decision: B1 computed ≠ UI → override or FX (family 3). B1 computed ≠ your expectation → B2 orphans, then fee column (`SELECT id, amount, fee FROM transactions WHERE account_id=:acct AND fee != 0;`).
|
||||
|
||||
### 3. Net worth wrong / currency wrong
|
||||
|
||||
Ground truth (`api/app/services/net_worth.py`):
|
||||
- `_to_eur` converts via `price_cache` row `EUR/<CCY>`. **If the rate row is missing, the asset contributes 0.0 to net worth** with a log warning `No EUR/<CCY> rate in price cache` (6b7d19b — deliberately no wrong-currency fallback).
|
||||
- If `EUR/USD` itself is missing, the *reported rate* falls back to 1.0 with a warning — and the same 1.0 fallback (`net_worth._get_eur_usd_rate`) is fed into funded-buy conversions at write time (gap U8, fin-ledger-analysis-toolkit), so a cross-currency funded buy booked while the cache was cold may carry a wrong amount permanently.
|
||||
- Investment/RSU `manual_override_balance` is converted assuming the account's declared currency (37bc101).
|
||||
- LSE pence: fetcher converts `GBp`/`GBX` → price/100, currency `GBP` (`price_fetcher.py`).
|
||||
|
||||
```sql
|
||||
-- N1: which FX rates exist, and how old
|
||||
SELECT symbol, price, currency, fetched_at FROM price_cache WHERE symbol LIKE 'EUR/%';
|
||||
-- N2: currencies in use that need a rate
|
||||
SELECT DISTINCT currency FROM accounts WHERE is_active=1
|
||||
UNION SELECT DISTINCT currency FROM price_cache WHERE symbol NOT LIKE 'EUR/%';
|
||||
```
|
||||
If an account/holding currency appears in N2 but has no `EUR/<CCY>` row in N1, that asset is being valued at €0. Check `docker compose logs api | grep 'No EUR/'`. Per-account drill-down: `fin networth --json | jq '.accounts[] | {name, currency, balance_native, balance_eur, is_override}'` — `balance_native` nonzero with `balance_eur` 0 is the smoking gun.
|
||||
|
||||
### 4. Prices stale or missing
|
||||
|
||||
Ground truth (`api/app/services/price_fetcher.py`, `scheduler.py`, as of 2026-07-05):
|
||||
- FX from `https://api.frankfurter.dev/v1/latest?from=EUR&to=USD` (and `to=GBP`). The old `api.frankfurter.app` host was retired — 582e39d. If FX dies again, suspect URL rot first.
|
||||
- Tickers via yfinance: tries bare ticker, then suffixes `.DE .L .AS .PA .MI .SW .F .VI` (bc8c8d5); price cached under the *original* symbol. 3 retries/candidate with 1/2/4 s sleeps.
|
||||
- On total failure: last cached price is served; `0.0` only when no cache row exists at all.
|
||||
- Scheduler (`0b5ba7c`): `price_refresh` interval job every 5 min + `daily_snapshot` cron at midnight, persisted in table `apscheduler_jobs`, running on a *private* engine.
|
||||
- Tracked = holdings with shares > 0 plus tickers of pending RSU grants.
|
||||
|
||||
Checks:
|
||||
```bash
|
||||
# P1: is anything updating? (fetched_at is UTC; should be < ~10 min old)
|
||||
sqlite3 fin-debug-copy.db "SELECT symbol, fetched_at FROM price_cache ORDER BY fetched_at DESC;"
|
||||
docker compose logs --tail 200 api | grep -E "Price refresh|Failed to fetch|All candidates|retrying"
|
||||
# P2: force one ticker end-to-end (writes to price_cache — prod-safe by design)
|
||||
fin prices fetch VWCE --json
|
||||
# P3: full refresh runs as a 202 background task (POST /api/prices/refresh)
|
||||
fin prices refresh
|
||||
```
|
||||
Never call `fetch_all_prices` synchronously in a request handler — it blocks and sleeps (CLAUDE.md).
|
||||
|
||||
### 5. Dates off by one day
|
||||
|
||||
Story: `new Date().toISOString().split('T')[0]` returns the **UTC** date; in UTC+1/+2 anything entered before ~1–2 AM was booked to *yesterday*. Fixed by `todayLocalISO()` in `web/src/lib/utils.ts` (bc256f6). Backend stores plain dates; only frontend default-date derivation was affected.
|
||||
|
||||
```bash
|
||||
# D1: any regressions? Should return only the explanatory comment in utils.ts.
|
||||
grep -rn "toISOString" /home/markuseisl/Documents/git/fin/web/src
|
||||
```
|
||||
If a user reports one specific wrong-dated row, fix the data with `fin tx update <id> --date YYYY-MM-DD` (allowed — it's not a linked leg... verify first: updates on legs with `linked_transaction_id` are rejected by the API, a1289f6).
|
||||
|
||||
### 6. Holdings don't match trades
|
||||
|
||||
Ground truth (`recompute_holding_from_stock_transactions`, `api/app/crud.py`): replay all `stock_transactions` for account+ticker in date,id order; buy/rsu_vest add shares and cost (incl. fee); sell removes shares at average cost; oversell raises → 422 and rollback. Delete triggers full recompute *before* commit; deleting a vest transaction also reverts the linked RSU grant to pending (bc256f6 — previously a FK deadlock made vested grants undeletable).
|
||||
|
||||
```sql
|
||||
-- H1: expected shares from the log vs the holdings row
|
||||
SELECT st.account_id, st.ticker,
|
||||
ROUND(SUM(CASE WHEN lower(st.type) IN ('buy','rsu_vest') THEN st.shares ELSE -st.shares END), 6) AS expected,
|
||||
h.shares AS stored
|
||||
FROM stock_transactions st
|
||||
LEFT JOIN holdings h ON h.account_id = st.account_id AND h.ticker = st.ticker
|
||||
GROUP BY st.account_id, st.ticker
|
||||
HAVING ABS(COALESCE(stored,0) - expected) > 1e-6;
|
||||
```
|
||||
Mismatch → trigger a recompute by the supported path: delete + re-add any one stock transaction for that ticker via `fin stock rm` / `fin stock add` (on a copy first). Never edit `holdings` rows directly.
|
||||
|
||||
### 7. 401s / frontend can't reach API
|
||||
|
||||
Ground truth: `api/app/auth.py` + `main.py`. Auth is **opt-in**: enforced only when env `FIN_API_TOKEN` is set on the API container, checked per-request (rotatable without restart... of code; the env var itself needs a container recreate). `/api/health` is always open; everything else, including `/api/openapi.json`, requires `Authorization: Bearer <token>`. The React frontend sends **no** Authorization header (`web/src/api/client.ts`) — setting the token locks the UI out until nginx injects the header. Nginx (`web/nginx.conf`) proxies `/api/` to `${API_UPSTREAM}` (default `api:8000`, set in `web/Dockerfile` and `docker-compose.yml`).
|
||||
|
||||
```bash
|
||||
curl -fsS http://<host>/api/health # up at all?
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://<host>/api/accounts # 401 → token enforced
|
||||
curl -s -H "Authorization: Bearer $FIN_API_TOKEN" http://<host>/api/accounts | head -c 200
|
||||
docker compose exec api printenv FIN_API_TOKEN # is it set on the container?
|
||||
```
|
||||
502 from nginx instead of 401 → `API_UPSTREAM` wrong or api container down.
|
||||
|
||||
### 8. Container won't start / unhealthy
|
||||
|
||||
Ground truth: `api/Dockerfile` CMD is `python -m app.preflight && alembic upgrade head && uvicorn ...` — a failing migration crash-loops the container *before* the app exists. Healthcheck (docker-compose.yml) curls `/api/health` every 30 s, 3 retries; `web` has `depends_on: api: condition: service_healthy`, so a sick api also keeps web down. Runs as non-root user `app` (uid 1000). Migration chain head as of 2026-07-05: `h8c9d0e1f2a3` (`api/alembic/versions/`).
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs --tail 50 api # alembic tracebacks appear here first
|
||||
docker compose exec api alembic current # applied revision vs files in api/alembic/versions/
|
||||
docker compose exec api ls -la /data # fin.db + -wal + -shm owned by app (uid 1000)?
|
||||
```
|
||||
`database is locked` in logs → something outside the two engines is holding the file (e.g. a host `sqlite3` shell on the live DB — that's exactly why you debug on a copy). Both the request engine and the scheduler's private engine set `PRAGMA journal_mode=WAL` and `foreign_keys=ON` per connection (`database.py`, `scheduler.py`).
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **"Spending is too high" that wasn't a spending bug** (e36d719): stock buys' bank legs were typed `expense` and receivable/payable reconciles typed `income`/`expense`, inflating both metrics for weeks. The fix retyped them `transfer`/`settlement` and shipped migration `h8c9d0e1f2a3` to rewrite history. Lesson: when a total looks wrong, first ask *what types feed this aggregate* (run S1), not *is the arithmetic wrong*.
|
||||
- **The payable flip-flop** (a9f045f → e61b254): payable accounts were first excluded from spending together with receivables, then deliberately re-included one commit later — money you owe and spent via I-Owe *is* consumption. Sonnet-class agents keep wanting to "fix" this. Don't; it's invariant 6 territory.
|
||||
- **Orphan legs from single-leg deletes** (a06f921, a1f109b, 9d06e1b): deleting one transfer leg silently skewed two balances. Fixes added paired delete, then `linked_transaction_id` for precise pairing (the old account-pair heuristic could grab a leg of a *different* transfer between the same accounts), then frontend refetch-after-delete. Legacy rows without `linked_transaction_id` still exist — the fallback matcher in `delete_transaction` handles them.
|
||||
- **Silent wrong-currency conversion** (6b7d19b): `_to_eur` used to fall back to the EUR/USD rate for *any* currency, quietly mis-valuing GBP assets. Now a missing rate excludes the asset (0 €) with a warning — so "asset disappeared from net worth" is usually a *missing FX row*, not lost data.
|
||||
- **FX provider URL rot** (582e39d): frankfurter moved to `api.frankfurter.dev/v1`; until fixed, the app served an ever-staler cached rate with no user-visible error. Stale-graceful design hides upstream breakage — check `fetched_at`, not just whether numbers exist.
|
||||
- **European ETFs invisible to yfinance** (bc8c8d5): bare ISIN-ish tickers (e.g. VWCE) return no data on Yahoo without an exchange suffix; the fetcher now walks `.DE .L .AS ...`. A ticker priced at 0 with "All candidates exhausted" in logs may simply trade under a suffix not in the list.
|
||||
- **Midnight-minus-two hours dates** (bc256f6): every QuickAdd flow defaulted its date via `toISOString()`, so late-evening entries landed on the previous day for UTC+ users. Grep D1 whenever a new date input is added.
|
||||
- **Undeletable vested grants** (bc256f6): deleting a vest's stock transaction hit the `vested_stock_transaction_id` FK (500), while deleting the grant was blocked (422) — a deadlock. `delete_stock_transaction` now reverts the grant to pending first.
|
||||
- **Auth that "worked in curl"** (082b608): enabling `FIN_API_TOKEN` broke only the browser UI, because the frontend sends no Authorization header. Test both paths after touching auth.
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
Verified against the working tree at commit 082b608 (2026-07-05). Docker/CLI commands verified statically (no docker/pip in authoring sandbox). Re-verify one-liners:
|
||||
|
||||
- Spending rules: `grep -n "notin_\|splitwise_partner_share\|Opening balance" api/app/crud.py`
|
||||
- Balance formula & _to_eur: `grep -n "amount - \|return 0.0" api/app/services/net_worth.py`
|
||||
- Frankfurter URL & suffixes: `grep -n "frankfurter\|_EXCHANGE_SUFFIXES" api/app/services/price_fetcher.py`
|
||||
- Scheduler cadence: `grep -n "minutes=5\|hour=0" api/app/services/scheduler.py`
|
||||
- Auth wiring: `grep -n "FIN_API_TOKEN" api/app/auth.py; grep -n "require_api_token" api/app/main.py`
|
||||
- Container CMD/healthcheck/volume: `grep -n "CMD" api/Dockerfile; grep -n "healthcheck\|sqlite_data\|API_UPSTREAM" -A2 docker-compose.yml`
|
||||
- Migration head: `ls api/alembic/versions/`
|
||||
- Date helper: `grep -rn "todayLocalISO" web/src/lib/utils.ts`
|
||||
- CLI commands: `grep -n "add_typer\|@app.command" cli/fin_cli/main.py`
|
||||
@@ -0,0 +1,295 @@
|
||||
---
|
||||
name: fin-diagnostics-and-tooling
|
||||
description: Read-only diagnostic scripts and API probes for the fin app. Load when you need to verify data integrity, quantify a suspected bug with real numbers, audit the ledger (linked transfer/settlement legs, holdings vs stock transactions, account balances), check price_cache or net-worth-snapshot freshness, independently recompute a month's spending, or gather before/after evidence around a change. The measurement layer — measure, don't eyeball.
|
||||
---
|
||||
|
||||
# fin — diagnostics and tooling
|
||||
|
||||
Five tested, stdlib-only, READ-ONLY Python scripts plus API probes that turn
|
||||
"the numbers look wrong" into a precise, diffable anomaly list. Every script:
|
||||
|
||||
- takes the SQLite DB path as `argv[1]` and opens it with
|
||||
`sqlite3.connect("file:...?mode=ro", uri=True)` — it **cannot** write
|
||||
- exits **0 = clean, 1 = anomalies found** (exception: `spending_recompute.py`
|
||||
is a recompute tool, exit 0 on success)
|
||||
- supports `--json` for machine-diffable output
|
||||
- needs only `/usr/bin/python3` (3.11+), no pip, no venv, no docker
|
||||
|
||||
The prod DB is sacred. These scripts are safe to point at a live copy, but
|
||||
prefer a copy anyway (below). Never run anything that writes against prod.
|
||||
|
||||
## Get a safe DB copy first
|
||||
|
||||
```bash
|
||||
# from the homelab host running the stack (full runbook: fin-run-and-operate)
|
||||
docker compose exec api python3 -c "import sqlite3; s=sqlite3.connect('file:/data/fin.db?mode=ro', uri=True); d=sqlite3.connect('/tmp/fin-copy.db'); s.backup(d); d.close()"
|
||||
docker compose cp api:/tmp/fin-copy.db ./fin-copy.db
|
||||
```
|
||||
|
||||
`sqlite3 .backup` snapshots WAL databases consistently. Note: a `mode=ro`
|
||||
open of a *live* WAL DB can fail if the `-wal`/`-shm` files need recovery
|
||||
and aren't writable — if that happens, use the backup copy, never loosen the
|
||||
mode. Backup/restore discipline and volume paths: see **fin-run-and-operate**.
|
||||
|
||||
## Diagnostic decision table
|
||||
|
||||
| Question | Tool | Healthy looks like |
|
||||
|---|---|---|
|
||||
| Is the API up / authed? | `curl -s $FIN_API_URL/api/health` (always open) | `{"status":"ok"...}` |
|
||||
| Is net worth plausible right now? | `fin networth --json \| jq '.total_eur, .eur_usd_rate'` | total matches expectation; rate ≈ market EUR/USD, not `1.0` |
|
||||
| Does a transfer/settlement have both legs? | `scripts/audit_linked_legs.py` | exit 0 |
|
||||
| Do holdings match the trade log? | `scripts/audit_holdings.py` | exit 0 |
|
||||
| What is each account's computed balance? Any masking overrides? | `scripts/audit_balances.py` | exit 0; computed values match the UI |
|
||||
| Are prices/FX fresh? Snapshot gaps? | `scripts/check_freshness.py` | exit 0; ticker ages are minutes (5-min scheduler job) |
|
||||
| Is the spending total right for month X? | `scripts/spending_recompute.py` vs `fin spending month X --json` | identical totals per category |
|
||||
| Prices stale in prod, why? | `check_freshness.py` on a copy + api logs (see fin-run-and-operate) | — |
|
||||
| Is the cause a known bug pattern? | not this skill — **fin-debugging-playbook** / **fin-failure-archaeology** | — |
|
||||
|
||||
DB scripts answer "is the stored data consistent with the code's rules?".
|
||||
API probes answer "what is the running system reporting?". A bug report needs
|
||||
both: if the DB audits are clean but the API number is wrong, the bug is in a
|
||||
service/router; if an audit fails, the data itself is damaged.
|
||||
|
||||
## Scripts
|
||||
|
||||
All paths below relative to this skill's `scripts/` directory. All five were
|
||||
tested (as of 2026-07-05) against a synthetic SQLite DB built from the
|
||||
`api/app/models.py` schema, seeded with healthy rows *and* one deliberately
|
||||
broken case per check — each script flagged exactly the broken cases and
|
||||
exited 0 on the clean DB (30/30 assertions passed).
|
||||
|
||||
### audit_linked_legs.py — two-leg integrity
|
||||
|
||||
```bash
|
||||
python3 scripts/audit_linked_legs.py fin-copy.db [--allow-legacy] [--json]
|
||||
```
|
||||
|
||||
Checks every transaction against the pairing rules in `crud.py`
|
||||
(`create_transfer`, `create_settlement`, `create_split_expense`,
|
||||
`create_splitwise_i_paid`, `create_funded_buy`, `delete_transaction`).
|
||||
|
||||
Interpretation:
|
||||
|
||||
- **CLEAN + counts line** — healthy. The counts tell you how many
|
||||
same-currency pairs were amount-checked, how many cross-currency pairs and
|
||||
funded-buy single legs were (correctly) skipped.
|
||||
- **`orphan-link`** — `linked_transaction_id` points at a deleted row. One leg
|
||||
of a pair was removed without the other: the surviving leg inflates/deflates
|
||||
one account's balance. This is invariant #1 damage (see CLAUDE.md).
|
||||
- **`self-link` / `self-account-link`** — a row references itself; corrupt
|
||||
write, never produced by crud.py.
|
||||
- **`amounts-dont-cancel`** — a *same-currency* transfer/settlement pair whose
|
||||
amounts don't sum to 0 (tolerance 0.005; fees live in `fee` and are
|
||||
excluded). One side was edited after creation. Cross-currency pairs are
|
||||
exempt: `create_transfer` writes `to_amount` on the destination leg.
|
||||
- **`non-reciprocal`** — A links to B but B doesn't link back. Deleting B via
|
||||
the API would still find A (delete follows links both ways), but the pair is
|
||||
malformed; expect an edit went through `PATCH /api/transactions`.
|
||||
- **`unlinked-leg`** — transfer/settlement with NULL `linked_transaction_id`.
|
||||
Rows written before that column existed are "legacy"; rerun with
|
||||
`--allow-legacy` to accept legacy rows that have a crud-style fallback match
|
||||
(reversed accounts + same date + same type). A leg that's still flagged with
|
||||
`--allow-legacy` is a genuine orphan.
|
||||
- **`missing-account` / `missing-linked-account`** — leg points at a deleted
|
||||
account id (accounts are normally deactivated, never deleted).
|
||||
- **Never flagged, by design**: funded-buy cash legs (single `transfer` linked
|
||||
to an investment/rsu account — `create_funded_buy` writes no partner row);
|
||||
`splitwise_pay` pair amounts (bank leg = −total, splitwise leg =
|
||||
+partner_share; they must NOT cancel); the split-expense `expense` row's
|
||||
one-way link into its transfer pair.
|
||||
|
||||
### audit_holdings.py — holdings vs stock_transactions
|
||||
|
||||
```bash
|
||||
python3 scripts/audit_holdings.py fin-copy.db [--json]
|
||||
```
|
||||
|
||||
Faithful reimplementation of `crud.recompute_holding_from_stock_transactions`
|
||||
(date-ASC/id-ASC replay; buy/rsu_vest add `shares*price + fee` to cost; sell
|
||||
removes at running average cost, clamped at 0; fractional shares supported),
|
||||
diffed against the `holdings` table.
|
||||
|
||||
Interpretation:
|
||||
|
||||
- **`shares-mismatch` / `cost-basis-mismatch`** (tolerance 1e-6 shares / 0.005
|
||||
cost) — the holding was mutated outside the recompute path, or a recompute
|
||||
was skipped (invariant #2). Fix by re-running the recompute via the API
|
||||
(delete+recreate a stock txn) — never by editing the holding row.
|
||||
- **`missing-holding`** — stock transactions exist but no holding row; the
|
||||
position is invisible to net worth.
|
||||
- **`stale-holding`** — holding row with zero stock transactions; recompute
|
||||
would delete it; it inflates net worth.
|
||||
- **`invalid-history`** — a sell exceeds shares held at that point in the
|
||||
replay. crud rejects this on write (422), so its presence means the history
|
||||
was edited/imported around validation.
|
||||
- **`unknown-stock-txn-type`** — type outside buy/sell/rsu_vest; crud silently
|
||||
ignores such rows, so the holding may still "match" while the ledger carries
|
||||
a dead row.
|
||||
- **WARNING `currency-drift`** (not exit-affecting) — holding currency differs
|
||||
from the first transaction's; crud never updates currency on existing
|
||||
holdings, so this can drift legitimately after deleting the first txn.
|
||||
|
||||
### audit_balances.py — computed balance per account
|
||||
|
||||
```bash
|
||||
python3 scripts/audit_balances.py fin-copy.db [--allow-overrides] [--json]
|
||||
```
|
||||
|
||||
Prints `SUM(amount - fee)` per account (invariant #5 — never `SUM(amount)`),
|
||||
with type, currency, txn count, and any `manual_override_balance`.
|
||||
|
||||
Interpretation:
|
||||
|
||||
- The table itself is the evidence: capture it before/after any change that
|
||||
touches transactions.
|
||||
- **`manual-override`** — the override *silently replaces* the computed value
|
||||
in `services/net_worth.py`. The printed `delta` is how much net worth
|
||||
differs from what the ledger says. Overrides may be intentional (e.g. a
|
||||
hand-set Splitwise balance) — use `--allow-overrides` once confirmed, but
|
||||
a large drifting delta means the ledger under the override has rotted.
|
||||
- **`orphan-transaction`** — txn whose `account_id` no longer exists; it is
|
||||
invisible to every balance and net worth calculation.
|
||||
- Investment/rsu accounts show computed ≈ 0 with 0 txns — normal; their value
|
||||
comes from holdings × price_cache, not transactions.
|
||||
|
||||
### check_freshness.py — prices, FX, snapshot gaps
|
||||
|
||||
```bash
|
||||
python3 scripts/check_freshness.py fin-copy.db [--hours-stale 26] [--days 30] [--json]
|
||||
```
|
||||
|
||||
Interpretation (thresholds as of 2026-07-05):
|
||||
|
||||
- **Age column** — the scheduler refreshes *every 5 minutes* plus a midnight
|
||||
job, so healthy ages are minutes while the api container is up. Hours old =
|
||||
container was down or yfinance/frankfurter kept failing (both fall back to
|
||||
the stale cache silently — check api logs, see fin-run-and-operate). The
|
||||
default 26 h threshold only catches hard outages; tighten with
|
||||
`--hours-stale 1` when investigating.
|
||||
- **`missing-fx` EUR/USD** — worst case: `calculate_net_worth` falls back to
|
||||
rate **1.0** and produces confidently wrong totals.
|
||||
- **`missing-fx` EUR/GBP** — flagged only when GBP data exists; then `_to_eur`
|
||||
returns 0 and GBP amounts vanish from net worth (deliberate — see
|
||||
fin-failure-archaeology before "fixing").
|
||||
- **`missing-ticker-price`** — a tracked ticker (holding with shares > 0, or a
|
||||
pending RSU grant's ticker) has no cache row: valued at €0. Fix:
|
||||
`fin prices fetch TICKER` or `POST /api/prices/refresh`.
|
||||
- **`zero-price`** — cached price ≤ 0; same €0 effect.
|
||||
- **`missing-snapshot`** — gap in the daily net-worth series (window ends
|
||||
today, container-local date). Today missing usually means the api container
|
||||
was down at midnight. Gaps make History charts lie by interpolation; the
|
||||
midnight job upserts, so a restart today backfills only today.
|
||||
|
||||
### spending_recompute.py — independent spending recompute
|
||||
|
||||
```bash
|
||||
python3 scripts/spending_recompute.py fin-copy.db 2026-06 [--account N] [--json]
|
||||
```
|
||||
|
||||
Line-by-line replication of `crud.get_monthly_spending`: expense +
|
||||
`splitwise_receive` at raw amount; `splitwise_pay` at
|
||||
`amount + splitwise_partner_share` (user's net share — intentional, do not
|
||||
"fix"); receivable accounts excluded from spending; income excludes
|
||||
receivable/payable accounts, `Opening balance` rows, **and rows with NULL
|
||||
description** (SQL `<>` semantics — replicated deliberately). No
|
||||
`is_active` filter — inactive accounts count, same as the API.
|
||||
|
||||
Interpretation — diff against the API:
|
||||
|
||||
```bash
|
||||
python3 scripts/spending_recompute.py fin-copy.db 2026-06 --json | jq -S '{total, income_total, categories: [.categories[] | {category_id, total}] | sort_by(.category_id)}' > /tmp/recomputed.json
|
||||
fin spending month 2026-06 --json | jq -S '{total, income_total, categories: [.categories[] | {category_id, total}] | sort_by(.category_id)}' > /tmp/api.json
|
||||
diff /tmp/recomputed.json /tmp/api.json
|
||||
```
|
||||
|
||||
- **Identical** — spending math is consistent between code and this spec; if
|
||||
the user still disputes the number, the disagreement is about *semantics*
|
||||
(load **fin-accounting-reference**), not a bug.
|
||||
- **Differs** — either the DB copy is older than the live DB (recopy!), or
|
||||
`get_monthly_spending` changed since 2026-07-05 (see Provenance), or you
|
||||
found a real bug — capture both JSONs as evidence.
|
||||
- Category *order* may differ (script sorts by total DESC; API preserves dict
|
||||
order) — always diff sorted by `category_id` as above.
|
||||
|
||||
## API-level probes
|
||||
|
||||
Full CLI surface: the **fin** skill. Always `--json | jq`. Curl equivalents
|
||||
(add `-H "Authorization: Bearer $FIN_API_TOKEN"` only if the API enforces it):
|
||||
|
||||
```bash
|
||||
fin networth --json | jq '{total: .total_eur, rate: .eur_usd_rate}' # rate==1.0 => EUR/USD cache row missing
|
||||
curl -s $FIN_API_URL/api/net-worth/current | jq '.total_eur'
|
||||
fin networth history --range 1M --json | jq 'length' # ~30 => no snapshot gaps
|
||||
curl -s "$FIN_API_URL/api/net-worth/history?range=1M" | jq 'length'
|
||||
fin spending month 2026-06 --json | jq '{total, income_total}'
|
||||
curl -s "$FIN_API_URL/api/spending/monthly?month=2026-06" | jq .
|
||||
fin prices fx --json # cached EUR/USD
|
||||
curl -s $FIN_API_URL/api/prices/fx-rate | jq .
|
||||
curl -s -X POST $FIN_API_URL/api/prices/refresh # on-demand refresh (202)
|
||||
fin accounts list --json | jq '[.[] | {id, name, type, balance}]' # cross-check vs audit_balances.py
|
||||
```
|
||||
|
||||
Cross-check pattern: API balance ≠ `audit_balances.py` computed balance for
|
||||
the same account → look for a `manual_override_balance` before suspecting
|
||||
crud.
|
||||
|
||||
## Evidence workflow — before/after a change
|
||||
|
||||
The project discipline (see **fin-validation-and-qa** for the evidence
|
||||
hierarchy): predict the numbers, then measure them.
|
||||
|
||||
```bash
|
||||
EV=/tmp/evidence-$(date +%Y%m%d-%H%M%S); mkdir -p $EV
|
||||
S=.claude/skills/fin-diagnostics-and-tooling/scripts
|
||||
for a in audit_linked_legs audit_holdings audit_balances check_freshness; do
|
||||
python3 $S/$a.py fin-copy.db --json > $EV/before-$a.json; echo "$a: exit $?"
|
||||
done
|
||||
python3 $S/spending_recompute.py fin-copy.db 2026-06 --json > $EV/before-spending.json
|
||||
fin networth --json > $EV/before-networth.json
|
||||
|
||||
# ... apply the change, re-copy the DB ...
|
||||
|
||||
for a in audit_linked_legs audit_holdings audit_balances check_freshness; do
|
||||
python3 $S/$a.py fin-copy.db --json > $EV/after-$a.json
|
||||
done
|
||||
python3 $S/spending_recompute.py fin-copy.db 2026-06 --json > $EV/after-spending.json
|
||||
fin networth --json > $EV/after-networth.json
|
||||
for f in $EV/before-*.json; do diff "$f" "${f/before-/after-}" && echo "unchanged: $f"; done
|
||||
```
|
||||
|
||||
Rules: state *which* numbers the change should move and by how much **before**
|
||||
running the after-pass; every diff line must be either predicted or explained;
|
||||
audits that were clean before must be clean after. Attach `$EV` contents to
|
||||
the change record (**fin-change-control**).
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Recording transactions / querying finances for the user → **fin**
|
||||
- What a formula *should* be (spending semantics, FX, RSU math) → **fin-accounting-reference**; proving ledger properties → **fin-ledger-analysis-toolkit**
|
||||
- Symptom→cause triage when you don't yet know what to measure → **fin-debugging-playbook**; historical bug context → **fin-failure-archaeology**
|
||||
- Backup/restore, env vars, container ops, prod logs → **fin-run-and-operate**
|
||||
- Deciding what evidence a change needs / writing tests → **fin-validation-and-qa**, **fin-test-coverage-campaign**
|
||||
- Design/invariant questions before changing code → **fin-architecture-contract**, **fin-change-control**
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
- Ground truth (read, not guessed): `api/app/models.py` (all tables/columns),
|
||||
`api/app/crud.py` (pairing rules, holding recompute, spending math),
|
||||
`api/app/services/price_fetcher.py` (EUR/USD + EUR/GBP + yfinance, tracked
|
||||
tickers), `api/app/services/net_worth.py` (balance formula, fallbacks),
|
||||
`api/app/services/scheduler.py` (5-min price job, midnight snapshot job).
|
||||
Written against repo HEAD `082b608`, 2026-07-05.
|
||||
- All five scripts tested 2026-07-05 on Python 3.13 against a synthetic DB
|
||||
(schema derived from models.py; healthy rows + one seeded defect per check;
|
||||
each script flagged exactly the seeded defects, exit codes verified).
|
||||
- Schema-drift check — run before trusting the scripts; any output means the
|
||||
ground truth moved and scripts/SKILL need re-verification:
|
||||
|
||||
```bash
|
||||
git -C /home/markuseisl/Documents/git/fin diff --stat 082b608 -- api/app/models.py api/app/crud.py api/app/services/price_fetcher.py api/app/services/net_worth.py api/app/services/scheduler.py
|
||||
```
|
||||
|
||||
- When drift is found: re-read the changed function, update the affected
|
||||
script *and* its interpretation section, re-run the synthetic-DB test
|
||||
(build a tiny DB with one broken case per check), and bump the commit hash
|
||||
and dates in this section.
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compute per-account balances (SUM(amount - fee)) and flag masking overrides.
|
||||
|
||||
READ-ONLY. Usage: python3 audit_balances.py /path/to/fin.db [--allow-overrides] [--json]
|
||||
|
||||
Ground truth: api/app/services/net_worth.py — for bank/splitwise/receivable/
|
||||
payable accounts, balance = SUM(amount - fee) over all transactions, UNLESS
|
||||
manual_override_balance is set, in which case the override silently replaces
|
||||
the computed value in net worth. Investment/rsu balances come from holdings x
|
||||
price_cache, not from transactions (any computed value shown for them here is
|
||||
just leftover cash legs and is not used by net worth).
|
||||
|
||||
Anomalies:
|
||||
- manual_override_balance set (masks the computed balance; suppress with
|
||||
--allow-overrides if the owner set it deliberately)
|
||||
- transactions whose account_id points at a nonexistent account (invisible
|
||||
to every balance and net-worth calculation)
|
||||
|
||||
Exit 0 = clean, 1 = anomalies found.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("db")
|
||||
ap.add_argument("--allow-overrides", action="store_true",
|
||||
help="do not count manual_override_balance as an anomaly")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
|
||||
rows = con.execute(
|
||||
"SELECT a.id, a.name, a.type, a.currency, a.is_active, "
|
||||
" a.manual_override_balance AS override, "
|
||||
" COUNT(t.id) AS txn_count, "
|
||||
" COALESCE(SUM(t.amount - t.fee), 0.0) AS computed "
|
||||
"FROM accounts a LEFT JOIN transactions t ON t.account_id = a.id "
|
||||
"GROUP BY a.id ORDER BY a.type, a.id").fetchall()
|
||||
|
||||
orphans = con.execute(
|
||||
"SELECT id, account_id, amount, date FROM transactions "
|
||||
"WHERE account_id NOT IN (SELECT id FROM accounts)").fetchall()
|
||||
|
||||
anomalies: list[dict] = []
|
||||
accounts = []
|
||||
for r in rows:
|
||||
d = dict(r)
|
||||
d["computed"] = round(d["computed"], 2)
|
||||
if d["override"] is not None:
|
||||
d["override_delta"] = round(d["override"] - d["computed"], 2)
|
||||
if not args.allow_overrides:
|
||||
anomalies.append({
|
||||
"check": "manual-override", "account_id": d["id"],
|
||||
"detail": (f"{d['name']} ({d['type']}): override "
|
||||
f"{d['override']} masks computed {d['computed']} "
|
||||
f"(delta {d['override_delta']:+})")})
|
||||
accounts.append(d)
|
||||
for o in orphans:
|
||||
anomalies.append({"check": "orphan-transaction", "account_id": o["account_id"],
|
||||
"detail": (f"txn {o['id']} ({o['date']}, {o['amount']}) points "
|
||||
f"at nonexistent account {o['account_id']}")})
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({"accounts": accounts, "anomalies": anomalies}, indent=2))
|
||||
else:
|
||||
print(f"{'id':>4} {'type':<12} {'active':<6} {'ccy':<4} "
|
||||
f"{'txns':>5} {'computed':>12} {'override':>10} name")
|
||||
for d in accounts:
|
||||
ov = "" if d["override"] is None else f"{d['override']:.2f}"
|
||||
print(f"{d['id']:>4} {d['type']:<12} {str(bool(d['is_active'])):<6} "
|
||||
f"{d['currency']:<4} {d['txn_count']:>5} {d['computed']:>12.2f} "
|
||||
f"{ov:>10} {d['name']}")
|
||||
for a in anomalies:
|
||||
print(f"ANOMALY [{a['check']}]: {a['detail']}")
|
||||
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
|
||||
return 1 if anomalies else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Recompute every holding from stock_transactions and diff against holdings rows.
|
||||
|
||||
READ-ONLY. Usage: python3 audit_holdings.py /path/to/fin.db [--json]
|
||||
|
||||
Faithful replication of crud.recompute_holding_from_stock_transactions
|
||||
(api/app/crud.py): iterate stock_transactions for each (account_id, ticker)
|
||||
ordered by date ASC, id ASC;
|
||||
buy / rsu_vest : shares += tx.shares; total_cost += tx.shares*price_per_share + fee
|
||||
sell : error if tx.shares > shares held at that point;
|
||||
total_cost -= tx.shares * (total_cost/shares); shares -= tx.shares;
|
||||
total_cost = max(0, total_cost)
|
||||
avg_cost_basis = total_cost/shares if shares > 0 else 0
|
||||
currency = first transaction's currency
|
||||
If no stock transactions remain, crud deletes the holding — so a holding row
|
||||
with zero stock transactions is stale.
|
||||
|
||||
Tolerances: shares 1e-6, avg cost basis 0.005 (float accumulation).
|
||||
Currency mismatch is a WARNING only (crud never updates currency on an
|
||||
existing holding, so drift is possible without a bug).
|
||||
|
||||
Exit 0 = clean, 1 = anomalies found.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
SHARES_EPS = 1e-6
|
||||
COST_EPS = 0.005
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("db")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
|
||||
txns_by_key: dict[tuple, list] = {}
|
||||
for r in con.execute(
|
||||
"SELECT id, account_id, ticker, type, shares, price_per_share, fee, "
|
||||
"date, currency FROM stock_transactions ORDER BY date ASC, id ASC"):
|
||||
txns_by_key.setdefault((r["account_id"], r["ticker"]), []).append(dict(r))
|
||||
|
||||
holdings = {(r["account_id"], r["ticker"]): dict(r) for r in con.execute(
|
||||
"SELECT id, account_id, ticker, shares, avg_cost_basis_per_share, currency "
|
||||
"FROM holdings")}
|
||||
|
||||
anomalies: list[dict] = []
|
||||
warnings: list[str] = []
|
||||
|
||||
def flag(check, key, detail):
|
||||
anomalies.append({"check": check, "account_id": key[0],
|
||||
"ticker": key[1], "detail": detail})
|
||||
|
||||
for key, txns in txns_by_key.items():
|
||||
shares = 0.0
|
||||
total_cost = 0.0
|
||||
currency = txns[0]["currency"]
|
||||
invalid = None
|
||||
for tx in txns:
|
||||
t = (tx["type"] or "").lower()
|
||||
if t in ("buy", "rsu_vest"):
|
||||
shares += tx["shares"]
|
||||
total_cost += tx["shares"] * tx["price_per_share"] + tx["fee"]
|
||||
elif t == "sell":
|
||||
if tx["shares"] > shares:
|
||||
invalid = (f"stock_txn {tx['id']} sells {tx['shares']} but only "
|
||||
f"{round(shares, 6)} held at that point in history")
|
||||
break
|
||||
if shares > 0:
|
||||
total_cost -= tx["shares"] * (total_cost / shares)
|
||||
shares -= tx["shares"]
|
||||
total_cost = max(0.0, total_cost)
|
||||
else:
|
||||
flag("unknown-stock-txn-type", key,
|
||||
f"stock_txn {tx['id']} has type {tx['type']!r} "
|
||||
f"(crud silently ignores it)")
|
||||
if invalid:
|
||||
flag("invalid-history", key, invalid)
|
||||
continue
|
||||
|
||||
avg = (total_cost / shares) if shares > 0 else 0.0
|
||||
h = holdings.get(key)
|
||||
if h is None:
|
||||
flag("missing-holding", key,
|
||||
f"{len(txns)} stock txns imply shares={round(shares, 6)}, "
|
||||
f"avg={round(avg, 4)}, but no holdings row exists")
|
||||
continue
|
||||
if abs(h["shares"] - shares) > SHARES_EPS:
|
||||
flag("shares-mismatch", key,
|
||||
f"holding {h['id']}: stored {h['shares']}, recomputed {round(shares, 6)}")
|
||||
if abs(h["avg_cost_basis_per_share"] - avg) > COST_EPS:
|
||||
flag("cost-basis-mismatch", key,
|
||||
f"holding {h['id']}: stored {h['avg_cost_basis_per_share']}, "
|
||||
f"recomputed {round(avg, 6)}")
|
||||
if h["currency"] != currency:
|
||||
warnings.append(f"WARNING [currency-drift] account {key[0]} {key[1]}: "
|
||||
f"holding says {h['currency']}, first txn says {currency}")
|
||||
|
||||
for key, h in holdings.items():
|
||||
if key not in txns_by_key:
|
||||
flag("stale-holding", key,
|
||||
f"holding {h['id']} ({h['shares']} shares) has no stock "
|
||||
f"transactions — recompute would delete it")
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({"holdings": len(holdings),
|
||||
"keys_with_txns": len(txns_by_key),
|
||||
"warnings": warnings, "anomalies": anomalies}, indent=2))
|
||||
else:
|
||||
print(f"audit_holdings: {len(holdings)} holdings, "
|
||||
f"{len(txns_by_key)} (account,ticker) keys with stock txns")
|
||||
for w in warnings:
|
||||
print(w)
|
||||
for a in anomalies:
|
||||
print(f"ANOMALY [{a['check']}] account {a['account_id']} "
|
||||
f"{a['ticker']}: {a['detail']}")
|
||||
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
|
||||
return 1 if anomalies else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Audit linked transaction legs (transfers, settlements, splitwise pairs).
|
||||
|
||||
READ-ONLY. Usage: python3 audit_linked_legs.py /path/to/fin.db [--allow-legacy] [--json]
|
||||
|
||||
Ground truth: api/app/crud.py (create_transfer, create_settlement,
|
||||
create_split_expense, create_splitwise_i_paid, create_funded_buy,
|
||||
delete_transaction legacy fallback) and api/app/models.py.
|
||||
|
||||
Rules encoded:
|
||||
- Every 'transfer'/'settlement' row must have a partner via linked_transaction_id,
|
||||
EXCEPT the funded-buy cash leg: a single 'transfer' row whose linked_account_id
|
||||
points to an investment/rsu account (create_funded_buy writes no partner row).
|
||||
- Rows written before linked_transaction_id existed are "legacy": NULL link but
|
||||
linked_account_id set. delete_transaction pairs them by reversed accounts +
|
||||
same date + same type. --allow-legacy accepts legacy rows that have such a match.
|
||||
- linked_transaction_id must resolve, must not be self, and for
|
||||
transfer/settlement/splitwise_pay the partner must link back (reciprocal).
|
||||
The split-expense 'expense' row links one-way into its transfer group by
|
||||
design and is exempt from reciprocity.
|
||||
- Same-currency transfer/settlement pairs must cancel: amount_a + amount_b == 0
|
||||
(fees live in the fee column and are excluded). Cross-currency transfer legs
|
||||
legitimately differ (to_amount conversion) and are skipped.
|
||||
- splitwise_pay pairs never cancel by design (bank leg = -total, splitwise leg
|
||||
= +partner_share) and are never amount-checked.
|
||||
|
||||
Exit 0 = clean, 1 = anomalies found.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
EPS = 0.005
|
||||
PARTNER_REQUIRED = ("transfer", "settlement")
|
||||
RECIPROCAL = ("transfer", "settlement", "splitwise_pay")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("db")
|
||||
ap.add_argument("--allow-legacy", action="store_true",
|
||||
help="accept legacy NULL-link legs that have a crud-style fallback match")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
|
||||
accounts = {r["id"]: dict(r) for r in con.execute(
|
||||
"SELECT id, name, type FROM accounts")}
|
||||
txns = {r["id"]: dict(r) for r in con.execute(
|
||||
"SELECT id, date, account_id, amount, currency, type, fee, "
|
||||
"linked_account_id, linked_transaction_id, description "
|
||||
"FROM transactions")}
|
||||
|
||||
anomalies: list[dict] = []
|
||||
info = {"transactions": len(txns), "checked_pairs": 0,
|
||||
"cross_currency_pairs": 0, "funded_buy_legs": 0, "legacy_matched": 0}
|
||||
|
||||
def flag(check, txn_id, detail):
|
||||
anomalies.append({"check": check, "txn_id": txn_id, "detail": detail})
|
||||
|
||||
for t in txns.values():
|
||||
tid, ttype = t["id"], t["type"]
|
||||
if t["account_id"] not in accounts:
|
||||
flag("missing-account", tid,
|
||||
f"account_id={t['account_id']} does not exist")
|
||||
la = t["linked_account_id"]
|
||||
if la is not None and la not in accounts:
|
||||
flag("missing-linked-account", tid,
|
||||
f"linked_account_id={la} does not exist")
|
||||
if la is not None and la == t["account_id"]:
|
||||
flag("self-account-link", tid, "linked_account_id == account_id")
|
||||
|
||||
lid = t["linked_transaction_id"]
|
||||
if lid is not None:
|
||||
if lid == tid:
|
||||
flag("self-link", tid, "linked_transaction_id == id")
|
||||
continue
|
||||
partner = txns.get(lid)
|
||||
if partner is None:
|
||||
flag("orphan-link", tid,
|
||||
f"linked_transaction_id={lid} does not exist")
|
||||
continue
|
||||
if ttype in RECIPROCAL and partner["linked_transaction_id"] != tid:
|
||||
flag("non-reciprocal", tid,
|
||||
f"partner {lid} links to {partner['linked_transaction_id']!r}, not back")
|
||||
if la is not None and partner["account_id"] != la:
|
||||
flag("partner-account-mismatch", tid,
|
||||
f"partner {lid} lives on account {partner['account_id']}, "
|
||||
f"but linked_account_id={la}")
|
||||
# amount cancellation — once per pair, same type, same currency only
|
||||
if (ttype in ("transfer", "settlement") and partner["type"] == ttype
|
||||
and tid < lid):
|
||||
if t["currency"] == partner["currency"]:
|
||||
info["checked_pairs"] += 1
|
||||
resid = t["amount"] + partner["amount"]
|
||||
if abs(resid) > EPS:
|
||||
flag("amounts-dont-cancel", tid,
|
||||
f"pair ({tid},{lid}): {t['amount']} + {partner['amount']} "
|
||||
f"= {resid:+.2f} {t['currency']}")
|
||||
else:
|
||||
info["cross_currency_pairs"] += 1
|
||||
elif ttype in PARTNER_REQUIRED:
|
||||
la_type = accounts.get(la, {}).get("type") if la is not None else None
|
||||
if ttype == "transfer" and la_type in ("investment", "rsu"):
|
||||
info["funded_buy_legs"] += 1 # single-leg by design
|
||||
continue
|
||||
legacy_ok = False
|
||||
if args.allow_legacy and la is not None:
|
||||
# replicate delete_transaction's legacy fallback match
|
||||
for o in txns.values():
|
||||
if (o["id"] != tid
|
||||
and o["account_id"] == la
|
||||
and o["linked_account_id"] == t["account_id"]
|
||||
and o["linked_transaction_id"] is None
|
||||
and o["date"] == t["date"]
|
||||
and o["type"] == ttype):
|
||||
legacy_ok = True
|
||||
break
|
||||
if legacy_ok:
|
||||
info["legacy_matched"] += 1
|
||||
else:
|
||||
flag("unlinked-leg", tid,
|
||||
f"{ttype} with NULL linked_transaction_id "
|
||||
f"(linked_account_id={la}, date={t['date']}, amount={t['amount']})")
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({"info": info, "anomalies": anomalies}, indent=2))
|
||||
else:
|
||||
print(f"audit_linked_legs: {info['transactions']} transactions, "
|
||||
f"{info['checked_pairs']} same-currency pairs amount-checked, "
|
||||
f"{info['cross_currency_pairs']} cross-currency pairs skipped, "
|
||||
f"{info['funded_buy_legs']} funded-buy single legs, "
|
||||
f"{info['legacy_matched']} legacy pairs accepted")
|
||||
for a in anomalies:
|
||||
print(f"ANOMALY [{a['check']}] txn {a['txn_id']}: {a['detail']}")
|
||||
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
|
||||
return 1 if anomalies else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check price_cache freshness, required FX rows, and net_worth_snapshot gaps.
|
||||
|
||||
READ-ONLY. Usage:
|
||||
python3 check_freshness.py /path/to/fin.db [--hours-stale 26] [--days 30] [--json]
|
||||
|
||||
Ground truth (as of 2026-07-05):
|
||||
- api/app/services/price_fetcher.py fetches EUR/USD and EUR/GBP from
|
||||
frankfurter.app plus one yfinance price per tracked ticker. Tracked =
|
||||
holdings with shares > 0 UNION tickers of pending (unvested) rsu_grants.
|
||||
- api/app/services/scheduler.py refreshes prices every 5 minutes and writes
|
||||
one net_worth_snapshot per calendar day at midnight (date.today() of the
|
||||
api container). fetched_at is datetime.utcnow().
|
||||
- api/app/services/net_worth.py falls back to EUR/USD = 1.0 when the row is
|
||||
missing (silently wrong totals) and values a ticker without a price row
|
||||
at 0.0. Missing EUR/GBP excludes GBP amounts entirely (_to_eur returns 0).
|
||||
|
||||
Anomalies:
|
||||
- price_cache row older than --hours-stale (default 26h; healthy is minutes
|
||||
while the api container is up, thanks to the 5-minute job)
|
||||
- price_cache row with price <= 0
|
||||
- EUR/USD row missing (always required)
|
||||
- EUR/GBP row missing while anything GBP exists (account, holding, or cached
|
||||
price in GBP)
|
||||
- tracked ticker with no price_cache row at all
|
||||
- missing snapshot days in the last --days calendar days (window ends today,
|
||||
local date)
|
||||
|
||||
Exit 0 = clean, 1 = anomalies found.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
|
||||
def parse_dt(s: str) -> datetime | None:
|
||||
try:
|
||||
return datetime.fromisoformat(s.replace("T", " "))
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("db")
|
||||
ap.add_argument("--hours-stale", type=float, default=26.0)
|
||||
ap.add_argument("--days", type=int, default=30)
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
now = datetime.now(timezone.utc).replace(tzinfo=None) # fetched_at is utcnow()
|
||||
anomalies: list[dict] = []
|
||||
|
||||
def flag(check, detail):
|
||||
anomalies.append({"check": check, "detail": detail})
|
||||
|
||||
prices = {r["symbol"]: dict(r) for r in con.execute(
|
||||
"SELECT symbol, price, currency, source, fetched_at FROM price_cache")}
|
||||
report = []
|
||||
for sym, p in sorted(prices.items()):
|
||||
dt = parse_dt(p["fetched_at"])
|
||||
age_h = round((now - dt).total_seconds() / 3600, 1) if dt else None
|
||||
report.append({"symbol": sym, "price": p["price"], "currency": p["currency"],
|
||||
"source": p["source"], "age_hours": age_h})
|
||||
if age_h is None:
|
||||
flag("unparseable-fetched-at", f"{sym}: fetched_at={p['fetched_at']!r}")
|
||||
elif age_h > args.hours_stale:
|
||||
flag("stale-price", f"{sym}: {age_h}h old (> {args.hours_stale}h)")
|
||||
if p["price"] is None or p["price"] <= 0:
|
||||
flag("zero-price", f"{sym}: cached price {p['price']} — net worth "
|
||||
f"values it at 0")
|
||||
|
||||
# required FX rows
|
||||
if "EUR/USD" not in prices:
|
||||
flag("missing-fx", "EUR/USD not in price_cache — net worth silently "
|
||||
"falls back to rate 1.0")
|
||||
gbp_seen = con.execute(
|
||||
"SELECT EXISTS(SELECT 1 FROM accounts WHERE currency='GBP') "
|
||||
"OR EXISTS(SELECT 1 FROM holdings WHERE currency='GBP') "
|
||||
"OR EXISTS(SELECT 1 FROM price_cache WHERE currency='GBP' "
|
||||
" AND symbol NOT LIKE 'EUR/%')").fetchone()[0]
|
||||
if gbp_seen and "EUR/GBP" not in prices:
|
||||
flag("missing-fx", "EUR/GBP not in price_cache but GBP data exists — "
|
||||
"GBP amounts are excluded from net worth (_to_eur -> 0)")
|
||||
|
||||
# tracked tickers must have a price row
|
||||
tracked = [r[0] for r in con.execute(
|
||||
"SELECT DISTINCT ticker FROM holdings WHERE shares > 0 "
|
||||
"UNION SELECT DISTINCT ticker FROM rsu_grants WHERE vested_at IS NULL")]
|
||||
for t in tracked:
|
||||
if t not in prices:
|
||||
flag("missing-ticker-price", f"{t} is tracked (holding or pending "
|
||||
f"grant) but has no price_cache row")
|
||||
|
||||
# snapshot gaps
|
||||
today = date.today()
|
||||
start = today - timedelta(days=args.days - 1)
|
||||
have = {r[0] for r in con.execute(
|
||||
"SELECT snapshot_date FROM net_worth_snapshots WHERE snapshot_date >= ?",
|
||||
(start.isoformat(),))}
|
||||
missing = [(start + timedelta(days=i)).isoformat()
|
||||
for i in range(args.days)
|
||||
if (start + timedelta(days=i)).isoformat() not in have]
|
||||
for m in missing:
|
||||
flag("missing-snapshot", f"no net_worth_snapshot for {m}")
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({"prices": report, "tracked_tickers": tracked,
|
||||
"snapshot_window": [start.isoformat(), today.isoformat()],
|
||||
"missing_snapshot_days": missing,
|
||||
"anomalies": anomalies}, indent=2))
|
||||
else:
|
||||
print(f"{'symbol':<12} {'price':>12} {'ccy':<4} {'source':<11} {'age_h':>7}")
|
||||
for p in report:
|
||||
print(f"{p['symbol']:<12} {p['price']:>12.4f} {p['currency']:<4} "
|
||||
f"{p['source']:<11} {p['age_hours'] if p['age_hours'] is not None else '?':>7}")
|
||||
print(f"tracked tickers: {', '.join(tracked) or '(none)'}")
|
||||
print(f"snapshot window {start} .. {today}: "
|
||||
f"{args.days - len(missing)}/{args.days} days present")
|
||||
for a in anomalies:
|
||||
print(f"ANOMALY [{a['check']}]: {a['detail']}")
|
||||
print("CLEAN" if not anomalies else f"{len(anomalies)} anomalies")
|
||||
return 1 if anomalies else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independently recompute one month's spending per category from raw transactions.
|
||||
|
||||
READ-ONLY. Usage:
|
||||
python3 spending_recompute.py /path/to/fin.db YYYY-MM [--account N] [--json]
|
||||
|
||||
Line-by-line replication of crud.get_monthly_spending (api/app/crud.py, as of
|
||||
2026-07-05), so the output can be diffed against `fin spending month YYYY-MM --json`:
|
||||
|
||||
- Query 1: type IN ('expense','splitwise_receive'), amount < 0, account type
|
||||
NOT IN ('receivable'), date in [month-start, next-month-start); SUM(amount)
|
||||
grouped by category_id. NOTE: no is_active filter — inactive accounts count.
|
||||
- Query 2: type = 'splitwise_pay', amount < 0, same account exclusion; each
|
||||
row contributes amount + COALESCE(splitwise_partner_share, 0) — the user's
|
||||
net share (e.g. -100 paid, partner owes 50 -> -50).
|
||||
- Categories whose merged total >= 0 are dropped. total = sum of abs values.
|
||||
- income_total: type = 'income', account type NOT IN ('payable','receivable'),
|
||||
description <> 'Opening balance'. SQL NULL semantics apply: rows with NULL
|
||||
description are ALSO excluded (matches SQLAlchemy's `!=`). Replicated as-is.
|
||||
- percentage = round(total/overall*100, 1).
|
||||
|
||||
This is a recompute tool, not an audit: exit 0 on success, 2 on usage error.
|
||||
Category order here is sorted by total DESC for readability; the API preserves
|
||||
dict insertion order — diff by category, not by position.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import date
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("db")
|
||||
ap.add_argument("month", help="YYYY-MM")
|
||||
ap.add_argument("--account", type=int, default=None)
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
try:
|
||||
year, mon = int(args.month[:4]), int(args.month[5:7])
|
||||
start = date(year, mon, 1).isoformat()
|
||||
end = (date(year + 1, 1, 1) if mon == 12 else date(year, mon + 1, 1)).isoformat()
|
||||
except ValueError:
|
||||
print("month must be YYYY-MM", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
con = sqlite3.connect(f"file:{args.db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
|
||||
acct_filter = "" if args.account is None else " AND t.account_id = :acct"
|
||||
params = {"start": start, "end": end, "acct": args.account}
|
||||
|
||||
# Query 1: expense + splitwise_receive at raw amount
|
||||
merged: dict = {}
|
||||
for r in con.execute(
|
||||
"SELECT t.category_id AS cid, SUM(t.amount) AS total "
|
||||
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
|
||||
"WHERE t.date >= :start AND t.date < :end "
|
||||
"AND t.type IN ('expense','splitwise_receive') AND t.amount < 0 "
|
||||
"AND a.type NOT IN ('receivable')" + acct_filter +
|
||||
" GROUP BY t.category_id", params):
|
||||
merged[r["cid"]] = merged.get(r["cid"], 0.0) + r["total"]
|
||||
|
||||
# Query 2: splitwise_pay at user's net share
|
||||
for r in con.execute(
|
||||
"SELECT t.category_id AS cid, t.amount AS amount, "
|
||||
"t.splitwise_partner_share AS share "
|
||||
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
|
||||
"WHERE t.date >= :start AND t.date < :end "
|
||||
"AND t.type = 'splitwise_pay' AND t.amount < 0 "
|
||||
"AND a.type NOT IN ('receivable')" + acct_filter, params):
|
||||
user_share = r["amount"] + (r["share"] or 0.0)
|
||||
merged[r["cid"]] = merged.get(r["cid"], 0.0) + user_share
|
||||
|
||||
cats = {r["id"]: dict(r) for r in con.execute(
|
||||
"SELECT id, name, color, icon FROM categories")}
|
||||
|
||||
categories = []
|
||||
overall = 0.0
|
||||
for cid, total in merged.items():
|
||||
if total >= 0:
|
||||
continue # no net expense for this category
|
||||
positive = abs(total)
|
||||
overall += positive
|
||||
c = cats.get(cid) if cid is not None else None
|
||||
categories.append({
|
||||
"category_id": cid,
|
||||
"category_name": c["name"] if c else None,
|
||||
"color": c["color"] if c else None,
|
||||
"icon": c["icon"] if c else None,
|
||||
"total": positive,
|
||||
"percentage": 0.0,
|
||||
})
|
||||
for item in categories:
|
||||
item["percentage"] = round(item["total"] / overall * 100, 1) if overall > 0 else 0.0
|
||||
categories.sort(key=lambda x: x["total"], reverse=True)
|
||||
|
||||
income = con.execute(
|
||||
"SELECT COALESCE(SUM(t.amount), 0.0) "
|
||||
"FROM transactions t JOIN accounts a ON t.account_id = a.id "
|
||||
"WHERE t.date >= :start AND t.date < :end AND t.type = 'income' "
|
||||
"AND a.type NOT IN ('payable','receivable') "
|
||||
"AND t.description <> 'Opening balance'" + acct_filter, params).fetchone()[0]
|
||||
|
||||
result = {"month": args.month, "total": overall,
|
||||
"income_total": income, "categories": categories}
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(f"spending_recompute {args.month}"
|
||||
+ (f" (account {args.account})" if args.account is not None else ""))
|
||||
for c in categories:
|
||||
print(f" {c['category_name'] or '(uncategorized)':<24} "
|
||||
f"{c['total']:>10.2f} {c['percentage']:>5.1f}%")
|
||||
print(f" {'TOTAL spending':<24} {overall:>10.2f}")
|
||||
print(f" {'income_total':<24} {income:>10.2f}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
name: fin-failure-archaeology
|
||||
description: The chronicle of every significant bug, dead end, and design correction in fin's git history. Load BEFORE fixing anything that looks like a bug, whenever a fix idea feels obvious ("just add a fallback", "just count that as an expense", "just delete the row"), when investigating why code is written a strange way (why _to_eur returns 0, why delete follows links both ways, why RSU is one pot), or before re-opening any settled design question (splitwise net-share math, transfer restrictions, FX fallbacks, RSU lots). If a change would touch spending semantics, linked transaction legs, currency conversion, or the RSU model — read this first.
|
||||
---
|
||||
|
||||
# fin — Failure Archaeology
|
||||
|
||||
Chronicle of every investigation, bug, dead end, and design retreat in this repo, mined from the full git history. Purpose: nobody re-fights a settled battle, and nobody "fixes" code back into a bug that was already fixed once.
|
||||
|
||||
**As of 2026-07-05.** History at time of writing: 107 commits, single `main` branch, **no reverts, no abandoned branches, no dead ends left in the tree** — every correction was made by a forward fix commit. The whole app was built in an intense burst: 2026-06-30 through 2026-07-03. That means fix commits are dense and each one is a real lesson, not noise.
|
||||
|
||||
Jargon used below:
|
||||
- **Leg** — one of the two `Transaction` rows a transfer/settlement writes (debit + credit), linked via `linked_account_id` and (since a1f109b) `linked_transaction_id`.
|
||||
- **Funded buy** — a stock buy that also debits a bank account for the cash.
|
||||
- **Net-share splitwise math** — counting only the user's share (`amount + splitwise_partner_share`) as spending, not the full bank outflow.
|
||||
- **RSU pot** — the aggregate of pending gross shares per RSU account (no per-grant schedule tracking).
|
||||
|
||||
## Settled questions — DO NOT re-open
|
||||
|
||||
These are the battles most likely to be re-fought by a well-meaning future engineer. Every one is intentional and evidence-backed.
|
||||
|
||||
| Settled question | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| `splitwise_pay` spending counts `amount + splitwise_partner_share` (user's net share), not the full bank outflow | INTENTIONAL — full amount double-counted the partner's half | 3e7f8ec |
|
||||
| Stock-buy cash legs and receivable/payable settles are typed `transfer`/`settlement`, never `expense`/`income` | INTENTIONAL — asset movement is not consumption/earnings | e36d719 + migration h8c9d0e1f2a3 |
|
||||
| No silent FX fallback in conversion paths; missing rate ⇒ skip/exclude with warning | REMOVED deliberately, twice. One residual 1.0 fallback survives in `net_worth._get_eur_usd_rate` (feeds funded buys) — a documented gap (U8, fin-ledger-analysis-toolkit), not precedent for adding more | 6b7d19b, 81091fc |
|
||||
| Transfers restricted to cash-type accounts (`bank`, `splitwise`, `receivable`, `payable`) with same currency (same-currency later relaxed via explicit cross-currency support) | INTENTIONAL | 225e69e (restriction), b2a7c7a/9a49be2 (explicit cross-currency path) |
|
||||
| RSU = one pot of pending gross shares per account, NOT per-grant vesting-schedule lots | DESIGN RETREAT — schedule generator was built (6385c06) and abandoned | c134801, c8dd938 |
|
||||
| Vest net shares are entered manually from the brokerage statement (the UI retreat in c134801); the API retains `floor(gross × (1 − withholding))` as the fallback when no explicit net shares are given (`crud.vest_rsu_grant`) | INTENTIONAL — brokerage statement is the source of truth; the computed path is a preview/fallback, not the authority | c134801, 225e69e |
|
||||
| Lending money ("I am owed") = bank→receivable **transfer**, not an income transaction | REDESIGNED — income version left cash untracked | e86df27 (spec), 16a19af (impl) |
|
||||
| `payable`-account expenses COUNT in spending; only `receivable` is excluded | INTENTIONAL — a9f045f excluded both, e61b254 deliberately re-included payable (expenses you owe her for are real consumption) | a9f045f → e61b254 |
|
||||
| Paired legs cannot be edited or deleted individually — delete cascades over the whole linked group, edit is blocked | INTENTIONAL | a06f921, a1f109b, a1289f6, e36d719 |
|
||||
| Holdings are always recomputed from full stock-transaction history, not incrementally patched | UNIFIED deliberately after the two paths diverged | c558999 |
|
||||
| Frankfurter API base URL is `https://api.frankfurter.dev/v1` (NOT `api.frankfurter.app`) | The old domain broke; do not "restore" it | 582e39d |
|
||||
| `npm install -g npm@11` before `npm ci` in web Dockerfile | Works around a real Alpine npm exit-handler bug — not cruft | 89d241a |
|
||||
| Category `type` (expense/income) is purely a UI picker concern; spending queries never filter transactions by category type | INTENTIONAL | e86df27 spec, 07ad335 |
|
||||
| Dates use local-time `todayLocalISO()`, never `toISOString().split('T')[0]` | UTC version booked entries to yesterday before ~1–2 AM CET | bc256f6, 00409f0, 9d06e1b |
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- Operating/deploying the app → `fin-run-and-operate`, `fin-build-and-env`
|
||||
- Using the `fin` CLI against the live API → `fin` skill
|
||||
- Making a new change safely (process, migrations, review gates) → `fin-change-control`
|
||||
- Live debugging of a current failure → `fin-debugging-playbook`, `fin-diagnostics-and-tooling`
|
||||
- Understanding the current architecture (not its history) → `fin-architecture-contract`
|
||||
- Accounting semantics reference (current rules, not how we got them) → `fin-accounting-reference`
|
||||
- Verifying data / analyzing the ledger → `fin-ledger-analysis-toolkit`, `fin-validation-and-qa`
|
||||
- Test writing → `fin-test-coverage-campaign`; open research → `fin-research-frontier`
|
||||
|
||||
Use THIS skill when the question is "why is it like this?", "is this a bug or a decision?", or "has this been tried before?".
|
||||
|
||||
---
|
||||
|
||||
## Chapter 1 — Ledger integrity and linked legs
|
||||
|
||||
The longest-running battle in the repo: keeping two-leg transactions atomic through create, delete, and edit. Fought in five escalating rounds.
|
||||
|
||||
| Round | Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Transfer could half-fail: one leg written, one lost | Frontend made two sequential `createTransaction` calls | c67d34f — atomic `POST /transactions/transfer` and `/settlement` writing both legs in one DB transaction | Settled |
|
||||
| 2 | Deleting one leg of a transfer orphaned the other and corrupted balances | `delete_transaction` deleted only the requested row | a06f921 — delete partner leg via account-pair query | Superseded by round 3 |
|
||||
| 3 | Account-pair partner lookup was ambiguous: could delete a leg of a *different* transfer between the same two accounts | No direct link between legs | a1f109b — `linked_transaction_id` column (migration e5f6a7b8c9d0); all paired writers `flush()` then cross-link both legs before commit. Legacy account-pair fallback retained | Settled; fallback further constrained by date/type/closest-id in bc256f6 |
|
||||
| 4 | Editing one leg desynced the pair | Edit endpoint didn't know about pairs | a1289f6 (backend blocks update on paired legs), 2bd4730 (frontend blocks edit UI on linked transactions) | Settled |
|
||||
| 5 | Split expense = 3 rows (expense + 2 transfer legs); deleting one left the others | Expense row wasn't linked to the transfer legs | bc256f6 (atomic `POST /transactions/split-expense`), e36d719 (`create_split_expense` links all rows; `delete_transaction` follows `linked_transaction_id` in BOTH directions and deletes the full group) | Settled |
|
||||
|
||||
Supporting fixes: 9d06e1b (frontend refetches after delete so orphan legs never linger visually), 082b608 (`fin tx add` in the CLI *refuses* transfer/settlement types to prevent single-leg writes over HTTP).
|
||||
|
||||
**Do not redo:** never add a "delete just this row" path, never let any client write a transfer as two separate transaction POSTs, and never remove the flush-then-cross-link dance in the paired writers — the IDs don't exist until flush.
|
||||
|
||||
## Chapter 2 — Spending and income semantics
|
||||
|
||||
The definition of "what counts as spending" was corrected four separate times. Current state is the sum of all four lessons.
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| Splitwise "I paid" double-counted: full €100 bank outflow shown as user's expense even on a 50/50 split | Spending summed raw `amount` for `splitwise_pay` | 3e7f8ec — sum `amount + splitwise_partner_share` (user's net share); also added the offsetting +half to the splitwise account so net worth reflects the receivable | Settled — invariant #6 in CLAUDE.md |
|
||||
| Receivable/payable bookkeeping rows polluted spending | Spending query didn't filter account types | a9f045f — exclude `payable` AND `receivable` account types | Partially reverted — see next row |
|
||||
| Real expenses vanished from spending after a9f045f | Expenses recorded on `payable` accounts (IOweFlow: partner paid, user owes their share) ARE real consumption | e61b254 — re-include `payable`; only `receivable` stays excluded | Settled. Do NOT "clean up" the asymmetry — it is the point |
|
||||
| Stock buys inflated spending; collecting a receivable inflated income | Funded-buy bank leg was typed `expense`; reconcile legs were typed `income`/`expense` | e36d719 — retyped to `transfer`/`settlement` (which spending queries already exclude); migration h8c9d0e1f2a3 retypes historical rows, matching funded-buy rows by the auto-generated `Buy <n> <TICKER>` description regex so user expenses like "Buy groceries" are untouched | Settled |
|
||||
| Lending money showed up as income | IAmOwedFlow wrote a single `income` transaction on the receivable and never touched the bank | e86df27 (spec) + 16a19af — redesigned as bank→receivable transfer; zero spending/income impact | Settled |
|
||||
|
||||
Also: 1e8e826 (original monthly-spending endpoint), 0a97ad0 (Earned/Spent/Net strip), b2a7c7a (income totals). Design docs for the pivots live in `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` and `docs/superpowers/plans/`.
|
||||
|
||||
**Do not redo:** if spending numbers look "too low", check the net-share math and the retyped internal flows before assuming a bug. If they look "too high", suspect a new flow writing `expense`-typed rows for asset movements — that exact bug shipped twice.
|
||||
|
||||
## Chapter 3 — Currency and FX
|
||||
|
||||
Theme: every implicit currency assumption ("it's probably USD", "EUR/USD is close enough", "1.0 is a safe default") turned out to be a bug.
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| GBP holdings valued as if USD | Only EUR/USD existed; ticker prices assumed USD | 61fc838 — store actual price currency in `price_cache`, add EUR/GBP rate | Settled |
|
||||
| Missing FX rate silently mis-converted using a *different* currency's rate | `_to_eur` fell back to EUR/USD for any unknown currency | 6b7d19b — return 0.0 with a logged warning; the amount is *excluded* from net worth rather than wrong | Settled. A dropped account value + warning log = missing rate, not a math bug |
|
||||
| Fresh DB cached a fake 1.0 FX rate forever | `fetch_eur_usd`/`fetch_eur_gbp` returned 1.0 when fetch failed and no cache existed | 81091fc — return `None`, skip the upsert entirely | Settled. Never reintroduce a numeric FX default |
|
||||
| GBP/other cash balances converted with hardcoded EUR/USD division | `net_worth.py` special-cased USD only | a06f921 — route all non-EUR cash through `_to_eur()` | Settled |
|
||||
| Manual override balances on investment/rsu accounts mis-converted the same way | Same hardcoded division survived in the override branch | 37bc101 — override branch also uses `_to_eur()` | Settled |
|
||||
| EUR/GBP fetch crashed instead of degrading | Error handling only existed for EUR/USD | 9b78d0e — mirrored the try/cache-fallback pattern | Settled |
|
||||
| Transfers/expenses hardcoded EUR regardless of account | QuickAdd flows had `currency: 'EUR'` literals | 424b926 (transfer uses source-account currency), 0496a58, bc256f6 (all splitwise/IOwe flows derive currency from selected account + mismatch guards) | Settled |
|
||||
| Funded buy double-converted for USD bank accounts | Bank debit always converted to EUR | b3062b7 — debit uses funding account's actual currency | Settled |
|
||||
| RSU vest transactions hardcoded USD | Literal `'USD'` in vest path | e36d719 — currency taken from `PriceCache` for the ticker | Settled |
|
||||
| LSE prices 100× too big | yfinance reports LSE in pence as `GBp`/`GBX` | Handled in fetcher (visible in bc8c8d5 diff): divide by 100, store GBP | Settled — keep the pence branch |
|
||||
|
||||
Cross-currency transfers were then added *explicitly* (b2a7c7a backend, 9a49be2 UI with FX pre-fill) rather than by loosening the same-currency guard silently.
|
||||
|
||||
**Do not redo:** any patch that makes a missing rate "just work" (defaults, nearest-rate fallbacks) reintroduces the class of bug this chapter killed four times.
|
||||
|
||||
## Chapter 4 — Dates and time
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| Trades/vests/settles entered before ~1–2 AM local time were booked to *yesterday* | `new Date().toISOString().split('T')[0]` is UTC; user is CET/CEST | 00409f0, 9d06e1b (IOwe/IAmOwed), bc256f6 (swept Dashboard, Portfolio, Spending, all QuickAdd flows to a local-time `todayLocalISO()` helper) | Settled — watch for regression in any NEW flow that defaults a date field |
|
||||
| Malformed `month` param → 500 | No validation on `YYYY-MM` parsing | a1b5763 — 400 on malformed month, validate custom range order | Settled |
|
||||
| Date-range queries subtly wrong | String date comparisons | dde3ee8 — proper date-range query handling (with indexes) | Settled |
|
||||
|
||||
## Chapter 5 — RSU model evolution (a design retreat)
|
||||
|
||||
The one place where the repo *built the sophisticated thing first and retreated to the simple thing*. Read this before proposing per-grant vesting schedules — it existed and was removed on purpose, same day it was built.
|
||||
|
||||
Timeline (all 2026-06-30):
|
||||
|
||||
1. 747e239 — unvested RSU grants: add, vest with withholding, delete.
|
||||
2. 6385c06 — **schedule generator**: cliff + quarterly tranches with preview table.
|
||||
3. 79d8220, 43a11fe — a cascade of fixes the schedule UI required: zero-price guards, blocked vested-delete, section gating, date parsing, per-row submitting state, server-side validation.
|
||||
4. a3487f9 — whole-share rounding, grant ticker price fetch, estimated values, configurable splitwise split.
|
||||
5. 225e69e — floor (not round) RSU preview net shares.
|
||||
6. c134801 — **retreat step 1**: schedule generator form replaced by a plain Add Grant form; vest takes *explicit net shares from the brokerage statement* instead of computing from a withholding % override. Rationale in commit: the brokerage is the source of truth for net shares.
|
||||
7. c8dd938 — **retreat step 2**: per-grant row UI replaced by one aggregated pot per RSU account (total pending gross shares by ticker) with inline Add-to-Pot and Vest actions; per-grant rows demoted to a collapsible Manage view.
|
||||
8. 51744a1 — post-vest guidance, per-holding error scoping.
|
||||
9. c835118 — withholding % had been hardcoded to 0 in the pot form (bug introduced by the simplification); now a UI field defaulting to 51%.
|
||||
10. babea72 — unvested value (after withholding) shown greyed-out on dashboard.
|
||||
11. bc256f6 — the **undeletable vested-grant deadlock**: deleting a vested grant returned 422, deleting its vest stock-transaction hit an FK 500 — a perfect deadlock. Fix: `delete_stock_transaction` reverts the grant to `pending` when its vest transaction is deleted.
|
||||
|
||||
**Do not redo:** rebuilding tranche schedules, computed withholding, or per-grant lot accounting. The complexity was tried; the maintenance cost (rounds 3–5 above) is why it was removed. The 51% withholding default is the user's real marginal rate — not a placeholder.
|
||||
|
||||
## Chapter 6 — Holdings and stock transactions
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| Cost basis drifted between create and delete paths | Create used incremental weighted-average update; delete used full recompute — two divergent implementations | c558999 — create ALSO uses `recompute_holding_from_stock_transactions`; deleted `_update_holding_buy`/`_update_holding_sell` | Settled. Do not reintroduce incremental holding math |
|
||||
| Over-sell silently clamped to zero shares | `sell_shares = min(tx.shares, shares)` hid data errors | c558999 — recompute raises `ValueError` (→ 422) if history over-sells at any point | Settled |
|
||||
| Stock tx delete / funded-buy bugs, fee ignored in balances | Assorted v1 backend bugs | 1480088 — over-sell guard, `SUM(amount - fee)` balance formula (now invariant #5), schema hardening, indexes | Settled |
|
||||
| Selling gave no way to receive proceeds | Funded path only supported buy | e36d719 — funded sell credits `shares × price − fee` to a bank account, typed `transfer` | Settled |
|
||||
|
||||
## Chapter 7 — Price fetching
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| All FX fetches suddenly failing | Frankfurter moved to `api.frankfurter.dev/v1`; old `api.frankfurter.app` URL dead | 582e39d | Settled — don't "fix" the URL back |
|
||||
| European ETF tickers (no exchange suffix) returned no data | yfinance needs `.DE`, `.L`, `.AS`, … suffixes | bc8c8d5 — auto-discovery: try bare ticker, then suffix list ordered by European-retail likelihood; cache under the *original* symbol; no-data ≠ transient error (only transient errors retry) | Settled |
|
||||
| Startup blocked / request handlers blocked on price fetches | Synchronous `fetch_all_prices` in startup and refresh endpoint (it sleeps 1s/2s/4s between retries) | ffa28bb — removed blocking startup fetch, made refresh async/queued | Settled — CLAUDE.md "What NOT to do" |
|
||||
| Refresh button lied ("updated: 0") | Queued refresh returned a fake count | a06f921 — honest "Refresh queued" message; bc256f6 — count only real updates | Settled |
|
||||
| Prices stale during the day | Only midnight scheduler | 0b5ba7c — 5-minute refresh interval | Settled |
|
||||
|
||||
## Chapter 8 — Build, packaging, environment
|
||||
|
||||
| Symptom | Root cause | Fix (SHA) | Status |
|
||||
|---|---|---|---|
|
||||
| `npm ci` crashed on node:20-alpine | npm exit-handler bug on Alpine | 89d241a — `npm install -g npm@11` before `npm ci`. Looks like cruft; is load-bearing | Settled |
|
||||
| Non-reproducible web builds | `npm install` from bare package.json | bfe7590 — `npm ci` + committed lockfile | Settled |
|
||||
| Builds failed outside corporate network | Lockfile pinned to Dynatrace's npm mirror | 3c2766e — re-resolved against the public registry | Settled |
|
||||
| Container crashed cryptically on unwritable volume / bad DATABASE_URL | No preflight; env-var handling scattered | d6001dc (unlabeled "fixes" — see Thin history below) — `app/db_config.py` (URL parsing, sqlite dir creation, writability check), `app/preflight.py` run before alembic in the Dockerfile CMD, `DATABASE_URL` default baked into the image, alembic env fallback chain. Error message explicitly mentions "volume mount permissions or Kubernetes fsGroup" — this was a real deploy failure | Settled |
|
||||
| Healthcheck failed although API was fine | `curl` doesn't exist in python:3.12-slim | 3e7f8ec — stdlib Python one-liner healthcheck | Settled |
|
||||
| Migrations not applied on deploy | Nothing ran alembic | 3e7f8ec — `alembic upgrade head` in Dockerfile CMD before uvicorn | Settled |
|
||||
| Alembic ignored DATABASE_URL | env.py read only alembic.ini | a8c94cc, extended by d6001dc | Settled |
|
||||
| Deploys served stale frontend | index.html browser-cached | bc256f6 — nginx `Cache-Control: no-cache` on index.html | Settled |
|
||||
| Container ran as root | v1 Dockerfile | ffa28bb — user `app` uid 1000 (invariant #4); bc256f6 adds `COPY --chown=app:app` | Settled |
|
||||
|
||||
## Chapter 9 — Input validation and UI correctness guards
|
||||
|
||||
Smaller but recurring: the frontend initially trusted inputs and hardcoded assumptions.
|
||||
|
||||
- 54d64e0 — `Literal` types for transaction/account type fields; readable 422 messages (garbage types previously reached the DB).
|
||||
- 256c27f — reject negative shares/price in add-trade form; 0a8a43f — block splitwise submit when computed user share is 0; bc256f6/e36d719 — reject 0% split and `partner_share >= total` server-side (422).
|
||||
- a3d9da7 — accept comma as decimal separator everywhere (European keyboards; "12,50" previously parsed as 12).
|
||||
- 00409f0 + 37bc101 — post-feature review sweeps: userShare guards, `initialAccountId` plumb-through, dialog state reset on `onOpenChange`, stale preview cleared on mode switch. Pattern: every big feature commit was followed by a same-day review-fix commit.
|
||||
- 424b926, 7794264, 9d06e1b — error handling on every page fetch; stale state cleared on mode switches.
|
||||
- dde3ee8 — FK enforcement ON for SQLite, security headers, category delete safety.
|
||||
|
||||
## Thin or ambiguous areas of history
|
||||
|
||||
- **No reverts, no branches**: `git log --all` shows 107 commits, all on `main`. Nothing was ever reverted; every correction is a forward fix. There is no abandoned-experiment branch to consult — the RSU schedule generator (Chapter 5) is the only "built then removed" feature, and it lives in mainline history.
|
||||
- **`9aae3e8` ("fix")** — inspected: removes the unused `hasReceivable` prop from Dashboard's `AccountCard`. Pure dead-code cleanup, 1 file, 4 lines. Its sibling 42308a1 finished the same cleanup. No behavior change.
|
||||
- **`d6001dc` ("fixes")** — inspected: NOT trivial despite the message. It is the deployment-hardening commit described in Chapter 8 (db_config.py, preflight.py, Dockerfile/compose changes). The Kubernetes fsGroup error message and `API_UPSTREAM` addition (0b5ba7c, same day) indicate the app was being adapted for the homelab/k8s deploy and hit a volume-permission failure in the wild. The commit message records none of this — the diff is the only evidence.
|
||||
- **Motivation gaps**: most one-line fix commits (e.g. 225e69e, 424b926) record *what* changed but not the incident that triggered it. Where this file states a symptom for those, it is inferred from the diff, and worded accordingly.
|
||||
- **Pre-history**: `PLAN.md` (commit a9de3c8) is the original design. Notable deviations that happened silently: `payable` account type added later (e3c8291), `manual_override_balance` largely replaced by balance-adjustment transactions (bf76e1d), and the 6-type transaction enum gained no new types but two retypings of existing rows (h8c9d0e1f2a3).
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
- Every SHA above was verified against this repo on **2026-07-05** by reading `git show <sha>` (commit message for all; full diff for the load-bearing ones: 6b7d19b, 81091fc, a1f109b, c558999, 225e69e, e61b254, e36d719/h8c9d0e1f2a3, bc8c8d5, d6001dc, 9aae3e8, 89d241a, bfe7590, 582e39d, a06f921, 37bc101, c835118). Design-pivot rationale from `docs/superpowers/specs/2026-07-01-category-types-receivable-redesign.md` and `docs/superpowers/plans/`.
|
||||
- Head commit at time of writing: `082b608` (fin CLI + opt-in bearer auth).
|
||||
- **To extend this chronicle**, find behavior-changing commits since the last audited SHA:
|
||||
```bash
|
||||
git log --oneline 082b608..HEAD
|
||||
git log 082b608..HEAD --grep='fix' -i --format='%h %s'
|
||||
# then, for each candidate:
|
||||
git show <sha> # read message AND diff before adding an entry
|
||||
```
|
||||
Also re-check for unlabeled fixes (messages like "fix"/"fixes"/"polish") — inspect their diffs, don't trust the subject line:
|
||||
```bash
|
||||
git log 082b608..HEAD --format='%h %s' | grep -viE 'feat|docs|chore'
|
||||
```
|
||||
- Entry format: Symptom → Root cause → Fix (SHA) → Status (settled / partially mitigated / watch for regression) → Do-not-redo note. Add new settled design questions to the top table. Never add an entry without reading the diff; never invent a symptom — mark it "inferred from diff" if the commit message doesn't state one.
|
||||
@@ -0,0 +1,269 @@
|
||||
---
|
||||
name: fin-ledger-analysis-toolkit
|
||||
description: Proof methods for fin's ledger. Load whenever you must PROVE an accounting property rather than eyeball it — verifying that balances/spending/net-worth are correct, deriving what a number SHOULD be before measuring it, computing the effect of a proposed change on aggregates before implementing it, adjudicating whether observed data is wrong vs expected, auditing a code path for currency correctness, checking a claimed fix actually fixed an aggregate, or stating/classifying ledger invariants (conservation, derivation, definition) as enforced vs unenforced. Also load before accepting any hypothesis about a money bug.
|
||||
---
|
||||
|
||||
# fin — Ledger Analysis Toolkit (proof methods)
|
||||
|
||||
House standard: **prove it, don't just eyeball it.** A number on a dashboard is a claim; this skill is the set of methods for deriving what the number *must* be from the transaction rows and the code, and for refusing conclusions that haven't survived an attempted refutation.
|
||||
|
||||
All facts below derived from code as of **2026-07-05** (HEAD = 082b608), with `file:function` citations. Worked examples use real commits via `git show`; where production amounts are not in the repo, numbers are **faithful synthetic reconstructions and labeled as such**.
|
||||
|
||||
Division of labor: `fin-accounting-reference` owns the *definitions and formulas* (balance formula, type-semantics table, two-leg mechanics). This skill owns the *proof methods built on them*. Don't re-derive definitions — cite that skill; do re-verify enforcement status, because that is this skill's subject.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- You need a formula or type-semantics definition → `fin-accounting-reference`.
|
||||
- Something is broken and you don't yet know where → `fin-debugging-playbook` first; come back here when you have a hypothesis to prove.
|
||||
- You want the history of a bug or whether a design question is settled → `fin-failure-archaeology`.
|
||||
- You need runnable audit scripts (the mechanized form of these methods) → `fin-diagnostics-and-tooling`.
|
||||
- You're deciding what evidence a change needs before "done" → `fin-validation-and-qa` (its evidence hierarchy is the acceptance policy; this skill supplies the proof techniques it ranks).
|
||||
- You're turning an invariant into a regression/property test → `fin-test-coverage-campaign`.
|
||||
- Design/refactor risk questions ("what breaks if I change this?") → `fin-architecture-contract`; change gating → `fin-change-control`.
|
||||
|
||||
---
|
||||
|
||||
## Method 1 — Invariant derivation & classification
|
||||
|
||||
**Purpose.** State a ledger invariant precisely enough to test it, then classify it honestly: ENFORCED (code prevents violation — cite where), INTENDED-UNENFORCED (design assumes it, nothing stops a violation — these are exactly where corruption enters), or STATISTICAL (expected to hold most of the time, gaps are normal).
|
||||
|
||||
**Inputs.** `api/app/crud.py`, `api/app/routers/transactions.py`, `api/app/schemas.py`, `api/app/models.py`, `api/app/services/net_worth.py`.
|
||||
|
||||
**Procedure.**
|
||||
1. Pick the quantity (a balance, a linked group, a holding) and write the invariant as an equation or a closed predicate over rows — no prose like "should be consistent". Three shapes exist in this codebase:
|
||||
- **Conservation**: a linked group of rows sums to a known constant (usually 0 or the recognized expense).
|
||||
- **Derivation**: a stored row is a pure function (fold) of other rows.
|
||||
- **Definition**: an aggregate is *defined* as a query; it cannot be "wrong", only mis-consumed.
|
||||
2. For each write path that can touch the rows (router → crud → raw POST), ask: what code would reject a violating write? If none, the invariant is unenforced no matter how much the happy-path code respects it.
|
||||
3. Check *at-rest* enforcement separately from *at-creation* enforcement: `create_transfer` writing both legs atomically does not stop a later single-leg write from another path.
|
||||
4. Record the classification with the citation. An invariant without a citation is a wish.
|
||||
|
||||
### Current invariant catalog (as of 2026-07-05)
|
||||
|
||||
**Conservation (linked groups; sums are over `amount` unless noted):**
|
||||
|
||||
| # | Invariant | Status |
|
||||
|---|---|---|
|
||||
| C1 | Same-currency transfer pair: `Σ amount = 0`; net-of-fee `Σ(amount−fee) = −fee` (fee sits on source leg only) | ENFORCED at creation (`crud.py:create_transfer` — `add_all` + single commit); **UNENFORCED at rest** (see U1) |
|
||||
| C2 | Cross-currency transfer: `−amount` in from-ccy, `+to_amount` in to-ccy; conservation holds only at the implied rate `amount/to_amount` | INTENDED-UNENFORCED — `to_amount` is a free field, no rate sanity check (`schemas.py:TransferRequest`, router only checks `> 0`) |
|
||||
| C3 | Settlement pair (`create_settlement`, `/reconcile`): `Σ amount = 0`, both fees 0 | ENFORCED at creation (`crud.py:create_settlement`, `routers/transactions.py:reconcile_accounts`); UNENFORCED at rest |
|
||||
| C4 | `splitwise_pay` pair: bank `−total`, splitwise `+partner_share` ⇒ `Σ amount = −(total − partner_share)` = the user's recognized expense | ENFORCED at creation (`crud.py:create_splitwise_i_paid`; router rejects `partner_share ≥ total`) |
|
||||
| C5 | Split-expense triple: `−user_share − partner_share + partner_share = −user_share` | ENFORCED at creation (`crud.py:create_split_expense`, atomic since bc256f6/e36d719) |
|
||||
| C6 | Funded buy/sell (cross-table): bank `transfer` leg + stock transaction satisfy `bank_Δ ≈ −cash_native/rate`; the "counter-leg" is the StockTransaction, not a Transaction row | INTENDED-UNENFORCED at rest — the bank leg is a deliberately **one-row** transfer (`crud.py:create_funded_buy`); deleting either side alone is allowed and leaves the other orphaned |
|
||||
|
||||
**Derivation:**
|
||||
|
||||
| # | Invariant | Status |
|
||||
|---|---|---|
|
||||
| D1 | `Holding(account,ticker)` = fold over all its StockTransactions ordered by `(date, id)`: buys/vests add `shares` and `shares×price+fee` to cost; sells remove at average cost; oversell at any historical point is rejected | ENFORCED — every insert/delete recomputes (`crud.py:recompute_holding_from_stock_transactions`, called from `create_stock_transaction`, `delete_stock_transaction`, `create_funded_buy`, `vest_rsu_grant`); oversell → ValueError → 422 |
|
||||
| D2 | Holding currency = currency of the **first** stock transaction; mixed-currency txns per ticker are not blocked | INTENDED-UNENFORCED (`recompute…`: `currency = txns[0].currency`) |
|
||||
| D3 | `StockTransaction.total_cost` is informational — the fold uses `shares×price_per_share+fee`, never `total_cost` | Definition/UNENFORCED — stored `total_cost` can diverge from the computed cost with no error |
|
||||
| D4 | Vested grant ↔ vest transaction: FK `rsu_grants.vested_stock_transaction_id`; deleting the vest txn reverts the grant to pending | ENFORCED (FK in `models.py:RSUGrant`; revert in `crud.py:delete_stock_transaction`, added in bc256f6) |
|
||||
|
||||
**Definition (aggregates — see fin-accounting-reference for full semantics):**
|
||||
|
||||
| # | Invariant | Where defined |
|
||||
|---|---|---|
|
||||
| B1 | balance ≡ `SUM(amount − fee)` per account, all types | `net_worth.py:calculate_net_worth` balance query |
|
||||
| S1 | monthly spending ≡ raw `expense`/`splitwise_receive` (amount<0, non-receivable accounts) + `splitwise_pay` at `amount + splitwise_partner_share` | `crud.py:get_monthly_spending` |
|
||||
| I1 | income ≡ `income` rows on non-payable/receivable accounts, excluding `description = 'Opening balance'` | `crud.py:get_monthly_spending` income query — string-match exclusion is fragile (a user-entered "Opening balance" income is silently excluded) |
|
||||
| N1 | net worth ≡ Σ per-account EUR balances (cash: B1; investment/rsu: `shares × cached_price` → `_to_eur`); missing FX rate ⇒ contribute **0.0** + warning | `net_worth.py:calculate_net_worth`, `_to_eur` |
|
||||
|
||||
**Integrity at rest (all INTENDED-UNENFORCED — the gap list):**
|
||||
|
||||
- **U1 — single-leg transfer/settlement via raw POST.** `POST /api/transactions` (`routers/transactions.py:create_transaction`) does **zero** validation beyond the Pydantic schema, and `schemas.py:TransactionBase` accepts `type` = any of the six including `transfer`/`settlement`, with `linked_account_id` freely settable and no partner row required. Verified truthfully: yes, a raw POST can create an unlinked `transfer` row. The CLI refuses (`cli/fin_cli/tx.py:TX_TYPES = ["expense","income","splitwise_pay","splitwise_receive"]`) — but that is client-side courtesy, not enforcement. (Small mercy: `linked_transaction_id` is NOT in the schema, so raw POST can't forge group links.)
|
||||
- **U2 — transaction currency vs account currency**: never checked anywhere; a USD row on a EUR account silently corrupts the native balance (balances are raw sums, no per-row FX).
|
||||
- **U3 — sign conventions**: raw POST accepts a positive-amount `expense` (drops out of spending via `amount < 0` filter but still credits the balance) and a negative `fee` (inflates balance via `amount − fee`).
|
||||
- **U4 — `linked_transaction_id` has no FK** (`models.py:Transaction` line 70 — plain `Integer`, not `ForeignKey`): dangling group links possible; `delete_transaction`'s graph walk silently skips missing partners.
|
||||
- **U5 — edit block covers only `linked_account_id`** (`routers/transactions.py:update_transaction`): the split-expense *expense* row carries only `linked_transaction_id`, so PUT can edit its amount, desyncing it from `total − partner_share` of its transfer legs.
|
||||
- **U6 — `PUT /api/accounts/{id}` can change `type`** with history attached (`crud.py:update_account` sets any field) — e.g. bank→investment makes its transactions invisible to net worth (investment balance comes from holdings).
|
||||
- **U7 — `splitwise_partner_share` unvalidated on raw POST**: can exceed `|amount|`, making the "user share" positive; `get_monthly_spending` then silently drops the category (`if total_amount >= 0: continue`).
|
||||
- **U8 — funded-buy FX fallback**: `routers/stock_transactions.py:create_funded_buy` gets its rate from `net_worth._get_eur_usd_rate`, which **still falls back to 1.0** when EUR/USD is missing — the 6b7d19b fix removed the fallback only in `_to_eur`. A cross-currency funded buy with a cold price cache converts at 1.0.
|
||||
|
||||
**Statistical:**
|
||||
|
||||
- **ST1 — snapshot continuity**: one `net_worth_snapshots` row per calendar day, written by the midnight cron (`services/scheduler.py:fetch_prices_and_snapshot`, upsert on `snapshot_date = date.today()`). Gaps mean the container was down at midnight — expected, not corruption. Never treat a missing day as a data bug; treat a *discontinuous jump* between adjacent days as a lead (diff `breakdown_json` per account).
|
||||
|
||||
**Most dangerous unenforced invariant: U1.** A single-leg `transfer` shifts one account's balance and net worth with no counterparty, and — because spending/income exclude `transfer` by type — it is invisible to the two aggregates a human actually reviews. It is the one corruption mode that evades every dashboard.
|
||||
|
||||
### Worked example — e36d719 as a precise classification violation
|
||||
|
||||
Real commit (`git show e36d719`): before the fix, `crud.py:create_funded_buy` wrote the bank cash leg as `type="expense"`, `description=f"Buy {shares} {ticker}"`, no `linked_account_id`. State the violated invariant precisely: *internal asset conversions must contribute 0 to spending and 0 to income* (they change asset composition, not net worth or consumption).
|
||||
|
||||
Faithful reconstruction (repo has no prod amounts): buy 10 VUAA at $100, fee $2 ⇒ `total_cost = 1002` USD; funding account EUR, `eur_usd_rate = 1.08`, no override ⇒ bank leg `amount = −1002/1.08 = −927.78 EUR`, type `expense`.
|
||||
|
||||
- Spending (S1): row matches `type in ('expense', …) AND amount < 0` ⇒ month total **inflated by 927.78** under category `None`. Violation: Δspending = +927.78, required 0.
|
||||
- Balance (B1): −927.78 either way — **unchanged by the fix**, because B1 sums all types.
|
||||
- Net worth (N1): unchanged (cash −927.78, holdings +~927.78 at cost).
|
||||
|
||||
The fix retyped the leg to `transfer` (excluded from S1 by type) and migration h8c9d0e1f2a3 retyped historical rows (regex `^Buy \d+(\.\d+)? [A-Za-z0-9.\-]{1,12}$` so "Buy groceries" is untouched). The signature of a *classification* bug: exactly one aggregate moves, balances and net worth stay fixed. If your fix for a classification bug changes a balance, you fixed the wrong thing.
|
||||
|
||||
**Failure modes of this method.** (1) Classifying by intent instead of code — "create_transfer is atomic" says nothing about at-rest enforcement; always check the raw POST path. (2) Missing a write path (migrations and the scheduler also write rows). (3) Catalog rot: enforcement status changes with every router/schema edit — re-verify citations before relying on a status, don't trust this table's date-stamped snapshot blindly.
|
||||
|
||||
---
|
||||
|
||||
## Method 2 — Delta analysis of a proposed change
|
||||
|
||||
**Purpose.** Compute the exact effect of a change on the three top-level aggregates — per-account balance (B1), monthly spending (S1/I1), net worth (N1) — **before** writing the code. The prediction is the acceptance test: after implementing, measured deltas must equal predicted deltas (fin-validation-and-qa's hypothesis-predicts-numbers discipline).
|
||||
|
||||
**Inputs.** The proposed diff (or migration), the aggregate queries in `crud.py:get_monthly_spending` and `net_worth.py:calculate_net_worth`, a small synthetic ledger.
|
||||
|
||||
**Procedure.**
|
||||
1. Identify every query whose row-set or per-row contribution the change touches. Grep for the column/type you're changing across `crud.py` and `net_worth.py` — a type retype touches S1/I1 (type filters) but not B1 (no type filter): read the filters, don't assume.
|
||||
2. Write each touched aggregate as an explicit formula over rows (copy the actual filter predicates).
|
||||
3. Build a 5-ish-row synthetic ledger that exercises every predicate branch the change touches (in-filter, out-of-filter, boundary).
|
||||
4. Hand-compute each aggregate before and after. Record the deltas, **including the aggregates predicted not to move** — those negative predictions are the cheapest bug detectors you own.
|
||||
5. After implementation, reconcile measured vs predicted (Method 3).
|
||||
|
||||
### Worked example — migration h8c9d0e1f2a3 (real file, synthetic ledger)
|
||||
|
||||
`api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py` does two UPDATEs: (a) `income`/`expense` rows with `linked_account_id IS NOT NULL` → `settlement` (legacy reconcile legs); (b) `expense` rows whose description matches the funded-buy regex → `transfer`.
|
||||
|
||||
Synthetic ledger, month 2026-06 (labeled reconstruction; account 1 = bank EUR, account 3 = receivable):
|
||||
|
||||
| id | acct | type (before) | amount | description | linked_acct |
|
||||
|---|---|---|---|---|---|
|
||||
| t1 | 1 | expense | −50.00 | groceries | — |
|
||||
| t2 | 1 | expense | −927.78 | Buy 10 VUAA | — |
|
||||
| t3 | 1 | income | +3000.00 | salary | — |
|
||||
| t4 | 1 | income | +200.00 | Received payment | 3 |
|
||||
| t5 | 3 | expense | −200.00 | Received payment | 1 |
|
||||
|
||||
Before: spending = 50 + 927.78 = **977.78** (t5 excluded — receivable account, S1's `Account.type.notin_(["receivable"])`); income = 3000 + 200 = **3200**; bank balance = −50 − 927.78 + 3000 + 200 = **2222.22**.
|
||||
|
||||
Migration hits: t2 (regex match) → `transfer`; t4, t5 (linked + income/expense) → `settlement`. t1, t3 untouched (no link, no regex match — "Buy groceries" would also survive, that's what the regex is for).
|
||||
|
||||
After: spending = **50.00** (Δ = −927.78); income = **3000.00** (Δ = −200.00); bank balance = **2222.22** (Δ = 0); net worth Δ = 0. The spending/income drop is **by design** — the migration exists to remove exactly those contributions; the zero deltas on balance and net worth are the safety proof that it is a pure re-labeling. If you had predicted these five numbers before running the migration on a DB copy, one `SELECT` per aggregate confirms it.
|
||||
|
||||
**Failure modes of this method.** (1) Synthetic ledger too clean — always include a row designed to be *incorrectly* caught (the "Buy groceries" decoy) and one on an excluded account type. (2) Forgetting month boundaries — S1 is per-month; a date-shifting change moves spending *between* months with zero total-across-months delta. (3) Predicting from the schema instead of the query — e.g. assuming income excludes opening balances "somehow" instead of reading the literal `description != 'Opening balance'` filter. (4) Not writing predictions down before measuring; post-hoc "yes that looks right" is eyeballing with extra steps.
|
||||
|
||||
---
|
||||
|
||||
## Method 3 — Reconciliation proof
|
||||
|
||||
**Purpose.** Prove a quantity by computing it three independent ways and showing agreement. **Rule of the house: any claim that a fix corrected an aggregate needs a reconciliation triple** — (1) the API's answer, (2) a raw-SQL recompute from first principles, (3) a hand calculation on the specific rows. Two agreeing numbers that share a code path prove nothing (the API and the frontend agree by construction).
|
||||
|
||||
**Inputs.** A **read-only copy** of the DB (prod DB is sacred — copy it out per fin-run-and-operate; never point tests or ad-hoc writes at it, per fin-validation-and-qa), the running API or `fin --json`, `sqlite3`.
|
||||
|
||||
**Procedure.**
|
||||
1. **API answer**: `curl -s http://<host>/api/net-worth/current | jq .` or `fin networth --json` / `fin spending --json` (CLI is a thin HTTP wrapper — same code path as the frontend, counts as ONE source).
|
||||
2. **Raw-SQL recompute** — re-derive from the definition, not from the API's implementation. Balances:
|
||||
```sql
|
||||
SELECT account_id, ROUND(SUM(amount - fee), 2) AS bal
|
||||
FROM transactions GROUP BY account_id;
|
||||
```
|
||||
Spending for a month (mirror S1's two parts *independently*):
|
||||
```sql
|
||||
SELECT ROUND(-SUM(t.amount), 2) FROM transactions t
|
||||
JOIN accounts a ON a.id = t.account_id
|
||||
WHERE t.date >= '2026-06-01' AND t.date < '2026-07-01'
|
||||
AND t.type IN ('expense','splitwise_receive') AND t.amount < 0
|
||||
AND a.type != 'receivable';
|
||||
SELECT ROUND(-SUM(t.amount + COALESCE(t.splitwise_partner_share,0)), 2)
|
||||
FROM transactions t JOIN accounts a ON a.id = t.account_id
|
||||
WHERE t.date >= '2026-06-01' AND t.date < '2026-07-01'
|
||||
AND t.type = 'splitwise_pay' AND t.amount < 0 AND a.type != 'receivable';
|
||||
```
|
||||
Holdings: fold `stock_transactions` per (account, ticker) ordered by `(date, id)` and compare to the `holdings` row (D1).
|
||||
3. **Hand calculation**: for the rows your change/bug touches (usually < 10), compute the contribution manually. This is the only leg that catches "the query and the API are consistently wrong together".
|
||||
4. Accept iff all three agree within **|Δ| < 0.01 per account/category** (money is `Float` everywhere — see Method 5; exact equality is not achievable and demanding it produces false alarms).
|
||||
5. Mechanized form: `.claude/skills/fin-diagnostics-and-tooling/` hosts executable audit scripts that are instances of these recomputes — use them for the SQL leg when they exist; the hand-calculation leg can never be mechanized away.
|
||||
|
||||
**Worked micro-example** (synthetic, from Method 2's ledger): claim "after migration, June spending is 50.00". Triple: `fin spending --month 2026-06 --json` → `total: 50.0`; SQL part 1 → 50.00, part 2 → NULL (no splitwise_pay rows); hand: only t1 qualifies, |−50| = 50. Agreement across three derivations ⇒ proven, not eyeballed.
|
||||
|
||||
**Failure modes of this method.** (1) Fake independence — recomputing with the ORM query copy-pasted from `crud.py` is source (1) again; write the SQL from the *definition*. (2) Reconciling against a moving target — the 5-minute price-refresh job (`scheduler.py`) mutates `price_cache`, so net-worth triples must pin prices (work on the DB copy, or capture `price_cache` first). (3) Currency blindness: the SQL balance is *native*; the API answer is EUR — convert with the same cached rate (`SELECT price FROM price_cache WHERE symbol='EUR/USD'`) before comparing, or you'll "find" an 8% bug. (4) Proving the aggregate but not the rows — totals can agree while two row-level errors cancel; when adjudicating a specific bug, reconcile the affected rows, not just the total.
|
||||
|
||||
---
|
||||
|
||||
## Method 4 — Currency-boundary analysis
|
||||
|
||||
**Purpose.** Audit any money-touching code path for currency correctness: where is a currency attached, converted, assumed, or silently defaulted?
|
||||
|
||||
**The conventions (as of 2026-07-05, from code):**
|
||||
- Every cached FX symbol is `EUR/X` meaning **X units per 1 EUR** (`price_fetcher.py:fetch_eur_usd` caches frankfurter's `from=EUR&to=USD` rate under `EUR/USD`). Therefore converting X→EUR is always a **division**: `_to_eur` does `amount_native / rate` (`net_worth.py:_to_eur`), and `create_funded_buy` does `cash_native / used_rate` (`crud.py`). Any multiplication by an `EUR/X` rate when going *to* EUR is a bug; multiplication is only correct going EUR→X.
|
||||
- Only **EUR/USD and EUR/GBP** are fetched (`price_fetcher.py:fetch_all_prices`). Any holding priced in CHF/JPY/SEK/… contributes **0.0** to net worth with a warning (N1). That is by design — see the worked example.
|
||||
- **GBp/pence handling**: yfinance reports LSE tickers in pence as `GBp`/`GBX`; `price_fetcher.py:_try_fetch_candidate` divides the price by 100 and stores currency `GBP`. If you ever see a UK holding ~100× too large, this normalization got bypassed.
|
||||
- Ticker prices are cached in **native currency** with the currency stored on the row (`price_cache.currency`); `calculate_net_worth` converts using the *price row's* currency (`ccy_map`), while cash balances convert using the *account's* declared currency. Transactions carry their own `currency` but balances never per-row convert (U2).
|
||||
|
||||
**Checklist — run this against any diff that touches money:**
|
||||
1. For every number, name its currency at every step (native → EUR is the only conversion direction in aggregates).
|
||||
2. Every conversion: is it a division by an `EUR/X` rate? If it multiplies, prove the direction.
|
||||
3. What happens when the rate is **missing**? Acceptable answers: exclude-with-warning (`_to_eur` → 0.0) or refuse (vest requires a cached price, `crud.py:vest_rsu_grant` → 422). Unacceptable: any silent numeric fallback. Known residual violation: **U8** — `_get_eur_usd_rate` still falls back to 1.0 and `routers/stock_transactions.py` feeds it to `create_funded_buy`.
|
||||
4. Could a *wrong currency's* rate be used? (The pre-6b7d19b bug class.)
|
||||
5. Pence/GBX: does the path consume `price_cache` (already normalized) or raw yfinance data (must normalize)?
|
||||
6. Does a currency default lurk? (`ccy = (price_ccy or "USD")` in `_to_eur`; `Transaction.currency` defaults `"EUR"`, `StockTransaction.currency` defaults `"USD"` — a caller omitting currency gets these silently.)
|
||||
|
||||
### Worked example — commit 6b7d19b (real diff; amounts reconstructed)
|
||||
|
||||
`git show 6b7d19b`: before, `_to_eur` with a missing `EUR/{ccy}` rate fell back to `amount_native / EUR_USD_rate` — converting with a **different currency's** rate. Reconstruction: 100 shares NESN.SW at 90 CHF = 9,000 CHF; cache has `EUR/USD = 1.08` but no `EUR/CHF` (true today — CHF is never fetched). Fallback books `9000 / 1.08 = 8333.33` "EUR"; the correct value at EUR/CHF ≈ 0.93 is `9000 / 0.93 = 9677.42`. Silent error: **−1344.09 EUR (−13.9%)**, buried inside a plausible-looking total.
|
||||
|
||||
Why returning 0 is *better* than returning the unconverted-or-misconverted number: (a) 8333.33 is plausible — no one investigates plausible; a 9,677 hole in net worth is investigated the same day; (b) the midnight job **persists** net worth into `net_worth_snapshots` — a plausible wrong value corrupts the historical series permanently, while an obvious dip is a visible, dateable, explainable artifact; (c) the warning log names the missing rate, turning a silent data bug into an ops task ("fetch EUR/CHF"). This is the codebase's settled loud-zero-over-plausible-lie policy (fin-failure-archaeology: no-1.0-fallback, removed twice) — U8 is the one place it hasn't reached yet.
|
||||
|
||||
**Failure modes of this method.** (1) Auditing only the read path — currency bugs are often planted at *write* time (RSU vest hardcoded `"USD"` until e36d719 fixed it to read `price_cache.currency`). (2) Trusting variable names — `eur_usd_rate` in `create_funded_buy` is used for *any* account-vs-stock currency pair, not just EUR/USD; the name lies when the account is GBP. (3) Checking direction with round-trip algebra on symmetric test values (rate 1.0, amount 100) — always test with an asymmetric rate like 1.08 so a flipped division actually shows.
|
||||
|
||||
---
|
||||
|
||||
## Method 5 — Boundary/edge catalog for money math
|
||||
|
||||
**Purpose.** The edges this codebase has historically gotten wrong, each with the standard probe to run against new money code. Every entry is a real commit.
|
||||
|
||||
| Edge | Incident (real commit) | Standard probe |
|
||||
|---|---|---|
|
||||
| Zero share / 0% split | 0a8a43f — SplitwiseShePayed with 0% computed user share submitted a 0-amount row; blocked client-side. Backend: split endpoints require `partner_share > 0` and `< total` (`schemas.py` `gt=0`, router checks); raw POST still accepts `amount = 0` (U3-adjacent) | Feed 0 and the value that *computes* to 0 (e.g. total = 2×partner_share rounding to user share 0.00) |
|
||||
| Negative inputs | 256c27f — Portfolio add-trade accepted negative shares/price (a "negative buy" is a disguised sell that corrupts avg cost). Backend now `gt=0` on `StockTransactionBase`; **`Transaction.amount`/`fee` remain unvalidated on raw POST** | Probe every numeric field with −1; decide reject-vs-meaning explicitly |
|
||||
| Comma decimals | a3d9da7 — `parseFloat("12,50")` silently returns **12**; German-locale entry lost cents/magnitude. QuickAdd inputs now normalize commas | Enter "12,50" and "1.234,56" in any new amount input; assert 12.50, not 12 |
|
||||
| UTC vs local midnight | bc256f6 — frontend default dates used `toISOString().split('T')[0]` (UTC); entries between 00:00 and ~01–02 CET booked to **yesterday**, silently moving spending across month boundaries. Fixed with `todayLocalISO()` (`web/src/lib/utils.ts`) | Test any date-defaulting path at simulated 00:30 local; grep new frontend code for `toISOString` |
|
||||
| Float money | No single incident — a standing condition: **every money column is `Float`** (`models.py`: `amount`, `fee`, `shares`, `price_per_share`, `total_cost`, `total_eur`, …). There is no Decimal and no backend rounding policy; the only rounding in the stack is frontend cents-rounding of `partner_share` (bc256f6) and display rounding (`round(pct,1)` in `get_monthly_spending`). Implications: (a) reconciliation must use tolerance |Δ| < 0.01, never equality (Method 3); (b) `recompute…` uses a **strict** `if tx.shares > shares` compare — selling exactly-all after fractional accumulations (0.1+0.2 ≠ 0.3) can 422 a legitimate sell-all, or conversely permit a dust oversell within epsilon | Sell-all after ≥3 fractional buys; sum 0.1+0.2-style amounts through an aggregate and check the API's raw JSON for 0.30000000000000004-type values |
|
||||
| RSU whole-share flooring | 225e69e — vest preview must **floor**, not round: `net = floor(gross × (1 − withholding))` (`crud.py:vest_rsu_grant`); schedule generator floors per tranche and pushes the remainder into the last (`crud.py:_generate_rsu_schedule`); `net_shares ≤ 0` → 422 | gross=10, pct=0.51 ⇒ 4 (floor of 4.9 — round would say 5); gross=1, pct=0.5 ⇒ 0 ⇒ must 422 |
|
||||
|
||||
**Failure modes of this catalog.** It is a list of *known* edges — passing all probes proves you didn't repeat history, not that no new edge exists. And probes at the frontend prove nothing about the API (comma handling, negative rejection are client-side in several places): probe the layer you changed.
|
||||
|
||||
---
|
||||
|
||||
## Method 6 — Adversarial self-refutation
|
||||
|
||||
**Purpose.** Before accepting any conclusion about ledger data ("the bug is X", "the data is actually correct", "the fix worked"), attack it yourself. Sonnet-and-owner failure mode this method exists to stop: the first mechanism that explains the headline observation gets accepted while a cheaper observation that contradicts it is never checked.
|
||||
|
||||
**Procedure.**
|
||||
1. Write down **every** observation the hypothesis must explain — explicitly including the negatives: *which aggregates did NOT change? which rows were NOT affected? who is NOT seeing the problem?* The negative space discriminates between mechanisms far better than the headline symptom (Method 2's classification signature — "spending moved, balance didn't" — is exactly such a negative).
|
||||
2. Construct at least **one alternative mechanism** that explains the headline observation. If you can't think of one, you haven't understood the system; check the layer above and below (frontend default vs backend default vs DB default all produce "wrong value in DB").
|
||||
3. Find a **discriminating check**: an observation the two mechanisms predict differently. Run it.
|
||||
4. Accept the hypothesis only when the alternative is *falsified by the check* — not when it merely feels less likely.
|
||||
5. For fixes: the prediction from Method 2 plus the triple from Method 3 is the acceptance; "the number looks right now" is not.
|
||||
|
||||
### Worked example — the bc256f6 off-by-one-day bug (real incident; investigation reconstructed faithfully from the commit message and diff)
|
||||
|
||||
Headline observation: trades, vests, and settlements entered via the UI sometimes carry **yesterday's** date.
|
||||
|
||||
Full observation list, including negatives:
|
||||
- O1: affected rows are dated one day early;
|
||||
- O2 (negative): amounts, accounts, balances all correct — only the `date` column;
|
||||
- O3: only entries made shortly after local midnight (before ~01–02 CET) are affected;
|
||||
- O4 (negative): rows created with an explicitly typed date, and CLI/API-created rows, are correct;
|
||||
- O5 (negative): the scheduler's snapshot dates are correct.
|
||||
|
||||
Hypothesis A (first instinct): the backend/server clock or timezone shifts dates. Hypothesis B: the **frontend's default date** is computed in UTC.
|
||||
|
||||
Discriminating checks: (i) A predicts O4 fails — the backend would shift *stored* dates regardless of who sent them; but explicitly-dated rows are stored verbatim (the `Date` column stores what it's given, `models.py:Transaction.date`), so A is falsified by O4. (ii) A predicts O5 fails too — the scheduler runs server-side (`scheduler.py` uses `date.today()`), and its snapshots were fine. (iii) B predicts the bug lives where defaults are computed client-side: `grep -rn "toISOString" web/src` found `toISOString().split('T')[0]` in Dashboard, Portfolio, Spending and the QuickAdd flows — JS `toISOString()` is UTC, so between local midnight and UTC midnight it yields yesterday. B explains O1–O5 with nothing left over; A is dead on two independent checks. The fix (`todayLocalISO()` helper, bc256f6) touched only the frontend — which is itself a post-hoc confirmation that B was the mechanism.
|
||||
|
||||
The refutation discipline is what made this cheap: checking O4 (one CLI insert with an explicit date) cost seconds and killed the expensive hypothesis (timezone archaeology on the container) before anyone spent an hour on it. Also note O3's precision — "before ~01–02 CET" is CET/CEST's UTC offset speaking; a hypothesis that can't derive the cutoff hour from its mechanism doesn't actually explain O3.
|
||||
|
||||
**Failure modes of this method.** (1) Strawman alternatives — the alternative must be one you'd genuinely believe if the check went the other way. (2) Confirmation-shaped checks — pick the observation the hypotheses *disagree* on, not another one they both predict. (3) Stopping at one bug: bc256f6 was a bug-*hunt* commit; after falsifying A for one symptom, re-run the loop per symptom — several independent mechanisms (currency hardcoding, non-atomic split expense, vest-unlink deadlock) were live simultaneously, and a single unified explanation for all of them would have been wrong.
|
||||
|
||||
---
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All claims verified against HEAD = 082b608, 2026-07-05. Re-verification one-liners (from repo root):
|
||||
|
||||
- Invariant catalog / U1: `sed -n '24,29p' api/app/routers/transactions.py` (create_transaction has no checks) and `grep -n "TX_TYPES" cli/fin_cli/tx.py` (CLI refusal).
|
||||
- U4: `grep -n "linked_transaction_id" api/app/models.py` — still plain `Integer`?
|
||||
- U5: `grep -n "linked_account_id is not None" api/app/routers/transactions.py` — edit block still ignores `linked_transaction_id`-only rows?
|
||||
- U8: `grep -n "_get_eur_usd_rate" api/app/routers/stock_transactions.py api/app/services/net_worth.py` — 1.0 fallback still feeding funded buys?
|
||||
- B1/S1/I1/N1 formulas: `grep -n "amount - models.Transaction.fee" api/app/services/net_worth.py`; `grep -n "splitwise_partner_share\|Opening balance" api/app/crud.py`.
|
||||
- FX direction & GBp: `grep -n "amount_native / rate" api/app/services/net_worth.py`; `grep -n "GBp\|GBX" api/app/services/price_fetcher.py`; fetched rates: `grep -n "fetch_eur_" api/app/services/price_fetcher.py`.
|
||||
- Float money: `grep -c "Float" api/app/models.py` (any Decimal yet?).
|
||||
- Worked-example commits still resolve: `git show --stat e36d719 6b7d19b bc256f6 225e69e a3d9da7 0a8a43f 256c27f | grep ^commit`.
|
||||
- Migration example: `sed -n '30,52p' api/alembic/versions/h8c9d0e1f2a3_retype_internal_cash_flows.py`.
|
||||
|
||||
If any check disagrees with this file, the code wins — update the catalog entry and its date stamp in the same commit as the change (per fin-change-control's docs-of-record rule).
|
||||
@@ -0,0 +1,635 @@
|
||||
---
|
||||
name: fin-research-frontier
|
||||
description: >-
|
||||
The ranked research-and-development frontier for the fin app — what to build
|
||||
next and why. Load when choosing the next significant project, when asked
|
||||
"what should I work on next", "what's next", "what's missing", or "what's
|
||||
the roadmap", and BEFORE starting any MCP-server, forecasting, budgeting,
|
||||
recurring-transaction, portfolio-analytics, backup-automation, or
|
||||
correctness-hardening work. Contains per-item first steps, milestones, and
|
||||
the anti-backlog of projects deliberately not worth doing.
|
||||
---
|
||||
|
||||
# fin — Research Frontier
|
||||
|
||||
The owner's four ambitions for this codebase, broken into concrete, ranked
|
||||
projects. Every repo claim below was verified by reading the code on
|
||||
2026-07-05 (commit 082b608, plus uncommitted `.claude/` work). Nothing here is
|
||||
described as done unless it is; nothing is described as easy unless the
|
||||
enabling code was actually read.
|
||||
|
||||
Ambitions: **(a)** provable ledger correctness, **(b)** agent-native finance,
|
||||
**(c)** analytics & forecasting, **(d)** practical backlog & polish.
|
||||
|
||||
How to use this skill:
|
||||
|
||||
1. Pick the highest-ranked `open` item in the theme you care about. Items are
|
||||
ranked by value÷risk within each theme.
|
||||
2. Re-verify the "why current state falls short" claim before starting — the
|
||||
code may have moved. Each item cites the exact file.
|
||||
3. Any behavior/schema change routes through `fin-change-control` (gates,
|
||||
migration discipline) and `fin-validation-and-qa` (evidence before "done").
|
||||
Design constraints live in `fin-architecture-contract` — several items
|
||||
below map directly onto its §5 "known weak points" table; the mapping is
|
||||
noted per item.
|
||||
|
||||
Status labels: `open` (ready to start), `candidate` (worth doing, but do the
|
||||
listed prerequisite or read-only probe first), `blocked-on-X` (a named thing
|
||||
must land first).
|
||||
|
||||
---
|
||||
|
||||
## Theme A — Provable ledger correctness
|
||||
|
||||
### A1. Double-entry hardening: make intended invariants enforced — `partly-done 2026-08-22`
|
||||
|
||||
> **Status update 2026-08-22.** The create half is closed: `TransactionCreate`
|
||||
> now narrows `type` to the single-leg kinds (expense/income/splitwise_pay/
|
||||
> splitwise_receive) and rejects a caller-supplied `linked_account_id`, so
|
||||
> `POST /api/transactions` can no longer write an orphan leg. Covered by
|
||||
> `cli/tests/test_idempotency.py::TestOrphanLegGuard`. Note `splitwise_pay`
|
||||
> stays allowed — a standalone row carrying `splitwise_partner_share` is a
|
||||
> legitimate record and is what `fin tx add --type splitwise_pay` writes.
|
||||
> **Still open:** only the measurement. The audit script itself already
|
||||
> exists as `fin-diagnostics-and-tooling/scripts/audit_linked_legs.py` and
|
||||
> already encodes the funded-buy whitelist described below; it has not been
|
||||
> run against a prod backup, so the "0 violations" half of the result
|
||||
> criterion is unmeasured.
|
||||
|
||||
**Why current state falls short.** The two-leg invariant is enforced only in
|
||||
the dedicated endpoints. The raw `POST /api/transactions` accepts
|
||||
`type: "transfer"` and `type: "settlement"` — both are in the
|
||||
`TransactionCreate` Literal (`api/app/schemas.py:77`) and the schema even
|
||||
accepts a free-standing `linked_account_id` (`schemas.py:83`) — and
|
||||
`crud.create_transaction` (`api/app/crud.py:152`) writes
|
||||
`models.Transaction(**model_dump)` verbatim with no linkage validation.
|
||||
Verified 2026-07-05: nothing stops a client from creating a single unlinked
|
||||
transfer leg, which is exactly the balance corruption that invariant #1 in
|
||||
CLAUDE.md exists to prevent. The router blocks *editing* linked legs
|
||||
(`routers/transactions.py:36-40`) but not *creating* orphans. Maps to
|
||||
`fin-architecture-contract` §3 "two-leg atomic linked writes".
|
||||
|
||||
**Tractable because.** Every legitimate multi-leg write already has a
|
||||
dedicated atomic endpoint (`/transfer`, `/settlement`, `/split-expense`,
|
||||
`/splitwise-i-paid`, `/reconcile`) — rejecting raw multi-leg types loses no
|
||||
capability. And the whole ledger is one SQLite file, so "are there existing
|
||||
orphans?" is one query.
|
||||
|
||||
**First three steps.**
|
||||
1. Grep all three clients for raw creates with these types before changing
|
||||
anything: `grep -n "createTransaction\|create_transaction" web/src cli/fin_cli -r`
|
||||
and check `web/src/components/QuickAdd/flows/` and `cli/fin_cli/tx.py`
|
||||
only ever send `expense`/`income` through the raw endpoint.
|
||||
2. In `api/app/routers/transactions.py::create_transaction`, reject
|
||||
`type in ("transfer", "settlement")` and any non-null `linked_account_id`
|
||||
with a 422 pointing at the dedicated endpoints. Router-level validation is
|
||||
the sanctioned layer for this (`fin-architecture-contract` §2).
|
||||
3. Add an e2e test in `cli/tests/test_cli.py` asserting the raw create of an
|
||||
unlinked transfer is rejected, and write the orphan-leg audit query (leg
|
||||
with `linked_account_id` set, `linked_transaction_id` null, no reciprocal
|
||||
partner) as a read-only script for `fin-diagnostics-and-tooling` —
|
||||
**caveat found while verifying**: funded-buy bank legs
|
||||
(`crud.create_funded_buy`, line 603-612) legitimately have
|
||||
`linked_account_id` set and `linked_transaction_id` null (their "partner"
|
||||
is a StockTransaction, not a Transaction). The audit must whitelist legs
|
||||
whose linked account is type `investment`/`rsu`, or it will false-positive
|
||||
on every funded buy.
|
||||
|
||||
**You have a result when** the new e2e test passes (raw unlinked
|
||||
transfer/settlement → 422) AND the orphan audit run against a prod backup
|
||||
reports 0 violations outside the funded-buy whitelist. Measurable: test count
|
||||
goes from 25 to ≥26 and the audit script prints a violation count.
|
||||
|
||||
### A2. Nightly self-audit job — `open`
|
||||
|
||||
**Why current state falls short.** No integrity check exists anywhere: no
|
||||
`api/tests/`, no CI (`fin-architecture-contract` §5 #1), and derived state
|
||||
(holdings) plus linked-leg groups can silently desync — e.g. the funded-buy
|
||||
pair gap in D3 below is invisible until a balance looks wrong. Errors are
|
||||
currently discovered by a human noticing a wrong number.
|
||||
|
||||
**Tractable because.** Every invariant is expressible as SQL over one SQLite
|
||||
file, and `api/app/services/scheduler.py` is a clean extension seam: verified
|
||||
2026-07-05, `create_scheduler` (line 84) registers jobs via `add_job`, and
|
||||
both existing jobs show the exact thread-safe pattern to copy — build a
|
||||
private engine via `_make_engine(database_url)` (WAL + FK pragmas, line
|
||||
19-28), open a session, `dispose()` in `finally`. A third job is ~10 lines of
|
||||
registration.
|
||||
|
||||
**First three steps.**
|
||||
1. Create `api/app/services/integrity.py` with pure functions
|
||||
`(db) -> list[violation dict]`: orphan linked legs (with the funded-buy
|
||||
whitelist from A1), holdings vs. a re-run of
|
||||
`recompute_holding_from_stock_transactions` fold, snapshot-date gaps,
|
||||
`price_cache.fetched_at` staleness, vested RSU grants whose
|
||||
`vested_stock_transaction_id` dangles. Reuse the query shapes from the
|
||||
`fin-diagnostics-and-tooling` scripts rather than inventing new ones.
|
||||
2. Register a `nightly_audit` job in `create_scheduler` (cron 00:30, after
|
||||
the 00:00 snapshot job), following the `fetch_prices_and_snapshot`
|
||||
engine-per-run pattern exactly; log findings at WARNING. Log-only first —
|
||||
no schema, no migration, trivially revertable.
|
||||
3. Expose `GET /api/integrity` (new router registered in `main.py` **with**
|
||||
`dependencies=_auth` — see `fin-architecture-contract` §5 #9) + a
|
||||
`FinClient.get_integrity()` method and `fin` subcommand, so both the owner
|
||||
and agents can pull the latest result on demand.
|
||||
|
||||
**You have a result when** a test fixture DB with one deliberately deleted
|
||||
transfer leg (raw SQL on a throwaway copy — never prod) yields exactly that
|
||||
one violation from the audit functions, and a clean fixture yields zero.
|
||||
Measurable: an e2e test asserts violation count 1 and 0 respectively.
|
||||
|
||||
### A3. Property-based ledger testing — `candidate` (execution lives in `fin-test-coverage-campaign`)
|
||||
|
||||
**Why current state falls short.** The 25 e2e tests in
|
||||
`cli/tests/test_cli.py` certify specific scenarios; nothing certifies the
|
||||
ledger under *arbitrary* operation sequences (create/delete transfers,
|
||||
funded buys, RSU vests in random order). crud.py is 995 lines with no direct
|
||||
tests (`fin-architecture-contract` §5 #1).
|
||||
|
||||
**Tractable because.** `FinClient` accepts an injected httpx transport /
|
||||
client (`cli/fin_cli/client.py:31-54`, verified), so a Hypothesis strategy
|
||||
can drive the full stack (FinClient → TestClient → SQLite) exactly as the
|
||||
existing suite does, and check invariants after every step: per-account
|
||||
balance = `SUM(amount − fee)`, transfer groups net to zero in same-currency
|
||||
cases, holdings equal the fold of stock_transactions, delete is a true
|
||||
inverse.
|
||||
|
||||
**First three steps + milestone.** Deliberately not duplicated here — this is
|
||||
ambition (a)'s executable flagship and is specified in
|
||||
`fin-test-coverage-campaign`. Start there; this entry exists only so the
|
||||
frontier list is complete. Result criterion (shared with that skill): a
|
||||
Hypothesis test that shrinks a real invariant violation to a minimal
|
||||
counterexample, or N≥100 random 20-op sequences passing all invariants.
|
||||
|
||||
---
|
||||
|
||||
## Theme B — Agent-native finance
|
||||
|
||||
### B1. MCP server wrapping FinClient — done (2026-07-09, uncommitted): read-only phase + gated writes implemented
|
||||
|
||||
**Why current state falls short.** Agents today reach fin only through the
|
||||
`fin` CLI (via the `fin` skill) — every query is a subprocess with text
|
||||
parsing, and there is no tool-schema contract. No MCP server exists in the
|
||||
repo (verified: `cli/` contains only `fin_cli/`, `tests/`, `pyproject.toml`).
|
||||
|
||||
**Tractable because.** This was designed for. Verified 2026-07-05:
|
||||
`cli/fin_cli/client.py` imports only `os`, `typing`, and `httpx` — zero
|
||||
typer/rich — and already covers every endpoint (net worth, accounts,
|
||||
transactions incl. all atomic multi-leg creators, holdings, RSU grants,
|
||||
spending, prices). The atomic endpoints are the safety asset: an agent
|
||||
calling `create_transfer` can never half-write a ledger group. Errors arrive
|
||||
as `FinClientError` with status codes — clean tool-error mapping.
|
||||
|
||||
**First three steps.**
|
||||
1. Sketch package: `cli/fin_mcp/server.py` (FastMCP from the `mcp` SDK)
|
||||
importing `fin_cli.client.FinClient`; add an optional dependency group
|
||||
`mcp` to `cli/pyproject.toml` and a `fin-mcp` entry in `[project.scripts]`
|
||||
so the core CLI install stays dependency-light. NOTE: the sandbox has no
|
||||
pip/venv — implement statically, verify on the homelab or in the api
|
||||
container (see `fin-build-and-env`).
|
||||
2. Register **read-only tools only**: `get_net_worth`, `list_accounts`,
|
||||
`list_transactions`, `get_spending_month`, `get_spending_summary`,
|
||||
`get_holdings`, `list_rsu_grants`, `get_fx_rate`. Each returns the API
|
||||
JSON untouched — same payloads `fin --json` already emits, so the `fin`
|
||||
skill's documented semantics carry over.
|
||||
3. Test without a network: reuse the existing suite's pattern of building
|
||||
`FinClient(http=...)` over FastAPI's TestClient and call the tool
|
||||
functions directly in `cli/tests/` (new `test_mcp.py`).
|
||||
|
||||
**You have a result when** an MCP client config pointing at `fin-mcp` lists
|
||||
the read-only tools and `get_net_worth` returns byte-equal JSON to
|
||||
`fin networth --json` against the same API. Measurable: one e2e test
|
||||
asserting tool output == FinClient output for three tools.
|
||||
|
||||
Write tools (`create_transaction`, `create_transfer`, …) are `candidate`:
|
||||
gate them behind an explicit env flag (e.g. `FIN_MCP_ALLOW_WRITES=1`) and do
|
||||
B3's safety rails first. For any deployment where the API is reachable beyond
|
||||
the LAN, writes are additionally **blocked-on-B2**.
|
||||
|
||||
### B2. Auth without locking out the frontend — `open` (prerequisite for exposed write access)
|
||||
|
||||
**Why current state falls short.** Verified and documented everywhere
|
||||
(CLAUDE.md; `fin-architecture-contract` §5 #4): setting `FIN_API_TOKEN` on
|
||||
the API locks out the React frontend entirely — `web/src/api/client.ts`
|
||||
sends no `Authorization` header and nginx injects nothing. So today the
|
||||
choice is "no auth" or "no web UI". Any agent-write story that leaves the
|
||||
LAN needs this fixed first.
|
||||
|
||||
**Tractable because.** The single-user model means one static bearer token is
|
||||
sufficient — no user table, no sessions. Nginx already fronts every `/api/*`
|
||||
call from the browser (`web/nginx.conf`), so injecting
|
||||
`proxy_set_header Authorization "Bearer <token>"` in that one proxy block
|
||||
covers the whole SPA without the token ever reaching the browser.
|
||||
|
||||
**First three steps.**
|
||||
1. Read `web/nginx.conf` and `web/Dockerfile` to pick the templating
|
||||
mechanism (the stock nginx image runs `envsubst` over
|
||||
`/etc/nginx/templates/*.template`; check whether the current conf is
|
||||
copied verbatim or templated — `API_UPSTREAM` handling shows the current
|
||||
approach).
|
||||
2. Add an optional env (e.g. `WEB_API_BEARER`) that, when set, injects the
|
||||
header in the `/api/` proxy block only; when unset, config is
|
||||
byte-identical to today. Wire it in `docker-compose.yml`.
|
||||
3. Update the configuration catalog in `fin-run-and-operate` §2 and the
|
||||
CLAUDE.md warning paragraph; add the threat-model caveat that the token
|
||||
then protects only direct `:8000` access, not the nginx-fronted UI.
|
||||
|
||||
**You have a result when**, with `FIN_API_TOKEN` set on the api container and
|
||||
`WEB_API_BEARER` set on web: the Dashboard loads and a QuickAdd expense
|
||||
saves; `curl` direct to the API without a token gets 401; and with both
|
||||
unset, behavior is unchanged. Three checks, all binary. (Runtime verification
|
||||
happens on the homelab — sandbox has no docker; see `fin-run-and-operate`.)
|
||||
|
||||
### B3. Natural-language entry safety rails: dry-run + idempotency — `partly-done 2026-08-22`
|
||||
|
||||
> **Status update 2026-08-22.** Idempotency shipped. Migration
|
||||
> `i9d0e1f2a3b4` adds a nullable `client_ref` to `transactions`, accepted by
|
||||
> `POST /api/transactions`, `/batch`, `/transfer` and `/settlement`, and
|
||||
> exposed on the MCP `create_transaction`/`create_transfer`/
|
||||
> `create_settlement` tools. Every row of one write shares the key and the
|
||||
> creators check `crud.find_by_client_ref` first, so a retry replays the
|
||||
> original rows. **Deviation from the plan below:** the index is NOT
|
||||
> unique-when-not-null — a transfer writes two rows and a batch many under
|
||||
> one key, so uniqueness is enforced by check-then-write instead, which is
|
||||
> sound because SQLite serialises writers. Covered by
|
||||
> `cli/tests/test_idempotency.py`.
|
||||
> **Still open:** the `dry_run` flag on the multi-leg creators, and the
|
||||
> duplicate-suspect diagnostic query.
|
||||
|
||||
**Why current state falls short.** NL parsing itself already works — the
|
||||
`fin` skill has Sonnet-class agents booking via CLI today. What the repo
|
||||
lacks is machinery that makes autonomous booking *safe*: there is no dry-run
|
||||
mode on any write endpoint (verified: no such param in
|
||||
`routers/transactions.py` or `schemas.py`) and no idempotency mechanism — an
|
||||
agent that retries a timed-out `create_transfer` books it twice, and nothing
|
||||
detects the duplicate.
|
||||
|
||||
**Tractable because.** The multi-leg creators are already pure
|
||||
"request-schema in → legs out" functions in crud.py; a dry-run that builds
|
||||
legs, returns them, and rolls back instead of committing reuses the exact
|
||||
code path. Duplicate detection has full history in one SQLite file to check
|
||||
against.
|
||||
|
||||
**First three steps.**
|
||||
1. Read-only probe first: a duplicate-suspect query (same account pair, same
|
||||
amount, same date, distinct link groups) as a `fin-diagnostics-and-tooling`
|
||||
script — measures whether double-booking has ever actually happened.
|
||||
2. Add `dry_run: bool = False` to `TransferRequest`/`SettlementRequest` in
|
||||
`schemas.py`; in the crud creators, on dry-run build the legs, serialize,
|
||||
`db.rollback()`, return with a marker. No schema migration needed.
|
||||
3. Idempotency needs a nullable `client_ref` column on `transactions` →
|
||||
Alembic revision chained after `h8c9d0e1f2a3` (this is the migration-gated
|
||||
half; route through `fin-change-control` first). Unique-when-not-null;
|
||||
creators return the existing group on a match instead of writing.
|
||||
|
||||
**You have a result when** an e2e test submits the same transfer twice with
|
||||
one `client_ref` and exactly one two-leg group exists afterwards, and a
|
||||
dry-run call leaves the transaction count unchanged while returning the
|
||||
would-be legs. Status detail: dry-run is startable now; idempotency is
|
||||
blocked-on a reviewed migration; both are most useful **after B1**, since
|
||||
MCP tool descriptions are where the confirm-before-write protocol gets
|
||||
enforced.
|
||||
|
||||
---
|
||||
|
||||
## Theme C — Analytics & forecasting
|
||||
|
||||
### C1. Category budget targets — `open`
|
||||
|
||||
**Why current state falls short.** No budget concept exists anywhere:
|
||||
`grep -ri budget api/app web/src cli/fin_cli` returns nothing (verified
|
||||
2026-07-05). The Spending page shows actuals only; there is no target to
|
||||
compare against, which blocks the most basic "am I on track this month"
|
||||
question.
|
||||
|
||||
**Tractable because.** `get_monthly_spending` (`api/app/crud.py:624`) already
|
||||
returns per-category totals with names/colors/icons — a budget is one
|
||||
nullable column and one extra field in that response. The Spending page
|
||||
already renders the per-category list to attach progress bars to.
|
||||
|
||||
**First three steps.**
|
||||
1. Add `monthly_budget = Column(Float, nullable=True)` to `Category` in
|
||||
`api/app/models.py`; generate the Alembic revision chained after
|
||||
`h8c9d0e1f2a3`, review it, and gate via `fin-change-control` (schema
|
||||
class).
|
||||
2. Surface it: `CategoryCreate`/`CategoryUpdate` in `schemas.py` (Optional,
|
||||
`exclude_unset` pattern), and add `budget`/`remaining` to each category
|
||||
dict in `get_monthly_spending`'s `category_data` loop (crud.py:714-721).
|
||||
3. Clients: `web/src/api/client.ts` types + Settings category form +
|
||||
Spending.tsx progress indicator; `fin_cli` category/spending commands.
|
||||
E2e test asserting budget math for a seeded month.
|
||||
|
||||
**You have a result when** `fin spending month --json` shows
|
||||
`budget`/`remaining` per category matching a hand-computed fixture in an e2e
|
||||
test, and the Spending page renders over-budget categories distinctly.
|
||||
|
||||
### C2. Recurring-transaction detection — `open` (read-only analysis FIRST, no schema)
|
||||
|
||||
**Why current state falls short.** Nothing in the repo knows that rent,
|
||||
salary, or subscriptions repeat — every forecast or "upcoming bills" feature
|
||||
is blocked on this, and there is no detection code anywhere in `api/app` or
|
||||
`cli/fin_cli` (verified).
|
||||
|
||||
**Tractable because.** The full transaction history lives in one SQLite file
|
||||
with `date`, `amount`, `description`, `category_id` — recurrence detection is
|
||||
a grouping problem over data already in hand, and it needs **zero** schema or
|
||||
API changes to evaluate.
|
||||
|
||||
**First three steps.**
|
||||
1. Write a read-only script (home: `fin-diagnostics-and-tooling` scripts
|
||||
collection, or the scratchpad) that pulls transactions via
|
||||
`FinClient.list_transactions` or reads a **backup copy** of the DB, groups
|
||||
by (normalized description, amount within ±10%), and flags groups with
|
||||
near-constant intervals (~7/14/30/365 days ± tolerance).
|
||||
2. Run it on real (backup) data; manually judge the top-20 candidates for
|
||||
precision. Tune tolerances. This step decides whether the idea graduates.
|
||||
3. Only if precision is acceptable: promote to
|
||||
`api/app/services/recurring.py` + `GET /api/spending/recurring` (router
|
||||
with `_auth`) returning candidates with period and next expected date. No
|
||||
stored state yet — recompute per request is fine at this scale.
|
||||
|
||||
**You have a result when** the script's top-10 candidates on real data are
|
||||
≥80% genuinely recurring by the owner's judgment (falsifiable: the owner
|
||||
labels them), with period and next-expected-date printed per candidate.
|
||||
|
||||
### C3. Cash-flow forecasting — `candidate` (baseline startable now; quality depends on C2)
|
||||
|
||||
**Why current state falls short.** The app answers "what happened" only.
|
||||
`get_spending_summary` (`api/app/crud.py:751`) produces a per-month series of
|
||||
`total` / `income_total` / `net_total`, but nothing projects forward, so
|
||||
"can I afford X in October" is unanswerable in-app.
|
||||
|
||||
**Tractable because.** Three assets: the monthly net series from
|
||||
`/api/spending/summary` (baseline signal); pending RSU grants in `rsu_grants`
|
||||
with **future `vest_date` and `gross_shares`** — deterministic, dated future
|
||||
income already in the DB (verified `models.py:115-126`); and
|
||||
`net_worth_snapshots` for backtesting any model against what actually
|
||||
happened.
|
||||
|
||||
**First three steps.**
|
||||
1. Baseline script (read-only, via `FinClient.get_spending_summary`):
|
||||
trailing-6-month median net per calendar month → project bank balances 3
|
||||
months out. No repo changes.
|
||||
2. Layer in known events: `list_rsu_grants` pending vests valued at cached
|
||||
ticker price, and C2's recurring items when available.
|
||||
3. Backtest: hindcast the last 3 months using only prior data; compare
|
||||
against actual monthly nets and the naive "last month repeats" baseline.
|
||||
Only promote to an API endpoint (`services/forecast.py`) if it beats
|
||||
naive.
|
||||
|
||||
**You have a result when** the backtest prints a mean absolute error over ≥3
|
||||
hindcast months that beats the naive baseline. If it cannot beat naive, the
|
||||
honest result is "publish the negative result here and stop" — do not ship a
|
||||
forecast worse than a constant.
|
||||
|
||||
### C4. Portfolio returns — money-weighted `open`; time-weighted `blocked-on-valuation-history`
|
||||
|
||||
**Why current state falls short.** The Portfolio page shows current value and
|
||||
P&L vs. cost basis only; no return *rate* exists anywhere. And be honest
|
||||
about the data: verified in `api/app/services/net_worth.py:104-175`, the
|
||||
snapshot `breakdown_json` stores **per-account** `balance_native`/
|
||||
`balance_eur` only — no per-holding values, no prices. `price_cache` keeps
|
||||
one row per symbol with no history (`fin-architecture-contract` §3). So
|
||||
per-holding valuation history does not exist and cannot be reconstructed.
|
||||
|
||||
**Tractable because (MWR).** `stock_transactions` is a complete dated
|
||||
cashflow log (buy/sell/rsu_vest with `total_cost`, `fee`, `currency`,
|
||||
verified `models.py:96-110`), and current value comes from holdings ×
|
||||
`price_cache`. That is exactly the input XIRR needs. Funded-buy bank legs
|
||||
(`crud.create_funded_buy`) even mark external cash flows into investment
|
||||
accounts.
|
||||
|
||||
**First three steps.**
|
||||
1. Read-only script: per investment account, build the cashflow vector from
|
||||
`stock_transactions` (buys negative, sells positive, rsu_vest as inflow at
|
||||
vest-day value) + terminal value from current holdings × cached prices;
|
||||
solve XIRR by bisection (no new dependency needed).
|
||||
2. Validate on a 3-trade hand-computed fixture to 4 decimal places.
|
||||
3. Promote to `GET /api/portfolio/returns` in a new
|
||||
`api/app/services/portfolio.py` + router with `_auth` + `client.ts` +
|
||||
Portfolio page display; e2e test with the fixture.
|
||||
|
||||
**You have a result when** the e2e fixture's MWR matches the hand-computed
|
||||
XIRR to 4 decimals, and the endpoint returns per-account MWR on real data
|
||||
without error.
|
||||
|
||||
**TWR unblock path (do NOT backfill):** extend the snapshot breakdown for
|
||||
investment/rsu accounts to include per-holding `{ticker, shares, price,
|
||||
value_eur}` — an additive JSON change in `calculate_net_worth`
|
||||
(`net_worth.py:126-153`), no migration, old snapshots simply stay
|
||||
account-level. TWR becomes computable only for dates after that ships;
|
||||
account-level TWR from existing daily snapshots + flow dates is a coarse
|
||||
interim approximation. Anyone claiming full-history TWR is wrong — the price
|
||||
history is gone.
|
||||
|
||||
---
|
||||
|
||||
## Theme D — Practical backlog & polish
|
||||
|
||||
Ranked by value÷risk. Verified backlog state first: the plan
|
||||
`docs/superpowers/plans/2026-07-01-investment-income-features-plan.md` is
|
||||
**fully implemented** — checked 2026-07-05 against the code: total-first buy
|
||||
and currency selector are in `StockTradeFlow.tsx`, cross-currency transfers
|
||||
in `crud.create_transfer` (`to_amount`/`to_currency`, lines 237-258),
|
||||
`IncomeFlow.tsx` exists, income totals are in `get_monthly_spending`/
|
||||
`get_spending_summary`, and both of the plan's open questions were resolved
|
||||
(opening balances excluded via the `'Opening balance'` description filter,
|
||||
crud.py:737; Spending.tsx guards relaxed to `total > 0 || income_total > 0`).
|
||||
The only remainder is D5's `exchange_rate_used` note. No TODO/FIXME markers
|
||||
exist anywhere in `api/app`, `web/src`, or `cli/fin_cli` (verified by grep).
|
||||
|
||||
### D1. Automated DB backups — `open` (highest value in this theme)
|
||||
|
||||
**Why current state falls short.** `fin-architecture-contract` §5 #6: the
|
||||
prod SQLite volume is sacred and unprotected — no backup automation exists
|
||||
anywhere in the repo, and snapshots are frozen JSON, so loss is
|
||||
unrecoverable. `fin-run-and-operate` §3 has a proven **manual** runbook
|
||||
(Python stdlib `sqlite3.backup()` inside the container — WAL-safe, and the
|
||||
image has no `sqlite3` binary, so the Python API is the only option); this
|
||||
item automates it.
|
||||
|
||||
**Tractable because.** The hard part (a WAL-safe consistent snapshot command)
|
||||
is already written and tested in the runbook; automation is scheduling +
|
||||
rotation + getting the file **off the volume**.
|
||||
|
||||
**First three steps.**
|
||||
1. Commit the runbook as a script: `deploy/backup.sh` wrapping the
|
||||
`docker compose exec api python -c "...src.backup(dst)..."` one-liner,
|
||||
then `docker cp`/volume-mount copy to a host directory, then rotation
|
||||
(keep last N, dated filenames). Repo-side only — runs on the homelab host.
|
||||
2. Schedule it host-side (cron or systemd timer — document the unit in
|
||||
`fin-run-and-operate`, since the repo cannot install host timers). An
|
||||
in-app APScheduler job is NOT sufficient alone: it can only write to the
|
||||
same volume it is trying to protect.
|
||||
3. Restore drill, once: restore a backup into a scratch container, run
|
||||
`alembic current` + `GET /api/health` + `fin networth --json` against it.
|
||||
Document the drill result in `fin-run-and-operate`.
|
||||
|
||||
**You have a result when** ≥2 dated backup files exist outside the
|
||||
`sqlite_data` volume, produced without manual action on consecutive days, and
|
||||
one restore drill has produced a working API against the restored copy.
|
||||
|
||||
### D2. Funded-buy pair delete consistency — `open` (verified gap, small fix)
|
||||
|
||||
**Why current state falls short.** Found while verifying this skill
|
||||
(2026-07-05): `create_funded_buy` (`crud.py:549-617`) writes a
|
||||
StockTransaction and a bank `transfer` leg with **no reference between
|
||||
them** — the bank leg gets `linked_account_id` but no
|
||||
`linked_transaction_id`, and the stock txn has no back-pointer. Consequence,
|
||||
confirmed by reading both delete paths: `delete_stock_transaction`
|
||||
(crud.py:522) deletes the stock txn and recomputes the holding but leaves the
|
||||
bank leg (cash stays deducted, position gone); `delete_transaction` on the
|
||||
bank leg falls into the legacy heuristic (crud.py:205-225), finds no partner
|
||||
on the investment account, and deletes only the bank leg (cash reappears,
|
||||
shares remain). Either order leaves the ledger inconsistent.
|
||||
|
||||
**Tractable because.** The pairing already exists at creation time inside one
|
||||
function — persisting it is one nullable FK.
|
||||
|
||||
**First three steps.**
|
||||
1. Alembic migration (chained after `h8c9d0e1f2a3`, via `fin-change-control`)
|
||||
adding nullable `funding_transaction_id` FK to `stock_transactions` (or
|
||||
the mirror-image column on `transactions` — pick one direction, document
|
||||
it in `fin-accounting-reference`).
|
||||
2. Set it in `create_funded_buy`; in `delete_stock_transaction`, delete the
|
||||
linked bank leg in the same commit; in `delete_transaction`, detect a bank
|
||||
leg referenced by a stock txn and either cascade (delete both +
|
||||
holding recompute) or 422 with "delete the stock transaction instead" —
|
||||
the 422 is simpler and safer; start there.
|
||||
3. E2e tests for both delete orders; extend the A2 audit with a
|
||||
"funded-buy halves both present" check for legacy rows (old pairs stay
|
||||
unlinked — the audit's date+amount matching flags survivors).
|
||||
|
||||
**You have a result when** deleting a funded buy through either entry point
|
||||
leaves both the bank balance and the holding at their pre-buy values (e2e
|
||||
asserted), and the audit reports 0 half-deleted pairs.
|
||||
|
||||
### D3. CSV export — `open`; import — `unblocked 2026-08-22, still gated`
|
||||
|
||||
> **Status update 2026-08-22.** B3's idempotency keys landed, so the
|
||||
> re-import double-booking objection no longer holds. Import is still gated
|
||||
> on A1's orphan-leg audit and on deciding how imported rows map to linked
|
||||
> pairs — do not treat this as simply open.
|
||||
|
||||
**Why current state falls short.** No CSV/export code exists in any router or
|
||||
CLI module (verified by grep, 2026-07-05). The data is only extractable as
|
||||
JSON via API, which is awkward for spreadsheets and for the owner's
|
||||
own-your-data expectations of a self-hosted app.
|
||||
|
||||
**Tractable because.** Export needs zero API changes: `FinClient.
|
||||
list_transactions` already returns everything, so a CLI-side `fin tx export`
|
||||
is pure presentation (stdlib `csv`, allowed in `fin_cli` command modules —
|
||||
only `client.py` must stay presentation-free).
|
||||
|
||||
**First three steps.**
|
||||
1. `fin tx export --output file.csv [--account ID] [--month YYYY-MM]` in
|
||||
`cli/fin_cli/tx.py`, columns = the Transaction schema fields incl. `type`,
|
||||
`fee`, `linked_account_id`, `splitwise_partner_share` (so the export is
|
||||
semantically complete, not just pretty).
|
||||
2. E2e test: seeded transactions → export → parse → per-account
|
||||
`SUM(amount − fee)` equals the API's reported balances.
|
||||
3. Same pattern for `stock_transactions`.
|
||||
|
||||
**You have a result when** the exported CSV re-summed per account matches
|
||||
API balances exactly in an e2e test. **Import stays blocked**: naive
|
||||
row-inserts through `POST /api/transactions` would create exactly the
|
||||
unlinked legs A1 forbids, and re-imports double-book without B3's
|
||||
idempotency keys. Do not build import before those land.
|
||||
|
||||
### D4. Fix synchronous `fetch-ticker` blocking — `open`
|
||||
|
||||
**Why current state falls short.** `fin-architecture-contract` §5 #5,
|
||||
re-verified: `POST /api/prices/fetch-ticker` calls `fetch_ticker_price`
|
||||
synchronously in the request handler, which retries 1s/2s/4s across up to 9
|
||||
symbol candidates — worst case blocks a threadpool worker for minutes, and
|
||||
CLAUDE.md explicitly forbids blocking price fetches on the hot path.
|
||||
`POST /api/prices/refresh` already shows the correct pattern
|
||||
(BackgroundTasks + 202).
|
||||
|
||||
**First three steps.** (1) Mirror the `/refresh` BackgroundTasks pattern in
|
||||
`api/app/routers/prices.py` for fetch-ticker, returning 202; (2) adjust the
|
||||
frontend caller in `client.ts`/Portfolio and `cli/fin_cli/prices.py` to poll
|
||||
the price via the cache after a beat instead of expecting the price in the
|
||||
response — this is a small behavior change, gate via `fin-change-control`;
|
||||
(3) e2e test asserting 202 and eventual cache population (fake transport).
|
||||
|
||||
**You have a result when** fetch-ticker returns in <100 ms regardless of
|
||||
yfinance behavior (assertable with a transport that sleeps) and the price
|
||||
still lands in `price_cache`.
|
||||
|
||||
### D5. Small polish items — `open`, batchable
|
||||
|
||||
Each is grounded and tiny; batch them into an idle session.
|
||||
|
||||
- **`exchange_rate_used` on cross-currency transfers**: the only remainder of
|
||||
the 2026-07-01 plan — `create_transfer` leaves it null (verified
|
||||
crud.py:239-258). Prefill from `price_cache` at write time so historical
|
||||
legs are auditable. Result: new cross-currency legs carry a rate; e2e
|
||||
asserts it.
|
||||
- **CLAUDE.md truth sync**: it still lists five account types (schemas.py has
|
||||
six — `payable`) and says holdings are "incrementally updated on insert"
|
||||
(code does a full recompute on insert too, crud.py:428-441/582). Result:
|
||||
CLAUDE.md matches code; `fin-architecture-contract` already carries the
|
||||
corrections.
|
||||
- **Router-layer mutation cleanup** (`fin-architecture-contract` §5 #2): move
|
||||
the `/reconcile` body (`routers/transactions.py:100-154`) and
|
||||
`adjust-balance` into crud.py functions — pure relocation, no behavior
|
||||
change, makes crud.py the complete mutation audit surface. Result:
|
||||
`grep -n "db.add" api/app/routers/*.py` returns nothing and the existing
|
||||
e2e tests still pass. Status: `candidate` — zero user value, do it when
|
||||
already touching those files.
|
||||
|
||||
---
|
||||
|
||||
## Anti-backlog — deliberately not worth doing (as of 2026-07-05)
|
||||
|
||||
Re-litigate only with new facts; each reason is structural, not taste.
|
||||
|
||||
| Rejected project | Why not |
|
||||
|---|---|
|
||||
| **Multi-user / per-user auth** | Single-user assumptions are baked into net worth calculation and account filtering (CLAUDE.md "What NOT to do" says exactly this). Every query in crud.py is implicitly "the one user". B2's single static token is the ceiling. |
|
||||
| **Async/AsyncSession rewrite** | Explicitly forbidden by CLAUDE.md; PLAN.md records the rationale (aiosqlite wraps a thread anyway; FastAPI threadpools sync handlers). Zero throughput need at single-user scale; guaranteed breakage of crud.py and the scheduler engine pattern. |
|
||||
| **Real-time / streaming / intraday prices, or a price-history table** | yfinance has no SLA (`fin-architecture-contract` §5 #5); the 5-minute refresh job already exceeds any single-user need; and a history table contradicts the load-bearing "one row per symbol, history lives in snapshots" design (§3). Valuation-history needs are served by the C4 breakdown extension instead. |
|
||||
| **Decimal/money-type migration** | Money is Float everywhere (§5 #7) and drift is cosmetic at this scale; the migration touches every column, every schema, every client, and every test for no user-visible gain. Revisit only if A2's audit measures drift beyond cents. |
|
||||
| **Postgres migration** | SQLite+WAL fits the two-writer (requests + scheduler) single-user reality; would invalidate the pragma listeners, the backup runbook, the preflight check, and the volume model, and buy nothing. |
|
||||
| **True journal/postings double-entry rewrite** | The linked-leg model already delivers the guarantees once A1+A2+D2 land, at a fraction of the risk. A rewrite re-opens every settled invariant in `fin-failure-archaeology` for zero new capability. |
|
||||
| **Bank auto-sync (PSD2/Plaid-style)** | Heavy external dependency + credential custody on a self-hosted box; providers churn. CSV export (D3) plus, later, a gated import covers the realistic need. |
|
||||
|
||||
---
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All repo claims verified 2026-07-05 against commit 082b608 by reading:
|
||||
`api/app/crud.py` (create/update/delete transaction, transfers, settlements,
|
||||
funded buy, delete_stock_transaction, spending/summary), `api/app/schemas.py`
|
||||
(type Literals, TransactionCreate fields), `api/app/models.py` (all tables),
|
||||
`api/app/services/{scheduler,net_worth}.py`, `api/app/routers/
|
||||
transactions.py`, `cli/fin_cli/client.py`, `cli/pyproject.toml`,
|
||||
`web/src/components/QuickAdd/flows/StockTradeFlow.tsx`, `web/src/pages/
|
||||
Spending.tsx`, `PLAN.md`, `docs/superpowers/plans/2026-07-01-investment-
|
||||
income-features-plan.md`, and sibling skills (`fin-architecture-contract`
|
||||
§5 weak points, `fin-run-and-operate` §3 backup runbook). Sibling skills
|
||||
`fin-test-coverage-campaign`, `fin-ledger-analysis-toolkit`, and the
|
||||
`fin-diagnostics-and-tooling` SKILL.md were being authored in parallel on the
|
||||
same date — their referenced content may still be landing.
|
||||
|
||||
Volatile facts to re-verify before acting on an item:
|
||||
|
||||
- A1 gap still open: `grep -n "transfer" api/app/routers/transactions.py`
|
||||
(look for a rejection in `create_transaction`) and schemas.py:77 Literal.
|
||||
- D2 gap still open: `grep -n "linked" api/app/crud.py | grep -i fund` and
|
||||
whether `stock_transactions` gained a funding FK (`models.py`).
|
||||
- Migration head still `h8c9d0e1f2a3`: `ls api/alembic/versions/`.
|
||||
- Frontend still tokenless: `grep -n "Authorization" web/src/api/client.ts`
|
||||
(expect no hits) — else B2 is done, unblock B1 writes.
|
||||
- MCP still absent: `ls cli/` (expect no `fin_mcp/`).
|
||||
- Budget still absent: `grep -ri budget api/app web/src cli/fin_cli`.
|
||||
- Test count baseline: `grep -c "def test" cli/tests/test_cli.py` (25 as of
|
||||
this writing).
|
||||
|
||||
Maintenance rule: when an item ships, do not delete it — flip its status to
|
||||
`done (YYYY-MM-DD, commit)` in one line and move the details to the relevant
|
||||
sibling skill (`fin-architecture-contract` for new invariants,
|
||||
`fin-run-and-operate` for new operational surfaces). When an item is
|
||||
attempted and abandoned, record why here — a documented dead end is this
|
||||
skill's second-most-valuable content after the ranking itself.
|
||||
@@ -0,0 +1,407 @@
|
||||
---
|
||||
name: fin-run-and-operate
|
||||
description: >
|
||||
Runbook for deploying, restarting, backing up, and operating the fin stack on
|
||||
the homelab. Load this skill when: deploying or rebuilding the Docker images,
|
||||
pushing to Harbor, restarting containers, configuring or auditing any env var
|
||||
(DATABASE_URL, FIN_API_TOKEN, FIN_API_URL, API_UPSTREAM), backing up or
|
||||
restoring the production SQLite DB, investigating the APScheduler jobs or
|
||||
stale prices in production, reading api logs, checking /api/health, or doing
|
||||
anything that touches the homelab deployment. Contains the complete
|
||||
configuration catalog for the project.
|
||||
---
|
||||
|
||||
# fin — Run and Operate
|
||||
|
||||
Operational runbook for the fin stack (FastAPI `api` + nginx `web`, Docker
|
||||
Compose, private Harbor registry). All commands below were **verified
|
||||
statically** against `docker-compose.yml`, `api/Dockerfile`, `web/Dockerfile`,
|
||||
and the source tree as of 2026-07-05 — this sandbox has no Docker, so nothing
|
||||
here was executed live. Homelab-specific facts the repo cannot show are marked
|
||||
**operator-supplied convention — confirm on the homelab**.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
| You actually want to… | Use instead |
|
||||
|---|---|
|
||||
| Drive the `fin` CLI to query/record finances | `fin` |
|
||||
| Change code, write migrations, review invariants | `fin-change-control`, `fin-architecture-contract` |
|
||||
| Debug wrong numbers / app bugs (incl. stale-price root-causing) | `fin-debugging-playbook` |
|
||||
| Understand past incidents | `fin-failure-archaeology` |
|
||||
| Set up a local build/dev environment | `fin-build-and-env` |
|
||||
| Inspect data ad hoc, run queries | `fin-diagnostics-and-tooling`, `fin-ledger-analysis-toolkit` |
|
||||
| Test/QA a change before deploy | `fin-validation-and-qa`, `fin-test-coverage-campaign` |
|
||||
| Accounting semantics (balances, spending math) | `fin-accounting-reference` |
|
||||
|
||||
## The one iron rule
|
||||
|
||||
**The production SQLite DB (`/data/fin.db` in volume `sqlite_data`) holds the
|
||||
owner's real financial history and is irreplaceable.** Before anything risky on
|
||||
the homelab — deploy with new migrations, alembic downgrade, manual DB surgery —
|
||||
take a backup (section below). Never point tests, dev servers, or the CLI's
|
||||
write commands at production casually. Local/dev work uses a different
|
||||
`DATABASE_URL`; see `fin-build-and-env`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Deploy anatomy
|
||||
|
||||
### docker-compose.yml walkthrough (repo-verified)
|
||||
|
||||
| Aspect | `api` service | `web` service |
|
||||
|---|---|---|
|
||||
| Image | `harbor.cloud.traunseenet.com/library/fin-api:latest` | `harbor.cloud.traunseenet.com/library/fin-web:latest` |
|
||||
| Build context | `./api` | `./web` |
|
||||
| Ports | none published (internal :8000 only) | `80:80` (host 80 → nginx 80) |
|
||||
| Volumes | `sqlite_data:/data` (named volume, the prod DB) | none |
|
||||
| Environment | `DATABASE_URL=sqlite:////data/fin.db` | `API_UPSTREAM=api:8000` |
|
||||
| Restart | `unless-stopped` | `unless-stopped` |
|
||||
| Healthcheck | see verbatim below | none |
|
||||
| depends_on | — | `api: condition: service_healthy` — web will not start until api's healthcheck passes |
|
||||
|
||||
Healthcheck, verbatim from `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
Note: the healthcheck hits `/api/health`, which is **always unauthenticated**
|
||||
(registered in `main.py` outside the token dependency), so enabling
|
||||
`FIN_API_TOKEN` never breaks the healthcheck or `depends_on`.
|
||||
|
||||
### What happens when the api container starts (repo-verified)
|
||||
|
||||
`api/Dockerfile` CMD, verbatim:
|
||||
|
||||
```dockerfile
|
||||
CMD sh -c "python -m app.preflight && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000"
|
||||
```
|
||||
|
||||
Sequence:
|
||||
|
||||
1. **Preflight** (`app/preflight.py`): resolves `DATABASE_URL`, creates the
|
||||
SQLite parent directory, and verifies it is writable (writes a temp file).
|
||||
Fails fast with "SQLite database directory is not writable" on bad volume
|
||||
permissions.
|
||||
2. **`alembic upgrade head`**: migrations apply automatically on every start.
|
||||
`alembic/env.py` reads `DATABASE_URL` first, so migrations always target the
|
||||
same DB the app uses. If a migration fails, the container never reaches
|
||||
uvicorn → healthcheck fails → web never (re)starts.
|
||||
3. **uvicorn** starts on `0.0.0.0:8000`, single process, no `--workers` flag
|
||||
→ exactly one worker. **Do not add `--workers N>1`**: APScheduler starts in
|
||||
the FastAPI lifespan, so multiple workers would run duplicate schedulers.
|
||||
4. **Lifespan** (`app/main.py`): `Base.metadata.create_all(bind=engine)`
|
||||
(first-boot convenience only, no-op afterwards — Alembic is the real
|
||||
migration mechanism), then seeds 10 default categories iff the `categories`
|
||||
table is empty, then builds and starts the APScheduler
|
||||
`BackgroundScheduler` with a `SQLAlchemyJobStore` persisted in the same DB
|
||||
(table `apscheduler_jobs`).
|
||||
|
||||
The container runs as non-root user `app` (uid 1000), created in the
|
||||
Dockerfile with `/data` chowned to it. Do not revert to root.
|
||||
|
||||
The web image is a two-stage build (node:20-alpine `npm ci && npm run build`
|
||||
→ nginx:alpine). `nginx.conf` is copied to
|
||||
`/etc/nginx/templates/default.conf.template`; the stock nginx:alpine
|
||||
entrypoint runs envsubst on templates, which is how `${API_UPSTREAM}` in
|
||||
`proxy_pass http://${API_UPSTREAM};` gets substituted. nginx serves the SPA,
|
||||
proxies `/api/*` to the api container, never caches `index.html`.
|
||||
|
||||
### Standard deploy runbook
|
||||
|
||||
On the dev machine (repo checkout):
|
||||
|
||||
```bash
|
||||
# 1. Build both images (repo-verified: compose has build: for both services)
|
||||
docker compose build
|
||||
|
||||
# 2. Push to Harbor (repo-verified: image: names point at the registry, so
|
||||
# `docker compose push` targets harbor.cloud.traunseenet.com/library/fin-{api,web}:latest.
|
||||
# Requires prior `docker login harbor.cloud.traunseenet.com` —
|
||||
# operator-supplied convention: credentials live on the operator's machines.)
|
||||
docker compose push
|
||||
```
|
||||
|
||||
On the homelab (**operator-supplied convention — confirm on the homelab**:
|
||||
where the compose file lives there and how it is kept in sync with the repo is
|
||||
not visible in this repo):
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
|
||||
Post-deploy verification (always do this):
|
||||
|
||||
```bash
|
||||
docker compose ps # api should be "healthy", web "running"
|
||||
docker compose logs --tail=50 api # look for "Database preflight passed",
|
||||
# alembic output, "APScheduler started"
|
||||
curl -s http://localhost/api/health # via nginx → {"status":"ok"}
|
||||
```
|
||||
|
||||
**Pre-deploy checklist when the deploy includes new Alembic migrations:**
|
||||
|
||||
- [ ] Backup the prod DB first (section 3).
|
||||
- [ ] Migration is backward-safe (old code can still run against the new
|
||||
schema), because the old container may restart against the migrated DB
|
||||
during a botched rollout — and rollback means old image + new schema.
|
||||
- [ ] Never plan to `alembic downgrade` on prod as a rollback strategy without
|
||||
a backup in hand.
|
||||
|
||||
---
|
||||
|
||||
## 2. Configuration catalog
|
||||
|
||||
Every configuration axis in the project (as of 2026-07-05). Exhaustive: a grep
|
||||
for `os.getenv`/`os.environ` across `api/` and `cli/` (excluding tests) finds
|
||||
only `DATABASE_URL`, `FIN_API_TOKEN`, and `FIN_API_URL`; everything else is a
|
||||
code constant or Docker-level setting.
|
||||
|
||||
| Axis | Where read | Default | Prod value / required? | Effect | Guard / caveat |
|
||||
|---|---|---|---|---|---|
|
||||
| `DATABASE_URL` | `api/app/db_config.py:12` (app+scheduler), `api/alembic/env.py` (~42, ~65) | `sqlite:///./fin.db` (code); Dockerfile bakes `sqlite:////data/fin.db` | Compose sets `sqlite:////data/fin.db` | DB location for app, Alembic, and the scheduler's private engine — all three read the same var | THE prod-safety axis. Tests/dev must use a different value. Note four slashes = absolute path |
|
||||
| `FIN_API_TOKEN` (API side) | `api/app/auth.py:15`, read **per request** | unset → auth disabled (no-op) | Optional | When set, every router requires `Authorization: Bearer <token>`; 401 otherwise. `/api/health` always open; `/api/openapi.json` is token-guarded | **Setting it locks out the React frontend** — `web/src/api/client.ts` sends no Authorization header — until nginx injects the header or the frontend learns the token. Per-request read means rotation needs no restart, only env change + container recreate |
|
||||
| `FIN_API_URL` (CLI side) | `cli/fin_cli/client.py:41` | none — **required**, deliberate | Required wherever `fin` runs | Base URL the CLI targets | No default so a misconfigured shell can never silently hit the wrong host |
|
||||
| `FIN_API_TOKEN` (CLI side) | `cli/fin_cli/client.py:47` | empty → no header sent | Only if API enforces auth | Adds `Authorization: Bearer …` to all CLI requests | Must match the API-side token exactly |
|
||||
| `API_UPSTREAM` | `web/nginx.conf:14` via nginx:alpine envsubst templating; ENV default in `web/Dockerfile:14` | `api:8000` | Compose sets `api:8000` | Where nginx proxies `/api/*` | Only change if the api service name/port changes |
|
||||
| Healthcheck params | `docker-compose.yml:10-14` | interval 30s, timeout 10s, retries 3 | as-is | Container "healthy" gate; web start depends on it | Hits `/api/health` from inside the container with python urllib — no curl in the image |
|
||||
| Price refresh interval | `api/app/services/scheduler.py:98-105` | `interval, minutes=5`, job id `price_refresh` | code constant | Fetches all prices every 5 min, no snapshot | Change in code only; jobs use `replace_existing=True` so redeploys update the persisted jobstore |
|
||||
| Daily snapshot time | `api/app/services/scheduler.py:108-116` | `cron, hour=0, minute=0`, job id `daily_snapshot` | code constant | Midnight: fetch prices + upsert one `net_worth_snapshots` row per calendar day | Container timezone is UTC (python:3.12-slim default) unless TZ is set — "midnight" means UTC midnight |
|
||||
| APScheduler jobstore | `scheduler.py:91-93` | `SQLAlchemyJobStore`, table `apscheduler_jobs` in the main DB | as-is | Jobs persist across restarts | Backups of fin.db include the jobstore — that is fine and expected |
|
||||
| Price fetch retries | `api/app/services/price_fetcher.py:153-201` | 3 attempts/candidate, backoff `2**attempt` = 1s/2s/4s; suffix fallbacks `.DE .L .AS .PA .MI .SW .F .VI`; falls back to cached price, then 0.0 | code constant | yfinance fetch resilience | This blocks+sleeps — never call `fetch_all_prices` synchronously in a request handler (POST /refresh correctly uses BackgroundTasks) |
|
||||
| FX fetch timeout | `price_fetcher.py:52-58, 80-86` | httpx `timeout=10` against `api.frankfurter.dev` (EUR/USD, EUR/GBP) | code constant | FX rates for net worth | On failure returns last cached rate |
|
||||
| SQLite pragmas | `api/app/database.py:18-24` and `scheduler.py:21-27` | `journal_mode=WAL`, `foreign_keys=ON` on **every** new connection, both engines | invariant | Concurrency + FK integrity | Never remove; the scheduler thread has its own engine on purpose |
|
||||
| uvicorn bind/workers | `api/Dockerfile:17` CMD | `--host 0.0.0.0 --port 8000`, no `--workers` → 1 | as-is | Single process runs both API and scheduler | Do not add workers (duplicate schedulers) |
|
||||
| nginx / host port | `web/Dockerfile` EXPOSE 80; compose `"80:80"` | 80 | as-is | Public entry point | — |
|
||||
| Container user | `api/Dockerfile:9-10` | user `app`, uid 1000, owns `/data` | invariant | Non-root runtime | Do not revert to root; volume files are uid 1000 |
|
||||
| CLI HTTP timeout | `cli/fin_cli/client.py:52` | 30.0s | code constant | CLI request timeout | — |
|
||||
| FastAPI docs | `api/app/main.py:51-57` | `docs_url/redoc_url/openapi_url=None` | as-is | No public Swagger; spec only at token-guarded `/api/openapi.json` | Deliberate — do not re-enable public docs |
|
||||
|
||||
### How to add a new configuration axis (checklist)
|
||||
|
||||
- [ ] Read it via `os.getenv` in exactly one place — `db_config.py` for
|
||||
DB-related, the owning service module otherwise. Give it a safe default
|
||||
so existing deployments keep working (follow the `FIN_API_TOKEN` opt-in
|
||||
pattern).
|
||||
- [ ] Thread it through `docker-compose.yml` `environment:` (and Dockerfile
|
||||
`ENV` if a baked default makes sense, mirroring `DATABASE_URL`).
|
||||
- [ ] If the scheduler needs it, pass it as a job arg like `database_url` —
|
||||
the scheduler runs in a background thread with its own engine.
|
||||
- [ ] Add a row to the env-var table in `README.md` (existing table around
|
||||
line 32).
|
||||
- [ ] Add a row to the catalog table in **this file** with file:line and
|
||||
caveats.
|
||||
- [ ] If it changes CLI behavior, update `.claude/skills/fin/SKILL.md`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Production DB protection (backup & restore)
|
||||
|
||||
Facts (repo-verified): the DB is SQLite in WAL mode at `/data/fin.db` inside
|
||||
the `api` container, on named volume `sqlite_data`. The image is
|
||||
`python:3.12-slim` with only pip packages installed — **the `sqlite3`
|
||||
command-line binary is NOT in the image** (no apt-get in `api/Dockerfile`), but
|
||||
Python's stdlib `sqlite3` module is. So backups use the Python backup API,
|
||||
which is WAL-safe (it takes a consistent snapshot without stopping writers).
|
||||
|
||||
### Backup runbook (online, WAL-safe — preferred)
|
||||
|
||||
Run on the homelab host:
|
||||
|
||||
```bash
|
||||
# 1. Create a consistent snapshot inside the container (stdlib backup API)
|
||||
docker compose exec api python -c "
|
||||
import sqlite3
|
||||
src = sqlite3.connect('/data/fin.db')
|
||||
dst = sqlite3.connect('/data/fin-backup.db')
|
||||
with dst:
|
||||
src.backup(dst)
|
||||
dst.close(); src.close()
|
||||
print('backup written')
|
||||
"
|
||||
|
||||
# 2. Copy it out and clean up
|
||||
docker compose cp api:/data/fin-backup.db ./fin-backup-$(date +%F).db
|
||||
docker compose exec api rm /data/fin-backup.db
|
||||
|
||||
# 3. Verify the copy is a sane database
|
||||
python3 -c "
|
||||
import sqlite3
|
||||
db = sqlite3.connect('fin-backup-$(date +%F).db')
|
||||
print(db.execute('PRAGMA integrity_check').fetchone())
|
||||
print('transactions:', db.execute('SELECT COUNT(*) FROM transactions').fetchone()[0])
|
||||
"
|
||||
```
|
||||
|
||||
Move the verified file off the homelab (**operator-supplied convention —
|
||||
confirm where offsite backups live**).
|
||||
|
||||
### Backup runbook (offline, raw copy — when the stack is stopped anyway)
|
||||
|
||||
A raw copy of a live WAL database is only consistent if you copy `fin.db` +
|
||||
`fin.db-wal` + `fin.db-shm` together **and** nothing is writing. Simplest safe
|
||||
form — stop the api first:
|
||||
|
||||
```bash
|
||||
docker compose stop api
|
||||
docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/backup alpine \
|
||||
tar czf /backup/fin-volume-$(date +%F).tar.gz -C /data .
|
||||
docker compose start api
|
||||
```
|
||||
|
||||
`<project>_sqlite_data` is the real volume name — compose prefixes the project
|
||||
name (directory name by default; check with `docker volume ls`).
|
||||
**Operator-supplied convention — confirm the actual volume name on the
|
||||
homelab.**
|
||||
|
||||
### Restore runbook
|
||||
|
||||
```bash
|
||||
docker compose stop api # 1. stop all writers
|
||||
|
||||
docker compose cp ./fin-backup-YYYY-MM-DD.db api:/data/fin.db # 2. replace the DB
|
||||
# If the container won't run stopped-exec on your compose version, use the
|
||||
# volume-mount approach instead:
|
||||
# docker run --rm -v <project>_sqlite_data:/data -v "$PWD":/backup alpine \
|
||||
# sh -c "cp /backup/fin-backup-YYYY-MM-DD.db /data/fin.db && rm -f /data/fin.db-wal /data/fin.db-shm && chown 1000:1000 /data/fin.db"
|
||||
|
||||
# 3. Remove stale WAL sidecars — a leftover -wal from the OLD database would
|
||||
# corrupt the restored one on first open
|
||||
docker compose exec api rm -f /data/fin.db-wal /data/fin.db-shm # (before start, via the volume-mount form above if api is stopped)
|
||||
|
||||
docker compose start api # 4. start; alembic upgrade head re-runs
|
||||
docker compose logs --tail=30 api # 5. verify preflight + migrations + scheduler start
|
||||
curl -s http://localhost/api/health
|
||||
```
|
||||
|
||||
File ownership matters: the api runs as uid 1000, so restored files must be
|
||||
readable/writable by uid 1000 (the `chown 1000:1000` above).
|
||||
|
||||
### Hard rules
|
||||
|
||||
1. **Never point tests at prod.** pytest, dev servers, and experiments use
|
||||
their own `DATABASE_URL`. The prod URL is exactly `sqlite:////data/fin.db`
|
||||
inside the api container and nowhere else.
|
||||
2. **Never run `alembic downgrade` on prod without a same-day backup.**
|
||||
3. **Migrations must be backward-safe** (additive first; destructive changes
|
||||
only after the code that needed the old shape is gone).
|
||||
4. **Never edit `/data/fin.db` with raw DDL or by hand** — CLAUDE.md rule; all
|
||||
schema changes go through Alembic, all data changes through the API/CLI.
|
||||
|
||||
---
|
||||
|
||||
## 4. Routine operations
|
||||
|
||||
| Task | Command | Expected result |
|
||||
|---|---|---|
|
||||
| Health check | `curl -s http://localhost/api/health` (via nginx) or from the host running compose: `docker compose exec api python -c "import urllib.request;print(urllib.request.urlopen('http://localhost:8000/api/health').read())"` | `{"status":"ok"}` — always open, even with auth enabled |
|
||||
| Manual price refresh (API) | `curl -s -X POST http://localhost/api/prices/refresh` (+ `-H "Authorization: Bearer $FIN_API_TOKEN"` if auth on) | `202` `{"status":"accepted","message":"Price refresh queued"}` — runs as a FastAPI BackgroundTask, verified in `api/app/routers/prices.py:19-23` |
|
||||
| Manual price refresh (CLI) | `fin prices refresh` | same 202 via `FinClient.refresh_prices()` |
|
||||
| Read api logs | `docker compose logs --tail=100 api` / `docker compose logs -f api` | See patterns below |
|
||||
| Restart api only | `docker compose restart api` | Preflight → alembic → uvicorn → scheduler again; web keeps running |
|
||||
| Verify daily snapshots | `fin networth history --json` or `curl -s http://localhost/api/net-worth/history?range=1M` (auth header if enabled) | One row per calendar day; the newest `snapshot_date` should be today or yesterday (UTC midnight job) |
|
||||
| Check scheduler health | `docker compose logs api \| grep -E "Price refresh|Daily job|APScheduler"` | `Price refresh: updated N price(s)` roughly every 5 min; `Daily job: created/updated snapshot for YYYY-MM-DD` once per day; `APScheduler started` on boot |
|
||||
|
||||
Healthy-startup log fingerprint (in order):
|
||||
`Database preflight passed` → Alembic `Running upgrade …` lines (or nothing if
|
||||
already at head) → `APScheduler started (daily_snapshot job scheduled at
|
||||
midnight)` → uvicorn `Application startup complete`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Data and artifact conventions
|
||||
|
||||
Verified: the application writes **no files other than the SQLite database and
|
||||
its WAL sidecars**. Everything lands in `/data` on volume `sqlite_data`:
|
||||
|
||||
- `fin.db` — all tables: accounts, transactions, holdings, stock_transactions,
|
||||
rsu_grants, categories, `price_cache` (one upserted row per symbol, incl.
|
||||
`EUR/USD` and `EUR/GBP`), `net_worth_snapshots` (one row per day: total_eur
|
||||
+ JSON breakdown), and `apscheduler_jobs` (persisted scheduler state).
|
||||
- `fin.db-wal`, `fin.db-shm` — WAL sidecars, present while the app runs.
|
||||
|
||||
No log files on disk (logs go to stdout → `docker compose logs`), no export
|
||||
files, no uploads. Backing up `/data` backs up everything.
|
||||
|
||||
---
|
||||
|
||||
## 6. Failure drills
|
||||
|
||||
**api container unhealthy / restart-looping**
|
||||
`docker compose logs --tail=200 api`. The CMD chain fails in order:
|
||||
1. `SQLite database directory is not writable` → volume permission problem;
|
||||
`/data` must be owned by uid 1000.
|
||||
2. Alembic traceback → a migration failed; the app never started, so the DB is
|
||||
whatever state the failed migration left. Do NOT retry blindly — restore
|
||||
from the pre-deploy backup or fix-forward with a corrected migration
|
||||
(see `fin-change-control`).
|
||||
3. Uvicorn import error → bad image; roll back to the previous image tag
|
||||
(**operator convention: only `:latest` is pushed per the compose file — if
|
||||
no older tag exists in Harbor, rebuild from the last good commit**).
|
||||
|
||||
**web up but SPA can't reach API**
|
||||
`depends_on: service_healthy` only gates startup ordering, not runtime. Check
|
||||
api health directly, then check whether `FIN_API_TOKEN` was recently set on
|
||||
the api — that 401s the frontend by design (catalog caveat above).
|
||||
|
||||
**Prices stale**
|
||||
Confirm the 5-minute `Price refresh` log lines are appearing. If the job runs
|
||||
but prices don't move, it is a fetcher/upstream problem (yfinance,
|
||||
frankfurter) — hand off to `fin-debugging-playbook`. If the job lines are
|
||||
absent entirely, the scheduler died with the process: restart api and check
|
||||
startup logs. Quick mitigation either way: `fin prices refresh`.
|
||||
|
||||
**Missing daily snapshot rows**
|
||||
The midnight job upserts by `snapshot_date`; a gap means the container was
|
||||
down at UTC midnight or the job errored (`Daily job: failed to upsert
|
||||
snapshot` / `Daily job: unexpected error` in logs). A missed day cannot be
|
||||
recomputed retroactively (it snapshots *current* net worth) — accept the gap;
|
||||
`POST /api/prices/refresh` does not write snapshots.
|
||||
|
||||
**Disk growth / huge fin.db-wal**
|
||||
WAL normally auto-checkpoints (SQLite default ~1000 pages). A persistently
|
||||
large `-wal` means a long-lived reader is blocking checkpoints. Force one:
|
||||
|
||||
```bash
|
||||
docker compose exec api python -c "
|
||||
import sqlite3
|
||||
db = sqlite3.connect('/data/fin.db')
|
||||
print(db.execute('PRAGMA wal_checkpoint(TRUNCATE)').fetchone())
|
||||
db.close()
|
||||
"
|
||||
```
|
||||
|
||||
`(0, N, N)` means success. If it stays blocked, `docker compose restart api`
|
||||
clears lingering connections, then checkpoint again.
|
||||
|
||||
---
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
All facts verified statically against the repo at commit `082b608` (2026-07-05).
|
||||
Re-verify before trusting, with one command each:
|
||||
|
||||
- Compose topology, healthcheck, ports, volume: `cat docker-compose.yml`
|
||||
- api start sequence + uvicorn args + uid: `cat api/Dockerfile` (CMD on line 17)
|
||||
- Lifespan (create_all, category seed, scheduler start): `sed -n '16,49p' api/app/main.py`
|
||||
- Scheduler timings (5 min interval, midnight cron, jobstore): `sed -n '84,119p' api/app/services/scheduler.py`
|
||||
- Env var inventory: `grep -rn "os.getenv\|os.environ" api/ cli/ --include="*.py" | grep -v tests`
|
||||
- Auth semantics: `cat api/app/auth.py`; open health route: `grep -n "api/health" api/app/main.py`
|
||||
- Price refresh endpoint (POST, 202, background): `sed -n '19,23p' api/app/routers/prices.py`
|
||||
- Retry/backoff constants: `sed -n '153,201p' api/app/services/price_fetcher.py`
|
||||
- nginx templating of API_UPSTREAM: `cat web/nginx.conf web/Dockerfile`
|
||||
- sqlite3 CLI still absent from image: `grep -n "apt-get\|sqlite3" api/Dockerfile` (expect no hits)
|
||||
|
||||
Operator-supplied conventions to re-confirm on the homelab (not derivable from
|
||||
the repo): where the compose file lives on the homelab and how it is synced;
|
||||
Harbor login credentials/procedure; the real docker volume name
|
||||
(`docker volume ls`); offsite backup destination; whether any tag other than
|
||||
`:latest` is pushed.
|
||||
|
||||
Update this file whenever: a new env var is added (catalog table + README),
|
||||
scheduler timings change, the Dockerfile CMD changes, or the registry/deploy
|
||||
path changes.
|
||||
@@ -0,0 +1,831 @@
|
||||
---
|
||||
name: fin-test-coverage-campaign
|
||||
description: The executable campaign to take fin from 25 e2e CLI tests to a real safety net — direct api-level tests for crud.py, regression tests for every historical bug family, property tests for ledger invariants, service tests (net worth, price fetcher, scheduler), Alembic migration tests, and CI. Load when asked to improve test coverage, harden crud.py or services against regressions, add CI or a test gate, write api/tests, measure coverage, or after ANY accounting bug is found or fixed (to lock down its whole family). Also load before refactoring crud.py — characterization tests come first.
|
||||
---
|
||||
|
||||
# fin — Test Coverage Campaign
|
||||
|
||||
Mission: the only tests are `cli/tests/` (25 e2e tests). `api/app/crud.py` — 995
|
||||
lines holding ALL DB mutation logic — has no direct tests. `services/`, router
|
||||
error mapping, and the 8-revision Alembic chain are untested. There is no CI.
|
||||
Every accounting bug in this project's history shipped because nothing failed
|
||||
(chronicle: `fin-failure-archaeology`). This campaign closes that, phase by
|
||||
phase, with a measurable gate at every step.
|
||||
|
||||
All facts and numbers date-stamped **2026-07-05** (HEAD `082b608`). The
|
||||
authoring sandbox had no pip/venv, so every command below is **verified
|
||||
statically** unless it appears in the git history or skill siblings as
|
||||
executed; the gates therefore say "record the number you observe; expected ≈N".
|
||||
If your observation differs from the expectation, follow the branch instruction
|
||||
— never shrug and continue.
|
||||
|
||||
Read first: `/CLAUDE.md` (root rulebook). Alignment contracts:
|
||||
|
||||
- **Promotion**: every phase lands as its own reviewed change through
|
||||
`fin-change-control` (§1 classification, §7 done-checklist). Tests-only diffs
|
||||
that touch no invariant file classify as behavior changes whose "test
|
||||
covering the new behavior" requirement is satisfied by the tests themselves.
|
||||
- **Evidence**: this campaign produces level-(a) evidence in the
|
||||
`fin-validation-and-qa` hierarchy. Follow its hypothesis-predicts-numbers
|
||||
discipline: write the expected number and mechanism as a comment BEFORE the
|
||||
first run of every new test.
|
||||
- **Regression targets**: Phase 2 converts the `fin-failure-archaeology`
|
||||
chronicle into named tests. Read that skill before writing Phase 2.
|
||||
- **Environment**: `fin-build-and-env` owns setup (venv, minimal deps,
|
||||
yfinance-stub rationale). Not duplicated here.
|
||||
- **Oracles**: `fin-diagnostics-and-tooling` and `fin-ledger-analysis-toolkit`
|
||||
(being written in parallel as of 2026-07-05) may provide audit scripts and
|
||||
derived invariants; check them before hand-rolling an oracle in Phase 3.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
| You actually want to… | Use instead |
|
||||
|---|---|
|
||||
| Add one e2e test for a change you just made | `fin-validation-and-qa` §4 |
|
||||
| Decide whether a change is allowed / gates | `fin-change-control` |
|
||||
| Set up the Python environment | `fin-build-and-env` |
|
||||
| Understand accounting math for expected values | `fin-accounting-reference` |
|
||||
| Look up a past bug before pinning it | `fin-failure-archaeology` |
|
||||
| Debug a live failure | `fin-debugging-playbook` |
|
||||
|
||||
Use THIS skill to execute or resume the coverage campaign, and whenever a new
|
||||
bug is fixed — its family gets a Phase-2-style regression test even after the
|
||||
campaign is "done".
|
||||
|
||||
## Ground truth — what exists (2026-07-05)
|
||||
|
||||
### crud.py public surface — the coverage target
|
||||
|
||||
35 top-level `def`s in `api/app/crud.py` (verify: `grep -cE '^def ' api/app/crud.py`),
|
||||
33 public + 2 private:
|
||||
|
||||
| Area | Functions (count) |
|
||||
|---|---|
|
||||
| Accounts (5) | `get_accounts`, `get_account`, `create_account`, `update_account`, `deactivate_account` |
|
||||
| Categories (5) | `get_categories`, `get_category`, `create_category`, `update_category`, `delete_category` |
|
||||
| Transactions (9) | `get_transactions`, `get_transaction`, `create_transaction`, `update_transaction`, `delete_transaction`, `create_transfer`, `create_settlement`, `create_split_expense`, `create_splitwise_i_paid` |
|
||||
| Holdings (2) | `get_holdings`, `get_holding` |
|
||||
| Stock transactions (5) | `create_stock_transaction`, `get_stock_transactions`, `recompute_holding_from_stock_transactions`, `delete_stock_transaction`, `create_funded_buy` |
|
||||
| Spending (2) | `get_monthly_spending`, `get_spending_summary` |
|
||||
| RSU helpers, private (2) | `_add_months`, `_generate_rsu_schedule` |
|
||||
| RSU grants (5) | `get_rsu_grants`, `get_rsu_grant`, `create_rsu_grant`, `delete_rsu_grant`, `vest_rsu_grant` |
|
||||
|
||||
Exercised by the existing 25 cli tests (13 of 33): `get_accounts`,
|
||||
`create_account`, `get_categories`, `create_category`, `get_transactions`,
|
||||
`get_transaction`, `create_transaction`, `delete_transaction`,
|
||||
`create_transfer`, `create_settlement`, `create_split_expense`,
|
||||
`create_splitwise_i_paid`, `get_monthly_spending`.
|
||||
|
||||
**Never executed by any test (20 public + 2 private)**: everything in the
|
||||
Accounts area except create/list, all category mutations, `update_transaction`
|
||||
(the one existing edit test is rejected at the router before crud runs), the
|
||||
entire Holdings, Stock-transactions, RSU areas, and `get_spending_summary`.
|
||||
CLAUDE.md invariant 2 (holdings recompute) is completely untested.
|
||||
|
||||
Also untested but NOT in crud.py: `reconcile_accounts` — a two-leg settlement
|
||||
writer that lives in `api/app/routers/transactions.py` (line ~100), the only
|
||||
paired-leg writer outside crud. It is a Phase 2 target too.
|
||||
|
||||
### Existing suite
|
||||
|
||||
`cli/tests/test_cli.py`: **25 `def test_` functions** (verify:
|
||||
`grep -c 'def test_' cli/tests/test_cli.py`), inventoried per-test in
|
||||
`fin-validation-and-qa` §2. `cli/tests/conftest.py` (59 lines): sets a
|
||||
throwaway `DATABASE_URL` before the `app.*` import chain, stubs `yfinance` via
|
||||
`sys.modules.setdefault`, autouse `drop_all`+`create_all` per test.
|
||||
|
||||
### Migration chain
|
||||
|
||||
8 revisions, linear, head `h8c9d0e1f2a3` (verify: `ls api/alembic/versions/`).
|
||||
Table in `fin-change-control` §4.2.
|
||||
|
||||
### Git remote — checked for Phase 6
|
||||
|
||||
```
|
||||
origin https://gitea.cloud.traunseenet.com/markuseisl/fin.git
|
||||
```
|
||||
|
||||
Self-hosted **Gitea**, not GitHub (observed 2026-07-05 via `git remote -v`).
|
||||
GitHub Actions is off the menu; see Phase 6.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Baseline (run before touching anything)
|
||||
|
||||
### 0.1 Environment
|
||||
|
||||
Follow `fin-build-and-env` §(b), then add the coverage tool:
|
||||
|
||||
```bash
|
||||
cd /home/markuseisl/Documents/git/fin
|
||||
python -m venv .venv && . .venv/bin/activate
|
||||
pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0" coverage
|
||||
pip install -e cli/
|
||||
```
|
||||
|
||||
Do NOT install yfinance (deliberately stubbed) or alembic yet (needed only
|
||||
from Phase 5).
|
||||
|
||||
### 0.2 Run the suite as-found
|
||||
|
||||
```bash
|
||||
python -m pytest cli/tests -q
|
||||
```
|
||||
|
||||
**Record**: test count, pass/fail, wall time.
|
||||
**Expected**: `25 passed`, well under 60 s (as of 2026-07-05).
|
||||
|
||||
**Gate 0a — branch table:**
|
||||
|
||||
- `25 passed` → continue.
|
||||
- More than 25 passed → someone added tests since 2026-07-05; re-read
|
||||
`cli/tests/`, update the ground-truth section counts, continue.
|
||||
- Any failure or error **as-found** → **STOP the campaign.** A broken baseline
|
||||
means either the environment is wrong (missing dep → `fin-build-and-env`
|
||||
traps table) or a real regression shipped untested. Fix or quarantine
|
||||
(`@pytest.mark.skip` with a linked reason) as its own change through
|
||||
`fin-change-control` first, and record the finding as a new entry candidate
|
||||
for `fin-failure-archaeology`. The campaign starts from green, always.
|
||||
|
||||
### 0.3 Baseline coverage
|
||||
|
||||
```bash
|
||||
python -m coverage run -m pytest cli/tests -q
|
||||
python -m coverage report --include='*/api/app/*' -m
|
||||
```
|
||||
|
||||
**Record** (into your session notes and the final phase commit message):
|
||||
per-file % for `crud.py`, `services/net_worth.py`, `services/price_fetcher.py`,
|
||||
`services/scheduler.py`, `routers/*.py`, and the total.
|
||||
|
||||
**Expected shape** (predicted statically from the exercised-function map above
|
||||
— record what you actually see): `crud.py` roughly 45–65% (20 of 33 public
|
||||
functions never run); `net_worth.py` moderate (cash paths only);
|
||||
`price_fetcher.py` and `scheduler.py` near import-only (≈15–35%); router
|
||||
coverage high for `transactions.py`/`accounts.py`, near-zero for
|
||||
`stock_transactions.py`/`rsu_grants.py`/`holdings.py`/`prices.py`.
|
||||
|
||||
**Gate 0b:** if `crud.py` reports > 80% at baseline, your `--include` filter is
|
||||
wrong (probably matching `fin_cli`) or the suite grew — investigate before
|
||||
trusting any later delta. If `coverage` reports "no data collected", you ran
|
||||
pytest without `coverage run` wrapping it.
|
||||
|
||||
The recorded table is the campaign's before-photo; every later phase gate is a
|
||||
delta against it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Infrastructure: `api/tests/` for direct crud/service tests
|
||||
|
||||
The cli suite tests the deployed HTTP contract; it cannot reach crud/service
|
||||
functions the routers don't expose (e.g. `recompute_holding_from_stock_transactions`
|
||||
mid-history, `get_spending_summary` edge modes, `_generate_rsu_schedule`), and
|
||||
it cannot seed `price_cache` directly. That is why a second, api-level suite is
|
||||
justified rather than growing `cli/tests` (which stays e2e-only per
|
||||
`fin-validation-and-qa` §4).
|
||||
|
||||
### 1.1 Create `api/tests/conftest.py` — exact content
|
||||
|
||||
Modeled on `cli/tests/conftest.py` mechanics: throwaway `DATABASE_URL` before
|
||||
the import chain, yfinance stub. Verified statically against
|
||||
`app/database.py` (reads `DATABASE_URL` at import time via `db_config`).
|
||||
|
||||
```python
|
||||
"""Direct api-layer fixtures: crud/services against a throwaway SQLite.
|
||||
|
||||
Same bootstrap discipline as cli/tests/conftest.py: DATABASE_URL must point at
|
||||
a throwaway file BEFORE anything imports app.database (which reads it at
|
||||
import time), and yfinance is stubbed so the heavy package is not required.
|
||||
Unconditional overwrite: a real DATABASE_URL in the shell must never leak in.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT / "api"))
|
||||
|
||||
sys.modules.setdefault("yfinance", types.ModuleType("yfinance"))
|
||||
|
||||
_db_dir = tempfile.mkdtemp(prefix="fin-api-tests-")
|
||||
os.environ["DATABASE_URL"] = f"sqlite:///{_db_dir}/test.db"
|
||||
|
||||
from app.database import Base, SessionLocal, engine # noqa: E402
|
||||
from app import models # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_schema():
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
Base.metadata.create_all(bind=engine)
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db():
|
||||
session = SessionLocal()
|
||||
try:
|
||||
yield session
|
||||
finally:
|
||||
session.rollback()
|
||||
session.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_account(db):
|
||||
def _make(name="Giro", type_="bank", currency="EUR"):
|
||||
acct = models.Account(name=name, type=type_, currency=currency)
|
||||
db.add(acct)
|
||||
db.commit()
|
||||
db.refresh(acct)
|
||||
return acct
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def seed_price(db):
|
||||
"""Insert a price_cache row directly — the only sanctioned way to get a
|
||||
price into tests (yfinance is a stub; never fetch)."""
|
||||
from datetime import datetime
|
||||
|
||||
def _seed(symbol, price, currency="USD", source="test"):
|
||||
db.add(models.PriceCache(symbol=symbol, price=price,
|
||||
currency=currency, source=source,
|
||||
fetched_at=datetime.utcnow()))
|
||||
db.commit()
|
||||
return _seed
|
||||
```
|
||||
|
||||
Coexistence note (verified statically): when both suites run in one pytest
|
||||
invocation, both conftests set throwaway URLs but `app.database` is imported
|
||||
once — the engine binds to whichever conftest imports first. Both URLs are
|
||||
throwaway temp dirs, so this is safe; both autouse fixtures drop/create the
|
||||
same engine's schema per test. If you ever see one suite's data in the other,
|
||||
that is a missing autouse fixture, not a conftest bug.
|
||||
|
||||
Rule inherited from `fin-change-control` §2.1: **tests use throwaway databases
|
||||
only** — the temp-dir-then-import pattern above is the load-bearing part. Do
|
||||
not "simplify" it.
|
||||
|
||||
### 1.2 One trivial proving test — `api/tests/test_crud_accounts.py`
|
||||
|
||||
```python
|
||||
from app import crud, schemas
|
||||
|
||||
|
||||
def test_create_account_with_opening_balance(db):
|
||||
acct = crud.create_account(db, schemas.AccountCreate(
|
||||
name="Giro", type="bank", currency="EUR", initial_balance=250.0))
|
||||
# create_account writes one opening 'income' transaction of 250
|
||||
txns = crud.get_transactions(db, account_id=acct.id)
|
||||
assert len(txns) == 1
|
||||
assert txns[0].amount == 250.0
|
||||
assert txns[0].type == "income"
|
||||
assert txns[0].description == "Opening balance"
|
||||
```
|
||||
|
||||
### 1.3 Gate 1
|
||||
|
||||
```bash
|
||||
python -m pytest api/tests -q # expect: 1 passed
|
||||
python -m pytest cli/tests -q # expect: still 25 passed
|
||||
python -m pytest cli/tests api/tests -q # expect: 26 passed — coexistence proven
|
||||
```
|
||||
|
||||
Branches:
|
||||
|
||||
- `ModuleNotFoundError: app` in api/tests → the `sys.path.insert` line is
|
||||
missing or the file moved; do not fix by installing the api package or
|
||||
editing cli's conftest.
|
||||
- cli/tests suddenly failing → your api conftest leaked state (most likely you
|
||||
made the `DATABASE_URL` assignment conditional, or imported `app.*` above the
|
||||
env assignment). Revert to the exact code above.
|
||||
- Combined run failing while individual runs pass → fixture-ordering conflict;
|
||||
check both autouse fixtures still drop/create and that neither conftest
|
||||
caches Session objects at module scope.
|
||||
|
||||
Land Phase 1 as its own change (conftest + one test). Classification:
|
||||
behavior-change gate per `fin-change-control` §1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Regression lockdown: the chronicle becomes tests
|
||||
|
||||
Convert `fin-failure-archaeology`'s bug families into named tests. Naming
|
||||
convention: `test_regress_<sha>_<slug>` so the pinned incident is greppable.
|
||||
For each family below: the exact scenario, the predicted numbers (write them as
|
||||
comments per `fin-validation-and-qa` §1), and the function it pins. Put crud-
|
||||
level tests in `api/tests/test_regressions.py` (split by area when it grows);
|
||||
anything asserting HTTP status codes belongs in `cli/tests` or an api-level
|
||||
`TestClient` test.
|
||||
|
||||
### Family A — spending/income inflation (`e36d719`, migration `h8c9d0e1f2a3`)
|
||||
|
||||
Pins: `create_funded_buy`, `get_monthly_spending`, router `reconcile_accounts`.
|
||||
|
||||
1. `test_regress_e36d719_funded_buy_not_spending` — EUR bank account +
|
||||
investment account; `crud.create_funded_buy` with
|
||||
`FundedBuyRequest(type="buy", shares=5, price_per_share=100, fee=0,
|
||||
total_cost=500, currency="USD", funded_by_account_id=bank.id, …)` and
|
||||
`eur_usd_rate=1.25`. Predict: bank leg `type == "transfer"`,
|
||||
`amount == -400.0` (−500/1.25), description `"Buy 5 <TICKER>"`;
|
||||
`get_monthly_spending` for that month → `total == 0.0` and
|
||||
`income_total == 0.0`.
|
||||
2. `test_regress_e36d719_funded_sell_not_income` — funded sell 5 @ 100, fee 2:
|
||||
bank leg `+498/1.25 == 398.4`, type `transfer`; income_total 0.
|
||||
3. `test_regress_e36d719_reconcile_not_income` — via `TestClient` POST
|
||||
`/api/transactions/reconcile` (bank + receivable account): two `settlement`
|
||||
legs, bank `+amount`, receivable `−amount`; spending total 0, income_total 0.
|
||||
4. `test_regress_a9f045f_e61b254_payable_counts_receivable_excluded` — expense
|
||||
−30 on a **payable** account counts (total 30); expense −30 on a
|
||||
**receivable** account is excluded (total 0). The asymmetry is intentional —
|
||||
do not "fix" it (settled question, archaeology top table).
|
||||
5. `test_opening_balance_not_income` — `create_account(initial_balance=500)` →
|
||||
`get_monthly_spending(current month)["income_total"] == 0.0` (the
|
||||
`description != 'Opening balance'` filter in crud).
|
||||
|
||||
### Family B — orphan legs / linked-group deletes (`a06f921`, `a1f109b`, `e36d719`)
|
||||
|
||||
Pins: `delete_transaction`, `create_transfer`, `create_settlement`,
|
||||
`create_split_expense`, `create_splitwise_i_paid`.
|
||||
|
||||
1. `test_regress_a06f921_delete_either_transfer_leg_removes_both` — create
|
||||
transfer, delete leg 1 → 0 rows remain; repeat deleting leg 2 first.
|
||||
2. `test_regress_e36d719_split_expense_delete_any_of_three` — three
|
||||
parametrized cases: deleting the expense row, the from-leg, or the to-leg
|
||||
each removes all 3 rows (the expense links via `linked_transaction_id` to
|
||||
the from-leg; delete follows links in both directions).
|
||||
3. `test_regress_a1f109b_legacy_fallback_picks_adjacent_leg` — two transfers
|
||||
between the same two accounts on the same date; null out
|
||||
`linked_transaction_id` on all four rows (simulating pre-migration legacy
|
||||
rows, direct UPDATE via the db fixture); delete one leg → exactly its
|
||||
closest-id partner dies with it, the other transfer's 2 rows survive.
|
||||
4. `test_zero_sum_after_any_group_delete` — after each delete above, no
|
||||
remaining row's `linked_transaction_id` references a missing id.
|
||||
|
||||
### Family C — FX missing means excluded, never mis-converted (`6b7d19b`, `81091fc`)
|
||||
|
||||
Pins: `services/net_worth._to_eur` via `calculate_net_worth`.
|
||||
|
||||
1. `test_regress_6b7d19b_missing_gbp_rate_excludes_not_converts` — GBP bank
|
||||
account with balance 100 (one income txn); seed only `EUR/USD = 1.10`.
|
||||
Predict: that account's `balance_eur == 0.0`, `total_eur == 0.0`, and
|
||||
`caplog` contains `"No EUR/GBP rate"`. Negative control in the same test:
|
||||
seed `EUR/GBP = 0.85` → `balance_eur == pytest.approx(117.647, rel=1e-4)`
|
||||
(100 / 0.85).
|
||||
2. `test_eur_usd_fallback_is_reported_not_used_for_conversion` — no price rows
|
||||
at all, one EUR account with 100: `eur_usd_rate == 1.0` (reported fallback,
|
||||
warning logged) but `total_eur == 100.0` untouched. This pins the current
|
||||
behavior; if the 1.0 report is ever removed, this test is the notification.
|
||||
|
||||
### Family D — holdings recompute (`c558999`, `1480088`, `bc256f6`)
|
||||
|
||||
Pins: `create_stock_transaction`, `delete_stock_transaction`,
|
||||
`recompute_holding_from_stock_transactions`, `vest_rsu_grant`. This is
|
||||
CLAUDE.md invariant 2's first-ever test coverage.
|
||||
|
||||
1. `test_regress_c558999_avg_cost_fold` — buy 10 @ 100 fee 5 → holding
|
||||
`shares == 10`, `avg_cost_basis_per_share == 100.5`; then sell 4 →
|
||||
`shares == 6`, avg stays `100.5` (weighted-average sell reduces cost
|
||||
proportionally).
|
||||
2. `test_regress_c558999_oversell_rejected_422` — sell 20 with 10 held:
|
||||
`crud.create_stock_transaction` raises `HTTPException` with
|
||||
`status_code == 422` and the holding is unchanged (rollback happened).
|
||||
3. `test_regress_1480088_delete_mid_history_recomputes` — buy 10, buy 10, sell
|
||||
5; delete the second buy → holding recomputed over the remaining history to
|
||||
`shares == 5` (10 − 5); predict the resulting avg cost by hand in the
|
||||
comment first. Then: buy 10, sell 8, delete the buy →
|
||||
`delete_stock_transaction` raises 422 ("Cannot sell 8"), the buy row still
|
||||
exists, holding unchanged.
|
||||
4. `test_delete_last_stock_txn_deletes_holding` — one buy, delete it → holding
|
||||
row gone (`get_holding` returns None).
|
||||
5. `test_regress_bc256f6_vest_delete_reverts_grant` — RSU account, grant
|
||||
(gross 100, withholding 0.5), seed price for the ticker, `vest_rsu_grant` →
|
||||
`net_shares == 50` (`floor(100 × 0.5)`), grant has `vested_at` and
|
||||
`vested_stock_transaction_id`; delete that stock transaction → grant back to
|
||||
pending (`vested_at is None`, link cleared), holding gone. This was the
|
||||
undeletable-grant deadlock.
|
||||
6. `test_vest_without_cached_price_is_422` — no price row →
|
||||
`vest_rsu_grant` raises 422 mentioning "No cached price". Never let a vest
|
||||
invent a price.
|
||||
|
||||
### Family E — atomic multi-row writes (`bc256f6`, `c67d34f`)
|
||||
|
||||
Pins: `create_split_expense`, `create_splitwise_i_paid`, cross-linking.
|
||||
|
||||
1. `test_regress_bc256f6_split_expense_three_rows_linked` — total 100, partner
|
||||
40: expense `amount == -60.0` typed `expense`; transfer legs −40/+40; the
|
||||
expense's `linked_transaction_id == from_leg.id`; both transfer legs
|
||||
cross-linked.
|
||||
2. `test_regress_3e7f8ec_splitwise_net_share_per_category` — two
|
||||
`splitwise_i_paid` in one month with different categories (100/45 and 60/30)
|
||||
→ spending total `55 + 30 == 85.0`, per-category totals 55 and 30. Extends
|
||||
the existing cli test to the multi-category merge path in
|
||||
`get_monthly_spending`.
|
||||
|
||||
### Family F — funded-buy currency handling (`b3062b7`)
|
||||
|
||||
1. `test_regress_b3062b7_usd_funding_account_no_double_conversion` — USD bank
|
||||
account funds a USD buy, total_cost 500: bank leg `amount == -500.0` exactly
|
||||
(currency match branch — no division by the rate).
|
||||
2. `test_funded_buy_zero_rate_guard` — `eur_usd_rate=0` and no
|
||||
`exchange_rate_used`, EUR funding account: bank amount falls back to
|
||||
`cash_native` un-divided (the `used_rate > 0` guard). Pins current behavior.
|
||||
|
||||
### Gate 2
|
||||
|
||||
```bash
|
||||
python -m coverage run -m pytest cli/tests api/tests -q
|
||||
python -m coverage report --include='*/api/app/*' -m
|
||||
```
|
||||
|
||||
- **Test count**: record it; expected ≈ 25 + 20–28 new.
|
||||
- **Coverage floor**: `crud.py` ≥ **75%**, and `coverage report -m` shows no
|
||||
crud function with an entirely-red body except the RSU-schedule helpers
|
||||
(`_generate_rsu_schedule` / `_add_months` — Phase 3/4 material) and pure
|
||||
getters you consciously deferred. List any deferred function by name in the
|
||||
commit body.
|
||||
- Branch: a regression test that passes without any product-code change is
|
||||
correct here (the bugs are already fixed — you are pinning). But if a
|
||||
regression test FAILS, you have found a live regression: stop the phase,
|
||||
handle it as a bug through `fin-debugging-playbook` + `fin-change-control`,
|
||||
and add the incident to `fin-failure-archaeology`.
|
||||
|
||||
Land Phase 2 as one change (or one per family if review size demands).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Invariant/property tests
|
||||
|
||||
Encode the ledger's algebraic invariants so *unknown future* bugs in the
|
||||
families above fail loudly. Derivations for these invariants live in
|
||||
`fin-accounting-reference` (and, when it lands, `fin-ledger-analysis-toolkit`
|
||||
— check it before re-deriving).
|
||||
|
||||
### Invariants to encode (file: `api/tests/test_properties.py`)
|
||||
|
||||
- **I1 — two-leg zero-sum**: every same-currency `transfer`/`settlement` group
|
||||
written by `create_transfer`/`create_settlement`/`reconcile` sums to 0 across
|
||||
its legs. Documented exceptions, assert them explicitly instead: cross-
|
||||
currency transfers (amounts differ by design), `splitwise_pay` pairs
|
||||
(−total / +partner_share — intentionally non-zero, invariant 6).
|
||||
- **I2 — balance formula equivalence**: for any bank account,
|
||||
`calculate_net_worth`'s `balance_native` equals a test-local Python fold of
|
||||
`sum(t.amount - t.fee)` over its transactions (CLAUDE.md invariant 5,
|
||||
computed by an independent implementation, not by re-querying the same SQL).
|
||||
- **I3 — holdings fold**: after any valid random buy/sell sequence,
|
||||
`recompute_holding_from_stock_transactions` equals a reference weighted-
|
||||
average fold implemented in the test (~15 lines — keep it independent).
|
||||
- **I4 — spending invariance under asset movement**: take any month's
|
||||
`get_monthly_spending` result, then add N random transfers/settlements/
|
||||
funded buys in that month → `total` and `income_total` unchanged.
|
||||
- **I5 — delete closure**: after deleting any randomly chosen transaction, no
|
||||
surviving row's `linked_transaction_id` points at a missing id, and no
|
||||
account's balance changed except those in the deleted group.
|
||||
|
||||
### Solution menu — ranked, with obligations
|
||||
|
||||
**Option 1 (preferred): `hypothesis`** — `pip install hypothesis` (dev-only;
|
||||
do not add to `api/requirements.txt`, which is the Docker image's dependency
|
||||
list).
|
||||
Obligations: define an action strategy (`st.sampled_from` over account/txn
|
||||
constructors with `st.floats(allow_nan=False, allow_infinity=False,
|
||||
min_value=0.01, max_value=1e6)` rounded to cents); `@settings(deadline=None)`
|
||||
(SQLite file IO makes per-example deadlines flaky); set
|
||||
`hypothesis.settings(database=None)` or gitignore `.hypothesis/` so shrink
|
||||
artifacts never enter the repo; promote every found counterexample to an
|
||||
explicit `@example(...)` so it reruns forever; `derandomize=True` in the CI
|
||||
profile (Phase 6) for reproducibility.
|
||||
|
||||
**Option 2: hand-rolled random-action generator** — no new dependency.
|
||||
Obligations you now own manually: seed from `os.environ.get("FIN_TEST_SEED")`
|
||||
or `random.randrange(2**32)`, ALWAYS print the seed in the failure message
|
||||
(`assert ok, f"seed={seed}: …"`), document replay
|
||||
(`FIN_TEST_SEED=<n> python -m pytest …`), and cap runtime (≤200 actions,
|
||||
≤20 iterations) so the suite stays under a minute. No shrinking — expect ugly
|
||||
failures.
|
||||
|
||||
**Option 3: exhaustive small-case enumeration** — enumerate all sequences of
|
||||
≤4 actions from a fixed small action set (itertools.product). Deterministic,
|
||||
dependency-free, but tiny state space; acceptable as a floor for I1/I5 only.
|
||||
|
||||
Choose 1 if the environment allows new dev dependencies, else 2. Record the
|
||||
choice and seed policy in the test module docstring.
|
||||
|
||||
### Gate 3
|
||||
|
||||
- At least I1–I5 implemented (any option).
|
||||
- **Prove the oracle bites**: temporarily flip a sign in one reference fold
|
||||
(e.g. I3's fee handling) and observe the property test fail with a
|
||||
reproducible seed/example; revert. A property test that never failed during
|
||||
development is unverified.
|
||||
- Full suite still green; runtime recorded — if the combined suite now exceeds
|
||||
~90 s, cut iteration counts, don't delete invariants.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Services: net worth, price fetcher, scheduler
|
||||
|
||||
File: `api/tests/test_services.py` (split later if large). NO live network,
|
||||
ever: yfinance stays a stub; `httpx.get` gets monkeypatched.
|
||||
|
||||
### 4.1 `services/net_worth.py` — synthetic `price_cache` rows
|
||||
|
||||
Using the `seed_price` fixture:
|
||||
|
||||
- USD cash: bank account (currency USD) with +110 income; seed
|
||||
`EUR/USD = 1.10` → `balance_eur == 100.0`, `total_eur == 100.0`.
|
||||
- Investment holding: create via `crud.create_stock_transaction` (buy 10
|
||||
VWCE @ 100), seed `VWCE = 100 (USD)` and `EUR/USD = 1.25` →
|
||||
account `balance_native == 1000.0`, `balance_eur == 800.0`.
|
||||
- GBP-priced holding: seed `TICK = 50` with `currency="GBP"` and
|
||||
`EUR/GBP = 0.85` → `balance_eur == 10*50/0.85`.
|
||||
- Missing ticker price → holding contributes 0 (price_map default), no crash.
|
||||
- `manual_override_balance` on an investment account wins over holdings and is
|
||||
converted via the account currency.
|
||||
- Unknown account type: insert an `Account` with `type="weird"` directly via
|
||||
the db fixture (the schema `Literal` blocks it at the API; crud/services must
|
||||
still not crash) → account skipped, warning logged.
|
||||
- Inactive accounts excluded from the total.
|
||||
|
||||
### 4.2 `services/price_fetcher.py` — mocked httpx and a rigged yfinance stub
|
||||
|
||||
The conftest already made `yfinance` an empty module; give it behavior
|
||||
per-test: `import yfinance` then `monkeypatch.setattr(yfinance, "Ticker",
|
||||
FakeTicker, raising=False)` (the module-scope `import yfinance as yf` in
|
||||
price_fetcher resolves to the same stub object). Build `FakeTicker` with a
|
||||
`fast_info` namespace (`last_price`, `currency`) and an `info` dict.
|
||||
|
||||
**Mandatory obligation: `monkeypatch.setattr(time, "sleep", lambda s: None)`**
|
||||
(patch `app.services.price_fetcher.time.sleep`) in every retry-path test — the
|
||||
real schedule sleeps 1+2+4 s per candidate across 9 candidates.
|
||||
|
||||
- `fetch_eur_usd` success: monkeypatch `httpx.get` to return a fake response
|
||||
(`raise_for_status` no-op, `json()` → `{"rates": {"USD": 1.1}}`) → returns
|
||||
1.1, upserts `EUR/USD` row with source `frankfurter`.
|
||||
- `fetch_eur_usd` failure with cache: httpx raises; pre-seed `EUR/USD = 1.08` →
|
||||
returns 1.08, `fetched_at` NOT bumped.
|
||||
- **`test_regress_81091fc_no_fake_rate_on_cold_failure`**: httpx raises, no
|
||||
cache → returns `None` and `price_cache` stays EMPTY (the 1.0-fallback bug).
|
||||
- **`test_regress_bc8c8d5_pence_division`**: FakeTicker reports
|
||||
`fast_info.currency == "GBp"`, price 12345 → cached price `123.45`, currency
|
||||
`GBP`.
|
||||
- Suffix discovery: bare ticker returns no data (`last_price is None`, `info`
|
||||
without `regularMarketPrice`), `TICK.DE` succeeds → row cached under the
|
||||
ORIGINAL symbol `TICK`.
|
||||
- All candidates fail → returns cached value if present, else `0.0`; no row
|
||||
written.
|
||||
- `get_tracked_tickers`: union of holdings with shares > 0 and pending grants;
|
||||
vested grants and zero-share holdings excluded.
|
||||
|
||||
### 4.3 `services/scheduler.py` — call the job function directly
|
||||
|
||||
Never `create_scheduler(...).start()` in tests. Call
|
||||
`fetch_prices_and_snapshot(database_url)` with the conftest's throwaway URL
|
||||
(`os.environ["DATABASE_URL"]`), monkeypatching
|
||||
`app.services.scheduler.fetch_all_prices` (it is imported into the scheduler
|
||||
module's namespace — patch it THERE) to `lambda db: 0`:
|
||||
|
||||
- One `NetWorthSnapshot` row for `date.today()` with `total_eur` matching
|
||||
seeded accounts and parseable `breakdown_json`.
|
||||
- Run twice same day → still exactly one row (update path), values refreshed.
|
||||
- `create_scheduler` unit check: returns a scheduler with job ids
|
||||
`{"price_refresh", "daily_snapshot"}` and correct triggers — WITHOUT calling
|
||||
`.start()`. Note (verified statically): `SQLAlchemyJobStore(url=…)` in
|
||||
`create_scheduler` touches the DB URL at construction; if this creates an
|
||||
unwanted `apscheduler_jobs` table in the shared test DB, point it at its own
|
||||
temp file via a distinct URL argument — the function takes the URL as a
|
||||
parameter precisely so you can.
|
||||
|
||||
### Gate 4
|
||||
|
||||
`coverage report --include='*/api/app/services/*' -m` — floors:
|
||||
`net_worth.py` ≥ **90%**, `price_fetcher.py` ≥ **80%**, `scheduler.py` ≥
|
||||
**70%** (the `create_scheduler` wiring may stay partially red). No single test
|
||||
over 2 s (if one is, you forgot the `time.sleep` patch). Full suite green.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Migration tests: the 8-revision chain
|
||||
|
||||
Now `pip install alembic` (first phase that needs it). File:
|
||||
`api/tests/test_migrations.py`.
|
||||
|
||||
Key mechanics (verified statically against `api/alembic/env.py`): env.py
|
||||
resolves the URL as `DATABASE_URL` env var → `sqlalchemy.url` → app default,
|
||||
and inserts the api dir into `sys.path` itself. `fileConfig` is skipped when
|
||||
the Config has no file. So a programmatic Config works from any cwd:
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _migrated_engine(monkeypatch, tmp_path):
|
||||
url = f"sqlite:///{tmp_path}/mig.db"
|
||||
# env.py reads DATABASE_URL FIRST — without this the conftest's shared
|
||||
# test-DB URL would be migrated instead. Always a fresh file per test.
|
||||
monkeypatch.setenv("DATABASE_URL", url)
|
||||
cfg = Config()
|
||||
cfg.set_main_option("script_location", str(REPO_ROOT / "api" / "alembic"))
|
||||
command.upgrade(cfg, "head")
|
||||
return sa.create_engine(url)
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
1. `test_chain_is_linear_with_8_revisions` — via
|
||||
`alembic.script.ScriptDirectory(str(REPO_ROOT / "api" / "alembic"))`:
|
||||
exactly one head, id `h8c9d0e1f2a3`, walk from base counts **8** revisions,
|
||||
no branchpoints. (If this fails because a NEW revision exists: update the
|
||||
expected head/count AND the table in `fin-change-control` §4.2 — that is
|
||||
maintenance, not a bug.)
|
||||
2. `test_empty_to_head_matches_metadata` — migrate a fresh temp DB, then
|
||||
compare against `Base.metadata`: build
|
||||
`{table: sorted(column names)}` from `sa.inspect(engine)` (excluding
|
||||
`alembic_version`) and from `Base.metadata.tables`; assert equal. **First
|
||||
run**: record any diff verbatim. Expected: empty. If NOT empty, you have
|
||||
found model/migration drift — the exact failure mode `fin-change-control`
|
||||
§4.3 warns about (`create_all` hides it in every other test). STOP, file it
|
||||
as a schema-change fix through change control; only allowlist a difference
|
||||
with an inline comment naming the reason.
|
||||
3. `test_downgrade_one_runs` — after upgrade, `command.downgrade(cfg, "-1")`
|
||||
then `command.upgrade(cfg, "head")` completes. This asserts the downgrade
|
||||
*runs*, not that it restores data (`h8c9d0e1f2a3`'s downgrade is an honest
|
||||
no-op by design).
|
||||
4. Stretch (optional, document if skipped):
|
||||
`alembic.autogenerate.compare_metadata` against the migrated DB expecting
|
||||
`[]` — catches type/index drift the name-comparison misses, but is noisier
|
||||
on SQLite; adopt only if the noise is manageable.
|
||||
|
||||
Gate 5: all migration tests green; the recorded schema diff is empty or each
|
||||
entry is allowlisted-with-reason. Full suite still green (migration tests must
|
||||
not have touched the shared conftest DB — if `cli/tests` now fails with an
|
||||
`alembic_version` artifact, a test forgot the `monkeypatch.setenv`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — CI
|
||||
|
||||
Checked first, per protocol: `git remote -v` → sole remote is
|
||||
`https://gitea.cloud.traunseenet.com/markuseisl/fin.git` — **self-hosted
|
||||
Gitea** (observed 2026-07-05). GitHub Actions does not apply.
|
||||
|
||||
Hard constraints for ANY option: no docker required, no live yfinance or
|
||||
frankfurter calls (the suite is already offline-by-construction — keep it
|
||||
that way), total runtime target < 3 minutes.
|
||||
|
||||
### Menu — ranked
|
||||
|
||||
**Option 1: Gitea Actions** (GitHub-Actions-compatible syntax). Preconditions
|
||||
you must VERIFY before choosing it — do not assume: (a) Actions is enabled on
|
||||
the instance and repo (repo Settings → Actions, or ask the owner — it is
|
||||
off by default on older Gitea), (b) an `act_runner` is registered. If either
|
||||
fails → Option 2. Workflow `.gitea/workflows/tests.yml`:
|
||||
|
||||
```yaml
|
||||
name: tests
|
||||
on: [push]
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
container: python:3.12-slim
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: pip install pytest fastapi httpx sqlalchemy apscheduler "pydantic>=2.0" coverage alembic hypothesis
|
||||
- run: pip install -e cli/
|
||||
- run: python -m coverage run -m pytest cli/tests api/tests -q
|
||||
- run: python -m coverage report --include='*/api/app/*' --fail-under=80
|
||||
```
|
||||
|
||||
(Verified statically — could not be executed from the sandbox; the dependency
|
||||
list mirrors Phase 0/5. Adjust `hypothesis` to your Phase 3 choice. If the
|
||||
runner lacks internet for pip, bake a runner image — that is
|
||||
`fin-run-and-operate` territory.)
|
||||
Obligations: pin `--fail-under` to your actual Gate-6 floor, not aspiration;
|
||||
add the badge/URL to README only once it has passed on a real push.
|
||||
|
||||
**Option 2: committed pre-push hook** (works with zero server support).
|
||||
`scripts/git-hooks/pre-push`:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
set -e
|
||||
python -m pytest cli/tests api/tests -q
|
||||
```
|
||||
|
||||
plus a one-time install command documented in README:
|
||||
`git config core.hooksPath scripts/git-hooks`. Obligations: keep it under a
|
||||
minute; document that `git push --no-verify` bypasses it (an emergency valve,
|
||||
not a habit); this protects only machines that ran the config line.
|
||||
|
||||
**Option 3: documented gate only** — the `fin-change-control` §7 checklist
|
||||
already mandates the pytest run; add the combined
|
||||
`cli/tests api/tests` command there and to CLAUDE.md. Weakest — no automation
|
||||
— acceptable only as an interim while Option 1's preconditions are pending.
|
||||
|
||||
Gate 6: state which option you chose and WHY (what you found when verifying
|
||||
Option 1's preconditions); demonstrate one real failing run blocking a push
|
||||
(introduce a deliberate test failure on a scratch branch, observe the block,
|
||||
revert). Record the evidence in the commit message.
|
||||
|
||||
---
|
||||
|
||||
## Promotion protocol (routes through fin-change-control)
|
||||
|
||||
1. **One phase = one reviewed change** (Phase 2 may split by family). Never
|
||||
bundle test phases with product-code fixes — if a phase uncovers a product
|
||||
bug, the fix is a separate change with its own class and gate.
|
||||
2. Classification per `fin-change-control` §1: Phases 1–5 are tests-only →
|
||||
behavior-change gate (compile + full suite green; the tests are their own
|
||||
coverage). Phase 6 adds repo config → same, plus README update for the hook
|
||||
install line. Nothing here is deploy-affecting: **no phase may touch
|
||||
`api/requirements.txt`, Dockerfiles, or compose** (dev-only deps like
|
||||
coverage/hypothesis/alembic-for-tests live in the venv / CI config, not the
|
||||
image).
|
||||
3. Every phase commit body records the gate numbers: test count before/after,
|
||||
coverage table delta, runtime. Commit style per `fin-change-control` §6.
|
||||
4. **CLAUDE.md** gains a single test-run line
|
||||
(`python -m pytest cli/tests api/tests -q`) only after Phase 2 has landed
|
||||
and stayed green across at least one subsequent unrelated change — not
|
||||
before (stability first, advertisement second). Update
|
||||
`fin-validation-and-qa` §2's inventory pointers at the same time.
|
||||
5. Campaign completion floors (measure, don't eyeball): `crud.py` ≥ **90%**,
|
||||
`api/app` total ≥ **80%**, every CLAUDE.md invariant (1, 2, 5, 6) named in
|
||||
at least one test docstring, migration chain tested, CI or hook active.
|
||||
Record the final coverage table in the last phase's commit.
|
||||
|
||||
## Fenced-off wrong paths — do not do these
|
||||
|
||||
- **Async test rewrite** (pytest-asyncio, AsyncSession, httpx.AsyncClient
|
||||
against the app): the engine is synchronous by design (CLAUDE.md). Any
|
||||
"modernization" here is architecture change, not testing.
|
||||
- **Pointing any test at the prod DB or a copy of it inside the test tree**:
|
||||
prod `/data/fin.db` is sacred (`fin-change-control` §2). Throwaway
|
||||
temp-file SQLite only, created by the conftest pattern above.
|
||||
- **Live yfinance/frankfurter in tests or CI**: the yfinance stub and httpx
|
||||
monkeypatching are load-bearing. A test that "just fetches one real price"
|
||||
makes the suite flaky, slow, and network-dependent — the exact failure CI
|
||||
must never have.
|
||||
- **Editing `cli/tests/conftest.py`'s sys.path/env bootstrapping casually**:
|
||||
the set-DATABASE_URL-before-import dance is what keeps real DBs out of
|
||||
reach. Extend by copying the pattern (Phase 1), never by "cleaning it up".
|
||||
- **`Base.metadata.create_all` as a migration-test substitute**: it creates a
|
||||
schema that matches the models by construction and proves nothing about the
|
||||
migration chain — the drift it hides is the point of Phase 5
|
||||
(`fin-change-control` §4.3).
|
||||
- **Refactoring crud.py "for testability" before characterization tests
|
||||
exist**: no extracting pure functions, no dependency injection, no splitting
|
||||
the file until Phase 2's floor is met. The current shape is testable as-is
|
||||
(the `db` fixture proves it); refactors before pinning behavior are how
|
||||
accounting semantics silently change.
|
||||
- **Chasing 100%**: the last percent (defensive `except` arms in
|
||||
`fetch_all_prices`, log-only branches) buys nothing. The floors above are
|
||||
the contract; spend surplus effort on Phase 3 invariants instead.
|
||||
- **Skipping/xfailing a red test to make a gate**: with no CI history, a
|
||||
skipped test is a deleted test (`fin-validation-and-qa` §5: fix or delete,
|
||||
never skip).
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
Authored 2026-07-05 against HEAD `082b608`; the authoring sandbox had no
|
||||
pip/venv/docker, so all run-commands are verified statically (files read in
|
||||
full: `api/app/crud.py`, `cli/tests/conftest.py`, `cli/tests/test_cli.py`,
|
||||
`api/app/services/{net_worth,price_fetcher,scheduler}.py`,
|
||||
`api/app/routers/transactions.py`, `api/app/{main,database}.py`,
|
||||
`api/app/schemas.py`, `api/alembic/env.py`; sibling skills: change-control,
|
||||
failure-archaeology, validation-and-qa, build-and-env).
|
||||
|
||||
Recount the volatile facts before trusting them:
|
||||
|
||||
```bash
|
||||
grep -cE '^def ' api/app/crud.py # 35 defs (33 public) as of 2026-07-05
|
||||
wc -l api/app/crud.py # 995
|
||||
grep -c 'def test_' cli/tests/test_cli.py # 25
|
||||
ls api/tests/ 2>/dev/null # campaign progress: which phases landed
|
||||
grep -rc 'def test_' api/tests/ 2>/dev/null # api-suite size
|
||||
ls api/alembic/versions/ # 8 revisions, head h8c9d0e1f2a3
|
||||
git remote -v # Gitea (Phase 6 premise)
|
||||
ls .gitea/workflows scripts/git-hooks 2>/dev/null # which Phase-6 option landed
|
||||
git log --oneline -5 # HEAD moved? re-audit archaeology for new families
|
||||
```
|
||||
|
||||
Maintenance rules: when a phase lands, update the ground-truth counts and mark
|
||||
the phase done in this file (same change). When a new bug is fixed anywhere in
|
||||
fin, add its regression test using the Phase 2 naming scheme and register the
|
||||
incident in `fin-failure-archaeology` — the campaign's steady state is that
|
||||
those two updates always travel together. If `crud.py`'s function inventory
|
||||
changes, regenerate the area table with
|
||||
`grep -nE '^def |^# ---' api/app/crud.py` and re-derive the
|
||||
exercised/never-executed split from the current test suites, not from memory.
|
||||
@@ -0,0 +1,280 @@
|
||||
---
|
||||
name: fin-validation-and-qa
|
||||
description: >
|
||||
Load BEFORE claiming any change to fin works, when writing or extending tests
|
||||
in cli/tests/, when deciding what evidence a fix or feature needs before it
|
||||
can be called done, or when reviewing someone else's change. Defines the
|
||||
evidence hierarchy (test > diagnostics numbers > fin --json > compile > eyeball),
|
||||
the golden inventory of what the e2e suite actually certifies, how to run and
|
||||
add tests safely (prod DB never a fixture), and the hypothesis-predicts-numbers
|
||||
discipline for accepting a fix.
|
||||
---
|
||||
|
||||
# fin — Validation and QA
|
||||
|
||||
Steady-state quality discipline for the fin repo. The only test suite is
|
||||
`cli/tests/` (e2e: `FinClient` → FastAPI `TestClient` → real `crud.py` →
|
||||
throwaway SQLite). There is NO CI — the green run on your machine IS the gate.
|
||||
The prod DB is sacred: no test, experiment, or "quick check" ever touches it.
|
||||
|
||||
Read `/home/markuseisl/Documents/git/fin/CLAUDE.md` first if you haven't.
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
| You want to… | Go to |
|
||||
|---|---|
|
||||
| Use the `fin` CLI against the live API | `fin` |
|
||||
| Expand test coverage as a campaign (direct crud.py tests, new layers) | `fin-test-coverage-campaign` |
|
||||
| Set up Python/pytest environment, install deps | `fin-build-and-env` |
|
||||
| Run the audit/diagnostics scripts themselves | `fin-diagnostics-and-tooling` |
|
||||
| Decide whether a change is allowed at all / merge gates | `fin-change-control` |
|
||||
| Debug a live misbehavior (not validate a fix) | `fin-debugging-playbook` |
|
||||
| Look up correct accounting math for expected values | `fin-accounting-reference` |
|
||||
| Check whether a bug class has happened before | `fin-failure-archaeology` |
|
||||
|
||||
## 1. Evidence hierarchy (strongest → weakest)
|
||||
|
||||
When you claim "X is fixed" or "X works", state which level your evidence is.
|
||||
|
||||
| Level | Evidence | Counts as proof for |
|
||||
|---|---|---|
|
||||
| (a) | Automated test in `cli/tests/` that **fails before** the change and **passes after** | Anything. The gold standard. |
|
||||
| (b) | Diagnostics-script numbers before/after, run on a **synthetic or copied** DB (see `fin-diagnostics-and-tooling`) | Accounting/balance logic when a test is genuinely impractical |
|
||||
| (c) | `fin --json` API-level checks against a throwaway/dev instance, with expected values written down first | Wiring, endpoint behavior, CLI plumbing |
|
||||
| (d) | `python -m py_compile …` / `tsc` via `vite build` passes | Nothing works — only that it parses/typechecks |
|
||||
| (e) | Manual UI walkthrough | Rendering and UX only |
|
||||
|
||||
**Hard rule: accounting-logic changes (balances, spending, transfers,
|
||||
settlements, splits, holdings, net worth) require level (a) or (b). Never
|
||||
eyeball a balance.**
|
||||
|
||||
### Hypothesis-predicts-numbers discipline
|
||||
|
||||
Before running ANY check (test, script, `fin --json` call):
|
||||
|
||||
1. Write down the exact number/output you expect and WHY (one sentence of mechanism).
|
||||
2. Run the check.
|
||||
3. If the number differs — even in a direction that "looks fixed" — the
|
||||
hypothesis is wrong or incomplete. Do not accept the fix.
|
||||
4. A fix is accepted only when **one mechanism explains ALL observations,
|
||||
including the negatives** (the checks that were expected to stay unchanged
|
||||
and did). A fix that explains the failing case but not why case Y was
|
||||
unaffected is a guess, not a fix.
|
||||
|
||||
## 2. Golden inventory — what a green run actually certifies
|
||||
|
||||
Source of truth: `/home/markuseisl/Documents/git/fin/cli/tests/test_cli.py`.
|
||||
**25 test functions as of 2026-07-05** (9 auth, 2 accounts, 2 transactions,
|
||||
3 transfer, 1 settlement, 1 split-expense, 1 splitwise-paid, 2 net-worth,
|
||||
1 spending, 3 CLI wiring). Re-count before trusting this table (see Provenance).
|
||||
|
||||
### Certified (locked by a test)
|
||||
|
||||
| Area | Test | Invariant it locks |
|
||||
|---|---|---|
|
||||
| Auth | `TestAuth::test_auth_off_allows_requests` | No `FIN_API_TOKEN` env → API open |
|
||||
| Auth | `test_health_ok`, `test_health_bypasses_auth` | `/api/health` always open, even with auth on |
|
||||
| Auth | `test_auth_on_rejects_missing_token`, `test_auth_on_rejects_wrong_token` | Auth on → 401 without/with wrong bearer token |
|
||||
| Auth | `test_auth_on_accepts_correct_token` | Correct token passes |
|
||||
| Auth | `test_bearer_scheme_is_case_insensitive` | `bearer` == `Bearer` |
|
||||
| Auth | `test_openapi_gated_when_auth_on`, `test_openapi_open_when_auth_off` | OpenAPI spec follows the auth gate |
|
||||
| Accounts | `TestAccounts::test_create_and_list` | Create/list roundtrip |
|
||||
| Accounts | `test_initial_balance_creates_opening_transaction` | `initial_balance` → one opening `income` txn of that amount |
|
||||
| Transactions | `TestTransactions::test_add_and_list_with_limit` | List is newest-first; `limit` respected |
|
||||
| Transactions | `test_limit_must_be_positive` | `limit=0` → 422 |
|
||||
| Transfer | `TestTransfer::test_two_legs_created_and_linked` | Two legs, ±amount, cross-linked via `linked_account_id` (CLAUDE.md invariant 1) |
|
||||
| Transfer | `test_deleting_one_leg_removes_whole_group` | Deleting either leg deletes both — no orphan legs |
|
||||
| Transfer | `test_editing_one_leg_is_rejected` | Editing one transfer leg → 422 |
|
||||
| Settlement | `TestSettlement::test_bank_to_splitwise_two_legs_linked` | bank→splitwise: two linked `settlement` legs, ±amount |
|
||||
| Split expense | `TestSplitExpense::test_three_rows_and_user_share_spending` | total 100/partner 40 → expense −60 + transfer ±40 (3 rows); spending counts only the 60 |
|
||||
| Splitwise paid | `TestSplitwisePaid::test_spending_counts_net_share` | bank −100, `splitwise_partner_share`=45, splitwise leg +45; spending = 55 (invariant 6) |
|
||||
| Net worth | `TestNetWorth::test_current_reflects_bank_balance` | Cash balance flows into `total_eur` |
|
||||
| Net worth | `test_fee_reduces_balance` | balance = SUM(amount − fee) (invariant 5): 100 − 50 − 2 = 48 |
|
||||
| Spending | `TestSpending::test_monthly_totals` | Monthly total + per-category breakdown |
|
||||
| CLI wiring | `TestCliWiring::test_accounts_list_json` | `fin accounts list --json` emits parseable JSON |
|
||||
| CLI wiring | `test_error_exits_nonzero` | API error → exit code 1 |
|
||||
| CLI wiring | `test_tx_add_refuses_transfer_type` | `tx add --type transfer` rejected client-side, nothing written |
|
||||
|
||||
### NOT certified (a green run says nothing about these)
|
||||
|
||||
| Area | Gap (as of 2026-07-05) |
|
||||
|---|---|
|
||||
| **Stocks/holdings** | Zero tests. `create_stock_transaction`, `delete_stock_transaction`, `recompute_holding_from_stock_transactions` (CLAUDE.md **invariant 2 is untested**), `create_funded_buy` |
|
||||
| **RSU** | Zero tests: grants, vest, schedule preview/generate, vest→holding linkage |
|
||||
| crud.py direct | No unit tests at all (~995 lines). Only paths reachable via the API calls above are exercised — `fin-test-coverage-campaign` owns closing this |
|
||||
| Mutations | `update_account`, `deactivate_account`, balance adjust/`reconcile`, category update/delete, `update_transaction` on plain txns, `delete_transaction` of a plain txn |
|
||||
| Settlement | `splitwise_to_bank` direction untested (only `bank_to_splitwise`) |
|
||||
| Spending | `get_spending_summary` (multi-month), spending with transfers/settlements/stock trades mixed in a month (the e36d719 bug class) |
|
||||
| Net worth | `services/net_worth.py` with holdings + `price_cache` (only pure-cash paths hit), history/snapshots, non-EUR accounts, FX conversion |
|
||||
| Prices | `price_fetcher.py`, `/api/prices/*` — yfinance is a stub module in tests; nothing about fetching is certified |
|
||||
| Scheduler | `services/scheduler.py` never starts in tests; midnight job untested |
|
||||
| Routers | Error mapping beyond the handful of 422/404 cases above |
|
||||
| **Frontend** | Zero automated tests of any kind — see §6 |
|
||||
| CLI | Most subcommands beyond the 3 wiring smoke tests; all human (non-`--json`) output |
|
||||
|
||||
Consult this map before trusting "tests pass" as evidence for your specific change.
|
||||
|
||||
## 3. Running the tests
|
||||
|
||||
Commands are statically verified against `conftest.py`/`pyproject.toml` — this
|
||||
authoring environment has no pip/pytest, so they were not executed here.
|
||||
|
||||
```bash
|
||||
cd /home/markuseisl/Documents/git/fin
|
||||
python -m pytest cli/tests -q
|
||||
```
|
||||
|
||||
- Working directory does not matter for imports: `conftest.py` computes
|
||||
`REPO_ROOT` from its own path and inserts `api/` and `cli/` into `sys.path`.
|
||||
Repo root is still the conventional place to run from.
|
||||
- `conftest.py` sets `DATABASE_URL` to `sqlite:///<mkdtemp>/test.db` **before**
|
||||
importing `app.*`, and stubs `yfinance` as an empty module. Never import
|
||||
`app.database` in a new conftest/helper before that env var is set.
|
||||
- Green run as of 2026-07-05: `25 passed` (dots, no warnings-as-errors config).
|
||||
- Requires `pytest`, `fastapi`, plus the api/ and cli/ dependencies installed.
|
||||
Environment failures (missing modules, wrong Python) → `fin-build-and-env`.
|
||||
- The suite never needs `FIN_API_URL`/`FIN_API_TOKEN`; the autouse fixture
|
||||
deletes `FIN_API_TOKEN` so a token in your shell can't poison auth tests.
|
||||
|
||||
## 4. Adding a test — checklist and skeleton
|
||||
|
||||
Where: a new `test_*.py` in `/home/markuseisl/Documents/git/fin/cli/tests/`
|
||||
(or a new class in `test_cli.py` if it fits an existing area). pytest discovers
|
||||
it automatically; `conftest.py` fixtures apply.
|
||||
|
||||
Checklist:
|
||||
|
||||
- [ ] Use the `client` fixture (a `FinClient` over in-process `TestClient`) or
|
||||
`make_client(token=...)` when testing auth behavior.
|
||||
- [ ] Rely on the autouse `clean_db` fixture: it `drop_all` + `create_all`s the
|
||||
throwaway DB before **every** test — tests start empty and must not
|
||||
depend on each other's data or ordering.
|
||||
- [ ] Exercise multi-step flows through the API surface only: create account →
|
||||
transact → assert balances via `client.get_net_worth()` /
|
||||
`client.list_transactions()` / `client.get_spending_month()`.
|
||||
- [ ] Do NOT import from `app.*` internals when the API surface suffices —
|
||||
these are e2e tests of the deployed contract. (Direct crud-layer tests
|
||||
are a different layer with different rules; that belongs to
|
||||
`fin-test-coverage-campaign`.)
|
||||
- [ ] No live prices, ever: `yfinance` is a stub. A test that triggers a real
|
||||
fetch (e.g. `client.refresh_prices()` reaching yfinance) will crash or
|
||||
hang — if you need a price, insert into `price_cache` via whatever
|
||||
seeding mechanism the campaign skill establishes, or don't test it here.
|
||||
- [ ] Write the expected numbers as literals with a comment stating the
|
||||
mechanism (see `test_fee_reduces_balance` citing invariant 5).
|
||||
- [ ] Assert on `FinClientError.status_code` for expected failures.
|
||||
|
||||
Worked skeleton (multi-step flow):
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from fin_cli.client import FinClientError
|
||||
|
||||
|
||||
class TestSettlementSplitwiseToBank:
|
||||
def test_splitwise_to_bank_direction(self, client):
|
||||
bank = client.create_account(
|
||||
{"name": "Giro", "type": "bank", "currency": "EUR",
|
||||
"initial_balance": 100.0}
|
||||
)
|
||||
sw = client.create_account(
|
||||
{"name": "Splitwise", "type": "splitwise", "currency": "EUR"}
|
||||
)
|
||||
legs = client.create_settlement(
|
||||
{"bank_account_id": bank["id"], "splitwise_account_id": sw["id"],
|
||||
"amount": 30.0, "date": "2026-06-20",
|
||||
"direction": "splitwise_to_bank"}
|
||||
)
|
||||
by_account = {leg["account_id"]: leg for leg in legs}
|
||||
# money moves toward the bank: bank +30, splitwise -30
|
||||
assert by_account[bank["id"]]["amount"] == 30.0
|
||||
assert by_account[sw["id"]]["amount"] == -30.0
|
||||
# settlements are not spending
|
||||
assert client.get_spending_month("2026-06")["total"] == pytest.approx(0.0)
|
||||
# net worth: 100 opening + 30 in - 30 out across own accounts = 100
|
||||
assert client.get_net_worth()["total_eur"] == pytest.approx(100.0)
|
||||
```
|
||||
|
||||
(Predict the numbers in comments BEFORE running — §1 discipline. Verify the
|
||||
expected sign convention against the router/crud before committing; the
|
||||
skeleton's ±30 is the hypothesis, the first run is the experiment.)
|
||||
|
||||
## 5. Acceptance thresholds and regression discipline
|
||||
|
||||
- **Every fixed bug gains a test that reproduces it** — written to fail on the
|
||||
pre-fix code. History shows why: the bug-hunt commits (`bc256f6`, `e36d719`)
|
||||
fixed accounting bugs without adding regression tests, and bugs recurred in
|
||||
families (spending-inflation class hit twice). Details: `fin-failure-archaeology`.
|
||||
- Tests must be **deterministic**: no network (yfinance stubbed, no httpx to
|
||||
the outside), no clock assumptions beyond the app's own local-date helpers —
|
||||
pass explicit `date` strings, never "today", when asserting on month buckets.
|
||||
- **Prod DB is never a fixture.** Do not point `DATABASE_URL` or `FIN_API_URL`
|
||||
at the live instance from a test, and never copy `/data/fin.db` into the test
|
||||
tree. Experiments on real data happen on a copied DB via
|
||||
`fin-diagnostics-and-tooling` / `fin-ledger-analysis-toolkit`, read-only.
|
||||
- A change that touches crud.py accounting paths and ships with only level
|
||||
(c)–(e) evidence is not done. Flag it in review.
|
||||
- No flaky-test tolerance: with 25 tests and no CI, a single intermittent
|
||||
failure erodes the only gate the project has. Fix or delete, never skip.
|
||||
|
||||
## 6. Frontend validation — honest current bar
|
||||
|
||||
**Zero frontend tests exist (as of 2026-07-05).** The bar is:
|
||||
|
||||
1. `vite build` (runs `tsc`) must pass — level (d) evidence only.
|
||||
2. Manual walkthrough of the critical flows — level (e).
|
||||
|
||||
Manual QA script for a frontend-touching change (worked numbers; exact
|
||||
accounting rules in `fin-accounting-reference`). Do this on a **dev/throwaway
|
||||
stack**, not prod. Start state: one bank account "Giro" with balance 1,000.00 €.
|
||||
|
||||
| # | Action (QuickAdd unless noted) | Expected result |
|
||||
|---|---|---|
|
||||
| 1 | Expense −50.00, fee 2.00, category Dining | Giro 948.00 (Δ −52: amount − fee) |
|
||||
| 2 | Spending page, current month | Total 50.00, Dining 50.00 (fee excluded from spending) |
|
||||
| 3 | Create second bank account "Savings" (0.00) | Net worth unchanged: 948.00 |
|
||||
| 4 | Transfer 100.00 Giro → Savings | Giro 848.00, Savings 100.00, net worth still 948.00, two linked rows on Transactions page |
|
||||
| 5 | Split expense: total 100.00, partner share 40.00, receivable acct | Giro 748.00 (−100), Receivable +40.00, Spending +60.00 → 110.00, net worth 888.00 |
|
||||
| 6 | Splitwise "I paid": total 100.00, partner share 45.00 | Giro 648.00, Splitwise +45.00, Spending +55.00 → 165.00, net worth 833.00 |
|
||||
| 7 | Portfolio: buy 2 AAPL @ 100.00 USD on investment acct | Holding row AAPL qty 2, avg cost 100.00 USD; cash leg only if funded-buy used |
|
||||
| 8 | RSU: vest a grant tranche | Holding appears/increments on the rsu account; grant's remaining unvested decreases |
|
||||
| 9 | Refresh prices (Settings) then Dashboard | Portfolio EUR value = qty × cached USD price ÷ EUR/USD rate; no NaN/0 flash |
|
||||
| 10 | Each add above fired the refresh event | Dashboard/Transactions updated without manual reload (`fin:transaction-added` bus intact) |
|
||||
|
||||
Any delta mismatch = stop, apply §1: write the mechanism you expected, find
|
||||
the one mechanism explaining all ten observations.
|
||||
|
||||
## 7. Idea lifecycle (methodology-lite)
|
||||
|
||||
Every "I think X is broken / X would fix it" follows this pipeline — no
|
||||
skipping stages:
|
||||
|
||||
1. **Hunch** — state it in one falsifiable sentence.
|
||||
2. **Discriminating check defined** — pick the observation that distinguishes
|
||||
your hypothesis from the nearest alternative, and write the predicted
|
||||
numbers down first.
|
||||
3. **Experiment** — run it on a synthetic DB (test fixtures) or a copied DB
|
||||
(`fin-diagnostics-and-tooling`). Never prod.
|
||||
4. **Test written** — the discriminating check becomes a `cli/tests/` test
|
||||
(fails pre-fix, passes post-fix).
|
||||
5. **Change lands** — through `fin-change-control` gates.
|
||||
6. **Archive** — if the investigation settled a question about how fin behaves
|
||||
(or used to misbehave), record it in `fin-failure-archaeology` so the next
|
||||
session doesn't re-derive it.
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
Ground truth as of 2026-07-05. Re-verify before relying on volatile facts:
|
||||
|
||||
- Test count / inventory: `grep -c "def test_" cli/tests/test_cli.py` (25) and
|
||||
`ls cli/tests/` (only `conftest.py` + `test_cli.py`) — re-read new files and
|
||||
update §2 tables when either changes.
|
||||
- Fixture semantics: re-read `cli/tests/conftest.py` (autouse `clean_db`,
|
||||
`make_client`, yfinance stub, `DATABASE_URL` set pre-import).
|
||||
- Run command: confirm against `cli/pyproject.toml` and `fin-build-and-env`
|
||||
(commands here were statically derived, not executed — no pip in the
|
||||
authoring sandbox).
|
||||
- Frontend still testless: `ls web/src/**/*.test.* web/vitest* 2>/dev/null`
|
||||
should return nothing; if it doesn't, §6 is stale.
|
||||
- Coverage gaps: cross-check §2 "NOT certified" against
|
||||
`fin-test-coverage-campaign` progress before repeating its claims.
|
||||
@@ -5,6 +5,11 @@ description: Interact with the user's fin personal-finance app (net worth, accou
|
||||
|
||||
# fin CLI
|
||||
|
||||
For MCP-capable agents: `fin-mcp` (install: `pip install -e 'cli/[mcp]'`) exposes the
|
||||
full fin API as structured MCP tools without subprocess overhead. Read-only by default;
|
||||
set `FIN_MCP_ALLOW_WRITES=1` for write access. The CLI (`fin`) remains the fallback for
|
||||
environments without the `mcp` extra or for one-off interactive queries.
|
||||
|
||||
`fin` talks HTTP to the fin API. Setup (usually already exported):
|
||||
|
||||
```bash
|
||||
|
||||
+2
-1
@@ -2,4 +2,5 @@
|
||||
**/__pycache__/
|
||||
api/fin.db
|
||||
web/dist/
|
||||
web/node_modules/
|
||||
web/node_modules/
|
||||
.claude/settings.local.json
|
||||
@@ -52,6 +52,16 @@ cli/
|
||||
|
||||
`fin` is a typer CLI that talks HTTP to the deployed API. Configure with `FIN_API_URL` (required, no default) and `FIN_API_TOKEN` (only if the API enforces auth — the API checks it only when its own `FIN_API_TOKEN` env var is set; `/api/health` is always open). **Warning**: enabling `FIN_API_TOKEN` on the API also locks out the React frontend — `web/src/api/client.ts` sends no `Authorization` header — until nginx injects the header or the frontend learns the token. Install with `pip install -e cli/`; every command supports `--json`. `fin_cli/client.py` is kept free of typer/rich imports so a future MCP server can reuse it as-is. Agent-facing usage docs live in `.claude/skills/fin/SKILL.md`.
|
||||
|
||||
`cli/fin_mcp/` wraps `FinClient` as an MCP server (`fin-mcp` console script), exposing the full API as 39 tools. Install the extra with `pip install -e 'cli/[mcp]'`.
|
||||
|
||||
- **Mode**: read-only (15 tools) by default; `FIN_MCP_ALLOW_WRITES=1` adds the 24 write tools.
|
||||
- **Name resolution**: `find_account` / `find_category` map a name to the integer id every other tool takes (exact match wins, else substring). Accounts and categories are also published as the resources `fin://accounts` and `fin://categories`. There is no `get_openapi_spec` tool — the tools already cover the API and the spec is too large to be worth a tool slot.
|
||||
- **Transport**: `FIN_MCP_TRANSPORT=stdio` (default, for clients that spawn the process) or `streamable-http`, which binds `FIN_MCP_HOST:FIN_MCP_PORT` and serves MCP at `FIN_MCP_PATH` (default `/mcp`). HTTP mode sets `stateless_http` so no session affinity is required.
|
||||
- **Deployment**: its own image (`cli/Dockerfile`) and its own Deployment + ClusterIP Service in k8s — it is not mounted into the API app. Reference manifest: `deploy/k8s/fin-mcp.yaml`. In-cluster endpoint is `http://fin-mcp.apps.svc.cluster.local:8000/mcp`.
|
||||
- **Tool params are typed**: enum-valued arguments use `Literal`, so the MCP client rejects bad values before the call. In particular `create_transaction` accepts only `expense`/`income` — the API itself accepts all six transaction types, and writing a `transfer` row through it would create an orphan leg with no `linked_account_id` (invariant #1). Two-leg writes must go through `create_transfer` / `create_settlement` / `create_split_expense` / `create_splitwise_i_paid`.
|
||||
|
||||
**Security**: `fin-mcp` has no auth of its own — it inherits whatever `FinClient` uses. With writes enabled it must stay unreachable from outside the cluster (no HTTPRoute) unless `FIN_API_TOKEN` is set on both the API and the MCP server.
|
||||
|
||||
## Backend conventions
|
||||
|
||||
- **Sync only**: SQLAlchemy is configured with a synchronous engine (`create_engine`, `SessionLocal`). Never introduce `async def` endpoints or `async with Session`. FastAPI handles the thread pool.
|
||||
@@ -73,13 +83,17 @@ cli/
|
||||
|
||||
`Transaction` belongs to one account and has a `type`: `expense`, `income`, `transfer`, `splitwise_pay`, `splitwise_receive`, or `settlement`. Transfers and settlements always produce two linked rows — a debit on one account and a credit on another — connected via `linked_account_id`. The `fee` column is included in balance calculations (`amount - fee`). For Splitwise pay transactions, `splitwise_partner_share` records the partner's portion (used in spending calculations to show only the user's net share).
|
||||
|
||||
**Idempotency**: every write that moves money accepts an optional `client_ref` — `POST /api/transactions`, `/batch`, `/transfer` and `/settlement`. Every row of one write carries the same key, and the creators check `crud.find_by_client_ref` before writing, so retrying a timed-out request replays the original rows instead of double-booking. The index is deliberately **not** unique (a transfer writes two rows under one key); duplicate suppression is a check-then-write, which is sound because SQLite serialises writers and the scheduler never writes transactions. A NULL `client_ref` means "no idempotency requested" and never collides.
|
||||
|
||||
`POST /api/transactions/batch` writes several single-leg rows sharing one account, date and currency in a single commit — all rows land or none do. It is the one write endpoint whose items carry **positive** amounts: the sign is derived server-side from the batch `type`, which makes a mixed-sign batch unrepresentable. Its `type` is restricted to `expense`/`income` for the same reason the MCP `create_transaction` tool is — a batched `transfer` would emit orphan legs (invariant #1). The UI entry point is the `Batch Expenses` QuickAdd flow.
|
||||
|
||||
`StockTransaction` records buy/sell/rsu_vest events and drives `Holding` state. Holdings are never edited directly — they are recomputed from the full stock transaction history on any delete, and incrementally updated on insert.
|
||||
|
||||
`PriceCache` holds the latest fetched price per symbol (one row, upserted). `NetWorthSnapshot` holds one row per calendar day with the total EUR value and a JSON breakdown by account.
|
||||
|
||||
## Key invariants — never break these
|
||||
|
||||
1. **Atomic two-leg writes**: `create_transfer` and `create_settlement` in crud.py use `db.add_all([leg1, leg2]); db.commit()`. Both legs must be written in a single transaction. Deleting one leg of a transfer without the other corrupts balances.
|
||||
1. **Atomic two-leg writes**: `create_transfer` and `create_settlement` in crud.py use `db.add_all([leg1, leg2]); db.commit()`. Both legs must be written in a single transaction. Deleting one leg of a transfer without the other corrupts balances. This is now enforced at the edge too: `TransactionCreate` rejects `type` of `transfer`/`settlement` and rejects a caller-supplied `linked_account_id`, so `POST /api/transactions` can no longer write half a pair. Keep `TransactionBase` wide — reads must still describe historical rows of every type.
|
||||
2. **Holding recompute on delete**: `delete_stock_transaction` calls `recompute_holding_from_stock_transactions` after the delete, before `db.commit()`. Skipping this leaves holdings inconsistent with the transaction log.
|
||||
3. **WAL mode**: `PRAGMA journal_mode=WAL` is applied on every new connection (both the main engine and the scheduler's private engine). The scheduler runs in a background thread and must not share the request-scoped session.
|
||||
4. **Non-root Docker user**: The API Dockerfile creates user `app` (uid 1000) and runs as that user. Do not revert to root.
|
||||
|
||||
@@ -30,8 +30,88 @@ Set in `docker-compose.yml` under the `api` service:
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | `sqlite:////data/fin.db` | SQLite path inside the container |
|
||||
| `FIN_API_TOKEN` | _(unset — open)_ | Bearer token the API requires; leave unset for no auth |
|
||||
|
||||
Data is persisted in a named Docker volume (`sqlite_data`). No other configuration is required.
|
||||
CLI / MCP environment variables (set in the shell or MCP client config):
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `FIN_API_URL` | _(required)_ | Base URL of the fin API, e.g. `https://fin.example.com` |
|
||||
| `FIN_API_TOKEN` | _(unset)_ | Bearer token for the API (only if the API enforces auth) |
|
||||
| `FIN_MCP_ALLOW_WRITES` | _(unset — read-only)_ | Set to `1` to enable the 24 write tools in the MCP server |
|
||||
| `FIN_MCP_TRANSPORT` | `stdio` | `stdio`, or `streamable-http` to run as a network service |
|
||||
| `FIN_MCP_HOST` / `FIN_MCP_PORT` | `0.0.0.0` / `8000` | Bind address, `streamable-http` only |
|
||||
| `FIN_MCP_PATH` | `/mcp` | Path MCP is served at, `streamable-http` only |
|
||||
|
||||
Data is persisted in a named Docker volume (`sqlite_data`).
|
||||
|
||||
## MCP server
|
||||
|
||||
`fin-mcp` exposes the full fin API as structured tools. Default mode is
|
||||
**read-only** (15 tools). Set `FIN_MCP_ALLOW_WRITES=1` to additionally enable
|
||||
24 write tools (create/update/delete transactions, accounts, stock trades, etc.).
|
||||
|
||||
Enum-valued arguments are typed, so the client rejects invalid values before the
|
||||
call. `create_transaction` deliberately accepts only `expense` and `income`:
|
||||
two-leg writes must go through `create_transfer`, `create_settlement`,
|
||||
`create_split_expense` or `create_splitwise_i_paid`, or the ledger ends up with
|
||||
an orphan leg.
|
||||
|
||||
Use `find_account` / `find_category` to turn a name into the integer id the
|
||||
other tools take. Both lists are also published as the resources
|
||||
`fin://accounts` and `fin://categories`.
|
||||
|
||||
`create_transaction`, `create_transfer` and `create_settlement` accept an
|
||||
optional `client_ref`. Retrying a failed call with the same key returns the
|
||||
write that already happened instead of booking it twice — worth setting on any
|
||||
write an agent might have to retry.
|
||||
|
||||
**Transports.** `stdio` (default) suits clients that spawn the process.
|
||||
`streamable-http` runs it as a service — it binds `FIN_MCP_HOST:FIN_MCP_PORT`
|
||||
and serves MCP at `FIN_MCP_PATH`:
|
||||
|
||||
```bash
|
||||
FIN_API_URL=http://localhost:8000 FIN_MCP_TRANSPORT=streamable-http fin-mcp
|
||||
```
|
||||
|
||||
In Kubernetes it runs as its own Deployment and ClusterIP Service — see
|
||||
`deploy/k8s/fin-mcp.yaml`. That manifest has no HTTPRoute on purpose: the MCP
|
||||
server has no auth of its own, so with writes enabled it must stay unreachable
|
||||
from outside the cluster unless `FIN_API_TOKEN` is set on both the API and the
|
||||
MCP server.
|
||||
|
||||
**Install:**
|
||||
|
||||
```bash
|
||||
pip install -e 'cli/[mcp]'
|
||||
```
|
||||
|
||||
**Run manually:**
|
||||
|
||||
```bash
|
||||
export FIN_API_URL=https://fin.example.com
|
||||
fin-mcp # stdio; connect via your MCP client
|
||||
```
|
||||
|
||||
**Claude Desktop / Claude Code config** (`~/.claude/mcp.json` or equivalent):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"fin": {
|
||||
"command": "fin-mcp",
|
||||
"env": {
|
||||
"FIN_API_URL": "https://fin.example.com",
|
||||
"FIN_API_TOKEN": "...",
|
||||
"FIN_MCP_ALLOW_WRITES": "0"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Remove `FIN_API_TOKEN` if the API runs without auth. Change `FIN_MCP_ALLOW_WRITES` to `"1"`
|
||||
to permit write operations (confirm amounts, accounts, and dates with the user before each call).
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
"""add client_ref idempotency key to transactions
|
||||
|
||||
Revision ID: i9d0e1f2a3b4
|
||||
Revises: h8c9d0e1f2a3
|
||||
Create Date: 2026-08-22
|
||||
|
||||
Adds a nullable, caller-supplied idempotency key. Every row written by one
|
||||
request carries the same value, so a retry after an ambiguous failure returns
|
||||
the existing rows instead of booking them twice.
|
||||
|
||||
The index is deliberately NOT unique: a transfer writes two legs and a batch
|
||||
writes many, all sharing one key. Duplicate suppression happens as a
|
||||
check-then-write inside the crud creators. Backward-safe — the column is
|
||||
nullable and every existing row keeps NULL, which the creators treat as
|
||||
"no idempotency requested".
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "i9d0e1f2a3b4"
|
||||
down_revision: Union[str, None] = "h8c9d0e1f2a3"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("transactions") as batch_op:
|
||||
batch_op.add_column(sa.Column("client_ref", sa.Text(), nullable=True))
|
||||
op.create_index("ix_transactions_client_ref", "transactions", ["client_ref"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_transactions_client_ref", table_name="transactions")
|
||||
with op.batch_alter_table("transactions") as batch_op:
|
||||
batch_op.drop_column("client_ref")
|
||||
@@ -149,9 +149,29 @@ def get_transaction(db: Session, transaction_id: int) -> models.Transaction | No
|
||||
)
|
||||
|
||||
|
||||
def find_by_client_ref(db: Session, client_ref: str | None) -> list[models.Transaction]:
|
||||
"""Return the rows a previous request already wrote under this key.
|
||||
|
||||
Empty when no key was supplied or nothing matches. Every creator that
|
||||
accepts a client_ref calls this first, so retrying an ambiguous request
|
||||
replays the original result instead of double-booking.
|
||||
"""
|
||||
if not client_ref:
|
||||
return []
|
||||
return (
|
||||
db.query(models.Transaction)
|
||||
.filter(models.Transaction.client_ref == client_ref)
|
||||
.order_by(models.Transaction.id)
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def create_transaction(
|
||||
db: Session, transaction: schemas.TransactionCreate
|
||||
) -> models.Transaction:
|
||||
existing = find_by_client_ref(db, transaction.client_ref)
|
||||
if existing:
|
||||
return existing[0]
|
||||
db_transaction = models.Transaction(**transaction.model_dump())
|
||||
db.add(db_transaction)
|
||||
db.commit()
|
||||
@@ -159,6 +179,41 @@ def create_transaction(
|
||||
return db_transaction
|
||||
|
||||
|
||||
def create_transactions_batch(
|
||||
db: Session, req: schemas.BatchTransactionRequest
|
||||
) -> list[models.Transaction]:
|
||||
"""Write every row of a batch in a single commit — all land or none do.
|
||||
|
||||
Items carry positive amounts and the sign is derived from req.type, so a
|
||||
batch cannot mix money in with money out. Only single-leg types are
|
||||
accepted (see BatchTransactionRequest), so this never produces the orphan
|
||||
legs that invariant #1 exists to prevent.
|
||||
"""
|
||||
existing = find_by_client_ref(db, req.client_ref)
|
||||
if existing:
|
||||
return existing
|
||||
sign = -1 if req.type == "expense" else 1
|
||||
rows = [
|
||||
models.Transaction(
|
||||
date=req.date,
|
||||
account_id=req.account_id,
|
||||
amount=sign * item.amount,
|
||||
currency=req.currency,
|
||||
type=req.type,
|
||||
category_id=item.category_id,
|
||||
description=item.description,
|
||||
fee=item.fee,
|
||||
client_ref=req.client_ref,
|
||||
)
|
||||
for item in req.items
|
||||
]
|
||||
db.add_all(rows)
|
||||
db.commit()
|
||||
for row in rows:
|
||||
db.refresh(row)
|
||||
return rows
|
||||
|
||||
|
||||
def update_transaction(
|
||||
db: Session, transaction_id: int, txn: schemas.TransactionUpdate
|
||||
) -> models.Transaction | None:
|
||||
@@ -234,6 +289,9 @@ def create_transfer(
|
||||
db: Session, req: schemas.TransferRequest
|
||||
) -> list[models.Transaction]:
|
||||
"""Write both legs of a transfer atomically."""
|
||||
existing = find_by_client_ref(db, req.client_ref)
|
||||
if existing:
|
||||
return existing
|
||||
dest_amount = req.to_amount if req.to_amount is not None else req.amount
|
||||
dest_currency = req.to_currency if req.to_currency is not None else req.currency
|
||||
from_leg = models.Transaction(
|
||||
@@ -245,6 +303,7 @@ def create_transfer(
|
||||
description=req.description,
|
||||
linked_account_id=req.to_account_id,
|
||||
fee=req.fee,
|
||||
client_ref=req.client_ref,
|
||||
)
|
||||
to_leg = models.Transaction(
|
||||
date=req.date,
|
||||
@@ -255,6 +314,7 @@ def create_transfer(
|
||||
description=req.description,
|
||||
linked_account_id=req.from_account_id,
|
||||
fee=0,
|
||||
client_ref=req.client_ref,
|
||||
)
|
||||
db.add_all([from_leg, to_leg])
|
||||
db.flush()
|
||||
@@ -274,6 +334,9 @@ def create_settlement(
|
||||
direction='splitwise_to_bank': she paid me — bank +amount, splitwise -amount
|
||||
direction='bank_to_splitwise': i paid her — bank -amount, splitwise +amount
|
||||
"""
|
||||
existing = find_by_client_ref(db, req.client_ref)
|
||||
if existing:
|
||||
return existing
|
||||
if req.direction == "splitwise_to_bank":
|
||||
bank_amount = req.amount
|
||||
splitwise_amount = -req.amount
|
||||
@@ -290,6 +353,7 @@ def create_settlement(
|
||||
description="Splitwise settlement",
|
||||
linked_account_id=req.splitwise_account_id,
|
||||
fee=0,
|
||||
client_ref=req.client_ref,
|
||||
)
|
||||
splitwise_leg = models.Transaction(
|
||||
date=req.date,
|
||||
@@ -300,6 +364,7 @@ def create_settlement(
|
||||
description="Splitwise settlement",
|
||||
linked_account_id=req.bank_account_id,
|
||||
fee=0,
|
||||
client_ref=req.client_ref,
|
||||
)
|
||||
db.add_all([bank_leg, splitwise_leg])
|
||||
db.flush()
|
||||
|
||||
@@ -68,6 +68,13 @@ class Transaction(Base):
|
||||
splitwise_partner_share = Column(Float, nullable=True)
|
||||
linked_account_id = Column(Integer, ForeignKey("accounts.id"), nullable=True)
|
||||
linked_transaction_id = Column(Integer, nullable=True)
|
||||
# Caller-supplied idempotency key. Shared by every row of one write, so a
|
||||
# retried request returns the rows it already made instead of double-booking.
|
||||
# Indexed but NOT unique: a two-leg or batched write puts the same value on
|
||||
# several rows. Duplicate suppression is a check-then-write inside the
|
||||
# creator, which is sound here because SQLite serialises writers and the
|
||||
# scheduler never writes transactions.
|
||||
client_ref = Column(Text, nullable=True, index=True)
|
||||
created_at = Column(DateTime, nullable=False, default=datetime.utcnow)
|
||||
|
||||
account = relationship("Account", back_populates="transactions", foreign_keys=[account_id])
|
||||
|
||||
@@ -28,6 +28,22 @@ def create_transaction(
|
||||
return crud.create_transaction(db, transaction)
|
||||
|
||||
|
||||
@router.post("/batch", response_model=list[schemas.TransactionRead], status_code=201)
|
||||
def create_transactions_batch(
|
||||
req: schemas.BatchTransactionRequest, db: Session = Depends(get_db)
|
||||
):
|
||||
if crud.get_account(db, req.account_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Account not found")
|
||||
# Validate category ids up front. The FK would reject them anyway, but as
|
||||
# an IntegrityError on commit — a 500 that does not say which row is wrong.
|
||||
for i, item in enumerate(req.items):
|
||||
if item.category_id is not None and crud.get_category(db, item.category_id) is None:
|
||||
raise HTTPException(
|
||||
status_code=422, detail=f"Row {i + 1}: category {item.category_id} not found"
|
||||
)
|
||||
return crud.create_transactions_batch(db, req)
|
||||
|
||||
|
||||
@router.put("/{transaction_id}", response_model=schemas.TransactionRead)
|
||||
def update_transaction(transaction_id: int, txn: schemas.TransactionUpdate, db: Session = Depends(get_db)):
|
||||
existing = crud.get_transaction(db, transaction_id)
|
||||
|
||||
+52
-2
@@ -1,7 +1,7 @@
|
||||
from datetime import date, datetime
|
||||
from typing import Literal, Optional
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -81,10 +81,32 @@ class TransactionBase(BaseModel):
|
||||
exchange_rate_used: Optional[float] = None
|
||||
splitwise_partner_share: Optional[float] = None
|
||||
linked_account_id: Optional[int] = None
|
||||
client_ref: Optional[str] = Field(default=None, max_length=100)
|
||||
|
||||
|
||||
class TransactionCreate(TransactionBase):
|
||||
pass
|
||||
"""A single, standalone transaction.
|
||||
|
||||
Narrower than TransactionBase, which must keep describing every historical
|
||||
row for reads. `transfer` and `settlement` always come in linked pairs, so
|
||||
one written here would be an orphan leg — the balance corruption invariant
|
||||
#1 exists to prevent. Both have dedicated atomic endpoints. `splitwise_pay`
|
||||
stays allowed: the two-leg creator uses that type, but a standalone row
|
||||
carrying splitwise_partner_share is also a legitimate record and is what
|
||||
`fin tx add --type splitwise_pay --partner-share` writes.
|
||||
"""
|
||||
|
||||
type: Literal["expense", "income", "splitwise_pay", "splitwise_receive"]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _reject_manual_linkage(self):
|
||||
if self.linked_account_id is not None:
|
||||
raise ValueError(
|
||||
"linked_account_id cannot be set on a standalone transaction — "
|
||||
"linked pairs are written by /transfer, /settlement, "
|
||||
"/split-expense and /splitwise-i-paid, which set both sides atomically"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class TransactionUpdate(BaseModel):
|
||||
@@ -95,6 +117,7 @@ class TransactionUpdate(BaseModel):
|
||||
|
||||
|
||||
class TransferRequest(BaseModel):
|
||||
client_ref: Optional[str] = Field(default=None, max_length=100)
|
||||
from_account_id: int
|
||||
to_account_id: int
|
||||
amount: float
|
||||
@@ -107,6 +130,7 @@ class TransferRequest(BaseModel):
|
||||
|
||||
|
||||
class SettlementRequest(BaseModel):
|
||||
client_ref: Optional[str] = Field(default=None, max_length=100)
|
||||
bank_account_id: int
|
||||
splitwise_account_id: int
|
||||
amount: float
|
||||
@@ -137,6 +161,32 @@ class SplitwiseIPaidRequest(BaseModel):
|
||||
description: str = "Splitwise: I paid"
|
||||
|
||||
|
||||
class BatchTransactionItem(BaseModel):
|
||||
"""One row of a batch. Amount is positive; the sign comes from the batch type."""
|
||||
|
||||
amount: float = Field(gt=0)
|
||||
category_id: Optional[int] = None
|
||||
description: Optional[str] = Field(default=None, max_length=500)
|
||||
fee: float = 0
|
||||
|
||||
|
||||
class BatchTransactionRequest(BaseModel):
|
||||
"""Several single-leg transactions sharing an account, date and currency.
|
||||
|
||||
`type` is restricted to the single-leg kinds. `transfer` and `settlement`
|
||||
always need two linked legs, and emitting them here would create orphans
|
||||
that corrupt balances (CLAUDE.md invariant #1) — those go through
|
||||
/transfer and /settlement instead.
|
||||
"""
|
||||
|
||||
account_id: int
|
||||
date: date
|
||||
currency: str = "EUR"
|
||||
type: Literal["expense", "income"] = "expense"
|
||||
client_ref: Optional[str] = Field(default=None, max_length=100)
|
||||
items: list[BatchTransactionItem] = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class TransactionRead(TransactionBase):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# fin_cli/client.py is the whole dependency surface; the mcp extra pulls the
|
||||
# server SDK. No build toolchain is needed, so this stays a single stage.
|
||||
COPY pyproject.toml ./
|
||||
COPY fin_cli/ ./fin_cli/
|
||||
COPY fin_mcp/ ./fin_mcp/
|
||||
|
||||
RUN pip install --no-cache-dir '.[mcp]'
|
||||
|
||||
RUN useradd -m -u 1000 app
|
||||
USER app
|
||||
|
||||
# Served over the network rather than spawned by a local client.
|
||||
ENV FIN_MCP_TRANSPORT=streamable-http \
|
||||
FIN_MCP_HOST=0.0.0.0 \
|
||||
FIN_MCP_PORT=8000 \
|
||||
FIN_MCP_PATH=/mcp
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["fin-mcp"]
|
||||
@@ -169,6 +169,9 @@ class FinClient:
|
||||
def create_settlement(self, data: dict[str, Any]) -> Any:
|
||||
return self._post("/api/transactions/settlement", json=data)
|
||||
|
||||
def create_transactions_batch(self, data: dict[str, Any]) -> Any:
|
||||
return self._post("/api/transactions/batch", json=data)
|
||||
|
||||
def create_split_expense(self, data: dict[str, Any]) -> Any:
|
||||
return self._post("/api/transactions/split-expense", json=data)
|
||||
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
"""fin-mcp — MCP server wrapping FinClient for the fin personal finance app."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,4 @@
|
||||
from .server import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large.
Load diff
+5
-1
@@ -9,12 +9,16 @@ dependencies = [
|
||||
"rich>=13",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
mcp = ["mcp[cli]>=1.10,<2"]
|
||||
|
||||
[project.scripts]
|
||||
fin = "fin_cli.main:app"
|
||||
fin-mcp = "fin_mcp.server:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["fin_cli"]
|
||||
packages = ["fin_cli", "fin_mcp"]
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Batch transaction entry — POST /api/transactions/batch.
|
||||
|
||||
Drives the real app through FinClient, same as the rest of the e2e suite.
|
||||
The property that earns this endpoint over a client-side loop is atomicity,
|
||||
so most of these tests assert that a rejected batch writes *nothing*.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from fin_cli.client import FinClientError
|
||||
|
||||
|
||||
def _bank(client, name="Checking"):
|
||||
return client.create_account({"name": name, "type": "bank", "currency": "EUR"})
|
||||
|
||||
|
||||
def _category(client, name="Groceries"):
|
||||
return client.create_category({"name": name, "color": "#22c55e", "icon": "X"})
|
||||
|
||||
|
||||
def _batch(account_id, items, **overrides):
|
||||
payload = {"account_id": account_id, "date": "2026-08-22", "items": items}
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
|
||||
class TestBatchSuccess:
|
||||
def test_writes_every_row_in_one_call(self, client):
|
||||
acct = _bank(client)
|
||||
cat = _category(client)
|
||||
|
||||
created = client.create_transactions_batch(
|
||||
_batch(
|
||||
acct["id"],
|
||||
[
|
||||
{"amount": 12.40, "category_id": cat["id"], "description": "Rewe"},
|
||||
{"amount": 4.20, "description": "coffee"},
|
||||
{"amount": 31.00, "category_id": cat["id"]},
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
assert len(created) == 3
|
||||
stored = client.list_transactions(account_id=acct["id"])
|
||||
assert len(stored) == 3
|
||||
assert {t["description"] for t in stored} == {"Rewe", "coffee", None}
|
||||
|
||||
def test_expense_amounts_are_stored_negative(self, client):
|
||||
"""Items are sent positive; the sign is derived from the batch type."""
|
||||
acct = _bank(client)
|
||||
|
||||
created = client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 12.40}, {"amount": 4.20}])
|
||||
)
|
||||
|
||||
assert sorted(t["amount"] for t in created) == [-12.40, -4.20]
|
||||
assert all(t["type"] == "expense" for t in created)
|
||||
|
||||
def test_income_batch_stores_positive(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
created = client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 900.0}], type="income")
|
||||
)
|
||||
|
||||
assert created[0]["amount"] == 900.0
|
||||
assert created[0]["type"] == "income"
|
||||
|
||||
def test_shared_date_and_account_apply_to_every_row(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
created = client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 1.0}, {"amount": 2.0}], date="2026-01-15")
|
||||
)
|
||||
|
||||
assert all(t["date"] == "2026-01-15" for t in created)
|
||||
assert all(t["account_id"] == acct["id"] for t in created)
|
||||
|
||||
def test_batch_moves_the_account_balance_by_the_total(self, client):
|
||||
"""Balance is SUM(amount - fee) over the account (CLAUDE.md invariant #5)."""
|
||||
acct = client.create_account(
|
||||
{"name": "Checking", "type": "bank", "currency": "EUR", "initial_balance": 100.0}
|
||||
)
|
||||
|
||||
client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 12.40}, {"amount": 4.20}, {"amount": 31.00}])
|
||||
)
|
||||
|
||||
txns = client.list_transactions(account_id=acct["id"])
|
||||
balance = sum(t["amount"] - t["fee"] for t in txns)
|
||||
assert balance == pytest.approx(100.0 - 47.60)
|
||||
|
||||
|
||||
class TestBatchIsAtomic:
|
||||
"""A rejected batch must leave the ledger untouched."""
|
||||
|
||||
def test_unknown_category_writes_nothing(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transactions_batch(
|
||||
_batch(
|
||||
acct["id"],
|
||||
[
|
||||
{"amount": 12.40},
|
||||
{"amount": 4.20, "category_id": 99999},
|
||||
{"amount": 31.00},
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
assert exc.value.status_code == 422
|
||||
assert client.list_transactions(account_id=acct["id"]) == []
|
||||
|
||||
def test_unknown_account_writes_nothing(self, client):
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transactions_batch(_batch(99999, [{"amount": 12.40}]))
|
||||
|
||||
assert exc.value.status_code == 404
|
||||
assert client.list_transactions() == []
|
||||
|
||||
def test_one_bad_amount_rejects_the_whole_batch(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 12.40}, {"amount": -5.0}])
|
||||
)
|
||||
|
||||
assert exc.value.status_code == 422
|
||||
assert client.list_transactions(account_id=acct["id"]) == []
|
||||
|
||||
|
||||
class TestBatchInvariantGuards:
|
||||
def test_two_leg_types_are_refused(self, client):
|
||||
"""A batch must never emit a transfer/settlement leg.
|
||||
|
||||
POST /api/transactions accepts all six transaction types, so an
|
||||
unrestricted batch endpoint could write single legs with no
|
||||
linked_account_id — orphans that corrupt balances and that
|
||||
paired-delete can never clean up (CLAUDE.md invariant #1).
|
||||
"""
|
||||
acct = _bank(client)
|
||||
|
||||
for bad_type in ("transfer", "settlement", "splitwise_pay", "splitwise_receive"):
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transactions_batch(
|
||||
_batch(acct["id"], [{"amount": 10.0}], type=bad_type)
|
||||
)
|
||||
assert exc.value.status_code == 422, f"{bad_type} was not rejected"
|
||||
|
||||
assert client.list_transactions(account_id=acct["id"]) == []
|
||||
|
||||
def test_empty_batch_is_refused(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transactions_batch(_batch(acct["id"], []))
|
||||
|
||||
assert exc.value.status_code == 422
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Idempotency keys — retrying a write must not double-book.
|
||||
|
||||
Without these, an agent or a flaky network that retries a timed-out create
|
||||
books the money twice and nothing detects it. A caller-supplied client_ref
|
||||
makes the retry replay the original result instead.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from fin_cli.client import FinClientError
|
||||
|
||||
|
||||
def _bank(client, name="Checking"):
|
||||
return client.create_account({"name": name, "type": "bank", "currency": "EUR"})
|
||||
|
||||
|
||||
def _expense(account_id, client_ref=None, amount=-12.40):
|
||||
data = {
|
||||
"account_id": account_id,
|
||||
"date": "2026-08-22",
|
||||
"amount": amount,
|
||||
"currency": "EUR",
|
||||
"type": "expense",
|
||||
}
|
||||
if client_ref is not None:
|
||||
data["client_ref"] = client_ref
|
||||
return data
|
||||
|
||||
|
||||
class TestSingleCreate:
|
||||
def test_repeat_with_same_key_writes_once(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
first = client.create_transaction(_expense(acct["id"], "trip-001"))
|
||||
second = client.create_transaction(_expense(acct["id"], "trip-001"))
|
||||
|
||||
assert first["id"] == second["id"]
|
||||
assert len(client.list_transactions(account_id=acct["id"])) == 1
|
||||
|
||||
def test_retry_replays_the_original_row_not_the_new_payload(self, client):
|
||||
"""The key identifies the operation; a differing body does not create a second row."""
|
||||
acct = _bank(client)
|
||||
|
||||
first = client.create_transaction(_expense(acct["id"], "trip-002", amount=-12.40))
|
||||
second = client.create_transaction(_expense(acct["id"], "trip-002", amount=-999.00))
|
||||
|
||||
assert second["id"] == first["id"]
|
||||
assert second["amount"] == -12.40
|
||||
assert len(client.list_transactions(account_id=acct["id"])) == 1
|
||||
|
||||
def test_different_keys_write_separately(self, client):
|
||||
acct = _bank(client)
|
||||
|
||||
client.create_transaction(_expense(acct["id"], "trip-003"))
|
||||
client.create_transaction(_expense(acct["id"], "trip-004"))
|
||||
|
||||
assert len(client.list_transactions(account_id=acct["id"])) == 2
|
||||
|
||||
def test_no_key_means_no_deduplication(self, client):
|
||||
"""NULL client_ref must not collide with other NULLs."""
|
||||
acct = _bank(client)
|
||||
|
||||
client.create_transaction(_expense(acct["id"]))
|
||||
client.create_transaction(_expense(acct["id"]))
|
||||
|
||||
assert len(client.list_transactions(account_id=acct["id"])) == 2
|
||||
|
||||
def test_client_ref_is_returned_on_reads(self, client):
|
||||
acct = _bank(client)
|
||||
client.create_transaction(_expense(acct["id"], "trip-005"))
|
||||
|
||||
stored = client.list_transactions(account_id=acct["id"])[0]
|
||||
assert stored["client_ref"] == "trip-005"
|
||||
|
||||
|
||||
class TestMultiLegWrites:
|
||||
def test_repeated_transfer_keeps_two_legs(self, client):
|
||||
a = _bank(client, "A")
|
||||
b = _bank(client, "B")
|
||||
payload = {
|
||||
"from_account_id": a["id"], "to_account_id": b["id"],
|
||||
"amount": 40.0, "currency": "EUR", "date": "2026-08-22",
|
||||
"client_ref": "xfer-001",
|
||||
}
|
||||
|
||||
first = client.create_transfer(payload)
|
||||
second = client.create_transfer(payload)
|
||||
|
||||
assert len(first) == 2 and len(second) == 2
|
||||
assert [t["id"] for t in first] == [t["id"] for t in second]
|
||||
assert len(client.list_transactions()) == 2, "retry created a second pair of legs"
|
||||
|
||||
def test_repeated_settlement_keeps_two_legs(self, client):
|
||||
bank = _bank(client, "Bank")
|
||||
sw = client.create_account({"name": "SW", "type": "splitwise", "currency": "EUR"})
|
||||
payload = {
|
||||
"bank_account_id": bank["id"], "splitwise_account_id": sw["id"],
|
||||
"amount": 25.0, "currency": "EUR", "date": "2026-08-22",
|
||||
"direction": "splitwise_to_bank", "client_ref": "settle-001",
|
||||
}
|
||||
|
||||
first = client.create_settlement(payload)
|
||||
second = client.create_settlement(payload)
|
||||
|
||||
assert [t["id"] for t in first] == [t["id"] for t in second]
|
||||
assert len(client.list_transactions()) == 2
|
||||
|
||||
def test_repeated_batch_writes_rows_once(self, client):
|
||||
acct = _bank(client)
|
||||
payload = {
|
||||
"account_id": acct["id"], "date": "2026-08-22", "client_ref": "batch-001",
|
||||
"items": [{"amount": 12.40}, {"amount": 4.20}, {"amount": 31.00}],
|
||||
}
|
||||
|
||||
first = client.create_transactions_batch(payload)
|
||||
second = client.create_transactions_batch(payload)
|
||||
|
||||
assert len(first) == 3
|
||||
assert [t["id"] for t in first] == [t["id"] for t in second]
|
||||
assert len(client.list_transactions(account_id=acct["id"])) == 3
|
||||
|
||||
def test_every_row_of_a_batch_carries_the_key(self, client):
|
||||
"""The whole group shares one key, which is how the retry finds all of it."""
|
||||
acct = _bank(client)
|
||||
client.create_transactions_batch({
|
||||
"account_id": acct["id"], "date": "2026-08-22", "client_ref": "batch-002",
|
||||
"items": [{"amount": 1.0}, {"amount": 2.0}],
|
||||
})
|
||||
|
||||
stored = client.list_transactions(account_id=acct["id"])
|
||||
assert all(t["client_ref"] == "batch-002" for t in stored)
|
||||
|
||||
|
||||
class TestOrphanLegGuard:
|
||||
"""A1 — the raw create endpoint must not be able to write half a pair."""
|
||||
|
||||
@pytest.mark.parametrize("bad_type", ["transfer", "settlement"])
|
||||
def test_two_leg_types_are_refused(self, client, bad_type):
|
||||
acct = _bank(client)
|
||||
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transaction({
|
||||
"account_id": acct["id"], "date": "2026-08-22", "amount": -10.0,
|
||||
"currency": "EUR", "type": bad_type,
|
||||
})
|
||||
|
||||
assert exc.value.status_code == 422
|
||||
assert client.list_transactions() == []
|
||||
|
||||
def test_single_leg_types_still_work(self, client):
|
||||
"""Narrowing must not break the flows that legitimately post single rows."""
|
||||
acct = _bank(client)
|
||||
sw = client.create_account({"name": "SW", "type": "splitwise", "currency": "EUR"})
|
||||
|
||||
for account_id, t in [
|
||||
(acct["id"], "expense"), (acct["id"], "income"),
|
||||
(sw["id"], "splitwise_receive"), (sw["id"], "splitwise_pay"),
|
||||
]:
|
||||
created = client.create_transaction({
|
||||
"account_id": account_id, "date": "2026-08-22", "amount": -5.0,
|
||||
"currency": "EUR", "type": t,
|
||||
})
|
||||
assert created["type"] == t
|
||||
|
||||
def test_manual_linkage_is_refused(self, client):
|
||||
"""A free-standing linked_account_id would fake half of a linked pair."""
|
||||
a = _bank(client, "A")
|
||||
b = _bank(client, "B")
|
||||
|
||||
with pytest.raises(FinClientError) as exc:
|
||||
client.create_transaction({
|
||||
"account_id": a["id"], "date": "2026-08-22", "amount": -10.0,
|
||||
"currency": "EUR", "type": "expense", "linked_account_id": b["id"],
|
||||
})
|
||||
|
||||
assert exc.value.status_code == 422
|
||||
assert client.list_transactions() == []
|
||||
@@ -0,0 +1,288 @@
|
||||
"""MCP server tests.
|
||||
|
||||
Skipped entirely on machines where `mcp` is not installed, so the existing
|
||||
25-test suite stays green without the optional extra.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("mcp")
|
||||
|
||||
from fin_mcp.server import create_server # noqa: E402
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tool_names(server):
|
||||
return {t.name for t in asyncio.run(server.list_tools())}
|
||||
|
||||
|
||||
def _call(server, name, arguments=None):
|
||||
"""Call a tool and return the raw FastMCP result list."""
|
||||
return asyncio.run(server.call_tool(name, arguments or {}))
|
||||
|
||||
|
||||
def _unwrap(result):
|
||||
"""Extract the {'result': ...} payload from a FastMCP call_tool response.
|
||||
|
||||
The return shape is SDK-version dependent:
|
||||
mcp >=1.10 -> list[Content]
|
||||
mcp >=1.12 -> tuple(list[Content], structured_dict)
|
||||
Both are handled so the suite does not pin a patch release.
|
||||
"""
|
||||
import json
|
||||
|
||||
# Newer SDKs return (content, structured_content); the second element is
|
||||
# already the deserialised return value.
|
||||
if isinstance(result, tuple):
|
||||
content, structured = result
|
||||
if isinstance(structured, dict):
|
||||
return structured
|
||||
result = content
|
||||
|
||||
item = result[0]
|
||||
text = getattr(item, "text", None)
|
||||
if text is None:
|
||||
text = item.content[0].text
|
||||
return json.loads(text)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestReadOnlyMode:
|
||||
def test_read_only_by_default(self, client, monkeypatch):
|
||||
"""Server created without flag exposes exactly 15 read tools."""
|
||||
monkeypatch.delenv("FIN_MCP_ALLOW_WRITES", raising=False)
|
||||
server = create_server(client=client)
|
||||
names = _tool_names(server)
|
||||
|
||||
# Required read tools present
|
||||
assert "get_net_worth" in names
|
||||
assert "list_accounts" in names
|
||||
assert "list_transactions" in names
|
||||
assert "get_spending_month" in names
|
||||
assert "get_fx_rate" in names
|
||||
assert "list_rsu_grants" in names
|
||||
assert "preview_rsu_schedule" in names
|
||||
assert "health" in names
|
||||
assert "find_account" in names
|
||||
assert "find_category" in names
|
||||
assert "get_net_worth_history" in names
|
||||
assert "list_categories" in names
|
||||
assert "get_holdings" in names
|
||||
assert "list_stock_transactions" in names
|
||||
assert "get_spending_summary" in names
|
||||
|
||||
# No write tools
|
||||
write_names = {n for n in names if n.startswith(("create_", "delete_", "update_", "deactivate_", "adjust_", "vest_", "generate_", "reconcile", "refresh_", "fetch_", "funded_"))}
|
||||
assert write_names == set(), f"Unexpected write tools in read-only mode: {write_names}"
|
||||
|
||||
assert len(names) == 15
|
||||
|
||||
def test_env_flag_enables_writes(self, client, monkeypatch):
|
||||
"""FIN_MCP_ALLOW_WRITES=1 adds 24 write tools for a total of 39."""
|
||||
monkeypatch.setenv("FIN_MCP_ALLOW_WRITES", "1")
|
||||
server = create_server(client=client)
|
||||
names = _tool_names(server)
|
||||
|
||||
assert "create_transfer" in names
|
||||
assert "delete_transaction" in names
|
||||
assert "refresh_prices" in names
|
||||
assert len(names) == 39
|
||||
|
||||
|
||||
class TestReadParity:
|
||||
def test_read_parity(self, client):
|
||||
"""Tool results match FinClient results for three read tools (B1 criterion)."""
|
||||
# Seed some data
|
||||
acct = client.create_account({"name": "Parity Bank", "type": "bank", "currency": "EUR"})
|
||||
account_id = acct["id"]
|
||||
import datetime
|
||||
today = datetime.date.today().isoformat()
|
||||
client.create_transaction({
|
||||
"account_id": account_id,
|
||||
"date": today,
|
||||
"amount": -25.0,
|
||||
"type": "expense",
|
||||
"currency": "EUR",
|
||||
})
|
||||
|
||||
server = create_server(client=client)
|
||||
|
||||
# list_accounts parity
|
||||
tool_result = _unwrap(_call(server, "list_accounts"))
|
||||
client_result = client.list_accounts()
|
||||
assert tool_result["result"] == client_result
|
||||
|
||||
# list_transactions parity
|
||||
tool_txns = _unwrap(_call(server, "list_transactions", {"account_id": account_id}))
|
||||
client_txns = client.list_transactions(account_id=account_id)
|
||||
assert tool_txns["result"] == client_txns
|
||||
|
||||
# get_net_worth parity
|
||||
tool_nw = _unwrap(_call(server, "get_net_worth"))
|
||||
client_nw = client.get_net_worth()
|
||||
assert tool_nw["result"] == client_nw
|
||||
|
||||
|
||||
class TestWriteRoundTrip:
|
||||
def test_write_round_trip(self, client):
|
||||
"""Writes-enabled server: create account + expense, then verify via FinClient."""
|
||||
server = create_server(client=client, allow_writes=True)
|
||||
|
||||
# Create account via tool
|
||||
result = _unwrap(_call(server, "create_account", {
|
||||
"name": "Write Test",
|
||||
"type": "bank",
|
||||
"currency": "EUR",
|
||||
}))
|
||||
account_id = result["result"]["id"]
|
||||
|
||||
# Create expense via tool
|
||||
import datetime
|
||||
today = datetime.date.today().isoformat()
|
||||
_call(server, "create_transaction", {
|
||||
"account_id": account_id,
|
||||
"date": today,
|
||||
"amount": -50.0,
|
||||
"type": "expense",
|
||||
"currency": "EUR",
|
||||
})
|
||||
|
||||
# Verify via FinClient
|
||||
txns = client.list_transactions(account_id=account_id)
|
||||
assert len(txns) == 1
|
||||
assert txns[0]["amount"] == -50.0
|
||||
|
||||
|
||||
class TestWriteErrors:
|
||||
def test_write_error_surfaces(self, client):
|
||||
"""Deleting a non-existent transaction surfaces the API error, not a crash.
|
||||
|
||||
Depending on SDK version the failure either raises ToolError or comes
|
||||
back as an error-flagged result; both must carry the upstream 404 so
|
||||
the caller can tell what went wrong.
|
||||
"""
|
||||
server = create_server(client=client, allow_writes=True)
|
||||
|
||||
try:
|
||||
result = _call(server, "delete_transaction", {"transaction_id": 99999})
|
||||
except Exception as exc: # ToolError on mcp >=1.12
|
||||
assert "404" in str(exc), f"Error lost the upstream status: {exc}"
|
||||
return
|
||||
|
||||
content = result[0] if isinstance(result, tuple) else result
|
||||
item = content[0]
|
||||
is_error = getattr(item, "isError", False) or getattr(item, "type", "") == "error"
|
||||
text = getattr(item, "text", "")
|
||||
assert is_error or "404" in text, (
|
||||
f"Expected an error for missing id 99999, got: {text}"
|
||||
)
|
||||
|
||||
|
||||
class TestInvariantGuards:
|
||||
"""The MCP surface must not be able to break ledger invariants."""
|
||||
|
||||
def test_create_transaction_rejects_two_leg_types(self, client):
|
||||
"""create_transaction cannot write a single leg of a transfer.
|
||||
|
||||
POST /api/transactions accepts all six transaction types, so a bare
|
||||
`type: str` would let a caller create a `transfer` row with no
|
||||
linked_account_id — an orphan leg that corrupts balances and that the
|
||||
paired-delete logic can never clean up. The tool schema must refuse it.
|
||||
"""
|
||||
server = create_server(client=client, allow_writes=True)
|
||||
|
||||
schema = None
|
||||
for tool in asyncio.run(server.list_tools()):
|
||||
if tool.name == "create_transaction":
|
||||
schema = tool.inputSchema
|
||||
break
|
||||
assert schema is not None, "create_transaction not registered"
|
||||
|
||||
# Resolve the enum, which may sit behind a $ref/$defs indirection.
|
||||
prop = schema["properties"]["type"]
|
||||
allowed = prop.get("enum")
|
||||
if allowed is None:
|
||||
ref = prop.get("$ref", "").rsplit("/", 1)[-1]
|
||||
allowed = schema.get("$defs", {}).get(ref, {}).get("enum")
|
||||
|
||||
assert allowed is not None, f"type is not constrained by an enum: {prop}"
|
||||
assert set(allowed) == {"expense", "income"}, (
|
||||
f"create_transaction must only allow expense/income, got: {allowed}"
|
||||
)
|
||||
|
||||
|
||||
class TestNameResolution:
|
||||
"""find_account / find_category turn a human name into the id tools need."""
|
||||
|
||||
def test_exact_match_beats_substring(self, client):
|
||||
"""An exact name must not be drowned out by longer names containing it."""
|
||||
client.create_account({"name": "Food", "type": "bank", "currency": "EUR"})
|
||||
client.create_account({"name": "Fast Food Fund", "type": "bank", "currency": "EUR"})
|
||||
|
||||
server = create_server(client=client)
|
||||
hits = _unwrap(_call(server, "find_account", {"name": "food"}))["result"]
|
||||
|
||||
assert [h["name"] for h in hits] == ["Food"]
|
||||
|
||||
def test_substring_returns_every_candidate(self, client):
|
||||
"""An ambiguous name returns all matches so the caller can disambiguate."""
|
||||
client.create_account({"name": "Savings A", "type": "bank", "currency": "EUR"})
|
||||
client.create_account({"name": "Savings B", "type": "bank", "currency": "EUR"})
|
||||
|
||||
server = create_server(client=client)
|
||||
hits = _unwrap(_call(server, "find_account", {"name": "savings"}))["result"]
|
||||
|
||||
assert sorted(h["name"] for h in hits) == ["Savings A", "Savings B"]
|
||||
|
||||
def test_no_match_is_empty_not_an_error(self, client):
|
||||
"""A miss must return [] rather than raising, so the model can ask the user."""
|
||||
server = create_server(client=client)
|
||||
hits = _unwrap(_call(server, "find_account", {"name": "no such account"}))["result"]
|
||||
|
||||
assert hits == []
|
||||
|
||||
def test_find_category_returns_usable_id(self, client):
|
||||
client.create_category({"name": "Groceries", "color": "#22c55e", "icon": "🛒"})
|
||||
|
||||
server = create_server(client=client)
|
||||
hits = _unwrap(_call(server, "find_category", {"name": "groceries"}))["result"]
|
||||
|
||||
assert len(hits) == 1
|
||||
assert hits[0]["name"] == "Groceries"
|
||||
assert isinstance(hits[0]["id"], int)
|
||||
|
||||
|
||||
class TestResources:
|
||||
"""Accounts and categories are also exposed as resources, not just tools."""
|
||||
|
||||
def test_reference_resources_are_listed(self, client):
|
||||
server = create_server(client=client)
|
||||
uris = {str(r.uri) for r in asyncio.run(server.list_resources())}
|
||||
|
||||
assert "fin://accounts" in uris
|
||||
assert "fin://categories" in uris
|
||||
|
||||
def test_accounts_resource_serves_live_json(self, client):
|
||||
import json
|
||||
|
||||
client.create_account({"name": "Resource Bank", "type": "bank", "currency": "EUR"})
|
||||
|
||||
server = create_server(client=client)
|
||||
contents = list(asyncio.run(server.read_resource("fin://accounts")))
|
||||
|
||||
assert len(contents) == 1
|
||||
assert contents[0].mime_type == "application/json"
|
||||
|
||||
payload = json.loads(contents[0].content)
|
||||
assert [a["name"] for a in payload] == ["Resource Bank"]
|
||||
assert payload == client.list_accounts()
|
||||
@@ -0,0 +1,110 @@
|
||||
# fin-mcp — MCP server for the fin API.
|
||||
#
|
||||
# This file is NOT applied from this repo. The cluster is reconciled by Flux
|
||||
# from the GitOps repo (Kustomization `fin-app`, namespace flux-system); copy
|
||||
# this alongside the existing fin-api / fin-web manifests there.
|
||||
#
|
||||
# Deliberately ClusterIP-only, with no HTTPRoute. Writes are enabled, and the
|
||||
# fin API has no authentication (FIN_API_TOKEN is unset on fin-api), so the
|
||||
# only thing keeping this safe is that nothing outside the cluster can reach
|
||||
# it. Do not add an HTTPRoute for this Service without turning on auth first.
|
||||
# This matches how karakeep-mcp, paperless-mcp and searxng-mcp are deployed.
|
||||
#
|
||||
# In-cluster endpoint: http://fin-mcp.apps.svc.cluster.local:8000/mcp
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: fin-mcp
|
||||
namespace: apps
|
||||
labels:
|
||||
app: fin
|
||||
component: mcp
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: fin
|
||||
component: mcp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: fin
|
||||
component: mcp
|
||||
spec:
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
containers:
|
||||
- name: mcp
|
||||
image: harbor.cloud.traunseenet.com/library/fin-mcp:latest
|
||||
imagePullPolicy: Always
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8000
|
||||
protocol: TCP
|
||||
env:
|
||||
# Talks to the API over the cluster Service, not the public route.
|
||||
- name: FIN_API_URL
|
||||
value: http://fin-api.apps.svc.cluster.local:8000
|
||||
- name: FIN_MCP_TRANSPORT
|
||||
value: streamable-http
|
||||
- name: FIN_MCP_HOST
|
||||
value: 0.0.0.0
|
||||
- name: FIN_MCP_PORT
|
||||
value: "8000"
|
||||
- name: FIN_MCP_PATH
|
||||
value: /mcp
|
||||
# 39 tools instead of 15. Safe only while this stays unreachable
|
||||
# from outside the cluster — see the header note.
|
||||
- name: FIN_MCP_ALLOW_WRITES
|
||||
value: "1"
|
||||
# Uncomment together with FIN_API_TOKEN on fin-api.
|
||||
# - name: FIN_API_TOKEN
|
||||
# valueFrom:
|
||||
# secretKeyRef:
|
||||
# name: fin-api-token
|
||||
# key: token
|
||||
# MCP has no unauthenticated health route, so probe the TCP socket
|
||||
# rather than an HTTP path — GET /mcp answers 406 without the MCP
|
||||
# Accept headers, which would read as a failure.
|
||||
livenessProbe:
|
||||
tcpSocket:
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
tcpSocket:
|
||||
port: http
|
||||
initialDelaySeconds: 3
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: fin-mcp
|
||||
namespace: apps
|
||||
labels:
|
||||
app: fin
|
||||
component: mcp
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: fin
|
||||
component: mcp
|
||||
ports:
|
||||
- name: http
|
||||
port: 8000
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
@@ -13,6 +13,24 @@ services:
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
mcp:
|
||||
image: harbor.cloud.traunseenet.com/library/fin-mcp:latest
|
||||
build: ./cli
|
||||
environment:
|
||||
- FIN_API_URL=http://api:8000
|
||||
- FIN_MCP_TRANSPORT=streamable-http
|
||||
- FIN_MCP_HOST=0.0.0.0
|
||||
- FIN_MCP_PORT=8000
|
||||
- FIN_MCP_PATH=/mcp
|
||||
- FIN_MCP_ALLOW_WRITES=1
|
||||
# No ports mapping on purpose: the API has no auth, so the MCP surface
|
||||
# stays reachable only from inside the compose network. Add a mapping
|
||||
# only if you also set FIN_API_TOKEN.
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
web:
|
||||
image: harbor.cloud.traunseenet.com/library/fin-web:latest
|
||||
build: ./web
|
||||
|
||||
@@ -136,6 +136,17 @@ export const api = {
|
||||
request<Transaction[]>('/transactions/splitwise-i-paid', { method: 'POST', body: JSON.stringify(data) }),
|
||||
createSplitExpense: (data: { bank_account_id: number; receivable_account_id: number; total_amount: number; partner_share: number; currency?: string; date: string; category_id?: number; description?: string }) =>
|
||||
request<Transaction[]>('/transactions/split-expense', { method: 'POST', body: JSON.stringify(data) }),
|
||||
// Items carry positive amounts; the backend derives the sign from `type`.
|
||||
// One commit server-side, so a rejected batch writes nothing.
|
||||
createTransactionBatch: (data: {
|
||||
account_id: number
|
||||
date: string
|
||||
currency?: string
|
||||
type?: 'expense' | 'income'
|
||||
items: { amount: number; category_id?: number; description?: string; fee?: number }[]
|
||||
}) => request<Transaction[]>('/transactions/batch', {
|
||||
method: 'POST', body: JSON.stringify(data),
|
||||
}),
|
||||
reconcileAccounts: (data: {
|
||||
bank_account_id: number; other_account_id: number;
|
||||
amount: number; date: string; description?: string;
|
||||
|
||||
@@ -2,6 +2,7 @@ import { useState, useEffect } from 'react'
|
||||
import { Account, Category } from '@/api/client'
|
||||
import { Dialog, DialogContent, DialogTitle, DialogDescription } from '@/components/ui/dialog'
|
||||
import ExpenseFlow from './flows/ExpenseFlow'
|
||||
import BatchExpenseFlow from './flows/BatchExpenseFlow'
|
||||
import IncomeFlow from './flows/IncomeFlow'
|
||||
import SplitwiseIPaidFlow from './flows/SplitwiseIPaidFlow'
|
||||
import SplitwiseShePayedFlow from './flows/SplitwiseShePayedFlow'
|
||||
@@ -14,6 +15,7 @@ import SplitExpenseFlow from './flows/SplitExpenseFlow'
|
||||
|
||||
export type FlowType =
|
||||
| 'expense'
|
||||
| 'batch_expense'
|
||||
| 'split_expense'
|
||||
| 'income'
|
||||
| 'splitwise_i_paid'
|
||||
@@ -35,6 +37,7 @@ function buildFlows(accounts: Account[]): { id: FlowType; label: string; descrip
|
||||
const hasPayable = accounts.some(a => a.type === 'payable' && a.is_active)
|
||||
const flows: { id: FlowType; label: string; description: string }[] = [
|
||||
{ id: 'expense', label: 'Expense', description: 'Record a personal expense' },
|
||||
{ id: 'batch_expense', label: 'Batch Expenses', description: 'Enter several expenses at once' },
|
||||
{ id: 'income', label: 'Income', description: 'Record money received' },
|
||||
{ id: 'splitwise_i_paid', label: 'Splitwise: I Paid', description: 'You paid, she owes her share' },
|
||||
{ id: 'splitwise_she_paid', label: 'Splitwise: She Paid', description: 'She paid, you owe your share' },
|
||||
@@ -94,6 +97,8 @@ function SelectedFlow({ flow, accounts, categories, onSuccess, onBack, initialAc
|
||||
switch (flow) {
|
||||
case 'expense':
|
||||
return <ExpenseFlow {...props} />
|
||||
case 'batch_expense':
|
||||
return <BatchExpenseFlow {...props} />
|
||||
case 'split_expense':
|
||||
return <SplitExpenseFlow {...props} />
|
||||
case 'income':
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import { useState } from 'react'
|
||||
import { api, Account, Category } from '@/api/client'
|
||||
import { Input } from '@/components/ui/input'
|
||||
import { parseAmount, todayLocalISO } from '@/lib/utils'
|
||||
|
||||
interface BatchExpenseFlowProps {
|
||||
accounts: Account[]
|
||||
categories: Category[]
|
||||
onSuccess: () => void
|
||||
onBack: () => void
|
||||
initialAccountId?: number
|
||||
}
|
||||
|
||||
type Row = { amountStr: string; categoryId: number | null; description: string }
|
||||
|
||||
const emptyRow = (): Row => ({ amountStr: '', categoryId: null, description: '' })
|
||||
|
||||
export default function BatchExpenseFlow({
|
||||
accounts,
|
||||
categories,
|
||||
onSuccess,
|
||||
onBack,
|
||||
initialAccountId,
|
||||
}: BatchExpenseFlowProps) {
|
||||
const bankAccounts = accounts.filter(a => a.type === 'bank' && a.is_active)
|
||||
const expenseCategories = categories.filter(c => c.type === 'expense')
|
||||
|
||||
const [rows, setRows] = useState<Row[]>([emptyRow()])
|
||||
const [selectedAccountId, setSelectedAccountId] = useState<number>(
|
||||
initialAccountId ?? bankAccounts[0]?.id ?? 0
|
||||
)
|
||||
const [dateStr, setDateStr] = useState(todayLocalISO())
|
||||
const [saving, setSaving] = useState(false)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
|
||||
const selectedAccount = bankAccounts.find(a => a.id === selectedAccountId)
|
||||
const currency = selectedAccount?.currency ?? 'EUR'
|
||||
|
||||
const updateRow = (i: number, patch: Partial<Row>) =>
|
||||
setRows(rs => rs.map((r, idx) => (idx === i ? { ...r, ...patch } : r)))
|
||||
const addRow = () => setRows(rs => [...rs, emptyRow()])
|
||||
const removeRow = (i: number) => setRows(rs => rs.filter((_, idx) => idx !== i))
|
||||
|
||||
const parsed = rows.map(r => ({ row: r, amount: parseAmount(r.amountStr) }))
|
||||
const valid = parsed.filter(p => p.amount !== null)
|
||||
const total = valid.reduce((sum, p) => sum + (p.amount ?? 0), 0)
|
||||
const fmt = (n: number) =>
|
||||
new Intl.NumberFormat('de-DE', { style: 'currency', currency }).format(n)
|
||||
|
||||
const handleSave = async () => {
|
||||
// A row left completely blank is just an unused slot; a row with text that
|
||||
// will not parse is a mistake worth naming.
|
||||
const bad = parsed.findIndex(p => p.amount === null && p.row.amountStr.trim() !== '')
|
||||
if (bad !== -1) { setError(`Row ${bad + 1}: enter a valid amount`); return }
|
||||
if (valid.length === 0) { setError('Add at least one row'); return }
|
||||
if (!selectedAccountId) { setError('Select an account'); return }
|
||||
|
||||
setSaving(true)
|
||||
setError(null)
|
||||
try {
|
||||
await api.createTransactionBatch({
|
||||
account_id: selectedAccountId,
|
||||
date: dateStr,
|
||||
currency,
|
||||
type: 'expense',
|
||||
items: valid.map(p => ({
|
||||
amount: p.amount as number,
|
||||
category_id: p.row.categoryId ?? undefined,
|
||||
description: p.row.description || undefined,
|
||||
})),
|
||||
})
|
||||
onSuccess()
|
||||
} catch (e) {
|
||||
// The write is atomic, so nothing was saved — leave the rows in place
|
||||
// for the user to fix and resubmit.
|
||||
setError(e instanceof Error ? e.message : 'Failed to save')
|
||||
} finally {
|
||||
setSaving(false)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="p-4 space-y-4">
|
||||
<div className="flex items-center gap-2 mb-2">
|
||||
<button onClick={onBack} className="text-muted-foreground text-sm">← Back</button>
|
||||
<h2 className="text-base font-semibold">Batch Expenses</h2>
|
||||
</div>
|
||||
|
||||
{bankAccounts.length === 0 && (
|
||||
<p className="text-sm text-yellow-600 bg-yellow-50 rounded-lg p-2">
|
||||
No bank accounts found. Please add one in Settings.
|
||||
</p>
|
||||
)}
|
||||
|
||||
{/* Account — shared by every row */}
|
||||
<div>
|
||||
<label className="text-xs text-muted-foreground mb-1 block">Account</label>
|
||||
<div className="flex flex-col gap-1">
|
||||
{bankAccounts.map(acc => (
|
||||
<button
|
||||
key={acc.id}
|
||||
onClick={() => setSelectedAccountId(acc.id)}
|
||||
className={`text-left px-3 py-2 rounded-lg border text-sm ${
|
||||
selectedAccountId === acc.id
|
||||
? 'border-primary bg-primary/10'
|
||||
: 'border-border bg-background'
|
||||
}`}
|
||||
>
|
||||
{acc.name}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Date — shared by every row */}
|
||||
<div>
|
||||
<label className="text-xs text-muted-foreground mb-1 block">Date</label>
|
||||
<Input type="date" value={dateStr} onChange={e => setDateStr(e.target.value)} />
|
||||
</div>
|
||||
|
||||
{/* Rows */}
|
||||
<div className="space-y-2">
|
||||
<label className="text-xs text-muted-foreground block">Expenses</label>
|
||||
{rows.map((row, i) => (
|
||||
<div key={i} className="rounded-lg border border-border p-2 space-y-2">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="text-xs text-muted-foreground w-5 shrink-0">{i + 1}.</span>
|
||||
<Input
|
||||
type="text"
|
||||
inputMode="decimal"
|
||||
placeholder="0.00"
|
||||
value={row.amountStr}
|
||||
onChange={e => updateRow(i, { amountStr: e.target.value })}
|
||||
/>
|
||||
{rows.length > 1 && (
|
||||
<button
|
||||
onClick={() => removeRow(i)}
|
||||
aria-label={`Remove row ${i + 1}`}
|
||||
className="text-muted-foreground text-lg leading-none px-2 shrink-0"
|
||||
>
|
||||
×
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex gap-2 pl-7">
|
||||
<select
|
||||
className="flex-1 min-w-0 border rounded px-2 py-1 text-sm bg-background"
|
||||
value={row.categoryId ?? ''}
|
||||
onChange={e =>
|
||||
updateRow(i, { categoryId: e.target.value ? Number(e.target.value) : null })
|
||||
}
|
||||
>
|
||||
<option value="">Uncategorized</option>
|
||||
{expenseCategories.map(c => (
|
||||
<option key={c.id} value={c.id}>{c.icon} {c.name}</option>
|
||||
))}
|
||||
</select>
|
||||
<input
|
||||
className="flex-1 min-w-0 border rounded px-2 py-1 text-sm bg-background"
|
||||
type="text"
|
||||
placeholder="Description"
|
||||
value={row.description}
|
||||
onChange={e => updateRow(i, { description: e.target.value })}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
|
||||
<button
|
||||
onClick={addRow}
|
||||
className="w-full py-2 rounded-lg border border-dashed border-border text-sm text-muted-foreground"
|
||||
>
|
||||
+ Add row
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="flex items-center justify-between text-sm">
|
||||
<span className="text-muted-foreground">Total</span>
|
||||
<span className="font-semibold">{fmt(total)}</span>
|
||||
</div>
|
||||
|
||||
{error && <p className="text-sm text-red-500">{error}</p>}
|
||||
|
||||
<button
|
||||
onClick={handleSave}
|
||||
disabled={saving || bankAccounts.length === 0}
|
||||
className="w-full py-3 rounded-lg bg-primary text-primary-foreground font-medium disabled:opacity-50"
|
||||
>
|
||||
{saving
|
||||
? 'Saving...'
|
||||
: `Save ${valid.length} expense${valid.length === 1 ? '' : 's'}`}
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -9,3 +9,10 @@ export function todayLocalISO(): string {
|
||||
const d = new Date()
|
||||
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(d.getDate()).padStart(2, '0')}`
|
||||
}
|
||||
|
||||
// Accepts both 12.40 and the German 12,40. Returns null for anything that is
|
||||
// not a usable positive amount, so callers can report which row is wrong.
|
||||
export function parseAmount(input: string): number | null {
|
||||
const n = parseFloat(input.replace(',', '.'))
|
||||
return Number.isFinite(n) && n > 0 ? n : null
|
||||
}
|
||||
Reference in new issue
Block a user