fin/.claude/skills/fin-validation-and-qa/SKILL.md

name, description

name

description

fin-validation-and-qa

Load BEFORE claiming any change to fin works, when writing or extending tests in cli/tests/, when deciding what evidence a fix or feature needs before it can be called done, or when reviewing someone else's change. Defines the evidence hierarchy (test > diagnostics numbers > fin --json > compile > eyeball), the golden inventory of what the e2e suite actually certifies, how to run and add tests safely (prod DB never a fixture), and the hypothesis-predicts-numbers discipline for accepting a fix.

fin — Validation and QA

Steady-state quality discipline for the fin repo. The only test suite is cli/tests/ (e2e: FinClient → FastAPI TestClient → real crud.py → throwaway SQLite). There is NO CI — the green run on your machine IS the gate. The prod DB is sacred: no test, experiment, or "quick check" ever touches it.

Read /home/markuseisl/Documents/git/fin/CLAUDE.md first if you haven't.

When NOT to use this skill

You want to…	Go to
Use the fin CLI against the live API	fin
Expand test coverage as a campaign (direct crud.py tests, new layers)	fin-test-coverage-campaign
Set up Python/pytest environment, install deps	fin-build-and-env
Run the audit/diagnostics scripts themselves	fin-diagnostics-and-tooling
Decide whether a change is allowed at all / merge gates	fin-change-control
Debug a live misbehavior (not validate a fix)	fin-debugging-playbook
Look up correct accounting math for expected values	fin-accounting-reference
Check whether a bug class has happened before	fin-failure-archaeology

1. Evidence hierarchy (strongest → weakest)

When you claim "X is fixed" or "X works", state which level your evidence is.

Level	Evidence	Counts as proof for
(a)	Automated test in cli/tests/ that fails before the change and passes after	Anything. The gold standard.
(b)	Diagnostics-script numbers before/after, run on a synthetic or copied DB (see fin-diagnostics-and-tooling)	Accounting/balance logic when a test is genuinely impractical
(c)	fin --json API-level checks against a throwaway/dev instance, with expected values written down first	Wiring, endpoint behavior, CLI plumbing
(d)	python -m py_compile … / tsc via vite build passes	Nothing works — only that it parses/typechecks
(e)	Manual UI walkthrough	Rendering and UX only

Hard rule: accounting-logic changes (balances, spending, transfers, settlements, splits, holdings, net worth) require level (a) or (b). Never eyeball a balance.

Hypothesis-predicts-numbers discipline

Before running ANY check (test, script, fin --json call):

1. Write down the exact number/output you expect and WHY (one sentence of mechanism).
2. Run the check.
3. If the number differs — even in a direction that "looks fixed" — the hypothesis is wrong or incomplete. Do not accept the fix.
4. A fix is accepted only when one mechanism explains ALL observations, including the negatives (the checks that were expected to stay unchanged and did). A fix that explains the failing case but not why case Y was unaffected is a guess, not a fix.

2. Golden inventory — what a green run actually certifies

Source of truth: /home/markuseisl/Documents/git/fin/cli/tests/test_cli.py. 25 test functions as of 2026-07-05 (9 auth, 2 accounts, 2 transactions, 3 transfer, 1 settlement, 1 split-expense, 1 splitwise-paid, 2 net-worth, 1 spending, 3 CLI wiring). Re-count before trusting this table (see Provenance).

Certified (locked by a test)

Area	Test	Invariant it locks
Auth	TestAuth::test_auth_off_allows_requests	No FIN_API_TOKEN env → API open
Auth	test_health_ok, test_health_bypasses_auth	/api/health always open, even with auth on
Auth	test_auth_on_rejects_missing_token, test_auth_on_rejects_wrong_token	Auth on → 401 without/with wrong bearer token
Auth	test_auth_on_accepts_correct_token	Correct token passes
Auth	test_bearer_scheme_is_case_insensitive	bearer == Bearer
Auth	test_openapi_gated_when_auth_on, test_openapi_open_when_auth_off	OpenAPI spec follows the auth gate
Accounts	TestAccounts::test_create_and_list	Create/list roundtrip
Accounts	test_initial_balance_creates_opening_transaction	initial_balance → one opening income txn of that amount
Transactions	TestTransactions::test_add_and_list_with_limit	List is newest-first; limit respected
Transactions	test_limit_must_be_positive	limit=0 → 422
Transfer	TestTransfer::test_two_legs_created_and_linked	Two legs, ±amount, cross-linked via linked_account_id (CLAUDE.md invariant 1)
Transfer	test_deleting_one_leg_removes_whole_group	Deleting either leg deletes both — no orphan legs
Transfer	test_editing_one_leg_is_rejected	Editing one transfer leg → 422
Settlement	TestSettlement::test_bank_to_splitwise_two_legs_linked	bank→splitwise: two linked settlement legs, ±amount
Split expense	TestSplitExpense::test_three_rows_and_user_share_spending	total 100/partner 40 → expense −60 + transfer ±40 (3 rows); spending counts only the 60
Splitwise paid	TestSplitwisePaid::test_spending_counts_net_share	bank −100, splitwise_partner_share=45, splitwise leg +45; spending = 55 (invariant 6)
Net worth	TestNetWorth::test_current_reflects_bank_balance	Cash balance flows into total_eur
Net worth	test_fee_reduces_balance	balance = SUM(amount − fee) (invariant 5): 100 − 50 − 2 = 48
Spending	TestSpending::test_monthly_totals	Monthly total + per-category breakdown
CLI wiring	TestCliWiring::test_accounts_list_json	fin accounts list --json emits parseable JSON
CLI wiring	test_error_exits_nonzero	API error → exit code 1
CLI wiring	test_tx_add_refuses_transfer_type	tx add --type transfer rejected client-side, nothing written

NOT certified (a green run says nothing about these)

Area	Gap (as of 2026-07-05)
Stocks/holdings	Zero tests. create_stock_transaction, delete_stock_transaction, recompute_holding_from_stock_transactions (CLAUDE.md invariant 2 is untested), create_funded_buy
RSU	Zero tests: grants, vest, schedule preview/generate, vest→holding linkage
crud.py direct	No unit tests at all (~995 lines). Only paths reachable via the API calls above are exercised — fin-test-coverage-campaign owns closing this
Mutations	update_account, deactivate_account, balance adjust/reconcile, category update/delete, update_transaction on plain txns, delete_transaction of a plain txn
Settlement	splitwise_to_bank direction untested (only bank_to_splitwise)
Spending	get_spending_summary (multi-month), spending with transfers/settlements/stock trades mixed in a month (the e36d719 bug class)
Net worth	services/net_worth.py with holdings + price_cache (only pure-cash paths hit), history/snapshots, non-EUR accounts, FX conversion
Prices	price_fetcher.py, /api/prices/* — yfinance is a stub module in tests; nothing about fetching is certified
Scheduler	services/scheduler.py never starts in tests; midnight job untested
Routers	Error mapping beyond the handful of 422/404 cases above
Frontend	Zero automated tests of any kind — see §6
CLI	Most subcommands beyond the 3 wiring smoke tests; all human (non---json) output

Consult this map before trusting "tests pass" as evidence for your specific change.

3. Running the tests

Commands are statically verified against conftest.py/pyproject.toml — this authoring environment has no pip/pytest, so they were not executed here.

cd /home/markuseisl/Documents/git/fin
python -m pytest cli/tests -q

• Working directory does not matter for imports: conftest.py computes REPO_ROOT from its own path and inserts api/ and cli/ into sys.path. Repo root is still the conventional place to run from.
• conftest.py sets DATABASE_URL to sqlite:///<mkdtemp>/test.db before importing app.*, and stubs yfinance as an empty module. Never import app.database in a new conftest/helper before that env var is set.
• Green run as of 2026-07-05: 25 passed (dots, no warnings-as-errors config).
• Requires pytest, fastapi, plus the api/ and cli/ dependencies installed. Environment failures (missing modules, wrong Python) → fin-build-and-env.
• The suite never needs FIN_API_URL/FIN_API_TOKEN; the autouse fixture deletes FIN_API_TOKEN so a token in your shell can't poison auth tests.

4. Adding a test — checklist and skeleton

Where: a new test_*.py in /home/markuseisl/Documents/git/fin/cli/tests/ (or a new class in test_cli.py if it fits an existing area). pytest discovers it automatically; conftest.py fixtures apply.

Checklist:

• Use the client fixture (a FinClient over in-process TestClient) or make_client(token=...) when testing auth behavior.
• Rely on the autouse clean_db fixture: it drop_all + create_alls the throwaway DB before every test — tests start empty and must not depend on each other's data or ordering.
• Exercise multi-step flows through the API surface only: create account → transact → assert balances via client.get_net_worth() / client.list_transactions() / client.get_spending_month().
• Do NOT import from app.* internals when the API surface suffices — these are e2e tests of the deployed contract. (Direct crud-layer tests are a different layer with different rules; that belongs to fin-test-coverage-campaign.)
• No live prices, ever: yfinance is a stub. A test that triggers a real fetch (e.g. client.refresh_prices() reaching yfinance) will crash or hang — if you need a price, insert into price_cache via whatever seeding mechanism the campaign skill establishes, or don't test it here.
• Write the expected numbers as literals with a comment stating the mechanism (see test_fee_reduces_balance citing invariant 5).
• Assert on FinClientError.status_code for expected failures.

Worked skeleton (multi-step flow):

import pytest
from fin_cli.client import FinClientError


class TestSettlementSplitwiseToBank:
    def test_splitwise_to_bank_direction(self, client):
        bank = client.create_account(
            {"name": "Giro", "type": "bank", "currency": "EUR",
             "initial_balance": 100.0}
        )
        sw = client.create_account(
            {"name": "Splitwise", "type": "splitwise", "currency": "EUR"}
        )
        legs = client.create_settlement(
            {"bank_account_id": bank["id"], "splitwise_account_id": sw["id"],
             "amount": 30.0, "date": "2026-06-20",
             "direction": "splitwise_to_bank"}
        )
        by_account = {leg["account_id"]: leg for leg in legs}
        # money moves toward the bank: bank +30, splitwise -30
        assert by_account[bank["id"]]["amount"] == 30.0
        assert by_account[sw["id"]]["amount"] == -30.0
        # settlements are not spending
        assert client.get_spending_month("2026-06")["total"] == pytest.approx(0.0)
        # net worth: 100 opening + 30 in - 30 out across own accounts = 100
        assert client.get_net_worth()["total_eur"] == pytest.approx(100.0)

(Predict the numbers in comments BEFORE running — §1 discipline. Verify the expected sign convention against the router/crud before committing; the skeleton's ±30 is the hypothesis, the first run is the experiment.)

5. Acceptance thresholds and regression discipline

• Every fixed bug gains a test that reproduces it — written to fail on the pre-fix code. History shows why: the bug-hunt commits (bc256f6, e36d719) fixed accounting bugs without adding regression tests, and bugs recurred in families (spending-inflation class hit twice). Details: fin-failure-archaeology.
• Tests must be deterministic: no network (yfinance stubbed, no httpx to the outside), no clock assumptions beyond the app's own local-date helpers — pass explicit date strings, never "today", when asserting on month buckets.
• Prod DB is never a fixture. Do not point DATABASE_URL or FIN_API_URL at the live instance from a test, and never copy /data/fin.db into the test tree. Experiments on real data happen on a copied DB via fin-diagnostics-and-tooling / fin-ledger-analysis-toolkit, read-only.
• A change that touches crud.py accounting paths and ships with only level (c)–(e) evidence is not done. Flag it in review.
• No flaky-test tolerance: with 25 tests and no CI, a single intermittent failure erodes the only gate the project has. Fix or delete, never skip.

6. Frontend validation — honest current bar

Zero frontend tests exist (as of 2026-07-05). The bar is:

1. vite build (runs tsc) must pass — level (d) evidence only.
2. Manual walkthrough of the critical flows — level (e).

Manual QA script for a frontend-touching change (worked numbers; exact accounting rules in fin-accounting-reference). Do this on a dev/throwaway stack, not prod. Start state: one bank account "Giro" with balance 1,000.00 €.

#	Action (QuickAdd unless noted)	Expected result
1	Expense −50.00, fee 2.00, category Dining	Giro 948.00 (Δ −52: amount − fee)
2	Spending page, current month	Total 50.00, Dining 50.00 (fee excluded from spending)
3	Create second bank account "Savings" (0.00)	Net worth unchanged: 948.00
4	Transfer 100.00 Giro → Savings	Giro 848.00, Savings 100.00, net worth still 948.00, two linked rows on Transactions page
5	Split expense: total 100.00, partner share 40.00, receivable acct	Giro 748.00 (−100), Receivable +40.00, Spending +60.00 → 110.00, net worth 888.00
6	Splitwise "I paid": total 100.00, partner share 45.00	Giro 648.00, Splitwise +45.00, Spending +55.00 → 165.00, net worth 833.00
7	Portfolio: buy 2 AAPL @ 100.00 USD on investment acct	Holding row AAPL qty 2, avg cost 100.00 USD; cash leg only if funded-buy used
8	RSU: vest a grant tranche	Holding appears/increments on the rsu account; grant's remaining unvested decreases
9	Refresh prices (Settings) then Dashboard	Portfolio EUR value = qty × cached USD price ÷ EUR/USD rate; no NaN/0 flash
10	Each add above fired the refresh event	Dashboard/Transactions updated without manual reload (fin:transaction-added bus intact)

Any delta mismatch = stop, apply §1: write the mechanism you expected, find the one mechanism explaining all ten observations.

7. Idea lifecycle (methodology-lite)

Every "I think X is broken / X would fix it" follows this pipeline — no skipping stages:

1. Hunch — state it in one falsifiable sentence.
2. Discriminating check defined — pick the observation that distinguishes your hypothesis from the nearest alternative, and write the predicted numbers down first.
3. Experiment — run it on a synthetic DB (test fixtures) or a copied DB (fin-diagnostics-and-tooling). Never prod.
4. Test written — the discriminating check becomes a cli/tests/ test (fails pre-fix, passes post-fix).
5. Change lands — through fin-change-control gates.
6. Archive — if the investigation settled a question about how fin behaves (or used to misbehave), record it in fin-failure-archaeology so the next session doesn't re-derive it.

Provenance and maintenance

Ground truth as of 2026-07-05. Re-verify before relying on volatile facts:

• Test count / inventory: grep -c "def test_" cli/tests/test_cli.py (25) and ls cli/tests/ (only conftest.py + test_cli.py) — re-read new files and update §2 tables when either changes.
• Fixture semantics: re-read cli/tests/conftest.py (autouse clean_db, make_client, yfinance stub, DATABASE_URL set pre-import).
• Run command: confirm against cli/pyproject.toml and fin-build-and-env (commands here were statically derived, not executed — no pip in the authoring sandbox).
• Frontend still testless: ls web/src/**/*.test.* web/vitest* 2>/dev/null should return nothing; if it doesn't, §6 is stale.
• Coverage gaps: cross-check §2 "NOT certified" against fin-test-coverage-campaign progress before repeating its claims.