MarkusandClaude Opus 5
f5dce6ce96
Closes SECURITY_ANALYSIS.md M-14, M-15 and M-16.
M-15: unavailable metrics stayed float("nan") and json.dumps emitted bare NaN,
which RFC 8259 does not permit — the committed report carried 120 of them. They
now serialize as null, and the test named as a CLI writer test invokes the real
CLI and parses with a strict parser instead of hand-writing a different object.
M-14: pick_default compared only overall nDCG and confidence intervals, so a
variant with a catastrophic temporal or update regression could still be
recommended. M-16: each variant got an isolated index but shared the configured
embedding cache, so the first real-model variant paid query embedding latency
that later variants read from cache, making p95 incomparable.
The synthetic smoke report is regenerated against the current code and renamed to
match the date it actually carries. rrf, rrf-inbox-1.0 and rrf-bm25-flat now
score identically, which is expected rather than the shared-state corruption an
earlier report showed: no seeded note is in inbox, and lexical subchannels fuse
by ordinal rank, so a BM25 weight change only moves the score if it reorders.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139vHnNpS2sLNELjrAHX5Cu
2.4 KiB
Eval harness (§11)
mdmem eval --queries eval/queries.yaml --config default # one config
mdmem eval --queries eval/queries.yaml --ablate # the whole matrix
Both write eval/results/<date>.json and print a markdown table plus a
recommendation. The recommendation only names a new default when its nDCG@10
95 % bootstrap CI clears the baseline's — a two-point difference on 50 queries
is noise, not a result.
--ablate covers: lex-only, sem-only, RRF, CC with α ∈ {0.3 … 0.9}, trigram
off, inbox weight 1.0, flat BM25 weights, and embedding dims 1024 / 512. Every
variant runs at the same MAX_CHARS_DEFAULT budget, because a retrieval
comparison at different context volumes measures the volume [R16]. Dim variants
build their own throwaway index and delete it afterwards.
calibrate_min_score() reports the MIN_SCORE that maximises abstention F1 on
the category: abstain queries. It is not applied automatically — set it in the
ConfigMap once you believe it.
Runs so far
| file | what it is | conclusion |
|---|---|---|
results/2026-09-13-synthetic-smoke.json |
18 synthetic queries over the 12 seeded notes in corpus/, run with --fake-embeddings |
The harness works; nothing about the config was learned. Vectors are hashed word bags, not Qwen3, and 18 queries cannot separate fusion variants — every CC arm's 95 % CI overlaps the RRF baseline's. Defaults stay as §2.5 specifies: FUSION=rrf, EMBED_DIM=2560, MIN_SCORE unset. |
rrf, rrf-inbox-1.0 and rrf-bm25-flat report the same nDCG@10 in that run.
That is expected here and is not the shared-state corruption an earlier report had:
none of the 12 seeded notes sit in inbox/, so the inbox weight has nothing to
act on, and the two lexical subchannels are fused by ordinal rank (§8.1), so a
BM25 weight change only moves the score if it reorders the hits — on a corpus this
small, flat weights do not.
The abstention F1 in that run is 0 because MIN_SCORE is unset by default, so
the two abstain queries still return their best (wrong) match. The calibration
block in the report names the threshold that would maximise F1 — on a real,
labelled set it is the number to put in the ConfigMap.
Before tuning anything: label 50–100 real queries in eval/queries.yaml,
run mdmem eval --ablate against the real Bifrost endpoint, and only then move
a default. A variant ships only when its nDCG@10 CI clears the baseline's and no
category regresses.