feat(p5): eval harness — metrics, bootstrap CIs, ablation matrix, MIN_SCORE calibration
Every variant runs at the same char budget; dim variants build and delete their own index. pick_default() only names a new default when its nDCG@10 CI clears the baseline's. First run is synthetic (fake embeddings, 18 queries) and is recorded as a machinery check, not a tuning result — defaults stay RRF/2560.
This commit is contained in:
Markus committed
1 parent
5853b7b1fa
commit
8e5e3e6309
22 files changed
+9244
-4
No files matched your search
Binary file not shown.
@@ -0,0 +1,37 @@
|
||||
# Eval harness (§11)
|
||||
|
||||
```bash
|
||||
mdmem eval --queries eval/queries.yaml --config default # one config
|
||||
mdmem eval --queries eval/queries.yaml --ablate # the whole matrix
|
||||
```
|
||||
|
||||
Both write `eval/results/<date>.json` and print a markdown table plus a
|
||||
recommendation. The recommendation only names a new default when its nDCG@10
|
||||
95 % bootstrap CI clears the baseline's — a two-point difference on 50 queries
|
||||
is noise, not a result.
|
||||
|
||||
`--ablate` covers: lex-only, sem-only, RRF, CC with α ∈ {0.3 … 0.9}, trigram
|
||||
off, inbox weight 1.0, flat BM25 weights, and embedding dims 1024 / 512. Every
|
||||
variant runs at the same `MAX_CHARS_DEFAULT` budget, because a retrieval
|
||||
comparison at different context volumes measures the volume [R16]. Dim variants
|
||||
build their own throwaway index and delete it afterwards.
|
||||
|
||||
`calibrate_min_score()` reports the `MIN_SCORE` that maximises abstention F1 on
|
||||
the `category: abstain` queries. It is not applied automatically — set it in the
|
||||
ConfigMap once you believe it.
|
||||
|
||||
## Runs so far
|
||||
|
||||
| file | what it is | conclusion |
|
||||
|---|---|---|
|
||||
| `results/2026-09-12-synthetic-smoke.json` | 18 synthetic queries over the 12 seeded notes in `corpus/`, run with `--fake-embeddings` | **The harness works; nothing about the config was learned.** Vectors are hashed word bags, not Qwen3, and 18 queries cannot separate fusion variants — every CC arm's 95 % CI overlaps the RRF baseline's. Defaults stay as §2.5 specifies: `FUSION=rrf`, `EMBED_DIM=2560`, `MIN_SCORE` unset. |
|
||||
|
||||
The abstention F1 in that run is 0 because `MIN_SCORE` is unset by default, so
|
||||
the two `abstain` queries still return their best (wrong) match. The calibration
|
||||
block in the report names the threshold that would maximise F1 — on a real,
|
||||
labelled set it is the number to put in the ConfigMap.
|
||||
|
||||
**Before tuning anything**: label 50–100 real queries in `eval/queries.yaml`,
|
||||
run `mdmem eval --ablate` against the real Bifrost endpoint, and only then move
|
||||
a default. A variant ships only when its nDCG@10 CI clears the baseline's and no
|
||||
category regresses.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000BFR
|
||||
title: Bifrost ist das AI-Gateway
|
||||
type: fact
|
||||
summary: Bifrost bündelt alle LLM-Aufrufe und hält die virtuellen Schlüssel.
|
||||
keywords: [bifrost, gateway, virtual key, api gateway, modellrouter]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Bifrost ist der einzige Eingang für LLM-Aufrufe und MCP-Clients. Es steht vor llama-swap.
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000TEM
|
||||
title: Der Modellrouter hieß früher LiteLLM
|
||||
type: fact
|
||||
summary: Vor Bifrost lief LiteLLM als Gateway. Ersetzt.
|
||||
keywords: [litellm, gateway, alt, modellrouter]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: 01M00000000000000000000BFR
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Historisch: LiteLLM war das Gateway bis 2026.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000EVX
|
||||
title: evo-x2
|
||||
type: entity
|
||||
summary: Die GPU-Box im Keller mit 128 GB unified memory.
|
||||
keywords: [evo-x2, gpu, hardware, workstation]
|
||||
tags: [homelab]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
evo-x2 ist die Inferenz-Box unter 10.0.0.59. Beherbergt llama-swap.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000FXC
|
||||
title: FluxCD deployt alles
|
||||
type: fact
|
||||
summary: Jede App kommt über FluxCD-Kustomizations aus dem fluxcd-Repo.
|
||||
keywords: [fluxcd, gitops, kustomization, deployment, ausrollen]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Der Cluster ist RKE2 mit FluxCD. Kein manuelles kubectl apply.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000HRB
|
||||
title: Harbor spiegelt die Images
|
||||
type: fact
|
||||
summary: harbor.cloud.traunseenet.com ist Proxy-Cache und Registry für eigene Images.
|
||||
keywords: [harbor, registry, images, proxy cache, container]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Eigene Images liegen unter library/, gespiegelte unter docker.io/ und ghcr.io/.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000KAF
|
||||
title: Kaffeemaschine entkalken
|
||||
type: procedure
|
||||
summary: Alle drei Monate mit Essigessenz entkalken.
|
||||
keywords: [kaffee, entkalken, wartung, descale, maintenance]
|
||||
tags: [haushalt]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
1. Tank mit 1:4 Essigessenz füllen. 2. Zwei Durchläufe. 3. Zweimal klar spülen.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000SWP
|
||||
title: llama-swap routet lokale Modelle
|
||||
type: fact
|
||||
summary: llama-swap tauscht Modelle auf evo-x2; Bifrost steht davor.
|
||||
keywords: [llama-swap, modellrouter, evo-x2, model router, inference]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
llama-swap läuft auf [[evo-x2]] und tauscht llama-server-Modelle. Bifrost adressiert es als llamaswap/<modell>.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000STG
|
||||
title: Longhorn liefert die Volumes
|
||||
type: fact
|
||||
summary: Longhorn ist die StorageClass im RKE2-Cluster, nicht democratic-csi.
|
||||
keywords: [longhorn, storage, pvc, storageclass, volumes, speicher]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Persistente Volumes kommen von longhorn-storage. RWO, repliziert.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000PKT
|
||||
title: PocketID macht das SSO
|
||||
type: fact
|
||||
summary: Alle Web-UIs authentifizieren sich per OIDC gegen PocketID.
|
||||
keywords: [pocketid, oidc, sso, login, authentifizierung]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Es gibt kein forward-auth im Cluster. Jede App macht OIDC selbst gegen PocketID.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000QWN
|
||||
title: Qwen3-Embedding-4B als Embedder
|
||||
type: decision
|
||||
summary: Q8_0-Quant, last-token pooling, 2560 Dimensionen.
|
||||
keywords: [qwen3, embedding, q8_0, pooling, vektoren, einbettung]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Entschieden: 4B statt 8B (kleiner Abstand) und Q8_0 statt Q4_K (Drift).
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000TRN
|
||||
title: Traunsee Wasserstand
|
||||
type: note
|
||||
summary: Pegel Gmunden liegt im Sommer meist um 422 m.
|
||||
keywords: [traunsee, pegel, wasserstand, lake level, gmunden]
|
||||
tags: [region]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Der Pegel Gmunden schwankt zwischen 421,5 m und 423 m.
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
---
|
||||
id: 01M00000000000000000000VSY
|
||||
title: VolSync sichert stündlich nach Restic
|
||||
type: decision
|
||||
summary: Backups laufen stündlich per VolSync-ReplicationSource in ein Restic-Repo.
|
||||
keywords: [volsync, restic, backup, snapshot, sicherung]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Entschieden am 2026-09-11: stündliche Snapshots statt sechsstündlich, weil der Vault handgeschrieben ist.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Query set for `mdmem eval` (§11).
|
||||
#
|
||||
# Copy to eval/queries.yaml and label 50–100 REAL queries — the ones you would
|
||||
# actually ask your assistant. The harness is only as meaningful as this file.
|
||||
#
|
||||
# Coverage the plan asks for (≥10 each):
|
||||
# lang: de · lang: en · cross-language (German query, English note)
|
||||
# category: temporal — needs since/until/valid_at to answer correctly
|
||||
# category: update — a superseded fact; only the successor may rank
|
||||
# category: abstain — nothing in memory answers it; `relevant: []`
|
||||
#
|
||||
# `relevant` holds note ULIDs. Find them in the UI (the id is on the note page)
|
||||
# or with: grep -l "Suchbegriff" /data/vault/active/*.md
|
||||
#
|
||||
# Fields: q (required), relevant, category, lang, types, tags,
|
||||
# since, until, valid_at, include_superseded.
|
||||
|
||||
- q: "Welcher Router verteilt meine lokalen Modelle?"
|
||||
relevant: [01J9Z4K8X2M3N5P7Q9R1S3T5V7]
|
||||
category: single
|
||||
lang: de
|
||||
|
||||
- q: "Which gateway sits in front of llama-swap?"
|
||||
relevant: [01J9Z4K8X2M3N5P7Q9R1S3T5V7]
|
||||
category: single
|
||||
lang: en # cross-language: the note is German, the query is English
|
||||
|
||||
- q: "Was habe ich im September 2026 über Backups entschieden?"
|
||||
relevant: []
|
||||
category: temporal
|
||||
lang: de
|
||||
since: "2026-09-01T00:00:00Z"
|
||||
until: "2026-10-01T00:00:00Z"
|
||||
|
||||
- q: "Wie heißt der Router jetzt?"
|
||||
relevant: [] # only the successor note may rank; list its id here
|
||||
category: update
|
||||
lang: de
|
||||
|
||||
- q: "Wie ist die Telefonnummer meines Zahnarztes?"
|
||||
relevant: []
|
||||
category: abstain
|
||||
lang: de
|
||||
@@ -0,0 +1,22 @@
|
||||
# Synthetic query set over eval/corpus (12 seeded notes). It exists so the
|
||||
# harness itself can be exercised offline with --fake-embeddings. It is NOT a
|
||||
# tuning set: the vectors are hashed word bags, not Qwen3, and 18 queries over
|
||||
# 12 notes cannot separate fusion variants. Real tuning needs eval/queries.yaml.
|
||||
- {q: "Welcher Router verteilt meine lokalen Modelle?", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
|
||||
- {q: "llama-swap", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
|
||||
- {q: "Which gateway sits in front of llama-swap?", relevant: ["01M00000000000000000000BFR"], category: single, lang: en}
|
||||
- {q: "Wo hält Bifrost die virtuellen Schlüssel?", relevant: ["01M00000000000000000000BFR"], category: single, lang: de}
|
||||
- {q: "Wie heißt die GPU-Box?", relevant: ["01M00000000000000000000EVX"], category: single, lang: de}
|
||||
- {q: "storage class for persistent volumes", relevant: ["01M00000000000000000000STG"], category: single, lang: en}
|
||||
- {q: "Welche StorageClass nutzt der Cluster?", relevant: ["01M00000000000000000000STG"], category: single, lang: de}
|
||||
- {q: "Wie oft laufen die Backups?", relevant: ["01M00000000000000000000VSY"], category: single, lang: de}
|
||||
- {q: "restic snapshot schedule", relevant: ["01M00000000000000000000VSY"], category: single, lang: en}
|
||||
- {q: "Wie melde ich mich bei den Web-UIs an?", relevant: ["01M00000000000000000000PKT"], category: single, lang: de}
|
||||
- {q: "Wie entkalke ich die Kaffeemaschine?", relevant: ["01M00000000000000000000KAF"], category: single, lang: de}
|
||||
- {q: "Pegel am Traunsee", relevant: ["01M00000000000000000000TRN"], category: single, lang: de}
|
||||
- {q: "Wie heißt der Modellrouter jetzt?", relevant: ["01M00000000000000000000BFR"], category: update, lang: de}
|
||||
- {q: "Welches Embedding-Modell und welche Quantisierung?", relevant: ["01M00000000000000000000QWN"], category: multi, lang: de}
|
||||
- {q: "Woher kommen die Container-Images und wer rollt sie aus?", relevant: ["01M00000000000000000000HRB", "01M00000000000000000000FXC"], category: multi, lang: de}
|
||||
- {q: "Was wurde am 11. September 2026 entschieden?", relevant: ["01M00000000000000000000VSY"], category: temporal, lang: de, since: "2026-09-01T00:00:00Z", until: "2026-12-31T00:00:00Z"}
|
||||
- {q: "Wie ist die Telefonnummer meines Zahnarztes?", relevant: [], category: abstain, lang: de}
|
||||
- {q: "Wann fährt der letzte Zug nach Wien?", relevant: [], category: abstain, lang: de}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -31,6 +31,11 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
ev.add_argument("--config", default="default", help="name for this run's config")
|
||||
ev.add_argument("--out", default=None, help="write the JSON report here")
|
||||
ev.add_argument("--ablate", action="store_true", help="run the ablation matrix")
|
||||
ev.add_argument(
|
||||
"--fake-embeddings",
|
||||
action="store_true",
|
||||
help="deterministic offline vectors — checks the harness, not the model",
|
||||
)
|
||||
|
||||
sub.add_parser("doctor", help="check config, index and embedding backend")
|
||||
return p
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
"""Retrieval evaluation (§11).
|
||||
|
||||
Vendor benchmark numbers do not transfer, and a retrieval comparison is
|
||||
meaningless unless the context budget is held constant [R16]. So: one labelled
|
||||
query set, one budget, every variant measured the same way, bootstrap CIs over
|
||||
queries so a two-point difference is not read as a win.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import random
|
||||
import statistics
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field, replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from mdmem.config import Settings
|
||||
from mdmem.service import MemoryService
|
||||
|
||||
log = logging.getLogger("mdmem.eval")
|
||||
|
||||
CATEGORIES = ("single", "multi", "temporal", "update", "abstain")
|
||||
BOOTSTRAP_ROUNDS = 2000
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvalQuery:
|
||||
q: str
|
||||
relevant: list[str] = field(default_factory=list)
|
||||
category: str = "single"
|
||||
lang: str = "de"
|
||||
types: list[str] | None = None
|
||||
tags: list[str] | None = None
|
||||
since: str | None = None
|
||||
until: str | None = None
|
||||
valid_at: str | None = None
|
||||
include_superseded: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class Variant:
|
||||
name: str
|
||||
fusion: str | None = None
|
||||
cc_alpha: float | None = None
|
||||
channels: tuple[str, ...] = ("lex", "sem")
|
||||
trigram: bool | None = None
|
||||
inbox_weight: float | None = None
|
||||
bm25_weights: tuple[float, float, float, float] | None = None
|
||||
embed_dim: int | None = None # requires a rebuilt index
|
||||
|
||||
@property
|
||||
def needs_rebuild(self) -> bool:
|
||||
return self.embed_dim is not None
|
||||
|
||||
|
||||
def load_queries(path: Path | str) -> list[EvalQuery]:
|
||||
raw = yaml.safe_load(Path(path).read_text(encoding="utf-8")) or []
|
||||
out: list[EvalQuery] = []
|
||||
for i, item in enumerate(raw):
|
||||
if "q" not in item:
|
||||
raise ValueError(f"query #{i} has no 'q'")
|
||||
cat = item.get("category", "single")
|
||||
if cat not in CATEGORIES:
|
||||
raise ValueError(f"query #{i} has unknown category {cat!r}, expected one of {CATEGORIES}")
|
||||
out.append(
|
||||
EvalQuery(
|
||||
q=item["q"],
|
||||
relevant=list(item.get("relevant") or []),
|
||||
category=cat,
|
||||
lang=item.get("lang", "de"),
|
||||
types=item.get("types"),
|
||||
tags=item.get("tags"),
|
||||
since=item.get("since"),
|
||||
until=item.get("until"),
|
||||
valid_at=item.get("valid_at"),
|
||||
include_superseded=bool(item.get("include_superseded", False)),
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# -- metrics ---------------------------------------------------------------
|
||||
def recall_at(ranked: list[str], relevant: list[str], k: int) -> float:
|
||||
if not relevant:
|
||||
return float("nan")
|
||||
hits = len(set(ranked[:k]) & set(relevant))
|
||||
return hits / len(relevant)
|
||||
|
||||
|
||||
def mrr_at(ranked: list[str], relevant: list[str], k: int = 10) -> float:
|
||||
if not relevant:
|
||||
return float("nan")
|
||||
for i, note_id in enumerate(ranked[:k], start=1):
|
||||
if note_id in relevant:
|
||||
return 1.0 / i
|
||||
return 0.0
|
||||
|
||||
|
||||
def ndcg_at(ranked: list[str], relevant: list[str], k: int = 10) -> float:
|
||||
if not relevant:
|
||||
return float("nan")
|
||||
dcg = sum(1.0 / math.log2(i + 1) for i, note_id in enumerate(ranked[:k], start=1) if note_id in relevant)
|
||||
ideal = sum(1.0 / math.log2(i + 1) for i in range(1, min(len(relevant), k) + 1))
|
||||
return dcg / ideal if ideal else 0.0
|
||||
|
||||
|
||||
def _mean(values: list[float]) -> float | None:
|
||||
clean = [v for v in values if not math.isnan(v)]
|
||||
return round(statistics.fmean(clean), 4) if clean else None
|
||||
|
||||
|
||||
def bootstrap_ci(values: list[float], rounds: int = BOOTSTRAP_ROUNDS, seed: int = 7) -> list[float] | None:
|
||||
clean = [v for v in values if not math.isnan(v)]
|
||||
if len(clean) < 2:
|
||||
return None
|
||||
rng = random.Random(seed)
|
||||
means = []
|
||||
n = len(clean)
|
||||
for _ in range(rounds):
|
||||
means.append(statistics.fmean(rng.choices(clean, k=n)))
|
||||
means.sort()
|
||||
lo = means[int(0.025 * rounds)]
|
||||
hi = means[int(0.975 * rounds) - 1]
|
||||
return [round(lo, 4), round(hi, 4)]
|
||||
|
||||
|
||||
# -- runner ----------------------------------------------------------------
|
||||
async def run_variant(
|
||||
svc: MemoryService, queries: list[EvalQuery], variant: Variant, max_chars: int
|
||||
) -> dict[str, Any]:
|
||||
per_query: list[dict[str, Any]] = []
|
||||
latencies: list[float] = []
|
||||
for query in queries:
|
||||
started = time.monotonic()
|
||||
resp = await svc.search(
|
||||
query.q,
|
||||
k=10,
|
||||
types=query.types,
|
||||
tags=query.tags,
|
||||
since=query.since,
|
||||
until=query.until,
|
||||
valid_at=query.valid_at,
|
||||
include_superseded=query.include_superseded,
|
||||
max_chars=max_chars,
|
||||
fusion=variant.fusion,
|
||||
cc_alpha=variant.cc_alpha,
|
||||
inbox_weight=variant.inbox_weight,
|
||||
channels=variant.channels,
|
||||
)
|
||||
latencies.append((time.monotonic() - started) * 1000)
|
||||
ranked = [r.id for r in resp.results]
|
||||
per_query.append(
|
||||
{
|
||||
"q": query.q,
|
||||
"category": query.category,
|
||||
"lang": query.lang,
|
||||
"ranked": ranked[:10],
|
||||
"recall@5": recall_at(ranked, query.relevant, 5),
|
||||
"recall@10": recall_at(ranked, query.relevant, 10),
|
||||
"mrr@10": mrr_at(ranked, query.relevant),
|
||||
"ndcg@10": ndcg_at(ranked, query.relevant),
|
||||
"is_abstain_query": not query.relevant,
|
||||
"abstained": not ranked,
|
||||
"chars": sum(len(r.snippet) for r in resp.results),
|
||||
"top_score": resp.results[0].score if resp.results else None,
|
||||
}
|
||||
)
|
||||
|
||||
report: dict[str, Any] = {
|
||||
"variant": variant.name,
|
||||
"config": {k: v for k, v in asdict(variant).items() if v is not None and k != "name"},
|
||||
"n_queries": len(queries),
|
||||
"overall": _aggregate(per_query),
|
||||
"per_category": {
|
||||
cat: _aggregate([p for p in per_query if p["category"] == cat])
|
||||
for cat in CATEGORIES
|
||||
if any(p["category"] == cat for p in per_query)
|
||||
},
|
||||
"abstention": _abstention(per_query),
|
||||
"mean_chars": round(statistics.fmean([p["chars"] for p in per_query]), 1) if per_query else 0,
|
||||
"latency_ms_p95": round(_p95(latencies), 2),
|
||||
"queries": per_query,
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
def _aggregate(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
if not rows:
|
||||
return {}
|
||||
out: dict[str, Any] = {}
|
||||
for metric in ("recall@5", "recall@10", "mrr@10", "ndcg@10"):
|
||||
values = [r[metric] for r in rows]
|
||||
out[metric] = _mean(values)
|
||||
ci = bootstrap_ci(values)
|
||||
if ci:
|
||||
out[f"{metric}_ci95"] = ci
|
||||
return out
|
||||
|
||||
|
||||
def _abstention(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""F1 of 'returned nothing' against the queries labelled with no relevant note."""
|
||||
tp = fp = fn = 0
|
||||
for row in rows:
|
||||
should = row["is_abstain_query"]
|
||||
did = row["abstained"]
|
||||
if should and did:
|
||||
tp += 1
|
||||
elif not should and did:
|
||||
fp += 1
|
||||
elif should and not did:
|
||||
fn += 1
|
||||
precision = tp / (tp + fp) if (tp + fp) else 0.0
|
||||
recall = tp / (tp + fn) if (tp + fn) else 0.0
|
||||
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
|
||||
return {
|
||||
"n_abstain_queries": tp + fn,
|
||||
"precision": round(precision, 4),
|
||||
"recall": round(recall, 4),
|
||||
"f1": round(f1, 4),
|
||||
}
|
||||
|
||||
|
||||
def _p95(values: list[float]) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
ordered = sorted(values)
|
||||
return ordered[min(len(ordered) - 1, int(0.95 * len(ordered)))]
|
||||
|
||||
|
||||
def calibrate_min_score(report: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Pick the MIN_SCORE that maximises abstention F1 without dropping answerable queries."""
|
||||
rows = report["queries"]
|
||||
thresholds = sorted({round(r["top_score"], 6) for r in rows if r["top_score"] is not None})
|
||||
best = {"min_score": None, "f1": _f1_at(rows, None)}
|
||||
for t in thresholds:
|
||||
f1 = _f1_at(rows, t)
|
||||
if f1 > best["f1"]:
|
||||
best = {"min_score": t, "f1": f1}
|
||||
return best
|
||||
|
||||
|
||||
def _f1_at(rows: list[dict[str, Any]], threshold: float | None) -> float:
|
||||
tp = fp = fn = 0
|
||||
for row in rows:
|
||||
top = row["top_score"]
|
||||
abstained = top is None or (threshold is not None and top < threshold)
|
||||
should = row["is_abstain_query"]
|
||||
if should and abstained:
|
||||
tp += 1
|
||||
elif not should and abstained:
|
||||
fp += 1
|
||||
elif should and not abstained:
|
||||
fn += 1
|
||||
precision = tp / (tp + fp) if (tp + fp) else 0.0
|
||||
recall = tp / (tp + fn) if (tp + fn) else 0.0
|
||||
return 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
|
||||
|
||||
|
||||
DEFAULT_VARIANTS: list[Variant] = [
|
||||
Variant("rrf", fusion="rrf"),
|
||||
Variant("lex-only", fusion="rrf", channels=("lex",)),
|
||||
Variant("sem-only", fusion="rrf", channels=("sem",)),
|
||||
*[Variant(f"cc-a{a}", fusion="cc", cc_alpha=a) for a in (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)],
|
||||
Variant("rrf-no-trigram", fusion="rrf", trigram=False),
|
||||
Variant("rrf-inbox-1.0", fusion="rrf", inbox_weight=1.0),
|
||||
Variant("rrf-bm25-flat", fusion="rrf", bm25_weights=(1.0, 1.0, 1.0, 1.0)),
|
||||
Variant("dim-1024", fusion="rrf", embed_dim=1024),
|
||||
Variant("dim-512", fusion="rrf", embed_dim=512),
|
||||
]
|
||||
|
||||
|
||||
async def run_matrix(
|
||||
settings: Settings,
|
||||
queries: list[EvalQuery],
|
||||
variants: list[Variant],
|
||||
*,
|
||||
embedder_factory: Any = None,
|
||||
max_chars: int = 6000,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Every variant is measured at the same char budget (§11, [R16])."""
|
||||
reports: list[dict[str, Any]] = []
|
||||
for variant in variants:
|
||||
run_settings = settings
|
||||
if variant.trigram is not None:
|
||||
run_settings = replace(run_settings, fts_trigram=variant.trigram)
|
||||
if variant.bm25_weights is not None:
|
||||
run_settings = replace(run_settings, bm25_weights=variant.bm25_weights)
|
||||
if variant.embed_dim is not None:
|
||||
run_settings = replace(
|
||||
run_settings,
|
||||
embed_dim=variant.embed_dim,
|
||||
index_db=settings.index_db.with_name(f"eval-dim{variant.embed_dim}.db"),
|
||||
)
|
||||
rebuild = variant.needs_rebuild or variant.trigram is not None
|
||||
embedder = embedder_factory(run_settings) if embedder_factory else None
|
||||
svc = MemoryService.build(run_settings, embedder=embedder)
|
||||
try:
|
||||
await svc.start(watch=False, embed=True)
|
||||
if rebuild:
|
||||
await svc.reindex(full=True)
|
||||
reports.append(await run_variant(svc, queries, variant, max_chars))
|
||||
finally:
|
||||
await svc.stop()
|
||||
if variant.embed_dim is not None:
|
||||
run_settings.index_db.unlink(missing_ok=True)
|
||||
for suffix in ("-wal", "-shm"):
|
||||
Path(str(run_settings.index_db) + suffix).unlink(missing_ok=True)
|
||||
return reports
|
||||
|
||||
|
||||
def render_table(reports: list[dict[str, Any]]) -> str:
|
||||
head = (
|
||||
"| variant | R@5 | R@10 | MRR@10 | nDCG@10 | abstain F1 | mean chars | p95 ms |\n"
|
||||
"|---|---|---|---|---|---|---|---|\n"
|
||||
)
|
||||
|
||||
def fmt(o: dict[str, Any], key: str) -> str:
|
||||
v = o.get(key)
|
||||
if v is None:
|
||||
return "–"
|
||||
ci = o.get(f"{key}_ci95")
|
||||
return f"{v:.3f} [{ci[0]:.3f},{ci[1]:.3f}]" if ci else f"{v:.3f}"
|
||||
|
||||
rows = []
|
||||
for r in reports:
|
||||
o = r["overall"]
|
||||
rows.append(
|
||||
f"| {r['variant']} | {fmt(o, 'recall@5')} | {fmt(o, 'recall@10')} | "
|
||||
f"{fmt(o, 'mrr@10')} | {fmt(o, 'ndcg@10')} | "
|
||||
f"{r['abstention']['f1']:.3f} | {r['mean_chars']:.0f} | "
|
||||
f"{r['latency_ms_p95']:.1f} |"
|
||||
)
|
||||
return head + "\n".join(rows)
|
||||
|
||||
|
||||
def pick_default(reports: list[dict[str, Any]], baseline: str = "rrf") -> dict[str, Any]:
|
||||
"""Ship a new default only if it beats the baseline outside the noise band (§11)."""
|
||||
base = next((r for r in reports if r["variant"] == baseline), None)
|
||||
if base is None:
|
||||
return {"recommendation": baseline, "reason": "baseline missing from the run"}
|
||||
base_score = base["overall"].get("ndcg@10") or 0.0
|
||||
base_ci = base["overall"].get("ndcg@10_ci95") or [base_score, base_score]
|
||||
best = base
|
||||
for r in reports:
|
||||
score = r["overall"].get("ndcg@10") or 0.0
|
||||
ci = r["overall"].get("ndcg@10_ci95") or [score, score]
|
||||
if score > (best["overall"].get("ndcg@10") or 0.0) and ci[0] > base_ci[1]:
|
||||
best = r
|
||||
if best is base:
|
||||
return {
|
||||
"recommendation": baseline,
|
||||
"reason": "no variant beat the baseline with non-overlapping 95 % CIs",
|
||||
"baseline_ndcg@10": base_score,
|
||||
}
|
||||
return {
|
||||
"recommendation": best["variant"],
|
||||
"reason": "beats the baseline with non-overlapping 95 % CIs",
|
||||
"baseline_ndcg@10": base_score,
|
||||
"variant_ndcg@10": best["overall"].get("ndcg@10"),
|
||||
}
|
||||
|
||||
|
||||
def run_eval_cli(settings: Settings, args: Any) -> int:
|
||||
queries_path = Path(args.queries)
|
||||
if not queries_path.exists():
|
||||
print(
|
||||
f"no query set at {queries_path}. Copy eval/queries.example.yaml and label "
|
||||
"50–100 real queries first (§11)."
|
||||
)
|
||||
return 2
|
||||
queries = load_queries(queries_path)
|
||||
|
||||
async def _main() -> int:
|
||||
variants = DEFAULT_VARIANTS if args.ablate else [Variant(args.config, fusion=settings.fusion)]
|
||||
factory = None
|
||||
if getattr(args, "fake_embeddings", False):
|
||||
from mdmem.embed import FakeEmbedder
|
||||
|
||||
log.warning(
|
||||
"running with deterministic fake embeddings: this measures the harness and the "
|
||||
"retrieval machinery, not the production model"
|
||||
)
|
||||
factory = lambda s: FakeEmbedder(s.embed_dim, model=f"fake-{s.embed_dim}") # noqa: E731
|
||||
reports = await run_matrix(
|
||||
settings,
|
||||
queries,
|
||||
variants,
|
||||
embedder_factory=factory,
|
||||
max_chars=settings.max_chars_default,
|
||||
)
|
||||
table = render_table(reports)
|
||||
decision = pick_default(reports)
|
||||
baseline = next((r for r in reports if r["variant"] == reports[0]["variant"]), reports[0])
|
||||
min_score = calibrate_min_score(baseline)
|
||||
print(table)
|
||||
print()
|
||||
print(json.dumps({"decision": decision, "min_score": min_score}, indent=2, ensure_ascii=False))
|
||||
out = Path(args.out) if args.out else Path("eval/results") / f"{time.strftime('%Y-%m-%d')}.json"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"date": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"embed_model": settings.embed_model,
|
||||
"embed_dim": settings.embed_dim,
|
||||
"max_chars": settings.max_chars_default,
|
||||
"n_queries": len(queries),
|
||||
"decision": decision,
|
||||
"min_score_calibration": min_score,
|
||||
"reports": reports,
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(f"\nwrote {out}")
|
||||
return 0
|
||||
|
||||
return asyncio.run(_main())
|
||||
+4
-3
@@ -99,6 +99,7 @@ def search(
|
||||
max_chars: int | None = None,
|
||||
min_score: float | None = None,
|
||||
fusion: str | None = None,
|
||||
channels: tuple[str, ...] = ("lex", "sem"),
|
||||
cc_alpha: float | None = None,
|
||||
inbox_weight: float | None = None,
|
||||
debug: bool = False,
|
||||
@@ -118,7 +119,7 @@ def search(
|
||||
|
||||
# 1. lexical -----------------------------------------------------------
|
||||
lex: dict[int, float] = {}
|
||||
match = to_match(query)
|
||||
match = to_match(query) if "lex" in channels else None
|
||||
if match:
|
||||
w = settings.bm25_weights
|
||||
rows = conn.execute(
|
||||
@@ -128,7 +129,7 @@ def search(
|
||||
).fetchall()
|
||||
for r in rows:
|
||||
lex[r["cid"]] = -float(r["s"]) # bm25() is negative; larger is better here
|
||||
if settings.fts_trigram:
|
||||
if settings.fts_trigram and "lex" in channels:
|
||||
tri = to_trigram_match(query)
|
||||
if tri and _has_table(conn, "chunks_tri"):
|
||||
rows = conn.execute(
|
||||
@@ -141,7 +142,7 @@ def search(
|
||||
|
||||
# 2. dense -------------------------------------------------------------
|
||||
sem: dict[int, float] = {}
|
||||
if query_vec is not None:
|
||||
if query_vec is not None and "sem" in channels:
|
||||
rows = conn.execute(
|
||||
"SELECT rowid AS cid, distance FROM chunks_vec "
|
||||
"WHERE embedding MATCH ? AND k = ? ORDER BY distance",
|
||||
|
||||
@@ -213,7 +213,7 @@ class MemoryService:
|
||||
vec: bytes | None = None
|
||||
from mdmem.ftsquery import to_match
|
||||
|
||||
if self.embedder is not None and to_match(query):
|
||||
if self.embedder is not None and to_match(query) and "sem" in kw.get("channels", ("lex", "sem")):
|
||||
try:
|
||||
v = await self.embedder.embed([query], is_query=True)
|
||||
vec = to_blob(v[0])
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
import json
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from mdmem.embed import FakeEmbedder
|
||||
from mdmem.evalharness import (
|
||||
EvalQuery,
|
||||
Variant,
|
||||
bootstrap_ci,
|
||||
calibrate_min_score,
|
||||
load_queries,
|
||||
mrr_at,
|
||||
ndcg_at,
|
||||
pick_default,
|
||||
recall_at,
|
||||
render_table,
|
||||
run_matrix,
|
||||
run_variant,
|
||||
)
|
||||
from mdmem.service import MemoryService
|
||||
|
||||
|
||||
def test_recall_mrr_ndcg():
|
||||
ranked = ["a", "b", "c"]
|
||||
assert recall_at(ranked, ["a", "z"], 5) == 0.5
|
||||
assert recall_at(ranked, ["a"], 1) == 1.0
|
||||
assert mrr_at(ranked, ["b"]) == 0.5
|
||||
assert mrr_at(ranked, ["z"]) == 0.0
|
||||
assert ndcg_at(ranked, ["a"]) == 1.0
|
||||
assert 0 < ndcg_at(ranked, ["c"]) < 1
|
||||
|
||||
|
||||
def test_metrics_are_nan_for_abstain_queries():
|
||||
import math
|
||||
|
||||
assert math.isnan(recall_at(["a"], [], 5))
|
||||
assert math.isnan(mrr_at(["a"], []))
|
||||
|
||||
|
||||
def test_bootstrap_ci_brackets_the_mean():
|
||||
values = [0.2, 0.4, 0.6, 0.8, 1.0]
|
||||
lo, hi = bootstrap_ci(values, rounds=500)
|
||||
assert lo < 0.6 < hi
|
||||
|
||||
|
||||
def test_load_queries_validates(tmp_path):
|
||||
p = tmp_path / "q.yaml"
|
||||
p.write_text("- q: hallo\n relevant: [01J]\n category: single\n", encoding="utf-8")
|
||||
qs = load_queries(p)
|
||||
assert qs[0].q == "hallo" and qs[0].relevant == ["01J"]
|
||||
|
||||
p.write_text("- relevant: []\n", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="no 'q'"):
|
||||
load_queries(p)
|
||||
|
||||
p.write_text("- q: x\n category: nonsense\n", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="unknown category"):
|
||||
load_queries(p)
|
||||
|
||||
|
||||
def test_the_shipped_example_query_set_parses():
|
||||
qs = load_queries("eval/queries.example.yaml")
|
||||
assert len(qs) >= 5
|
||||
assert {q.category for q in qs} >= {"single", "temporal", "update", "abstain"}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
async def seeded(settings):
|
||||
svc = MemoryService.build(settings, embedder=FakeEmbedder(settings.embed_dim))
|
||||
await svc.start(watch=False)
|
||||
ids = {}
|
||||
for title, body, kw in [
|
||||
("llama-swap routet Modelle", "Bifrost fronts llama-swap auf evo-x2", ["router"]),
|
||||
("Kaffeemaschine entkalken", "alle drei Monate mit Essig entkalken", ["kaffee"]),
|
||||
("Longhorn Storage", "Longhorn liefert RWO Volumes im Cluster", ["storage"]),
|
||||
]:
|
||||
r = await svc.write_note(title=title, body=body, type="fact", summary=title, keywords=kw)
|
||||
await svc.set_state(r["id"], "active")
|
||||
ids[title] = r["id"]
|
||||
yield svc, ids
|
||||
await svc.stop()
|
||||
|
||||
|
||||
async def test_run_variant_scores_a_query_set(seeded):
|
||||
svc, ids = seeded
|
||||
queries = [
|
||||
EvalQuery(q="llama-swap", relevant=[ids["llama-swap routet Modelle"]], category="single"),
|
||||
EvalQuery(q="entkalken", relevant=[ids["Kaffeemaschine entkalken"]], category="single"),
|
||||
EvalQuery(q="Zahnarzttermin nächste Woche", relevant=[], category="abstain"),
|
||||
]
|
||||
report = await run_variant(svc, queries, Variant("rrf", fusion="rrf"), max_chars=6000)
|
||||
assert report["n_queries"] == 3
|
||||
assert report["overall"]["recall@5"] == 1.0
|
||||
assert report["overall"]["recall@5_ci95"]
|
||||
assert report["abstention"]["n_abstain_queries"] == 1
|
||||
assert report["latency_ms_p95"] > 0
|
||||
assert report["mean_chars"] > 0
|
||||
|
||||
|
||||
async def test_every_variant_runs_at_the_same_budget(seeded):
|
||||
svc, ids = seeded
|
||||
queries = [EvalQuery(q="llama-swap Modelle", relevant=[ids["llama-swap routet Modelle"]])]
|
||||
reports = []
|
||||
for v in (Variant("rrf", fusion="rrf"), Variant("lex", fusion="rrf", channels=("lex",)),
|
||||
Variant("sem", fusion="rrf", channels=("sem",)), Variant("cc", fusion="cc", cc_alpha=0.5)):
|
||||
reports.append(await run_variant(svc, queries, v, max_chars=500))
|
||||
assert all(r["mean_chars"] <= 800 for r in reports)
|
||||
assert {r["variant"] for r in reports} == {"rrf", "lex", "sem", "cc"}
|
||||
table = render_table(reports)
|
||||
assert "| rrf |" in table and "R@5" in table
|
||||
|
||||
|
||||
async def test_sem_only_still_ranks_without_lexical_overlap(seeded):
|
||||
svc, ids = seeded
|
||||
q = [EvalQuery(q="alle drei Monate mit Essig entkalken", relevant=[ids["Kaffeemaschine entkalken"]])]
|
||||
report = await run_variant(svc, q, Variant("sem", fusion="rrf", channels=("sem",)), max_chars=6000)
|
||||
assert report["overall"]["recall@5"] == 1.0
|
||||
|
||||
|
||||
async def test_matrix_rebuilds_for_a_dim_variant_and_cleans_up(settings, seeded):
|
||||
svc, ids = seeded
|
||||
queries = [EvalQuery(q="Longhorn", relevant=[ids["Longhorn Storage"]])]
|
||||
reports = await run_matrix(
|
||||
settings,
|
||||
queries,
|
||||
[Variant("rrf", fusion="rrf"), Variant("dim-8", fusion="rrf", embed_dim=8)],
|
||||
embedder_factory=lambda s: FakeEmbedder(s.embed_dim),
|
||||
max_chars=6000,
|
||||
)
|
||||
assert len(reports) == 2
|
||||
assert reports[1]["config"]["embed_dim"] == 8
|
||||
assert not settings.index_db.with_name("eval-dim8.db").exists()
|
||||
|
||||
|
||||
def test_pick_default_keeps_the_baseline_when_nothing_clears_the_noise():
|
||||
reports = [
|
||||
{"variant": "rrf", "overall": {"ndcg@10": 0.70, "ndcg@10_ci95": [0.60, 0.80]}},
|
||||
{"variant": "cc-a0.5", "overall": {"ndcg@10": 0.74, "ndcg@10_ci95": [0.64, 0.84]}},
|
||||
]
|
||||
out = pick_default(reports)
|
||||
assert out["recommendation"] == "rrf" and "overlap" in out["reason"]
|
||||
|
||||
|
||||
def test_pick_default_switches_on_a_clear_win():
|
||||
reports = [
|
||||
{"variant": "rrf", "overall": {"ndcg@10": 0.50, "ndcg@10_ci95": [0.45, 0.55]}},
|
||||
{"variant": "cc-a0.5", "overall": {"ndcg@10": 0.80, "ndcg@10_ci95": [0.75, 0.85]}},
|
||||
]
|
||||
assert pick_default(reports)["recommendation"] == "cc-a0.5"
|
||||
|
||||
|
||||
def test_min_score_calibration_finds_a_separating_threshold():
|
||||
report = {
|
||||
"queries": [
|
||||
{"is_abstain_query": False, "top_score": 0.9, "ranked": ["a"], "recall@10": 1.0},
|
||||
{"is_abstain_query": False, "top_score": 0.8, "ranked": ["b"], "recall@10": 1.0},
|
||||
{"is_abstain_query": True, "top_score": 0.2, "ranked": ["c"], "recall@10": float("nan")},
|
||||
{"is_abstain_query": True, "top_score": 0.1, "ranked": ["d"], "recall@10": float("nan")},
|
||||
]
|
||||
}
|
||||
best = calibrate_min_score(report)
|
||||
assert best["f1"] == 1.0 and 0.2 < best["min_score"] <= 0.8
|
||||
|
||||
|
||||
async def test_cli_reports_and_writes_a_json_file(settings, seeded, tmp_path, capsys):
|
||||
svc, ids = seeded
|
||||
qfile = tmp_path / "q.yaml"
|
||||
qfile.write_text(
|
||||
f"- q: llama-swap\n relevant: [{ids['llama-swap routet Modelle']}]\n category: single\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
out = tmp_path / "report.json"
|
||||
|
||||
from mdmem.evalharness import run_matrix as _rm
|
||||
|
||||
reports = await _rm(
|
||||
replace(settings),
|
||||
load_queries(qfile),
|
||||
[Variant("rrf", fusion="rrf")],
|
||||
embedder_factory=lambda s: FakeEmbedder(s.embed_dim),
|
||||
)
|
||||
out.write_text(json.dumps({"reports": reports}), encoding="utf-8")
|
||||
assert json.loads(out.read_text())["reports"][0]["overall"]["recall@5"] == 1.0
|
||||
Reference in new issue
Block a user