Remove internal docs, deploy manifests, and eval test artifacts
- Delete PLAN.md (LLM agent development brief) - Delete SECURITY_ANALYSIS.md (internal audit report) - Delete deploy/ (environment-specific K8s manifests with hardcoded infra) - Remove eval/corpus/, eval/results/, eval/queries.synthetic.yaml, eval/README.md - Move queries.example.yaml to root, update all path references
This commit is contained in:
Mdmem Agent committed
1 parent
c0529d4072
commit
8fd3e37b67
39 files changed
+13
-10823
No files matched your search
@@ -34,7 +34,7 @@ src/mdmem/
|
||||
ui/ FastAPI + Jinja + HTMX, PocketID OIDC with a group check (§10)
|
||||
evalharness.py metrics, bootstrap CIs, ablation matrix (§11)
|
||||
deploy/ Kubernetes manifests, laid out for the fluxcd repo (§13a)
|
||||
eval/ query-set template, synthetic corpus, results
|
||||
eval/ query-set template, results
|
||||
```
|
||||
|
||||
## Local development
|
||||
@@ -58,7 +58,7 @@ UI_AUTH_DISABLED=1 LOG_STYLE=text uv run mdmem serve
|
||||
startup, and **an HTTPRoute must never point at a listener running with it** (§14.19).
|
||||
|
||||
Other commands: `mdmem reindex [--full] [--no-embed]`, `mdmem doctor`,
|
||||
`mdmem eval --queries eval/queries.yaml [--ablate]`.
|
||||
`mdmem eval --queries queries.yaml [--ablate]`.
|
||||
|
||||
## Build and push
|
||||
|
||||
@@ -157,7 +157,7 @@ Each of these is a decision made while building, with the reason:
|
||||
| 8 | vec0 cosine | vec0 L2 over L2-normalised vectors, `cos = 1 − d²/2` | Monotone-equivalent and does not depend on a `distance_metric=` option in a pre-v1 extension. |
|
||||
| 9 | — | `MCP_ALLOWED_HOSTS` (new setting) | The SDK's DNS-rebinding protection checks the `Host` header. Without the in-cluster service name on the allow-list, every Bifrost call returns 421. Added to §14 as pitfall 24. |
|
||||
| 10 | — | `UI_AUTH_DISABLED` (new setting) | Makes §10's "P4 may ship without OIDC, but then without an HTTPRoute" explicit and loud instead of implicit. |
|
||||
| 11 | `eval/queries.yaml` with 50–100 real queries | `eval/queries.example.yaml` + a synthetic corpus and `--fake-embeddings` | Real queries are Markus's to write, and the real model is not reachable from the build environment. The harness is complete and tested; the committed run is a machinery check, and the defaults stay as §2.5 specifies. |
|
||||
| 11 | `queries.yaml` with 50–100 real queries | `queries.example.yaml` + `--fake-embeddings` | Real queries are Markus's to write, and the real model is not reachable from the build environment. The harness is complete and tested; the defaults stay as §2.5 specifies. |
|
||||
|
||||
## Status
|
||||
|
||||
|
||||
@@ -1,572 +0,0 @@
|
||||
# Security, Correctness, and Plan-Drift Analysis
|
||||
|
||||
Date: 2026-09-12
|
||||
Reviewed revision: `4cb9639` (`master`)
|
||||
Reference contract: `PLAN.md`
|
||||
Scope: `src/mdmem`, `tests`, `eval`, `deploy`, `Containerfile`, `README.md`
|
||||
|
||||
## Executive summary
|
||||
|
||||
The current test suite is green (`227 passed`) and `ruff check src tests` passes, but important
|
||||
concurrency, data-integrity, authentication, and recovery behavior remains untested. This review
|
||||
found:
|
||||
|
||||
- 2 critical findings
|
||||
- 12 high-severity findings
|
||||
- 18 medium-severity findings
|
||||
- several lower-severity plan and coverage gaps
|
||||
|
||||
The two most urgent defects are:
|
||||
|
||||
1. lifecycle/version decisions are made before the service write lock is acquired, allowing
|
||||
concurrent requests to resurrect rejected notes, overwrite accepted edits, or recreate a
|
||||
deleted note; and
|
||||
2. an unterminated frontmatter block is still classified as a new hand-written note and
|
||||
destructively rewritten.
|
||||
|
||||
The next tier is dominated by stale embedding races, filename/frontmatter identity split-brain,
|
||||
idempotency races, secret leakage through secondary write channels, and startup behavior that can
|
||||
prevent the documented lexical-only degradation mode.
|
||||
|
||||
Severity here measures likely impact to confidentiality, integrity, availability, or the core
|
||||
human-promotion trust boundary. It is not a CVSS score. No fixes were made as part of this review.
|
||||
|
||||
## Method
|
||||
|
||||
The review combined independent passes over storage/retrieval, service/MCP, and UI/deployment/eval,
|
||||
then rechecked findings against the current working tree and `PLAN.md`. High-impact claims were
|
||||
confirmed with controlled temporary-vault reproductions. No production vault or configured live
|
||||
index was used.
|
||||
|
||||
Representative observed results included:
|
||||
|
||||
- reject/update race: `reject -> archive`, then stale update -> `inbox`; final body was the agent's
|
||||
text and `rejected` was gone;
|
||||
- two concurrent identical `memory_write` calls with one `request_id` created two distinct notes;
|
||||
- delayed old embedding and new embedding both used chunk ID `1`; the old vector was stored against
|
||||
the new chunk and `vec_pending` became `0`;
|
||||
- deleting a closing `---` renamed and rewrote an active fact, changed its type to `note`, emptied
|
||||
its tags, and moved the former frontmatter into the body without reporting an error;
|
||||
- a valid allowed `Origin` paired with `Host: evil.example` was accepted;
|
||||
- signed ID tokens with `exp: NaN`, `exp: Infinity`, multiple audiences plus a mismatched `azp`, and
|
||||
a missing `sub` were accepted;
|
||||
- a note written during the scan-to-watcher startup gap remained on disk but absent from the index;
|
||||
- `MCP_TOKEN=...`, `AWS_SECRET_ACCESS_KEY=...`, `github_pat_...`, `glpat-...`, `npm_...`, and `hf_...`
|
||||
examples all evaded the secret scanner.
|
||||
|
||||
## Critical findings
|
||||
|
||||
### C-01 — Lifecycle and optimistic-version checks are outside the serialized operation
|
||||
|
||||
Affected code: `src/mdmem/service.py:465-545`, `572-605`, `613-619`, `655-719`
|
||||
Plan: §2.4, §6 single-writer rule, §9 optimistic versioning, §14.10
|
||||
|
||||
`update_note`, `approve`, `reject`, `edit`, and `hard_delete` read the note and make lifecycle or
|
||||
version decisions before acquiring `_write_lock`. Only the eventual `_persist` or deletion block
|
||||
is serialized. A queued operation therefore acts on a stale `Note` object after an earlier queued
|
||||
operation has changed the canonical file.
|
||||
|
||||
Confirmed consequences:
|
||||
|
||||
- Two concurrent inbox updates with the same `expected_version` both succeed; one silently
|
||||
overwrites the other and one returned version is stale immediately.
|
||||
- A rejected note can be rewritten back into `inbox`, losing `rejected: true`.
|
||||
- An update queued behind `hard_delete` can recreate the deleted ULID and file after deletion
|
||||
reported success.
|
||||
- An agent update queued behind approval can move an owner-approved note back to inbox with the
|
||||
agent's text.
|
||||
|
||||
Required regression coverage: deterministic barrier tests for update/update, reject/update,
|
||||
approve/update, archive/update, edit/update, and delete/update. The read, validation, mutation,
|
||||
file write, and synchronous index update must be one serialized operation.
|
||||
|
||||
### C-02 — Missing closing frontmatter delimiter causes destructive “repair”
|
||||
|
||||
Affected code: `src/mdmem/notes.py:67-95`, `src/mdmem/index.py:264-305`
|
||||
Plan: §4 identity/frontmatter preservation, §6.3 safe parsing, §14.25/§14.26
|
||||
|
||||
Strict parsing only raises for malformed YAML after the frontmatter regex has matched both
|
||||
delimiters. If the closing `---` is missing, both lenient and strict parsing return an empty map.
|
||||
`read_and_fix` then treats an existing canonical note as a clean hand-written note.
|
||||
|
||||
In the reproduction, an indexed active `fact` was renamed and rewritten, its real title was
|
||||
replaced by the old filename stem, tags were erased, and the former YAML became body text. The
|
||||
degraded metadata also replaced the good index row. Clearing `superseded_by` this way can make an
|
||||
obsolete fact searchable again. The history snapshot made during repair contains the already
|
||||
broken text and is not a dependable last-good recovery point.
|
||||
|
||||
Required regression coverage: remove only the closing delimiter, then exercise both full scan and
|
||||
watcher sync. The file, path, identity, and last-good index row must remain unchanged while a clear
|
||||
error is reported.
|
||||
|
||||
## High-severity findings
|
||||
|
||||
### H-01 — Delayed embeddings can be attached to newer text
|
||||
|
||||
Affected code: `src/mdmem/index.py:24-29,61-127,360-367`,
|
||||
`src/mdmem/service.py:227-242,714-719`
|
||||
Plan: §6 single-writer consistency, §7 recovery, §14.7
|
||||
|
||||
`PendingEmbed` records `key_hash`, but `_embed_items` discards it and `store_vectors` stores solely
|
||||
by numeric chunk ID. Embedding happens after `_write_lock` is released. Reindexing deletes and
|
||||
reinserts chunks, and SQLite can immediately reuse the old integer ID.
|
||||
|
||||
A delayed response for hash A can therefore overwrite the vector for current hash B and mark B as
|
||||
complete. This affects ordinary writes, background retries, and `reembed_all`. If the old ID is not
|
||||
reused, the code may instead attempt to create an orphan vec row.
|
||||
|
||||
Required regression: hold an old embed request open, complete an edit/reindex, then release it.
|
||||
Store a vector only when both chunk ID and `embed_text_hash` still match in the same transaction.
|
||||
|
||||
### H-02 — Filename/frontmatter disagreement creates identity and lifecycle split-brain
|
||||
|
||||
Affected code: `src/mdmem/vaultio.py:55-69`, `src/mdmem/index.py:210-305`,
|
||||
`src/mdmem/db.py:25-32`
|
||||
Plan: §4 identity and folder lifecycle source of truth
|
||||
|
||||
`read_and_fix` trusts any syntactically valid frontmatter ULID without checking the ULID encoded in
|
||||
the filename. Confirmed failure modes:
|
||||
|
||||
- Editing a canonical file's frontmatter ID to another ULID makes indexing fail on the unique path
|
||||
constraint while preserving a stale row under the filename ID. The file's new ID cannot be found
|
||||
by `Vault.find`.
|
||||
- An active original plus an archive copy carrying the original frontmatter ID is accepted with no
|
||||
conflict; last-file-wins indexing can hide the active original as archived while `Vault.find`
|
||||
still reads the active file.
|
||||
- A file such as `active/manual.md` with a valid frontmatter ID is indexed, but `Vault.find(id)` and
|
||||
`memory_get` cannot locate it because lookup uses the filename ID.
|
||||
|
||||
Required regression: reject mismatched IDs and duplicate lifecycle copies without changing the
|
||||
last-good row. Also test valid-frontmatter-ID files with noncanonical names.
|
||||
|
||||
### H-03 — Normal vault deletion leaves inbound links invisibly dangling
|
||||
|
||||
Affected code: `src/mdmem/index.py:153-158`, `src/mdmem/service.py:180-190`
|
||||
Plan: §5 unresolved links, §6 vanished-file handling, §10 health report
|
||||
|
||||
The UI hard-delete path now reindexes inbound sources, but `Indexer.remove`—used by ordinary
|
||||
filesystem deletion through scan/watcher—deletes only outgoing links. Inbound rows retain the
|
||||
deleted ULID rather than reverting to `unresolved:<target>`, so the unresolved health count remains
|
||||
zero and the broken reference is invisible until its source is reindexed for another reason.
|
||||
|
||||
Required regression: delete a target through both scan and watcher paths; surviving source links
|
||||
must become unresolved and flags for the vanished note must be handled explicitly.
|
||||
|
||||
### H-04 — Service updates can write NFC to disk while indexing NFD
|
||||
|
||||
Affected code: `src/mdmem/notes.py:98-130`, `src/mdmem/vaultio.py:88-119`,
|
||||
`src/mdmem/service.py:531-545,655-684,714-719`
|
||||
Plan: §4 and §14.8 NFC normalization
|
||||
|
||||
`dumps` normalizes a deep copy, not the live `Note`. `_persist` writes the normalized serialization
|
||||
and then indexes the original object. An inbox update with NFD text produced NFC on disk but NFD in
|
||||
`chunks.body` and in the embedding input. Because the indexed file hash is the canonical disk hash,
|
||||
the watcher considers the service event already handled and does not repair the mismatch.
|
||||
|
||||
Required regression: update and UI-edit with NFD in body, frontmatter, relations, and link targets;
|
||||
assert the live note, disk, chunks, FTS/trigram text, links, and embedding input all use NFC.
|
||||
|
||||
### H-05 — Scan-to-watcher startup gap can permanently miss changes
|
||||
|
||||
Affected code: `src/mdmem/service.py:117-157`
|
||||
Plan: §6 startup scan/watcher, P1 live-edit latency
|
||||
|
||||
The watcher starts only after scan, secret scan, self-test, and pending embeddings. A file changed
|
||||
after its scan pass but before watcher registration emits no event. A controlled reproduction wrote
|
||||
a second valid note while `embed_pending` was blocked; after startup finished, the file existed but
|
||||
the note was not indexed.
|
||||
|
||||
Required regression: mutate a file after initial scan while startup embedding is paused. Start
|
||||
watching before reconciliation or run a final scan after watcher registration.
|
||||
|
||||
### H-06 — A Bifrost outage can put Kubernetes into a cold-start restart loop
|
||||
|
||||
Affected code: `src/mdmem/server.py:67-72`, `src/mdmem/service.py:117-157`,
|
||||
`src/mdmem/embed.py:160-199`, `deploy/kubernetes/apps/apps/mdmem/mdmem/app/deployment.yaml:58-69`
|
||||
Plan: §7, §13a.3, §14.17 lexical-only degradation
|
||||
|
||||
The process awaits `svc.start()` before either listener binds. Startup synchronously awaits an
|
||||
embedding self-test and pending embeddings. Each can spend three 60-second attempts. The deployment
|
||||
has liveness starting after 30 seconds, no `startupProbe`, and the default failure threshold.
|
||||
Kubernetes can kill the pod before `/healthz` ever binds, repeatedly preventing the promised
|
||||
lexical-only mode.
|
||||
|
||||
Required regression: start the real server with a blocking/blackholed embedder and prove liveness,
|
||||
readiness, and lexical search bind before the probe deadline. Deployment tests must cover the
|
||||
startup probe budget.
|
||||
|
||||
### H-07 — Common explicit credential forms evade the secret scanner
|
||||
|
||||
Affected code: `src/mdmem/secrets.py:11-29`
|
||||
Plan: §9 and §14.12
|
||||
|
||||
The generic assignment regex requires a word boundary before names such as `token` and `secret`.
|
||||
Because underscore is a word character, common environment variable names evade it. Provider
|
||||
patterns are also incomplete. Confirmed misses include `MCP_TOKEN=...`,
|
||||
`AWS_SECRET_ACCESS_KEY=...`, `OPENAI_API_KEY=...`, `GITHUB_TOKEN=...`, `github_pat_...`, `glpat-...`,
|
||||
`npm_...`, and `hf_...`.
|
||||
|
||||
Required regression: current provider token formats and standard environment assignments,
|
||||
including `export`, quoting, mixed case where applicable, and every serialized note field.
|
||||
|
||||
### H-08 — Secondary persistent channels bypass secret rejection and can leak into audit
|
||||
|
||||
Affected code: `src/mdmem/service.py:556-569`, `src/mdmem/mcp_server.py:90-128,350-354`
|
||||
Plan: §4 untrusted `source`, §9 audit, §14.12
|
||||
|
||||
`flag.reason` is agent-controlled and persisted in SQLite without a scan. `request_id` is stored and
|
||||
logged raw without a size or secret check. Audit `source` is copied from self-reported note metadata
|
||||
before note validation. A write rejected for `source.harness=AKIAIOSFODNN7EXAMPLE` still copied that
|
||||
full key into `audit.jsonl`; the same value in a flag reason was stored and returned by `memory_get`.
|
||||
|
||||
This is also provenance confusion: the audit UI presents an untrusted, caller-asserted source as
|
||||
attribution.
|
||||
|
||||
Required regression: inspect the audit file after rejected source/request ID values, and test flag
|
||||
reason storage. Store bounded/redacted identifiers, not unchecked caller strings.
|
||||
|
||||
### H-09 — Idempotency is race-prone and has inconsistent key semantics
|
||||
|
||||
Affected code: `src/mdmem/mcp_server.py:83-129`, `src/mdmem/db.py:58-59`,
|
||||
`src/mdmem/service.py:777-813`
|
||||
Plan: §9 replay guarantee
|
||||
|
||||
The guard performs `get -> side effect -> put` without synchronization. Two concurrent identical
|
||||
writes with one request ID both missed and created different notes. Concurrent different arguments
|
||||
also both succeeded instead of returning `IDEMPOTENCY_CONFLICT`.
|
||||
|
||||
The table primary key is `request_id`, while lookup treats `(request_id, tool)` as the key.
|
||||
`INSERT OR REPLACE` for a second tool deletes the first tool's replay record. Replaying the original
|
||||
operation can then repeat its side effect.
|
||||
|
||||
Required regression: concurrent same/different arguments, same ID across tools, in-progress
|
||||
requests, and failure after mutation but before idempotency persistence. Claim and commit an
|
||||
idempotency key atomically before performing a mutating operation.
|
||||
|
||||
### H-10 — Revision approval has no target-version or target-lifecycle check
|
||||
|
||||
Affected code: `src/mdmem/service.py:494-528,572-605`
|
||||
Plan: §2.4 and §9 revision review
|
||||
|
||||
A proposal records only `revises`, not the version it was based on. `_merge_revision` blindly loads
|
||||
the current target and overwrites it. Confirmed outcomes:
|
||||
|
||||
- active v1 -> agent proposal v2 -> owner edit v3 -> approve proposal results in v2, silently losing
|
||||
the newer owner edit;
|
||||
- active target -> proposal -> reject/archive target -> approve proposal rewrites the archived
|
||||
target while retaining rejection state, so a later restore promotes the agent text.
|
||||
|
||||
Required regression: target edited, archived, rejected, deleted, or superseded after proposal
|
||||
creation. Approval should refuse stale proposals or require an explicit owner conflict resolution.
|
||||
|
||||
### H-11 — Supersede proposals bypass lifecycle and exclusivity invariants
|
||||
|
||||
Affected code: `src/mdmem/service.py:548-554,572-593`
|
||||
Plan: §2.4, §2.6, §9
|
||||
|
||||
`supersede` accepts archived, rejected, inbox, and already-superseded targets. A rejected item can be
|
||||
re-proposed through this path despite the explicit archived-note guard on update. Two successor
|
||||
proposals can also both be approved: both remain active and point back to the old fact, while the
|
||||
old note's singular `superseded_by` points only to the second. Search can expose contradictory
|
||||
“current” facts.
|
||||
|
||||
Required regression: archived/rejected/inbox/already-superseded/nonexistent targets, concurrent and
|
||||
sequential competing successors, and bidirectional chain consistency.
|
||||
|
||||
### H-12 — Zero and non-finite embeddings are accepted as healthy
|
||||
|
||||
Affected code: `src/mdmem/embed.py:39-42,178-188`, `src/mdmem/search.py:72-74`
|
||||
Plan: §2.1 known all-zero failure, §7 startup self-test, §14.1
|
||||
|
||||
Normalization leaves zero vectors at zero and propagates NaN/Infinity. Shape checks accept them,
|
||||
including during startup self-test. Two zero vectors have distance zero, which search converts to
|
||||
cosine `1.0`, creating a falsely perfect semantic match.
|
||||
|
||||
Required regression: zero, NaN, and infinite vectors for self-test, documents, and queries. Invalid
|
||||
vectors must not be cached or stored, and search must degrade to lexical.
|
||||
|
||||
## Medium-severity findings
|
||||
|
||||
### M-01 — Flags violate the Markdown source-of-truth and recovery contract
|
||||
|
||||
Affected code: `src/mdmem/db.py:53-56`, `src/mdmem/service.py:556-569`
|
||||
Plan: §0 Markdown canonical; SQLite rebuildable
|
||||
|
||||
Flags exist only in SQLite. Deleting/rebuilding `index.db` from an unchanged vault changes
|
||||
`flags_for(id)` from populated to empty. Either make flags canonical in Markdown or explicitly
|
||||
change the backup/recovery contract and back up this mutable database state.
|
||||
|
||||
Missing test: flags survive full index recreation.
|
||||
|
||||
### M-02 — Temporal values are unvalidated strings compared lexicographically
|
||||
|
||||
Affected code: `src/mdmem/service.py:417-450`, `src/mdmem/search.py:194-199,327-332`
|
||||
Plan: §2.6, §4 UTC timestamps, §8 filters
|
||||
|
||||
Malformed values, offsets, and `valid_from >= valid_until` are accepted. Lexical comparison is not
|
||||
instant comparison: for example, equivalent offset and `Z` timestamps can filter differently.
|
||||
|
||||
Missing tests: RFC3339 validity, UTC normalization, ordering, equivalent instants, and boundary
|
||||
semantics for stored values and search arguments.
|
||||
|
||||
### M-03 — Unicode and trigram lexical hits are not merged by better rank
|
||||
|
||||
Affected code: `src/mdmem/search.py:120-150`
|
||||
Plan: §8.1
|
||||
|
||||
The implementation mixes raw BM25 magnitudes from different FTS tables and ignores trigram whenever
|
||||
the same chunk has a unicode61 hit. This does not implement “keep the better rank per chunk.”
|
||||
|
||||
Missing test: controlled rank lists where a chunk is rank 50 in unicode61 and rank 1 in trigram.
|
||||
|
||||
### M-04 — Late link resolution misses equivalent slug spellings
|
||||
|
||||
Affected code: `src/mdmem/index.py:182-207`
|
||||
Plan: §5 exact-title then slug resolution
|
||||
|
||||
Initial resolution slugifies a target, but late resolution updates only two literal unresolved
|
||||
keys. `[[Later Note]]` remains unresolved when a later note titled `Later Note` arrives, although
|
||||
both slugify identically.
|
||||
|
||||
Missing tests: whitespace, punctuation, capitalization, and umlaut variants on forward links.
|
||||
|
||||
### M-05 — Search accepts invalid bounds and can exceed `max_chars`
|
||||
|
||||
Affected code: `src/mdmem/search.py:303-318`
|
||||
Plan: §8.7, §14.11
|
||||
|
||||
The first result is always emitted even when it exceeds the requested budget; zero or negative
|
||||
budgets still return one result. A negative internal `k` uses Python slicing and can return nearly
|
||||
all candidates. The current test explicitly permits an 800-character overrun, contradicting the
|
||||
plan.
|
||||
|
||||
Missing tests: `k <= 0`, `max_chars <= 0`, oversized first title/snippet, exact-budget boundary, and
|
||||
structured MCP validation.
|
||||
|
||||
### M-06 — One long token bypasses the chunk target
|
||||
|
||||
Affected code: `src/mdmem/chunker.py:50-89`
|
||||
Plan: §6.4
|
||||
|
||||
`_split_words` does not split a single token longer than the limit. Agent content can reach 8,000
|
||||
characters and hand-authored notes are unbounded, producing oversized chunks.
|
||||
|
||||
Missing test: a no-whitespace token larger than the chunk limit, with bounded lossless output.
|
||||
|
||||
### M-07 — `is_ulid` accepts overflow encodings rejected by the ULID library
|
||||
|
||||
Affected code: `src/mdmem/ids.py:13,38-39`
|
||||
Plan: §4/§6 valid identity
|
||||
|
||||
Validation checks only length and alphabet. Values such as
|
||||
`ZZZZZZZZZZZZZZZZZZZZZZZZZZ` pass but `ULID.from_str` rejects them as overflow.
|
||||
|
||||
Missing property: `is_ulid(value)` implies successful parsing by the canonical ULID implementation.
|
||||
|
||||
### M-08 — OIDC claim validation accepts non-finite expiry, missing subject, and bad `azp`
|
||||
|
||||
Affected code: `src/mdmem/ui/auth.py:210-245`
|
||||
Plan: §10 fail closed
|
||||
|
||||
Signed tokens with `exp: NaN` and `exp: Infinity` pass because any int/float is accepted and
|
||||
non-finite comparisons do not expire them. `sub` is not required, allowing an empty-subject session.
|
||||
When `aud` contains multiple values, `azp` is not required or checked; a token with mdmem among its
|
||||
audiences but `azp` naming another client was accepted.
|
||||
|
||||
Missing tests: finite expiry, required nonempty subject, and multi-audience authorized-party rules.
|
||||
|
||||
### M-09 — Scalar group claim is accepted contrary to the fail-closed plan
|
||||
|
||||
Affected code: `src/mdmem/ui/auth.py:60-90`, `tests/test_ui_auth.py:59-60`
|
||||
Plan: §10, P4, §14.21
|
||||
|
||||
The code converts a scalar string group to a list, and a test deliberately blesses it. The plan
|
||||
requires a present-but-wrong-type group claim to return 403.
|
||||
|
||||
Required change to coverage: only `list[str]` should pass; scalar, null, mapping, and mixed lists
|
||||
must fail.
|
||||
|
||||
### M-10 — UI POST protection still does not validate Host
|
||||
|
||||
Affected code: `src/mdmem/ui/app.py:85-105,464-480`
|
||||
Plan: §10 Origin/Host allowlist
|
||||
|
||||
The recent Origin/Referer fix correctly rejects headerless writes, but when configured origins are
|
||||
present, Host is ignored. A request with allowed `Origin` and `Host: evil.example` was accepted.
|
||||
|
||||
Missing tests: invalid Host with valid Origin, proxy-forwarded host behavior, and the exact deployed
|
||||
host allowlist.
|
||||
|
||||
### M-11 — JWKS is cached forever, breaking key rotation until restart
|
||||
|
||||
Affected code: `src/mdmem/ui/auth.py:149-165`
|
||||
Plan: §10 operationally reliable OIDC
|
||||
|
||||
An unknown `kid` does not trigger one forced JWKS refresh. New logins fail after PocketID signing-key
|
||||
rotation until the pod restarts.
|
||||
|
||||
Missing test: cache key A, publish key B, present a B-signed token, and verify one refresh succeeds.
|
||||
|
||||
### M-12 — Audit hashes omit actual arguments and unexpected errors vanish
|
||||
|
||||
Affected code: `src/mdmem/mcp_server.py:83-129,167,335-339`
|
||||
Plan: §9 every tool call logged with argument hash, source, latency, IDs
|
||||
|
||||
Search hashes only query and `k`, omitting filters. Flag hashes only ID, omitting reason. Different
|
||||
calls therefore receive identical audit fingerprints. `_guard` catches only `MdmemError`; an
|
||||
injected database/programming exception produced no audit entry at all.
|
||||
|
||||
Missing tests: vary every argument and require a hash change; inject unexpected exceptions and
|
||||
require `ok:false`, latency, request ID, and a bounded error class.
|
||||
|
||||
### M-13 — `memory_flag` is non-idempotent
|
||||
|
||||
Affected code: `src/mdmem/mcp_server.py:328-339`
|
||||
Plan: §9 idempotency
|
||||
|
||||
The mutating flag tool has no `request_id`. A transport retry creates duplicate review items. Search
|
||||
and get also omit the plan's broadly stated optional request ID, though they have no side effect.
|
||||
|
||||
Missing test: replaying one flag request creates exactly one flag.
|
||||
|
||||
### M-14 — Evaluation can recommend a variant with a catastrophic category regression
|
||||
|
||||
Affected code: `src/mdmem/evalharness.py:340-372`
|
||||
Plan: §11 “without a regression in any category beyond noise”
|
||||
|
||||
`pick_default` checks only overall nDCG and confidence intervals. It never examines `per_category`.
|
||||
|
||||
Missing test: a candidate with a clear overall win but severe temporal or update regression must
|
||||
remain unselected.
|
||||
|
||||
### M-15 — Eval output is not standards-compliant JSON
|
||||
|
||||
Affected code: `src/mdmem/evalharness.py:90-117,160-174,395-425`,
|
||||
`eval/results/2026-09-12-synthetic-smoke.json:538-541`
|
||||
Plan: §11 committed JSON result artifact
|
||||
|
||||
Unavailable metrics remain `float("nan")`; `json.dumps` emits raw `NaN`, which RFC 8259 does not
|
||||
permit. The test named as a CLI writer test manually writes a different object and never calls
|
||||
`run_eval_cli`, leaving serialization false-green.
|
||||
|
||||
Missing test: invoke the actual CLI and parse with a strict JSON parser; unavailable values should
|
||||
serialize as `null`.
|
||||
|
||||
### M-16 — Eval latency is order-biased by a shared embedding cache
|
||||
|
||||
Affected code: `src/mdmem/evalharness.py:293-312`, `src/mdmem/service.py:99-115`
|
||||
Plan: §11 fair ablation comparison
|
||||
|
||||
Each variant gets an isolated index but retains the configured `cache_db`. The first real-model
|
||||
variant pays query embedding latency while later variants reuse cached query embeddings, making p95
|
||||
latency incomparable.
|
||||
|
||||
Missing test: instrument backend query calls and prove equivalent cache state or perform an explicit
|
||||
warm-up excluded from every variant's timing.
|
||||
|
||||
### M-17 — Required Prometheus signals are absent
|
||||
|
||||
Affected code: `src/mdmem/ui/app.py:75-145`
|
||||
Plan: §13a.7
|
||||
|
||||
Embedding error count, search-latency histogram, and MCP calls by tool/outcome are not exported.
|
||||
Search/write totals are Gauges rather than Counters. Existing tests only check that generic metrics
|
||||
exist and omit note content.
|
||||
|
||||
Missing test: scrape and assert all required metric families and bounded label cardinality.
|
||||
|
||||
### M-18 — UI state filtering and state transitions do not enforce the displayed workflow
|
||||
|
||||
Affected code: `src/mdmem/ui/app.py:261-300`, `src/mdmem/service.py:638-653`
|
||||
Plan: §2.4, §10 Search filters and review workflow
|
||||
|
||||
Search `state=inbox` includes both active and inbox notes, while `state=archive` includes active,
|
||||
inbox, and archive. In addition, the generic state endpoint can promote an inbox revision or
|
||||
supersede proposal directly to active, bypassing `approve` merge/chain behavior if called manually.
|
||||
|
||||
Missing tests: exact results for every state filter and forbidden lifecycle transitions that must
|
||||
go through approval.
|
||||
|
||||
## Lower-severity plan drift and test-suite gaps
|
||||
|
||||
- Health has no “run eval” action required by §10 (`src/mdmem/ui/templates/health.html`,
|
||||
`src/mdmem/ui/app.py:443-460`).
|
||||
- History compares one snapshot only to current, not any two revisions as required by §10
|
||||
(`src/mdmem/ui/app.py:313-324`).
|
||||
- Browse does not offer sorting by tags as required by §10.
|
||||
- Health audit rows do not display the recorded `source` field.
|
||||
- “Edit & approve” converts an intentionally empty body to `None`, approving the old body instead
|
||||
(`src/mdmem/ui/app.py:235-239`). Empty must be saved or explicitly rejected, not ignored.
|
||||
- `AuditLog.recent` reads persisted history only while its in-memory list is completely empty. After
|
||||
restart and one new call, Health shows only that call rather than the last 100 across restart
|
||||
(`src/mdmem/logging.py:77-92`).
|
||||
- `memory_supersede` omits relations, validity bounds, and source metadata available to
|
||||
`memory_write`; either complete the §9 schema or narrow the contract.
|
||||
- Bearer configuration accepts whitespace/non-ASCII values that clients cannot reliably present;
|
||||
startup should validate a trimmed ASCII token with an explicit minimum length.
|
||||
- `validate_id` is dead code. ID arguments have no schema pattern or maximum length and are echoed
|
||||
in `NOT_FOUND` responses.
|
||||
- The session test named “round trip and expiry” never advances time or tests expiration.
|
||||
- No route-level OIDC test covers discovery, state, PKCE, nonce carriage, callback errors, cookie
|
||||
flags, expiry, logout, or signing-key rotation.
|
||||
- The OIDC HTTP client created in `build_ui_app` has no application shutdown hook calling
|
||||
`OidcClient.aclose`.
|
||||
- CLI tests cover help/version but not serve, doctor, reindex, eval, contradictory mode flags,
|
||||
startup failures, or graceful shutdown.
|
||||
- The note round-trip property test generates a body but does not assert its round trip and does not
|
||||
property-test arbitrary unknown-key order/comments.
|
||||
- The clean manifest template is not directly buildable: `app/kustomization.yaml` references the
|
||||
intentionally absent encrypted `secret.yaml`; image digest and PocketID client ID remain release
|
||||
placeholders. These are documented setup blockers, but there is no CI smoke test for the rendered
|
||||
deployment overlay.
|
||||
- The only committed evaluation is a synthetic fake-embedding smoke run. It proves harness wiring,
|
||||
not the P5 acceptance criterion of 50–100 real labelled queries and a production-model tuning
|
||||
report.
|
||||
|
||||
## Test integrity assessment
|
||||
|
||||
The suite is useful for sequential happy paths but systematically underrepresents interleavings,
|
||||
recovery from canonical Markdown, deployment startup, and strict external-format validation.
|
||||
|
||||
Highest-value missing test groups, in order:
|
||||
|
||||
1. deterministic service concurrency tests around the whole read/check/write operation;
|
||||
2. chunk-hash-guarded embedding races, including background retry and re-embed;
|
||||
3. malformed frontmatter delimiter and filename/frontmatter identity conflict cases;
|
||||
4. startup with a blocked embedding backend plus a mutation during watcher registration;
|
||||
5. concurrent idempotency and crash-between-side-effect-and-record cases;
|
||||
6. secret scanning and audit redaction across every persistent channel;
|
||||
7. full OIDC route tests with strict claims and signing-key rotation;
|
||||
8. actual CLI eval serialization through a strict JSON parser;
|
||||
9. disaster rebuild proving all canonical state survives; and
|
||||
10. deployment render/probe smoke tests.
|
||||
|
||||
Some current tests actively encode plan drift rather than merely omitting it:
|
||||
|
||||
- `tests/test_ui_auth.py` accepts a scalar group claim despite §14.21;
|
||||
- `tests/test_search.py` permits the first result to exceed `max_chars` by up to one snippet;
|
||||
- `tests/test_evalharness.py::test_cli_reports_and_writes_a_json_file` does not invoke the CLI it
|
||||
names.
|
||||
|
||||
## Recommended remediation order
|
||||
|
||||
1. Serialize lifecycle/version operations end-to-end and add deterministic race tests.
|
||||
2. Reject unterminated/mismatched frontmatter without touching the file or last-good index.
|
||||
3. Guard vector writes by chunk ID plus embedding-text hash.
|
||||
4. Make startup bind in lexical mode independently of Bifrost and close the scan/watch gap.
|
||||
5. Make idempotency atomic and reconcile its database key.
|
||||
6. Close all secret-bearing secondary channels and redact audit fields.
|
||||
7. Enforce revision/supersede target version, lifecycle, and exclusivity.
|
||||
8. Validate embeddings, timestamps, ULIDs, OIDC claims, Host, and response bounds.
|
||||
9. Restore the Markdown-only recovery guarantee for flags or explicitly redesign it.
|
||||
10. Repair eval selection/serialization/fairness, observability, and the lower-priority UI drift.
|
||||
|
||||
## Baseline notes
|
||||
|
||||
The following earlier findings are fixed in the reviewed revision and are intentionally not listed
|
||||
as open defects above: full-field note secret scanning, mandatory `expected_version`, archived-note
|
||||
update rejection in the sequential case, approval validation, audit-file wiring, ordinary
|
||||
hard-delete link repair, background retry of pending vectors, safe re-embed on a simple outage,
|
||||
trigram backfill after toggling it off/on, missing-Origin CSRF rejection, nonce/`exp`/`iat` presence
|
||||
checks, eval index isolation, reciprocal synthetic supersede data, connection/cache shutdown, and
|
||||
the direct fusion formula tests.
|
||||
@@ -1,19 +0,0 @@
|
||||
{
|
||||
"_howto": "Add this object to mcp.client_configs in kubernetes/apps/apps/bifrost/bifrost/app/config.json. That file is a configMapGenerator input, so editing it is the whole change — the generated ConfigMap's hash suffix moves and Flux rolls the Deployment. Commit it only AFTER mdmem itself is healthy, so a failure is attributable to one change.",
|
||||
"name": "mdmem",
|
||||
"connection_type": "http",
|
||||
"connection_string": "http://mdmem.apps.svc.cluster.local:8080/mcp",
|
||||
"auth_type": "headers",
|
||||
"headers": { "Authorization": "env.MDMEM_AUTH_HEADER" },
|
||||
"is_ping_available": false,
|
||||
"is_code_mode_client": false,
|
||||
"_comment": "Long-term agent memory. Code Mode deliberately off: memory bodies are untrusted text, and code mode hands tool output to a model that then writes executable code. Revisit only after the prompt-injection behaviour is tested with a local model. tools_to_execute is an explicit allow-list, not '*', so a future mdmem release cannot add a destructive tool that silently becomes reachable.",
|
||||
"tools_to_execute": [
|
||||
"memory_search",
|
||||
"memory_get",
|
||||
"memory_write",
|
||||
"memory_update",
|
||||
"memory_supersede",
|
||||
"memory_flag"
|
||||
]
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- ./mdmem/ks.yaml
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: mdmem
|
||||
labels:
|
||||
app: mdmem
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # SQLite + watchfiles need a single writer
|
||||
selector:
|
||||
matchLabels:
|
||||
app: mdmem
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: mdmem
|
||||
spec:
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
fsGroup: 1000
|
||||
fsGroupChangePolicy: OnRootMismatch # Longhorn: else /data comes up root-owned
|
||||
containers:
|
||||
- name: mdmem
|
||||
# Pin tag AND digest. Renovate does not track library/*, so the digest
|
||||
# is bumped by hand in the same commit that pushes a build (§13a.2).
|
||||
image: harbor.cloud.traunseenet.com/library/mdmem:0.1.0@sha256:4b0c21bec82f12ae5f6e3b5398fc6af31fecf3bfb5c0d96b44acd85bf638d1f4
|
||||
imagePullPolicy: IfNotPresent
|
||||
args: ["serve"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
env:
|
||||
- name: HOME
|
||||
value: /tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /tmp
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: mdmem-env
|
||||
- secretRef:
|
||||
name: mdmem # from Vault via the ExternalSecret
|
||||
ports:
|
||||
- name: mcp
|
||||
containerPort: 8080
|
||||
- name: ui
|
||||
containerPort: 8081
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: ui
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: ui
|
||||
periodSeconds: 5
|
||||
failureThreshold: 36
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /readyz # never checks Bifrost (§14.17)
|
||||
port: ui
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: mdmem-data
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: mdmem
|
||||
spec:
|
||||
refreshInterval: 5m
|
||||
secretStoreRef:
|
||||
name: vault-backend
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: mdmem
|
||||
dataFrom:
|
||||
# KV-v2 mount is `secret`, so this reads secret/apps/mdmem:
|
||||
# MCP_TOKEN, BIFROST_VK, OIDC_CLIENT_SECRET, SESSION_SECRET
|
||||
- extract:
|
||||
key: apps/mdmem
|
||||
@@ -1,25 +0,0 @@
|
||||
# Only the UI port is routed. :8080 (MCP) gets no HTTPRoute — Bifrost reaches it
|
||||
# at mdmem.apps.svc.cluster.local:8080, same namespace (§13a.5).
|
||||
#
|
||||
# Do not apply this before the OIDC group check is live (§14.19): *.cloud is
|
||||
# split-horizon, not private.
|
||||
---
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: mdmem
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: traefik
|
||||
namespace: networking
|
||||
sectionName: websecure-cloud
|
||||
hostnames:
|
||||
- memory.cloud.traunseenet.com
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
type: PathPrefix
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: mdmem
|
||||
port: 8081
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: apps
|
||||
resources:
|
||||
- ./externalsecret.yaml
|
||||
- ./secret.yaml # SOPS: restic-mdmem only (§12.1)
|
||||
- ./pvc.yaml
|
||||
- ./deployment.yaml
|
||||
- ./service.yaml
|
||||
- ./httproute.yaml
|
||||
- ./networkpolicy.yaml
|
||||
- ./servicemonitor.yaml
|
||||
configMapGenerator:
|
||||
- name: mdmem-env
|
||||
literals:
|
||||
# --- storage ---------------------------------------------------------
|
||||
- VAULT_DIR=/data/vault
|
||||
- INDEX_DB=/data/index.db
|
||||
- CACHE_DB=/data/emb_cache.db
|
||||
- HISTORY_DIR=/data/.history
|
||||
- AUDIT_LOG=/data/audit.jsonl
|
||||
# --- embeddings ------------------------------------------------------
|
||||
- BIFROST_URL=http://bifrost.apps.svc.cluster.local:8080
|
||||
# provider-prefixed; a bare alias 404s and looks like an mdmem bug (§7, §14.14)
|
||||
- EMBED_MODEL=llamaswap/qwen3-embedding-4b
|
||||
- EMBED_DIM=2560
|
||||
# --- retrieval -------------------------------------------------------
|
||||
- FUSION=rrf
|
||||
- CC_ALPHA=0.5
|
||||
- FTS_TRIGRAM=1
|
||||
- INBOX_WEIGHT=0.8
|
||||
- BM25_WEIGHTS=3,2,1.5,1
|
||||
- INCLUDE_INBOX_DEFAULT=true
|
||||
- MIN_SCORE=
|
||||
- MAX_CHARS_DEFAULT=6000
|
||||
- DUP_THRESHOLD=0.90
|
||||
- MAX_NOTE_CHARS=8000
|
||||
# --- listeners -------------------------------------------------------
|
||||
- MCP_ADDR=0.0.0.0:8080
|
||||
- UI_ADDR=0.0.0.0:8081
|
||||
# the MCP SDK checks the Host header; Bifrost dials the service name (§14.24)
|
||||
- MCP_ALLOWED_HOSTS=mdmem.apps.svc.cluster.local:8080,mdmem:8080,localhost:8080
|
||||
# --- ui / auth -------------------------------------------------------
|
||||
- UI_BASE_URL=https://memory.cloud.traunseenet.com
|
||||
- UI_ALLOWED_ORIGINS=https://memory.cloud.traunseenet.com
|
||||
- OIDC_ISSUER=https://pocketid.cloud.traunseenet.com
|
||||
- OIDC_SCOPE=openid email profile groups
|
||||
- OIDC_CLIENT_ID=dc2aa40e-4386-46cc-bae6-0259a2e7490e
|
||||
- OIDC_GROUPS_CLAIM=groups
|
||||
- OIDC_REQUIRED_GROUP=mdmem_admins
|
||||
- SESSION_TTL_HOURS=12
|
||||
# --- misc ------------------------------------------------------------
|
||||
- LOG_LEVEL=info
|
||||
- LOG_STYLE=json
|
||||
- TZ=Europe/Vienna
|
||||
@@ -1,49 +0,0 @@
|
||||
# Required, not optional (§13a.5a). newt's egress policy allows the entire apps
|
||||
# namespace on every port and reaches pods directly rather than through Traefik,
|
||||
# so without this policy "not published through Pangolin" is a promise that
|
||||
# lives in a remote dashboard instead of in git — and :8080, whose only control
|
||||
# is a bearer token, would be one click from the internet.
|
||||
#
|
||||
# After applying, verify the two things it can silently break: the pod stays
|
||||
# Ready (kubelet probes) and the Prometheus target stays UP (§14.23).
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: mdmem-ingress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: mdmem
|
||||
policyTypes: ["Ingress"]
|
||||
ingress:
|
||||
# MCP: Bifrost only, same namespace
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: apps
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: bifrost
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
# UI: Traefik only — deliberately NOT newt, so Pangolin cannot bypass the gateway
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: networking
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
# /metrics scrape
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
@@ -1,38 +0,0 @@
|
||||
---
|
||||
apiVersion: volsync.backube/v1alpha1
|
||||
kind: ReplicationDestination
|
||||
metadata:
|
||||
name: mdmem-restore
|
||||
spec:
|
||||
trigger:
|
||||
manual: restore-once
|
||||
restic:
|
||||
repository: restic-mdmem
|
||||
copyMethod: Snapshot
|
||||
storageClassName: longhorn-storage
|
||||
volumeSnapshotClassName: longhorn-snapshot
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
capacity: 5Gi
|
||||
cacheCapacity: 1Gi
|
||||
enableFileDeletion: false
|
||||
moverSecurityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: mdmem-data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: longhorn-storage
|
||||
# Do NOT set numberOfReplicas: 1 here. The vault is the primary artefact of
|
||||
# this service; an hourly restic snapshot is not block-level redundancy.
|
||||
dataSourceRef:
|
||||
kind: ReplicationDestination
|
||||
apiGroup: volsync.backube
|
||||
name: mdmem-restore
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
@@ -1,17 +0,0 @@
|
||||
# Rename to secret.yaml and SOPS-encrypt before committing. This is the ONE
|
||||
# secret that stays out of Vault (§12.1): the restore path must work when Vault
|
||||
# itself is what is being restored.
|
||||
#
|
||||
# sops --encrypt --age age1q3mu6g0l3kr7spszks69yqrvlxvlgfrqgw9r00qqt8cj9xpjd9mqdkd7um \
|
||||
# --encrypted-regex '^(data|stringData)$' --in-place secret.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: restic-mdmem
|
||||
type: Opaque
|
||||
stringData:
|
||||
RESTIC_REPOSITORY: s3:https://s3.example/restic-mdmem
|
||||
RESTIC_PASSWORD: REPLACE
|
||||
AWS_ACCESS_KEY_ID: REPLACE
|
||||
AWS_SECRET_ACCESS_KEY: REPLACE
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: mdmem
|
||||
labels:
|
||||
app: mdmem
|
||||
spec:
|
||||
selector:
|
||||
app: mdmem
|
||||
ports:
|
||||
- name: mcp
|
||||
port: 8080
|
||||
targetPort: mcp
|
||||
- name: ui
|
||||
port: 8081
|
||||
targetPort: ui
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: mdmem
|
||||
labels:
|
||||
app: mdmem
|
||||
spec:
|
||||
endpoints:
|
||||
- port: ui
|
||||
path: /metrics
|
||||
interval: 15s
|
||||
honorLabels: true
|
||||
namespaceSelector:
|
||||
matchNames: ["apps"]
|
||||
selector:
|
||||
matchLabels:
|
||||
app: mdmem
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: apps
|
||||
resources:
|
||||
- ./pvc-backup.yaml
|
||||
@@ -1,30 +0,0 @@
|
||||
# Hourly, not six-hourly: the vault accumulates irreplaceable hand-written
|
||||
# content between snapshots, unlike a re-derivable app database (§13a.4).
|
||||
# The whole PVC is backed up — excluding index.db/emb_cache.db would only buy a
|
||||
# full re-embed at restore time.
|
||||
---
|
||||
apiVersion: volsync.backube/v1alpha1
|
||||
kind: ReplicationSource
|
||||
metadata:
|
||||
name: mdmem
|
||||
spec:
|
||||
sourcePVC: mdmem-data
|
||||
trigger:
|
||||
schedule: "0 * * * *"
|
||||
restic:
|
||||
repository: restic-mdmem
|
||||
copyMethod: Snapshot
|
||||
storageClassName: longhorn-storage
|
||||
volumeSnapshotClassName: longhorn-snapshot
|
||||
cacheCapacity: 1Gi
|
||||
pruneIntervalDays: 1
|
||||
retain:
|
||||
hourly: 5
|
||||
daily: 4
|
||||
weekly: 3
|
||||
monthly: 2
|
||||
yearly: 1
|
||||
moverSecurityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
@@ -1,58 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: mdmem-app
|
||||
namespace: flux-system
|
||||
spec:
|
||||
targetNamespace: apps
|
||||
commonMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: mdmem
|
||||
dependsOn:
|
||||
- name: traefik-base
|
||||
- name: longhorn-addons
|
||||
- name: cert-manager-addons
|
||||
- name: volsync
|
||||
- name: harbor-app # the image is pulled from in-cluster Harbor
|
||||
- name: hashicorp-vault-app # the ExternalSecret needs Vault reachable
|
||||
- name: external-secrets-app
|
||||
- name: bifrost-app # embeddings + the MCP front door
|
||||
path: ./kubernetes/apps/apps/mdmem/mdmem/app
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: fluxcd-kubernetes
|
||||
namespace: flux-system
|
||||
wait: true
|
||||
interval: 15m
|
||||
retryInterval: 15m
|
||||
timeout: 15m
|
||||
healthChecks:
|
||||
- apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: mdmem
|
||||
namespace: apps
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: mdmem-backup
|
||||
namespace: flux-system
|
||||
spec:
|
||||
targetNamespace: apps
|
||||
commonMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: mdmem
|
||||
dependsOn:
|
||||
- name: mdmem-app
|
||||
- name: volsync
|
||||
path: ./kubernetes/apps/apps/mdmem/mdmem/backup
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: fluxcd-kubernetes
|
||||
namespace: flux-system
|
||||
interval: 15m
|
||||
retryInterval: 15m
|
||||
timeout: 15m
|
||||
@@ -1,25 +0,0 @@
|
||||
# Add to llama-swap's config on evo-x2 (10.0.0.59:9000) — the box Bifrost's
|
||||
# `llamaswap` provider already points at. Not managed by Flux; llama-swap runs
|
||||
# off-cluster (§7).
|
||||
#
|
||||
# --pooling last is mandatory: Qwen3 embeddings use last-token pooling and the
|
||||
# wrong pooling degrades retrieval silently (§14.1, [R2][R3]).
|
||||
# Never add --reranking to this instance: with both flags on, a reported
|
||||
# llama.cpp bug returns all-zero embeddings (§14.2, [R5]).
|
||||
# Q8_0, not Q4_K: Q4_K measured 0.974 cosine against the reference (§14.3, [R4]).
|
||||
models:
|
||||
qwen3-embedding-4b:
|
||||
cmd: >
|
||||
llama-server --port ${PORT}
|
||||
-m /models/Qwen3-Embedding-4B-Q8_0.gguf
|
||||
--embedding --pooling last
|
||||
-ub 8192 -c 8192 -ngl 99
|
||||
ttl: 0 # keep resident; an embedding call must not evict the chat model
|
||||
|
||||
# Keep it co-resident with the chat model rather than swapping. Check the group
|
||||
# syntax against your llama-swap version:
|
||||
# groups:
|
||||
# resident:
|
||||
# swap: false
|
||||
# exclusive: false
|
||||
# members: ["qwen3-embedding-4b", "<your chat model>"]
|
||||
@@ -1,44 +0,0 @@
|
||||
# Eval harness (§11)
|
||||
|
||||
```bash
|
||||
mdmem eval --queries eval/queries.yaml --config default # one config
|
||||
mdmem eval --queries eval/queries.yaml --ablate # the whole matrix
|
||||
```
|
||||
|
||||
Both write `eval/results/<date>.json` and print a markdown table plus a
|
||||
recommendation. The recommendation only names a new default when its nDCG@10
|
||||
95 % bootstrap CI clears the baseline's — a two-point difference on 50 queries
|
||||
is noise, not a result.
|
||||
|
||||
`--ablate` covers: lex-only, sem-only, RRF, CC with α ∈ {0.3 … 0.9}, trigram
|
||||
off, inbox weight 1.0, flat BM25 weights, and embedding dims 1024 / 512. Every
|
||||
variant runs at the same `MAX_CHARS_DEFAULT` budget, because a retrieval
|
||||
comparison at different context volumes measures the volume [R16]. Dim variants
|
||||
build their own throwaway index and delete it afterwards.
|
||||
|
||||
`calibrate_min_score()` reports the `MIN_SCORE` that maximises abstention F1 on
|
||||
the `category: abstain` queries. It is not applied automatically — set it in the
|
||||
ConfigMap once you believe it.
|
||||
|
||||
## Runs so far
|
||||
|
||||
| file | what it is | conclusion |
|
||||
|---|---|---|
|
||||
| `results/2026-09-13-synthetic-smoke.json` | 18 synthetic queries over the 12 seeded notes in `corpus/`, run with `--fake-embeddings` | **The harness works; nothing about the config was learned.** Vectors are hashed word bags, not Qwen3, and 18 queries cannot separate fusion variants — every CC arm's 95 % CI overlaps the RRF baseline's. Defaults stay as §2.5 specifies: `FUSION=rrf`, `EMBED_DIM=2560`, `MIN_SCORE` unset. |
|
||||
|
||||
`rrf`, `rrf-inbox-1.0` and `rrf-bm25-flat` report the *same* nDCG@10 in that run.
|
||||
That is expected here and is not the shared-state corruption an earlier report had:
|
||||
none of the 12 seeded notes sit in `inbox/`, so the inbox weight has nothing to
|
||||
act on, and the two lexical subchannels are fused by ordinal rank (§8.1), so a
|
||||
BM25 weight change only moves the score if it reorders the hits — on a corpus this
|
||||
small, flat weights do not.
|
||||
|
||||
The abstention F1 in that run is 0 because `MIN_SCORE` is unset by default, so
|
||||
the two `abstain` queries still return their best (wrong) match. The calibration
|
||||
block in the report names the threshold that would maximise F1 — on a real,
|
||||
labelled set it is the number to put in the ConfigMap.
|
||||
|
||||
**Before tuning anything**: label 50–100 real queries in `eval/queries.yaml`,
|
||||
run `mdmem eval --ablate` against the real Bifrost endpoint, and only then move
|
||||
a default. A variant ships only when its nDCG@10 CI clears the baseline's and no
|
||||
category regresses.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000BFR
|
||||
title: Bifrost ist das AI-Gateway
|
||||
type: fact
|
||||
summary: Bifrost bündelt alle LLM-Aufrufe und hält die virtuellen Schlüssel.
|
||||
keywords: [bifrost, gateway, virtual key, api gateway, modellrouter]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: [01M00000000000000000000TEM]
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Bifrost ist der einzige Eingang für LLM-Aufrufe und MCP-Clients. Es steht vor llama-swap.
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000TEM
|
||||
title: Der Modellrouter hieß früher LiteLLM
|
||||
type: fact
|
||||
summary: Vor Bifrost lief LiteLLM als Gateway. Ersetzt.
|
||||
keywords: [litellm, gateway, alt, modellrouter]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: 01M00000000000000000000BFR
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Historisch: LiteLLM war das Gateway bis 2026.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000EVX
|
||||
title: evo-x2
|
||||
type: entity
|
||||
summary: Die GPU-Box im Keller mit 128 GB unified memory.
|
||||
keywords: [evo-x2, gpu, hardware, workstation]
|
||||
tags: [homelab]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
evo-x2 ist die Inferenz-Box unter 10.0.0.59. Beherbergt llama-swap.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000FXC
|
||||
title: FluxCD deployt alles
|
||||
type: fact
|
||||
summary: Jede App kommt über FluxCD-Kustomizations aus dem fluxcd-Repo.
|
||||
keywords: [fluxcd, gitops, kustomization, deployment, ausrollen]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Der Cluster ist RKE2 mit FluxCD. Kein manuelles kubectl apply.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000HRB
|
||||
title: Harbor spiegelt die Images
|
||||
type: fact
|
||||
summary: harbor.cloud.traunseenet.com ist Proxy-Cache und Registry für eigene Images.
|
||||
keywords: [harbor, registry, images, proxy cache, container]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Eigene Images liegen unter library/, gespiegelte unter docker.io/ und ghcr.io/.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000KAF
|
||||
title: Kaffeemaschine entkalken
|
||||
type: procedure
|
||||
summary: Alle drei Monate mit Essigessenz entkalken.
|
||||
keywords: [kaffee, entkalken, wartung, descale, maintenance]
|
||||
tags: [haushalt]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
1. Tank mit 1:4 Essigessenz füllen. 2. Zwei Durchläufe. 3. Zweimal klar spülen.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000SWP
|
||||
title: llama-swap routet lokale Modelle
|
||||
type: fact
|
||||
summary: llama-swap tauscht Modelle auf evo-x2; Bifrost steht davor.
|
||||
keywords: [llama-swap, modellrouter, evo-x2, model router, inference]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
llama-swap läuft auf [[evo-x2]] und tauscht llama-server-Modelle. Bifrost adressiert es als llamaswap/<modell>.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000STG
|
||||
title: Longhorn liefert die Volumes
|
||||
type: fact
|
||||
summary: Longhorn ist die StorageClass im RKE2-Cluster, nicht democratic-csi.
|
||||
keywords: [longhorn, storage, pvc, storageclass, volumes, speicher]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Persistente Volumes kommen von longhorn-storage. RWO, repliziert.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000PKT
|
||||
title: PocketID macht das SSO
|
||||
type: fact
|
||||
summary: Alle Web-UIs authentifizieren sich per OIDC gegen PocketID.
|
||||
keywords: [pocketid, oidc, sso, login, authentifizierung]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Es gibt kein forward-auth im Cluster. Jede App macht OIDC selbst gegen PocketID.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000QWN
|
||||
title: Qwen3-Embedding-4B als Embedder
|
||||
type: decision
|
||||
summary: Q8_0-Quant, last-token pooling, 2560 Dimensionen.
|
||||
keywords: [qwen3, embedding, q8_0, pooling, vektoren, einbettung]
|
||||
tags: [homelab, ai]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Entschieden: 4B statt 8B (kleiner Abstand) und Q8_0 statt Q4_K (Drift).
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000TRN
|
||||
title: Traunsee Wasserstand
|
||||
type: note
|
||||
summary: Pegel Gmunden liegt im Sommer meist um 422 m.
|
||||
keywords: [traunsee, pegel, wasserstand, lake level, gmunden]
|
||||
tags: [region]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Der Pegel Gmunden schwankt zwischen 421,5 m und 423 m.
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
---
|
||||
id: 01M00000000000000000000VSY
|
||||
title: VolSync sichert stündlich nach Restic
|
||||
type: decision
|
||||
summary: Backups laufen stündlich per VolSync-ReplicationSource in ein Restic-Repo.
|
||||
keywords: [volsync, restic, backup, snapshot, sicherung]
|
||||
tags: [homelab, k8s]
|
||||
relations: {}
|
||||
supersedes: []
|
||||
superseded_by: null
|
||||
revises: null
|
||||
possible_duplicates: []
|
||||
observed_at: 2026-09-11T08:00:00Z
|
||||
valid_from: null
|
||||
valid_until: null
|
||||
source: {harness: seed}
|
||||
created: 2026-09-11T08:00:00Z
|
||||
updated: 2026-09-11T08:00:00Z
|
||||
---
|
||||
Entschieden am 2026-09-11: stündliche Snapshots statt sechsstündlich, weil der Vault handgeschrieben ist.
|
||||
@@ -1,22 +0,0 @@
|
||||
# Synthetic query set over eval/corpus (12 seeded notes). It exists so the
|
||||
# harness itself can be exercised offline with --fake-embeddings. It is NOT a
|
||||
# tuning set: the vectors are hashed word bags, not Qwen3, and 18 queries over
|
||||
# 12 notes cannot separate fusion variants. Real tuning needs eval/queries.yaml.
|
||||
- {q: "Welcher Router verteilt meine lokalen Modelle?", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
|
||||
- {q: "llama-swap", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
|
||||
- {q: "Which gateway sits in front of llama-swap?", relevant: ["01M00000000000000000000BFR"], category: single, lang: en}
|
||||
- {q: "Wo hält Bifrost die virtuellen Schlüssel?", relevant: ["01M00000000000000000000BFR"], category: single, lang: de}
|
||||
- {q: "Wie heißt die GPU-Box?", relevant: ["01M00000000000000000000EVX"], category: single, lang: de}
|
||||
- {q: "storage class for persistent volumes", relevant: ["01M00000000000000000000STG"], category: single, lang: en}
|
||||
- {q: "Welche StorageClass nutzt der Cluster?", relevant: ["01M00000000000000000000STG"], category: single, lang: de}
|
||||
- {q: "Wie oft laufen die Backups?", relevant: ["01M00000000000000000000VSY"], category: single, lang: de}
|
||||
- {q: "restic snapshot schedule", relevant: ["01M00000000000000000000VSY"], category: single, lang: en}
|
||||
- {q: "Wie melde ich mich bei den Web-UIs an?", relevant: ["01M00000000000000000000PKT"], category: single, lang: de}
|
||||
- {q: "Wie entkalke ich die Kaffeemaschine?", relevant: ["01M00000000000000000000KAF"], category: single, lang: de}
|
||||
- {q: "Pegel am Traunsee", relevant: ["01M00000000000000000000TRN"], category: single, lang: de}
|
||||
- {q: "Wie heißt der Modellrouter jetzt?", relevant: ["01M00000000000000000000BFR"], category: update, lang: de}
|
||||
- {q: "Welches Embedding-Modell und welche Quantisierung?", relevant: ["01M00000000000000000000QWN"], category: multi, lang: de}
|
||||
- {q: "Woher kommen die Container-Images und wer rollt sie aus?", relevant: ["01M00000000000000000000HRB", "01M00000000000000000000FXC"], category: multi, lang: de}
|
||||
- {q: "Was wurde am 11. September 2026 entschieden?", relevant: ["01M00000000000000000000VSY"], category: temporal, lang: de, since: "2026-09-01T00:00:00Z", until: "2026-12-31T00:00:00Z"}
|
||||
- {q: "Wie ist die Telefonnummer meines Zahnarztes?", relevant: [], category: abstain, lang: de}
|
||||
- {q: "Wann fährt der letzte Zug nach Wien?", relevant: [], category: abstain, lang: de}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,6 +1,6 @@
|
||||
# Query set for `mdmem eval` (§11).
|
||||
#
|
||||
# Copy to eval/queries.yaml and label 50–100 REAL queries — the ones you would
|
||||
# Copy to queries.yaml and label 50–100 REAL queries — the ones you would
|
||||
# actually ask your assistant. The harness is only as meaningful as this file.
|
||||
#
|
||||
# Coverage the plan asks for (≥10 each):
|
||||
+1
-1
@@ -32,7 +32,7 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
reindex.add_argument("--no-embed", action="store_true", help="lexical index only")
|
||||
|
||||
ev = sub.add_parser("eval", help="run the retrieval eval harness (§11)")
|
||||
ev.add_argument("--queries", default="eval/queries.yaml")
|
||||
ev.add_argument("--queries", default="queries.yaml")
|
||||
ev.add_argument("--config", default="default", help="name for this run's config")
|
||||
ev.add_argument("--out", default=None, help="write the JSON report here")
|
||||
ev.add_argument("--ablate", action="store_true", help="run the ablation matrix")
|
||||
|
||||
@@ -404,7 +404,7 @@ def run_eval_cli(settings: Settings, args: Any) -> int:
|
||||
queries_path = Path(args.queries)
|
||||
if not queries_path.exists():
|
||||
print(
|
||||
f"no query set at {queries_path}. Copy eval/queries.example.yaml and label "
|
||||
f"no query set at {queries_path}. Copy queries.example.yaml and label "
|
||||
"50–100 real queries first (§11)."
|
||||
)
|
||||
return 2
|
||||
@@ -435,7 +435,7 @@ def run_eval_cli(settings: Settings, args: Any) -> int:
|
||||
print(table)
|
||||
print()
|
||||
print(dumps_json({"decision": decision, "min_score": min_score}, indent=2, ensure_ascii=False))
|
||||
out = Path(args.out) if args.out else Path("eval/results") / f"{time.strftime('%Y-%m-%d')}.json"
|
||||
out = Path(args.out) if args.out else Path("results") / f"{time.strftime('%Y-%m-%d')}.json"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(
|
||||
dumps_json(
|
||||
|
||||
+1
-1
@@ -377,7 +377,7 @@ def _merge_lexical_hits(
|
||||
can. The trigram pass keeps its 0.5 discount as a rank weight: substring
|
||||
matching is the fuzzier signal, so a trigram hit has to be clearly better
|
||||
ranked than a whole-word hit to displace it, rather than winning a tie.
|
||||
Measured on eval/corpus, dropping the discount cost the rrf baseline about
|
||||
Measured on a synthetic corpus, dropping the discount cost the rrf baseline about
|
||||
0.03 nDCG@10 by promoting trigram hits over better whole-word ones.
|
||||
"""
|
||||
weight = 1.0 if source == "fts" else TRIGRAM_RANK_WEIGHT
|
||||
|
||||
+4
-4
@@ -620,20 +620,20 @@ def build_ui_app(svc: MemoryService) -> FastAPI: # noqa: C901 - one place, many
|
||||
`asyncio.run` from and keeps the response fast.
|
||||
|
||||
Both the query set and the report live on the data volume, not in the
|
||||
working directory: the pod runs with `readOnlyRootFilesystem: true`
|
||||
(§13a.3), so a relative `eval/results/...` path would fail at the write
|
||||
working directory: the pod runs with `readOnlyRootFilesystem: true`,
|
||||
so a relative `results/...` path would fail at the write
|
||||
with nothing but a log line to show for it.
|
||||
"""
|
||||
from mdmem.evalharness import run_eval_cli
|
||||
|
||||
data_dir = settings.index_db.parent
|
||||
queries = data_dir / "eval" / "queries.yaml"
|
||||
queries = data_dir / "results" / "queries.yaml"
|
||||
if not queries.exists():
|
||||
# §11: a run needs 50-100 labelled queries. Say so on the page
|
||||
# instead of starting a job that prints a hint into the log and
|
||||
# exits 2.
|
||||
return RedirectResponse(f"/ui/health?eval=missing:{queries}", status_code=303)
|
||||
out = data_dir / "eval" / f"{time.strftime('%Y-%m-%dT%H-%M-%SZ', time.gmtime())}.json"
|
||||
out = data_dir / "results" / f"{time.strftime('%Y-%m-%dT%H-%M-%SZ', time.gmtime())}.json"
|
||||
args = SimpleNamespace(
|
||||
queries=str(queries), config="ui", out=str(out), ablate=False, fake_embeddings=False
|
||||
)
|
||||
|
||||
@@ -62,7 +62,7 @@ def test_load_queries_validates(tmp_path):
|
||||
|
||||
|
||||
def test_the_shipped_example_query_set_parses():
|
||||
qs = load_queries("eval/queries.example.yaml")
|
||||
qs = load_queries("queries.example.yaml")
|
||||
assert len(qs) >= 5
|
||||
assert {q.category for q in qs} >= {"single", "temporal", "update", "abstain"}
|
||||
|
||||
|
||||
Reference in new issue
Block a user