Compare commits

...
2 Commits
Author SHA1 Message Date
Mdmem Agent 8fdec33037 Fix stale references to deleted files in README and source
- Remove references to PLAN.md, SECURITY_ANALYSIS.md, deploy/ paths
- Delete test_deploy.py (tests deleted K8s manifests)
- Clean up eval/ references in comments and docstrings
2026-09-14 13:40:09 +00:00
Mdmem Agent 8fd3e37b67 Remove internal docs, deploy manifests, and eval test artifacts
- Delete PLAN.md (LLM agent development brief)
- Delete SECURITY_ANALYSIS.md (internal audit report)
- Delete deploy/ (environment-specific K8s manifests with hardcoded infra)
- Remove eval/corpus/, eval/results/, eval/queries.synthetic.yaml, eval/README.md
- Move queries.example.yaml to root, update all path references
2026-09-14 13:31:12 +00:00
44 changed files with 48 additions and 11044 deletions

No files matched your search

-1183
View File
File diff suppressed because it is too large. Load diff
+32 -34
View File
@@ -4,16 +4,14 @@ Markdown-canonical agent memory service. One self-hosted process that gives ever
harness (Claude Code, claude.ai, Open WebUI, Hermes, opencode…) a single shared
long-term memory over MCP, with markdown files as the only source of truth.
Full design and its evidence: [`PLAN.md`](PLAN.md). Section numbers below refer to it.
- **Markdown is canonical.** `/data/vault/{inbox,active,archive}/*.md`. The SQLite
index is derived and rebuildable; delete it and the service rebuilds from the files.
- **Hybrid retrieval.** FTS5 (BM25, `unicode61 remove_diacritics 2`, optional trigram)
+ sqlite-vec KNN, fused with RRF (default) or convex combination. No graph traversal
in the retrieval path (§2.7).
in the retrieval path.
- **Agents propose, the owner disposes.** Agent writes land in `inbox/`; an agent
editing an approved note creates a *proposed revision*, never an in-place change.
There is no delete tool (§2.4).
There is no delete tool.
- **Memory content is untrusted data.** Tool descriptions and the MCP `instructions`
say so; the UI renders markdown with raw HTML disabled.
@@ -21,20 +19,19 @@ Full design and its evidence: [`PLAN.md`](PLAN.md). Section numbers below refer
```
src/mdmem/
config.py settings from the environment (§12)
config.py settings from the environment
notes.py frontmatter round-trip (ruamel): unknown keys, order and comments survive
vaultio.py atomic write → fsync → replace, pre-write history snapshot
chunker.py atomic notes are one chunk; long ones split on H2/H3 (§6)
db.py schema, sqlite-vec, single-writer rule (§5)
chunker.py atomic notes are one chunk; long ones split on H2/H3
db.py schema, sqlite-vec, single-writer rule
index.py vault → rows, links, duplicate hints
embed.py Bifrost client, content-hash cache, FakeEmbedder for tests (§7)
search.py the retrieval algorithm (§8)
embed.py Bifrost client, content-hash cache, FakeEmbedder for tests
search.py the retrieval algorithm
service.py the one code path every write goes through — MCP and UI alike
mcp_server.py six memory_* tools, bearer auth, idempotency, audit log (§9)
ui/ FastAPI + Jinja + HTMX, PocketID OIDC with a group check (§10)
evalharness.py metrics, bootstrap CIs, ablation matrix (§11)
deploy/ Kubernetes manifests, laid out for the fluxcd repo (§13a)
eval/ query-set template, synthetic corpus, results
mcp_server.py six memory_* tools, bearer auth, idempotency, audit log
ui/ FastAPI + Jinja + HTMX, PocketID OIDC with a group check
evalharness.py metrics, bootstrap CIs, ablation matrix
tests/ pytest suite
```
## Local development
@@ -55,14 +52,14 @@ UI_AUTH_DISABLED=1 LOG_STYLE=text uv run mdmem serve
```
`UI_AUTH_DISABLED=1` is for a port-forwarded local UI only. It logs an error on
startup, and **an HTTPRoute must never point at a listener running with it** (§14.19).
startup, and **an HTTPRoute must never point at a listener running with it**.
Other commands: `mdmem reindex [--full] [--no-embed]`, `mdmem doctor`,
`mdmem eval --queries eval/queries.yaml [--ablate]`.
`mdmem eval --queries queries.yaml [--ablate]`.
## Build and push
There is no CI runner in the cluster, so builds are manual from the workstation:
Builds are manual from the workstation:
```bash
podman build -t harbor.cloud.traunseenet.com/library/mdmem:0.1.0 -f Containerfile .
@@ -72,13 +69,14 @@ podman inspect --format '{{index .RepoDigests 0}}' harbor.cloud.traunseenet.com/
Put tag **and** digest in `deployment.yaml`. Never `:latest` + `imagePullPolicy: Always`
for this service: without a runner, an unpinned tag means the running code is not in
git, and a pod restart becomes an unplanned upgrade of your memory store (§13a.2, §14.20).
git, and a pod restart becomes an unplanned upgrade of your memory store.
## Deploy (first install)
## Deployment (FluxCD on RKE2)
`deploy/kubernetes/apps/apps/mdmem/**` mirrors the fluxcd repo's nesting; copy it in.
The manifests were written from §13a and validated with `kubectl kustomize`, but **not**
against a live cluster — verify each step.
Deployment manifests are maintained in a separate [fluxcd repo](https://github.com/traunseenet/fluxcd).
The mdmem manifests live under `kubernetes/apps/apps/mdmem/` and are managed by FluxCD.
### First install
1. Build and push the image; record the digest and put it in `deployment.yaml`.
2. Seed Vault:
@@ -96,8 +94,8 @@ against a live cluster — verify each step.
assume. Client id → `mdmem-env`, client secret → Vault.
4. Create the restic repo for `restic-mdmem`; copy `secret.example.yaml` to
`secret.yaml`, fill it in, SOPS-encrypt it.
5. Commit `kubernetes/apps/apps/mdmem/**`. Check: ExternalSecret `Ready`, PVC `Bound`,
Deployment healthy, `/readyz` 200.
5. Commit `kubernetes/apps/apps/mdmem/**` to the fluxcd repo. Check: ExternalSecret
`Ready`, PVC `Bound`, Deployment healthy, `/readyz` 200.
6. Verify embeddings end to end from inside the cluster:
```bash
kubectl -n apps run curl --rm -it --restart=Never \
@@ -111,19 +109,19 @@ against a live cluster — verify each step.
While you are there, confirm Bifrost's `semantic_cache` plugin does **not** intercept
`/v1/embeddings` — send two slightly different inputs and check the vectors differ. A
cache hit there would return a stale vector for changed text and silently corrupt the
index (§14.13). `uv run pytest -m live` checks this and the pooling/prefix behaviour.
7. Only then add `deploy/bifrost-mcp-client.json` to Bifrost's `config.json` and commit,
index. `uv run pytest -m live` checks this and the pooling/prefix behaviour.
7. Only then add the Bifrost MCP client configuration to Bifrost's `config.json` and commit,
so a failure is attributable to one change at a time. Create a VK scoped to the six
`memory_*` tools.
8. Apply the NetworkPolicy and re-check the two things it can silently break: the pod
stays Ready (kubelet probes) and the Prometheus target stays `UP` (§14.23).
stays Ready (kubelet probes) and the Prometheus target stays `UP`.
9. Run a `ReplicationSource` and **test a restore** into a throwaway PVC before the
vault holds anything you would miss.
### Rotating MCP_TOKEN
The token lives in two Vault paths. Rotating one and not the other gives every agent a
401 that looks like an mdmem bug (§14.15):
401 that looks like an mdmem bug:
```bash
NEW=$(openssl rand -hex 32)
@@ -145,7 +143,7 @@ All harnesses go through Bifrost — mdmem's own port is never exposed.
a browser-side connector flow may need `https://claude.ai` added.
- **Hermes / opencode** — remote MCP config, same URL + VK header.
## Where the implementation deviates from PLAN.md
## Where the implementation deviates from the plan
Each of these is a decision made while building, with the reason:
@@ -155,9 +153,9 @@ Each of these is a decision made while building, with the reason:
| 5 | `chunks_fts`/`chunks_tri` are `content=''` | ordinary FTS5 tables | Contentless FTS5 cannot delete a row before SQLite 3.43 (bookworm ships 3.40) and cannot `snippet()`. The index is rewritten on every edit, so deletes are the hot path. Costs a few MB of duplicated text. |
| 6 | "one asyncio task owns all SQLite writes (queue)" | one write connection behind a lock; async paths use `asyncio.to_thread` | Same invariant (exactly one writer), and it also holds for the CLI and the tests, which have no event loop of the service's. |
| 8 | vec0 cosine | vec0 L2 over L2-normalised vectors, `cos = 1 − d²/2` | Monotone-equivalent and does not depend on a `distance_metric=` option in a pre-v1 extension. |
| 9 | — | `MCP_ALLOWED_HOSTS` (new setting) | The SDK's DNS-rebinding protection checks the `Host` header. Without the in-cluster service name on the allow-list, every Bifrost call returns 421. Added to §14 as pitfall 24. |
| 10 | — | `UI_AUTH_DISABLED` (new setting) | Makes §10's "P4 may ship without OIDC, but then without an HTTPRoute" explicit and loud instead of implicit. |
| 11 | `eval/queries.yaml` with 50–100 real queries | `eval/queries.example.yaml` + a synthetic corpus and `--fake-embeddings` | Real queries are Markus's to write, and the real model is not reachable from the build environment. The harness is complete and tested; the committed run is a machinery check, and the defaults stay as §2.5 specifies. |
| 9 | — | `MCP_ALLOWED_HOSTS` (new setting) | The SDK's DNS-rebinding protection checks the `Host` header. Without the in-cluster service name on the allow-list, every Bifrost call returns 421. Added to pitfalls as pitfall 24. |
| 10 | — | `UI_AUTH_DISABLED` (new setting) | Makes the "may ship without OIDC, but then without an HTTPRoute" rule explicit and loud instead of implicit. |
| 11 | `queries.yaml` with 50–100 real queries | `queries.example.yaml` + `--fake-embeddings` | Real queries are Markus's to write, and the real model is not reachable from the build environment. The harness is complete and tested; the defaults stay as specified. |
## Status
@@ -166,13 +164,13 @@ Bifrost endpoint). P6 is written but unverified: no cluster, registry, Vault or
was reachable from the build environment, so the image was never built, the manifests
were never applied, and the live checks in the deploy list above are still to do.
The implementation has been through a multi-agent review against `PLAN.md`, which found
The implementation has been through a multi-agent review which found
35 defects — four of them data-losing (a YAML typo or a UTF-8 BOM destroying a note's
identity; `Vault.delete()` resolving the target from file contents and `rmtree`-ing
outside the vault; `Vault.write()` escaping the vault via a caller-supplied id; and
`mdmem eval --ablate` dropping the trigram table from the live index). All are fixed,
each with a regression test that was checked to fail when the fix is reverted. The
lessons are recorded as pitfalls 26–31 in `PLAN.md` §14.
lessons are recorded as pitfalls 26–31.
Three tests were also found to be green for the wrong reason — a dense-retrieval test
whose query was the note body verbatim, a double-index test that slept less than the
-572
View File
@@ -1,572 +0,0 @@
# Security, Correctness, and Plan-Drift Analysis
Date: 2026-09-12
Reviewed revision: `4cb9639` (`master`)
Reference contract: `PLAN.md`
Scope: `src/mdmem`, `tests`, `eval`, `deploy`, `Containerfile`, `README.md`
## Executive summary
The current test suite is green (`227 passed`) and `ruff check src tests` passes, but important
concurrency, data-integrity, authentication, and recovery behavior remains untested. This review
found:
- 2 critical findings
- 12 high-severity findings
- 18 medium-severity findings
- several lower-severity plan and coverage gaps
The two most urgent defects are:
1. lifecycle/version decisions are made before the service write lock is acquired, allowing
concurrent requests to resurrect rejected notes, overwrite accepted edits, or recreate a
deleted note; and
2. an unterminated frontmatter block is still classified as a new hand-written note and
destructively rewritten.
The next tier is dominated by stale embedding races, filename/frontmatter identity split-brain,
idempotency races, secret leakage through secondary write channels, and startup behavior that can
prevent the documented lexical-only degradation mode.
Severity here measures likely impact to confidentiality, integrity, availability, or the core
human-promotion trust boundary. It is not a CVSS score. No fixes were made as part of this review.
## Method
The review combined independent passes over storage/retrieval, service/MCP, and UI/deployment/eval,
then rechecked findings against the current working tree and `PLAN.md`. High-impact claims were
confirmed with controlled temporary-vault reproductions. No production vault or configured live
index was used.
Representative observed results included:
- reject/update race: `reject -> archive`, then stale update -> `inbox`; final body was the agent's
text and `rejected` was gone;
- two concurrent identical `memory_write` calls with one `request_id` created two distinct notes;
- delayed old embedding and new embedding both used chunk ID `1`; the old vector was stored against
the new chunk and `vec_pending` became `0`;
- deleting a closing `---` renamed and rewrote an active fact, changed its type to `note`, emptied
its tags, and moved the former frontmatter into the body without reporting an error;
- a valid allowed `Origin` paired with `Host: evil.example` was accepted;
- signed ID tokens with `exp: NaN`, `exp: Infinity`, multiple audiences plus a mismatched `azp`, and
a missing `sub` were accepted;
- a note written during the scan-to-watcher startup gap remained on disk but absent from the index;
- `MCP_TOKEN=...`, `AWS_SECRET_ACCESS_KEY=...`, `github_pat_...`, `glpat-...`, `npm_...`, and `hf_...`
examples all evaded the secret scanner.
## Critical findings
### C-01 — Lifecycle and optimistic-version checks are outside the serialized operation
Affected code: `src/mdmem/service.py:465-545`, `572-605`, `613-619`, `655-719`
Plan: §2.4, §6 single-writer rule, §9 optimistic versioning, §14.10
`update_note`, `approve`, `reject`, `edit`, and `hard_delete` read the note and make lifecycle or
version decisions before acquiring `_write_lock`. Only the eventual `_persist` or deletion block
is serialized. A queued operation therefore acts on a stale `Note` object after an earlier queued
operation has changed the canonical file.
Confirmed consequences:
- Two concurrent inbox updates with the same `expected_version` both succeed; one silently
overwrites the other and one returned version is stale immediately.
- A rejected note can be rewritten back into `inbox`, losing `rejected: true`.
- An update queued behind `hard_delete` can recreate the deleted ULID and file after deletion
reported success.
- An agent update queued behind approval can move an owner-approved note back to inbox with the
agent's text.
Required regression coverage: deterministic barrier tests for update/update, reject/update,
approve/update, archive/update, edit/update, and delete/update. The read, validation, mutation,
file write, and synchronous index update must be one serialized operation.
### C-02 — Missing closing frontmatter delimiter causes destructive “repair”
Affected code: `src/mdmem/notes.py:67-95`, `src/mdmem/index.py:264-305`
Plan: §4 identity/frontmatter preservation, §6.3 safe parsing, §14.25/§14.26
Strict parsing only raises for malformed YAML after the frontmatter regex has matched both
delimiters. If the closing `---` is missing, both lenient and strict parsing return an empty map.
`read_and_fix` then treats an existing canonical note as a clean hand-written note.
In the reproduction, an indexed active `fact` was renamed and rewritten, its real title was
replaced by the old filename stem, tags were erased, and the former YAML became body text. The
degraded metadata also replaced the good index row. Clearing `superseded_by` this way can make an
obsolete fact searchable again. The history snapshot made during repair contains the already
broken text and is not a dependable last-good recovery point.
Required regression coverage: remove only the closing delimiter, then exercise both full scan and
watcher sync. The file, path, identity, and last-good index row must remain unchanged while a clear
error is reported.
## High-severity findings
### H-01 — Delayed embeddings can be attached to newer text
Affected code: `src/mdmem/index.py:24-29,61-127,360-367`,
`src/mdmem/service.py:227-242,714-719`
Plan: §6 single-writer consistency, §7 recovery, §14.7
`PendingEmbed` records `key_hash`, but `_embed_items` discards it and `store_vectors` stores solely
by numeric chunk ID. Embedding happens after `_write_lock` is released. Reindexing deletes and
reinserts chunks, and SQLite can immediately reuse the old integer ID.
A delayed response for hash A can therefore overwrite the vector for current hash B and mark B as
complete. This affects ordinary writes, background retries, and `reembed_all`. If the old ID is not
reused, the code may instead attempt to create an orphan vec row.
Required regression: hold an old embed request open, complete an edit/reindex, then release it.
Store a vector only when both chunk ID and `embed_text_hash` still match in the same transaction.
### H-02 — Filename/frontmatter disagreement creates identity and lifecycle split-brain
Affected code: `src/mdmem/vaultio.py:55-69`, `src/mdmem/index.py:210-305`,
`src/mdmem/db.py:25-32`
Plan: §4 identity and folder lifecycle source of truth
`read_and_fix` trusts any syntactically valid frontmatter ULID without checking the ULID encoded in
the filename. Confirmed failure modes:
- Editing a canonical file's frontmatter ID to another ULID makes indexing fail on the unique path
constraint while preserving a stale row under the filename ID. The file's new ID cannot be found
by `Vault.find`.
- An active original plus an archive copy carrying the original frontmatter ID is accepted with no
conflict; last-file-wins indexing can hide the active original as archived while `Vault.find`
still reads the active file.
- A file such as `active/manual.md` with a valid frontmatter ID is indexed, but `Vault.find(id)` and
`memory_get` cannot locate it because lookup uses the filename ID.
Required regression: reject mismatched IDs and duplicate lifecycle copies without changing the
last-good row. Also test valid-frontmatter-ID files with noncanonical names.
### H-03 — Normal vault deletion leaves inbound links invisibly dangling
Affected code: `src/mdmem/index.py:153-158`, `src/mdmem/service.py:180-190`
Plan: §5 unresolved links, §6 vanished-file handling, §10 health report
The UI hard-delete path now reindexes inbound sources, but `Indexer.remove`—used by ordinary
filesystem deletion through scan/watcher—deletes only outgoing links. Inbound rows retain the
deleted ULID rather than reverting to `unresolved:<target>`, so the unresolved health count remains
zero and the broken reference is invisible until its source is reindexed for another reason.
Required regression: delete a target through both scan and watcher paths; surviving source links
must become unresolved and flags for the vanished note must be handled explicitly.
### H-04 — Service updates can write NFC to disk while indexing NFD
Affected code: `src/mdmem/notes.py:98-130`, `src/mdmem/vaultio.py:88-119`,
`src/mdmem/service.py:531-545,655-684,714-719`
Plan: §4 and §14.8 NFC normalization
`dumps` normalizes a deep copy, not the live `Note`. `_persist` writes the normalized serialization
and then indexes the original object. An inbox update with NFD text produced NFC on disk but NFD in
`chunks.body` and in the embedding input. Because the indexed file hash is the canonical disk hash,
the watcher considers the service event already handled and does not repair the mismatch.
Required regression: update and UI-edit with NFD in body, frontmatter, relations, and link targets;
assert the live note, disk, chunks, FTS/trigram text, links, and embedding input all use NFC.
### H-05 — Scan-to-watcher startup gap can permanently miss changes
Affected code: `src/mdmem/service.py:117-157`
Plan: §6 startup scan/watcher, P1 live-edit latency
The watcher starts only after scan, secret scan, self-test, and pending embeddings. A file changed
after its scan pass but before watcher registration emits no event. A controlled reproduction wrote
a second valid note while `embed_pending` was blocked; after startup finished, the file existed but
the note was not indexed.
Required regression: mutate a file after initial scan while startup embedding is paused. Start
watching before reconciliation or run a final scan after watcher registration.
### H-06 — A Bifrost outage can put Kubernetes into a cold-start restart loop
Affected code: `src/mdmem/server.py:67-72`, `src/mdmem/service.py:117-157`,
`src/mdmem/embed.py:160-199`, `deploy/kubernetes/apps/apps/mdmem/mdmem/app/deployment.yaml:58-69`
Plan: §7, §13a.3, §14.17 lexical-only degradation
The process awaits `svc.start()` before either listener binds. Startup synchronously awaits an
embedding self-test and pending embeddings. Each can spend three 60-second attempts. The deployment
has liveness starting after 30 seconds, no `startupProbe`, and the default failure threshold.
Kubernetes can kill the pod before `/healthz` ever binds, repeatedly preventing the promised
lexical-only mode.
Required regression: start the real server with a blocking/blackholed embedder and prove liveness,
readiness, and lexical search bind before the probe deadline. Deployment tests must cover the
startup probe budget.
### H-07 — Common explicit credential forms evade the secret scanner
Affected code: `src/mdmem/secrets.py:11-29`
Plan: §9 and §14.12
The generic assignment regex requires a word boundary before names such as `token` and `secret`.
Because underscore is a word character, common environment variable names evade it. Provider
patterns are also incomplete. Confirmed misses include `MCP_TOKEN=...`,
`AWS_SECRET_ACCESS_KEY=...`, `OPENAI_API_KEY=...`, `GITHUB_TOKEN=...`, `github_pat_...`, `glpat-...`,
`npm_...`, and `hf_...`.
Required regression: current provider token formats and standard environment assignments,
including `export`, quoting, mixed case where applicable, and every serialized note field.
### H-08 — Secondary persistent channels bypass secret rejection and can leak into audit
Affected code: `src/mdmem/service.py:556-569`, `src/mdmem/mcp_server.py:90-128,350-354`
Plan: §4 untrusted `source`, §9 audit, §14.12
`flag.reason` is agent-controlled and persisted in SQLite without a scan. `request_id` is stored and
logged raw without a size or secret check. Audit `source` is copied from self-reported note metadata
before note validation. A write rejected for `source.harness=AKIAIOSFODNN7EXAMPLE` still copied that
full key into `audit.jsonl`; the same value in a flag reason was stored and returned by `memory_get`.
This is also provenance confusion: the audit UI presents an untrusted, caller-asserted source as
attribution.
Required regression: inspect the audit file after rejected source/request ID values, and test flag
reason storage. Store bounded/redacted identifiers, not unchecked caller strings.
### H-09 — Idempotency is race-prone and has inconsistent key semantics
Affected code: `src/mdmem/mcp_server.py:83-129`, `src/mdmem/db.py:58-59`,
`src/mdmem/service.py:777-813`
Plan: §9 replay guarantee
The guard performs `get -> side effect -> put` without synchronization. Two concurrent identical
writes with one request ID both missed and created different notes. Concurrent different arguments
also both succeeded instead of returning `IDEMPOTENCY_CONFLICT`.
The table primary key is `request_id`, while lookup treats `(request_id, tool)` as the key.
`INSERT OR REPLACE` for a second tool deletes the first tool's replay record. Replaying the original
operation can then repeat its side effect.
Required regression: concurrent same/different arguments, same ID across tools, in-progress
requests, and failure after mutation but before idempotency persistence. Claim and commit an
idempotency key atomically before performing a mutating operation.
### H-10 — Revision approval has no target-version or target-lifecycle check
Affected code: `src/mdmem/service.py:494-528,572-605`
Plan: §2.4 and §9 revision review
A proposal records only `revises`, not the version it was based on. `_merge_revision` blindly loads
the current target and overwrites it. Confirmed outcomes:
- active v1 -> agent proposal v2 -> owner edit v3 -> approve proposal results in v2, silently losing
the newer owner edit;
- active target -> proposal -> reject/archive target -> approve proposal rewrites the archived
target while retaining rejection state, so a later restore promotes the agent text.
Required regression: target edited, archived, rejected, deleted, or superseded after proposal
creation. Approval should refuse stale proposals or require an explicit owner conflict resolution.
### H-11 — Supersede proposals bypass lifecycle and exclusivity invariants
Affected code: `src/mdmem/service.py:548-554,572-593`
Plan: §2.4, §2.6, §9
`supersede` accepts archived, rejected, inbox, and already-superseded targets. A rejected item can be
re-proposed through this path despite the explicit archived-note guard on update. Two successor
proposals can also both be approved: both remain active and point back to the old fact, while the
old note's singular `superseded_by` points only to the second. Search can expose contradictory
“current” facts.
Required regression: archived/rejected/inbox/already-superseded/nonexistent targets, concurrent and
sequential competing successors, and bidirectional chain consistency.
### H-12 — Zero and non-finite embeddings are accepted as healthy
Affected code: `src/mdmem/embed.py:39-42,178-188`, `src/mdmem/search.py:72-74`
Plan: §2.1 known all-zero failure, §7 startup self-test, §14.1
Normalization leaves zero vectors at zero and propagates NaN/Infinity. Shape checks accept them,
including during startup self-test. Two zero vectors have distance zero, which search converts to
cosine `1.0`, creating a falsely perfect semantic match.
Required regression: zero, NaN, and infinite vectors for self-test, documents, and queries. Invalid
vectors must not be cached or stored, and search must degrade to lexical.
## Medium-severity findings
### M-01 — Flags violate the Markdown source-of-truth and recovery contract
Affected code: `src/mdmem/db.py:53-56`, `src/mdmem/service.py:556-569`
Plan: §0 Markdown canonical; SQLite rebuildable
Flags exist only in SQLite. Deleting/rebuilding `index.db` from an unchanged vault changes
`flags_for(id)` from populated to empty. Either make flags canonical in Markdown or explicitly
change the backup/recovery contract and back up this mutable database state.
Missing test: flags survive full index recreation.
### M-02 — Temporal values are unvalidated strings compared lexicographically
Affected code: `src/mdmem/service.py:417-450`, `src/mdmem/search.py:194-199,327-332`
Plan: §2.6, §4 UTC timestamps, §8 filters
Malformed values, offsets, and `valid_from >= valid_until` are accepted. Lexical comparison is not
instant comparison: for example, equivalent offset and `Z` timestamps can filter differently.
Missing tests: RFC3339 validity, UTC normalization, ordering, equivalent instants, and boundary
semantics for stored values and search arguments.
### M-03 — Unicode and trigram lexical hits are not merged by better rank
Affected code: `src/mdmem/search.py:120-150`
Plan: §8.1
The implementation mixes raw BM25 magnitudes from different FTS tables and ignores trigram whenever
the same chunk has a unicode61 hit. This does not implement “keep the better rank per chunk.”
Missing test: controlled rank lists where a chunk is rank 50 in unicode61 and rank 1 in trigram.
### M-04 — Late link resolution misses equivalent slug spellings
Affected code: `src/mdmem/index.py:182-207`
Plan: §5 exact-title then slug resolution
Initial resolution slugifies a target, but late resolution updates only two literal unresolved
keys. `[[Later Note]]` remains unresolved when a later note titled `Later Note` arrives, although
both slugify identically.
Missing tests: whitespace, punctuation, capitalization, and umlaut variants on forward links.
### M-05 — Search accepts invalid bounds and can exceed `max_chars`
Affected code: `src/mdmem/search.py:303-318`
Plan: §8.7, §14.11
The first result is always emitted even when it exceeds the requested budget; zero or negative
budgets still return one result. A negative internal `k` uses Python slicing and can return nearly
all candidates. The current test explicitly permits an 800-character overrun, contradicting the
plan.
Missing tests: `k <= 0`, `max_chars <= 0`, oversized first title/snippet, exact-budget boundary, and
structured MCP validation.
### M-06 — One long token bypasses the chunk target
Affected code: `src/mdmem/chunker.py:50-89`
Plan: §6.4
`_split_words` does not split a single token longer than the limit. Agent content can reach 8,000
characters and hand-authored notes are unbounded, producing oversized chunks.
Missing test: a no-whitespace token larger than the chunk limit, with bounded lossless output.
### M-07 — `is_ulid` accepts overflow encodings rejected by the ULID library
Affected code: `src/mdmem/ids.py:13,38-39`
Plan: §4/§6 valid identity
Validation checks only length and alphabet. Values such as
`ZZZZZZZZZZZZZZZZZZZZZZZZZZ` pass but `ULID.from_str` rejects them as overflow.
Missing property: `is_ulid(value)` implies successful parsing by the canonical ULID implementation.
### M-08 — OIDC claim validation accepts non-finite expiry, missing subject, and bad `azp`
Affected code: `src/mdmem/ui/auth.py:210-245`
Plan: §10 fail closed
Signed tokens with `exp: NaN` and `exp: Infinity` pass because any int/float is accepted and
non-finite comparisons do not expire them. `sub` is not required, allowing an empty-subject session.
When `aud` contains multiple values, `azp` is not required or checked; a token with mdmem among its
audiences but `azp` naming another client was accepted.
Missing tests: finite expiry, required nonempty subject, and multi-audience authorized-party rules.
### M-09 — Scalar group claim is accepted contrary to the fail-closed plan
Affected code: `src/mdmem/ui/auth.py:60-90`, `tests/test_ui_auth.py:59-60`
Plan: §10, P4, §14.21
The code converts a scalar string group to a list, and a test deliberately blesses it. The plan
requires a present-but-wrong-type group claim to return 403.
Required change to coverage: only `list[str]` should pass; scalar, null, mapping, and mixed lists
must fail.
### M-10 — UI POST protection still does not validate Host
Affected code: `src/mdmem/ui/app.py:85-105,464-480`
Plan: §10 Origin/Host allowlist
The recent Origin/Referer fix correctly rejects headerless writes, but when configured origins are
present, Host is ignored. A request with allowed `Origin` and `Host: evil.example` was accepted.
Missing tests: invalid Host with valid Origin, proxy-forwarded host behavior, and the exact deployed
host allowlist.
### M-11 — JWKS is cached forever, breaking key rotation until restart
Affected code: `src/mdmem/ui/auth.py:149-165`
Plan: §10 operationally reliable OIDC
An unknown `kid` does not trigger one forced JWKS refresh. New logins fail after PocketID signing-key
rotation until the pod restarts.
Missing test: cache key A, publish key B, present a B-signed token, and verify one refresh succeeds.
### M-12 — Audit hashes omit actual arguments and unexpected errors vanish
Affected code: `src/mdmem/mcp_server.py:83-129,167,335-339`
Plan: §9 every tool call logged with argument hash, source, latency, IDs
Search hashes only query and `k`, omitting filters. Flag hashes only ID, omitting reason. Different
calls therefore receive identical audit fingerprints. `_guard` catches only `MdmemError`; an
injected database/programming exception produced no audit entry at all.
Missing tests: vary every argument and require a hash change; inject unexpected exceptions and
require `ok:false`, latency, request ID, and a bounded error class.
### M-13 — `memory_flag` is non-idempotent
Affected code: `src/mdmem/mcp_server.py:328-339`
Plan: §9 idempotency
The mutating flag tool has no `request_id`. A transport retry creates duplicate review items. Search
and get also omit the plan's broadly stated optional request ID, though they have no side effect.
Missing test: replaying one flag request creates exactly one flag.
### M-14 — Evaluation can recommend a variant with a catastrophic category regression
Affected code: `src/mdmem/evalharness.py:340-372`
Plan: §11 “without a regression in any category beyond noise”
`pick_default` checks only overall nDCG and confidence intervals. It never examines `per_category`.
Missing test: a candidate with a clear overall win but severe temporal or update regression must
remain unselected.
### M-15 — Eval output is not standards-compliant JSON
Affected code: `src/mdmem/evalharness.py:90-117,160-174,395-425`,
`eval/results/2026-09-12-synthetic-smoke.json:538-541`
Plan: §11 committed JSON result artifact
Unavailable metrics remain `float("nan")`; `json.dumps` emits raw `NaN`, which RFC 8259 does not
permit. The test named as a CLI writer test manually writes a different object and never calls
`run_eval_cli`, leaving serialization false-green.
Missing test: invoke the actual CLI and parse with a strict JSON parser; unavailable values should
serialize as `null`.
### M-16 — Eval latency is order-biased by a shared embedding cache
Affected code: `src/mdmem/evalharness.py:293-312`, `src/mdmem/service.py:99-115`
Plan: §11 fair ablation comparison
Each variant gets an isolated index but retains the configured `cache_db`. The first real-model
variant pays query embedding latency while later variants reuse cached query embeddings, making p95
latency incomparable.
Missing test: instrument backend query calls and prove equivalent cache state or perform an explicit
warm-up excluded from every variant's timing.
### M-17 — Required Prometheus signals are absent
Affected code: `src/mdmem/ui/app.py:75-145`
Plan: §13a.7
Embedding error count, search-latency histogram, and MCP calls by tool/outcome are not exported.
Search/write totals are Gauges rather than Counters. Existing tests only check that generic metrics
exist and omit note content.
Missing test: scrape and assert all required metric families and bounded label cardinality.
### M-18 — UI state filtering and state transitions do not enforce the displayed workflow
Affected code: `src/mdmem/ui/app.py:261-300`, `src/mdmem/service.py:638-653`
Plan: §2.4, §10 Search filters and review workflow
Search `state=inbox` includes both active and inbox notes, while `state=archive` includes active,
inbox, and archive. In addition, the generic state endpoint can promote an inbox revision or
supersede proposal directly to active, bypassing `approve` merge/chain behavior if called manually.
Missing tests: exact results for every state filter and forbidden lifecycle transitions that must
go through approval.
## Lower-severity plan drift and test-suite gaps
- Health has no “run eval” action required by §10 (`src/mdmem/ui/templates/health.html`,
`src/mdmem/ui/app.py:443-460`).
- History compares one snapshot only to current, not any two revisions as required by §10
(`src/mdmem/ui/app.py:313-324`).
- Browse does not offer sorting by tags as required by §10.
- Health audit rows do not display the recorded `source` field.
- “Edit & approve” converts an intentionally empty body to `None`, approving the old body instead
(`src/mdmem/ui/app.py:235-239`). Empty must be saved or explicitly rejected, not ignored.
- `AuditLog.recent` reads persisted history only while its in-memory list is completely empty. After
restart and one new call, Health shows only that call rather than the last 100 across restart
(`src/mdmem/logging.py:77-92`).
- `memory_supersede` omits relations, validity bounds, and source metadata available to
`memory_write`; either complete the §9 schema or narrow the contract.
- Bearer configuration accepts whitespace/non-ASCII values that clients cannot reliably present;
startup should validate a trimmed ASCII token with an explicit minimum length.
- `validate_id` is dead code. ID arguments have no schema pattern or maximum length and are echoed
in `NOT_FOUND` responses.
- The session test named “round trip and expiry” never advances time or tests expiration.
- No route-level OIDC test covers discovery, state, PKCE, nonce carriage, callback errors, cookie
flags, expiry, logout, or signing-key rotation.
- The OIDC HTTP client created in `build_ui_app` has no application shutdown hook calling
`OidcClient.aclose`.
- CLI tests cover help/version but not serve, doctor, reindex, eval, contradictory mode flags,
startup failures, or graceful shutdown.
- The note round-trip property test generates a body but does not assert its round trip and does not
property-test arbitrary unknown-key order/comments.
- The clean manifest template is not directly buildable: `app/kustomization.yaml` references the
intentionally absent encrypted `secret.yaml`; image digest and PocketID client ID remain release
placeholders. These are documented setup blockers, but there is no CI smoke test for the rendered
deployment overlay.
- The only committed evaluation is a synthetic fake-embedding smoke run. It proves harness wiring,
not the P5 acceptance criterion of 50–100 real labelled queries and a production-model tuning
report.
## Test integrity assessment
The suite is useful for sequential happy paths but systematically underrepresents interleavings,
recovery from canonical Markdown, deployment startup, and strict external-format validation.
Highest-value missing test groups, in order:
1. deterministic service concurrency tests around the whole read/check/write operation;
2. chunk-hash-guarded embedding races, including background retry and re-embed;
3. malformed frontmatter delimiter and filename/frontmatter identity conflict cases;
4. startup with a blocked embedding backend plus a mutation during watcher registration;
5. concurrent idempotency and crash-between-side-effect-and-record cases;
6. secret scanning and audit redaction across every persistent channel;
7. full OIDC route tests with strict claims and signing-key rotation;
8. actual CLI eval serialization through a strict JSON parser;
9. disaster rebuild proving all canonical state survives; and
10. deployment render/probe smoke tests.
Some current tests actively encode plan drift rather than merely omitting it:
- `tests/test_ui_auth.py` accepts a scalar group claim despite §14.21;
- `tests/test_search.py` permits the first result to exceed `max_chars` by up to one snippet;
- `tests/test_evalharness.py::test_cli_reports_and_writes_a_json_file` does not invoke the CLI it
names.
## Recommended remediation order
1. Serialize lifecycle/version operations end-to-end and add deterministic race tests.
2. Reject unterminated/mismatched frontmatter without touching the file or last-good index.
3. Guard vector writes by chunk ID plus embedding-text hash.
4. Make startup bind in lexical mode independently of Bifrost and close the scan/watch gap.
5. Make idempotency atomic and reconcile its database key.
6. Close all secret-bearing secondary channels and redact audit fields.
7. Enforce revision/supersede target version, lifecycle, and exclusivity.
8. Validate embeddings, timestamps, ULIDs, OIDC claims, Host, and response bounds.
9. Restore the Markdown-only recovery guarantee for flags or explicitly redesign it.
10. Repair eval selection/serialization/fairness, observability, and the lower-priority UI drift.
## Baseline notes
The following earlier findings are fixed in the reviewed revision and are intentionally not listed
as open defects above: full-field note secret scanning, mandatory `expected_version`, archived-note
update rejection in the sequential case, approval validation, audit-file wiring, ordinary
hard-delete link repair, background retry of pending vectors, safe re-embed on a simple outage,
trigram backfill after toggling it off/on, missing-Origin CSRF rejection, nonce/`exp`/`iat` presence
checks, eval index isolation, reciprocal synthetic supersede data, connection/cache shutdown, and
the direct fusion formula tests.
-19
View File
@@ -1,19 +0,0 @@
{
"_howto": "Add this object to mcp.client_configs in kubernetes/apps/apps/bifrost/bifrost/app/config.json. That file is a configMapGenerator input, so editing it is the whole change — the generated ConfigMap's hash suffix moves and Flux rolls the Deployment. Commit it only AFTER mdmem itself is healthy, so a failure is attributable to one change.",
"name": "mdmem",
"connection_type": "http",
"connection_string": "http://mdmem.apps.svc.cluster.local:8080/mcp",
"auth_type": "headers",
"headers": { "Authorization": "env.MDMEM_AUTH_HEADER" },
"is_ping_available": false,
"is_code_mode_client": false,
"_comment": "Long-term agent memory. Code Mode deliberately off: memory bodies are untrusted text, and code mode hands tool output to a model that then writes executable code. Revisit only after the prompt-injection behaviour is tested with a local model. tools_to_execute is an explicit allow-list, not '*', so a future mdmem release cannot add a destructive tool that silently becomes reachable.",
"tools_to_execute": [
"memory_search",
"memory_get",
"memory_write",
"memory_update",
"memory_supersede",
"memory_flag"
]
}
@@ -1,5 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./mdmem/ks.yaml
@@ -1,88 +0,0 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mdmem
labels:
app: mdmem
spec:
replicas: 1
strategy:
type: Recreate # SQLite + watchfiles need a single writer
selector:
matchLabels:
app: mdmem
template:
metadata:
labels:
app: mdmem
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch # Longhorn: else /data comes up root-owned
containers:
- name: mdmem
# Pin tag AND digest. Renovate does not track library/*, so the digest
# is bumped by hand in the same commit that pushes a build (§13a.2).
image: harbor.cloud.traunseenet.com/library/mdmem:0.1.0@sha256:4b0c21bec82f12ae5f6e3b5398fc6af31fecf3bfb5c0d96b44acd85bf638d1f4
imagePullPolicy: IfNotPresent
args: ["serve"]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
env:
- name: HOME
value: /tmp
- name: XDG_CACHE_HOME
value: /tmp
envFrom:
- configMapRef:
name: mdmem-env
- secretRef:
name: mdmem # from Vault via the ExternalSecret
ports:
- name: mcp
containerPort: 8080
- name: ui
containerPort: 8081
volumeMounts:
- name: data
mountPath: /data
- name: tmp
mountPath: /tmp
livenessProbe:
httpGet:
path: /healthz
port: ui
initialDelaySeconds: 30
periodSeconds: 30
startupProbe:
httpGet:
path: /healthz
port: ui
periodSeconds: 5
failureThreshold: 36
readinessProbe:
httpGet:
path: /readyz # never checks Bifrost (§14.17)
port: ui
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
volumes:
- name: data
persistentVolumeClaim:
claimName: mdmem-data
- name: tmp
emptyDir: {}
@@ -1,17 +0,0 @@
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: mdmem
spec:
refreshInterval: 5m
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: mdmem
dataFrom:
# KV-v2 mount is `secret`, so this reads secret/apps/mdmem:
# MCP_TOKEN, BIFROST_VK, OIDC_CLIENT_SECRET, SESSION_SECRET
- extract:
key: apps/mdmem
@@ -1,25 +0,0 @@
# Only the UI port is routed. :8080 (MCP) gets no HTTPRoute — Bifrost reaches it
# at mdmem.apps.svc.cluster.local:8080, same namespace (§13a.5).
#
# Do not apply this before the OIDC group check is live (§14.19): *.cloud is
# split-horizon, not private.
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: mdmem
spec:
parentRefs:
- name: traefik
namespace: networking
sectionName: websecure-cloud
hostnames:
- memory.cloud.traunseenet.com
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: mdmem
port: 8081
@@ -1,56 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: apps
resources:
- ./externalsecret.yaml
- ./secret.yaml # SOPS: restic-mdmem only (§12.1)
- ./pvc.yaml
- ./deployment.yaml
- ./service.yaml
- ./httproute.yaml
- ./networkpolicy.yaml
- ./servicemonitor.yaml
configMapGenerator:
- name: mdmem-env
literals:
# --- storage ---------------------------------------------------------
- VAULT_DIR=/data/vault
- INDEX_DB=/data/index.db
- CACHE_DB=/data/emb_cache.db
- HISTORY_DIR=/data/.history
- AUDIT_LOG=/data/audit.jsonl
# --- embeddings ------------------------------------------------------
- BIFROST_URL=http://bifrost.apps.svc.cluster.local:8080
# provider-prefixed; a bare alias 404s and looks like an mdmem bug (§7, §14.14)
- EMBED_MODEL=llamaswap/qwen3-embedding-4b
- EMBED_DIM=2560
# --- retrieval -------------------------------------------------------
- FUSION=rrf
- CC_ALPHA=0.5
- FTS_TRIGRAM=1
- INBOX_WEIGHT=0.8
- BM25_WEIGHTS=3,2,1.5,1
- INCLUDE_INBOX_DEFAULT=true
- MIN_SCORE=
- MAX_CHARS_DEFAULT=6000
- DUP_THRESHOLD=0.90
- MAX_NOTE_CHARS=8000
# --- listeners -------------------------------------------------------
- MCP_ADDR=0.0.0.0:8080
- UI_ADDR=0.0.0.0:8081
# the MCP SDK checks the Host header; Bifrost dials the service name (§14.24)
- MCP_ALLOWED_HOSTS=mdmem.apps.svc.cluster.local:8080,mdmem:8080,localhost:8080
# --- ui / auth -------------------------------------------------------
- UI_BASE_URL=https://memory.cloud.traunseenet.com
- UI_ALLOWED_ORIGINS=https://memory.cloud.traunseenet.com
- OIDC_ISSUER=https://pocketid.cloud.traunseenet.com
- OIDC_SCOPE=openid email profile groups
- OIDC_CLIENT_ID=dc2aa40e-4386-46cc-bae6-0259a2e7490e
- OIDC_GROUPS_CLAIM=groups
- OIDC_REQUIRED_GROUP=mdmem_admins
- SESSION_TTL_HOURS=12
# --- misc ------------------------------------------------------------
- LOG_LEVEL=info
- LOG_STYLE=json
- TZ=Europe/Vienna
@@ -1,49 +0,0 @@
# Required, not optional (§13a.5a). newt's egress policy allows the entire apps
# namespace on every port and reaches pods directly rather than through Traefik,
# so without this policy "not published through Pangolin" is a promise that
# lives in a remote dashboard instead of in git — and :8080, whose only control
# is a bearer token, would be one click from the internet.
#
# After applying, verify the two things it can silently break: the pod stays
# Ready (kubelet probes) and the Prometheus target stays UP (§14.23).
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: mdmem-ingress
spec:
podSelector:
matchLabels:
app: mdmem
policyTypes: ["Ingress"]
ingress:
# MCP: Bifrost only, same namespace
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: apps
podSelector:
matchLabels:
app: bifrost
ports:
- protocol: TCP
port: 8080
# UI: Traefik only — deliberately NOT newt, so Pangolin cannot bypass the gateway
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: networking
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8081
# /metrics scrape
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
ports:
- protocol: TCP
port: 8081
@@ -1,38 +0,0 @@
---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
name: mdmem-restore
spec:
trigger:
manual: restore-once
restic:
repository: restic-mdmem
copyMethod: Snapshot
storageClassName: longhorn-storage
volumeSnapshotClassName: longhorn-snapshot
accessModes: ["ReadWriteOnce"]
capacity: 5Gi
cacheCapacity: 1Gi
enableFileDeletion: false
moverSecurityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mdmem-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: longhorn-storage
# Do NOT set numberOfReplicas: 1 here. The vault is the primary artefact of
# this service; an hourly restic snapshot is not block-level redundancy.
dataSourceRef:
kind: ReplicationDestination
apiGroup: volsync.backube
name: mdmem-restore
resources:
requests:
storage: 5Gi
@@ -1,17 +0,0 @@
# Rename to secret.yaml and SOPS-encrypt before committing. This is the ONE
# secret that stays out of Vault (§12.1): the restore path must work when Vault
# itself is what is being restored.
#
# sops --encrypt --age age1q3mu6g0l3kr7spszks69yqrvlxvlgfrqgw9r00qqt8cj9xpjd9mqdkd7um \
# --encrypted-regex '^(data|stringData)$' --in-place secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: restic-mdmem
type: Opaque
stringData:
RESTIC_REPOSITORY: s3:https://s3.example/restic-mdmem
RESTIC_PASSWORD: REPLACE
AWS_ACCESS_KEY_ID: REPLACE
AWS_SECRET_ACCESS_KEY: REPLACE
@@ -1,17 +0,0 @@
---
apiVersion: v1
kind: Service
metadata:
name: mdmem
labels:
app: mdmem
spec:
selector:
app: mdmem
ports:
- name: mcp
port: 8080
targetPort: mcp
- name: ui
port: 8081
targetPort: ui
@@ -1,18 +0,0 @@
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: mdmem
labels:
app: mdmem
spec:
endpoints:
- port: ui
path: /metrics
interval: 15s
honorLabels: true
namespaceSelector:
matchNames: ["apps"]
selector:
matchLabels:
app: mdmem
@@ -1,6 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: apps
resources:
- ./pvc-backup.yaml
@@ -1,30 +0,0 @@
# Hourly, not six-hourly: the vault accumulates irreplaceable hand-written
# content between snapshots, unlike a re-derivable app database (§13a.4).
# The whole PVC is backed up — excluding index.db/emb_cache.db would only buy a
# full re-embed at restore time.
---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: mdmem
spec:
sourcePVC: mdmem-data
trigger:
schedule: "0 * * * *"
restic:
repository: restic-mdmem
copyMethod: Snapshot
storageClassName: longhorn-storage
volumeSnapshotClassName: longhorn-snapshot
cacheCapacity: 1Gi
pruneIntervalDays: 1
retain:
hourly: 5
daily: 4
weekly: 3
monthly: 2
yearly: 1
moverSecurityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
@@ -1,58 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: mdmem-app
namespace: flux-system
spec:
targetNamespace: apps
commonMetadata:
labels:
app.kubernetes.io/name: mdmem
dependsOn:
- name: traefik-base
- name: longhorn-addons
- name: cert-manager-addons
- name: volsync
- name: harbor-app # the image is pulled from in-cluster Harbor
- name: hashicorp-vault-app # the ExternalSecret needs Vault reachable
- name: external-secrets-app
- name: bifrost-app # embeddings + the MCP front door
path: ./kubernetes/apps/apps/mdmem/mdmem/app
prune: true
sourceRef:
kind: GitRepository
name: fluxcd-kubernetes
namespace: flux-system
wait: true
interval: 15m
retryInterval: 15m
timeout: 15m
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: mdmem
namespace: apps
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: mdmem-backup
namespace: flux-system
spec:
targetNamespace: apps
commonMetadata:
labels:
app.kubernetes.io/name: mdmem
dependsOn:
- name: mdmem-app
- name: volsync
path: ./kubernetes/apps/apps/mdmem/mdmem/backup
prune: true
sourceRef:
kind: GitRepository
name: fluxcd-kubernetes
namespace: flux-system
interval: 15m
retryInterval: 15m
timeout: 15m
-25
View File
@@ -1,25 +0,0 @@
# Add to llama-swap's config on evo-x2 (10.0.0.59:9000) — the box Bifrost's
# `llamaswap` provider already points at. Not managed by Flux; llama-swap runs
# off-cluster (§7).
#
# --pooling last is mandatory: Qwen3 embeddings use last-token pooling and the
# wrong pooling degrades retrieval silently (§14.1, [R2][R3]).
# Never add --reranking to this instance: with both flags on, a reported
# llama.cpp bug returns all-zero embeddings (§14.2, [R5]).
# Q8_0, not Q4_K: Q4_K measured 0.974 cosine against the reference (§14.3, [R4]).
models:
qwen3-embedding-4b:
cmd: >
llama-server --port ${PORT}
-m /models/Qwen3-Embedding-4B-Q8_0.gguf
--embedding --pooling last
-ub 8192 -c 8192 -ngl 99
ttl: 0 # keep resident; an embedding call must not evict the chat model
# Keep it co-resident with the chat model rather than swapping. Check the group
# syntax against your llama-swap version:
# groups:
# resident:
# swap: false
# exclusive: false
# members: ["qwen3-embedding-4b", "<your chat model>"]
-44
View File
@@ -1,44 +0,0 @@
# Eval harness (§11)
```bash
mdmem eval --queries eval/queries.yaml --config default # one config
mdmem eval --queries eval/queries.yaml --ablate # the whole matrix
```
Both write `eval/results/<date>.json` and print a markdown table plus a
recommendation. The recommendation only names a new default when its nDCG@10
95 % bootstrap CI clears the baseline's — a two-point difference on 50 queries
is noise, not a result.
`--ablate` covers: lex-only, sem-only, RRF, CC with α ∈ {0.3 … 0.9}, trigram
off, inbox weight 1.0, flat BM25 weights, and embedding dims 1024 / 512. Every
variant runs at the same `MAX_CHARS_DEFAULT` budget, because a retrieval
comparison at different context volumes measures the volume [R16]. Dim variants
build their own throwaway index and delete it afterwards.
`calibrate_min_score()` reports the `MIN_SCORE` that maximises abstention F1 on
the `category: abstain` queries. It is not applied automatically — set it in the
ConfigMap once you believe it.
## Runs so far
| file | what it is | conclusion |
|---|---|---|
| `results/2026-09-13-synthetic-smoke.json` | 18 synthetic queries over the 12 seeded notes in `corpus/`, run with `--fake-embeddings` | **The harness works; nothing about the config was learned.** Vectors are hashed word bags, not Qwen3, and 18 queries cannot separate fusion variants — every CC arm's 95 % CI overlaps the RRF baseline's. Defaults stay as §2.5 specifies: `FUSION=rrf`, `EMBED_DIM=2560`, `MIN_SCORE` unset. |
`rrf`, `rrf-inbox-1.0` and `rrf-bm25-flat` report the *same* nDCG@10 in that run.
That is expected here and is not the shared-state corruption an earlier report had:
none of the 12 seeded notes sit in `inbox/`, so the inbox weight has nothing to
act on, and the two lexical subchannels are fused by ordinal rank (§8.1), so a
BM25 weight change only moves the score if it reorders the hits — on a corpus this
small, flat weights do not.
The abstention F1 in that run is 0 because `MIN_SCORE` is unset by default, so
the two `abstain` queries still return their best (wrong) match. The calibration
block in the report names the threshold that would maximise F1 — on a real,
labelled set it is the number to put in the ConfigMap.
**Before tuning anything**: label 50–100 real queries in `eval/queries.yaml`,
run `mdmem eval --ablate` against the real Bifrost endpoint, and only then move
a default. A variant ships only when its nDCG@10 CI clears the baseline's and no
category regresses.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000BFR
title: Bifrost ist das AI-Gateway
type: fact
summary: Bifrost bündelt alle LLM-Aufrufe und hält die virtuellen Schlüssel.
keywords: [bifrost, gateway, virtual key, api gateway, modellrouter]
tags: [homelab, ai]
relations: {}
supersedes: [01M00000000000000000000TEM]
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Bifrost ist der einzige Eingang für LLM-Aufrufe und MCP-Clients. Es steht vor llama-swap.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000TEM
title: Der Modellrouter hieß früher LiteLLM
type: fact
summary: Vor Bifrost lief LiteLLM als Gateway. Ersetzt.
keywords: [litellm, gateway, alt, modellrouter]
tags: [homelab, ai]
relations: {}
supersedes: []
superseded_by: 01M00000000000000000000BFR
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Historisch: LiteLLM war das Gateway bis 2026.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000EVX
title: evo-x2
type: entity
summary: Die GPU-Box im Keller mit 128 GB unified memory.
keywords: [evo-x2, gpu, hardware, workstation]
tags: [homelab]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
evo-x2 ist die Inferenz-Box unter 10.0.0.59. Beherbergt llama-swap.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000FXC
title: FluxCD deployt alles
type: fact
summary: Jede App kommt über FluxCD-Kustomizations aus dem fluxcd-Repo.
keywords: [fluxcd, gitops, kustomization, deployment, ausrollen]
tags: [homelab, k8s]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Der Cluster ist RKE2 mit FluxCD. Kein manuelles kubectl apply.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000HRB
title: Harbor spiegelt die Images
type: fact
summary: harbor.cloud.traunseenet.com ist Proxy-Cache und Registry für eigene Images.
keywords: [harbor, registry, images, proxy cache, container]
tags: [homelab, k8s]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Eigene Images liegen unter library/, gespiegelte unter docker.io/ und ghcr.io/.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000KAF
title: Kaffeemaschine entkalken
type: procedure
summary: Alle drei Monate mit Essigessenz entkalken.
keywords: [kaffee, entkalken, wartung, descale, maintenance]
tags: [haushalt]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
1. Tank mit 1:4 Essigessenz füllen. 2. Zwei Durchläufe. 3. Zweimal klar spülen.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000SWP
title: llama-swap routet lokale Modelle
type: fact
summary: llama-swap tauscht Modelle auf evo-x2; Bifrost steht davor.
keywords: [llama-swap, modellrouter, evo-x2, model router, inference]
tags: [homelab, ai]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
llama-swap läuft auf [[evo-x2]] und tauscht llama-server-Modelle. Bifrost adressiert es als llamaswap/<modell>.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000STG
title: Longhorn liefert die Volumes
type: fact
summary: Longhorn ist die StorageClass im RKE2-Cluster, nicht democratic-csi.
keywords: [longhorn, storage, pvc, storageclass, volumes, speicher]
tags: [homelab, k8s]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Persistente Volumes kommen von longhorn-storage. RWO, repliziert.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000PKT
title: PocketID macht das SSO
type: fact
summary: Alle Web-UIs authentifizieren sich per OIDC gegen PocketID.
keywords: [pocketid, oidc, sso, login, authentifizierung]
tags: [homelab, k8s]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Es gibt kein forward-auth im Cluster. Jede App macht OIDC selbst gegen PocketID.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000QWN
title: Qwen3-Embedding-4B als Embedder
type: decision
summary: Q8_0-Quant, last-token pooling, 2560 Dimensionen.
keywords: [qwen3, embedding, q8_0, pooling, vektoren, einbettung]
tags: [homelab, ai]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Entschieden: 4B statt 8B (kleiner Abstand) und Q8_0 statt Q4_K (Drift).
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000TRN
title: Traunsee Wasserstand
type: note
summary: Pegel Gmunden liegt im Sommer meist um 422 m.
keywords: [traunsee, pegel, wasserstand, lake level, gmunden]
tags: [region]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Der Pegel Gmunden schwankt zwischen 421,5 m und 423 m.
@@ -1,20 +0,0 @@
---
id: 01M00000000000000000000VSY
title: VolSync sichert stündlich nach Restic
type: decision
summary: Backups laufen stündlich per VolSync-ReplicationSource in ein Restic-Repo.
keywords: [volsync, restic, backup, snapshot, sicherung]
tags: [homelab, k8s]
relations: {}
supersedes: []
superseded_by: null
revises: null
possible_duplicates: []
observed_at: 2026-09-11T08:00:00Z
valid_from: null
valid_until: null
source: {harness: seed}
created: 2026-09-11T08:00:00Z
updated: 2026-09-11T08:00:00Z
---
Entschieden am 2026-09-11: stündliche Snapshots statt sechsstündlich, weil der Vault handgeschrieben ist.
-22
View File
@@ -1,22 +0,0 @@
# Synthetic query set over eval/corpus (12 seeded notes). It exists so the
# harness itself can be exercised offline with --fake-embeddings. It is NOT a
# tuning set: the vectors are hashed word bags, not Qwen3, and 18 queries over
# 12 notes cannot separate fusion variants. Real tuning needs eval/queries.yaml.
- {q: "Welcher Router verteilt meine lokalen Modelle?", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
- {q: "llama-swap", relevant: ["01M00000000000000000000SWP"], category: single, lang: de}
- {q: "Which gateway sits in front of llama-swap?", relevant: ["01M00000000000000000000BFR"], category: single, lang: en}
- {q: "Wo hält Bifrost die virtuellen Schlüssel?", relevant: ["01M00000000000000000000BFR"], category: single, lang: de}
- {q: "Wie heißt die GPU-Box?", relevant: ["01M00000000000000000000EVX"], category: single, lang: de}
- {q: "storage class for persistent volumes", relevant: ["01M00000000000000000000STG"], category: single, lang: en}
- {q: "Welche StorageClass nutzt der Cluster?", relevant: ["01M00000000000000000000STG"], category: single, lang: de}
- {q: "Wie oft laufen die Backups?", relevant: ["01M00000000000000000000VSY"], category: single, lang: de}
- {q: "restic snapshot schedule", relevant: ["01M00000000000000000000VSY"], category: single, lang: en}
- {q: "Wie melde ich mich bei den Web-UIs an?", relevant: ["01M00000000000000000000PKT"], category: single, lang: de}
- {q: "Wie entkalke ich die Kaffeemaschine?", relevant: ["01M00000000000000000000KAF"], category: single, lang: de}
- {q: "Pegel am Traunsee", relevant: ["01M00000000000000000000TRN"], category: single, lang: de}
- {q: "Wie heißt der Modellrouter jetzt?", relevant: ["01M00000000000000000000BFR"], category: update, lang: de}
- {q: "Welches Embedding-Modell und welche Quantisierung?", relevant: ["01M00000000000000000000QWN"], category: multi, lang: de}
- {q: "Woher kommen die Container-Images und wer rollt sie aus?", relevant: ["01M00000000000000000000HRB", "01M00000000000000000000FXC"], category: multi, lang: de}
- {q: "Was wurde am 11. September 2026 entschieden?", relevant: ["01M00000000000000000000VSY"], category: temporal, lang: de, since: "2026-09-01T00:00:00Z", until: "2026-12-31T00:00:00Z"}
- {q: "Wie ist die Telefonnummer meines Zahnarztes?", relevant: [], category: abstain, lang: de}
- {q: "Wann fährt der letzte Zug nach Wien?", relevant: [], category: abstain, lang: de}
File diff suppressed because it is too large. Load diff
@@ -1,6 +1,6 @@
# Query set for `mdmem eval` (§11).
#
# Copy to eval/queries.yaml and label 50–100 REAL queries — the ones you would
# Copy to queries.yaml and label 50–100 REAL queries — the ones you would
# actually ask your assistant. The harness is only as meaningful as this file.
#
# Coverage the plan asks for (≥10 each):
+1 -1
View File
@@ -32,7 +32,7 @@ def build_parser() -> argparse.ArgumentParser:
reindex.add_argument("--no-embed", action="store_true", help="lexical index only")
ev = sub.add_parser("eval", help="run the retrieval eval harness (§11)")
ev.add_argument("--queries", default="eval/queries.yaml")
ev.add_argument("--queries", default="queries.yaml")
ev.add_argument("--config", default="default", help="name for this run's config")
ev.add_argument("--out", default=None, help="write the JSON report here")
ev.add_argument("--ablate", action="store_true", help="run the ablation matrix")
+2 -2
View File
@@ -404,7 +404,7 @@ def run_eval_cli(settings: Settings, args: Any) -> int:
queries_path = Path(args.queries)
if not queries_path.exists():
print(
f"no query set at {queries_path}. Copy eval/queries.example.yaml and label "
f"no query set at {queries_path}. Copy queries.example.yaml and label "
"50–100 real queries first (§11)."
)
return 2
@@ -435,7 +435,7 @@ def run_eval_cli(settings: Settings, args: Any) -> int:
print(table)
print()
print(dumps_json({"decision": decision, "min_score": min_score}, indent=2, ensure_ascii=False))
out = Path(args.out) if args.out else Path("eval/results") / f"{time.strftime('%Y-%m-%d')}.json"
out = Path(args.out) if args.out else Path("results") / f"{time.strftime('%Y-%m-%d')}.json"
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(
dumps_json(
+1 -1
View File
@@ -377,7 +377,7 @@ def _merge_lexical_hits(
can. The trigram pass keeps its 0.5 discount as a rank weight: substring
matching is the fuzzier signal, so a trigram hit has to be clearly better
ranked than a whole-word hit to displace it, rather than winning a tie.
Measured on eval/corpus, dropping the discount cost the rrf baseline about
Measured on a synthetic corpus, dropping the discount cost the rrf baseline about
0.03 nDCG@10 by promoting trigram hits over better whole-word ones.
"""
weight = 1.0 if source == "fts" else TRIGRAM_RANK_WEIGHT
+1 -1
View File
@@ -90,7 +90,7 @@ class MemoryService:
# retry loop, embed_pending/_embed_items). Cancelling the asyncio task
# that awaits one of these does NOT stop the worker thread, so stop()
# must wait for them to actually finish before closing the db out from
# under them (§ SECURITY_ANALYSIS.md, segfault in Indexer.pending_embeds).
# under them (segfault risk in Indexer.pending_embeds).
self._inflight: set[asyncio.Future[Any]] = set()
self._secret_warnings: list[str] = []
self.metrics = {"search": 0, "writes": 0, "embed_calls": 0}
+4 -4
View File
@@ -620,20 +620,20 @@ def build_ui_app(svc: MemoryService) -> FastAPI: # noqa: C901 - one place, many
`asyncio.run` from and keeps the response fast.
Both the query set and the report live on the data volume, not in the
working directory: the pod runs with `readOnlyRootFilesystem: true`
(§13a.3), so a relative `eval/results/...` path would fail at the write
working directory: the pod runs with `readOnlyRootFilesystem: true`,
so a relative `results/...` path would fail at the write
with nothing but a log line to show for it.
"""
from mdmem.evalharness import run_eval_cli
data_dir = settings.index_db.parent
queries = data_dir / "eval" / "queries.yaml"
queries = data_dir / "results" / "queries.yaml"
if not queries.exists():
# §11: a run needs 50-100 labelled queries. Say so on the page
# instead of starting a job that prints a hint into the log and
# exits 2.
return RedirectResponse(f"/ui/health?eval=missing:{queries}", status_code=303)
out = data_dir / "eval" / f"{time.strftime('%Y-%m-%dT%H-%M-%SZ', time.gmtime())}.json"
out = data_dir / "results" / f"{time.strftime('%Y-%m-%dT%H-%M-%SZ', time.gmtime())}.json"
args = SimpleNamespace(
queries=str(queries), config="ui", out=str(out), ablate=False, fake_embeddings=False
)
-184
View File
@@ -1,184 +0,0 @@
"""Smoke tests for the committed Kubernetes overlay (PLAN §13a).
The overlay is not directly buildable: `secret.yaml` is SOPS-encrypted and kept out
of the repo. The image digest and the PocketID client id are filled in, but both are
release-time values — the digest is bumped by hand on every build (§13a.2) — so their
placeholder spellings stay on the allowlist below rather than being treated as a
defect if they reappear.
Nothing else checked that the overlay stayed coherent: a renamed environment
variable, a probe pointing at a port the container does not expose, or a resource
dropped from the kustomization would only surface on a cluster.
"""
from pathlib import Path
import pytest
import yaml
APP = Path("deploy/kubernetes/apps/apps/mdmem/mdmem/app")
DEPLOYMENT = APP / "deployment.yaml"
# Documented, intentional gaps: the overlay cannot render as committed.
ABSENT_BY_DESIGN = {"./secret.yaml"}
RELEASE_PLACEHOLDERS = ("REPLACE_WITH_DIGEST", "REPLACE-WITH-THE-POCKETID-CLIENT-UUID")
def _load(path: Path) -> dict:
"""The overlay has multi-document files; take the first real document."""
docs = [d for d in yaml.safe_load_all(path.read_text(encoding="utf-8")) if d]
return docs[0]
def _load_all(path: Path) -> list[dict]:
return [d for d in yaml.safe_load_all(path.read_text(encoding="utf-8")) if d]
@pytest.fixture(scope="module")
def deployment() -> dict:
return _load(DEPLOYMENT)
@pytest.fixture(scope="module")
def container(deployment) -> dict:
return deployment["spec"]["template"]["spec"]["containers"][0]
@pytest.fixture(scope="module")
def kustomization() -> dict:
return _load(APP / "kustomization.yaml")
@pytest.fixture(scope="module")
def env_literals(kustomization) -> dict[str, str]:
generator = next(g for g in kustomization["configMapGenerator"] if g["name"] == "mdmem-env")
out = {}
for literal in generator["literals"]:
key, _, value = literal.partition("=")
out[key] = value
return out
def test_every_manifest_in_the_overlay_parses():
for path in sorted(APP.glob("*.yaml")):
docs = _load_all(path)
assert docs, f"{path} is empty or unparseable"
for doc in docs:
assert doc.get("kind") or "kustomize" in str(doc.get("apiVersion", "")), path
def test_kustomization_resources_exist_apart_from_the_sops_secret(kustomization):
for resource in kustomization["resources"]:
if resource in ABSENT_BY_DESIGN:
# ...but its template must be committed so the gap is fixable
assert (APP / "secret.example.yaml").exists()
continue
assert (APP / resource).exists(), f"{resource} is referenced but missing"
def test_the_only_unrendered_placeholders_are_the_documented_release_values():
"""If a new placeholder creeps in, it must be declared here, not discovered
during a deploy."""
found = set()
for path in sorted(APP.glob("*.yaml")):
text = path.read_text(encoding="utf-8")
for token in ("REPLACE_WITH", "REPLACE-WITH", "CHANGEME", "TODO:"):
if token in text:
found.update(w for w in text.split() if token in w)
unexpected = {f for f in found if not any(p in f for p in RELEASE_PLACEHOLDERS)}
assert not unexpected, f"undocumented placeholders in the overlay: {sorted(unexpected)}"
def test_configmap_env_keys_are_all_read_by_the_application(env_literals):
"""Catches a variable renamed in config.py but not in the overlay (or vice versa):
a silently ignored EMBED_MODEL would serve lexical-only in production."""
src = "\n".join(p.read_text(encoding="utf-8") for p in Path("src/mdmem").rglob("*.py"))
unread = [key for key in env_literals if f'"{key}"' not in src]
assert not unread, f"the overlay sets variables the app never reads: {unread}"
def test_secret_supplied_variables_are_not_duplicated_in_the_configmap(env_literals):
"""MCP_TOKEN and friends come from Vault via the ExternalSecret; a plaintext
copy in the ConfigMap would both leak and win or lose unpredictably."""
secret_keys = {"MCP_TOKEN", "BIFROST_VK", "OIDC_CLIENT_SECRET", "SESSION_SECRET"}
assert not (secret_keys & set(env_literals))
def test_embed_model_carries_its_provider_prefix(env_literals):
"""A bare alias 404s at Bifrost and looks like an mdmem bug (§7, §14.14)."""
assert "/" in env_literals["EMBED_MODEL"]
def test_listener_addresses_match_the_declared_container_ports(env_literals, container):
ports = {p["name"]: p["containerPort"] for p in container["ports"]}
assert env_literals["MCP_ADDR"].endswith(f":{ports['mcp']}")
assert env_literals["UI_ADDR"].endswith(f":{ports['ui']}")
# the MCP SDK checks Host, so every allowed host must name the MCP port
for host in env_literals["MCP_ALLOWED_HOSTS"].split(","):
assert host.strip().endswith(f":{ports['mcp']}"), host
def test_probes_point_at_ports_the_container_actually_exposes(container):
names = {p["name"] for p in container["ports"]}
for probe in ("livenessProbe", "readinessProbe", "startupProbe"):
assert container[probe]["httpGet"]["port"] in names
def test_mdmem_deployment_has_startup_probe_budget(container):
"""§14.17: the pod must be allowed to come up in lexical-only mode while the
embedding backend is unreachable, instead of being restarted mid-startup."""
probe = container["startupProbe"]
assert probe["httpGet"] == {"path": "/healthz", "port": "ui"}
assert probe["periodSeconds"] * probe["failureThreshold"] >= 180
def test_liveness_is_dumb_and_readiness_never_depends_on_bifrost(container):
assert container["livenessProbe"]["httpGet"]["path"] == "/healthz"
assert container["readinessProbe"]["httpGet"]["path"] == "/readyz"
def test_the_pod_stays_locked_down(deployment, container):
pod_sc = deployment["spec"]["template"]["spec"]["securityContext"]
assert pod_sc["runAsNonRoot"] is True
sc = container["securityContext"]
assert sc["readOnlyRootFilesystem"] is True
assert sc["allowPrivilegeEscalation"] is False
assert sc["capabilities"]["drop"] == ["ALL"]
def test_read_only_root_filesystem_has_somewhere_to_write(deployment, container):
"""readOnlyRootFilesystem only holds if HOME/XDG_CACHE_HOME point at a mounted
writable path."""
mounts = {m["name"]: m["mountPath"] for m in container["volumeMounts"]}
env = {e["name"]: e["value"] for e in container["env"]}
volumes = {v["name"] for v in deployment["spec"]["template"]["spec"]["volumes"]}
assert mounts["tmp"] == "/tmp" and "tmp" in volumes
assert env["HOME"] == "/tmp" and env["XDG_CACHE_HOME"] == "/tmp"
assert mounts["data"] == "/data" and "data" in volumes
def test_every_persistent_path_lives_on_the_data_volume(env_literals, container):
mounts = {m["name"]: m["mountPath"] for m in container["volumeMounts"]}
for key in ("VAULT_DIR", "INDEX_DB", "CACHE_DB", "HISTORY_DIR", "AUDIT_LOG"):
assert env_literals[key].startswith(mounts["data"] + "/"), key
def test_sqlite_requires_a_single_writer(deployment):
assert deployment["spec"]["replicas"] == 1
assert deployment["spec"]["strategy"]["type"] == "Recreate"
def test_the_servicemonitor_scrapes_the_ui_port_where_metrics_live():
monitor = _load(APP / "servicemonitor.yaml")
service = _load(APP / "service.yaml")
endpoint = monitor["spec"]["endpoints"][0]
assert endpoint["path"] == "/metrics"
assert endpoint["port"] in {p["name"] for p in service["spec"]["ports"]}
assert monitor["spec"]["selector"]["matchLabels"] == service["metadata"]["labels"]
def test_the_service_targets_the_containers_named_ports(container):
service = _load(APP / "service.yaml")
names = {p["name"] for p in container["ports"]}
for port in service["spec"]["ports"]:
assert port["targetPort"] in names
+1 -1
View File
@@ -62,7 +62,7 @@ def test_load_queries_validates(tmp_path):
def test_the_shipped_example_query_set_parses():
qs = load_queries("eval/queries.example.yaml")
qs = load_queries("queries.example.yaml")
assert len(qs) >= 5
assert {q.category for q in qs} >= {"single", "temporal", "update", "abstain"}
+1 -1
View File
@@ -6,7 +6,7 @@ from mdmem.logging import AuditLog
def test_recent_merges_persisted_history_across_restart_without_duplicates(tmp_path):
"""`AuditLog.recent` must return the merged tail of persisted history plus
in-memory entries across a restart, newest-first, without duplicating the
entries that show up in both (SECURITY_ANALYSIS.md, logging.py:77-92)."""
entries that show up in both (logging.py:77-92)."""
path = tmp_path / "audit.jsonl"
first = AuditLog(path)
for i in range(5):
+2 -2
View File
@@ -1,4 +1,4 @@
"""Regression tests for service-layer write-path and security findings (PLAN.md §2.4, §9, §14.9-14.12)."""
"""Regression tests for service-layer write-path and security findings."""
from __future__ import annotations
@@ -361,7 +361,7 @@ async def test_flag_reason_carrying_a_secret_is_rejected_before_it_is_stored(svc
# -- C-01: the whole read/check/write operation is one serialized step -------
#
# These are the deterministic barrier tests SECURITY_ANALYSIS.md asked for. Every
# These are the deterministic barrier tests for concurrency-critical paths.
# other "race" test in this file is sequential and would still pass if the write
# lock were narrowed back to just the persist call, which is exactly the defect
# C-01 described. The barrier below wedges the first operation *inside* its
+2 -2
View File
@@ -428,9 +428,9 @@ async def test_run_eval_is_auth_and_csrf_protected_like_the_other_health_actions
async def test_run_eval_keeps_its_files_on_the_data_volume_and_reports_status(ui, monkeypatch):
"""The pod runs with readOnlyRootFilesystem, so a relative eval/results path
"""The pod runs with readOnlyRootFilesystem, so a relative results path
would fail at the write with only a log line. Both sides of the run must sit
next to index.db on the mounted volume (§13a.3, §11)."""
next to index.db on the mounted volume."""
client, svc = ui
data_dir = svc.settings.index_db.parent