CARC Knowledge Store

An AI-readable knowledge store for CARC (HPC center) — the grounding layer for CARC's AI support assistant over Zammad tickets and Coldfront workflows. This repo holds the knowledge base content, a working ingestion and hybrid-search pipeline, a read-only MCP server exposing search over it, the MCP gateway that fronts that search (and Zammad/Grafana) per agent identity, and a read-only Zammad ticket fetch/draft workflow for content conversion. It has no write access to Zammad or Coldfront and no Coldfront integration — see Roadmap.

Companion repo — carc-agents (git.repo.alliance.unm.edu/CARC/carc-agents, also at ~/carc-agents on the serving host) owns everything this one doesn't: local model serving (vLLM behind a LiteLLM gateway on a two-DGX-Spark pooled fabric), the network boundaries around it, the NemoClaw agent roster, the agent/triage.py classify→retrieve→draft→verify pipeline, and the evals. Its ARCHITECTURE.md is the system-wide design rationale; this repo is a shareable CARC asset with CI, that one is host-specific infrastructure. The two share exactly one contract: carc-agents' agents call this repo's carc-gateway (identity → policy → tools) and its gateway/agents.yaml is the authorization policy both sides read.

Architecture at a glance

kb/*.md (Markdown + YAML frontmatter)
        │  kb validate  — lint frontmatter, flag stale/empty content
        ▼
kbstore/parser.py        — split frontmatter/body, chunk by ## heading
        │  kb ingest
        ▼
kbstore/embedders.py     — swappable embedding backend (hash / voyage / local / vllm)
        ▼
kb.index.sqlite          — FTS5 (BM25 keyword) + stored vectors, per-article content hash
        │  kb search
        ▼
kbstore/search.py        — hybrid search (BM25 + cosine, reciprocal rank fusion)
                            filtered by system / category / audience / status

Why SQLite instead of a vector database: a KB of a few hundred articles doesn't need one. SQLite is stdlib, needs no daemon, gives BM25 keyword search via FTS5 for free, and brute-force cosine similarity over numpy on a few thousand chunks runs in milliseconds. Revisit only if the KB grows 100×.

Access boundary: audience (users / staff / admins) is enforced in the SQL query via --max-audience, not filtered after the fact — a future user-facing ticket agent running with --max-audience users cannot retrieve staff or admin content into a user-visible reply. status works the same way: only approved articles are retrievable by default, so unreviewed content can't leak into agent responses.

Repo layout

kb/                 the knowledge base itself (see kb/templates/ and CONTRIBUTING.md)
kb/facts/           provenance-tagged cluster fact sheets (see Ground-truth facts below)
kb/procedures/      staff procedures converted from raw notes (see Procedure notes below)
kbstore/            the Python package: schema, parser, embedders, index, search, CLI
scripts/            fetch_zammad_tickets.py, collect_cluster_facts.sh, import_carc_docs.py, embedder smoke test
kb/docs/            GENERATED mirror of carc.unm.edu/docs (see Published documentation below); never hand-edit
facts/raw/          raw command output backing kb/facts/*.md, one dated dir per collection run
notes/procedures/   raw process notes backing kb/procedures/*.md (see Procedure notes below)
tests/              pytest suite for kbstore
gateway/            per-identity authorization policy (agents.yaml) for the MCP gateway;
                    the gateway itself is kbstore/gateway.py (console script carc-gateway),
                    production-verified against real Zammad/Grafana — see gateway/README.md

Quickstart

python3 -m venv .venv
.venv/bin/pip install -e ".[dev]"

.venv/bin/kb validate                              # lint all articles
.venv/bin/kb ingest                                # parse, chunk, embed, index
.venv/bin/kb search "cannot submit jobs" --include-drafts
.venv/bin/kb status                                # index stats

Run tests with .venv/bin/pytest.

CLI reference

• kb validate [--kb-root kb] — lint every article's frontmatter and body. Exits nonzero on schema errors (missing fields, bad enum values, missing escalate_if); warns (exit 0) on stale last_reviewed and empty sections. Suitable as a CI/pre-commit gate.
• kb ingest [--kb-root kb] [--db kb.index.sqlite] [--embedder hash|voyage|local|vllm] [--full] — parses, chunks, embeds, and upserts articles into the SQLite index. Incremental by default: articles whose content hash hasn't changed are skipped. Switching --embedder forces a full reindex, since a search can't mix vectors from two different embedding spaces. Articles deleted from disk are removed from the index.
• kb search "<query>" [-k 5] [--db kb.index.sqlite] [--embedder ...] [--system slurm] [--category runbook] [--max-audience users|staff|admins] [--include-drafts] [--json] — hybrid search: FTS5 BM25 plus cosine similarity, merged by reciprocal rank fusion. Every hit carries a citation (file path + heading). Defaults to approved-only, uncapped audience. --json output is the data contract the MCP server below wraps.
• kb status [--db kb.index.sqlite] — article/chunk counts by category and status, embedder in use, last ingest time, and count of articles overdue for review (last_reviewed >12 months old).

MCP server

kb-mcp-server (install with pip install -e ".[mcp]") exposes the KB over the Model Context Protocol as two read-only tools: kb_search and kb_status. It's a thin wrapper around the same search/index code the CLI uses, run over stdio for an agent client to call.

Configuration is via environment variables, read once at server startup — not as tool parameters:

• KB_DB_PATH (default kb.index.sqlite)
• KB_EMBEDDER (default: whatever kb ingest used)
• KB_MAX_AUDIENCE — users / staff / admins, or unset for uncapped
• KB_INCLUDE_DRAFTS — true to include drafts, default false

This split is deliberate: kb_search's tool schema only exposes query, k, system, and category — there is no argument an agent can pass to widen its own audience or draft access. A user-facing support agent gets a server instance launched with KB_MAX_AUDIENCE=users; the operator sets that once at deploy time, and no prompt or tool call from the agent side can override it. kb validate/kb ingest are intentionally not exposed as MCP tools — this server is read-only.

KB_DB_PATH=kb.index.sqlite KB_MAX_AUDIENCE=users kb-mcp-server

MCP gateway

kb-mcp-server above is the KB alone over stdio. carc-gateway (kbstore/gateway.py, console script carc-gateway) is the multi-backend front door: it reads MCP_AGENT_IDENTITY at launch, resolves that identity's policy from gateway/agents.yaml, serves KB search/status in-process, spawns every other enabled backend (mcp-zammad, mcp-grafana) as a stdio subprocess with per-identity credentials resolved from gateway/credentials.yaml, and aggregates their tools behind one MCP server — logging every call, allowed or denied, to gateway_audit.sqlite. Identity is fixed at process launch and nothing the agent does at runtime can widen it, the same property kb-mcp-server's KB_MAX_AUDIENCE already has.

support, ops, and engineering are verified end-to-end against production Zammad and Grafana with dedicated service-account tokens; kb-researcher and ticket-triage are KB-only, zero-credential identities used by the NemoClaw sandbox. Full design and the per-identity policy table are in gateway/README.md.

Embedding backend

Select with --embedder or the KB_EMBEDDER env var. CARC's serving stack uses vllm (carc-embed, Qwen3-Embedding-0.6B, via the LiteLLM gateway on carc-agents' Spark fabric); hash stays the default so the pipeline runs offline, in CI, and in tests with no model or credentials. The GPU embedder's edge over the hash baseline is corpus-size dependent — negligible on a toy corpus, +11.6 points recall@5 at ~220 articles (carc-agents FINDINGS).

• hash (default) — deterministic hashing-trick embedder, zero dependencies, zero credentials, zero model downloads. Retrieval quality is intentionally poor on the vector side; FTS5 keyword search carries relevance. This is what makes the whole pipeline runnable offline, in CI, and in tests.
• vllm — embeddings from a local OpenAI-compatible endpoint (vLLM behind LiteLLM). Reads KB_VLLM_BASE_URL, KB_VLLM_API_KEY, KB_VLLM_MODEL (default carc-embed), optional KB_VLLM_DIM (Matryoshka truncation). Needs requests (any of the voyage/zammad extras pull it in). No KB text leaves the host. .env.carc sets this up on the serving host.
• voyage — hosted embeddings via the Voyage AI API. Requires VOYAGE_API_KEY; install with pip install -e ".[voyage]".
• local — local sentence-transformers model, no network calls at query time. Install with pip install -e ".[local]".

To compare local against the hash baseline without installing torch/sentence-transformers on your own machine, scripts/local_embedder_smoke_test.py ingests kb/ with a chosen embedder and checks a handful of known queries against expected articles. scripts/slurm_local_embedder_smoke.sbatch runs it as a Slurm job — edit its --account/--partition and submit with sbatch. Note that sentence-transformers downloads model weights on first use, so this needs a node with outbound internet (see the script's comments).

Content: placeholder vs. real

Phase 1 shipped with 11 fully placeholder articles (one per category, spanning support and admin audiences) to exercise the pipeline and model the writing style, since CARC's existing docs were sparse, scattered, and out of date. Those remain status: draft — they were never meant to ship as real guidance, just to prove the pipeline end-to-end.

Content conversion from real, solved Zammad tickets follows the ticket-to-article workflow in CONTRIBUTING.md. The first batch of 11 real tickets has gone through a full staff review pass against their source tickets: 10 were approved (status: approved) after fact-checking, one correction pass, and in a couple of cases real edits where the draft had gotten ahead of what the ticket actually supported; one was dropped entirely for asserting a causal mechanism the ticket never confirmed. Converting more of the real backlog beyond this first batch is ongoing — see Roadmap.

scripts/fetch_zammad_tickets.py pulls closed tickets from a Zammad group (read-only — GET requests only) for review as article candidates, and scripts/draft_kb_articles_from_tickets.py runs the whole ticket-to-article workflow end to end (fetch → drop noise → dedup-flag → local-LLM draft → kb validate → write status: draft). Both read ZAMMAD_URL / ZAMMAD_TOKEN:

pip install -e ".[zammad]"
python scripts/fetch_zammad_tickets.py --group "HPC Support" --limit 20

Zammad access, in order of preference:

1. The read-only egress proxy — on the serving host, carc-agents' bin/carc-zammad-proxy (127.0.0.1:8010) holds a scoped service-account token and refuses every mutating HTTP method and every non-allowlisted read path at the network layer, outside the agent. Point the scripts at it with ZAMMAD_URL=http://127.0.0.1:8010 and ZAMMAD_TOKEN set to the proxy's own inbound key (carc-agents/keys/zammad-proxy.key; the proxy accepts it in the Token token= header the scripts already send, then injects the real Zammad token upstream). A missing or wrong key is a 401. This is the enforceable boundary and the preferred path.
2. A fresh personal token, scoped to knowledge_base.reader + ticket.agent. The personal admin-capable PAT this repo used to carry in a gitignored .env was retired 2026-08-25 when the gateway's dedicated zammad_support_token / zammad_ops_token service accounts came online; self-service a new one via Keycloak if the ad hoc review workflow is needed directly.

See CONTRIBUTING.md for the workflow these feed into.

Published documentation (kb/docs/)

CARC's public documentation at carc.unm.edu/docs is an OKF v0.2 bundle published for agents as well as people (its agent guide and robots.txt name the endpoints). scripts/import_carc_docs.py consumes the docs-bundle corpus file, https://carc.unm.edu/docs/llms-full.txt, and writes one generated article per page under kb/docs/<section>/<slug>.md with evidence: published_doc:

.venv/bin/python scripts/import_carc_docs.py            # fetch, convert, write, manifest
.venv/bin/python scripts/import_carc_docs.py --dry-run  # report only
.venv/bin/kb validate && (set -a; source .env.carc; set +a; .venv/bin/kb ingest --embedder vllm)

Publication is the review. Pages import as approved; attribution comes from the OKF contract: a page carrying verified: {by: "human:<netid>", at} records that reviewer and date, any other page records the publication and corpus date. OKF status: deprecated imports as deprecated (out of default retrieval, tagged legacy), status: draft as draft. Relative links are rewritten to canonical page URLs so a drafted reply can cite them. Every file starts with a GENERATED marker: never edit kb/docs/ by hand -- fix the page in github.com/UNM-CARC/docs, let it publish, re-run the import. Pages that leave the corpus are removed here; kb/docs/IMPORT.json records the corpus, fetch time, and per-page hashes, and the git diff of kb/docs/ is the correction log (CONTRIBUTING, "Correcting the KB").

Ground-truth facts

kb/facts/easley.md and kb/facts/hopper.md are provenance-tagged reference sheets built by directly observing each cluster over SSH, not by inferring from ticket history. Every fact is tagged observed (captured command output), attested (stated by staff, not independently verified), or inferred (generalized from limited evidence) — this exists because some facts inferred from tickets turned out to be wrong or incomplete (e.g. an early draft of storage-tiers-and-quotas.md had several open questions about whether a given quota number was cluster-specific or universal).

scripts/collect_cluster_facts.sh easley   # or hopper

Read-only by construction (same discipline as the Zammad fetch script) — runs an inspection-only command battery (sinfo, scontrol, quotas, df, etc.) over SSH and writes each command's output to facts/raw/<host>/<date>/, one file per command plus a manifest. Verified idempotent: rerunning against the same cluster only changes timestamp/load-dependent lines. Even a command's own displayed banner text isn't automatically trustworthy just because it's "observed" rather than "attested" — one such banner was caught displaying a stale cleanup window and had to be corrected at the source, see git history on kb/facts/easley.md for that example. Scope any account/quota lookups to the observing user (e.g. --uid $USER, not --all) before adding a new command to the battery — an early version of this script briefly captured ~550 other users' storage usage cluster-wide before that was caught and fixed.

Roadmap

Done since the initial phase: the MCP gateway (built, production-verified); the vllm embedder on CARC's Spark fabric; read-only Zammad and Grafana MCP access per identity via Zammad-MCP and mcp-grafana, spawned by the gateway (no custom clients); the scripted ticket-to-article and procedure-from-notes drafting workflows.

Still open, in this repo:

• Nightly cron ingestion
• A Coldfront MCP server (no Coldfront integration exists yet)
• Zammad write actions (the gateway's support identity has a Zammad permission group scoped for ticket read + reply-note, but nothing in this repo posts to Zammad)
• Converting the rest of the real backlog of solved tickets and wiki content into KB articles
• Closing the "Needs staff confirmation" items in kb/facts/*.md (e.g. per-partition memory defaults beyond each cluster's general partition, backup policy specifics, Hopper's condo-partition access model)

Agent orchestration, the Zammad poll trigger, escalation mechanics, and the approval-gate audit trail live in the carc-agents repo — see its ARCHITECTURE.md ("Target: the ticket triage loop") and FINDINGS.md.