# aria-eval-suite — Brain-Quality-Benchmarks

KAR-187 (Phase-1 gbrain-Adoption). Inspired by gbrain-evals.

## Status: Foundation

**Done in Phase 1**:
- gbrain-evals cloned to `/tmp/gbrain-evals/` (2.7 MB corpus: amara-life-v1, world-v1, source-swamp-v1, longmemeval, gold)
- Adapter-Plan: Aria-Side wraps `aria-brain-search.py` als gbrain-Replacement
- Aria-eigene Eval-Fixtures skeleton

**Deferred to Phase 2** (großer Run, Cost ~$5-10 für Full-Reproduce):
- Full BrainBench-Reproduce: ingest amara-life-v1 in Aria-Brain → run BrainBench gegen Aria-Search
- 50-100 Aria-spezifische Queries mit Expected-Slugs schreiben

## Verzeichnis-Struktur

```
/root/aria/eval-suite/
├── README.md                           # diese Datei
├── aria-queries.jsonl                  # Aria-spezifische Eval-Queries (TBD)
├── adapters/
│   └── aria-brain-adapter.py          # gbrain-evals → Aria-Brain-Search Bridge (TBD)
└── results/
    └── {YYYY-MM-DD}-brainbench-run.md # Reports
```

## Sandbox-Verify Plan (Foundation)

### Phase 2.1 — Adapter bauen
Schreibe `aria-brain-adapter.py` der gbrain-evals' Query-Interface erfüllt:

```python
# gbrain-evals erwartet:
brain.query(text, top_k=5) -> [{"slug": ..., "score": ...}]

# Aria-Mapping:
def query(text, top_k=5):
    # Wraps aria-brain-search.py mit --json
    result = subprocess.run(["python3", "aria-brain-search.py", text, 
                            "--top", str(top_k), "--json"], capture_output=True)
    return parse_search_results(result.stdout)
```

### Phase 2.2 — Corpus migrieren
Aria-Brain hat 192 eigene Notes. gbrain-evals' amara-life-v1 ist 4 MB fiktive
Personen-Daten. Optionen:

a) **Direkt-Reproduce** (cleaner, vergleichbar mit gbrain):
   - Ingest amara-life-v1 in temporäre Brain-Section
   - Run BrainBench
   - Erwartung: gbrain P@5 49,1% reproducible bei intakter Implementation
   - Cost: ~$2-5 für Embeddings

b) **Aria-Specific** (relevanter, eigene Domain):
   - 50 Aria-Queries gegen 192 Aria-Notes
   - "Wo steht X über BMW Zipse?" → Expected: 02-Wissen/bmw-allgemein/01_Index.md
   - Manuelles Labeling
   - Cost: ~$0.10

### Phase 2.3 — Eval-Run
```bash
cd /tmp/gbrain-evals
# Adapter konfigurieren
EVAL_BRAIN_ADAPTER=/root/aria/eval-suite/adapters/aria-brain-adapter.py
bun run eval/runner/all.ts --corpus amara-life-v1
```

### Phase 2.4 — Report
- Aria P@5/R@5 vs gbrain Baseline
- Wenn ±5pp: Implementation OK
- Wenn 10pp+ schlechter: Investigieren (RRF-Weights, Chunk-Size, Frontmatter-Boost)

## Aria-Specific Test-Queries (Beispiel-Skeleton)

Datei `aria-queries.jsonl` (Format: 1 JSON pro Zeile):

```jsonl
{"query": "Wer ist Oliver Zipse?", "expected_slugs": ["02_Oliver_Zipse_TUM_Vorlesungen_Kernaussagen", "01_BMW_Allgemein_Index"], "category": "person-lookup"}
{"query": "BMW CO2-Ziele 2030", "expected_slugs": ["07_BMW_Nachhaltigkeit_CO2_Kreislaufwirtschaft", "04_BMW_Zahlen_Daten_Fakten_Jahresvergleich"], "category": "factual"}
{"query": "Aria Identity Layer", "expected_slugs": ["IDENTITY", "SOUL", "USER"], "category": "self-knowledge"}
```

Initial 10-Query-Set für Phase 2.1 reicht für ersten Run.

## Cost-Budget

- Sandbox-Verify Phase 2: ~$3-7 einmalig (Embedding amara-life-v1)
- Reguläre Quartals-Runs: ~$0.50 pro Run gegen Aria-Specific-Set

## Verwandte
- gbrain-evals: https://github.com/garrytan/gbrain-evals
- KAR-187 in Linear
- Phase-1 Plan-Note: 01-Projekte/aria-phase1-gbrain-adoption/