6NjW.8dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl m Z e dZ ejjddj!dZejjd dZejjd dZd Zd Zd ZdZesdD]Zej2j5er}e ej7j9D]VZej=dsej?dddjAjAdjAdZnesneredndZ!d"dZ"eefd#dZ#d$d%dZ$d&dZ%e fd'dZ&d(dZ'd)dZ(d)dZ)d)dZ*d)dZ+d)d Z,e-d!k(rej\e,yy)*u{aria-brain-embed — One-shot + incremental Embedding für brain_chunks (KAR-120). Schreibt Brain-Files in chunks → embedded sie via OpenAI text-embedding-3-large (3072 dim) → upsertet sie in `brain_chunks` (Supabase aria-DB pgvector). Embedding-Modell-Wahl (Sektion 12.3 der Gibson-Analyse): - text-embedding-3-large: 3072 dim, ~$0.13 per 1M Token, general-purpose - One-shot full-brain (200 Files, 30 MB): geschätzt ~$0.13 Voraussetzungen: - OPENAI_API_KEY in env oder /root/.aria-secrets/openai.env - pgvector + brain_chunks Table vorhanden (siehe /tmp/aria-pgvector-schema.sql, KAR-120-Setup) CLI: python3 aria-brain-embed.py status # zeige welche Files embedded sind python3 aria-brain-embed.py backfill # full re-embed alles python3 aria-brain-embed.py incremental # nur neu/geändert (default) python3 aria-brain-embed.py file # einzelnes File ) annotationsN)Pathz/root/aria/brain SUPABASE_URL/SUPABASE_SERVICE_ROLE_KEYOPENAI_API_KEYz$https://api.openai.com/v1/embeddingsztext-embedding-3-largei)z/root/.aria-secrets/openai.envz/root/aria/.envzOPENAI_API_KEY=="'z5/rest/v1/brain_chunks?on_conflict=file_path,chunk_idxc&tdtdddS)NBearer application/jsonzresolution=merge-duplicates)apikey Authorization Content-TypePrefer)r&/root/aria/scripts/aria-brain-embed.py_supabase_headersr<s!+"#<"=>*/  rc|j}t||kr|r|gSgSg}d}|t|krt||zt|}|t|kr |jd||}|||dzzkDr|}|j |||j|t|k\rn!t ||z |dz}|t|kr|Dcgc]}|s| c}Scc}w)zZSimple character-window chunking. Brain-Files sind meist <30KB, 1-15 chunks pro Datei.r r )striplenminrfindappendmax)texttargetoverlapchunksposendnlcs r chunk_textr+Es :: ?  4  jj;?@GGIG EH1% ..(( w'.~.>%?$6! )C''R'8 ,Azz!&&(+ ,26v,?$D%? ?(  Cl#BCC , ,?||%% Hxx37W#4   >'A+ayUSVKWZ![\ 5!EAIr*  sD AE2$EE% E1EE EG'4A'G"!G""G'cts td||||d}tjj tt j |jitddid}tjj|d5}d |jcxkxrd knccd d d S#1swYy xYw) uLUpsert via Supabase REST. Unique(file_path, chunk_idx) — resolution=merge.zERROR: SUPABASE_URL missing) file_path chunk_idxcontentr8rz*resolution=merge-duplicates,return=minimalr0r1r9r6r i,N) REST_BRAIN_CHUNKSr=rBrCrDr>r?rArrFstatus)r_r`rar8rowrZr[s r upsert_chunkres 677  C ..  3 6 6 8_$&_2^_ ! C   R  0%Aahh$$%%%s B44B=cg}|jdD]C}td|jtjDr3|j |Et |S)z:Liste alle .md-Files unter BRAIN_ROOT, ohne .git/.archive.z*.mdc3pK|].}|jdxs|dvxs|jd0yw).) __pycache__Archivz.archiveN) startswithendswith).0parts r zbrain_files..s=Tjnts#et/H'HeDMMZdLeeTs46)rglobany relative_to BRAIN_ROOTpartsr!sorted) only_underrTmds r brain_filesrxs} Cv& TrtsAsABLsMsSsST T  2 #;rc &tsiSi}d\}} td}tjj |it d|d||zdz dd} tjj |d 5}tj|j}d d d D] }|||d< t||kr|S||z }#1swY.xYw#t$r2}tjjd |d |d|cYd }~Sd }~wwxYw)udWas ist in brain_chunks_files (welche Files schon embedded). Paginiert (KAR-876, 08.07.2026): PostgREST/Supabase kappt Antworten bei max-rows=1000 — limit=10000 half nicht. Ohne Pagination sah der Incremental-Lauf nur 1000 von ~2960 registrierten Files und hat ~1900 Files faelschlich als 'missing' re-embedded (Kosten + inflierte Logs).)riz8/rest/v1/brain_chunks_files?select=*&order=file_path.ascitems-r )z Range-UnitRangeGET)r3r4r6Nz&[embed] status query failed at offset : rr_)rbrrBrCrDrrFr>rGrH ExceptionrLrMrNr) rToffsetpageurlrZr[rowsr]rds rdb_files_statusrs6  CLFD VWnn$$ ?(*?'!'&4-!*;)<=? %   ''R'8 ,Azz!&&(+ ,  (C$'CK ! ( t9t J$%  , ,  JJ  EfXRPSuTVW XJ s6!C8$C C CC D'D D Dcht}t}tdt|dtdt|dtdt|t|z d|jr>|D]9}t |j t}||vrdnd}td |d |;y ) Nz Brain-Files: z totalz DB-Indexed: z filesz Missing: z files not embedded yetu✓u—  r)rxrprintrverbosestrrrrs)argsfilesdb_idxfrelrcs r cmd_statusrs ME  F M#e*V ,- M#f+f -. M#e*s6{233J KL || 'AammJ/0C!VmUF Bvhau% & ' rcht|j}|js t|z }|j s t d|dt jyt|jt}|jd}t|}|st |dyt |d t|d t jt|}tt||D];\}\}}t!||||} t d |d | rd nd t j=y)NzERROR: z not foundfiler replaceerrorsz: empty, skippedrr chunksz chunk OKFAIL)rpath is_absolutersexistsrrLrMrrr read_textr+rr< enumeratezipre) rrrrar& embeddingsrUr#emboks rcmd_filers  TYYA ==? N 88: s*%CJJ7 ammJ' (Ckkk+G  F  %&' SECK= (szz:V$J#C $;<I;D# #q$ , 2bdf56SZZHI rc t}tdt|dtdtj d}d}|D]}t |jt}|jd}t|}|sA t|}tt||D]\} \} } t|| | | |t|z }t!d |Ddz } || dzdz z }td |dt|dtj t#j$dtd|dt|d|dtj y#t$r,} td |d | tj Yd} ~ Kd} ~ wwxYw#t$r/} td |d | d | tj Yd} ~ $d} ~ wwxYw)Nz Embedding z brain-files via z...rrgrrr EMBED-FAIL: #z UPSERT-FAIL: c32K|]}t|yw)N)r)rmr*s rrozcmd_backfill..s3qCF3sgp= ף?i@Brr皙?z Done. z chunks embedded across z files. Estimated cost: $z.4f)rxrrr@rLrMrrrrsrr+r<rrrresumrOrP)rr total_chunkstotal_cost_estrrrar&rr]rUr#r approx_tokenss r cmd_backfillrs ME Js5zl"3K= D3::VLN !-- +,++Y+/G$  $V,J(FJ(?@ JNA{c JS!T3/ J F # 3F33a7 -$.:: 3%r#f+g.SZZ@ 3+, H\N":3u:,G,S1 39<E !  Bse=.SZZ @    J3%q>#7cjjII Js0 E,F F$!F  F G $GG c t}t}|Dcgc]%}t|jt|vs$|'}}t dt |dt |dtj|sy|D]}t|jt}|jd}t|}|sAt d|d t |d tj t|}d} tt||D]\} \} } t|| | | t d|dt || z dt |dtjt!j"dycc}w#t$r,} t d|d | tjYd} ~ 0d} ~ wwxYw#t$r3} | d z } t d|d | d| tjYd} ~ d} ~ wwxYw)Nz Incremental: z new/missing files (of z total)rrrrrz : embedding z chunks ...rr [z] UPSERT-FAIL: rrz chunks okr)rxrrrrrsrrrLrMrr+r<rrrrerOrP)rrrrpendingrrar&rr]failedrUr#rs rcmd_incrementalrs ME  FPQ3q}}Z'@#A#OqPGP M#g,'>s5zl' RY\YcYcd  !-- +,++Y+/G$  3%|CK= <3::N $V,J'FJ(?@ KNA{c KS!T3/ K 3%r#f+./qV ZHszzZ 3)* 3Q  Bse=.SZZ @   K! 3%q?3%8szzJJ Ks:%E9E9+ E>F6> F3!F..F36 G2?)G--G2c,tjd}|jdd}|jdd}|j d d d |j t |jdd}|j t |jdd}|j t |jdd}|j dd|j t |j}|j|S)NzAria Brain Embedding (KAR-120)) descriptioncmdT)destrequiredrczshow indexed vs missing files)helpz-vz --verbose store_true)action)funcbackfillzfull re-embed all brain files incrementalzembed only new/changed filesrzembed a single filerzabsolute or BRAIN_ROOT-relative) argparseArgumentParseradd_subparsers add_parser add_argument set_defaultsrrrr parse_argsr)parsersubp_stp_bfp_incp_frs rmainr%s  $ $1Q RF   UT  :C >>()H> IDdK =:& >>*+J> KD<( NN=/MN NE O, ..&;. osrLrOurllib.requestrBpathlibrrsenvirongetrstriprrr rEr@CHUNK_TARGET_CHARS CHUNK_OVERLAPenv_filerrr splitlineslinerksplitrrbrr+r<rerxrrrrrr__name__exitrrrrs&# $ % zz~~nb188= JJNN+FK 0"59&  I 77>>( #X002==? ??#45%)ZZQ%7%:%@%@%B%H%H%M%S%STW%XN   nIJ);=$,&DR%&$.D   * @ >, z CHHTVr