z1jKdZddlmZddlZej j ddddlmZ e dZ ddl m Z ddlZddlZddlZddlZddlZddlZddlZddlZddlmZmZdd lmZed Zd!d Zd"d Zd#d Zd$dZd%dZd&dZd'dZ d(dZ!d)dZ"d*dZ#d+dZ$d,dZ%d-dZ&d.dZ'd/dZ(d0dZ)d1dZ*d2d3dZ+d4dZ,e-dk(r7e j]dej^e,ej`d dyy)5u_ Aria Knowledge Pipeline (AKP) — Stage 1: Ingest KAR-74 · 2026-05-12 Liest Channel-Liste aus brain/05-Referenzen/youtube-channel-subscriptions.md, holt neue Videos via yt-dlp --flat-playlist (kein YouTube-Data-API-Key noetig), filtert per Cheap-Filter (title-blocklist + duration), pullt Transcripts via youtube_transcript_api (timedtext-Endpoint, ~KB/Video — KAR-742), mit yt-dlp Auto-Subs nur noch als gecappten Fallback, schreibt JSON nach brain/youtube/raw//.json, trackt State in brain/youtube/.state.sqlite. Discovery-Bypass: liest *.url / *.txt aus brain/youtube/discover/. ) annotationsNz/root/aria/lib) get_loggerzaria-akp-ingest)audit)datetimetimezone)Pathz%/root/aria/brain/youtube/.config.yamlctj5}tj|cdddS#1swYyxYwN) CONFIG_PATHopenyaml safe_load)fs %/root/aria/scripts/aria-akp-ingest.py load_configrs1    !q~~a !!!s4=cZtj|}tj|_|Sr )sqlite3connectRow row_factory)db_pathcons rdb_connr"s //' "CkkCO Jcg}tjd}t|5}|D]h}|j|j }|s%|j d}|j dj}|j||dj dddt}g} |D]/} | d|vr |j| d| j| 1| S#1swYKxYw)zJParse channel list from markdown memory file. Returns [{handle/id, slug}].z"^-\s+(@\w[\w.-]*|UC[\w-]{20,24})\b@)identslugNr) recompiler matchstripgrouplstriplowerappendsetadd) memory_pathoutline_rerlinemrrseendedupedcs rparse_channelsr2(sCjj>?G k 7a 7D djjl+AGGAJE<<$**,D JJ5 6  77 5DG  W:   7q  N77s A.CC%cb|jdrd|dS|jdrd|dS|S)Nrzhttps://www.youtube.com/z/videosUCz https://www.youtube.com/channel/) startswith)rs r channel_urlr6>sA )%88 1%@@ Lrc ddddt|ddddg }|r t|jr|d |gz }|r|d |gz }|j| t j |d d d }|jdk7r|jjsgS tj|j}t|tsgS|jdxsg}g}|D]} | r| jddvr| jd} | s-|j| | jdxsd| jd| jdxsd| | jdxs| jdxsd| jdxsdd|S#tj $rgcYSwxYw#tj$rgcYSwxYw)u?Fetch flat-playlist of channel — only metadata, no downloads.yt-dlpz--flat-playlistz-Jz--playlist-endz--ignore-errors --no-warnings--socket-timeout30 --cookies--proxyTxcapture_outputtexttimeoutrentries_type)NurlidtitledurationrEz https://www.youtube.com/watch?v=uploaderchannel upload_date)video_idrGrIrErJrL)strrexistsr' subprocessrunTimeoutExpired returncodestdoutr#jsonloadsJSONDecodeError isinstancedictget) rE max_itemscookiesproxycmdrdatarCr+evids r yt_dlp_flatrcFs  #i.D C4='') W%%  5!!JJsO NN3t$ L ||q!1 zz!((# dD ! hhy!'RG C   AEE'N-7 eeDk  UU7^)rj)55<K%EcU#Kj)CQUU9-=C55/52     J5  $ $     s$FF6F32F36G Gcddl}|jd}gd}|r|d|gz }|r t|jr|d|gz } dg|d d |} t j |d d d }|j jrtj|j nd}dg|d dddddd|d|} t j | d d d } t!t|j#d} | s| j$xsdjj'ddxsdg} t)d|d| dddt*j$d|ft|jD]} | j t|jSt-| dj/d}||ft|jD]} | j t|jS#tjtjf$rd}YwxYw#tj$rnd|fcYt|jD]"} | j#t$rY wxYw t|jS#t$rYSwxYwwxYw#t$rYXwxYw#t$rYSwxYw#t$rY wxYw#t$rYSwxYw#t|jD]"} | j#t$rY wxYw t|jw#t$rYwwxYwxYw)a Pull auto-subs + metadata. Returns (transcript_text, metadata_dict). Two-call pattern: first metadata via --dump-json (lightweight), then sub-only download. Combining --write-sub with -J is unreliable because yt-dlp tries to extract video formats too. rNzakp-)prefix)r9r:60r=r<r8--skip-download --dump-jsonT<r?z--write-auto-subz --sub-langz en.*,en,dez --sub-formatvttz-oz/%(id)s.%(ext)sr>z*.vttrHz [no_transcript] : filereplaceerrors)tempfilemkdtemprrOrPrQrTr#rUrVrRrWiterdirunlinkOSErrorrmdirsortedglobstderr splitlinesprintsys vtt_to_text read_text)rEr\r]rrtmp base_argsmeta_cmdmrmetasub_cmdsrp vtt_fileserrrAs ryt_dlp_transcriptrus   &  )C;I i'' 4='')k7++ $PyP*;P]PCP DRTUB,.IIOO,=4::bii(4D          '    "    E)     4QTUB49>>'23 99?))+668=E"C &se2c!fTcl^<3:: N:c""$ A      IOO 9Q<111CDTzc""$ A      IOO ;))4+?+?@ D (( : c""$ A        IOO    #            c""$ A        IOO    s LAH'LH?BL;KK(!L&K"9K2%H<8L;H<<L?KL4J JJJ00 J<;J<?KL KK KK" K/.K/2 K>=K>M*L0/M*0 L< 9M*;L< <M*MM* M'$M*&M''M*cg}d}|jD]}|j}|r|jdr'd|vstjd|rBtj dd|}tj dd|j}|s||k7s|j ||}dj|S) z=Strip VTT timestamps + cue numbers, dedupe consecutive lines.rH)WEBVTTNOTEzKind:z Language:z-->z^\d+$z<[^>]+>z&\w+;  )r{r#r5r r"subr'join)rj out_lineslastrawr-cleaneds rr~r~sI D~~ yy{t'OP  D=BHHXt4 &&R.&&30668 w$   W %D  99Y rc : ddlm}m}m}m}m}m}m} m} m } ddl m } |r | ||nd}|| } |j!|t#| }dj3d|D}|r|dfSdS#t$r(} td| tjYd} ~ yd} ~ wwxYw#||f$r>} td |d t%| j&tjYd} ~ y d} ~ w|$r t)t+|j-|d}|Yy|j!}nT#||f$rYYy t.$r?} td|d t%| j&tjYd} ~ Yyd} ~ wwxYwY-||| | | f$r>} td|d t%| j&tjYd} ~ yd} ~ wt.$rM} td|d t%| j&d t1| ddtjYd} ~ yd} ~ wwxYw)a]KAR-742: primary transcript path via youtube_transcript_api. Trifft nur den `timedtext`-Endpoint (~KB statt MB/Video via yt-dlp Watch-Page). Proxy als GenericProxyConfig (unser Webshare-Setup nutzt IP-Auth ohne User/Pass, deshalb NICHT WebshareProxyConfig). Returns (text_or_None, status) mit status in: "ok" transcript geholt "none" Video hat nachweislich kein Transcript -> skip, kein yt-dlp-Fallback "blocked" Proxy quota/IP-Block -> Lauf abbrechen (Bandbreite schonen) "error" transienter Fehler -> optionaler, gecappter yt-dlp-Fallback erlaubt r) YouTubeTranscriptApiRequestBlocked IpBlockedTranscriptsDisabledNoTranscriptFoundVideoUnavailableVideoUnplayable AgeRestrictedInvalidVideoId)GenericProxyConfigz [ytapi_import_fail] rmN)Nerror)http_url https_url) proxy_config) languagesz [ytapi_blocked] rl)Nblocked)Nnonez [ytapi_none] z [ytapi_error] r>rc3K|]:}t|ddjs|jj<yw)rArHN)getattrr#rA).0ss r z#ytapi_transcript..s.W628N8T8T8VQVV\\^Ws AAok)youtube_transcript_apirrrrrrrrryoutube_transcript_api.proxiesr ImportErrorr|r}rzfetchtupletype__name__nextiterlist ExceptionrNr)rMr]rrrrrrrrrrrrapcapifetchedfirstrAs rytapi_transcriptrs    F AF Ue <4B B /C))Hi0@)A. 99WWW WDD$<3^3?  &qc*< I & "8*BtAw/?/?.@A S    chhx0148E}#kkmG * #" OH:RQ0@0@/AB T   !1?MSa b zDG,<,<+=>SZZP    "T!W-=-=,>bQOVYV`V`asA2B&2 B#;BB#&H-4C&&H/&D(D('H(E9/H2E9:4E4.H4E99H? H4G H AHHc~gd}|r|d|gz }|r t|jr|d|gz }|j| tj|ddd}|j j rtj|j SdS#tjtjf$rYywxYw)uMetadata-only fetch (--dump-json) fuer Discovery-Items (selten, manuell). Channel-Videos brauchen das NICHT — deren Metadaten kommen aus --flat-playlist. )r8r9r:rfrgrhr=r<Trir?N) rrOr'rPrQrTr#rUrVrRrW)rEr\r]r^r_s r yt_dlp_metars bC  5!!4='') W%%JJsO NN3t$ K'(xx~~'7tzz!((#ATA  % %t';'; <sABB#B<;B<c|jdxsd}|jd}|dd}|r&tj||tjry|||ddkry ||dd kDry y) z#Return skip-reason or None if pass.rGrHrIingesttitle_blocklist_regextitle_blocklistNduration_min_seconds too_shortduration_max_secondstoo_long)rZr search IGNORECASE)videocfgrGrI blocklists r cheap_filterrs~ IIg  $"Eyy$HH 56IRYYy%?  c(m$:; ; c(m$:; ; rcL|jd|f}|jduS)Nz'SELECT 1 FROM ingested WHERE video_id=?)executefetchone)rrMcurs ralready_ingestedrs& ++?( MC <<> %%rc |jd|d||jdd|jd|jdd|f|jy)NzuINSERT OR IGNORE INTO ingested (video_id, channel, title, duration_seconds, url, skipped_reason) VALUES (?,?,?,?,?,?)rMrGrHrIrErrZcommit)rr channel_slugreasons r store_skippedrsVKK z L%))GR*@%))JBWY^YbYbchjlYmouvJJLrc |jd|d||jd||jdd|jd|jdd|jdd|||jd d f |jy) NzINSERT OR REPLACE INTO ingested (video_id, channel, channel_handle, title, duration_seconds, published_at, url, transcript_chars, raw_path, source) VALUES (?,?,?,?,?,?,?,?,?,?)rMrJrGrHrIrLrE_sourcerKr)rrrraw_pathtranscript_charss rstore_ingestedr sKK { *   IIj, / IIgr " IIj ! IImR ( IIeR   IIi + JJLrct|dd|z }|jdd||ddz }|d||jdxs||jdd |jd |jd d |jd d |t|t j t jj|jd d|r|jdnd|r|jdnd|r|jdnd|r|jdnd|r|jdxsgddnddd }|jtj|ddt|S)Npathsraw_dirT)parentsexist_okrMz.jsonrJrGrHrIrErLrrK view_count like_count channel_id categoriestags)rrrrr) rMrchannel_handlerGduration_secondsrErL transcriptr ingested_atsourceyt_metaF)indent ensure_ascii) rmkdirrZlenrnowrutc isoformat write_textrUdumpsrN)rrrrrout_dirout_pathpayloads r write_rawr3sS3w< *+l:G MM$M.E*-.e44H*%$))J/?<7B'!IIj1yy#yy3  O||HLL1;;=))Iy148$((<0d48$((<0d48$((<0d48$((<0d59TXXf%+Sb1t  G(  715IJ x=rc &g}t|dd}|js|Stjd}|j dD]W}|j d}|j |D]/}|j|jddt|d 1Y|j d D]W}|j d}|j |D]/}|j|jddt|d 1Y|S) zBRead discovery folder for *.url and *.txt files with YouTube URLs.r discover_dirz?https?://[^\s'\"<>]+(?:youtube\.com/watch\?v=|youtu\.be/)[\w-]+z*.urlrorprdiscover)rEr _drop_filez*.txt) rrOr r!ryrfinditerr'r$rN)rr+rurl_rerrAr.s rcollect_discoveryrOsCG ^45L     ZZZ [F   w 'Y{{){,& YA JJqwwqzjPSTUPVW X YY  w 'Y{{){,& YA JJqwwqzjPSTUPVW X YY JrcXtjd|}|r|jdSdS)Nz(?:v=|youtu\.be/)([\w-]{6,})r)r rr$)rEr.s rvideo_id_from_urlras( 137A1771:$$rc\|jd}|j|jS)Nz@INSERT INTO run_log (stage, status) VALUES ('ingest', 'running'))rr lastrowid)rrs r log_run_startrfs$ ++X YCJJL ==rcR|jd|||||f|jy)NzUPDATE run_log SET finished_at=datetime('now'), status='completed', items_processed=?, items_skipped=?, items_failed=?, notes=? WHERE id=?)rr)rrun_id processedskippedfailednotess rlog_run_finishrls-KK U GVUF3JJLrc  t}|dd}t|}t|dd}t|}|dj d}t j j dxs|dj dxsd}|dj dxsgd }t|dj d d } d } d } |dd} d|vr|dd} d} d|vrG|jd}|dzt|kr%||dzjjd} | r|Dcgc] }|d| k(s |}}t|}d }d }d }g}|D]1}| rn,t|d}t|| ||}|s|j|dd?|D]}| rHt!||drt#||}|rt%|||d||dz }BCLMG(m$@AOtj/*FGL4jj" 7SY a=..077! QT3?sAw#j/BQ   < ! ( ( *S+Z3 7FDIIk)rErNr\r?r]r?r<ztuple[str | None, dict | None])rjrNr<rN)rMrNr]r?r list[str]r<ztuple[str | None, str])rErNr\r?r]r?r< dict | None)rrYrrYr<r?)rr=rMrNr<bool) rr=rrYrrNrrNr<None) rr=rrYrrNrrNrr"r<rC) rrYrrNrrNrrArrYr<rN)rrYr<r>)rErNr<r?)rr=r<r")rH)rr=r r"r r"r r"r r"rrNr<rC)r(r@r<r")1__doc__ __future__rr}_syspathinsert aria_loggingr _get_logger_log aria_auditr_auditrUr r rrPr$r rrpathlibrr rrr2r6rcrr~rrrrrrrrrrrr9reventexitr(rrrRs # $%2$%&9999':; !  ,,^2j $34l$ & &8$%  } @ zJJ~ CHHT#((12,  r