*pj,i%SrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r SSK J r Jr SSKJrJr \R"S5\R"S54r\ R&R)S5r\R,"S5R.r\R,"S 5R.r\"S \"S 555rS rS r0r\ \!\!4\"S'\RF"5H7r$\%"\$RL5S:XdM\$RLS\\$RN'M9 S2S\S\(S\)\!S\(S\ \!\44 Sjjr*S\S\+\ \!\4\ \!\,444Sjr-\R\S\+\ \!\4\ \!\,444Sj5r/S\ \!\44Sjr0S\ \!\4S\ \!\)\+\!S-\\!444Sjr1\R\S\ \!\)\+\!S-\\!444Sj5r2S\!S\!S-4Sjr3S\!S\44Sjr5S\ \!\,44Sjr6\R\S\ \!\44S j5r7\R\S\4S!j5r8"S"S#5r9S3S$\9S%S&S'\!S\,4S(jjr:\;"1S)k5r<\;\!\"S*'S+r=S,r>S-r?S4S.S/.S\S\!S$\9S-S0\4S\+\,\!S-44 S1jjjr@g)5zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)_kernel) dot_packed pack_profile)REGISTRYlookup_encodingi)z.soz.pydz>Iz>dc#4# UHoS;aSOSv M g7f)) rN).0bs S/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/chardet/models/__init__.py r9s ISA8 8Aa?ssCMD2sCRM1_SINGLE_LANG_MAPrdataoffsetnames chunk_sizereturnc[U5nUS-n[R"5n0n[5nSn Un [U5n Sn [U5U:aS[U5- n UR(aUR URU 5nONX:a&X X-nU [U5- n UR X5nO#U (dSn UR 5nU(dOnOOlU [U5- n X- n[U5S:Xa*[U5Xr[U5'UR5 O[U5S:aO[U5U:aM[U5U:XaSnUR(aUR URS5nU(d[UR(dJX:aEX X-nU [U5- n UR US5nU(dUR(dX:aMEU(d(UR(dU (dUR 5nU [U5- n X:wd[U5U:waSU SU3n[U5eU$) uDecompress the model tables from ``data[offset:]``, one per name. Each model is stored as its own bytes object rather than a memoryview slice of one big blob: mypyc compiles bytes indexing in the scoring hot loop to a native C array access, while memoryview indexing goes through a boxed generic call. Decompression is incremental, one 64 KB table at a time — materializing the whole multi-megabyte blob and slicing it would transiently double the allocation and strand the freed pages in process RSS. Trailing compressed bytes are ignored, as with whole-blob ``zlib.decompress``; the decompressed size is validated instead. :raises ValueError: If the decompressed size is not exactly ``len(names) * 65536``. rFTrz&corrupt models.bin: decompressed size z != expected ) lenzlib decompressobj bytearrayunconsumed_tail decompressflushbytescleareof ValueError)rrrr num_models expected_sizedecompmodelstableproducedposendflushedneedpiecechunkextramsgs r_decompress_tablesr:Is"UJ&M    !F!F KEH C d)CG f+ "s5z!  ! !%%f&<& >| LC >> D   '   2v T ""r c[5S$)zLoad all bigram models from the bundled models.bin file. Each model is a bytes object of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. rr^rr r load_modelsras  q !!r r/c0nUR5H<up#URSS5upEURU/5RXCU45 M> [ U5H n[ U5nUcMXq;dMXX'M" U$)zBuild a grouped index from a models dict. :param models: Mapping of ``"lang/encoding"`` keys to 65536-byte tables. :returns: Mapping of encoding name to ``[(lang, model, model_key), ...]``. /r)itemssplit setdefaultrClistr)r/indexkeymodellangencenc_name canonicals r_build_enc_indexros=?Elln IIc1%  b!(($s);<% K#H-  Y%;$E  Lr c([[55$)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].)rorarr r get_enc_indexrqs KM **r encodingc,[RU5$)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )rgetrrs rinfer_languagervs    ))r cU[5;$)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rqrus rhas_model_variantsrx s } &&r c[5S$)zAReturn cached L2 norms for all models, keyed by model key string.rr`rr r_get_model_normsrzs  q !!r c ^[5n[RRS5nUR S5R 5n[ R"UR S5R 55R5nSnUSS[:XaWUSUU:XaN[U5U[U5S--:Xa0[U5VVs0sHupVXbXES--XES -S--_M snn$U(a[R"S [ S S 9 UR#5VV^s0sH'unmU[%U4S j['SSS555_M) snn$![[4a SnSnNf=fs snnfs snnf)uReturn per-model row-maximum tables for upper-bound prescreening. For each model, entry ``b1`` of its 256-byte table holds the maximum weight in the model's row for lead byte ``b1``. Because every bigram weight is bounded by its row maximum, a dot product against the row maxima (256 terms) upper-bounds the dot product against the full table (65536 terms) — statistical scoring uses this to rule out candidate models without scoring them fully. Loads the precomputed ``rowmax.bin`` (written by ``scripts/train.py`` in the same model order as ``models.bin``). The file starts with a ``CRM1`` magic and the SHA-256 of the ``models.bin`` it was derived from: a stale or mismatched file would silently under-estimate row maxima and break the upper bound that pruning depends on, so anything that does not match the *current* ``models.bin`` byte-for-byte is rejected and the tables are derived from the models directly (slower, but always correct). rOz rowmax.binrPr $Nr<rrzpchardet rowmax.bin is missing or does not match models.bin; deriving row maxima from the models (slower startup)rQrRc3D># UHn[TXS-5v M g7f)rN)max)rstartr0s rrget_rowmax..Gs$U@Tu3uUS[122@Ts rr)rarTrUrVrWrXhashlibsha256digestFileNotFoundErrorOSError _ROWMAX_MAGICr! enumeraterYrZr[rdr(r@)r/rVr models_digest header_sizerrir0s `r get_rowmaxrs{&]F    % %&6 7E~~l+668 NN< ( 3 3 5 &(  K RaM! ; = 0 Is6{S'88 8 $F+ + kG+kUcM.IJ J+    C   !,,. (JC UUaPS@TU UU( + w '   sA EE3'.E9E0/E0c[RRS5RS5nUR 5n[ U5S:wa,[ R"S[ U5S3[SS9 SS-$U$) uReturn a 65536-byte IDF weight table for bigram profile construction. Loads a precomputed table from ``idf.bin`` (generated at training time). For each bigram index, the weight reflects how discriminative that bigram is across all models: - Bigrams in every model (common ASCII) → weight 1 (minimal signal) - Bigrams in one model → weight 255 (maximum signal) - Bigrams not in any model → weight 1 (unknown, treat as neutral) rOzidf.binrz chardet idf.bin has wrong size (z"), falling back to uniform weightsrQrR) rTrUrVrWrXr!rYrZr[r\s rget_idf_weightsrLsv    # #$4 5 > >y IC >> D 4yE .s4yk:. .    Kr c \rSrSrSrSrS\SS4SjrS\\ S \\ S \\ S \ SS4 S jr \ S \ \ \ 4SS4Sj5r Srg) BigramProfileieuPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Each bigram is weighted by its IDF (inverse document frequency) across all models — bigrams unique to few models get high weight, bigrams common to all models get weight 1. ``nonzero`` lists the indices carrying weight, in first-encounter order. The weights are reachable two ways, and which one a profile fills depends on how it was built: * ``idx_arr``/``val_arr`` — parallel ``array('i')`` buffers, what :func:`score_with_profile` reads for a streaming profile. Filled by the streaming constructor only. * ``values`` — a plain list parallel to ``nonzero``, filled by :meth:`from_weighted_freq` for the small focused profiles confusion resolution builds, which are scored inline instead. ``row_freq`` aggregates the weights by lead byte (256 entries) and ``nonzero_rows`` lists the lead bytes with non-zero total; together with per-model row maxima (:func:`get_rowmax`) they let statistical scoring compute a cheap upper bound on a model's score. **Input limit.** Weights are packed as int32, which holds any value an input of at most 16 MB can produce (``255`` per occurrence against a 2.1-billion ceiling). Detection truncates to ``max_bytes`` long before that; construct a profile directly from a larger buffer and packing raises :exc:`OverflowError`. ) freqidx_arr input_normnonzero nonzero_rowsrow_freqval_arrvalues weight_sumrrNc[U5S- nUS::aE/Ul/Ul/Ul[uUlUl/Ul/UlSUl SUl g[5nS/S-n/nSn[U5HXnXnXS-n X:Xa[U(aM#US-U -n X:n XJS:XaURU 5 XJ==U - ss'Xk- nMZ URXE/U5 g)aCompute the bigram frequency distribution for *data*. Each bigram is weighted by its IDF (inverse document frequency) across all loaded models. Bigrams unique to few models get high weight; bigrams common to all models get weight 1. :param data: The raw byte data to profile. rrNrr=)r!rrr _EMPTY_PACKEDrrrrrrrr@_ASCII_WHITESPACE_TABLErC_finish) selfr total_bigramsidfrrw_sumrb1b2idxws r__init__BigramProfile.__init__sD A A #%DI&(DL%'DK)6 &DL$,')DM+-D #$DO%(DO #+}%AB!eB x3B77b.CAyA~s# INI JE&$ TB.r rrrrc`X lX0lX@lSnS/S-nU(a9[[ U55H nX7nXXU-- nXbUS- ==U- ss'M" O#UHn XnXXU-- nXiS- ==U- ss'M [ R "U5UlX`l[S5V s/sHoU (dMU PM sn Ul U(d [(d[uUl Ul O[X!5uUl Ul U(d [(a/UlgUUlgs sn f)aStore the frequency data and derive the norm and row aggregates. Single finalization path shared by both constructors so pruning fields cannot silently diverge between them. Exactly one of *freq* and *values* carries the weights, never both. The streaming constructor passes the dense *freq* table it had to build and leaves *values* empty; the sparse constructor fills *values* parallel to *nonzero* and passes no table, which is what lets it skip a 65536-entry allocation per call. Neither is retained. *freq* is read here to derive the norm, the row aggregates and the packed buffers, and is then dropped -- scoring reads ``idx_arr``/``val_arr`` or ``values``, never the dense table, so keeping it alive would pin 512 KB per profile for nothing. rrr=N)rrrr@r!mathsqrtrrr_KERNEL_COMPILEDrrrrr) rrrrrnorm_sqrrvrrs rrBigramProfile._finishs.  $ cCi 3w<(Iq5 q)Q.)) Iq5 "a'"))G, */*E*B R*E )))6 &DL$,)5g)D &DL$,!%5%5B D Fs , D+=D+ weighted_freqcU"S5n/n/nSnUR5H4upgU(dMURU5 URU5 XW- nM6 UR/X4U5 U$)uCreate a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the stored fields. Deliberately does not build the dense 65536-entry ``freq`` table the streaming constructor uses. Callers here pass a handful of bigrams — confusion resolution's focused profiles hold a median of eight — and allocating a 65536-element list per call cost about 24us, which measured as roughly 40% of the whole bigram-rescore stage. Scoring reads ``nonzero``/``values``, so the table is never needed. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. r r)rdrCr)clsrprofilerrrrcounts rfrom_weighted_freq BigramProfile.from_weighted_freqsl$c('--/JCus# e$ 0 GU3r )__name__ __module__ __qualname____firstlineno____doc__ __slots__r(rrgintr classmethoddictr__static_attributes__rr rrres> I,/U,/t,/\1A3i1Ac1AS 1A  1A  1AftCH~/r rrrjzbytes | bytearray | memoryview model_keyc[U[5(d [U5nURS:Xag[5nU(aUR U5OSnUc>Sn[ S5HnXnU(dMXWU-- nM [ R"U5nUS:XagURnURn U (a,Sn [ [U55HnXXX-- n M OP[(a"[URURU5n O#Sn URn UHn XU X-- n M XUR-- $)aZScore a pre-computed bigram profile against a single model using cosine similarity. ``bytearray``/``memoryview`` tables are accepted for compatibility but copied to ``bytes`` first: the narrow type lets mypyc compile the dot-product loop with native byte indexing (and keeps compiled and pure-Python installs accepting the same argument types). rNrr) isinstancer(rrzrtr@rrrrr!rrrrr) rrjrrG model_normsq_sumrrrrdotrrs rscore_with_profilers2 eU # #e S   E)29%JuAAqa%YYv& SooG ^^F s7|$A $vy0 0C%  '//5A||C : ) )C w111 22r >brcygagdRARE_LANGUAGESzxxgQ?F)demote_thin_rarerclU(dUcg[5nURU5nUcgUc [U5nSnSnSnSn UHGupn [X+U 5n X:aU nU nU(dM$U [;dM0U [ :wdM<X:dMCU nU n MI U(aUbU[;aU bXh- [ :aU nXg4$)aScore data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :param demote_thin_rare: Pass true only when the caller has judged the *original* input thin (under :data:`_THIN_RARE_MAX_BYTES` before any transcoding). A :data:`RARE_LANGUAGES` winner that leads the best prevalent-language variant by less than the measured noise band is then reported under the prevalent language instead. The length judgment deliberately lives with the caller: this function may receive transcoded bytes or a profile without its source data, so ``len(data)`` here is not a reliable proxy for input size. Language-fill callers pass this; encoding-ranking callers must not, so that candidate ordering stays byte-identical. :returns: A ``(score, language)`` tuple. The score is always the best cosine similarity across variants; the language matches it except when ``demote_thin_rare`` fires, in which case the label is the best prevalent-language variant's while the score remains the rare winner's. N)rNr)rqrtrrr ART_LANGUAGE_THIN_RARE_MARGIN)rrrrrrhvariants best_score best_langbest_prevalentbest_prevalent_langrkrjrss rscore_best_languager]sD GO OEyy"H%J IN&*"*Y wy 9 >JI  N* $"N"& #+   !  '  +  '*; ;'   r )i))N)Ararray functoolsrimportlib.resourcesrTrrErYr"chardetrchardet._kernelrrchardet.registryrrr__file__endswithrStruct unpack_fromr?rBr(r@rr>rrrstr__annotations__r_encr! languagesrJrrgr:tuplefloatrMcacher^rarorqrvboolrxrzrrrr frozensetrr_THIN_RARE_MAX_BYTESrrrr rrs   46S!5;;s#34 ##,,_=t$00--%11  INs    $&$sCx.% OO D 4>>a&*nnQ&7#  CIE EE%)#YE