*pj)2%SrSSKrSSKJrJrJr SrSrSrSr Sr S r S r S r S \-r\\S 'S\S\S\4SjrS\S\S-4SjrS\S\S-4SjrS\S\S-4SjrS\S\4SjrSS\S\S\4Sjjrg)aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??g?gffffff?g333333?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncNU[:agURS[5(+$)ucReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r s T/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/chardet/pipeline/utf1632.py_is_null_separator_patternr=s%00~~d$;< <<crUS[n[U5[:ag[U5nUbU$[ U5$)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresults rdetect_utf1632_patternsrQsD-< F 6{%%& !F    rc ^[T5[T5S-- nU[:agTSUmUS-n[U4Sj[S[T5S555n[U4Sj[S[T5S555nX2:Xa:XB- S:a2TR S5n[ U5(a[ S[SS9$[U4S j[S [T5S555n[U4S j[S [T5S555nXb:Xa;Xr- S:a3TR S 5n[ U5(a[ S [SS9$gg![a Nf=f![a gf=f)avCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc3>># UHnTUS:XdMSv M g7frN.0ir s r _check_utf32..{J#9aT!W\#9  rc3D># UHnTUS-S:XdMSv M g7f)r!rNr"r#s rr&r'}s$O$:qd1q5kQ>N$:s  rz utf-32-beencoding confidencelanguagec3>># UHnTUS:XdMSv M g7fr r"r#s rr&r'sI"8QDGqLqq"8r)c3>># UHnTUS:XdMSv M g7fr r"r#s rr&r'r(r)z utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nulls` rrrgsvd)s4y1}-K%%   Dq IJ5CIq#9JJMOE!SY$:OON!n&@3&F ;;{+D%%&(7!&I%3t9a"8IILJ5CIq#9JJM ]%>%D ;;{+D%%&(7!& )"   ""    s$/E"/E% E"!E"% E21E2cx^[[T5[5nXS--nU[:agUS-n[ U4Sj[ SUS555n[ U4Sj[ SUS555nX2- nXB- nU[ :=(a [TSUU5(+nU[ :=(a [TSUU5(+nU(dU(dgSXg4SXX4/n XV:aU R5 Sn S n S n Sn S nU HcunnnTSURU5nU(aS n[U5(dM7U S- n [U5nUXS:a[OS -:dM]Un Un Un Me U cgU (aU S:H=(d U [:nOU=(a U [:nU(a[U [ SS 9$g![a Mf=f)aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. The null pattern only establishes that the data is UTF-16-like. Byte order is always chosen by decoding both ways and comparing text-quality scores, with the null signal breaking near-ties (see below). r2Nc3>># UHnTUS:XdMSv M g7fr r"r#s rr&_check_utf16..K#:ad1gl#:r)rc3>># UHnTUS:XdMSv M g7fr r"r#s rr&rBrCr)r!z utf-16-lez utf-16-begFTgr+)minrrrr4r5_UTF16_MIN_NULL_FRACTIONrreverser6r8r7 _text_quality_QUALITY_TIE_MARGIN_MIN_TEXT_QUALITYrr)r sample_lenr: be_null_count le_null_countbe_fracle_frac le_qualified be_qualifiedsides best_encoding best_qualitybest_qualifiedviablequalified_side_decodedr,_frac qualifiedr=qualityaccepteds` rrrs SY -Jq. J$$aIK5J#:KKMK5J#:KKM'G'G ++ G*4 +O.O"/  I"   sF++ F98F9r=cjU(dgUSSn[SU55nU[U5- [:$)z9Quick check: is decoded text mostly printable characters.FNc3^# UH#oR5(dUS;dMSv M% g7f) r!N) isprintable)r$cs rr&#_looks_like_text.. s Jv!AMAAvs- -)r4r_MIN_PRINTABLE_FRACTION)r=r printables rr7r7s8  $3ZFJvJJI s6{ "%< <10% control characters or >20% combining marks (category ``M*``). NrgLr!MZsr_Cg?g?r)r unicodedatacategoryord) r=rernlettersmarksspacescontrols ascii_lettersracatscores rrHrHs(&5\F F AAvG E FHM ""1% q6S= qLG1v|"  Vs] QJE D[AM aKF Vs] MH|c y3 KE a3 &&E2v&1*   Lr)r])__doc__rmchardet.pipelinerrrrr3rrFrJrIrcrr bytes__annotations__floatboolrrrrstrr7intrHr"rrrs XX   $")+;!;;=U=u==( % Od,B ,5u54!75pdud4!7dN=3=4=99C9%9r