JKixSrSSKJr SSKrSSKrSSKrSSKJr SSKJ r J r J r J r J r Jr SSKrSSKJr SSKJrJr SSKJr SS KJr \ (aSS KJrJrJrJr SS KJr SSK r S r!SSK#J$r$J%r% SSKJ&r&J'r' S r(SSK)J*r* S r+"SS\ 5r,S S.SSjjr-SS S.SSjjr."SS5r/"SS5r0\"5"SS\55r1g!\"a S r!Nzf=f!\"a S r(Nuf=f!\"a S r+Nzf=f)zHTML text splitters.) annotationsN)StringIO)IO TYPE_CHECKINGAnyLiteral TypedDictcast)beta)BaseDocumentTransformerDocument)override)RecursiveCharacterTextSplitter)CallableIterableIteratorSequence) ResultSetTF) BeautifulSoupTag)NavigableString PageElement)etreecB\rSrSr%SrS\S'S\S'S\S'S\S'S rg ) ElementType5zElement type as typed dict.strurlxpathcontentdict[str, str]metadataN)__name__ __module__ __qualname____firstlineno____doc____annotations____static_attributes__r#_/var/www/html/dynamic-report/venv/lib/python3.13/site-packages/langchain_text_splitters/html.pyrr5s% H J Lr+r recursivec"URSUS9$)NT)stringr.find_all)tagr.s r,_find_all_stringsr4Bs <

Introduction

Welcome to the introduction section.

Background

Some background details here.

Conclusion

Final thoughts.

""" documents = splitter.split_text(html_content) # 'documents' now contains Document objects reflecting the hierarchy: # - Document with metadata={"Main Topic": "Introduction"} and # content="Introduction" # - Document with metadata={"Main Topic": "Introduction"} and # content="Welcome to the introduction section." # - Document with metadata={"Main Topic": "Introduction", # "Sub Topic": "Background"} and content="Background" # - Document with metadata={"Main Topic": "Introduction", # "Sub Topic": "Background"} and content="Some background details here." # - Document with metadata={"Main Topic": "Conclusion"} and # content="Conclusion" # - Document with metadata={"Main Topic": "Conclusion"} and # content="Final thoughts." ``` c[USS9Ul[UR5UlURVVs/sHup4UPM snnUlX lgs snnf)aInitialize with headers to split on. Args: headers_to_split_on: A list of `(header_tag, header_name)` pairs representing the headers that define splitting boundaries. For example, `[("h1", "Header 1"), ("h2", "Header 2")]` will split content by `h1` and `h2` tags, assigning their textual content to the `Document` metadata. return_each_element: If `True`, every HTML element encountered (including headers, paragraphs, etc.) is returned as a separate `Document`. If `False`, content under the same header hierarchy is aggregated into fewer `Document` objects. c$[USSS5$)Nr)int)xs r,1HTMLHeaderTextSplitter.__init__..ss1Q48}r+)keyN)sortedheaders_to_split_ondictheader_mapping header_tagsreturn_each_element)selfrDrHr3_s r,__init__HTMLHeaderTextSplitter.__init__sY.$* %<$  #4#;#;<.2.F.FG.FFCC.FG#6 HsAc6UR[U55$)a4Split the given text into a list of `Document` objects. Args: text: The HTML text to split. Returns: A list of split `Document` objects. Each `Document` contains `page_content` holding the extracted text and `metadata` that maps the header hierarchy to their corresponding titles. split_text_from_filerrItexts r, split_text!HTMLHeaderTextSplitter.split_texts(($88r+c SSKJn U"USSS9 [R"U4SU0UD6nUR 5 UR UR 5$)aFetch text content from a URL and split it into documents. Args: url: The URL to fetch content from. timeout: Timeout for the request. **kwargs: Additional keyword arguments for the request. Returns: A list of split `Document` objects. Each `Document` contains `page_content` holding the extracted text and `metadata` that maps the header hierarchy to their corresponding titles. Raises: requests.RequestException: If the HTTP request fails. r)validate_safe_urlFT) allow_private allow_httptimeout))langchain_core._security._ssrf_protectionrUrequestsgetraise_for_statusrRrQ)rIrrXkwargsrUresponses r,split_text_from_url*HTMLHeaderTextSplitter.split_text_from_urlsL&  #UtD<<?W??!!#x}}--r+c[U[5(a$[R"U5R SS9nOUR 5n[ URU55$)aaSplit HTML content from a file into a list of `Document` objects. Args: file: A file path or a file-like object containing HTML content. Returns: A list of split `Document` objects. Each `Document` contains `page_content` holding the extracted text and `metadata` that maps the header hierarchy to their corresponding titles. zutf-8)encoding) isinstancerpathlibPath read_textreadlist_generate_documents)rIfile html_contents r,rO+HTMLHeaderTextSplitter.split_text_from_filesN dC "<<-777IL99;LD,,\:;;r+c #^^# [(d Sn[U5e[US5nUR=(d Un0m/mSUU4SjjnU/nU(Ga;UR 5n[ UR 5nURS[U555 [USS5n U (dMd[USS9V s/sHn [U 5R5PM n n S RS U 55n U (dM[[ UR55n XR ;aUR"(dU"5nU(aUv [%U S S5nTR)5VVVVs/sHununnnUU:dMUPM nnnnnUHnTU M UR*U nXU 4TU'TR)5VVs0sH unnUUS _M nnn[-U US9v OTR)5VVVs/sHunu nnU U:dMUPM nnnnUHnTU M UR"(a6TR)5VVs0sH unnUUS _M nnn[-U US9v OTR/U 5 U(aGM;UR"(dU"5nU(aUv gggs sn f![&a S nGNhf=fs snnnnfs snnfs snnnfs snnf7f)aPrivate method that performs a DFS traversal over the DOM and yields. Document objects on-the-fly. This approach maintains the same splitting logic (headers vs. non-headers, chunking, etc.) while walking the DOM explicitly in code. Args: html_content: The raw HTML content. Yields: Document objects as they are created. Raises: ImportError: If BeautifulSoup is not installed. zEUnable to import BeautifulSoup. Please install via `pip install bs4`. html.parserc>T(dgSRST55nTR5 UR5(dgTR5VVs0sH upXS_M nnn[ XS9$s snnf)z6Finalize the accumulated chunk into a single Document.Nz c3R# UHoR5(dMUv M g7fNstrip).0lines r, UHTMLHeaderTextSplitter._generate_documents..finalize_chunk..s$Tmdzz|TTms' 'r page_contentr")joinclearrsitemsr ) final_textkv final_metaactive_headers current_chunks r,finalize_chunkBHTMLHeaderTextSplitter._generate_documents..finalize_chunk ss $Tm$TTJ    !##%%.<.B.B.DE.Dda!qT'.DJEI IFsA8c3T# UHn[U[5(dMUv M g7frq)rcr)rtchilds r,rv=HTMLHeaderTextSplitter._generate_documents..s#5%E39O#5s( (r5NFr- c36# UHo(dM Uv M g7frqr#)rtelems r,rvr(s H-$4-s  r=i'rrx)returnzDocument | None)_HAS_BS4 ImportErrorrbodypoprhchildrenextendreversedgetattrr4rrsrzlenparentsrGrHr> ValueErrorr|rFr append)rIrkmsgsouprrstacknoderr3r text_elements node_text dom_depthdoclevelr~rJlvldheaders_to_removerB header_namer header_metaheaders_out_of_scopemetarrs @@r,ri*HTMLHeaderTextSplitter._generate_documentss xW c" "\=9yy D ;=#%  J J99;DDMM*H LL#+H#5 $-C1B$RW0X0XuE   "0X  H- HHID./I&&&//(*C! !ABLE -;,@,@,B%,B.![aacUlA,B"%-C&s+-#11#6 /8.K{+4B3G3G3IJ3I41aq!A$w3I JI LL+9*>*>*@(*@,!YaAIPQMA*@%(0C&s+0++0>0D0D0FG0F1AqtG0FDG" DII"((3eD'' "C (m("! E!%K(HsyB:K*>#J7!A3K*J<#K*8K K :K* K $K*KK4K*K$&(K*+K*< K K* K  K*)rFrGrDrHN)F)rDlist[tuple[str, str]]rHboolrNonerQrrlist[Document]) )rrrXr>r]rrr)rjz str | IO[str]rr)rkrrzIterator[Document]) r$r%r&r'r(rKrRr_rOrir*r#r+r,r9r9Ssh;@%*727"7  7< 9(*..!$.58. .8<$sr+r9cz\rSrSrSrS SjrS SjrSSjrSSSjjrSSjr SS jr SS jr S r g)HTMLSectionSplitteriazTSplitting HTML files based on specified tag and font sizes. Requires lxml package. c [U5Ul[R"[5R S- R 5UlX lg)a^Create a new `HTMLSectionSplitter`. Args: headers_to_split_on: List of tuples of headers we want to track mapped to (arbitrary) keys for metadata. Allowed header values: `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, e.g.: `[("h1", "Header 1"), ("h2", "Header 2"]`. **kwargs: Additional optional arguments for customizations. zxsl/converting_to_header.xsltN) rErDrdre__file__parentabsolute xslt_pathr])rIrDr]s r,rKHTMLSectionSplitter.__init__gs? $((;#< LL " ) ),K K (*  r+c//p2UH9nURUR5 URUR5 M; URX#S9n[ S0UR D6nUR U5$)zSplit documents. Args: documents: Iterable of `Document` objects to be split. Returns: A list of split `Document` objects. ) metadatasr#)rryr"create_documentsrr]split_documents)rI documentstextsrrresults text_splitters r,r#HTMLSectionSplitter.split_documents}spryC LL)) *   S\\ *'''C6EE ,,W55r+c6UR[U55$)zeSplit HTML text string. Args: text: HTML text Returns: A list of split `Document` objects. rNrPs r,rRHTMLSectionSplitter.split_texts(($88r+NcU=(d 0/[U5-n/n[U5HupVURU5Hn[R"X55nUR H*n UR U S:XdMUSUR U 'M, 0UEUR En[ URUS9n URU 5 M M U$)zCreate a list of `Document` objects from a list of texts. Args: texts: A list of texts to be split and converted into documents. metadatas: Optional list of metadata to associate with each document. Returns: A list of `Document` objects. #TITLE#Titlerx) r enumeraterRcopydeepcopyr"r ryr) rIrr metadatas_rirQchunkr"rBnew_docs r,r$HTMLSectionSplitter.create_documentss32$U"3   'GA.==7 >>C~~c*i7.6w.?s+*:h9%..9"0B0BXV  )/(r+cr[(d Sn[U5e[US5n[URR 55n/n[ US/UQS9n[U5HupxUS:XaSn Sn /n O(URR5n URn /n URHJn US-[U5:a XUS-:Xa O,[U [5(dM9U RU 5 ML S R!U 5R5n U (dMURU U U S .5 M U$) aSplit an HTML document into sections based on specified header tags. This method uses BeautifulSoup to parse the HTML content and divides it into sections based on headers defined in `headers_to_split_on`. Each section contains the header text, content under the header, and the tag name. Args: html_doc: The HTML document to be split into sections. Returns: A list of dictionaries representing sections. Each dictionary contains: * `'header'`: The header text or a default title for the first section. * `'content'`: The content under the header. * `'tag_name'`: The name of the header tag (e.g., `h1`, `h2`). Raises: ImportError: If BeautifulSoup is not installed. zzUnable to import BeautifulSoup/PageElement, please install with `pip install bs4`.rnrr5rrh1r=r)headerr tag_name)rrrrhrDkeysr7rrQrsr5 next_elementsrrcrrrz)rIhtml_docrr header_namessectionsheadersrrcurrent_headercurrent_header_tagsection_contentelementr s r,split_html_by_headers)HTMLSectionSplitter.split_html_by_headerss.,xCc" "X}5D4499;< 02 V,Cl,CD"7+IAAv!*%)"-/!'!2!2!4%+[[""$!//q53w<'Gq1u~,Egs++#**73 0 hh/557Gw"0#*$6#,2r+c[(d Sn[U5e[R"SS9n[R"SSSS9n[R R n[R"[U5U5n[R"URU5n[R"XuS9nU"U5n [U 5$)aConvert specific HTML tags to headers using an XSLT transformation. This method uses an XSLT file to transform the HTML content, converting certain tags into headers for easier parsing. If no XSLT path is provided, the HTML content is returned unchanged. Args: html_content: The HTML content to be transformed. Returns: The transformed HTML content as a string. Raises: ImportError: If the `lxml` library is not installed. z>Unable to import lxml, please install with `pip install lxml`.T) no_networkF)resolve_entitiesrload_dtd)access_control) _HAS_LXMLrr HTMLParser XMLParserXSLTAccessControlDENY_ALLparserrXSLTr) rIrkr html_parser xslt_parseractree xslt_tree transformresults r,convert_possible_tags_to_header3HTMLSectionSplitter.convert_possible_tags_to_headers yRCc" "&&$7 oo"te  $ $ - -{{8L1;?KK < JJy< 46{r+c UR5nURU5nURU5nUVs/sH8n[[ SUS5UR [ US5US0S9PM: sn$s snf)zSplit HTML content from a file into a list of `Document` objects. Args: file: A file path or a file-like object containing HTML content. Returns: A list of split `Document` objects. rr rr)r")getvaluerrr r rDr)rIrj file_contentrsections r,rO(HTMLSectionSplitter.split_text_from_files}} ;;LI --l;$  $ UGI./,,S1D-EF I $   s?A9)rDr]r)rDrr]rrr)rzIterable[Document]rrrrq)r list[str]rzlist[dict[Any, Any]] | Nonerr)rrrzlist[dict[str, str | None]])rkrrr)rjrrr) r$r%r&r'r(rKrrRrrrrOr*r#r+r,rrasg 2  ,6& 9JN+F 4;z#J r+rcN\rSrSrSrSSSSSSSSSSSSSSSSSS .SS jjrSS jr\SS j5rSS jr \ SSj5r SSjr SSjr SSjrSSjrSSjr\ SSj5rSrg)HTMLSemanticPreservingSplitteri1aSplit HTML content preserving semantic structure. Splits HTML content by headers into generalized chunks, preserving semantic structure. If chunks exceed the maximum chunk size, it uses `RecursiveCharacterTextSplitter` for further splitting. The splitter preserves full HTML elements and converts links to Markdown-like links. It can also preserve images, videos, and audio elements by converting them into Markdown format. Note that some chunks may exceed the maximum size to maintain semantic integrity. !!! version-added "Added in `langchain-text-splitters` 0.3.5" Example: ```python from langchain_text_splitters.html import HTMLSemanticPreservingSplitter def custom_iframe_extractor(iframe_tag): ``` Custom handler function to extract the 'src' attribute from an