9i'SSKrSSKJr SSKrSSKJrJr SSKr SSK r SSK J r SSKJr SSKJr SSKJr SSKJr SSKJrJr SS KJrJr SS KJr \ R<S :a'\ R>R@r \ RB"5 \"5r"S /r#\RH"\RJ\ RLS 9"SS \55r'g)N)AnyDict) Pipelines)Model) OutputKeys)Pipeline) PIPELINES)Config ConfigFields) ModelFileTasks) get_loggerz2.0LanguageIdentificationPipeline) module_namec^\rSrSrSrS\4U4SjjrS\S\4SjrS\S\ \\ 44Sjr S\ \\ 4S\ \\ 44S jr S \ \\ 4S\ \\ 44S jr S rU=r$) ru Language Identification Pipeline. Examples: >>> from modelscope.pipelines import pipeline >>> from modelscope.utils.constant import Tasks >>> pipeline_ins = pipeline(Tasks.text_classification, 'damo/nlp_language_identification-classification-base') >>> pipeline_ins('Elon Musk, co-founder and chief executive officer of Tesla Motors.\n' \ >>> 'Gleichzeitig nahm die Legion an der Befriedung Algeriens teil, die von.\n' \ >>> '使用pipeline推理及在线体验功能的时候，尽量输入单句文本，如果是多句长文本建议人工分句。' >>> { >>> "labels":[ >>> "en", >>> "de", >>> "zh" >>> ], >>> "scores":[ >>> [('en', 0.99)], >>> [('de', 1.0)], >>> [('zh', 1.0)] >>> ] >>> } modelc 0>[TU]"SSU0UD6 UnSUl[R"[ R RU[R55Ul [ R RX0R[RS5n/n/n[[US55HKupxUR5nUR!S5nUR#X45 UR#Xx45 MM [)U5Ul[)U5UlUR*R/SS5UlUR*R/S S 5Ul[ R RX0R[RS 5n [)[[U S S S95VVs/sHupxXxR54PM snn5UlSUl:"5 [8R<"SS9n SU R>l [8RB"U S9Ul"[8RFRHRKURD[8RFRLRN/U5 [8RP"5n UR(a;U RS5H'n ['U RTU RW55 M) U RYS5Ul-U RYS5n U RYS5nU US.Ul.[8R^"5n[8R`"5nURDRcUU/5 [8RFRHRKURD[8RFRLRN/U5 g![$a" UR(a ['SX5 GMTf=fs snnf)z~Build a language identification pipeline with a model dir or a model id in the model hub. Args: model: A Model instance. rFvocabrbzutf-8z error vocab:zrlabelrutf8)encodingunkT)allow_soft_placement)configz src_cid:0zoutput_label:0zpredict_score:0) output_ids output_scoreN)2super__init__debugr from_fileospathjoinr CONFIGURATIONcfgr preprocessor enumerateopenstripdecodeappendUnicodeDecodeErrorprintdictr vocab_reversegetunk_idpad_idr unk_labeltfreset_default_graph ConfigProto gpu_options allow_growthSession_session saved_modelloaderload tag_constantsSERVINGget_default_graphget_operationsnamevaluesget_tensor_by_name input_idsoutputglobal_variables_initializerlocal_variables_initializerrun)selfrkwargs export_dirjoint_vocab_file vocabfilesvocabfiles_reverseiwjoint_label_file tf_config default_graphop output_labelr!init local_init __class__s x/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/pipelines/nlp/language_identification_pipline.pyr$'LanguageIdentificationPipeline.__init__:s& /u//  ## GGLLY%<%< =?77<< !:!:;GDF d#3T:;DA A HHW%!!1&)"))1&1 <*% !"45jjnnWa0 jjnnVQ/ 77<< !:!:;GDFi !3 87:;7:daAwwy>7:;<   NN= -1 * )4  ""4==$&NN$@$@$H$H#I#- /,,. ::#224bggryy{+5'99+F$778HI $778IJ '(  ..0335  4,- ""4==$&NN$@$@$H$H#I#- /Q& ::.!/  ;s5O#P #'PPinputreturnc PUR5nSn[R"USU5nSn[R"USU5nSn[R"USU5nSnSnSRUVs/sHnX;aU"U5OSPM sn5n[R"S[R 5n [R"U SU5nSRUR 5V s/sHQn [[R"S U 55(a'[[R"S U 55(aMOU PMS sn 5n/n UR5Hln URRXR5n [U 5S :a$XR:XaU S UR:XaM[U RU 5 Mn [U 5S :aU S UR:XaU S Sn [U 5S :aU S UR:XaU SS n U $s snfs sn f)Nz/<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});z \S+[./]\S+\s?z \S*@\S*\s?c[U5nSU:dUS:aUS-nOUS:XaSnOUS;aSnOUS;aS n[U5$) Nii_ii0 )i0i0i i i i ")i i i i ')ordchr)uchar inside_codes r` stringpartQ2BELanguageIdentificationPipeline._lid_preprocess..stringpartQ2Bs^e*K #{V';v% &$ !%  @@$ {# #uV,-+"'\&.!=:;°·$«»|±[]{}_?<>~^*/%#@()，。！《》？、` …‼️ u`[😀-🙏🌀-🗿🚀-🛿🇠-🇿🤦-🤷𐀀-􏿿✂-➰♀-♂☀-⭕‍⏏⏩⌚️〰]+z\dz^[a-z0-9+-_]+$rr)lowerresubr)compileUNICODEsplitboolsearchmatchr/rr6r7lenr1)rPrbsentenceCLEANRURLREEMAILRErn m_noisyCharscEMOJIREitemoutidsrWtmps r`_lid_preprocess.LanguageIdentificationPipeline._lid_preprocess}s;;=C66&"h/ 66%X.66'2x0 $w 77 !" 5M! 3 >  **  JJ 66'2x088%^^- -T5$/00BHH%6=> -  !A**..KK0C6 KK/F2J$++4M MM#  " v;?vayDKK7ABZF v;?vbzT[[8CR[F M & s3H-AH#?H#c URS5nUVs/sH*nUR5S:wdMURU5PM, nnUR(a[ X$5Hpup5[ SU5 [ SSR UVs/sH8nURRX`R5RSS5PM: sn55 Mr [UVs/sHn[U5PM sn5nUH.nURUR/U[U5- -5 M0 [R "U5nSU0n U $s snfs snfs snf)N rezraw:zres:rrqrK)rxr/rr%zipr3r)r5r6r7replacemaxr|extendr8nparray) rPrb sentenceltr} input_ids_ltrKwididsmaxlenresults r` preprocess)LanguageIdentificationPipeline.preprocesss<[[& ;E ;Ex~~2% +D  *:  ::'*:'D#fh'BGG5>%5>c**..sKK@HH#S*5>%(E,7,3c#h,78C JJ }S(9: ; HH\* y) ) %8sEE ?E"EcURR5 URUS0nURRURUS9nUsSSS5 $!,(df  g=f)NrK) feed_dict)r@ as_defaultrKrOrL)rPrbr sess_outputss r`forward&LanguageIdentificationPipeline.forwardsQ ]] % % '{);.s!rpT)keyreverser)rg?g{Gz?) setrrrIr1sortedr|roundrLABELSSCORES)rProutput_scores_rawsupported_104_langlabels_scores_lt output_labelsr!tmpltsl labels_scoresrscore output_scoresrs r` postprocess*LanguageIdentificationPipeline.postprocesss@">2 "   -LEL***;*;*=>. aV$?5ndCBQGE5zQ"   # #E *  q! -./?@.>]/<M.;leut|35%q/2.;M.> @   }   }  M@s6 D DD#DD) r@r+r%rKrrLr8r7r9rr5)__name__ __module__ __qualname____firstlineno____doc__strr$listrrrrrr__static_attributes__ __classcell__)r_s@r`rrs4A/cA/FBSBTBHS#X0 T#s(^ S#X $$sCx.$T#s(^$$rp)(r'os.pathr(osprttypingrrnumpyr tensorflowr:modelscope.metainformodelscope.models.basermodelscope.outputsrmodelscope.pipelines.basermodelscope.pipelines.builderr modelscope.utils.configr r modelscope.utils.constantr r modelscope.utils.loggerr __version__compatv1disable_eager_executionlogger__all__register_moduletext_classificationlanguage_identificationrr"rpr`r?s  )().286.>>U B  + ,  9+L+LNdXdNdrp