KKiQE:SrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r SSK Jr SS KJr SS KJr SS KJr SS KJr \R."\5r"S S\\ 5r"SS\5r"SS5r"SS\\5r"SS\\5r"SS\\5r g)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)Any)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrg) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____firstlineno____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATION__static_attributes__r%e/var/www/html/dynamic-report/venv/lib/python3.13/site-packages/langchain_classic/evaluation/schema.pyrrs& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5Ir@rc6\rSrSrSr\\SSj55rSrg) LLMEvalChainKz,A base class for evaluators that use an LLM.c g)z#Create a new evaluator from an LLM.Nr%)clsllmkwargss rAfrom_llmLLMEvalChain.from_llmNr@r%N)rGr rHrreturnrC) r&r'r(r)r* classmethodrrIr?r%r@rArCrCKs622r@rCc\rSrSrSr\S Sj5r\S Sj5r\S Sj5r\S Sj5r S S Sjjr S r g)_EvalArgsMixinTz(Mixin for checking evaluation arguments.cgz2Whether this evaluator requires a reference label.Fr%selfs rArequires_reference!_EvalArgsMixin.requires_referenceWr@cg)0Whether this evaluator requires an input string.Fr%rSs rArequires_input_EvalArgsMixin.requires_input\rWr@c6SURRS3$)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r&rSs rA_skip_input_warning"_EvalArgsMixin._skip_input_warningas$DNN$;$;#<^^,,--GHCS/ !  d&9&9 ))a 8  " "y'8^^,,--JKCS/ !  )@)@ --! <*A r@r%rLboolrLstr)NN)ri str | NonerjrrrLNone) r&r'r(r)r*propertyrUrZr`rcrlr?r%r@rArOrOTs2 VV  !%!===  ==r@rOc\rSrSrSr\S Sj5r\S Sj5r\SSS.SSjj5r SSS.SSjjr SSS.SS jjr SSS.SS jjr S r g)StringEvaluatorz}String evaluator interface. Grade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels. c.URR$)zThe name of the evaluation.r^rSs rAevaluation_nameStringEvaluator.evaluation_names~~&&&r@cgrRr%rSs rArU"StringEvaluator.requires_referencerWr@Nriinputc g)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr%rT predictionrir~rHs rA_evaluate_strings!StringEvaluator._evaluate_stringsrKr@cT# [SUR4UUUS.UD6IShvN $N7f)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrrir~)r rrs rA_aevaluate_strings"StringEvaluator._aevaluate_stringssD.%   " " "       s (&(c LURX#S9 UR"SUUUS.UD6$)a}Evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. rirjrr%)rlrrs rAevaluate_strings StringEvaluator.evaluate_stringss@& ##i#F%% !    r@ch# URX#S9 UR"SUUUS.UD6IShvN $N7f)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. rrNr%)rlrrs rAaevaluate_strings!StringEvaluator.aevaluate_stringssM& ##i#F,, !      s )202r%rprn) rz str | Anyristr | Any | Noner~rrHrrLdict) rrqrirrr~rrrHrrLr)r&r'r(r)r*rtryrUrrrrrr?r%r@rArvrvs@ '' '+"&   $         8'+"&   $        H!%           >!%            r@rvc\rSrSrSr\SSS.S Sjj5rSSS.S SjjrSSS.S SjjrSSS.S Sjjr S r g) PairwiseStringEvaluatori zDCompare the output of two models (or two outputs of the same model).Nr}c g)Evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nr%rTr prediction_brir~rHs rA_evaluate_string_pairs.PairwiseStringEvaluator._evaluate_string_pairs rKr@cV# [SUR4UUUUS.UD6IShvN $N7f)Asynchronously evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nrrrir~)r rrs rA_aevaluate_string_pairs/PairwiseStringEvaluator._aevaluate_string_pairs#sG*%   ' ' "%      )')c NURX4S9 UR"SUUUUS.UD6$)rrrr%)rlrrs rAevaluate_string_pairs-PairwiseStringEvaluator.evaluate_string_pairsBsC* ##i#F** !%     r@cj# URX4S9 UR"SUUUUS.UD6IShvN $N7f)rrrNr%)rlrrs rAaevaluate_string_pairs.PairwiseStringEvaluator.aevaluate_string_pairs`sP* ##i#F11 !%       *313r%) rrqrrqrirrr~rrrHrrLr) r&r'r(r)r*rrrrrr?r%r@rArr s=N !%             6!%            H!%            F!%             r@rc\rSrSrSr\S Sj5r\SS.S Sjj5rSS.S Sjjr SS.S Sjjr SS.S S jjr S r g) AgentTrajectoryEvaluatoriz,Interface for evaluating agent trajectories.cg)rYTr%rSs rArZ'AgentTrajectoryEvaluator.requires_inputsr@N)ric g)8Evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. Nr%rTragent_trajectoryr~rirHs rA_evaluate_agent_trajectory3AgentTrajectoryEvaluator._evaluate_agent_trajectoryrKr@cV# [SUR4UUUUS.UD6IShvN $N7f)GAsynchronously evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. N)rrrir~)r rrs rA_aevaluate_agent_trajectory4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysG,%   + + "-      rc NURXCS9 UR"SUUUUS.UD6$)rrrr~rrir%)rlrrs rAevaluate_agent_trajectory2AgentTrajectoryEvaluator.evaluate_agent_trajectorysC, ##i#F.. !-     r@cj# URXCS9 UR"SUUUUS.UD6IShvN $N7f)rrrNr%)rlrrs rAaevaluate_agent_trajectory3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysP, ##i#F55 !-       rr%rn) rrqrz!Sequence[tuple[AgentAction, str]]r~rqrirrrHrrLr) r&r'r(r)r*rtrZrrrrrr?r%r@rArrsE6 !%   <          :!%   <         L!%   <         J!%   <          r@r)!r* __future__rloggingabcrrcollections.abcrenumrtypingrwarningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configr langchain_classic.chains.baser getLoggerr&loggerrqrrCrOrvrrr%r@rArs9"#$-<;/   8 $3JC3Jl2521=1=h~ nc~ Bs ncs l| ~s| r@