=KiSrSSKJr SSKrSSKrSSKrSSKrSSKJr SSK J r J r SSK J r SSKJrJrJrJrJrJr SSKJrJrJrJrJr SS KJr SS KJr SS KJ r SS K!J"r"J#r#J$r$J%r% \RL"\'5r("S S\5r)"SS\SS9r*"SS\5r+"SS\SS9r,"SS5r-\\+\,\.4r/"SS\5r0\\0\.4r1"SS\-5r2S'Sjr3Sr4S(Sjr5"S S!5r6S)S"jr7S*S#jr8S+S$jr9S,S%jr:\\\ \ RJ\ \ RH/\\+\,44\\;\ RJ\;\ RH/\\+\,444rr?r@rBrCrDr model_configrrPr*r"r+r,r6r67s H@E:0E:8#Hn#8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> )H-L'"#r+r6c$\rSrSr%SrS\S'Srg)EvaluationResultsazeBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsr"Nr#r"r+r,rUrUas $#!r+rUcb\rSrSrSr\SSSjj5rSSSjjrSrg) RunEvaluatorlzEvaluator interface class.Ncg)zEvaluate an example.Nr")rOrunexampleevaluator_run_ids r, evaluate_runRunEvaluator.evaluate_runosr+c^^^^^# [R"5mUUUUU4Sjn[R"5R SU5IShvN $N7f)z#Evaluate an example asynchronously.c>[R"S0TD6 TRTTT5sSSS5 $!,(df  g=f)Nr")rhtracing_contextr_)current_contextr^r]r\rOsr,_run_with_context5RunEvaluator.aevaluate_run.._run_with_contexts3##6o6((g7GH766s 5 AN)rcget_tracing_contextasyncioget_running_looprun_in_executor)rOr\r]r^rfres```` @r, aevaluate_runRunEvaluator.aevaluate_runxsG002 I I--/??FWXXXXsAAAAr"NNr\rr]Optional[Example]r^Optional[uuid.UUID]rR*Union[EvaluationResult, EvaluationResults]) r$r%r&r'r(rr_rlr*r"r+r,rYrYls$&*04 # ###. # 4 ##&*04 Y  Y# Y. Y 4 Y Yr+rYcP\rSrSr%SrS\S'S\S'SrS\S 'SrS \S 'S rg) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. r r8z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNrArBz6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r;r") r$r%r&r'r(r)rBr;r*r"r+r,rtrts8 H@ 33459M296FJG CJr+rtc^\rSrSrSrS SSjjrSSSjjrSSjrSSjr\ SSj5r SSSjjr SSU4S jjjr S SS jjr SS jrS rU=r$)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. c^[U5unmU(a[U5unmS U4Sjjn[U5"U5 SSKJn Ub&UR X#S9Ul[ USS5Ul[R"U5(a5Ub [S5eUR XS9Ul[ USS5UlgUR [[[[[/[ 4U5US9Ul[ USS5Ulg) zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. cf>TcU$T"URS5URS55u pU$)Nr\r]getinputs_ traced_inputsprepare_inputss r,process_inputs4DynamicRunEvaluator.__init__..process_inputss<% $2 5!6::i#8% !Q! r+rrNrr$rxFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.rr>rRr>)_normalize_evaluator_funcr langsmithrensure_traceableafuncgetattr_nameinspectiscoroutinefunction TypeErrorr r rr r_RUNNABLE_OUTPUTfuncrOrrrrrs @r,__init__DynamicRunEvaluator.__init__s(";4!@~ &?&F #UN ! d D)  $556DJ! 4IJDJ  & &t , , 3 %556DJ!z3HIDJ#44XsHW$568HHI4P-5DI!z3HIDJr+cv^[T[5(aTR(dUTlT$T(d[ST35eST;aU(aURTS'[ U4SjS55(a[ST35e[S0SU0TED6$![ an[ST35UeSnAff=f) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r8c3,># UH oT;v M g7fNr").0kresults r, @DynamicRunEvaluator._coerce_evaluation_result..sJ,IqF?,I)r9rr;zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got rBz[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r")rIr6rB ValueErrorrallr)rOrrB allow_no_keyes ` r,_coerce_evaluation_result-DynamicRunEvaluator._coerce_evaluation_results f. / /'''4$M  FFLXOF"| $ u J,IJJJ OOUhX$Q &P&PQ Q 44:8=  sA'B B8$B33B8cSU;aAUR5nUSVs/sHnURXBS9PM snUS'[S0UD6$UR[[U5USS9$s snf)NrW)rBT)rBrr")copyrrUr r>)rOrWrBcprs r,_coerce_evaluation_results.DynamicRunEvaluator._coerce_evaluation_resultss  B!++A..q.N+ByM%*r* *-- w }4.  sA'c[U[5(aUR(dX!lU$[U5nUR X5$r)rIr6rB_format_evaluator_resultr)rOrrBs r,_format_result"DynamicRunEvaluator._format_resultsA f. / /'''4$M)&1..vEEr+c[US5$zCheck if the evaluator function is asynchronous. Returns: bool: `True` if the evaluator function is asynchronous, `False` otherwise. rhasattrrNs r,is_asyncDynamicRunEvaluator.is_async$tW%%r+c[US5(dU[R"5nUR5(a [ S5eUR UR X55$Uc[R"5nSUR0n[USS5(a[UR5US'URUUX5S.S9nURXc5$) aUEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.NrC session_id experimentrun_idr:langsmith_extra)rriget_event_loop is_running RuntimeErrorrun_until_completerluuiduuid4idrr rrr)rOr\r]r^ running_loopr:rs r,r_ DynamicRunEvaluator.evaluate_run-s"tV$$"113L&&(("R $66t7I7I#7WXX  ##zz| $3SVV#< 3 d + +%(%8H\ "  '7N  ""6<# [US5(d[TU] X5IShvN $Uc[R"5nSUR 0n[ USS5(a[UR5US'URUUX4S.S9IShvN nURXS5$NN7f)a|Evaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rNrCrrrr) rsuperrlrrrrr rrr)rOr\r]r^r:r __class__s r,rl!DynamicRunEvaluator.aevaluate_runSs&tW%%.s<< <  ##zz| $3SVV#< 3 d + +%(%8H\ "zz  '7N"   ""6<<=  s"$B,B(A*B,B*B,*B,c$URX5$)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )r_)rOr\r]s r,__call__DynamicRunEvaluator.__call__ts  ..r+c"SURS3$))Represent the DynamicRunEvaluator object.zrrNs r,__repr__DynamicRunEvaluator.__repr__s&tzzl!44r+rrrr)rXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]rzIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]rB uuid.UUIDrboolrRr6)rWzUnion[dict, EvaluationResults]rBrrRrr)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]rBrrRrrrRrrnro)r\rr]rpr^rq)r\rr]rprRrrrRr )r$r%r&r'r(rrrrpropertyrr_rlrrr* __classcell__)rs@r,rxrxsE, 8J 8J 8J|# -!   < / !  4  " F  F ! F 4 F&&&*04 $= $=#$=. $= 4 $=R&*04 = =#=. ==D6://!2/ 3/"55r+rxc[U5$)zeCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rxrs r, run_evaluatorrs t $$r+i'c`[U5n[U5[:aUS[S- S-nU$)Nz...))reprlen_MAXSIZE)objss r, _maxsize_reprrs1 S A 1v n1  & Hr+c\rSrSrSrS SSjjr\SSj5rS SSjjrS SSjjr S SSjjr SS jr \ SS j5r SS jrS rg)DynamicComparisonRunEvaluatoriz4Compare predictions (as traces) from 2 or more runs.Nc(^[U5unmU(a[U5unmS U4Sjjn[U5"U5 SSKJn Ub&UR X#S9Ul[ USS5Ul[R"U5(a5Ub [S5eUR XS9Ul[ USS5UlgUR [[[[[[ /["4U5US9Ul[ USS5Ulg) zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. cf>TcU$T"URS5URS55u pU$)Nrunsr]r|r~s r,r>DynamicComparisonRunEvaluator.__init__..process_inputss<% $2 6"FJJy$9% !Q! r+rrNrr$rxrr)$_normalize_comparison_evaluator_funcrrrrrrrrrrr r rrr r_COMPARISON_OUTPUTrrs @r,r&DynamicComparisonRunEvaluator.__init__s%("Fd!K~ &J5&Q #UN ! d D)  $556DJ! 4IJDJ  & &t , , 3 %556DJ!z3HIDJ#44!#(9:*,  .5 DI!z3HIDJr+c[US5$rrrNs r,r&DynamicComparisonRunEvaluator.is_asyncrr+cd[US5(dU[R"5nUR5(a [ S5eUR UR X55$[R"5nURU5nURUUXES.S9nURXdU5$)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rrrtagsr) rrirrrr acompare_runsrr _get_tagsr_format_results)rOrr]rrBrrs r, compare_runs*DynamicComparisonRunEvaluator.compare_runsstV$$"113L&&(("R $66&&t5 ~~d#  '4C  ##F4@@r+c# [US5(dURX5$[R"5nUR U5nUR UUX4S.S9IShvN nUR XSU5$N7f)anEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rrrN)rrrrrrr)rOrr]rBrrs r,r+DynamicComparisonRunEvaluator.acompare_runs s| tW%%$$T3 3 ~~d#zz  '4C"   ##F4@@  sAA:!A8"A:c$URX5$)ayMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rOrr]s r,r&DynamicComparisonRunEvaluator.__call__'s  //r+c"SURS3$)rz>r+c/nUHenURS[UR5-5 [USS5(dM>URS[UR5-5 Mg U$)zExtract tags from runs.zrun:rNz experiment:)appendr rrr)rrr\s r,r'DynamicComparisonRunEvaluator._get_tags<s\C KKSVV, -sL$// MC,??@ r+c[U[5(aUR(dX!lU$[U[5(a:[ X15VVs0sHupEUR U_M snnUR US.nO<[U[5(aSU;aUR US'OSU<3n[U5e[S0SU0UED6$s snnf![an[SU35UeSnAff=f)N)rvr8rBr8zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=rBzExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r") rIrtrBlistziprrr>rr)rOrrBrr\r9msgrs r,r-DynamicComparisonRunEvaluator._format_resultsGs  f8 9 9'''4$M  % %;>t;LM;LZS3665=;LMzz!.F  % %F" $ u -%+I/ S/ ! -"M >J >J >J@&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r+rc[U5$)z.Create a comaprison evaluator from a function.)rrs r,comparison_evaluatorr ls ) ..r+c ^^^ ^ ^ Sm [R"T5m T RR5VVs/sH#upURUR :wdM!UPM% nnnT RR5VVs/sH0upUR [RRLdM.UPM2 snnm U(aB[U U 4SjU55(d8[UVs/sH oDT ;dM UPM sn5S:waST S3n[U5e[U U 4SjU55(aUSS/:XaTS4$[R"T5(aOSU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$SU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$s snnfs snnfs snf)N)r\r]routputsreference_outputs attachmentsc3D># UHoT;=(d UT;v M g7frr"rpnameargs_with_defaultssupported_argss r,r,_normalize_evaluator_func..& PXu^ # Bu0B'B BPX UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3D># UHoT;=(d UT;v M g7frr"rs r,rr%LT5>5,>#>>Hrr\r]c>UUU(a URO0UR=(d 0U(aUR=(d 0O0U(aUR=(d 0O0S.n0n/n0nTRR 5HeupgXb;dM UR UR UR4;aURX&5 OX&X6'US;a [X&5OX&XV'Mg XCU4$N)r\r]rrrr)r\r] rrr parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrr r\r]arg_mapkwargsargsr param_nameparamsigs r,_prepare_inputs2_normalize_evaluator_func.._prepare_inputs&07gnnR"{{0b@G7#6#6#<"RBI)>Br  " ),)=)=)?%J!, ::!77!11*!KK(;<181DF. *-??*'*=>!(!4&1*@]22r+cF># T"X5up#nT"U0UD6IShvN $N7frr"r\r]r)r(rr-rs r,awrapper+_normalize_evaluator_func..awrappers-%4C$A!q!4262222 !!r$c>UUU(a URO0UR=(d 0U(aUR=(d 0O0U(aUR=(d 0O0S.n0n/n0nTRR 5HeupgXb;dM UR UR UR4;aURX&5 OX&X6'US;a [X&5OX&XV'Mg XCU4$rr r&s r,r-r.r/r+c*>T"X5up#nT"U0UD6$rr"r1s r,wrapper*_normalize_evaluator_func..wrappers"$3C$A!qT,V,,r+)r\rr]rprRtuple[list, dict, dict])r\rr]rprRrr signaturer!r"r# VAR_KEYWORDdefault Parameteremptyrrrrrrr$ rrpall_argsarr2r7r-rr,rs ` @@@@r,rrvs9N   D !C&)nn&:&:&<X&<(%!--@W& 3 3#4 3! 3 34,,j)&&   o. . 3 3#4 3( 3> - - 4,,j)%%   _- -kYEs# G4G4-G:6G:+ H8Hc^^^ ^ ^ Sm [R"T5m T RR5VVs/sH#upURUR :wdM!UPM% nnnT RR5VVs/sH0upUR [RRLdM.UPM2 snnm U(aB[U U 4SjU55(d8[UVs/sH oDT ;dM UPM sn5S:waST S3n[U5e[U U 4SjU55(aUSS/:XaTS4$[R"T5(aOSU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$SU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$s snnfs snnfs snf)Nrr]rrrc3D># UHoT;=(d UT;v M g7frr"rs r,r7_normalize_comparison_evaluator_func..rrrrrc3D># UHoT;=(d UT;v M g7frr"rs r,rrG rrrr]c>UUU(a URO0UVs/sHo"R=(d 0PM snU(aUR=(d 0O0S.n0n/n0nT RR5HeupxXs;dM URUR UR 4;aURX75 OX7XG'US;a [X75OX7Xg'Mg XTU4$s snfNrE)rr] rrr!r"r#r$r%rr rr]r\r'r(r)rr*r+r,s r,r-=_normalize_comparison_evaluator_func.._prepare_inputs*!&07gnnR=ABTc 1r 1TBBI)>Br  " ),)=)=)?%J!, ::!77!11*!KK(;<181DF. *-@@*'*=>!(!4&1*@]22+ CC) cF># T"X5up#nT"U0UD6IShvN $N7frr"rr]r)r(rr-rs r,r26_normalize_comparison_evaluator_func..awrapperHs-%4D$B!q!4262222r4r$c>UUU(a URO0UVs/sHo"R=(d 0PM snU(aUR=(d 0O0S.n0n/n0nT RR5HeupxXs;dM URUR UR 4;aURX75 OX7XG'US;a [X75OX7Xg'Mg XTU4$s snfrJrKrLs r,r-rMWrNrOc*>T"X5up#nT"U0UD6$rr"rQs r,r75_normalize_comparison_evaluator_func..wrapperus$%4D$B!qT,V,,r+)rr r]rprRr9)rr r]rprRrr:r@s ` @@@@r,rrsPSN   D !C&)nn&:&:&<X&<(%!--@W&)rxs r,r+_format_evaluator_result..s71:a&&sz8Expected a list of dicts or EvaluationResults. Received .rWrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rIrrKrJrrrr r>)rs r,rrs&4,--6"* M);;A( D   FD ! !7777J6(RST V$ M FC 6" M FD ! ! M &&,X /  r+c^^^^ Sm [R"T5mTRR5VVs/sHupUPM nnnTRR5VVs/sH0upUR[R R LdM.UPM2 snnmU(aB[UU 4SjU55(dH[UVs/sH oDT;dM UPM sn5S:wa!ST S3nU(a USUS3- n[U5e[U 4SjU55(aUSS /:XaT$S UU4S jjn[TS 5(a[TS 5Ul U$URUl U$s snnfs snnfs snf) Nrexamplesrrrc3D># UHoT;=(d UT;v M g7frr"rs r,r/_normalize_summary_evaluator..rrrrrZz Received arguments c3,># UH oT;v M g7frr")rrrs r,rr_s?hU.(hrrr]c $>UUUVs/sHo"RPM snUVs/sHo3R=(d 0PM snUVs/sHo"R=(d 0PM snS.n0n/nT RR5HMupxXt;dM URUR UR 4;aURXG5 MGXGXW'MO T "U0UD6n [U [5(aU $[U 5$s snfs snfs snf)Nr\) rrr!r"r#r$r%rrIr6r) rr]r]r\r'r(r)r*r+rrr,s r,r7-_normalize_summary_evaluator..wrappers$9ABg>>B9=>#KK-2->KS%T8oo&;&;8%T GFD%(^^%9%9%;! (zz33--& G$78-4-@*&<4*6*F&"233 +F3 3)C>%TsDD D r$)rzSequence[schemas.Run]r]zSequence[schemas.Example]rRrr) rr;r!r"r=r>r?rrrrrr$) rrrArBrCrr7rr,rs ` @@@r,_normalize_summary_evaluatorrcsSN   D !C&)nn&:&:&<=&<(%&EsE7(-E=E= FF)rr)rr)rrrRr)rr rRztuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]])rr rRztuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]])rz;Union[EvaluationResults, dict, str, int, bool, float, list]rRzUnion[EvaluationResults, dict])rr rRSUMMARY_EVALUATOR_T)>r( __future__rrirloggingrabcrcollections.abcrr functoolsrtypingrr r r r r pydanticrrrrrtyping_extensionsrrrrcrlangsmith.schemasrrrr getLoggerr$rLrr.r6rUrYr>rrtrrxrrrrr rrrrrdrcr"r+r,rosE" /TS''BB   8 $'y' 6Ye 6'y'T" "YY8)+r+