langchain.smith.evaluation.runner_utils.arun_on_dataset¶
async langchain.smith.evaluation.runner_utils.arun_on_dataset(client: Client, dataset_name: str, llm_or_chain_factory: Union[Callable[[], Union[Chain, Runnable]], BaseLanguageModel, Callable[[dict], Any], Runnable, Chain], *, evaluation: Optional[RunEvalConfig] = None, concurrency_level: int = 5, num_repetitions: int = 1, project_name: Optional[str] = None, verbose: bool = False, tags: Optional[List[str]] = None, input_mapper: Optional[Callable[[Dict], Any]] = None) → Dict[str, Any][source]¶
Asynchronously run the Chain or language model on a dataset
and store traces to the specified project name.
Parameters
client – LangSmith client to use to read the dataset, and to
log feedback and run traces.
dataset_name – Name of the dataset to run the chain on.
llm_or_chain_factory – Language model or Chain constructor to run
over the dataset. The Chain constructor is used to permit
independent calls on each example without carrying over state.
evaluation – Optional evaluation configuration to use when evaluating
concurrency_level – The number of async tasks to run concurrently.
num_repetitions – Number of times to run the model on each example.
This is useful when testing success rates or generating confidence
intervals.
project_name – Name of the project to store the traces in.
Defaults to {dataset_name}-{chain class name}-{datetime}.
verbose – Whether to print progress.
tags – Tags to add to each run in the project.
input_mapper – A function to map to the inputs dictionary from an Example
to the format expected by the model to be evaluated. This is useful if
your model needs to deserialize more complex schema or if your dataset