prompttools.prompttest.runner package

Submodules

prompttools.prompttest.runner.prompt_template_runner module

prompttools.prompttest.runner.runner module

class prompttools.prompttest.runner.runner.PromptTestRunner

Bases: object

Base class for prompt test runners. Please use the subclass instead.s

evaluate(key, metric_name, eval_fn, expected=None)

Evaluates the test results using the given eval_fn.

Parameters:

• key (str) –

• metric_name (str) –

• eval_fn (Callable) –

• expected (Optional[str]) –

Return type:

None

run(*args, **kwargs)

Runs the test if it has not already been run.

Return type:

str

scores(key)

Returns the scores for the underlying experiment at the given key.

visualize(key)

Evaluates the test results using the given eval_fn.

Parameters:

key (str) –

Return type:

None

prompttools.prompttest.runner.runner.run_prompttest(metric_name, eval_fn, threshold, threshold_type, prompts, results, expected)

Runs the prompt test evaluation.

Parameters:

• metric_name (str) –

• eval_fn (Callable) –

• threshold (float) –

• threshold_type (ThresholdType) –

• prompts (List[str]) –

• results (List[str]) –

• expected (Optional[List[str]]) –

Return type:

int

prompttools.prompttest.runner.system_prompt_runner module

Module contents