iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Evals

By · Reviewed & last updated · Part of LangChain Tutorial

LLM evals are how you keep a chain honest as prompts, models, and tools change. The reliable pattern is: a fixed dataset of inputs with reference outputs (or graded rubrics), an evaluator that scores each prediction, and a regression gate in CI. LangSmith hosts this end-to-end, but the shape applies to any framework.

Run an evaluator over a small dataset

EXAMPLE
from langsmith import Client
from langsmith.evaluation import evaluate
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

client = Client()

# 1) Define or upload a dataset (one-time setup)
dataset_name = 'support-faq-v1'
examples = [
    ('How do I reset my password?',
     'Use the Forgot password link on the sign-in page.'),
    ('What are your support hours?',
     'Support is available Monday to Friday, 9am to 6pm AEST.'),
]
if not client.has_dataset(dataset_name=dataset_name):
    ds = client.create_dataset(dataset_name=dataset_name)
    for q, a in examples:
        client.create_example(inputs={'question': q},
                              outputs={'answer': a},
                              dataset_id=ds.id)

# 2) The chain under test
llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer the support question in one sentence.'),
    ('human', '{question}'),
])
chain = prompt | llm

def predict(inputs: dict) -> dict:
    msg = chain.invoke(inputs)
    return {'answer': msg.content}

# 3) Evaluators
def exact_match(run, example):
    return {'key': 'exact', 'score': int(
        run.outputs['answer'].strip().lower()
        == example.outputs['answer'].strip().lower())}

def contains_key_phrase(run, example):
    ref = example.outputs['answer'].lower()
    pred = run.outputs['answer'].lower()
    return {'key': 'overlap', 'score': int(any(
        w in pred for w in ref.split() if len(w) > 4))}

results = evaluate(predict,
    data=dataset_name,
    evaluators=[exact_match, contains_key_phrase],
    experiment_prefix='faq-eval')

print('See results in LangSmith UI.')

Why it matters

Cheap deterministic evaluators (exact match, contains) catch regressions fast and cost nothing per run. Use LLM-as-judge sparingly — for nuanced quality where rules cannot encode the rubric — because it adds latency, cost, and its own variance.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# Use LangSmith datasets:
# 1) Capture good answers
# 2) Define evaluators (string match, LLM-as-judge)
# 3) Compare model versions / prompts side-by-side
Try it Yourself »

Discussion

Loading…