Section 5 · Testing, Deployment, and Maintenance · 10% of exam
5.2 Understanding Testing Center Evaluations
Exam objective
Explain how Testing Center evaluations assess routing, action selection, and response quality, and how to interpret test results.
Evaluations answer different questions
An evaluation, or scorer, measures a particular aspect of an agent’s behavior. Routing assertions ask whether the agent selected the expected topic or subagent. Action assertions check whether expected actions were selected or completed according to the test. Response evaluations assess whether the reply addresses the user’s need. Optional quality scorers can measure areas such as completeness, coherence, conciseness, instruction adherence, or latency. The exact names and availability depend on the Testing Center version and test type. Select measures that match the objective of the test rather than enabling every metric without a reason.
Expected responses and LLM judges
Some response evaluations use an LLM judge to compare the agent’s reply with an expected result and assign a score. Other evaluations may return pass/fail assertions for selected subagents or actions. Provide a useful expected response that states the facts or outcome that should be present. Do not turn the expected response into a list of evaluator instructions such as ‘be polite’ or ‘mention the order number’; Salesforce warns that mixing instructions into the expected answer can reduce scoring accuracy. Since generated wording can vary, evaluate the important content and behavior instead of demanding an exact sentence unless exact text is a genuine requirement. Read scores with context
A score is evidence about a test case, not a guarantee of production quality. Look at the scorer definition, the test utterance, expected result, actual response, selected subagent, and actions executed. A low response score may result from missing knowledge or an unclear expected answer. A failed action assertion might mean the agent did not use a required operation, or that the test case was configured with the wrong expected action. Repeated failures across related cases are more informative than a single isolated score. Use details and reasoning in the result panel to form a diagnosis, then manually reproduce the scenario in Builder.
Use evaluations to guide improvement
If routing assertions fail, refine descriptions, scopes, or deterministic transitions. If action assertions fail, inspect action availability, filters, instructions, inputs, and dependencies. If the response lacks accurate information, check grounding and source freshness. If the answer is factually correct but incomplete or inconsistent in tone, improve the relevant instructions and measure the same criterion again. Custom scorers can be useful for organization-specific requirements such as approved language or a defined compliance rule, but their criteria must be clear and tested. Establish acceptance thresholds before a release run, compare results with earlier versions, and investigate recurring gaps rather than optimizing one score in isolation.
TIP: Match the failed scorer to the symptom: routing, action selection, factual response, completeness, instruction adherence, or latency.
Default and optional evaluations vary between the newer Studio Testing Center and legacy Setup Testing Center. Check which scorer definitions apply to the test you are running.
Extra concept: Evaluation scores are most useful alongside the case details and trace; the number alone rarely identifies the root cause.
Official references:
- Choose Evaluations https://help.salesforce.com/s/articleView?id=ai.agent_testing_center_evaluations.htm&language=en_US&type=5
- Use Test Results https://help.salesforce.com/s/articleView?id=ai.agent_testing_center_use_test_results.htm&language=en_US&type=5
- Select Scorers https://help.salesforce.com/s/articleView?id=ai.agent_studio_testing_center_evaluations.htm&language=en_US&type=5
Key takeaways
- Scorers measure distinct behaviors such as routing, action choice, response quality, or latency.
- LLM-judge response scores require a clear expected outcome and should allow acceptable wording variation.
- Diagnose recurring failures with case details and traces, then rerun the affected tests.
