Section 6 · Governance and Observability · 10% of exam
6.4 Using Observations to Guide Improvements
Exam objective
Describe a repeatable process for turning agent analytics, feedback, and session observations into validated improvements.
Start with an outcome and evidence
An observation becomes useful when it is tied to a defined outcome, such as resolving a task accurately, respecting a verification rule, reducing avoidable escalation, or responding within an acceptable time. Establish a baseline for the relevant agent, channel, and intents. Combine aggregate analytics with session transcripts, trace details, evaluation scores, user feedback, and release history. A metric alone may show where to look but not why performance changed. Confirm that the observation is representative and that the data source covers the sessions in question. Note any changes to models, data, instructions, channels, or permissions during the period. Translate findings into a change
Classify the issue before choosing a remedy. A knowledge gap may need an authoritative source update or retrieval adjustment. Incorrect routing may need clearer classification descriptions or topic scope. A missed action may require better instructions, a variable mapping, filter logic, or action configuration. A security issue may require permission or authentication changes, not just a prompt edit. A latency issue may call for trace analysis of a slow step. Keep the proposed change focused and record the expected effect. If the evidence does not identify a cause, gather more sessions or reproduce the behavior before editing production.
Validate before and after release
Use a sandbox or controlled testing environment to validate the proposed change. Add the representative failed interaction and neighboring scenarios to Testing Center. Rerun the test cases that measure the desired outcome and check that other important paths still work. Review the detailed trace and action results, not just an aggregate score. Promote the approved change through the organization’s release process, activate the intended agent version, and confirm the target environment has the needed data, permissions, and connections. After deployment, compare the same measures against the baseline and look for unintended side effects.
Build a continuous improvement loop
A useful monitoring plan defines what to watch, how often to review it, who owns each signal, and what action follows a threshold or trend. For example, a sustained rise in failed tasks might trigger session review; repeated low-quality answers for one intent might trigger a knowledge or instruction review; a trust signal might require immediate governance review. Add resolved incidents to the regression suite and update operational documentation. Close the loop by checking whether the change improved the intended outcome and whether users still encounter the issue. Avoid optimizing one metric at the expense of safety, accuracy, or a good handoff to a human.
Use a closed loop: observe a pattern, confirm it in session details, identify the responsible layer, test a focused fix, deploy it, and measure the result.
The data collected for monitoring can contain sensitive content. Maintain access controls and data masking while preserving enough context for authorized investigation.
Extra concept: A practical success measure is defined before the fix; choosing a threshold only after seeing results makes it harder to judge improvement objectively.
Official references:
- Build an Agent Testing and Observability Plan https://help.salesforce.com/s/articleView?id=ai.agent_test_prep.htm&language=en_US&type=5
- Observe Phase Observability Tools https://help.salesforce.com/s/articleView?id=ai.agentforce_observability_observe_phase_tools.htm&language=en_US&type=5
- Set Up Agentforce Session Tracing https://help.salesforce.com/s/articleView?id=ai.generative_ai_session_trace_setup.htm&language=en_US&type=5
Key takeaways
- Connect observations to defined outcomes and verify the evidence before changing configuration.
- Test targeted fixes and related paths before release, then compare production behavior with a baseline.
- Assign owners and follow-up actions to important monitoring thresholds and recurring patterns.
