Manually evaluate an AI Agent - Review each AI Agent output individually and submit a thumbs-up or thumbs-down rating to score agent performance based on human judgment.

Automation 360

Review each AI Agent output individually and submit a thumbs-up or thumbs-down rating to score agent performance based on human judgment.

Before you begin

You must have the Manage Evaluation permission to run and score manual evaluations. Users with View-only access can view completed evaluations but cannot submit feedback or scores.

About this task

Manual evaluation provides a human-in-the-loop method for assessing AI Agents output quality. Unlike automatic evaluation, no NLP metrics or LLM-as-a-judge scoring is applied. You review each test case and submit a thumbs-up (pass) or thumbs-down (fail) rating. The evaluation aggregates these ratings into a summary showing total likes and dislikes across all reviewed outputs. Each result is logged with your name, a timestamp, and the score for audit and trend analysis.

Procedure

  1. Start a new evaluation from one of the following entry points:
    • From the AI Evaluations page, select Run Evaluation.
    • From the AI Agent page, select Evaluate > Run Evaluation. The agent is pre-populated in the next step.
    The evaluation configuration wizard opens.
  2. Select the AI Agent to evaluate.
  3. Click Next.
  4. Under Evaluation Method, select Manual Evaluation.
    Selecting this option configures the evaluation for human review. No automated metrics are generated.
  5. Add your dataset using one of the following options:
    • Select Upload file to enter a name and choose a CSV file (maximum 100 KB).
    • Select Use existing data to select a previously uploaded dataset.
    • Select Enter data manually to provide a name and enter input variables and expected outputs directly.
  6. Click Run evaluation.
    The evaluation saves and begins running. Processing time depends on dataset size. You receive a notification with a link when the evaluation is complete.
  7. Navigate to the completed evaluation via the Evaluations tab or the link in the notification.
    The evaluation summary page opens. The header displays run-level metadata including the AI Agent name and version, evaluation method, date, the user who ran it, the model evaluated, total time, and total tokens. The Output details table lists all test cases with columns for ID, tools used, a preview of the agent output, user feedback status, time taken, and tokens consumed.
  8. In the Output details table, click a row to open its detail view.
    The AI Agent evaluation detail page opens, displaying the full agent output, the final tool call name and status (Success, Failed, or Canceled), the input variables passed to the final tool call, the full tool call sequence table, and the dataset input and expected output variables.
  9. Review the output by examining the following sections:
    1. Read the Agent output to assess the quality and accuracy of the response.
    2. Check Final tool call and Input variables for final tool call to confirm the agent completed the correct action with the expected inputs.
    3. Review the Tool calls table to inspect the sequence of tools the agent invoked, their call types (such as AI agent, API task, Process, Task bot, or Human in the loop), and the inputs passed to each.
      Use the Type filter and Search field above the table to focus on specific call types. Use the pagination controls to navigate to additional tool calls.
    4. Expand the Dataset section to compare the agent input variables and expected output variables against the actual output.
  10. Under User feedback, rate the output by selecting one of the following:
    • Click the thumbs-up icon to indicate the output is acceptable (pass).
    • Click the thumbs-down icon to indicate the output needs improvement (fail).
  11. Click Back to evaluation to return to the Output details table.
  12. Repeat the previous review steps for each remaining output in the Output details table.
  13. After reviewing all outputs, view the aggregated results in the summary panel at the top of the evaluation page.
    The summary displays:
    • Total count of Feedback received (for example, 30 of 45 outputs reviewed)
    • Total thumbs-up count (Likes)
    • Total thumbs-down count (Dislikes)
    • Average time taken
    • Average tokens consumed across all reviewed outputs