Refer to this FAQ for common questions about AI Evaluations, including purpose, permissions, credit consumption, running evaluations, and interpreting results.
Purpose and scope
- How does AI Evaluation fit into AI strategy?
- Similar to testing code with regression testing, AI Evaluation enables the iterative testing of AI performance. Evaluations can provide scoring through A:B testing to support greater confidence in the integral development of AI Skills and AI Agents to truly improve the operation.
- What is AI Evaluations?
- A built-in quality-assurance tool that scores AI Skill and AI Agent output against criteria such as accuracy, relevance, and factual correctness before agents go to production.
- Does it test Task bots or task automations?
- No. AI Evaluations only evaluates generative AI capabilities (AI Skills and AI Agents), not traditional task bots.
- Is it available to all customers?
- The core feature requires an Enterprise Platform license. The detailed Metrics view is restricted to specific customers. Contact your Automation Anywhere account team for access.
Access and permissions
- What permissions do users need?
-
- View AI Evaluations: Allows users to see scores for assigned folders and AI Skills only.
- Manage AI Evaluations: Required to run evaluations and manage data sets.
Users who run evaluations need both permissions.
- A user cannot see results for a specific AI Skill. Why?
- View access is scoped to folders the user is assigned to. Check folder-level assignments, not just the role permission.
- What is needed to view audit logs?
-
Both permissions must be assigned explicitly:
- View AI governance logs for general information.
- View AI governance log details for full session data.
Credits and costs
- Does every evaluation consume credits?
- No. Only automatic evaluations consume Automation AI Credits. Manual (human-reviewed) evaluations are free.
- What drives credit consumption?
- Token usage (input plus output tokens) drives consumption, not the number of runs. Larger data sets and AI Agent evaluations, that involve multi-step reasoning, cost more than AI Skill evaluations.
- What happens when credits run out?
- A 10% grace buffer prevents mid-run disruption. After that, evaluations will not run. Admins can monitor balances on the Control Room Licenses page.
Running evaluations
- Manual vs. automatic — which should I use?
- Automatic uses LLM-as-a-judge for speed and scale; it works best when you have a data set with expected outputs. Manual lets a human rate each output, which is useful for nuanced review or when expected outputs are unavailable.
- What format does the data set need to be?
- CSV, maximum 100 KB. You can also enter data manually or reuse a previously uploaded data set.
Note: Data set upload is not yet available for AI Agent evaluations.
- I submitted manual ratings but see no final score. What is wrong?
- Refresh the Evaluation page after submitting all ratings. Results are not updated in real time.
Interpreting results
- What do the score thresholds mean?
-
- Above 80% = Good
- 50–79% = Fair
- Below 50% = Poor
Metrics shown in red indicate a failing result. Low numbers always indicate low performance.
- What are the four AI Agent metrics?
-
- Tool Selection (0–100%): Measures whether the correct tools were chosen at each step.
- Goal Completion (Pass/Fail): Indicates whether the task objective was met.
- Trajectory Accuracy (Pass/Fail): Indicates whether the expected step sequence was followed.
- Human Input (0–100%): Measures the percentage of steps that required human intervention.
- Some rows show N/A across all metrics. What does that mean?
- Typically, when N/A displays across all metrics the test case could not be evaluated due to a data validation error. Hover over the error badge on that row for the specific message.
- What if I did not provide expected outputs in my data set?
- NLP metrics (BLEU, ROUGE) cannot compute without expected outputs. The LLM-as-a-judge will fall back to predefined metrics, giving you a less complete picture.