Understand how AI Evaluations consume credits based on token usage across AI Skills and AI Agents evaluation scenarios.

Licensing model for AI Evaluations

AI Evaluations consume Automation AI Credits, leveraging the same underlying credit framework used by AI Agents. However, consumption for Evaluations is driven by evaluation metrics and execution scope, rather than by agent task execution.

  • Enterprise platform license includes a base pool of Automation AI Credits.

  • These credits are shared across AI-powered services, including AI Agents, AI Skills, and AI Evaluations.

Automation AI Credits

Automation AI Credits are deducted when an AI Evaluations runs successfully. AI Evaluations consume credits based on:
  • AI Evaluations consume AI credits when automatic evaluations are selected (no consumption for manual evaluation).
  • Tokens consumed for evaluating AI Agents when executed from public workspace.
  • For AI Skills, additional tokens are consumed when out of box models are used.

  • The credit consumption depends on the size of the data set.

Each evaluation can trigger multiple model calls and agent executions depending on the size of the data set, which directly impacts the credit consumption.

Key interpretation

  • Token-based billing model:

    AI Evaluations consumption is driven entirely by token usage (input + output) rather than execution count.

  • Cost hierarchy:

    • AI Skills < AI Agents

    • Input tokens < Output tokens

  • Why the difference:

    • AI Skills evaluations typically involve single-step model responses

    • AI Agents evaluations involve multi-step reasoning, tool use, and orchestration, increasing token utilization and cost.

  • Practical implication:

    The total credit consumption for an evaluation can be estimated as:

    Total Credits = 
  (Input Tokens × Input Rate) + 
  (Output Tokens × Output Rate)

    Rates vary by evaluation type (Skills vs Agents).

  • Also consumes credits for Agent executions when executed from public workspace, and credits consumed for using out of the box model connections. This consumption varies based on the size of the dataset.

Grace policy

If purchased credits are exhausted, up to 10% additional credits can be consumed as a grace buffer to prevent disruption of evaluation workflows.

Visibility

Administrators can monitor evaluation-related credit usage in the Control Room Licenses page, including:

  • Total purchased credits

  • Used credits

  • Remaining balance

  • Grace usage

Evaluation consumption rules
Charged (credits deducted) Not charged (no deduction)
AI Evaluations charged for both public and private when using automatic evalution. Evaluations are not charged from public and private when using manual evalution.
All successful evaluations. Failed evaluations.