Enhance your AI Governance dashboard experience by reviewing data for Top Automations using AI, Most used models, Gen AI model calls, Model utilization, Token consumption, and Token consumption details widgets to view, monitor, and analyze AI data derived from model interactions during automation development and automations executions across your enterprise.

Permissions

GRC Leads, Automation Leads, and Pro Developers would use the AI Governance dashboard via these settings that needs to be enabled by the Automation Admin:
  • Roles > AI > View AI dashboards
  • Roles > AI > View AI prompt logs > View AI prompt log details
  • Roles > AI > View AI event logs > View AI event log details
Note:
Enabling the View AI dashboards setting displays the AI Governance dashboard. To further drill-down and access detailed logs in the AI prompt log and Event log tabs, these additional settings must be enabled:
  • AI > View AI prompt logs > View AI prompt details
  • AI > View AI event logs > View AI event details

See Assign roles and permissions to enable AI Governance.

Top Automations using AI widget

This dashboard widget displays a bar graph showing the top ten frequently run automations that uses AI interactions. You can see the name of the top ten automations and their number of executions in the Last 24 hours. Hovering over an automation bar in the widget shows you a blurb with the automation name and the total number of execution runs.

Click a bar graph to drill-down and view additional details. Clicking an automation bar navigates you to the AI > AI governance > AI prompt log tab in the Control Room where you can view a list of all the executions the automation ran listed by sessions. Clicking each sessions displays the details. You can change the time filter in this screen.

See, AI prompt log.

Note: You will not see the prompt-text, prompt-response, and model parameter setting data in the automation execution session details if the Pro Developer has disabled the Data logging toggle for a specific AI Skill from the AI Skills screen.

Most used models widget

The GRC Lead and Automation Lead would use this widget to monitor and track the Model utilization for the Most used models within automations to effectively manage subscriptions. This will help them implement the use of approved and supported models from the hyperscaler vendors.

The Most used models widget gives visibility into the most popularly used top five models within automations using the Generative AI Package and AI Skills Packageto execute automations in the Production environment.

Let us review the details of the Most used models widget. The data is represented in a bar format and is sorted by the automation name, hyperscaler vendor model name, package name, and their usage. By default, the Most used models widget shows data for the Last 24 hours. Each bar represents the following:
  • The automation name with the package used, whether it used the Generative AI Packages or the AI Skills package
  • The model used from the hyperscaler vendor, for example: GPT-4 from OpenAI

Hover over a bar to see a summary of the details.

Clicking on a bar graph navigates you to the AI > AI governance > Event log tab in the Control Room where you can view a list of logs filtered by the model name. This action performs an exact search (for example, "=GPT-3.5 (Turbo)") of that model and displays the associated event logs, categorized by Event type. The logs are displayed by Event type. Click an Event type to view additional details for the model interactions that took place within the automation execution.

See Event log.

  1. In addition to the drill-down option from the bar graph, click the ellipses at the top-right corner to select View details for a detail view.
  2. The AI Governance dashboard screen displays the Model utilization tab details.

GenAI model calls widget

The GenAI Model Calls widget provides a view of your organization's GenAI model usage. This dashboard displays the top 5 most frequently called GenAI models, based on the volume of prompts executed. It specifically tracks calls made by automations running in public workspaces, encompassing both successful and failed runs, as well as attended and unattended executions.

The widget excludes connection and authentication calls, focusing solely on prompt execution within GenAI command actions and AI Skill command actions. Importantly, if a command action is within a loop, each prompt execution within that loop counts as a separate model call. You can select a time window to analyze model usage trends. Clicking a model name within the widget takes you to the AI Governance Event Log, pre-filtered to show all events associated with that specific model, enabling deeper analysis and troubleshooting.

Model utilization widget

The Model utilization widget displays the top five automations. By default, it shows data for the Last 24 hours which can be changed to a different time filter at the top of the screen.

Clicking on a model name navigates you to the AI > AI governance > Event log tab in the Control Room where you can view a list of logs filtered by the model name. This action performs an exact search (for example, "=GPT-3.5 (Turbo)") of that model and displays the associated event logs, categorized by Event type. The logs are displayed by Event type. Click an Event type to view additional details for the model interactions that took place within the automation execution.

Export reports based on the data represented in the widget by clicking Export items to .csv.

When you click View more to see all the model utilization, the Model utilization details screen for Most used modelsappears. The following table provides details of each of the columns:

Table 1. Model utilization details screen for Most used models
Column header Description
Model name This is the model used in a specific automation such as GPT-4 from OpenAI. You can click the model to go to AI > AI governance > Event log tab in the Control Room where you can view all logs filtered by the model name.

See Event log.
Publisher Supported hyperscaler vendor such as Amazon Bedrock, Google Vertex AI, Azure OpenAI, or OpenAI.
Automations Number of generative AIcommand actions used in automation executions using the Generative AI Package and AI Skills Package.
AI Skills Number of calls automations executed using AI Skills.
Model calls Number of calls made to the model.
Devices Number of devices used during the model interaction.

Token consumption widget

The Token consumption widget provides visibility into your top automations consuming most number of tokens aggregated per model. To derive the total count for a model's Token consumption, the metrics is calculated based on the Automation name, Folder path, and the model name from the hyperscaler vendor. Hover over a bar in the widget to see the Token consumption details of a specific automation.

By default, the Token consumption widget shows data for the Last 24 hours which can be changed to a different time filter at the top of the screen.

The GRC leads and automation leads would use this widget to monitor and track the token usage of the models during their interactions and execution within automations to effectively manage cost and subscription. This will help them project Token consumption based on model preference.

Token consumption details widget

The Token consumption details widget displays the top five automations. By default, it shows data for the Last 24 hours which can be changed to a different time filter at the top of the screen.

Note: You can filter the Token consumption details table based on automation name, folder path, publisher, model name, or workspace. To search the exact phrase, enclose the search phrase within double quotes (for example, "Amazon Bedrock")

Export reports based on the data represented in the widget by clicking Export items to .csv.

When you click View more to see all the token consumption, the Token consumption details screenappears. The following table provides details of each of the columns:

Table 2. Token consumption details screen
Column header Description
Automation name Name of the automation that was executed. You can click the name to go to AI > AI governance > AI prompt log tab in the Control Room where you can view all logs filtered by the automation name.

See AI prompt log.
Folder path The folder location of the automation from where it was executed.
Publisher Supported hyperscaler vendor such as Amazon Bedrock, Google Vertex AI, Azure OpenAI, or OpenAI.
Model name This is the model used in a specific automation such as GPT-4 from OpenAI.
Users Number of users.
Workspace This can be the Private or Public folder from where the automation was executed.
AI Skills Number of AI Skills used in the automation.
Tokens Total token count used by the model.