Generative Recorder

When web applications undergo changes or have dynamic UI elements, the bots automating them might not work reliably. The Recorder actions could fail because they might not find the UI elements at runtime. With the new Generative Recorder fallback mechanism, you can automate such dynamic web pages more reliably.

Note: Generative AI models can produce errors and/or misrepresent the information they generate. It is advisable to verify the accuracy, reliability, and completeness of the content generated by the AI model.

See also Generative Recorder Product Club.

Architecture

The Generative Recorder architecture diagram illustrates the system's framework, highlighting the integration of generative AI for resilient UI automation.

Generative Recorder architecture

Data flow

The Generative Recorder flow diagram outlines the sequence of operations, showcasing how generative AI is employed to detect and adapt to interface changes, ensuring continuous and resilient automation.

Generative Recorder data flow

Availability

An Automator AI Platorm Base license is required to enable the Generative Recorder capability. A license is available to all Cloud regions. When the license is assigned, the Generative Recorder capability will be available to all users with the following roles and device license:
  • Role: AAE_Basic and AAE_Bot Developer
  • Device License: Bot Creator or Citizen Developer
Note:
  • This feature is currently available only with AWS.
  • Ensure that the minimum Bot Agent version and the Control Room version on your system is the version that was released with Automation 360 v.32.
  • This feature is currently supported only for HTML technology in Google Chrome and Microsoft Edge browsers.
  • Ensure that you use Bot compatibility version 6 to leverage Generative AI and Native fallbacks for enhanced automation resiliency. If you set your Bot compatibility version to 5, use any of the following conditions in your automation, and run it in Debug mode, you might encounter issues while updating the DOMXPath value to the recommended fallback value during bot run whenever a successful fallback occurs:

    Recorder conditions such as Object exists and Object does not exist in If, Else if, Wait for condition, and Loop while actions.

Supportability

Recorder Native and Generative AI-based fallback also supports the Object exists and Object does not exist conditions in the following actions:
  • If action
  • Loop > While action
  • Wait > Wait for condition action
Note:
  • When you select the Advanced time out > Stop the bot and display an error message option, the bot will not display an error message.
  • This capability is supported for Google Chrome and Microsoft Edge browsers.
  • Generative Recorder supports resiliency when there is an anchor UI element to fall back on, and when that UI element label appears only one time on the page.

Benefits of generative AI

  • Fallback: When Recorder is not able to automate a web UI element, leverage generative AI and native fallbacks to get the latest UI element details and automate.
  • Secure: The data specific to the UI element is used only after redaction and anonymization. Business application data that is not related to the automated UI element is not used.
  • Validate: Process generative AI response through another layer of validation to help improve fallback accuracy.
  • Optional: If you do not want to use the fallback option, you can opt out.
  • Traceable: Citizen developer, Professional developers, and Automation admins will be notified on the successful use of the generative AI fallback.

Benefits of Generative Recorder

Fallback notification pop-ups for automation

When a fallback occurs in native, text, or vision-based automation, the fallback notification pop-up enables you to quickly identify and manage dynamic properties, streamlining troubleshooting and enhancing your automation experience.

Dynamic properties shown in the fallback pop-up

Following are some of the key features of this fallback pop-up:
  • Visibility of dynamic properties: All dynamic properties related to the action are now displayed in the fallback pop-up. This enables you to view which properties have changed and could potentially cause the automation to fail.
    Note: The fallback popup displays dynamic properties from the selected object properties, with the exception of DOMXPath which is always shown even if it is unchecked or modified.
  • Copying values: You can copy the New value of any property directly from the pop-up and paste it elsewhere, such as in Notepad.
  • Editable and read-only properties:
    • To update the dynamic properties, ensure you have selected them using the check box and then click the Update properties option. You can undo or redo your actions before saving your automation.
    • When you deselect a specific property, it becomes read-only, but you can still copy its value.
  • Saving and executing again: After you save the new values and run the automation again, fallback is not triggered for those properties unless they change again. This prevents repeated fallbacks and streamlines your automation runs.
  • DOMXPath property: The DOMXPath property is always selected and set to read-only by default, as it must be included in the Object properties, regardless of the scenario.
  • Pop-up display logic: If the saved value and the new value for a property are the same, that property is not shown in the fallback pop-up. Only properties with changes are shown to reduce clutter.
  • Updating object properties: The pop-up only allows you to update the object properties. It does not let you add or remove properties from the criteria and must be done manually. The check box is solely for updating values of existing properties.
  • Value input: Always enter plain text values in the value field; do not use variables.
  • Run mode vs. Debug mode: In the Run mode, you can copy and paste property values, but you cannot update properties directly as you can in Debug mode.

These fallback pop-ups make it easier to identify, update, and manage dynamic properties, reducing repeated fallback interruptions and improving the reliability of your automations across native, text, and vision-based scenarios.

Resources