Using Find image in window action

Use the Find image in window action to search for a UI element in an application window (haystack) using a target image (needle). The target image is an existing image that you can use to search for a UI element.

Procedure

To find an image inside an application window, follow these steps:

  1. Double-click or drag the Find image in window action from the Image Recognition package in the Actions palette.
  2. Specify the target image (needle) that you want to find in the application window.
    • Control Room file: Uses an image file that is available on the Control Room.
    • Desktop file: Uses an image file that is available on your device.
    • Variable: Uses a file variable to specify the location of the image file you want to use.
    Note: Images of .jpeg, .jpg, .jpe, .jfif, .bmp, .png, and .gif formats are supported.
  3. Specify the window in which you want to find the source image (haystack):
    Choose from the Application, Browser, or Variable tab.
    OptionDescription
    Application Select from a list of currently active windows. This option shows a list of all the application and browser windows that are open on the Bot Creator device.
    Browser Select from a list of supported browser tabs.
    Note: This option supports Google Chrome, Chromium-based Microsoft Edge, and Internet Explorer browsers. For all other browsers, use the Application option.
    Variable Select an existing window variable to specify the title of the application window title.
    Insert a wildcard character (*) in the Window title field to search for window titles that can change. You can specify one or more wildcards (*) at the start, middle, or end of the value. For example, to perform an operation on any window that has Microsoft in the title, use a wildcard to indicate any string by adding it before or after the value: *Microsoft*. The bot first searches for the exact window title (Microsoft), and if it does not find a match, it searches for windows with the term Microsoft anywhere in the title.

    You can use a regular expression (regex) to identify a window title. Select the regular expression condition as either String or a Pattern and then enter the value. By default, the Window title field is case-sensitive.

    To identify a browser title, you can enable the Case sensitive option to match the letter case, or disable the option to identify a window title where the letter case does not match.
    Note: Ensure that the minimum Bot Agent version installed on your system is the version that was released with Automation 360 v.26. If you have a Bot Agent version released with Automation 360 v.25 or earlier, then by default, the window title will be identified only with matching letter case (case-sensitive).
    For example, to match all window title names beginning with a word followed by - Notepad, you can enter the String value as * -Notepad, or use Pattern value as ((\w)\s)*document-Notepad for only those window titles that match this pattern.
    Note:
    • When you use a regular expression of type String with -title as the input value for a window title in any action, the bot might encounter a run time error. To prevent the run time error, ensure that when you add the string input as *-title, the application with the specific title is open. The asterisk (*) is just a character, not a wild card character.
    • When you use a regular expression of type Pattern with -title as the input value for a window title in any action, the bot might encounter a run time error because the input value *-title is incorrect.

      Ensure that you use valid regex patterns.

  4. Optional: Select the Resize window option to specify the window dimensions.
    This option delivers a more reliable bot. It resizes the window to the dimensions at which the task was recorded, which enhances the ability of the botto identify the target object.
    • If the window is maximized when you record the task, this option is not selected.
    • If the window is not at the maximum size when you record the task, this option is selected, by default, and the width and height fields are automatically filled with the dimensions of the window.
    Note: This option is available only for windows that can be resized. It is not available for the Desktop or Taskbar options.
  5. Click Capture region.
    The selected window appears.
  6. Drag the mouse to select the area and right-click when done.
    The captured area appears in the Preview section with the image coordinates underneath.
  7. In the Wait in ms before capturing the image field, specify the delay (in milliseconds) before searching for the image.
  8. Select the Repeat if image not found check box if you want the system to retry searching for the target image if it is not found.
    1. In the Times field, specify the number of times the system must repeat the process to find the target image.
    2. In the Wait between repeats (ms) field, specify the time period the system must wait before repeating the process of finding the target image.
  9. In the Match percentage field, specify the acceptable percentage of matching pixels between the two images.
    For example, if you specify 20 in the field as the match percentage, the system considers the images as matching even if there is up to 80% of pixel mismatch between the two images.
  10. In the Selected match number field, enter a value to specify the occurrence of the target image on which you want to perform this action.
    You can insert a variable when you do not know the number of times the image might appear on the screen. Ensure you assign variables that support numeric values.
  11. Use the Preview option if you want to select an image that is relative (offset) to the existing target image you are searching on the application screen, or you want to select a specific occurrence if there are multiple occurrences of the target image.
  12. Click Preview.
    If a target image occurs multiple times in the Preview window, each occurrence is highlighted in a rectangular box and displays a number against each occurrence.
  13. Select the specific highlighted occurrence, and you can position your click location relative to an existing image.
    Note: If there are multiple images on an application screen that you are automating and when the target image is not easily searchable; in that case, you can select an easily identifiable image in the captured region and position your click relative to it.
  14. If the target image is found multiple times in the captured region. In that case, you can select the occurrence of a particular target image or select the occurrence number from the drop-down in the Match number (optional) field on the right pane.
    The Match number (optional) and the Target Offset fields on the right pane display the image occurrence you selected and its offset coordinates of the target image.
  15. Click Submit.
  16. In the Target coordinate selections field, the coordinate options Center of match or Offset from match will be selected automatically based on the action you performed in the Preview window.
    • Center of match: The image is selected based on the coordinates at the center of the matched image.
    • Offset from match: The image is selected based on the coordinates relative to that of the matched image.
    Note: The offset coordinates measure the number of pixels from the top left corner of the image.
  17. Select an option from the Action to take on target coordinates list to specify the action you want to perform on the matched image in the application window.
  18. Click Save.