Using Find image in window action
Use the Find image in window action to search for a UI element in an application window (haystack) using a target image (needle). The target image is an existing image that you can use to search for a UI element.
To find an image inside an application window, follow these steps:
- Double-click or drag the Find image in window action from the Image Recognition package in the Actions palette.
Specify the target image (needle) that you want to find in the application
Note: Images of .jpeg, .jpg, .jpe, .jfif, .bmp, .png, and .gif formats are supported.
- Control Room file: Uses an image file that is available on the Control Room.
- Desktop file: Uses an image file that is available on your device.
- Variable: Uses a file variable to specify the location of the image file you want to use.
Specify the window in which you want to find the source image (haystack):
Choose from the Application, Browser, or Variable tab.
Option Description Application Select from a list of currently active windows. This option shows a list of all the application and browser windows that are open on the Bot Creator device. Browser Select from a list of supported browser tabs.Note: This option supports Google Chrome, Chromium-based Microsoft Edge, and Internet Explorer browsers. For all other browsers, use the Application option. Variable Select an existing window variable to specify the title of the application window title.Insert a wildcard character (*) in the Window title field to search for window titles that can change. You can specify one or more wildcards (*) at the start, middle, or end of the value. For example, to perform an operation on any window that has Microsoft in the title, use a wildcard to indicate any string by adding it before or after the value: *Microsoft*. The bot first searches for the exact window title (Microsoft), and if it does not find a match, it searches for windows with the term Microsoft anywhere in the title.
You can use a regular expression (regex) to identify a window title. Select the regular expression either as a String or Pattern for each regular expression condition and enter the value. By default, the Window title field is case-sensitive. You can enable Case insensitive regex flag to identify a window title that is not case-sensitive.
For example, you can enter the String value as ((\w*)-Notepad for all the window title names starting with
((\w*)-Notepad, or use Pattern such as ((\w)\s)*document-Notepad for only those window titles that match this pattern.
Select the Resize window option to specify the window
This option delivers a more reliable bot. It resizes the window to the dimensions at which the task was recorded, which enhances the ability of the botto identify the target object.
Note: This option is available only for windows that can be resized. It is not available for the Desktop or Taskbar options.
- If the window is maximized when you record the task, this option is not selected.
- If the window is not at the maximum size when you record the task, this option is selected, by default, and the width and height fields are automatically filled with the dimensions of the window.
Click Capture region.
The selected window appears.
Drag the mouse to select the area and right-click when done.
The captured area appears in the Preview section with the image coordinates underneath.
- In the Wait in ms before capturing the image field, specify the delay (in milliseconds) before searching for the image.
Select the Repeat if image not
found check box if you want the system to retry searching for
the target image if it is not found.
- In the Times field, specify the number of times the system must repeat the process to find the target image.
- In the Wait between repeats (ms) field, specify the time period the system must wait before repeating the process of finding the target image.
In the Match percentage field, specify the acceptable
percentage of matching pixels between the two images.
For example, if you specify 20 in the field as the match percentage, the system considers the images as matching even if there is up to 80% of pixel mismatch between the two images.
In the Selected match number field, enter a value to
specify the occurrence of the target image on which you want to perform this action.
You can insert a variable when you do not know the number of times the image might appear on the screen. Ensure you assign variables that support numeric values.
- Use the Preview option if you want to select an image that is relative (offset) to the existing target image you are searching on the application screen, or you want to select a specific occurrence if there are multiple occurrences of the target image.
If a target image occurs multiple times in the Preview window, each occurrence is highlighted in a rectangular box and displays a number against each occurrence.
Select the specific highlighted occurrence, and you can position your click
location relative to an existing image.
Note: If there are multiple images on an application screen that you are automating and when the target image is not easily searchable; in that case, you can select an easily identifiable image in the captured region and position your click relative to it.
If the target image is found multiple times in the captured region. In that
case, you can select the occurrence of a particular target image or select the
occurrence number from the drop-down in the Match number
(optional) field on the right pane.
The Match number (optional) and the Target Offset fields on the right pane display the image occurrence you selected and its offset coordinates of the target image.
- Click Submit.
In the Target coordinate selections field, the
coordinate options Center of match or Offset
from match will be selected automatically based on the action you performed in the Preview
Note: The offset coordinates measure the number of pixels from the top left corner of the image.
- Center of match: The image is selected based on the coordinates at the center of the matched image.
- Offset from match: The image is selected based on the coordinates relative to that of the matched image.
- Select an option from the Action to take on target coordinates list to specify the action you want to perform on the matched image in the application window.
- Click Save.