How to use Screen Scraping - the basics
UiPath has pioneered the screen scraping of running desktop apps with 100% accuracy in under 16 milliseconds. Prior to UiPath, screen scraping was a synonym for slow, fragile, inaccurate OCR on the display, and had a low rate of accuracy.
UiPath features an innovative technique for extracting text from running apps, even if they are hidden or covered by another app.
Let's learn how to use the various screen scraping techniques.
The aim of this tutorial is to provide you with the best and fastest way to achieve 100% accuracy in scraping any desktop application.
Scraping a single element in a desktop application using Get Text
Let's take ExpenseIT's sample application as an example.
We will try to extract the total expense amount. The number 277 is in image format and cannot be selected through a simple mouse click. So the regular cursor Select Activity won't work with this one.
To scrape this data, we can use Get Text. This allows you to get the text value of the specified single UI object. To use this Activity, the first option is to simply search for this Activity in the Activities panel and drag and drop it in your Workflow.
The second option is by using the Macro Recorder.
1. Start by pulling up the Macro Recorder from the Standard Toolbar and select Desktop.
2. Once the Recorder is ready, select Copy and click Copy Text.
3. This will activate the Recording mode with a blue hand cursor. Mouse hover to the number 277 which will be highlighted automatically then click on it.
4. That's it! You can now save the recording by clicking Save & Exit. This will automatically create a recording sequence.
This technique is 100% accurate and will work in any desktop application.
Scraping using the Built-in Screen Scraping Tool
This method is very effective when dealing complex UIs. The Screen Scraping Tool can handle both individual text elements, groups of text and blocks of text.
One good example is when you are trying to scrape text in table format in an application.
The Screen Scraping Tool automatically detects regions on the screen. The active region will be processed once it is selected. This will open the Screen Scraper wizard to save the activity.
Let's say you want to scrape a block of data from the same application.
1. Click on the Screen Scraping button from the Design menu.
2. Highlight the full application window and click on it. What you are trying to do here is scrape all the possible text. This works with individual elements as well. But in this case, to capture the text in the tables, the whole window needs to be selected.
3. This will pull up the Screen Scraping window with scraped data. The default scraping method will automatically be selected depending on the application you are trying to scrape.
How to Choose the right Scraping Method
The Screen Scraping tool will automatically select the most appropriate scraping method for a particular application. Full Text is currently set, as shown on top of the wizard window.
There are 3 different methods we can use in scraping text:
Full Text - allows you to scrape all visible text on the UI object.
Native - aside from scraping the text, it also captures the position of the text and other text properties such as font style and colour. Think of it like your clipboard doing the copy-paste for you.
OCR - OCR should be used when scraping virtual desktops and Citrix Applications.
4. Click the Continue button to save the scraping activity.
5. That's it. The new sequence will be added to your Workflow. This sequence includes the scraping activity being used in this sequence. In the example above, it uses the Get Full Text Activity.
How to scrape text with hidden elements
UiPath's Scraping Tool is capable of scraping both visible and non-visible UI object. Please note that this will only work using the Native method of screen scraping. This can only work in native Windows application like Windows Command prompt or terminal applications.
To scrape the full text from a terminal window, follow the following steps:
1. Click on Screen Scraping button from the Design Menu.
2. Highlight the full application window. Click on it.
3. This will pull up the Screen Scraping window with the scraped data. The Native scraping method will automatically be selected.
4. This will show all the possible text inside the command prompt including the special characters. Hit once done.
5. That's it. An appropriate Activity will be created based on Native Scraping.
Screen Scraping with Citrix Applications using OCR
The built-in screen scraper will not work with Citrix Apps since they are running in a Virtual Machine. A different approach will be used for scraping text from of a Citrix Application. We will be using the OCR screen scraping method.
Let's say the same application is running in a Virtual Machine. We will scrape the total expense inside this virtual desktop.
1. From the Design menu, click on Record -> Citrix.
2. Once the Recorder is pulled up, click on Screen Scraping -> Scrape Relative.
3. One the Recording mode is active, highlight the "Total Expenses Region" and then indicate the region near it that covers the number
4. This will automatically run the OCR and will capture the text being highlighted.
5. Click Continue and you are done.
It's a simple yet a powerful technique to capture text from images.