Streamlining operations with UiPath Object Detection model

Introduction

UiPath offers a comprehensive suite of tools and technologies that can significantly enhance various aspects of business operations. By integrating Object Detection, UiPath Apps, Data Services, Twilio, and Druid AI chatbot, organizations can create a streamlined and automated system to optimize their processes.

UiPath AI Center™ Object Detection model, powered by artificial intelligence (AI), is designed to identify and classify objects within images. When integrated with UiPath Apps, this model can be used to process images and extract valuable information. Leveraging data services, organizations can provide seamless data integration and management, further enhancing the capabilities of the system.

In this setup, the Druid AI chatbot can play a pivotal role. Through its advanced natural language processing capabilities, it can effectively communicate with users, understand their requests, and provide relevant responses. By integrating Twilio, the chatbot can extend its reach by sending notifications and updates via SMS, enhancing communication and user engagement.

Concepts overview

Together, these technologies enable the creation of a powerful end-to-end solution. This solution can be customized to various business contexts, such as customer support, data analysis, insurance settlement, and more. By automating processes, improving communication, and harnessing the capabilities of AI, organizations can achieve higher efficiency, accuracy, and overall improved operational workflows. Let's delve into the distinct components of this comprehensive use case.

UiPath Object Detection model is a technology that empowers systems to recognize and categorize objects within images. This model can enable businesses to automate complex tasks and optimize resource allocation.

UiPath Apps is a versatile platform that serves as the gateway for launching automated operations. Customers can seamlessly upload images of interest, and through the harmonious integration of UiPath Object Detection model, data processing is streamlined.

UiPath Data Service is the backbone of this automation journey. It can work tirelessly behind the scenes. Through orchestrated interactions with UiPath Apps and the Object Detection model, data transformation and analysis unfold seamlessly.

Twilio is an integral communication tool that provides a seamless flow of information. By facilitating the exchange of estimated repair costs between customers and insurance helpdesks, Twilio elevates the user experience.

Druid AI chatbot is powered by advanced AI and takes user interaction to new heights. Engaging customers effortlessly; it facilitates submitting repair cost estimates to insurance helpdesks, opening a fluid approval process.

How to train the Object Detection model on your own data

Here we explore the steps involved in deploying UiPath Object Detection model in UiPath AI Center™. We use "Accident Vehicle Claim Processing" use case as an example to walk through this process. Users can input accident images, which are processed by the ML model to detect and identify various vehicle damage, such as CarFrontMajorDamage, CarBackWindDamage and so on.

Step 1: Collect and prepare dataset for training and testing.

Data plays a crucial role in the entire process, as the accuracy of the machine learning model heavily relies on the quality and relevance of the data used during full pipelines. The success of the model hinges upon the data it learns from.

Collecting the dataset

The UiPath Object Detection model comes pre-trained on extensive datasets like Common Objects in Context (COCO). However, for our specific use case of Damage Detection, it's essential to train the model using a curated dataset focused on such incidents. Acquiring the right dataset from various sources, both free and paid, will be vital for the training process.

Step 2: Annotate accident vehicle images

To begin, acquire a dataset of vehicle images involved in accidents. Afterwards, proceed to download the LabelImg tool by accessing the official documentation provided by UiPath. Once the package has been successfully downloaded, install it using Python. Next, open the application through the Windows Command Prompt and proceed to annotate the images depicting damages. Check out about Object Detection model ‌and read more on GitHub.

Step 3: Choose an ML model and train It using a dataset of annotated images.

Begin by creating a project within AI Center™. Next, select the Object Detection model to establish the pipeline for the dataset that you have already prepared. It's crucial to provide that the dataset includes the respective .xml file for each image, as the training pipeline may fail otherwise.

The.XML file contains comprehensive details of images, including information such as filename, file path, height, width, depth, and annotation name.

Upload dataset folder to the AI Center™ for creating pipeline

Create full pipeline

The optimal configuration for an effective UiPath Object Detection model entails the establishment of a full pipeline. This requires the provisioning of a dataset in a specific ratio to derive accurate confidence scores. Training dataset - consisting of 500 annotated images, encompassing 80% of the entire dataset.

Evaluation dataset - Comprising 50 annotated images, equivalent to 20% of the overall dataset.

It is imperative to emphasize that the images within the evaluation dataset mustn't be drawn from the training dataset to ensure the integrity and reliability of the evaluation process. This practice safeguards against potential biases and guarantees a robust assessment of the model's performance.

Once the full pipeline demonstrates a high level of confidence and accuracy, it can be considered for deployment as the final version of the machine learning Model. This version can then be used to create the ML skill, ready for integration into the intended application or system.

Step 4: Create an ML skill and use it in UiPath Studio to process data After the successful completion of the training pipeline, an ML skill should be created using the trained ML package. This ML skill can be integrated into UiPath Studio to process images of vehicles.

Create a project in UiPath Studio Once the ML skill is prepared, proceed to create a project in UiPath Studio and install the necessary package. Thoroughly test the workflow and finally push it to UiPath Orchestrator to start the process from within the Apps interface. Packages required

• UiPath.MLServices.Activities

• UiPath.WebAPI.Activities

• UiPathTeam.EncodeDecodeActivities.Activities

Step 5: Create entity in Data Service to store the customer's records

To efficiently store customer records, using a reliable database is essential. UiPath Data Service emerges as an optimal choice for storing customer records and seamlessly integrating them with UiPath to ensure swift and responsive data retrieval. Upon logging into their respective accounts, customers will be able to conveniently search and retrieve their policy details through the Apps.

Step 6: Create storage bucket to store images

To securely store customer-uploaded images, we can use a storage bucket, such as UiPath storage bucket, which provides a reliable and secure environment for downloading and uploading files. UiPath storage bucket is renowned for its high level of security and reliability, making it an excellent choice for storing various types of files.

How to build a front-end app with UiPath Apps Step 1: Design a Front-End with UiPath Apps

Design a UiPath application that allows customers to input images of damaged vehicles. The application will process the images using a machine learning model and display the results to the customer. The application will facilitate customer interaction with the backend system and provide a user-friendly interface for displaying the response.

AppsPage1 : Home

The home page of the application provides users with the ability to log into their accounts and connect with an AI chatbot. Users can also obtain instant quotations, automatically renew their insurance, and efficiently settle claims for their damaged vehicles. Additional options can be incorporated as per the company's specific requirements.

AppsPage2 : AI chatbot

To enable ‌company-specific response, the Druid AI chatbot must be trained using appropriate data. Once trained, the AI chatbot can be seamlessly integrated into applications to address customer queries promptly.

AppsPage3 : Get instant quotation

To obtain an instant quotation for their vehicle, customers are required to upload the latest picture of the vehicle and provide certain personal details. Afterward, they can submit the information by clicking on the specified button. In the background, a UiPath ML modell will process the vehicle's image and accurately determine whether it's sustained any damage or remains undamaged. Based on this assessment, the robot will promptly send the estimated quotation to both the customer and the insurer. The estimation will be conducted according to the predefined rules established by the insurer. Afterwards, customers have the option to respond to the quotation via email, and the robot will promptly relay their response to the relevant team for further approval.

AppsPage4 : Search policy details

To initiate the claim processing, the customer should first search for their policy details and verify the policy status to determine whether it is currently active or expired. If the policy has expired, the customer should navigate back to the home page and ask for the renewal of their existing policy. Conversely, if the policy is active, they may proceed with the next steps to process their damage claim.

AppsPage5 : Upload most recent picture

To start the claim settlement process, customers are required to upload the most recent photograph of the damaged vehicle using the specified upload section. Afterward, they should proceed by clicking on the "Estimate Damage Cost" button.

AppsPage6 : Estimated damaged cost

After the uploaded photograph is processed by the ML model, the predefined rules will calculate the cost of damages and display the result on the same screen, providing a detailed breakdown of the offered amount to the customer for their damaged vehicle. If the customer wishes to receive the breakdown on their registered mobile number, they can input their mobile number and send it as a text message using Twilio integration. Additionally, if the customer wants to submit it for approval, they can send it to the appropriate department for further action.

AppsPage7 : Estimation breakup 

AppsPage8 : Send SMS 

Conclusion 

In this article, we explored the usage of UiPath AI Center™ in streamlining the business process through the implementation of an Object Detection ML model. If you want to keep yourself updated with other use cases, deep dive into UiPath Community Forum in the use case repository.