SmartMail Business User Acceptance Testing

This document outlines the recommended User Acceptance Testing (UAT) guidelines for business users to best analyze auto-classification or indexing of emails.

Objective

The purpose of SmartMail Business UAT is for the business experts to test the SmartMail Language Intelligence models before production deployment.

Data Testing

Business experts can guide the model by selecting misclassifications and providing feedback on how the misclassifications should have been classified. Below are some guidelines around this piece of UAT testing.

1. Testing Data Requirements for UAT: We recommend handpicking production data that has been accurately categorized by a business expert. Data should span over every single category. Selected data samples should be new to Ushur and ideally very recent.

2. Data Collection Cycle Length for UAT: Batches of test sample data should be collected over at least one or two weeks to ensure that there is a healthy variety of samples.

3. Number of work hours required for UAT: 3-5 work hours every two weeks will be required. This includes vetting the data samples, preparing performance analysis, and reviewing misclassifications to identify opportunities for improvement of the SmartMail Language Intelligence models.

4. Volume Requirements: The ideal volume per cycle of UAT should be at least 40-50 samples per issue type.

5. Accuracy: Remember that part of the goal of this UAT testing is to review misclassifications to improve the model prior to deployment. If the volume of samples collected is significantly lower than the volume of samples used to determine overall model accuracy, the reports provided by Ushur will not match the accuracy rate for the business UAT testing. This is expected and should not become a blocker for production deployment.

Integration Testing

The business experts should also provide feedback on the end-to-end workflow. Below are some considerations for this aspect of the testing.

1. Incoming Emails: In the UAT area, business experts should test sending emails to the UAT email endpoint once a week.

2. CRM: If the email triggers a CRM case creation, business experts should confirm that email triggers such a case creation.

3. Routing: Business experts should confirm that routing occurs, whether it be to an email inbox or labelling of a case within the CRM.