Maintenance

Once the solution is deployed, periodic audits should be performed of the auto classified categories for any errors in the classification. The results of the audit should be collated and used to retrain the model with accurate labels. This will ensure that the natural change that occurs in human language syntax or use over time is addressed. This phase is described in depth in Continuous Learning Loop in Operations.

This is the post-production activity, which should happen regularly to update the Ushur LI engine to fine-tune the classification or update any process change. This involves steps such as:

• Generate Confusion Matrix

• Assess Misclassification 

• Relabel

• Training data

Generate Confusion Matrix

The following is the sample of our Confusion Matrix. The horizontal bar shows the Ushur AI’s classification (predicted label)  and  the vertical bar shows the True label provided in the test data. The value that appears on the coincidence of true, as opposed to, predicted label is the correctly classified content and the rest of the contents are misclassified.

Assess Misclassification

There could be some topics that are not performing up to the mark and might need modification. Understanding the success or failure rate from the confusion matrix can help you identify the topics that need attention to prepare data for the next step. 

Relabel

This is the part where any previously mislabeled or addition of new labeling happens. Based on the confusion matrix and previous step (Assessing Misclassification), data must be tweaked for better results.  

Training Data

After relabeling, this is the stage where the packaging of the data starts. We prepare a new .csv file with complete data for training and pass the file to Ushur for retraining the LI model. There are multiple methods of training, which can be referred from our SmartMail Integration Guide.