Continuous annotation to improve data quality

Continuous annotation to improve data quality

Let’s say in a bid to improve your model, You have searched for optimal hyperparameters, regularized and even tested several architectures, but yet model performance is still not satisfactory.

Because Machine learning solutions are cascading in nature, the quality of the training data affects model performance. How can I check errors in a training dataset labelled by experts? I don’t even have domain knowledge, and that is twice the cost in time? Why should I hire a new set of annotators to verify initial my annotation? Well, you don’t have to do this, You can use an algorithm if you have extra computing, if use case is worth the extra compute. This algorithm is confident learning(implementation by clean lab). Take a leaf from OpenAI’s book and clean your initial dataset to have a high-fidelity training dataset

Further on, the process of continuously identifying label issues in order to improve model quality is called continuous annotation.

The state-of-the-art method applied is called confidence learning, which is an algorithm which learns a joint distribution using originally labelled data and out-of-sample predicted probabilities(Probabilities obtained by training a model using cross-validation).

Although computationally intensive, this algorithm smartly identifies issues with apriori labelled data, by returning indices of possible corrupted labels. In addition to human-in-the-loop training to correct these suspicious annotations, a model’s performance can be improved with this method. Tools that can be used are Labelstudio, for annotation and cleanlab to identify quality issues.