Role of Data in the lifecycle of a machine learning model

Data Quality

Data Quality refers to the ratio of signal to noise in a training dataset. Is this data pure? Are the data points and corresponding labels accurate? What is the fidelity of this dataset.

Noise in a dataset takes many form but ultimately reduces the quality of a dataset. Sometimes it can be a positive thing, as good model performance on such dataset corresponds to a certain degree of model robustness which will be further discussed in this article. Other times, when there is a lot of noise in a dataset, it has a negative effect on the trained machine learning model. Neural networks seem to be the most sensitive of machine models possibly due to the intricate, and meticulous nature of the deep learning.

Depending on the machine learning task at hand, noise can be introduced to input data in many forms. For NLP tasks, it can arise from input variables e.g misspellings in text, mix in languages, poor preprocessing, or input tags such as wrong entity tags for NER such as Cities being tagged as names, presence of rare words not present in a tokenizer’s vocabulary leading it to be replaced it <unk>. For Image processing tasks, this can be blurred images, poorly exposed images, bounding boxes which encapsulate multiple objects for object detection tasks. This list is not exhaustive as forms of noise in input data differ from task to task.

It can also arise from labels as in wrong and incomplete annotations. Lack of standardisation, poor preprocessing, and missing data are also common reasons for poor quality in data and anomalies in a training dataset. Logically, more data reduces the impact of noise while training neural nets, but in a real life application where annotations can become expensive, we can’t always have access to a lot of data therefore it is imperative that the smaller the dataset, the higher the quality should be.

In cases where there are few or no labelled data prior to model training. Techniques such as domain adaptation and unsupervised self-training with pre-trained models may be employed otherwise extra annotation is also a plausible option. Others options which can be explored is Data Augmentation — libraries like NlpAug, TextAttack can be used for token perturbation in NLP while libraries like PIL or OpenCV for image flipping, adding noise, blurring for image processing tasks.

Inter Annotator Agreement

Exploring extra annotation means there are couple of considerations to be made. A major one is the domain expertise of the annotator. This is because all model training and tasks downstream of this depend on the quality of the annotation.

There are several mitigating checks to ensure the veracity of annotation, e.g use of multiple annotators — a super annotator is the gold standard and other annotators as predictions. Metrics such as cohen’s kappa/fleiss kappa and IoU(Intersection over Union) measure inter-annotation agreement for varying tasks and penalise agreement by chance. These metrics utilise probabilities of annotation to verify that annotators are not selecting labels at random.

Traditional software engineering has unit tests in order to debug and test the behaviour of software programs in edge cases. Analogous to this for machine learning models are integration, regression and directionality tests. These tests serve the purpose of assessing the robustness of a machine-learning model before deployment. By robustness, I mean how well this model performs on data representative of real-life scenarios.

Integration test: This is a particular kind of test where a trained machine learning model is tested on real-world examples. Such examples are gathered with human help in order to verify the representativeness of such data. By performing this test repetitively and cyclically. Machine learning models can be versioned into better ones by adding recent, representative data to the training dataset.

Regression test: This is a type of test usually performed to critically investigate and compare the performance of model versions. Let’s say there are two models, v1 and v2, where v2 is an iterative improvement on v1. In a bid to quantitatively affirm this, we write regression tests. Regression tests assess the performance of new model v2 on data points where model v1 was wrong. Model v2 is also assessed on data points that were predicted correctly in order to examine if this performance is consistent with model v1. An ideal circumstance is one where model v2 correctly predicts/classifies all of the data points model v1 correctly predicts in addition to a portion of the outliers. In this case, metric for evaluation depends on the task at hand. By investigating right and wrongly predicted data points across the model versions, hypothesis and inferences can be made, that inform future experimentation.

Directionality test: Directionality tests are quite similar to regression and integration tests, but different in some light. Directionality tests attempt to examine the performance of a model under the condition of noisy inputs. As stated earlier, depending on the task at hand, noise can be introduced to a dataset in several ways. Usually, this is set as a baseline, to examine the impact of data augmentation. Data augmentation is sometimes used to increase the amount of data in a training set, where inputs are intentionally modified/ corrupted, but the labels remain the same in a bid to teach the model how to classify such input on the occasion that it receives it.

Metric for evaluation here is the batch metric, and key here is observing if classification changes. If it does, back to the drawing board. During this phase of repeated experiments. Key metrics and parameters across multiple pipeline runs can be easily tracked and changed using a tool called Metaflow

Continuous annotation to improve data quality

Further on, the process of continuously identifying label issues in order to improve model quality is called continuous annotation. The state-of-the-art method applied is called confidence learning, which is an algorithm which learns a joint distribution using noisy labels(originally labelled data) and out-of-sample predicted probabilities(Probabilities obtained by training a model using cross-validation).

Although computationally intensive, this algorithm smartly identifies issues with apriori labelled data, by returning indices of possible corrupted labels. In addition to human-in-the-loop training to correct these suspicious annotations, a model’s performance can be improved with this method. Tools that can be used is Labelstudio, and cleanlab.