In the previous session, we explored active learning query scenarios. We learned that the query scenario varies depending on the data type and environment of the problem to be solved. In this session, we will look into the criteria for finding a good active learning strategy.

What makes a good active learning strategy? Active learning is a technique that selects data samples with the greatest impact on model learning from unlabeled samples. Therefore, a good active learning strategy can be seen as one that effectively selects data that greatly contributes to model learning. Let’s think more about what it means to significantly help model learning.

To significantly help model learning means that by learning certain data, the model’s performance (accuracy) improves substantially. In other words, if the model learns data it hasn’t learned before, its accuracy would increase dramatically.

Let’s consider an example. Imagine there is an image classification model that distinguishes between apples, pears, and grapes. What would happen if we input a banana into this model? It will likely produce an incorrect result because the model has never been trained on banana data and doesn’t know what a banana is. Now, what if we added banana data to the model and retrained it? After retraining, if we input a banana, the model would correctly classify it as a banana.

In this way, data that the model has not yet learned and has a large error is considered data that significantly helps model learning. The criteria used to evaluate this data is called informativeness. Informativeness is measured by the model’s uncertainty. A representative metric for uncertainty is entropy, a concept derived from information theory. In the previous example, the banana image is the most informative image. To summarize, the higher the uncertainty, the more informative the data is, and thus, it should be prioritized.

The second criterion is representativeness. As the name suggests, the data selected by active learning should represent the overall characteristics of the entire dataset. For example, suppose you have 50 images of apples, pears, and grapes from the front, bottom, and top views. As shown in [Figure 2], if you train the model only with front-facing images, its performance is likely to be poor. A model trained with images from all views — front, bottom, and top — will perform better.