Let me introduce Query-By-Committee (QBC), the most representative method of Active Learning. Literally translated, “Query-By-Committee” means asking questions to a committee, and its mechanism depends on what the committee consists of and what questions are asked.

In general, the committee is composed of multiple machine learning models.
These models can be of various types — for instance, Random Forest or YOLO.
If the task is for classification, classification models are used; if it is for regression, regression models are used.

Once the committee is organized, an unlabeled sample is presented to the committee (a query). Each model in the committee makes its own prediction, and by analyzing the differences among these predictions, the information content (or uncertainty) of the sample can be estimated.

If we measure information content based on uncertainty, then when all models make the same prediction, it indicates that the models are already confident about that data — hence, low uncertainty. Conversely, if the models’ predictions differ, it suggests that the models are uncertain and confused about that data point — leading to high uncertainty.
The process of checking whether the predictions are consistent or not is called a vote.

A representative metric used to measure uncertainty is entropy, which quantifies the degree of uncertainty in information.

The formula for entropy is as follows:

Let’s look at an example.

Task : Classification

Class : Apple, Pear, Grape

Committee : YOLO, VGG, RCNN

Vote : Hard voting

Uncertainty : Entropy

Final model : YOLO

Unlabeled set : Three images(Apple 1, Pear 1, Grape 1)

It is generally better to set the committee size as an odd number.

If the number of committee members is even, a tie may occur during voting, which makes it difficult to determine a priority or final decision. The committee can consist of the same model type (e.g., multiple YOLO models) or different models such as VGG, RCNN, and others.

However, if you build the committee using the same model architecture, it is important that their weights (parameters) are different.

If all models share identical weights, they will always produce identical predictions, making the query process meaningless.

In this example, let’s assume we use our final trained model as the base for the committee.

Now that the environment is ready, let’s begin Active Learning. First, we query the committee with an image of an apple. Each model in the committee predicts its class label.

If all models unanimously classify it as “apple”, then the uncertainty for this sample becomes 0, indicating that the committee is fully confident in its prediction.

Next, let’s ask the committee about the grape. The VGG model classified it as grape, while the other two models classified it as pear. In this case, the uncertainty is 0.579.

Finally, we query the committee with a pear. Similarly, only VGG classified it as grape, while the other models predicted pear. Here again, the uncertainty is 0.579.

Now, since both grape and pear have the same uncertainty, we could apply a more complex tie-breaking rule to determine priority — but for simplicity, we’ll choose the order randomly, setting it as grape → pear.

Thus, the final labeling order becomes as follows:

        Pear → Grape→ Apple

Now, either a human annotator or the model can label the samples in this order, and the model can be retrained with the newly labeled data.

The overall framework of this Active Learning process can be summarized as follows:

1. Build a committee of models.

2. Query the committee with unlabeled samples.

3. Measure uncertainty based on disagreement (e.g., entropy).

4. Rank samples by uncertainty.

5. Label and retrain the model with the selected data.

6. Press enter or click to view image in full size

This concludes the series on Active Learning.

There are many more variations and metrics if you’d like to explore deeper — feel free to look them up if you’re interested.

In the next post, I’ll return with a new topic.

[References]

[1] Settles, Burr. “Active learning literature survey.” (2009).

[2] D. Wu, “Pool-Based Sequential Active Learning for Regression,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 5, pp. 1348–1359, May 2019, doi: 10.1109/TNNLS.2018.2868649.