Last time, we briefly introduced what Active Learning (AL) is and discussed its necessity. Before diving into the techniques, I would like to explain the query scenario, which was briefly mentioned earlier. Let’s first look at what a query scenario means before going into more detail.
What is a Query Scenario?
A query scenario refers to a method of querying data to select data that could be helpful for training a machine learning model from an unlabeled dataset. In AL, it's like asking the data a question. As shown in Figure 1, you're asking, "How much information do you have?" . The data then provides a response, indicating, "I have 123.444." Based on this response, you can determine the importance or relevance of the data. This process helps prioritize which data points should be labeled and used for training the model. The value 123.444 is an example, and there are various techniques to calculate this value. We called these techniques as active learning.
There are various forms of data, such as image, table, and language, and the method of querying varies depending on the characteristics of the data. The most representative query scenarios are pool-based and stream-based(population-based).
Pool-based Scenario
This is the figure 2 we saw earlier when discussing Active Learning. In this scenario, the person in the middle labels data from an unlabeled pool, and the labeled data is then used to train the machine learning model. In the image, the data in the unlabeled pool is queried by AL, and if it is deemed meaningful, the person labels it. This method is referred to as the pool-based scenario.
The term “pool-based” literally refers to performing from a data pool, with “pool” implying that the data is static or stagnant. You can think of it as a collection of data that is readily available but not actively changing. For example, imagine a dataset containing 100 cat photos and 100 dog photos. This dataset remains static, and AL is applied to select and label the most informative data from this pool.
Population-based Scenario
The population-based scenario, refers to a case where the distribution of data is known in advance, allowing for the generation of data. The concept of “knowing the distribution in advance” may sound confusing, but you can think of it as data that follows a specific function, like a polynomial function. For example, imagine a complex polynomial, as shown in Figure 2 below. I’ll call this the “peak function.” For such a peak function, if an x-value is given, you can calculate the corresponding y-value. Moreover, since the range of x-values is [-∞, +∞], you can compute a y-value for any x-value you input. When the range of x-values is infinite, and the distribution (function) is known in advance, you can apply the population-based scenario.
It’s not only possible to use the stream-based approach for function-based data. You can also create a pool of 1,000 x-values in advance and use a pool-based approach. However, in such cases, you must carefully select the x-values when generating the pool. If you limit yourself to 1,000 values in an infinite range, you might miss out on more meaningful data (x-values) not included in the pool. That’s why in these cases, a population-based approach is usually recommended.
Stream-based Scenario
“Stream-based” refers to stream-based selective sampling or sequential active learning. As the term “stream” suggests, this method is used in situations where unlabeled data can be continuously obtained for free. When new data arrives, the learner must decide whether to label the data or discard it. Since new data arrives one instance at a time, it is also called sequential active learning. This method is particularly suitable for real-world problems where data is continuously generated in real time.
Based on the explanations so far, can you start to see which scenarios could be applied to typical ML problems like regression and classification? In regression problems, population-based scenarios are generally used, as in the earlier example of the peak-function. But can the Boston housing price dataset, a well-known regression problem, use the population-based approach? The answer is NO. You can’t know the data distribution in advance, nor is the range of independent variables infinite. Therefore, for table data like the Boston dataset, a pool-based approach is recommend. In short, the population-based approach is mostly used with function-based data in regression problems.
What about classification? In classification problems, pool-based methods are typically used. When real-time driving is used to obtain road-driving image data and a model for classifying objects from road images is being built, it is possible to select and label the images collected using a stream-based approach. Except for such cases, a pool-based approach is used.
Additionally, regression problems are slightly more challenging than classification problems. Think about it. In classification tasks, using pool-based or stream-based scenarios usually involves selecting from a predefined pool or a limited set of data candidates. However, in regression problems, the range of x is infinite, meaning the number of options is unlimited. As a result, there are more factors to consider compared to classification, which can make regression slightly more difficult.
Today, we explored the scenarios for applying AL. This explanation turned out longer than expected, but I hope everything is clear. In the next session, we will cover the criteria used in AL.
[References]
[1] Settles, Burr. “Active learning literature survey.” (2009).
[2] D. Wu, “Pool-Based Sequential Active Learning for Regression,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 5, pp. 1348–1359, May 2019, doi: 10.1109/TNNLS.2018.2868649.
[3] Jiang, Ping, et al. “A novel sequential exploration-exploitation sampling strategy for global metamodeling.” IFAC-PapersOnLine 48.28 (2015): 532–537.
GitHub
LinkedIn