Recently, Google AI has launched a data challenge called CATS4ML (Crowdsourcing Adverse Test Sets for Machine Learning). This challenge improves the evaluation datasets for Machine Learning by encouraging searching the existing ML benchmarks.
CATS4ML Data Challenge is up till 30 April 2021
CATS4ML data challenge asks the challengers to use different methods to find the examples of Unknown Unknowns in ML models. So, when this technology will get more experience, the object recognition technology of Google will perform better.
In the object recognition tasks, CATS4ML will challenge the ability of Machine learning. The test set has many examples that are difficult to solve with algorithms. The main purpose of CATS4ML is to give a data set for the developers to examine the weaknesses of the algorithm.
Many evaluation datasets have easy-to-evaluate items, but they miss the natural ambiguity of real context. Evaluating ML models without real-world examples is difficult to test machine learning performance. And this causes ML models to develop “weak spots”.
Google AI’s CATS4ML Data Challenge at HCOMP 2020 shows the difficulty of identifying the ML model’s weaknesses. The main aim of this challenge is to put the bar in ML evaluation sets to spot new data examples and about this machine learning is confident. The outcomes of this challenges will help identify and avoid future errors.
Weak Spots in Machine Learning Models
Weak Spots are examples that are difficult for a model to evaluate correctly. This happens because the dataset does not include the classes of examples.
The researchers continue to study the “Known Unknowns” in an Active Learning domain. The community has found a solution to get a new label from people on random examples. Like, if a model is not sure of the subject of a photo is cat or not, a person is directed to verify that photo. And if the model is sure about the photo, then the person is not asked.
The real-world examples can give better results to a model’s failures in its performance. Hence, the CATS4ML data challenge tries to collect unmanipulated samples that humans can read but the models make mistakes.
The CATS4ML data challenge is open till 30 April 2021 for researchers and developers globally. The participants can register on the Challenge website, download the target images and dataset, and provide the pictures.
