ObjectNet: A bias-controlled dataset object recognition

What is your paper about?

It was well known that large-scale datasets such as ImageNet which are became the gold standard in the vision community suffer from various biases. This is because these benchmarks typically include images that are extracted from the internet and, not surprising, the images that people tend to upload are typically ‘clean’ in many ways. For example, objects in ImageNet typically appear in the right context (e.g. kitchen utensils appear in the kitchen). It is well known that models learn these biases and that helps them to achieve very high performance on the benchmarks. In this project we address this issue by constructing a dataset, ObjectNet, in a principled manner which controls several of these biases. We show that by removing some of the biases, object recognition models which perform very well on ImageNet, fail miserably on ObjectNet. Humans, on the other hand don’t. The goal is to push the community to develop models that are robust to such changes in the same way that human are.

What is new and significant about your paper?

We suggest a new way to collect data for benchmarking computer vision models which controls for biases. This is a standard way of performing experiments in other fields of science yet in the AI community, because of the difficulty to collect data, this issue has been pushed under the rug in many cases.

What will the impact be on the real world?

A more robust way to benchmark computer vision models that will be a better predictor for their performance in the real-world.

What would be the next steps?

The next steps are to develop methods to control biases which we are not addressing at the moment, such as occlusions. In addition, we are planning to launch a challenge to the computer vision community, in the form of a competition for the model which achieves the best accuracy in object recognition over ObjectNet.

What surprised you the most about your findings?

We expected models to achieve sub-human performance on ObjectNet, but we did not expect such a large decrease in performance compared to ImageNet.

What was the most challenging part of your research?

Coming up with a principled way to collect the data and making sure that workers who were crowd-sourced to collect the data are not ‘gaming’ the system.

What made you most excited about this paper?

The fact that we are addressing such a fundamental issue in the methodology that is currently being widely used by the community.