Seeing What a GAN Cannot Generate

Despite the success of Generative Adversarial Networks (GANs), mode collapse remains a serious issue during GAN training. Worse yet, little work has focused on understanding and quantifying which modes have been dropped by a model. In this work, we take a first step and present two analytic methods for systematically studying this phenomenon. First, we deploy a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set. Differences in segmentation statistics reveal object classes that are omitted by a GAN. Second, given the identified omitted object classes, we further visualize what the GAN is doing instead. In particular, we compare specific differences between individual photos and their approximate reconstructions by a GAN model. To this end, we propose a new image reconstruction method based on inverting the layers of a generator. Finally, we use our framework to analyze several state-of-the-art GANs trained on multiple datasets and identify the typical failure cases of existing models.