Paper | ArXiv | Project | Demo
Example inpainting results of our method on images of natural scene (Places2), face (CelebA) and object (ImageNet). Missing regions are shown in white. In each pair, the left is input image and right is the direct output of our trained generative neural networks without any post-processing.
Training/testing code and models will be released soon. Please stay tuned.
Visualization on TensorBoard for training and validation is supported.
CC 4.0 Attribution-NonCommercial International
- Can other types of GANs work in current setting?
The proposed contextual attention module is independent of GAN losses. We experimented with other types of GANs in Section 5.1 and WGAN-GP is used by default. Intuitively, during training, pixel-wise reconstruction loss directly regresses holes to the current ground truth image, while WGANs implicitly learn to match potentially correct images and supervise the generator with adversarial gradients. Both WGANs and reconstruction loss measure image distance with pixel-wise L1.
- How to determine if G is converged?
We use a slightly modified WGAN-GP for adversarial loss. WGANs are demonstrated to have more meaningful convergent curves than others, which is also confirmed in our experiments.
- The split of train/test data in experiments.
We use the default training/validation data split from Places2 and CelebA. For CelebA, training/validation have no identity overlap. For DTD texture dataset which has no default split, we sample 30 images for validation.
- How does random mask generated?
Please check out function random_bbox and bbox2mask in file inpaint_ops.py.
- Parameters to handle the memory overhead.
Please check out function contextual_attention in file inpaint_ops.py.
@article{yu2018generative,
title={Generative Image Inpainting with Contextual Attention},
author={Yu, Jiahui and Lin, Zhe and Yang, Jimei and Shen, Xiaohui and Lu, Xin and Huang, Thomas S},
journal={arXiv preprint arXiv:1801.07892},
year={2018}
}
