Fine-grained Image-to-Image Transformation towards Visual Recognition

Wei Xiong1, Yutong He1, Yixuan Zhang1, Wenhan Luo2, Lin Ma2, Jiebo Luo1

1University of Rochester

2Tencent AI Lab


Existing image-to-image transformation approaches primarily focus on synthesizing visually pleasing data. Generating images with correct identity labels is challenging yet much less explored. It is even more challenging to deal with image transformation tasks with large deformation in poses, viewpoints, or scales while preserving the identity, such as face rotation and object viewpoint morphing. In this paper, we aim at transforming an image with a fine-grained category to synthesize new images that preserve the identity of the input image, which can thereby benefit the subsequent fine-grained image recognition and few-shot learning tasks. The generated images, transformed with large geometric deformation, do not necessarily need to be of high visual quality but are required to maintain as much identity information as possible. To this end, we adopt a model based on generative adversarial networks to disentangle the identity related and unrelated factors of an image. In order to preserve the fine-grained contextual details of the input image during the deformable transformation, a constrained nonalignment connection method is proposed to construct learnable highways between intermediate convolution blocks in the generator. Moreover, an adaptive identity modulation mechanism is proposed to transfer the identity information into the output image effectively. Extensive experiments on the CompCars and Multi-PIE datasets demonstrate that our model preserves the identity of the generated images much better than the state-of-the-art image-to-image transformation models, and as a result significantly boosts the visual recognition performance in fine-grained few-shot learning.


Overall Architecture

Constrained Nonalignment Connection and its Attention Visualization

Experiment Results

Identity Classification on Generated Images

Visual Results

Visual results on the CompCars dataset.
Visual results on the Multi-PIE dataset.


  title={Fine-grained Image-to-Image Transformation towards Visual Recognition},
  author={Xiong, Wei and He, Yutong and Zhang, Yixuan and Luo, Wenhan and Ma, Lin and Luo, Jiebo},
  booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
Paper: [Paper][Supp]