Training with only a few but high resolution image pairs

[yingzwang]

I want to generate watercolor paintings from pencil sketches. I have perfectly paired training data, but only a few pairs.

Q: When we have paired training data, pix2pix always works better than CycleGAN. Is that correct?

I trained pix2pix model on only two training pairs, and the output is amazing. Please see below my training pairs, and the generated new paintings.



The results above are obtained by training on images of size 512 x 512. The original scans of my paintings have much higher resolution, i.e., 4096 x 4096. I feel that the results can get much better if I take advantage of the high resolution images.

Q: Will it help if I apply a random crop on the original image of size 4096 x 4096, to create many 512 x 512 training pairs?

[junyanz]

Your results look really cool. Data augmentation might help! See this paper for training a pix2pix model with only one image. They have some tricks!

[yingzwang]

Thanks for your reply. I'll check the paper and try their tricks.