Image synthesis through feature-based mixing of contents

Abstract

In this project we approach the problem of synthesizing an image by combining several features from two sample images. One of this features is the content, which makes reference to shapes and spatial distribution of the elements present in the image. The other is the style, which includes colors, brush strokes, illumination... This problem is known as the 'neural style' problem. We start with an overview of the state of art to analyze some of the most popular methods and tools to implement a solution for this problem: machine learning, convolutional networks, gradient descent... Also, we describe the chosen methods and tools that we will use in our implementation. In particular, the object that will do the feature extraction will be a speci c convolutional network called VGG16. Afterwards, we explain all the entire mathematical structure behind the idea of how the convolutional network can act as the extractor. Firstly, we analyze the general problem of image synthesis abstractly. Secondly, a concrete example is used to show how we can implement the method. Finally, we provide an implementation of the neural style transfer method using the machine learning approach. Lastly, the output of our implementation is used to study how the convolutional network works. In order to do it, we have designed some experiments, divided in three categories. In the rst one, we analize the inside of the network, getting some insights about its architechture and what it does exactly. In the second one, we try to use the network combined with our program to generate a morphing e ect. The results of this experiment did not come up quite good, but they leave plenty of room open for improvement. In the third category, we study what happens when we modify some of the characteristics of the implementation. To sum up, the results obtained from the experiments are satisfying enough: they allow us to meet our objectives for the project while giving some room for improvements in future work.