In previous tests we presented some initial results showing what it takes to create a deepfake. The basic parameters were identified based on the number of images, lighting, and the angle of the source material.

Since then we have also performed tests in the following categories:

• Resolution of source material
• Differing facial structure
• Overlapping objects

The underlying model had been trained with images of Donald Trump and Nicolas Cage. The results are presented in line with the last round of tests. That means running topically relevant test cases through a set of specifically defined categories. Two videos will be shown for each of these test cases. The first case shows the results of merging the faces using the standard parameters (default). In the second case, each of the parameters for merging the faces is manually tweaked until they produce the best possible results. The YouTube video Bush’s Best Speech was used as the output video for all tests.

⇒ YouTube Video

Resolution of source material

The question here is how will the standard resolutions of the video affect the fake. To test the effect of the standard resolution on the deepfake, different renderings were performed with videos in 360p and 720p resolution.

Source video 360p ⇒ output video 360p, 7 seconds (168 images)

First, the results from the source and output video in 360p resolution were compared with the source and output video in 720p. The newly created 360p fake was compared with the video from the case source video 500 images ⇒ output video 7 seconds (168 images) in 720p resolution, which was covered in the recent article. Apart from the poorer quality of the new 360p video, which was basically to be expected, and the somewhat blurrier edges, it still achieved decent results. Less obvious was the somewhat darker face compared to the video in 720p resolution, although nothing about the images, except for the resolution, was modified for either fake.

Source video 360p ⇒ output video 720p, 7 seconds (168 images)

Next we tried to determine whether the quality of the fake deteriorated when the new video material was only available in 360p resolution and the video being modified had a resolution of 720p. The resulting video was compared with the same source video with 500 images⇒ output video 7 seconds discussed in the most recent article. If the images are compared side-by-side, subtle differences in facial structure are noticeable. When comparing the output videos, however, these differences do not immediately stand out. Nevertheless, for best results, the maximum resolution should be used for the video.

Differing facial structures

The main question here is whether you can achieve good results when the subjects have completely different facial structures. To test this, videos of Donald Trump and Hillary Clinton were used for face-swapping. Care was taken to select image material containing a good mix of images with frontal and side angles to ensure this would not be a factor in the test results.

Source video Donald Trump ⇒ output video George Bush, 7 seconds (168 images)

The goal was to use two faces with very different facial structures. Donald Trump has a much rounder face than George Bush. Care was taken to ensure that the lighting and angle of the shots was as similar as possible in both cases to allow a close comparison of structural differences. The evaluation of this particular deepfake showed it to be a total failure. Trump’s and Bush’s contrasting skin tones emphasized their very different facial structures in the manual test case. In the default case, the problem is with the heavy flickering, as the most recent article pointed out. What’s more, the artificially rendered nose does not look anything like Donald Trump’s nose.

Source video Hillary Clinton ⇒ output video George Bush, 7 seconds (168 images)

Here, the purpose of the test was to see whether you can swap a female face for a male face. Videos of Hillary Clinton were used for the female face. The resulting fake does not look real and gives the impression that something is not quite right with this face, but we have no conclusive reason why that is.

Overlapping objects

The aim of these tests is to determine how to deal with hands in front of the face. In this category, we also performed a test in which we swapped a face with a beard for a face without a beard.

Source video 10% of images with overlapping objects ⇒ output video 7 seconds (168 images)

Of the 500 source images used, 50 had George Clooney with hands in front of his face. This already created a problem during the processing of the faces required for the actual rendering of the deepfake. Before the rendering, the first step is to try to extract a face from each image. This extraction failed for a quarter of the facial images in which there was a hand in front of the face. To produce the deepfake, all images containing a recognizable face were used. The video differs slightly from the case using source video 500 images ⇒ output video 7 seconds discussed in the most recent article. For example, the left eye appears to tremble more than it does where there are no overlapping objects.

Source video 33% of images with overlapping objects ⇒ output video 7 seconds (168 images)

As with the previous source material, here there were also problems extracting faces when there were objects in front of them. Again, no face was recognized in a quarter of the facial images which featured obstructions. Likewise, the quality was poorer than without overlapping objects.

Source video with beard ⇒ output video without beard, 7 seconds (168 images)

In this case, the task was to determine how well the deepfake algorithm can handle beards. For this purpose, we used a source video of a bearded George Clooney and found that the beard could not be recreated convincingly in the deepfake. The beard looks as if it is glued on, and it lacks depth. The individual beard hairs do not have enough detail. As it currently stands, reproducing hair in images is still quite difficult.

Glasses

While researching the production of deepfakes, we came across an article featuring a transformation of Jimmy Fallon into John Oliver. The resulting deepfake is a compelling indication that glasses are still a problem. The faces pass muster, but the frames of the glasses look glued on, much like the beard before. Upon closer inspection, the viewer will notice that the temple of the glasses is missing.

Conclusion

As the glasses example and Donald Trump both demonstrate, only part of the face is modified. Even under the best conditions, the underlying deepfake algorithm is unable to properly transform the forehead, ears and hair. To create a truly convincing deepfake (that is, one that is indistinguishable from the original), the haircut and general shape of the face of the “new” person must at least be similar to that of the person in the source material. Moreover, we found that glasses, overlapping objects, and hair are still generally problematic.

About the Author

Andrea Hauser graduated with a Bachelor of Science FHO in information technology at the University of Applied Sciences Rapperswil. She is focusing her offensive work on web application security testing and the realization of social engineering campaigns. Her research focus is creating and analyzing deepfakes. (ORCID 0000-0002-5161-8658)

Links

• https://hackernoon.com/exploring-deepfakes-20c9947c22d9