Digital forensics, also known as IT forensics uses scientific procedures and methods to reliably capture digital evidence so that it can be used in court. It involves the examination of data stored on various digital devices such as computers, mobile phones, servers, memory cards and other storage media.

The importance of IT forensics extends beyond law enforcement and is also increasing due to the growing digitisation in other sectors. Companies and organisations use IT forensics to investigate and detect incidents such as data breaches, fraud and breaches of data protection policies. Through the use of IT forensics, digital evidence can be collected, analysed and evaluated for use in court. Forensic analysis of images and documents is a main component of IT forensics. An introduction and related techniques can be read in the previous article: IT forensics Analysis of Images and Documents Another essential aspect of IT forensics is video analysis, which helps to verify the authenticity of video recordings and extract valuable information from the videos. Various techniques and methods are used for this.

Video analysis

The analysis of videos is an important part of IT forensics and requires specific knowledge and skills. It is important to note that the use of video as evidence in IT forensics must follow certain guidelines and protocols to ensure that the integrity of the evidence is not compromised. This includes ensuring the chain of custody of evidence and following best practices for evidence preservation and storage. As a rule, electronic evidence is first appropriately secured by the authorities. This is done by seizing devices such as hard drives, USB sticks or other storage media. In the process, a protocol is drawn up that contains detailed information about the find, the condition of the evidence, the location and the persons involved. This protocol serves as proof of proper seizure. The electronic evidence is then transported in a safe and controlled manner to avoid loss or damage. Specialised means of transport or encrypted storage media can be used for this purpose. After seizure, the evidence is sealed to ensure that it is not tampered with during storage. This can be done by using seals, security labels or digital sealing technology. When evidence is stored securely, measures are taken to prevent unauthorised access, damage or loss.

Authenticity analysis

Authenticity analysis of digital content plays an increasingly important role in a time when manipulation and forgery of information and media are widespread. Especially in the field of forensic investigations and media analysis, authenticity verification is of great importance to ensure the integrity of evidence and the trustworthiness of information. Authenticity analysis is concerned with the verification of characteristics and properties of a digital object to determine whether it is indeed what it claims to be and whether it has remained unchanged throughout the creation, transmission and storage process. This process involves examining various factors such as metadata, data structures, file formats, digital signatures and other characteristic features of the digital object.

Metadata

Metadata plays a central role in authenticity analysis as it can provide information about the origin, creation and editing of the digital object. For example, image files contain metadata such as date taken, camera model, GPS coordinates and possibly the name of the photographer. By analysing and checking this metadata, irregularities, contradictions or manipulations can be detected. For example, a possible manipulation could be that a photo was given a later date to support a particular story.

Technical analyses

Technical analyses of digital content can provide promising evidence of manipulation. In the case of images, for example, artefacts can appear that indicate the removal or addition of objects or the alteration of pixels. For videos, analysis of frame rate, motion patterns and audio playback can provide clues to editing or synchronisation problems. These technical analyses often require specialised software tools and expertise.

Digital signatures or certificates

Another important aspect of authenticity analysis is the verification of digital signatures or certificates. Digital signatures are used to guarantee the authenticity and integrity of a digital object. They are often used to verify digital documents or transactions. By checking the digital signature, it can be determined whether the object has remained unchanged since it was signed and whether the signer is trustworthy.

Automated methods

The development of artificial intelligence and deep learning algorithms also has an impact on authenticity analysis. Automated methods and machine learning algorithms are increasingly being used to detect manipulations in images and videos. These techniques make it possible to identify changes in pixels, artefacts from image processing software and unusual patterns. Machine learning algorithms can be trained to learn from examples which features indicate manipulation. This can help make the process of authenticity analysis more efficient and quickly identify potential forgeries.

Application areas

Authenticity analysis is used in various fields. In the forensic investigation of crimes, digital evidence such as images, videos or documents can be checked to ensure that they have not been manipulated and can be used in court. In journalism and media, authenticity verification is crucial to provide trustworthy information and avoid misinformation or fake content. In the security industry, authenticity analysis is used to detect manipulation of surveillance video or other security-related footage. However, it is important to note that authenticity analysis does not always provide conclusive results. Sometimes tampering can be done so cleverly that it is difficult to detect. In addition, some techniques for forging digital content can lead to it being classified as authentic, even with careful analysis. It is therefore advisable to combine different analysis methods and make a comprehensive assessment to increase the likelihood of a successful authenticity analysis.

Frame-by-frame analysis

Frame-by-frame analysis is a method used in various fields such as IT forensics, video surveillance and the film industry to extract detailed information from individual frames of a video, gather evidence or gain valuable insights. In this type of analysis, each frame is viewed sequentially and examined for possible features, events or patterns. The process of frame-by-frame analysis begins by splitting the video into its individual frames. This can be done using specialised software or tools. Each frame is then analysed individually and examined in detail. This analysis can be done manually or using automated techniques such as machine learning and image processing algorithms to automatically detect certain features or events. These algorithms are trained with large amounts of training data to identify specific patterns or objects in the video. During frame-by-frame analysis, different aspects of the video can be examined. This includes identifying people, vehicles or objects, patterns of movement, identifying faces, gestures or actions, analysing text or symbols in the video, detecting changes in the background or finding evidence of criminal activity or accidents. Each individual frame is carefully considered to find possible clues or evidence that may be relevant to the context. Documentation of the results of the analysis is crucial to ensure the integrity of the evidence found and to be able to reproduce the results later.

IT Forensics

In IT forensics, frame-by-frame analysis can be used to gather evidence. This can include identifying suspects by their appearance or behaviour, tracking movements before, during and after a crime, or reconstructing an accident scenario. The detailed examination of each frame can help to gain important information that is crucial in identifying perpetrators or reconstructing events.

Video surveillance

In video surveillance, frame-by-frame analysis is used to detect suspicious activity, identify security breaches or monitor unusual events. By looking closely at each frame, deviations from normal patterns or behaviours can be detected. This can facilitate the detection of intrusions, vandalism, theft or other unwanted events.

Film industry

In the film industry, frame-by-frame analysis is used to create visual effects, edit special shots or enhance scenes. By examining each frame, visual elements such as background detail, colour correction, image stabilisation or the integration of CGI effects (Computer Generated Imagery effects are often used to create realistic visual representations of fantasy worlds, alien creatures, special effects, explosions, virtual environments and more) can be adjusted. This allows precise control over the visual appearance of a film and provides seamless integration of effects or changes in a particular scene.

Challenges

However, there are also some challenges with frame-by-frame analysis. The accurate viewing and analysis of large amounts of video material requires a lot of time and resources. In addition, the quality of the video, such as resolution, image noise or compression, can affect the accuracy of the analysis. Furthermore, motion blur, fast movements or unclear footage can make the detection and analysis of details harder.

Temporal analysis

Temporal analysis is also an important part of the forensic examination of videos. It deals with checking the frame rate, movement patterns, synchronisation of audio and video, and time stamps to detect manipulations and irregularities. The analysis helps confirm the authenticity of videos and reconstruct the exact timing of events. Machine learning and artificial intelligence are increasingly supporting automated analysis here as well. However, temporal analysis can be challenging. Achieving accurate results and ensuring the integrity of videos requires expertise and specialised equipment.

Voice forensics

Voice forensics is a field that deals with the scientific analysis of vocal characteristics. It is used to confirm the identity of a person based on their voice or to detect vocal manipulation. Voice forensics plays an important role in criminal justice, counter-eavesdropping and media analysis. By analysing pitch, timbre and other vocal characteristics, forensic experts can draw conclusions and obtain accurate results. It is important to note, however, that voice forensics has certain limits and manipulation cannot always be ruled out here either.

Detection of deepfakes

Deepfakes are trained deep-learning models that make it easy to exchange a face of a third person with a face in any video. In the article Deepfakes an Introduction Andrea gives an introduction to the topic. Already in it she mentions the first forensic tool of the US Defense Advanced Research Project Agency (DARPA) to be able to identify deepfakes. In 2021, the paper Exposing Manipulated Photos and Videos in Digital Forensics Analysis was published together with a promising Autopsy plugin: Photo and video manipulations detector. As the name of the plugin suggests, it is supposed to be able to detect manipulations, including deepfakes, in photos and videos. Autopsy is an open source platform for forensic work with digital devices and files.

Example – Is Selenski really standing here in the middle of Kiev?

One month after the Russian invasion of 24 February 2022, Ukrainian President Selenski called in a video on all free people of this earth to take a stand for freedom, peace and thus – he concluded – also for Ukraine, is Selenski really standing here in the middle of Kiev? But a close look at the video raises doubts as to whether Selenki is really standing in the open street from the Kiev government building.

Because of the light shining into Selenski’s face, there should be a bright warm-white lamp (~3300 Kelvin) to his left. This should have an influence on the shadow cast by the soldier in the background. However, this is not the case. In addition, the shadows cast by the two lamps directly in front of Selenski are missing and can be recognised by the reflection in his eyes.

The light visible in the background, which grazes his left cheek, does not have the expected effect on his whiskers. The so-called volumetric shimmer is not visible. It can also be seen that at time point 03:27 of the video, the soldier’s shadow moves as he changes his position. After the cut, however, he is still standing still in his old position. At time point 06:24 it is also noticeable that the soldier in the background remains in the same position, which means that this so-called jump cut is not real, but merely a zoom made afterwards in the cut. In addition, the jump cut has the strange effect that the soldier’s neck immediately shifts to the left. This is only possible if the cut comes from two different shots. However, both the soldier and the pulsating light remain true to their cadence. Thus, the background is the same continued shot. Also interesting is the lack of ambient noise. This is only possible if a directional microphone was used and noise cancelling or strong compression was used. Furthermore, the sound effect of Selenski’s voice, such as reverb, does not correspond to the spacious street. This is only possible if an accurate directional microphone or a lapel microphone was used. Neither is visible on the video. All these features point to a fabricated video and the use of a green screen.

Summary

Video analysis in IT forensics plays an important role in the investigation of digital video data. The aim is to extract relevant information, identify manipulations and ensure the integrity of the evidence. This analysis finds application in areas such as law enforcement, litigation, digital forensics and security audits. Various techniques and methods are used to perform a comprehensive video analysis. First, an authenticity analysis is performed to verify the authenticity of the video and the presence of possible tampering. Metadata such as timestamps, camera settings and file information are examined to detect any discrepancies or deviations. Furthermore, frame-by-frame analysis plays an important role. Here, individual frames of the video are analysed to detect features such as compression artefacts, colour differences, irregularities in the edge structures and signs of editing. These examinations help to identify manipulations. Temporal analysis looks at the temporal aspects of the video. This involves checking the frame rate, analysing the synchronisation of audio and video, and examining movement patterns. Deviations in these areas can also provide indications of manipulation. Another important aspect is voice forensics, which focuses on checking the authenticity of audio recordings in the video. Characteristics of the human voice are analysed to detect manipulation or synthetically generated voices. Advanced techniques also include the creation of 3D models to analyse perspectives, shadows and lighting in the video. This can reveal anomalies or irregularities. By using specialised techniques and tools, forensic experts can identify tampering, verify the authenticity of videos and thus help to ensure the integrity of the evidence. A particular challenge is that tampering techniques are becoming increasingly advanced. There are a variety of tools and software programs that enable even inexperienced users to manipulate digital content and create forgeries. It is therefore important that authenticity analysis methods are constantly evolving to keep up with the latest manipulation techniques.

About the Authors

Michèle Trebo has a Bachelor of Information Technology at ZHAW and worked six years as a police officer in the field of cyber crime investigations. She is responsible for criminal research topics like darknet analysis, cyber threat intelligence, fraud investigation, and forensics. (ORCID 0000-0002-6968-8785)

Ralph Meier completed an apprenticeship as an application developer, with a focus on web development with Java, at a major Swiss bank and then completed a Bachelor of Science in Computer Science UAS Zurich at the ZHAW School of Engineering. His primary task is doing security-related analysis of web applications and services. (ORCID 0000-0002-3997-8482)

Links

• https://github.com/saraferreirascf/Photo-and-video-manipulations-detector
• https://www.autopsy.com/
• https://www.itsec.techfak.fau.de/files/2019/07/Einfuehrung_Digitale_Forensik_Leseprobe.pdf
• https://www.mdpi.com/2313-433X/7/7/102
• https://www.tagesanzeiger.ch/was-steckt-hinter-diesem-panzerwrack-totenkopf-439914937537/2#event-list
• https://www.technologyreview.com/2018/08/07/66640/the-defense-department-has-produced-the-first-tools-for-catching-deepfakes/