Combating Digital Forgeries: Advanced AI Techniques for Detecting Forgeries of Diverse Data

Digital forgery

Forgery is the act of deliberately altering, creating, or imitating documents, signatures, artworks, or other items with the intent to deceive, mislead, or defraud. The rise of powerful editing tools and Artificial Intelligence (AI)-generated content has made it easier than ever to create sophisticated forgeries that can deceive even trained professionals. Among the most concerning types of digital forgeries are those involving identity documents, general images, and deepfake videos as shown in the following Figure 1. These forgeries pose significant risks, including identity theft, misinformation, financial fraud, and lack of trust in digital media.

• Identity Document Forgeries: Forged identity documents can facilitate a wide range of criminal activities, from illegal immigration to financial fraud. The ability to detect these forgeries accurately and swiftly is crucial for maintaining security and trust in digital and physical identification processes.

• General Forgery Images: Manipulated images are prevalent in digital platforms, often used to spread false information, manipulate public opinion, or tarnish reputations. Detecting these forgeries is essential to uphold the integrity of visual information and prevent the harmful consequences of misinformation.

• Deepfake Videos: Deepfakes represent one of the most challenging types of digital forgeries. These AI-generated videos are convincingly altered appearances and speeches of individuals, creating realistic but entirely fake content. Deepfakes have been used for malicious purposes, including political manipulation, blackmail, and misinformation campaigns. As deepfake technology continues to advance, the need for effective detection methods becomes increasingly urgent.

Given the diverse nature and serious implications of these forgeries, there is an immediate need to develop advanced AI techniques capable of detecting forgeries across different types of data. Traditional methods are often inadequate against the sophistication of modern forgeries. Thus, leveraging state-of-the-art AI models offers a promising solution.

Forgery detection system

Structured workflow for detecting and managing digital forgeries, enhancing the reliability of digital media is shown in Figure 2. The major steps involved are

• Collecting digital media (images or videos) from various sources such as standard databases, social media platforms, news websites, or personal archives.

• Converting data to a consistent format to carry out experiments, i.e., normalization and standardization of data. The common preprocessing techniques involve remove noise and artifacts, resizing images, and standardizing video frames.

• Identifying and extracting relevant features from the media.

• Applying machine learning or deep learning models to the extracted features to detect anomalies or inconsistencies that indicate forgery.

• Cross-validating the results using multiple models or techniques to ensure robustness.

Motivation

The motivation for this research is rooted in the desire to leverage advanced AI techniques to protect against the multifaceted threats of digital forgeries. By addressing the challenges posed by identity document forgeries, general forgery images, and deepfake videos, developing a robust and comprehensive framework for forgery detection, we address the following:

• Enhance security measures for identity verification processes by accurately detecting forged identity documents.

• Combat the spread of misinformation by identifying manipulated images.

• Protect individuals and institutions from the threats posed by deepfake videos.

Document image forgery detection

Morphing attacks present a significant threat to face recognition systems, particularly in border control scenarios. These attacks exploit the vulnerabilities of facial recognition systems by altering facial features, making it difficult to distinguish between legitimate and morphed images. To combat this, various morphing attack detection (MAD) algorithms and performance metrics have been developed, offering innovative solutions to detect and reject morphed images during enrollment or verification [4]. One approach utilizes patch-level features and lightweight networks, which significantly improve detection accuracy while maintaining computational efficiency and robustness [5]. Another method enhances detection by analyzing RGB color channels and using error-level analysis combined with selective kernel networks. This approach captures critical feature differences and improves classification accuracy without adding excessive parameters [6].

Despite advances in supervised detection methods, they often struggle with diverse attacks due to limited dataset diversity. To address this, an unsupervised solution using self-paced anomaly detection and convolutional autoencoders has been proposed [7]. This method analyses reconstruction behaviours in attack and bona fide samples, allowing better generalization to unknown attacks. Additionally, new neural network training schemes enhance robustness by modifying training data to improve generality and resistance to morphing attacks [8]. By employing layer-wise relevance propagation, these training schemes ensure more reliable models across all image regions, essential for security applications. Overall, these advancements in detection techniques offer promising solutions for enhancing the security of face recognition systems.

Face morphing attacks compromise face recognition systems (FRS), hence the need for development of MAD techniques. The fairness of six single image-based MAD (S-MAD) methods across four ethnic groups was evaluated in [9] and bias was revealed through fairness discrepancy rate (FDR) underscoring the need for more equitable MAD approaches. A survey [10] reviewed progress in face morphing, covering image generation techniques and state-of-the-art MAD algorithms in the context of public databases, vulnerability assessments, benchmarking, and future challenges in biometrics.

Image forgery detection

Traditional approaches to tampered image detection often involve extracting handcrafted features that capture specific properties of images, such as texture and color. These features are then used to train machine learning classifiers to distinguish between authentic and tampered images. In a review of recent deep learning (DL) techniques [11] for detecting image forgeries, particularly copy-move and splicing attacks, as well as deepfake content, the authors highlighted the superior performance of DL methods on benchmark datasets and possible future research directions in DL architectures, evaluation methods, and dataset development.

Image splicing is detected by analyzing and characterizing texture content within different regions of an image [12]. Detection of copy-move and splicing forgeries has been described in [13] by analyzing texture changes in the Cb and Cr components of YCbCr images using a standard deviation filter and higher-order texture descriptors, followed by SVM classification. A study [14] explored feature extraction and classification techniques for detecting multimedia forgery, using a combination of GLCM, GLDM, GLRLM, and Local Binary Pattern (LBP) features. Transform domain techniques combined with LBP for tampered image detection using machine learning techniques were proposed in [15]. Utilizatiob of Haralick-based textural descriptors and Extreme Learning Machine (ELM) to detect image tampering by applying LBP on YCbCr color bands and classifying features with ELM has been described in [16]. Segmentation-based fractal texture analysis (SFTA) with K-nearest neighbors (KNN) was proposed for copy-move forgery detection in [17]. Extracting and combining SIFT and KAZE features from the HSV color channels for detecting copy-move forgery (CMF) was proposed in [18]. Spliced image forgery detection using adaptive over-segmentation combined with AKAZE, ORB, and SIFT feature descriptors was proposed in [19]. Detecting and localizing CMF by applying SIFT key points on CLAHE-enhanced sub-images in RGB and Lab^∗ color spaces was presented in [20]. A CMF detection method using color features and hierarchical feature point matching was presented in [21].

Deepfake forgery detection

Various digital image tamper detection methods, focusing on the challenges posed by advanced editing tools and DL techniques like GANs, analyze both traditional and DL approaches for detecting image forgeries, their strengths and limitations are elucidated in [22]. A comprehensive review [23] of deepfake detection techniques using DL algorithms categorizes the methods of video, image, audio, and hybrid multimedia tamper detection. It also posits how deepfakes are created and detected, recent advancements, weaknesses in current security methods, and areas for further research. The results highlight convolutional neural networks (CNN) as the most commonly used DL approach, with video deepfake detection being the primary focus, particularly on improving accuracy. The use of DL in creating and detecting deepfakes, which are often difficult to distinguish from real images has been explored in [24]. With growing concerns about privacy and security, various deepfake detection methods have emerged and a DL-based image enhancement technique to improve the quality of generated deepfakes has also been proposed.

A novel framework for detecting fake videos using transfer learning with autoencoders and a hybrid model of CNN and recurrent neural networks (RNN) is worth noting [25]. Convolutional vision transformer for detecting deepfakes, combining CNN for feature extraction with vision transformers (ViT) for classification using an attention mechanism has been proposed [26]. Robust methods for detecting deepfake videos by analyzing teeth and mouth movements, which are challenging to replicate accurately are presented in [27] by incorporating multi-transfer learning techniques with models such as DenseNet121, DenseNet169, EfficientNetB0, EfficientNetB7, InceptionV3, MobileNet, ResNet50, VGG16, VGG19, and Xception. Detecting deepfake videos using CNNs with transfer learning has been explored in [28].

Using the recent advancements in ViT for face forgery detection, a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules, enabling the model to extract global and local forgery clues simultaneously was proposed in [29]. Generalized multi-scenario deepfake detection framework (GM-DF) presented in [30] showed strong performance across five datasets on unseen scenarios. GM-DF uses hybrid expert modelling for domain-specific features, CLIP for cross-domain alignment, and a masked reconstruction mechanism to capture detailed forgeries. Latent space data augmentation model proposed in [31], expanded the forgery feature space by simulating diverse forgeries within the latent space. This approach enhanced domain-specific features and smoothed transitions across forgery types, resulting in a robust, generalizable deepfake detector that outperformed state-of-the-art models on multiple benchmarks. A framework that captures broader forgery clues by combining diverse local representations into a global feature, guided by local and global information integrity principles to optimize feature relevance was proposed in [32]. Deepfake detection network combining spatial and noise domains proposed in [33], used a dual feature fusion module and a local enhancement attention module, which captured complementary forgery features, achieving superior performance across mainstream datasets.

Dataset description

Dataset1 (Aadhaar Card Images Dataset): The images in the dataset were sourced from roboflow.com platform [34] to create a custom dataset for research purposes [35]. These images were resized from 640 × 640 to 300 × 300 pixels to standardize their dimensions. Additionally, faces within the Aadhaar card images were morphed using faceswapper.ai tool [36], ensuring that the identities of individuals in the images remain unidentifiable. In order to further enhance the dataset’s diversity, various image augmentation techniques were applied to generate 3085 images consisting of 1532 real images and 1553 morphed images for use in our study.

Dataset2 (CASIA 2.0 image tampering detection): This dataset [37] consists of 7492 authentic images; 5125 tampered images and 5123 ground truth images. Input images are of resolution 384 × 256 or 256 × 384. Authentic images are of 8 different categories: animal, architecture, art, characters, indoor, nature, plants, and texture. Tampered images comprise spliced and copy-move forgeries. The database is made publicly available in Kaggle for researchers to compare and evaluate their proposed tampering detection techniques.

Dataset3 (Small-scale Deepfake Forgery Video Dataset-SDFVD): This custom dataset, which includes both actual and deepfake videos in a variety of scenarios, was developed to research and evaluate deepfake detection algorithms [38]. There are 106 videos in all, 53 of which are original and 53 of which are deepfakes. The original videos were obtained from Pexels, a popular source of stock footage and photos [39]. These videos depict a wide range of events with persons of all ages and genders from diverse origins. Remaker AI [40] was used to create the deepfake videos by employing face-swapping techniques.

Traditional techniques

Traditional approaches for forgery detection such as texture-based and color-based, often rely on analyzing the intrinsic properties of digital media, leveraging various statistical and structural characteristics. Texture-based techniques are widely used due to their effectiveness in capturing fine details and patterns that can indicate tampering, whereas color-based techniques are essential for detecting image forgeries as they analyze the color properties and distributions within an image.

AI techniques

AI techniques have proven to be highly effective in identifying tampered images. This approach typically involves two primary steps: feature extraction and classification. Pretrained models such as ResNet50, VGG16, and EfficientNetB0 are employed for feature extraction, and CNNs are utilized for classification.

3.3.1.

Feature extraction with pretrained models

Pretrained models like ResNet-50, VGG16, and EfficientNetB0 are CNNs trained on large datasets (e.g., ImageNet). When using these models for feature extraction, we remove the final classification layers and use the output of the last convolutional layer or a fully connected layer (before softmax layer) as features. Computations at convolutional layer, max pooling layer and then feature concatenation are expressed as

(13)

where:

• X_{i, j, c} is the input image pixel at position (i, j) in channel c,

• W_{m, n, c, k} is the filter weight at position (m, n) for channel ccc and filter k,

• b_k is the bias for filter k,

• Z_{i, j, k} is the output feature map at position (i, j) for filter k.

  (14)

where P_{i, j, k} is the pooled feature map.

After the convolutional and pooling layers, the feature maps are flattened into a 1D feature vector.

(15)

where ∥ denotes concatenation.

3.3.1.1.

ResNet50

ResNet50 has a deep CNN architecture as shown in the following Figure 3 and is widely used for image classification tasks, for its depth and efficiency in learning complex patterns [48]. Some of the key components are given below.

• Residual Blocks: Introduces skip connections (shortcuts) that allow the network to learn residual mappings instead of directly fitting to desired mappings. This helps in tackling the vanishing gradient problem during training.

• Convolutional Layers: Uses filters to convolve over the input data, extracting features at multiple levels of abstraction.

• Pooling Layers: Performs downsampling to reduce the spatial dimensions of the input, aiding in feature extraction and computational efficiency.

• Fully Connected Layers: Integrate extracted features and produce the final classification output.

3.3.1.2.

Visual Geometry Group16 (VGG16)

VGG16 is a deep CNN [50] known for its high accuracy in image classification. The architecture of vgg16 is given in Figure 4 and the key components are as follows.

• Convolution layer applies filters that perform a dot product with areas of the input image to produce an output.

• Pooling layer reduces dimensionality and complexity by down-sampling the activation map, which helps to speed up processing and prevent overfitting.

• Max Pooling selects the maximum value from each region in the input image, similar to the convolution layer, but focusing on reducing the size of the data.

• Softmax layer outputs a probability distribution over the labels, representing the likelihood of each label being correct.

3.3.1.3.

EfficientNet B0

EfficientNet B0 architecture is designed to achieve state-of-the-art performance while maintaining computational efficiency [52, 53]. Following are the key components of efficientnet B0.

• Compound Scaling to balance network depth (number of layers), width (number of channels), and resolution (input image size). This scaling strategy optimizes both accuracy and efficiency across different computational constraints.

• Convolutional Blocks uses a combination of depthwise separable convolutions and standard convolutions to extract features efficiently. Depthwise separable convolutions reduce the number of parameters and computational cost compared to traditional convolutions.

• Squeeze-and-Excitation (SE) Blocks improve feature representation by adaptively recalibrating the responses of channel-wise features. This enhances the model’s focus on informative features and boosts overall performance.

• Efficient Scaling refers to scaling the network architecture uniformly in all dimensions (depth, width, and resolution) based on a predefined compound scaling method, which ensures a balanced trade-off between accuracy and computational efficiency.

3.3.3.

Classification with CNNs

Once the features are extracted, the next step is to classify the image as either authentic or tampered. This is achieved using a CNN designed for binary classification [55]. The architecture of a CNN for binary classification is shown in Figure 5.

• The CNN consists of multiple convolutional layers, pooling layers, and fully connected layers that further process the feature vectors to learn distinguishing patterns.

• The final layer of the CNN is sigmoid layer which outputs the probability of the image being authentic or tampered.

• The CNN is trained using a labelled dataset of authentic and tampered images. The network learns to minimize the classification error by adjusting its weights through backpropagation.

Forgery detection using traditional techniques

This process typically includes texture-based and color-based feature extraction methods, followed by classification using machine learning algorithms as shown in Figure 6 and the steps involved are as follows:

Step 1: Read input data containing authentic and tampered images/videos (Aadhaar card images, CASIA 2.0, SDFVD).

Step 2: Apply preprocessing techniques to the input data such as standardize the size of images, converting them to grayscale (for texture-based techniques) and split input videos into frames.

Step 3: Extract features using following methods

• LBP: Compute the LBP for each pixel and generate histogram of LBP values.

• GLCM: Compute GLCM for specified distances and angles, extract properties such as contrast, dissimilarity, homogeneity, energy and correlation.

• Color histograms: Compute the histogram for each color channel (RGB) and normalize the histograms.

• RGB to HSV Conversion: Convert the image from RGB to HSV color space. Extract and normalize histograms for each channel (H, S, V).

Step 4: Generate feature vectors from LBP, GLCM, Color histograms and RGB to HSV Conversion separately and use them for further processing.

Step 5: The data is split into training and testing sets with 20% of the data reserved for testing.

Step 6: Use the extracted features for training the classifiers like Random Forest, SVM and GBM.

Step 7: Evaluate the models on the test sets to check the accuracy and performance with the help of metrics like precision, recall, F1 score and accuracy.

Forgery detection using AI techniques

This process typically includes feature extraction from pretrained models, followed by classification using DL algorithms as shown in Figure 7 and the steps are as follows:

Step 1: Read input data containing authentic and tampered images/videos (Aadhaar card images, CASIA 2.0, SDFVD).

Step 2: Apply preprocessing techniques to the input data such as resizing images to 224x224 pixels and split input videos into frames.

Step 3: Features are extracted from the images using pretrained models such as ResNet-50, VGG16 and EfficientnetB0 without the top classification layer. Later on, we have also ensembled these pretrained models for feature extraction and concatenated all the features to form a single feature vector for each image. PCA is applied to reduce the number of features.

Step 4: The data is divided into training and testing sets, with 20% of the data used for testing.

Step 5: A simple CNN classifier was constructed with an input layer for the feature vectors, two hidden layers using ReLU activation and dropout for regularization, and an output layer with a sigmoid activation function for binary classification. The model was compiled using the Adam optimizer and binary cross-entropy loss.

Step 6: Custom CNN classifier is built with: convolutional layers with 32, 64 and 128 filters, followed by max pooling layers, flattened the model, followed by dense layer by 256 and 128 units, dropout layers, an output layer with a sigmoid activation function for binary classification.

Step 7: Classification report is generated to evaluate the performance of the classifier, providing precision, recall, and F1 score for each class.

Step 8: Compare the performance of both simple and custom CNN models on individual features extracted as well as on ensembled features from the pretrained models.

Evaluation metrics

Accuracy, recall, precision, and F1 score are the assessment metrics that are used to gauge how well the suggested algorithm performs.

∙  Accuracy: It is determined by applying the following equation to the percentage of images that can be correctly identified.

∙  Precision: The algorithm’s positive predictions are accurate to a certain degree, which is measured by precision. In other words, it is the proportion to find positivity of the predictions by considering true positives and both true and false positives.

∙  Recall: Recall, also known as sensitivity or true positive rate, is a metric used to assess an algorithm’s ability to find all positive cases. It evaluates appropriately detected positive examples. Along with actual positive values, it also takes into account misleading negative values.

∙  F1 Score: It is calculated as the harmonic mean of recall and precision values. The value of F1 score ranges from 0 to 1, prediction values near to 1 means model is performing really well.

Experimental results

Detailed results and analysis of our forgery detection experiments using both traditional feature extraction techniques combined with machine learning classifiers, and advanced AI-based methods leveraging pretrained convolutional neural networks (CNNs) are summarized in Tables 1, 2 and 3 along with performance metrics Precision, Recall, F1 score and Accuracy for various classifiers SVM, RF, GBM applied to different traditional feature extraction techniques LBP, GLCM, Color Histogram, RGB to HSV conversion evaluated on three different datasets respectively.

Figure 8 illustrates the performance of the GBM classifier when applied to various feature extraction techniques on Dataset1. The comparative analysis highlights the effectiveness of each technique in detecting forgeries and shows that LBP provides good results. Figure 9 also presents a comprehensive comparison of the accuracy of different classifiers; SVM, RF, GBM across three datasets. It demonstrates how each classifier performs with traditional feature extraction techniques, providing insights into their relative strengths and weaknesses.

Tables 4, 5 and 6 detail the performance metrics of pretrained CNN models ResNet50, VGG16, EfficientNetB0 and custom CNN architectures, along with ensembled features with PCA, on dataset1, dataset2 and dataset3 respectively. Performance metrics include Precision, Recall, F1 score, and Accuracy. Comparative analysis of these techniques is represented in Figures 10, 11 and 12 respectively.

Table 7 presents a comparative analysis of the proposed system for forgery detection against existing methods described in the literature. The comparison covers different datasets, types of data such as images or videos, methods employed, and their corresponding detection results in terms of accuracy. The proposed system demonstrates superior performance compared to existing methods across different datasets and types of data. Its higher accuracy rates 95.23% for image datasets and 93.34% for video datasets highlight the effectiveness of combining ensembled features with PCA for forgery detection. This approach not only improves detection accuracy but also maintains computational efficiency, making it a robust solution for both image and video forgery detection tasks.

Role of AI tools in fraud detection

∙  Pretrained Models (ResNet-50, VGG16, EfficientNetB0) for Deep Feature Extraction: Pretrained models, originally trained on large-scale datasets like ImageNet, can extract deep, high-level features from images. Unlike traditional handcrafted features, these deep features capture complex patterns, textures, and structures within the images, which are critical for identifying subtle manipulations. For images and videos, pretrained models can distinguish between genuine and forged images by recognizing anomalies in the texture or layout that simpler techniques might overlook.

∙  Ensembling of Models for Enhanced Feature Representation: Ensembling different pretrained models, leveraging the strengths of each model, for instance, ResNet-50 may capture detailed spatial hierarchies, VGG16 may focus on edge and texture information, and EfficientNetB0 balances performance and computational efficiency. Combining these features provides a richer and more comprehensive representation of the image. In fraud detection involving normal images, such as detecting tampered photos on social media, an ensemble of these models would provide a robust detection system capable of identifying a wide range of manipulations, from minor to major alterations.

∙  CNN Classifier: After feature extraction, a custom CNN is employed to classify images as “fake” or “authentic.” The CNN architecture is specifically tailored to process the combined feature vector and make accurate predictions. For videos, the custom CNN can detect deepfakes by analyzing subtle inconsistencies in facial expressions, lighting, or shadows that may not align perfectly due to the manipulation.

Advantages over traditional methods

∙  Higher Accuracy and Robustness: AI tools, particularly DL models, have demonstrated superior accuracy in tasks like image recognition and classification compared to traditional methods (LBP, GLCM). They are better equipped to handle the complexity and variety of real-world data. Traditional methods might fail to detect sophisticated forgeries involving small, nuanced changes, but a DL-based approach can pick up on these subtleties due to its ability to learn complex patterns from the data.

∙  Adaptability: DL models can be fine-tuned on specific datasets, making them highly adaptable to different types of fraud detection tasks. This adaptability ensures that the models remain effective even as the nature of forgeries evolves. In a scenario where new types of document forgeries emerge, the AI models can be retrained or fine-tuned on updated datasets to quickly adapt to new fraud patterns.

∙  Automation and Scalability: AI tools automate the process of image identification and fraud detection, which is crucial for large-scale applications like scanning millions of images or videos on social media platforms. This automation reduces the need for manual inspection, making the process faster and more scalable. In the financial sector, AI tools can automatically scan thousands of transaction documents daily to detect any signs of forgery, significantly improving efficiency and accuracy.

Real-time scenarios

∙  Consider a scenario where government-issued IDs are being forged. The AI tools analyze the texture, layout, and fonts in the document images, detect inconsistencies, such as slight deviations in font style or spacing, which are indicative of forgery. This leads to the early detection of forged documents, preventing potential identity theft or fraud.

∙  On social media platforms, AI tools are used to detect fake images that have been manipulated for malicious purposes (e.g., spreading misinformation). Such AI applications can identify signs of tampering including unnatural lighting or shadows, that may not be detectable by the naked eye. The platform can automatically flag or remove these images, maintaining the integrity of the content shared with users.

∙  Deepfakes represent one of the most challenging forms of video manipulation. The AI tools analyze facial movements, expressions, and other subtle cues in the video. This helps in preventing the spread of malicious deepfake videos, which could be used for blackmail or misinformation.

By leveraging advanced AI tools like pretrained DL models and custom CNNs, the proposed system provides a powerful and scalable solution for fraud detection. These tools are well-suited to handle the complexities and evolving nature of digital forgeries, offering significant advantages in terms of accuracy, robustness, and adaptability. The examples provided illustrate how these tools can be effectively applied across different scenarios.