Rethinking Cross-Domain Evaluation for Face Forgery Detection with Semantic Fine-grained Alignment and Mixture-of-Experts

Face forgery detection has evolved from handcrafted feature methods to deep learning-driven paradigms. With the development of convolutional neural networks (CNNs), CNN-based methods became the mainstream of deepfake detection research. Chollet et al.[4] proposed Xception, which became the most widely used backbone for early forgery detection; Tan et al.[22] further put forward EfficientNet series for better accuracy-efficiency balance, and Lin et al.[14] applied dynamic data augmentation strategies. However, limited by the inherent local receptive field, CNNs fail to detect high-quality forgeries with subtle artifacts.

To address this limitation, Vision Transformer (ViT)[9] has gradually become the core backbone paradigm for face forgery detection. Subsequent studies optimized ViT for forgery detection tasks: Chen et al.[3] proposed a local relation learning framework to fuse RGB and frequency domain features; Luo et al.[15] devised a generalizable model to enhance high-frequency forgery feature capture. In recent years, cross-modal learning represented by CLIP[18] has become a key research direction, bringing breakthroughs to the interpretability and generalization of forgery detection. Khan et al. first introduced adaptive prompt tuning into CLIP for universal deepfake detection; Zhang et al.[28] designed fine-grained text prompts to help CLIP focus on subtle forgery artifacts; Zhang et al.[29] constructed the DD-VQA dataset to train a BLIP-based model for both authenticity judgment and interpretable explanation. Meanwhile, works[21; 27; 10] have been done to improve CLIP. These works verified the potential of ViT models, while existing methods still lack targeted optimization for facial regional heterogeneous feature extraction and semantically fine-grained detection.

Evaluation metrics are the core guidance for the development of face forgery detection technologies. Early research directly adopted general metrics including Accuracy, Precision, Recall and F1-score, which remain auxiliary indicators in most studies. However, due to the widespread class imbalance in forgery detection datasets, these threshold-sensitive metrics cannot comprehensively reflect the model’s overall detection ability.

To solve this problem, the Area Under the Receiver Operating Characteristic Curve (AUC) has gradually become the core evaluation metric in this field. AUC comprehensively quantifies the model’s ability to distinguish fake and real samples with strong robustness to imbalanced datasets. Rossler[19] et al. took AUC as the core metric in the FaceForensics++ benchmark, and it has become the most widely accepted standard. Subsequently, mainstream benchmarks including Celeb-DF[13], DFDC[7] and DFDCP[8] all adopted AUC as the primary evaluation indicator. Although researchers have supplemented a series of targeted metrics on the basis of AUC, the current evaluation system still relies heavily on intra-dataset AUC, lacking a unified and systematic metric to accurately quantify the model’s cross-domain generalization ability in real open scenarios.

Our framework is built upon the pre-trained CLIP model, which learns aligned visual–text representations from large-scale image–text data and provides strong semantic priors for visual understanding. However, directly applying CLIP to forgery detection has two limitations. First, CLIP performs image–text alignment mainly at the global level, which may overlook subtle local manipulation artifacts. Second, the vision encoder applies a shared transformation to all patches, lacking region-aware specialization for different facial components. To address these issues, we introduce two complementary modules. PaITA extends CLIP with patch-level image–text alignment to capture fine-grained manipulation cues, while FaRMoE introduces region-aware expert routing to model organ-specific forgery patterns.

To encourage the model to focus on manipulated facial regions during training, we introduce a mask-guided hybrid augmentation strategy that synthesizes diverse forged samples while providing explicit spatial supervision.

Given a real image $I_{r}$ and a fake image $I_{f}$, we generate a binary authenticity mask $M\in\{0,1\}^{H\times W}$ based on facial landmarks. Several facial regions (e.g., eyes, nose, mouth, or half-face) are randomly selected as manipulated regions. Using the mask, a synthetic forged image $\tilde{I}$ is constructed as

where $\odot$ denotes element-wise multiplication.

To further increase appearance diversity, we incorporate Self-Blended Images (SBI)[20]. A transformed version of the real image $T(I_{r})$ is blended with the original image using a smoothed mask $\tilde{M}$:

During training, the two augmentation strategies are randomly sampled to generate diverse forged samples. The generated mask $M$ explicitly indicates manipulated regions and will later be downsampled to the patch resolution to supervise patch-level alignment.

To capture region-specific forgery patterns, we introduce FaRMoE, a facial region Mixture-of-Experts module integrated into the CLIP vision encoder. Fig. 3 illustrates the structure of our targeted optimized vision encoder with FarMoE module.

Given an input image, the vision encoder divides it into $N$ patches as: $\{x_{1},x_{2},\dots,x_{N}\},\quad x_{i}\in\mathbb{R}^{d}.$ Using facial landmarks, each patch is assigned a region label $r_{i}\in\{1,\dots,K\}$, where $K$ denotes the number of facial regions. FaRMoE consists of a set of region experts $\{E_{1},\dots,E_{K}\}$ that specialize in modeling different facial components. Each patch feature is routed to the corresponding expert according to its region label:

We integrate FaRMoE into the CLIP vision encoder by replacing the key projection in selected self-attention layers:

This design enables the Transformer to capture organ-specific forgery artifacts while preserving global contextual modeling.

To further capture subtle manipulation cues, we extend CLIP’s global image–text alignment to a patch-level visual–semantic alignment framework. Given an input image, the vision encoder outputs a global feature $f_{cls}$ and patch features $\{f_{1},f_{2},\dots,f_{P}\}.$ On the text side, we construct two prompts describing authentic and forged content. Let $t_{real}$ and $t_{fake}$ denote their embeddings. The visual–text similarity is computed using cosine similarity

The forgery probability is then obtained as

Global prediction is computed from the class token $f_{cls}$, while patch-level probabilities $\{p_{i}\}$ are computed for each patch feature $f_{i}$. The augmentation mask is downsampled to the patch resolution to obtain patch labels $M\in\{0,1\}^{B\times P}$.

We first conduct ablation experiments on the three core components of our SFAM framework to explore the effect of each component. The experimental results are shown in Table IV. Key observations from the ablation results are summarized as follows: First, the CLIP baseline achieves the worst performance across all metrics, with a Cross-AUC Average of only 0.740 and a Cross-AUC Minimum of 0.601. This demonstrates that the CLIP model lacks optimization and cannot be directly applied to deepfake detection without targeted structural and training strategy modifications. Second, mask-guided hybrid data augmentation serves as the foundation for performance improvement. By adding this module, the Cross-AUC Average increases significantly from 0.740 to 0.861, and the Cross-AUC Minimum rises from 0.601 to 0.725. This improvement validates that the mask-guided hybrid data augmentation enriches the diversity of fake samples, and explicitly guides the model to focus on forged regions, thus fundamentally enhancing the model’s cross-domain generalization ability. Third, patch-level alignment is the core module for further cross-domain performance enhancement. On the basis of mask-based augmentation, adding patch-level alignment further lifts the Cross-AUC Average to 0.881 and Cross-AUC Minimum to 0.749. This result proves that the fine-grained cross-modal semantic matching between local image patches and regional text prompts enables the model to capture subtle local forgery cues that are overlooked by global-only alignment.

Finally, the full SFAM model with all three modules achieves the highest Original AUC Average of 0.886, while reaching a highest Cross-AUC Average of 0.885 and Cross-AUC Minimum of 0.747, which is nearly identical to Variant 2. This indicates that the FaRMoE module further enhances the model’s organ-specific fine-grained feature extraction and overall detection ability, without sacrificing cross-domain generalization performance or the lower bound of detection stability. In summary, the three core modules have a significant effect, and their addition continuously improves the model’s detection accuracy, cross-domain generalization ability, and practical deployment stability. All ablation results fully validate the rationality and necessity of each module’s design in our SFAM framework.

We also conduct experiments on the hyperparameters $\lambda_{1}$ and $\lambda_{2}$ of the loss function in section 3.4 to explore the best configuration. The experiment is conducted without the FaRMoE module and the results are shown in Table V. The results show that the combination of ($\lambda_{1}$=0.3 and $\lambda_{2}$=0.2) achieves the highest Original AUC Average (0.880) and Cross-AUC Average (0.881) among all tested configurations. The intra-image ranking loss $L_{rank\_intra}$ (weighted by $\lambda_{1}$) guides the model to distinguish forged and authentic regions within a single image, which is the foundation of detection ability. We assign a slightly higher weight of 0.3 to ensure the model can capture subtle local forgery cues. The cross-sample ranking loss $L_{rank\_real\_fake}$ (weighted by $\lambda_{2}$) helps the model learn universal forgery features and improve cross-domain generalization, so we set a weight of 0.2 to provide effective supervision without affecting the basic feature learning. The weight setting of ($\lambda_{1}$=0.3 and $\lambda_{2}$=0.2) achieves the best balance between the two ranking loss terms. All other weight configurations lead to obvious performance degradation. For configuration of ($\lambda_{1}$=0.2 and $\lambda_{2}$=0.3), the reversed weight allocation makes the model ignore fine-grained local forgery details, resulting in a sharp drop in overall performance. The equal weight setting fails to capture the different priorities of the two loss terms, and cannot reach the optimal performance. For the last configuration of ($\lambda_{1}$=0.35 and $\lambda_{2}$=0.15), excessive weight on intra-image loss makes the model overfit to local details of the training set, which damages the cross-domain generalization ability.

Besides, the optimal configuration has a negligible gap between Original AUC and Cross-AUC, which means the model achieves the best balance between closed-set detection performance and real-world cross-domain generalization. This perfectly matches the core design goal of our SFAM framework.

We perform t-distributed Stochastic Neighbor Embedding (t-SNE)[11] visualization on the feature representations learned by four methods: the CLIP baseline, Effort, Forensics Adapter, and our proposed SFAM framework, with the results presented in Figure 4. As observed from the figure, the features of real and forged images extracted by the CLIP baseline are severely intertwined with no clear discriminative boundary between the two categories. This indicates that the baseline fails to effectively distinguish between authentic and fake samples. In contrast, Effort and Forensics Adapter form relatively well-defined class-wise distributions in the feature space. However, the features extracted by these methods exhibit obvious inter-dataset distribution gaps: samples from different data sources form isolated clusters in the feature space. This phenomenon reveals that the models rely to dataset-specific biases, which may impair their generalization ability in real-world open scenarios.

Meanwhile, our proposed method exhibits superior cross-dataset generalization capability. Samples from different data sources are distributed more compactly in the feature space, while the clear separation between authentic and fake categories is well preserved. This demonstrates that our framework effectively mitigates the adverse effects of dataset bias and learns more generalizable forgery-related feature representations, thereby enhancing the model’s robustness in complex real-world deployment scenarios.