Micro-DualNet: Dual-Path Spatio–Temporal Network for
Micro-Action Recognition

As shown in Fig. 2, given an input video $\mathbf{V}\in\mathbb{R}^{T\times H\times W\times 3}$ with $T$ frames, height $H$, width $W$, and 3 color channels, along with corresponding body joints, our framework processes micro-actions through four components: (1) a Spatial Entity Module (SEM) that extracts anatomically-grounded entity representations from Convolutional Neural Network (CNN) features, (2) dual Spatial-Temporal (ST) and Temporal-Spatial (TS) pathways that capture complementary spatio-temporal patterns, (3) entity-level adaptive routing that learns per-entity processing preferences, and (4) Mutual Action Consistency (MAC) loss that enforces cross-path coherence while preserving specialized processing.

For each frame $t\in\{1,...,T\}$, we extract features $\mathbf{F}_{t}\in\mathbb{R}^{C\times H^{\prime}\times W^{\prime}}$ from the penultimate layer of ResNet-101 with Temporal Shift Module (TSM) [24], where $C=2048$, $H^{\prime}=H/32$, $W^{\prime}=W/32$. We also use 25 keypoints (joint coordinates) represented as $\{(\mathbf{k}_{j},c_{j})\}_{j=1}^{25}$ where $\mathbf{k}_{j}\in\mathbb{R}^{2}$ denotes pixel coordinates and $c_{j}\in[0,1]$ indicates detection confidence. Key points can be generated by any human pose detection architecture. In the current study, we used OpenPose [2] for this purpose.

We define $K$ anatomical groupings using key points. For MA-52, we use $K=6$ entities (head, face, left_hand, right_hand, torso, lower_body) to capture whole-body micro-actions. For iMiGUE focusing on upper-body micro-gestures, we use $K=5$ entities, excluding lower_body as the dataset contains seated subjects. Please see Sec. A in Supplementary Material (Supp.) for detailed body joint-to-entity mappings.

Entity bounding boxes are computed dynamically as:

where $\mathcal{J}_{i}$ denotes keypoint indices for entity $i$, $\mathcal{V}_{i}=\{j\in\mathcal{J}_{i}|c_{j}>\theta\}$ are visible keypoints with confidence threshold $\theta=0.3$, and $\text{BBox}(\cdot)$ computes the minimum enclosing rectangle with 10% padding.

Entity features are extracted via ROIAlign [15] followed by entity-specific refinement:

where ROIAlign extracts fixed-size features from arbitrary bounding box regions $\mathbf{B}_{i,t}$ using bilinear interpolation, avoiding the quantization artifacts of ROI pooling; $\mathcal{E}_{i}$ consists of depthwise separable convolutions projecting to $D=256$ dimensions, and $\mathbf{p}_{i}\in\mathbb{R}^{D}$ is a learnable position embedding encoding entity identity.

Given entity features $\mathbf{X}\in\mathbb{R}^{B\times T\times K\times D}$ where $B$ is batch size, $T$ is number of frames, $K$ is number of spatial entities, and $D=256$ is feature dimension, we construct dual paths to capture complementary spatio-temporal patterns in micro-actions.

To ensure the dual paths learn consistent representations while maintaining their complementary strengths, we employ entity-aware contrastive learning between the ST and TS pathways.

For each entity $i$ and temporal position $t$, we enforce that representations from both paths align for the same spatio-temporal location while contrasting with different temporal positions:

where $\mathbf{z}_{i,t}^{ST},\mathbf{z}_{i,t}^{TS}$ are $\ell_{2}$-normalized features for entity $i$ at time $t$ from respective paths, and $\tau=0.07$ is the temperature parameter. This formulation ensures that while both paths process the same entity differently (spatial-first vs temporal-first), they maintain agreement about which temporal segments are most relevant for each entity. The total MAC loss aggregates across all visible entities and frames:

where $c_{i}^{t}$ represents the keypoint confidence for entity $i$ at time $t$, naturally down-weighting occluded or uncertain entities. Note that MAC loss operates on the raw path outputs $\mathbf{X}^{ST}$ and $\mathbf{X}^{TS}$ before adaptive routing (section III-C4), providing a training signal that encourages temporal coherence between paths while allowing the routing module to independently learn entity-specific fusion strategies.

Given the dual-path outputs $\mathbf{X}^{ST},\mathbf{X}^{TS}\in\mathbb{R}^{B\times T\times K\times D}$, we apply entity-level adaptive routing to obtain fused representations where each entity $i$ is combined according to its learned preferences:

where $\mathbf{X}_{i}^{\text{fused}}$ is computed via Eq. (13)-(15). The video-level entity representation is obtained by averaging across temporal and entity dimensions:

This is concatenated with global appearance features for final classification:

where $\mathbf{f}_{\text{CNN}}\in\mathbb{R}^{C}$ represents global context obtained via Global Average Pooling (GAP) over the CNN feature maps, capturing scene-level information that complements localized entity features. The model is trained with classification and consistency objectives:

where $\mathcal{C}$ is a two-layer MLP classifier, $\mathcal{L}_{\text{CE}}$ is cross-entropy loss, $y$ denotes micro-action labels, and $\lambda=0.1$ balances the objectives. Importantly, $\mathcal{L}_{\text{MAC}}$ is computed on raw path outputs before routing, ensuring both paths receive gradient signals regardless of learned routing preferences. This design separates representation learning (via MAC) from adaptive fusion (via routing), allowing each component to fulfill its distinct role.

Architecture Configuration. We employ ResNet-101 with TSM [24] as our backbone, pretrained on Kinetics-400 [17]. SEM extracts $K=6$ entities for MA-52 and $K=5$ entities for iMiGUE datasets with dimension $D=256$ each. Both ST and TS paths use 3-layer transformers with 8 attention heads, hidden dimension 1024, and dropout 0.1. For entity-level adaptive routing, each of the $K$ entities has a dedicated routing network $\mathcal{R}_{i}$ consisting of two linear layers ($2D\rightarrow D/2\rightarrow 2$, i.e., $512\rightarrow 128\rightarrow 2$) with LayerNorm, ReLU activation, and dropout 0.1. The learnable entity-type bias $\mathbf{b}_{i}$ is initialized to zero. The routing temperature $\tau=1.0$ provides soft routing that allows continuous blending between paths. This module adds approximately 0.5M parameters ($\sim$3% overhead). The final classifier uses a 2-layer MLP (512→256→$C$) with GeLU [16] activation and dropout 0.5, where $C$ is the number of action classes (52 for MA-52, 32 for iMiGUE as shown in Fig. 4). For MAC loss computation, we use temperature $\tau=0.07$ and $\lambda_{MAC}=0.1$.

Table I compares our method with state-of-the-art approaches. On MA-52, we achieve 68.72% F1${}_{\text{mean}}$, competitive with PCAN [20] (69.97%) while using simple end-to-end training instead of PCAN’s complex 3-stage pipeline. On iMiGUE, we achieve state-of-the-art 76.88% Top-1 accuracy, surpassing PoseConv3D [7] by 12.50%.

Our adaptive keypoint-guided entities outperform MANet’s fixed regions [14] by 3.13% F1${}_{\text{mean}}$. While 3D CNNs (SlowFast [8]: 63.09%) and Transformers (UniFormer [22]: 64.43%) achieve reasonable performance, they require substantially more computation. Pure skeleton methods struggle—CTR-GCN [5] achieves only 56.48%, confirming that appearance cues are indispensable for micro-actions.

The bottom rows of table I show modality ablations: pose-only (65.20%) surpasses all skeleton baselines, RGB-only (67.11%) demonstrates entity-aware processing benefits, and their fusion (68.72%) confirms complementarity. Even single-modality variants outperform heavier baselines, validating our dual-path design and MAC regularization. The 12.50% improvement on iMiGUE versus 3.13% on MA-52 reveals that adaptive entity extraction particularly excels for concentrated upper-body micro-gestures, where our keypoint guidance maintains semantic alignment across viewpoints while fixed regions fail.

To evaluate the potential clinical utility, we applied Micro-DualNet to an in-house dataset of 290 individuals (ages 5–52) recorded during 2–3 minute conversations with a research staff member [The study which includes this dataset was reviewed and approved by the Institutional Review Board at CHOP.]. Participants received licensed psychologist-supervised diagnostic evaluations and were classified into three groups: ASD (autism spectrum disorder, $n$=120), PSY (non-autistic psychiatric conditions, $n$=46), and TDC (typically developing, $n$=124). For the ten most frequent micro-actions, we conducted pairwise group comparisons using two-part (hurdle) analysis: (a) probability of engagement via logistic GLM (Prob.), and (b) intensity among engagers via fractional logit (Int.). Table V summarizes significant group differences. Notably, the PSY group showed elevated “retracting feet” intensity compared to both ASD ($p<0.001$) and TDC ($p<0.01$), while “turning head” intensity was lower in PSY than both ASD and TDC ($p<0.05$). Fig. 3 illustrates intensity distributions for these two micro-actions; see Supp. for all comparisons. Although analyses controlling for demographics are needed for definitive interpretation, these results provide initial evidence that micro-action detection can be used to identify behavioral differences in psychiatric conditions as part of a larger computational behavior analysis research.

t-SNE visualizations (Fig. 4) show that single-path models yield complementary patterns—ST groups position-based actions while TS separates motion-based ones and Micro-DualNet combines these strengths with improved overall clustering. This aligns with Fig. 5: Micro-DualNet shows modest 3% gains on easy actions but 31% improvement on hard actions, suggesting that complementary entity-centric processing most benefits challenging micro-actions where single paths struggle.