C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis

We employ the Qwen3-Omni-30B model as a teacher model to automatically generate CoT reasoning for each sample in the multimodal sentiment dataset. Specifically, by designing prompts, we guide this model to sequentially analyze emotional cues from the textual, audio, and visual modalities, progressively deducing the final sentiment judgment and outputting a regression score. The model adheres to a strict output format, generating sequences containing <polarity>, <think>, and <score> tags. Subsequently, we filter the generated CoT data pairs based on the rationality of the reasoning chain and the consistency between the predicted results and the ground-truth labels. Through this filtering mechanism, we retain high-quality data pairs and replace the generated score within them with the real score label from the dataset. This constructs a training dataset $\mathcal{D}_{\text{CoT}}$ that possesses both a reliable reasoning process and genuine supervisory signals.

By explicitly including the <polarity> tag within the generated reasoning chain, we obtain a reward signal verifiable early in the generated sequence. This design enables the RL policy to receive immediate feedback at the first step, specifically addressing potential initial polarity prediction errors, especially for hard samples. Based on this, for hard samples, we leverage the ground-truth <polarity> label as hint information during GRPO training to guide the model in generating samples that contribute positively to training during the sampling stage, thereby improving optimization efficiency.

The filtered dataset $\mathcal{D}_{\text{CoT}}$ is used for supervised fine-tuning of the Qwen2.5-Omni-7B base model. The standard next-token prediction loss is applied to the concatenated sequence of the reasoning chain and the final answer, conditioned on the multimodal input $x$:

$${\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{(x,y_{\text{cot}},y_{a})\sim\mathcal{D}_{\text{CoT}}}\left[\sum_{t}\log p_{\theta}(y_{t}|x,y_{<t})\right]}$$

(1)

where $x$ denotes the multimodal input, $y_{\text{cot}}$ the reasoning chain tokens, $y_{a}$ the answer tokens, and $\theta$ the model parameters. This stage produces a model $\pi_{\text{SFT}}$ capable of basic interpretable predictions.

The reward function comprises three components: format reward, polarity reward, and score reward, designed to guide the model to follow the specified output structure, ensure correct macro sentiment, and achieve accurate fine-grained score regression, respectively. The specific design is as follows:

$${R_{\text{total}}=\lambda_{1}R_{\text{format}}+\lambda_{2}R_{\text{polarity}}+\lambda_{3}R_{\text{score}}}$$

(2)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters used to balance the contribution of each reward component.

Format Reward ($R_{\text{format}}$): Ensures the model strictly adheres to the designed C2F structured reasoning, i.e., generating the complete sequence of <polarity>, <think>, <score> tags in order. Its calculation is as follows:

$${R_{\text{format}}=\begin{cases}1,&\text{if format is correst }\\ 0,&\text{if format is incorrest}\end{cases}}$$

(3)

Polarity Reward ($R_{\text{polarity}}$): This reward encourages the model to make correct macro sentiment polarity judgments. Its calculation is as follows:

$${R_{\text{polarity}}=\begin{cases}1,&\text{if }p_{\text{pred}}=p_{\text{true}}\\ 0,&\text{otherwise}\end{cases}}$$

(4)

where $p_{\text{pred}}$ and $p_{\text{true}}$ represent the predicted and true sentiment polarity (e.g., “positive”, “negative” or “neutral”).

Score Reward ($R_{\text{score}}$): Under the premise of consistent polarity, it encourages the predicted score to be close to the true value. Its design is as follows:

$${R_{\text{score}}=\begin{cases}1-\tanh\left(\frac{|s_{\text{pred}}-s_{\text{true}}|}{r_{\text{max}}-r_{\text{min}}}\right),&\text{if }p_{\text{pred}}=p_{\text{true}}\\ 0,&\text{otherwise}\end{cases}}$$

(5)

where $s_{\text{pred}}$ and $s_{\text{true}}$ represent the predicted and true sentiment scores, and $r_{\text{max}}$ and $r_{\text{min}}$ are the upper and lower bounds of score annotations in the dataset.

We employ the GRPO algorithm for RL training of the model. GRPO is an online learning algorithm whose core mechanism is to generate multiple candidate outputs for the same prompt and provide optimization signals through intra-group relative comparisons. Specifically, in each training step, given an input prompt $x$, the current policy model $\pi_{\theta}$ generates a group containing $G$ candidate sequences, denoted as $\{o_{i}\}^{G}_{i=1}$. Subsequently, the total reward $r_{i}$ for each sequence $o_{i}$ is calculated according to the reward function defined above. To enable “relative” comparison and compute optimization signals at each token level, GRPO normalizes the raw rewards within the group to obtain the advantage for each sequence: $\hat{A}_{i}=\frac{r_{i}-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$. The final optimization objective is defined by the following loss function:

$\displaystyle\begin{aligned} \mathcal{L}_{\text{GRPO}}(\theta)=-\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{D},\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot|x)\end{subarray}}&\left[\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\left(\hat{A}_{i,t}\log\pi_{\theta}(o_{i,t}|x,o_{i,<t})\right.\right.\\ &\quad\left.\left.-\beta\mathbb{D}_{\text{KL}}[\pi_{\theta}(\cdot|x,o_{i,<t})\|\pi_{\text{ref}}(\cdot|x,o_{i,<t})]\right)\right]\end{aligned}$

(6)

where $\mathbb{D}_{\text{KL}}[\pi_{\theta}\|\pi_{\text{ref}}]$ is the KL divergence between the current policy and the reference policy $\pi_{\text{ref}}$, with coefficient $\beta$ controlling the conservativeness of policy updates.

Our C2F-Thinker aims to solve the problem of sparse rewards when training on hard samples. We provide polarity hints only for hard samples. First, we have the model generate candidate outputs. If the rewards for all outputs fall below a threshold, meaning all are incorrect, the sample is deemed a hard sample. Subsequently, we provide the ground-truth sentiment polarity label (e.g., positive, negative, or neutral) to the model as a hint. This hint guides the model to regenerate the output from the correct starting point. This hint is implemented via in-answer injection, the polarity label is pre-positioned at the beginning of the output sequence, allowing the model to continue writing the reasoning chain and score based on it, thereby ensuring consistency between training and inference input formats.

For hard samples, the hint provides crucial initial correct information, enabling the model to produce partially correct output sequences, thereby obtaining non-zero rewards and effective gradients. This mechanism significantly improves data utilization for hard samples while preventing simple samples from skipping necessary reasoning due to excessive prompting. In this way, the model continuously receives stable optimization signals during reinforcement learning, ultimately improving overall reasoning performance and prediction accuracy.

The CMU-MOSI dataset consists of 93 YouTube videos segmented into 2,199 clips. Each clip is annotated with an emotion score on a 7-point scale from strongly negative (-3) to strongly positive (+3). Similarly, the CMU-MOSEI dataset contains 23,453 video clips collected from various online platforms and adopts the same annotation scheme.

The CH-SIMS dataset includes 2,281 carefully selected video clips sourced from movies, TV series, and variety shows. Each clip is annotated with an emotion score ranging from negative (-1) to positive (+1). Building on this, the CH-SIMSv2 dataset expands the corpus to include 4,402 supervised clips and 10,161 unsupervised clips (totaling 14,563 clips) from 11 different scenarios, such as vlogs, interviews, and talk shows. While retaining the original annotation methodology, it places greater emphasis on rich non-verbal behaviors.

To verify the effectiveness of the two-stage progressive training framework, we compared the performance of the base model under zero-shot prompting, after first-stage SFT, and after second-stage Hint-guided GRPO reinforcement learning optimization. The results are presented in Table V.

Under the zero-shot setting, the model followed only instruction prompts to generate structured outputs, but all metrics remained low. Despite its general multimodal capabilities, the base model struggled to produce high-quality reasoning chains without fine-tuning, often suffering from format errors or logical inconsistencies, especially under noisy audio or visual inputs.

After first-stage SFT, performance improved substantially. Compared to zero-shot, Acc5 increased by 16.2 percentage points, Acc3 by 21.3 points, Acc2 by 16.7 points and F1 score by 16.1 points. These gains demonstrate that supervised fine-tuning on teacher-generated CoT data enables the model to master structured sentiment reasoning, establish reliable polarity and score predictions, and provide a strong initial policy for reinforcement learning.

After second-stage Hint-guided GRPO optimization, performance improved further. Relative to the SFT version, Acc5 rose by 1.6 points, Acc3 by 1.1 points, Acc2 by 0.8 points, and F1 score by 0.9 points. This confirms that Hint-guided GRPO effectively alleviates reward sparsity in continuous sentiment space by providing polarity hints for hard samples, enabling more efficient policy exploration. Notably, GRPO outperformed SFT across all metrics, indicating that reinforcement learning offers more than repetitive fine-tuning. It enhances discrimination on complex samples, validating the synergy of the two-stage progressive training framework.

To validate the core role of directional hints in the Hint-guided GRPO mechanism, we conducted an ablation experiment comparing model performance with hints (w/ Hint) and without hints (w/o Hint). Table VI reports the results.

It is noteworthy that under the standard GRPO setting without hints, model performance experienced a comprehensive decline compared to the first-stage SFT version. This phenomenon suggests that when the model encounters a large number of challenging samples during training, the extremely low reward signals generated by these samples can lead to chaotic policy gradient directions. This causes the optimization process to fall into local oscillations or even regress, ultimately harming the model’s existing performance.

After introducing Hint guidance, model performance was significantly restored and improved. Compared to the w/o Hint version, Acc5 increased by 8.1 percentage points, Acc3 by 10.2 points, Acc2 by 5.0 points, and F1 score by 6.4 points. Moreover, the model with hints comprehensively surpassed the first-stage SFT version, validating the effectiveness of the Hint mechanism. This demonstrates that by providing guiding hints during the polarity discrimination phase for challenging samples, Hint-guided GRPO effectively constrains the exploration range of reinforcement learning within a reasonable region. It transforms potentially harmful noise gradients into beneficial policy updates, thereby enabling continuous optimization of fine-grained sentiment prediction accuracy while maintaining interpretable reasoning.

To investigate the role of challenging samples in reinforcement learning training, we conducted a further ablation experiment comparing two training strategies: GRPO optimization using only easy samples (w/o Hard), and training that included challenging samples with Hint guidance (w/ Hard). The results are shown in Table VII.

The experimental results indicate that completely excluding challenging samples during the reinforcement learning phase leads to a significant performance gap compared to the version that includes challenging samples supplemented with Hint guidance. Specifically, the w/o Hard version achieved an Acc3 of 77.0%, slightly lower than the w/ Hard version’s 77.2%; Acc2 and F1 scores were also lower than those of the w/ Hard version; the difference in regression metrics was particularly pronounced: MAE increased from 0.331 to 0.447, and the correlation coefficient decreased from 0.780 to 0.715. This phenomenon suggests that relying solely on easy samples for optimization, while avoiding the risk of reward sparsity from challenging samples, prevents the model from learning the ability to distinguish complex sentiment boundaries from more difficult instances, thereby limiting its generalization performance.

To more intuitively demonstrate the optimization process of Hint-guided GRPO, we plotted the reward curves during training in Figure 3. As shown in the figure, in the early stages of training, the reward values for the w/ Hint and w/o Hint versions were similar. As training progressed, the w/o Hint version, lacking directional guidance, exhibited severe reward fluctuations and slow growth when processing challenging samples. In contrast, the reward curve for the w/ Hint version maintained a steady upward trend, converged faster, and ultimately reached a higher stable value. Meanwhile, the w/o Hard version, while avoiding the risk of reward sparsity from challenging samples, showed a relatively stable reward curve with minor fluctuations. However, relying solely on easy samples for optimization prevented the model from learning to distinguish complex sentiment boundaries from more challenging instances, resulting in insufficient exploration and a limited performance ceiling.

Combining this result with the previous ablation experiment (where standard GRPO without hints led to a comprehensive performance decline) yields a more complete conclusion. In reinforcement learning, simply discarding challenging samples is not optimal, and introducing them without guidance can also harm performance. Only by providing effective directional guidance via the Hint mechanism can challenging samples be transformed from an optimization burden into a key performance gain. This enables the model to continuously improve fine-grained sentiment prediction accuracy while maintaining interpretable reasoning, fully validating the necessity and effectiveness of the proposed Hint-guided GRPO mechanism in handling challenging samples.