ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability

Abstract

Reasoning models have focused heavily on accuracy and token efficiency, often overlooking trustworthiness. ReFIne is a training framework that makes large reasoning models more interpretable, faithful, and reliable without sacrificing utility. It uses a two-stage pipeline—supervised fine-tuning to learn a structured, tag-based reasoning format, followed by GRPO to reinforce structural compliance, explicit cross-section references, and calibrated self-assessed confidence. Across math benchmarks and model scales, ReFIne produces clearer traces, more transparent use of information, and more informative confidence estimates, while remaining competitive in accuracy and improving reasoning efficiency.

Figure 1: Overview comparing standard LRMs vs. ReFIne with gains in interpretability, faithfulness, and reliability.

Trustworthy Reasoning: Definition and Motivation

While prior works on LRM have largely emphasized accuracy and efficiency, we argue that a reasoning model is trustworthy only if it satisfies the following three dimensions:

• #1 Interpretability: The reasoning trace should be presented in a clear, well-organized structure that allows humans to easily follow.
• #2 Faithfulness: The reasoning trace should accurately reflect the actual process by which the model arrives at its answer.
• #3 Reliability: The model should perform an explicit self-assessment to judge whether each step of its derivation is rigorous and produce a well-calibrated estimate of the likelihood that its final answer is correct.

The ReFIne Framework

ReFIne trains models in two stages, combining supervised fine-tuning for structured output and RL for enhancing trustworthy behavior.

Stage 1. SFT for Structural Reasoning Format

During fine-tuning, the model learns to produce structured outputs that explicitly follow the ReFIne reasoning format. The output is organized into the following hierarchical sections:

• Preparation Phase:
  1. Problem Understanding, understanding: The model restates the problem in its own words, clarifying what is being asked.
  2. List Facts, facts: The model lists all known conditions, variables, and constraints.
  3. Stepwise Plan, plan: The model outlines a high-level roadmap for solving the problem.
• Reasoning Phase, think: The model performs detailed step-by-step derivations, explicitly referencing relevant parts from understanding, facts, and plan. If it changes its approach, it must identify and justify the switch.
• Answer Phase, final_answer: The model states the final result and includes a short justification.
• Evaluation Phase, self_assessment: The model audits the soundness of its reasoning and reports a confidence score (0–10) indicating belief in correctness.

This structured format is taught via supervised fine-tuning on curated data where each example adheres to this template.

Stage 2. GRPO for Enhancing Trustworthy Reasoning

In the second stage, Group Relative Policy Optimization (GRPO) refines the model using a multi-component reward function. For each prompt x and gold answer a, a generated trace y is scored on four dimensions:

• (1) Correctness: Reward for answer accuracy using a robust verifier Verify(y,a).
• (2) Structural Integrity: Reward for correctly producing all expected tags (understanding ... self_assessment) exactly once and in the right order.
• (3) Cross-Section References: Reward for explicitly referencing understanding, facts, and plan inside the think section.
• (4) Confidence Estimation: Reward based on how well the confidence score (0–10) in self_assessment aligns with actual correctness, and penalty if missing.

The total reward is the weighted sum of these components:

where all weights are set equally (\(\alpha=\beta=\gamma=\zeta=0.25\)). This balanced objective reinforces correctness, structural compliance, cross-section coherence, and reliable confidence reporting.

ReFIne Model Output Showcase

Experiments

We train three ReFIne models—ReFIne-Qwen3-1.7B, ReFIne-Qwen3-4B, and ReFIne-Qwen3-8B—using the two-stage pipeline described above. Each model undergoes supervised fine-tuning on 10k structured traces followed by GRPO reinforcement on 2k problems. For comparison, baseline models Plain-Qwen3-{1.7B, 4B, 8B} are trained with identical data and compute budgets but using normal reasoning traces and correctness-only rewards. All other training hyperparameters are held constant to isolate the effects of structured reasoning and multi-component rewards.

We evaluate performance on four mathematical reasoning benchmarks—AIME-2024, GPQA-Diamond, MATH-500, and GSM8K—spanning diverse difficulty levels. We assess models along five complementary dimensions: interpretability, faithfulness, reliability, accuracy, and efficiency.

1. Interpretability

ReFIne improves interpretability by producing structured reasoning that explicitly links sections and maintains coherence. We assess two metrics: Format & References and Readability.

Table 1 (Format & References): Percentage of think sections that explicitly reference understanding / facts / plan. GRPO substantially strengthens the cross-section referencing behavior..

Figure 2 (Readability): Readability comparison across datasets, judged by QwQ-32B. ReFIne produces clearer, easier-to-follow traces.

2. Faithfulness

Faithfulness measures whether reasoning traces genuinely reflect the internal solving process and stay grounded in disclosed context. We evaluate Disclosure Faithfulness (transparency about decisive information) and Commitment Faithfulness (consistency with self-declared premises).

Table 2 (Disclosure Faithfulness): ReFIne is more transparent to disclose any premise (hints in the question).

Table 3 (Commitment faithfulness): We report the fraction of traces where think strictly follows understanding / facts / plan. ReFIne models almost always follow their own prior commitments.

3. Reliability

Reliability quantifies the model’s self-awareness—its ability to verbalize a confidence estimate for its answer, and align those confidence values with actual correctness. We therefore assess reliability along Confidence Verbalization, AUROC (discrimination), and ECE (calibration).

Table 4 (Confidence Verbalization): Confidence verbalization rate. ReFIne nearly always reports confidence, unlike Plain.

Table 5 (Discrimination): AUROC, showing how well confidence distinguishes correct from incorrect answers. ReFIne achieves high AUROC. Plain models on AIME-2024 is marked in red since it rarely outputs confidence, making its AUROC unreliable.

Table 6 (Calibration): Expected Calibration Error (ECE). ReFIne demonstrates high confidence calibration. Plain models on AIME-2024 is marked in red since it rarely outputs confidence, making its ECE unreliable.

4. Accuracy and Efficiency

We further assess task-level utility, comparing final-answer accuracy and reasoning-token efficiency across datasets.

Figure 3 (Accuracy): Accuracy across benchmarks and scales. ReFIne matches Plain models' performance.

Figure 4 (Efficiency): Average reasoning length (tokens). ReFIne generates more concise reasoning traces.

Conclusion

We introduce ReFIne, the first training framework for Large Reasoning Models (LRMs) explicitly optimized for interpretability, faithfulness, and reliability—the three pillars of trustworthy reasoning. ReFIne improves interpretability by 44.0%, faithfulness by 18.8%, and reliability by 42.4% across four benchmarks and three model sizes, while maintaining similar accuracy and achieving 1.16× better reasoning efficiency.

Cite this work

Chung-En Sun, Ge Yan, Akshay Kulkarni, Tsui-Wei Weng. "ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability", arXiv 2025.

@inproceedings{refine,
  title     = {ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability},
  author    = {Sun, Chung-En and Yan, Ge and Kulkarni, Akshay and Weng, Tsui-Wei},
  booktitle = {arXiv},
  year      = {2025}
}

            

This webpage template was recycled from here.
Accessibility