The question

Could preference optimization improve how a vision-language model reasons through a problem, not only the answer it eventually produces?

I explored that question using LLaVA, reinforcement learning from AI feedback, and Direct Preference Optimization. Along the way, I created a multimodal chain-of-thought preference dataset so the reasoning traces, not just final responses, could become part of the training signal.

What stayed interesting

The project surfaced limitations in applying DPO to multimodal reasoning. A preferred answer is not necessarily evidence of a better reasoning process, and vision-language models introduce failure modes that are easy to miss when evaluation only looks at the final text.

The dataset and experiments became a way to study that gap directly: how preference data is constructed, what the optimization actually rewards, and where apparent improvements stop being trustworthy.