SWIFT 多模态 GRPO 训练实战从 ClevrCount 到几何问答与 Open-R1 多模态数据集的完整实验流程【免费下载链接】swiftUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600 LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300 MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).项目地址: https://gitcode.com/GitHub_Trending/swift1/swift本文基于 SWIFTms-swift仓库中的多模态 GRPO 最佳实践文档系统讲解如何把视觉语言模型VLM接入 GRPO 强化学习流程以Qwen2.5-VL-3B-Instruct为基座覆盖自定义数据集预处理器、外部奖励函数插件、外部 vLLM rollout 部署以及三组完整可复现的训练命令ClevrCount 计数、GEOQA 几何问答、Open-R1 多模态推理数据集并给出每组的训练曲线观察结论。读完后你可以直接复制命令跑通多模态 GRPO 训练并理解num_generations、beta、max_grad_norm、MAX_PIXELS等关键参数的工程取舍。背景与整体流程GRPO 用同一问题采样多条 completion以组内奖励均值/标准差构造优势advantage相比 PPO 省去价值模型。对于多模态任务SWIFT 的 GRPO 管线需要解决三个环节数据集把「图片 问题 可验证答案」组织成images/messages/solution结构其中solution不进入模型输入而是直接透传给奖励函数奖励函数内置的format奖励保证think/answer输出格式自定义的准确性奖励如external_r1v_acc通过--external_plugins插件机制注册到orms字典rollout 与训练用独立的 vLLM 服务swift rollout加速采样训练侧用swift rlhf --rlhf_type grpo配合 DeepSpeed ZeRO-3 完成全参更新。这套流程的基座选择了Qwen2.5-VL-3B-Instruct。文档中明确说明选择Instruct版本而非 base 模型的原因指令微调过的模型能更快拿到 format 奖励让训练尽早进入准确性提升阶段。任务一ClevrCount 图像计数数据集定义ClevrCount 基于clevr_cogen_a_train数据集目标是让模型输出图中物体数量。由于原始数据集的 query 没有要求特定的思考/答案标签格式需要重写一个预处理器来改写 query。class ClevrPreprocessor(ResponsePreprocessor): def preprocess(self, row: Dict[str, Any]) - Dict[str, Any]: query row.get(query, ) query f{query} Output the thinking process in think /think and final answer (number) in answer /answer tags. row.update({query: query}) return super().preprocess(row) register_dataset( DatasetMeta( ms_dataset_idAI-ModelScope/clevr_cogen_a_train, subsets[ SubsetDataset( namedefault, subsetdefault, split[train], ), ], preprocess_funcClevrPreprocessor(), tags[qa, math]))从源码看ResponsePreprocessor定义在 preprocessor/core.py。它维护了三类列名映射system如system_prompt、query如question、problem、response含solution等。preprocess会把response/query/system等键弹出并转换为标准字段其余字段包括自定义的solution原样保留在行数据中最终随 batch 一起透传给奖励函数——这正是数据集自定义字段可直接作为 ORM 入参的底层依据。改写后的数据集样本形如{ images: [image_path1, image_path2], messages: [ { role: user, content: How many items are there in the image? Output the thinking process in think /think and\n final answer (number) in answer /answer tags. } ], solution: answer 3 /answer }两个要点原文档明确提示样本中若带有{role: assistant, content: answer 3 /answer}这样的 assistant 消息GRPOTrainer 会将其移除可以忽略solution字段会直接进入 ORM自定义数据集时images字段需组织为路径列表[image_path1, image_path2]。奖励函数内置 format 自定义准确性奖励ClevrCount 使用两个奖励函数formatDeepSeek-R1 风格的结构化格式奖励已内置于 SWIFT用--reward_funcs format直接启用。对应实现见 rewards/orm.py 中的Format类其校验正则要求输出严格为^think.*?/think\s*answer.*?/answer(?![\s\S])即整个 completion 必须从think块开始、以answer块结束external_r1v_acc自定义准确性奖励通过external_plugin机制注入。代码放在 plugin/plugin.py 中。MultiModalAccuracyORM的实现如下采用先符号校验、失败再回退字符串匹配的双层验证策略class MultiModalAccuracyORM(ORM): def __call__(self, completions, solution, **kwargs) - List[float]: Reward function that checks if the completion is correct. Args: completions (list[str]): Generated outputs solution (list[str]): Ground Truths. Returns: list[float]: Reward scores rewards [] from math_verify import parse, verify for content, sol in zip(completions, solution): reward 0.0 # Try symbolic verification first try: answer parse(content) if float(verify(answer, parse(sol))) 0: reward 1.0 except Exception: pass # Continue to next verification method if this fails # If symbolic verification failed, try string matching if reward 0.0: try: # Extract answer from solution if it has think/answer tags sol_match re.search(ranswer(.*?)/answer, sol) ground_truth sol_match.group(1).strip() if sol_match else sol.strip() # Extract answer from content if it has think/answer tags content_match re.search(ranswer(.*?)/answer, content) student_answer content_match.group(1).strip() if content_match else content.strip() # Compare the extracted answers if student_answer ground_truth: reward 1.0 except Exception: pass # Keep reward as 0.0 if both methods fail rewards.append(reward) return rewards orms[external_r1v_acc] MultiModalAccuracyORM从源码结构看奖励插件机制的工作方式是ORM 基类定义在 rewards/orm.py__call__接收completions模型生成文本列表加上数据集透传字段此处为solution子类只需实现打分逻辑再通过orms[name] Class注册到全局奖励字典训练侧用--external_plugins path --reward_funcs name即可生效。plugin/plugin.py 文件头部的注释也给出了这三步标准流程定义奖励类 → 注册到orms→ 通过参数引用。由于completions和solution都是列表一个 batch 的所有 completion 可一次算完任务变化时只需同步修改数据集字段与奖励函数。训练命令与实验记录实验在 8 卡上进行SWIFT GRPO 支持多 GPU 部署以加速 rollout此处用 2 张卡起 vLLM 数据并行服务6 张卡训练。若qwen2.5-vl在 vLLM 上遇到部署报错可查阅 vLLM 社区 issue 处理。先启动外部 vLLM rollout 服务CUDA_VISIBLE_DEVICES6,7 \ swift rollout \ --model Qwen/Qwen2.5-VL-3B-Instruct \ --vllm_data_parallel_size 2再启动训练。任务简单max_completion_length取 1024学习率与beta分别为1e-6与0.001batch_size与num_generations的配比逻辑可参考 GRPO 完整流程文档WANDB_API_KEYyour_wandb_api_key \ CUDA_VISIBLE_DEVICES0,1,2,3,4,5 \ NPROC_PER_NODE6 \ swift rlhf \ --rlhf_type grpo \ --model Qwen/Qwen2.5-VL-3B-Instruct \ --external_plugins examples/train/grpo/plugin/plugin.py \ --reward_funcs external_r1v_acc format \ --use_vllm true \ --vllm_mode server \ --vllm_server_host 127.0.0.1 \ --vllm_server_port 8000 \ --tuner_type full \ --torch_dtype bfloat16 \ --dataset AI-ModelScope/clevr_cogen_a_train \ --load_from_cache_file true \ --max_completion_length 1024 \ --num_train_epochs 1 \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --learning_rate 1e-6 \ --gradient_accumulation_steps 2 \ --save_strategy steps \ --eval_strategy steps \ --eval_steps 1000 \ --save_steps 1000 \ --save_total_limit 10 \ --logging_steps 1 \ --output_dir output/GRPO_CLEVR_COUNTDOWN \ --warmup_ratio 0.01 \ --dataloader_num_workers 4 \ --num_generations 24 \ --temperature 1.0 \ --system examples/train/grpo/prompt.txt \ --deepspeed zero3 \ --log_completions true \ --report_to wandb \ --num_iterations 1 \ --async_generate false \ --beta 0.001 \几个值得注意的参数--vllm_mode server--vllm_server_host/--vllm_server_port训练进程不本地拉起 vLLM而是连接上面swift rollout启动的独立服务实现采样与训练的卡数解耦--num_generations 24GRPO 组内优势估计的采样条数本任务简单用较多采样换取更稳定的组内统计--system examples/train/grpo/prompt.txt从文件读取 system prompt。prompt.txt 的内容正是要求 Assistant 先推理再作答、并把过程与答案分别包在think与answer标签中与format奖励的正则严格对应--beta 0.001KL 正则项权重配合1e-6的小学习率保持训练平稳--deepspeed zero33B 全参多模态训练下用 ZeRO-3 分摊显存。实验观察由于数据集与任务都比较简单模型在约 500 步后收敛文档给出的关键观察自定义准确性奖励external_r1v_acc持续上升任务成功率从初始 0.4 提升至接近 1证明模型确实学会了计数任务format奖励全程稳定在 1——因为所有样本的 query 格式一致模型早期就掌握了输出结构reward_std稳定在 0.1 以下说明组内采样趋于同质都答对优势信号自然衰减completion 长度最终稳定在 60–80 token 区间模型收敛出逐个物体计数的固定输出模式。任务二GEOQA 几何问答数据集与奖励函数几何问答任务要求给定一张几何图形回答与之相关的数学问题。数据源自相关论文并经 R1-V 项目预处理为problem-solution格式图片保留在image字段。因此无需自定义预处理器直接--dataset AI-ModelScope/GEOQA_R1V_Train_8K即可奖励函数也直接复用上一节的MultiModalAccuracyORM无需改动。训练参数两个关键差异基座模型与大部分超参与 ClevrCount 相同主要差异有两处--num_iterations 2允许一次 rollout 的采样结果被多次复用做参数更新等价于用更少的采样换取更多更新步提升训练吞吐--max_grad_norm 0.5实验中发现数学类任务训练可能不稳定、甚至崩溃——表现为所有奖励骤降、loss、grad_norm与 KL 散度快速攀升且无法恢复。截断梯度范数到 0.5 是文档给出的防崩溃手段原文同时提醒该不稳定性有一定随机性并非必然出现。此外通过环境变量MAX_PIXELS401408控制单图最大像素从而限制视觉 token 数、控制显存从源码结构看MAX_PIXELS等环境变量在多模态模型的 processor 配置中被映射为图像 token 上限见 model/models/qwen.py。WANDB_API_KEYyour_wandb_api_key \ CUDA_VISIBLE_DEVICES0,1,2,3,4,5 \ MAX_PIXELS401408 \ NPROC_PER_NODE6 \ swift rlhf \ --rlhf_type grpo \ --model Qwen/Qwen2.5-VL-3B-Instruct \ --external_plugins examples/train/grpo/plugin/plugin.py \ --reward_funcs external_r1v_acc format \ --use_vllm true \ --vllm_mode server \ --vllm_server_host 127.0.0.1 \ --vllm_server_port 8000 \ --tuner_type full \ --torch_dtype bfloat16 \ --dataset AI-ModelScope/GEOQA_R1V_Train_8K \ --load_from_cache_file true \ --max_completion_length 1024 \ --num_train_epochs 1 \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --learning_rate 1e-6 \ --gradient_accumulation_steps 2 \ --save_strategy steps \ --eval_strategy steps \ --eval_steps 400 \ --save_steps 400 \ --save_total_limit 10 \ --logging_steps 1 \ --output_dir output/GRPO_GEOQA \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --num_generations 8 \ --temperature 1.0 \ --repetition_penalty 1.1 \ --system examples/train/grpo/prompt.txt \ --deepspeed zero3 \ --log_completions true \ --report_to wandb \ --num_iterations 2 \ --async_generate false \ --beta 0.001 \ --max_grad_norm 0.5 \相比 ClevrCount还新增了--repetition_penalty 1.1抑制几何推理中常见的循环复述num_generations从 24 降到 8任务更难、rollout 更贵。实验观察相比 ClevrCount几何问答难度大得多Accuracy Reward 收敛明显更慢、震荡更大completion_length呈现清晰的增长趋势最终稳定在 250 token 左右——模型学会了写更长的推理链奖励最终收敛reward_std在总共约 1300 步后稳定在 0.2 附近。训练后的 completion 示例几何切线题答案 29°User: If AB is the tangent to circle O with the tangent point A, and BO intersects circle O at point C, and point D is located on circle O, what is the measure of angle ADC if the degree of angle ABO is 32.0? Assistant: To find the measure of angle ADC, we first need to understand the properties of the tangent and the circle. Since AB is a tangent to the circle at point A, the line segment OA (which is perpendicular to the tangent AB) is perpendicular to AB. This means that angle OAB is 90 degrees. Given that angle ABO is 32 degrees, we can find angle BAO by subtracting angle ABO from 90 degrees: ∠BAO 90° - ∠ABO 90° - 32° 58° Since angle BAO is an angle in the minor arc AD, the angle ADC, which is an angle in the minor arc AD, is half the angle BAO. Therefore, angle ADC is: ∠ADC 1/2 × ∠BAO 1/2 × 58° 29° So, the measure of angle ADC is 29 degrees. answer The measure of angle ADC is 29 degrees. /answer任务三Open-R1 多模态推理数据集数据集说明该实验对应 open-r1-multimodal 项目的设置使用数据集lmms-lab/multimodal-open-r1-8k-verified聚焦多模态数学推理由 GPT4o 基于 Math360K 与 Geo170K 数据集生成包含推理路径与可验证答案。数据集自带image、problem、solution字段无需改写 prompt也无需重新定义数据集奖励函数继续复用MultiModalAccuracyORM。训练参数应对 OOM与上一组实验几乎一致唯一调整是针对OOM显存溢出问题将MAX_PIXELS降到262144以减少视觉 token 带来的显存占用WANDB_API_KEYyour_wandb_api_key \ CUDA_VISIBLE_DEVICES0,1,2,3,4,5 \ MAX_PIXELS262144 \ MASTER_PORT29600 \ NPROC_PER_NODE6 \ swift rlhf \ --rlhf_type grpo \ --model Qwen/Qwen2.5-VL-3B-Instruct \ --external_plugins examples/train/grpo/plugin/plugin.py \ --reward_funcs external_r1v_acc format \ --use_vllm true \ --vllm_mode server \ --vllm_server_host 127.0.0.1 \ --vllm_server_port 8000 \ --tuner_type full \ --torch_dtype bfloat16 \ --dataset lmms-lab/multimodal-open-r1-8k-verified \ --load_from_cache_file true \ --max_completion_length 1024 \ --num_train_epochs 1 \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --learning_rate 1e-6 \ --gradient_accumulation_steps 2 \ --save_strategy steps \ --eval_strategy steps \ --eval_steps 400 \ --save_steps 400 \ --save_total_limit 10 \ --logging_steps 1 \ --output_dir output/GRPO_GEOQA \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --num_generations 8 \ --temperature 1.0 \ --repetition_penalty 1.1 \ --system examples/train/grpo/prompt.txt \ --deepspeed zero3 \ --log_completions true \ --report_to wandb \ --num_iterations 2 \ --async_generate false \ --beta 0.001 \ --max_grad_norm 0.5 \实验观察8k 样本规模下共训练1268 步Accuracy Reward 收敛到0.5 左右——开放推理任务的答案空间更大准确率天花板低于闭式问答completion_length稳定在 200 token 附近reward_std最终在 0.2 上下震荡。训练后 completion 示例地图分区统计题User: Based on the map, which state falls into the lowest percentage range of lots, and what is that percentage range? Assistant: The image provided shows a color-coded map of the USA indicating different ranges of lot percentages across various states. According to the legend at the bottom, the lowest percentage range (6.0% - 6.5%) is represented by white. In the image, Alabama (AL) is shaded in white, which corresponds to the 6.0% - 6.5% category. Therefore, based on the map, the state that falls into the lowest percentage range of lots is Alabama, with the percentage range of 6.0% - 6.5%. Therefore, the answer is 6.0% - 6.5%. answerAlabama/answer关键要点总结数据集字段契约messagesimages是模型输入solution以及任意自定义列由ResponsePreprocessor透传给 ORM自定义多模态数据集时images必须是路径列表。相关实现在 dataset/preprocessor/core.py 与 dataset/register.py。奖励插件三步法继承 ORM 基类 实现__call__(completions, **dataset_fields)→orms[name] MyORM注册 →--external_plugins--reward_funcs引用。format、math_accuracy等常用奖励均已内置可直接组合使用。rollout/训练解耦swift rollout独立起 vLLM 服务支持--vllm_data_parallel_size多卡并行训练端通过--vllm_mode server连接卡数、显存、采样吞吐均可独立调优。稳定性三板斧来自两组数学类实验的教训小学习率1e-6 小beta0.001打底数学任务加--max_grad_norm 0.5防崩溃图像任务用MAX_PIXELS环境变量压视觉 token 防 OOM。超参随任务难度调整简单任务用大num_generations24获得更稳的组内优势难任务降采样8并用--num_iterations 2复用 rollout、加repetition_penalty抑制重复推理。收敛判据准确性奖励曲线单调上行且reward_std走低ClevrCount 中低于 0.1说明任务被吃掉若reward_std长期在 0.2 附近且 accuracy 停在 0.5通常是任务本身答案开放度或数据规模决定的上限而非训练配置问题。更多 GRPO 配置细节batch 与num_generations配比、总步数推算、KL 项讨论见 docs/source_en/BestPractices/GRPO.md更多奖励函数与外部奖励模型插件示例见 examples/train/grpo/plugin/plugin.py。【免费下载链接】swiftUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600 LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300 MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).项目地址: https://gitcode.com/GitHub_Trending/swift1/swift创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考