[Image Generation & Edit] Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
[Video Generation] FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

Why This Project

• Critics are the bottleneck. The FIRM series is built around a simple belief: RL for visual generation only works when the reward model is faithful, stable, and hard to hack.
• Three task-specific data pipelines. FIRM-Edit uses a difference-first scoring pipeline, FIRM-Gen uses a plan-then-score pipeline to reduce MLLM hallucinations, and FIRM-Video uses a check-before-score pipeline to ground video scores in verified evidence.
• Benchmarks for critic reliability. FIRM-Bench provides human-annotated evaluation for editing and generation critics, while FIRM-Video-Bench extends this evaluation to video reward models.
• Reward shaping that actually holds up. CME and QMA are designed to prevent the shortcut behavior that appears when rewards are naively combined.

FIRM At A Glance

Track	Component
Image Editing	FIRM-Edit-370K, FIRM-Edit-8B, FIRM-Qwen-Edit
Image Generation	FIRM-Gen-293K, FIRM-Gen-8B, FIRM-SD-3.5
Video Generation	FIRM-Video-90K, FIRM-Video-8B, FIRM-Video-Bench
Benchmarking	FIRM-Bench, FIRM-Video-Bench

Repository Layout

TrustYourCritic/
├── generation/   # Image Generation RL training and reward serving
└── editing/      # Image Edit RL training, reward serving, reproduction scripts
└── video_generarion/   # Reserved for our planned Video Generation RL code release — stay tuned

Important Notes

• To avoid Python package conflicts, install and run GenRL/EditRL in separate environments.

Quick Start

1) Image Generation RL

cd generation
conda create -n FIRM-Gen python=3.10 -y
conda activate FIRM-Gen
pip install -e .

i ) Launch Reward Server First

python generation/flow_grpo/reward_model_server.py

ii ) Change the Training Configuration

• generation/config/nft_flux2_klein.py
• generation/config/nft_qwen_image.py
• generation/config/nft_zimage_turbo.py
• generation/config/nft.py

iii ) Start Training

bash generation/scripts/train_sd35_sharegpt_qwenvl.sh

2) Image Edit RL

cd editing
conda create -n FIRM-Edit python=3.10 -y
conda activate FIRM-Edit
pip install -e .

i ) Launch Reward Server First

## Change the default ip and port to your perference
python editing/reward_server/reward_server_qwen3_vl_8b_sft.py

ii ) Change the Training Configuration

• editing/config/kontext_nft_qwen3vl_8b_sft.py
• editing/config/kontext_nft_qwen3vl_8b.py
• editing/config/kontext_nft_qwen25vl_32b_non_logits.py

iii ) Start Training

bash editing/examples/train_qwen_image_edit.sh

3) Video Generation RL

Coming soon.

Data Perparation

Image Generation RL

Expected JSON file like:

[
  {"input_prompt": "A cinematic portrait of a fox in snow."}
]

Image Edit RL

Expected dataset layout:

dataset-root/
├── images/
├── train_metadata.jsonl
└── test_metadata.jsonl

Each JSONL line:

{"prompt": "make the sky sunset orange", "image": "images/example.jpg", "requirement": "preserve identity"}

Video Generation RL

Coming soon.

Evaluation

The code and data for FIRM-Bench&FIRM-Bench-Video are hosted on Hugging Face.

We provide inference and evaluation scripts for FIRM-Bench&FIRM-Bench-Video. We recommend deploying the model with vLLM for inference.

FIRM-Bench-Gen

Inference

python FIRM-Bench-Gen/vllm_infer.py \
  --input FIRM-Bench-Gen/bench_v1.jsonl \
  --output FIRM-Bench-Gen/result/xxx.jsonl \
  --image-root FIRM-Bench-Gen/ \
  --api-url xxxxx

MAE Calculation

python FIRM-Bench-Gen/gen_mae.py \
  --gt FIRM-Bench-Gen/result/human_bench_v1.jsonl \
  --pred FIRM-Bench-Gen/result/xxx.jsonl

FIRM-Bench-Edit

Inference

python FIRM-Bench-Edit/vllm_infer.py \
  --input FIRM-Bench-Edit/bench_v1.jsonl \
  --output FIRM-Bench-Edit/result/xxx.jsonl \
  --image-root FIRM-Bench-Edit/ \
  --api-url xxxxx

MAE Calculation

python FIRM-Bench-Edit/edit_mae.py \
  --gt FIRM-Bench-Edit/result/human_bench_v1.jsonl \
  --pred FIRM-Bench-Edit/result/xxx.jsonl

FIRM-Bench-Video

Inference

bash scripts/infer.sh all 32 qwen3vl8b \
  data/firm-video-bench.json \
  http://127.0.0.1:8000/v1 \
  Qwen3-VL-8B-Instruct

Metric Calculation

python scripts/metric.py \
  --gt_file data/firm-video-bench.json \
  --results_dir results

Acknowledgements

This repository was shaped by several open-source projects that pushed RL for image generation and image editing forward:

• flow-grpo
• DiffusionNFT
• Edit-R1

Citation

If you find this project, models, datasets, or benchmarks useful in your research, please cite the corresponding papers:

@misc{zhao2026trustcriticrobustreward,
  title         = {Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation},
  author        = {Xiangyu Zhao and Peiyuan Zhang and Junming Lin and Tianhao Liang and Yuchen Duan and Shengyuan Ding and Changyao Tian and Yuhang Zang and Junchi Yan and Xue Yang},
  year          = {2026},
  eprint        = {2603.12247},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2603.12247}
}

@misc{zhang2026firmvideocheckscorereliable,
  title         = {FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling},
  author        = {Peiyuan Zhang and Xiangyu Zhao and Hongbo Liu and Xiaoxing Hu and Mingxin Liu and Shuran Ma and Yunhang Shen and Jian Hu and Haihan Gao and Haoyu Cao and Xue Yang},
  year          = {2026},
  eprint        = {2608.21839},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.21839}
}