Training a Text-Editing Generator with RL

Fine-tuning a text-editing diffusion model with reinforcement learning against a checkable reward, using document forgery as the testbed. In progress at DFKI.

Status: in progress, with Prof. Andreas Dengel and Dr. Sheraz Ahmed (DFKI)

The question

Can reinforcement learning train a text-editing diffusion model against a reward we can actually check? Document forgery is the testbed: an edit is good if it is realistic, correctly localized, says what it should (checked by OCR), and is hard for detectors to catch.

Evidence so far

  • Detectors have a shortcut to target. In BMVC, grid-aligned copy-move raises DTD’s detection-failure rate from 0.125 to 0.512 on DocTamper.
  • The generator. A DiffUTE reproduction (in progress), a diffusion model for document text editing.
  • The reward has to be learned. Heuristic rewards gave no usable training signal, so a learned critic (a reward model) scores each generated edit.
Copy-move, splicing and generative forgeries from our BMVC 2026 paper. This project learns to generate the generative case.

Next step

Fine-tune the generator with RL against the critic, and pair every generated forgery with grid-shifted variants of itself, so detectors trained on the output must give the same verdict across the pair. The baseline to beat is random-shift augmentation: a detector trained with random JPEG-grid shifts, compared on failure rate under the BMVC interventions.

  • Black et al., Training diffusion models with reinforcement learning (DDPO), ICLR 2024.
  • Wallace et al., Diffusion model alignment using direct preference optimization, CVPR 2024.
  • Chen et al., DiffUTE: universal text editing diffusion model, NeurIPS 2023.
  • Qu et al., Towards robust tampered text detection in document image: new dataset and new solution (DocTamper), CVPR 2023.

← Directions I’m exploring · Homepage