DiffUTE Reproduction

A paper-faithful reproduction of DiffUTE (NeurIPS 2023), a diffusion model for universal scene and document text editing.

Code on GitHub

A from-scratch reproduction of DiffUTE: Universal Text Editing Diffusion Model (Chen et al., NeurIPS 2023). It is groundwork for my research on generative document forgery.

The target is the paper, not the official repository. The official code retrains on a different dataset (AnyText-3M) and skips parts of the published recipe. This reproduction follows the paper instead:

  • Stage 1: progressive-training (PTT) fine-tuning of the VAE so it can reconstruct text, using the paper’s hyper-parameters (lr 1e-5, effective batch size 256).
  • Stage 2: UNet training with a frozen TrOCR glyph encoder, with the input layout kept compatible with SD-2 inpainting weights.
  • Data: the paper’s mix (CLDA, XFUND, PubLayNet, ICDAR 2013/2015/2017-MLT/2019-MLT), plus a documented public substitute for the paper’s unreleased 5M web-crawled images.
  • Evaluation: OCR-accuracy tooling, PaddleOCR annotation, and multi-GPU launch scripts.

Every departure from the paper, and every dataset source and license, is documented in the repository.