InstructPix2Pix

InstructPix2Pix, from Tim Brooks, Aleksander Holynski, and Alexei Efros at UC Berkeley (paper arXiv 2211.09800, November 2022; CVPR 2023), defined instruction-based image editing as a task. The trick was in the data: they combined GPT-3 and Stable Diffusion to synthesize a large dataset of (image, instruction, edited image) triplets, then trained a conditional diffusion model on top of Stable Diffusion v1.5. At inference it takes an input image plus a plain instruction like 'turn him into a cyborg' and produces the edit in one forward pass, no per-image fine-tuning or inversion. Two guidance scales let users trade instruction strength against fidelity to the original.

It has no published standard benchmark scores, the paper's case is qualitative results across diverse images and instructions, but its influence is hard to overstate: nearly every instruction-editing model and product since (including FLUX.1 Kontext) descends from this formulation. The official code is on GitHub under timothybrooks/instruct-pix2pix with checkpoints fine-tuned from SD v1.5, and it is wired into diffusers and most editing UIs. Limitations are the ones the field is still working through: edits can drift on identity and fine detail, and complex multi-object instructions sometimes partially apply.

No published standard benchmark scores; evaluated qualitatively in the paper across diverse images and instructions