Skip to content
/ oft Public

Diffusers Implementation of Controlling Text-to-Image Diffusion by Orthogonal Finetuning

License

Notifications You must be signed in to change notification settings

tripplyons/oft

Repository files navigation

OFT (Orthogonal Fine-Tuning)

Diffusers Implementation of Controlling Text-to-Image Diffusion by Orthogonal Finetuning (https://arxiv.org/pdf/2306.07280.pdf)

Setup (tested on Linux with an Nvidia GPU)

  1. Install PyTorch
  2. pip install -r requirements.txt

Usage

Training

  1. Create folders config and output if they don't already exist.
  2. Create a config file at config/config.json. See example.config.json for an example. Right now it only supports fine-tuning Diffusers models.
  3. Run python train.py

Merging

To merge the adapter weight with a base model, run python merge_to_original.py --processor_path /path/to/attn_processors.pt --output_path /path/to/merged_model.ckpt --model_path /path/to/base_model.ckpt.

Right now this file only supports original Stable Diffusion (non-Diffusers) model as base models.

Implementation Details

  • Parameterized each skew-symmetric matrix as a weight matrix minus its transpose.
  • For constrained orthogonal fine-tuning (COFT), the norm of the weights of each skew-symmetric matrix is given a maximum value.
  • All matrices are kept in the block-diagonal form for as many operations as possible for efficiency.

About

Diffusers Implementation of Controlling Text-to-Image Diffusion by Orthogonal Finetuning

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages