Inter-Architecture Knowledge Transfer

iatransfer is a PyTorch package for transferring pretrained weights between models of different architectures instantaneously.

Drastically speed up your training process using two additional lines of code.

Installation

pip install iatransfer

Usage

• simple

import torch
from iatransfer.toolkit import IAT

transfer = IAT()

# run training on Model1()
model_from: nn.Module = Model1()

train(model_from)

# instantiate new model
model_to: nn.Module = Model2() 

# enjoy high-accuracy initialization
transfer(model_from, model_to)

• parametrization

from iatransfer.toolkit import IAT


iat = IAT(standardization='blocks', matching='dp', score='autoencoder', transfer='trace')

# ==== or

iat = IAT(matching=('dp', {'param': 'value'}))

# ==== or

from iatransfer.toolkit.matching.dp_matching import DPMatching

iat = IAT(matching=DPMatching())

• plugins

from iatransfer.toolkit.base_matching import Matching


class CustomMatching(Matching):

    def match(self, from_module, to_module, *args, **kwargs)
        # provide your implementation


# This will instantiate the above CustomMatching in IAT
iat = IAT(matching='custom') 

Documentation

Citation

When referring to or using iatransfer in a scientific publication, please consider including citation to the following thesis:

@manual{
 iat2021,
 title = {Inter-Architecture Knowledge Transfer},
 author = {Maciej A. Czyzewski and Daniel Nowak and Kamil Piechowiak},
 note = {Transfer learning between different architectures},
 organization = {Poznan University of Technology},
 type = {Bachelor’s Thesis},
 address = {Poznan, Poland},
 year = {2021}
}

Development

Init:

./dev/init.sh

Run tests:

nosetests tests

Install in edit mode:

pip install -e .

Research reproduction:

Copy the source code to the GCP cloudshell or install iatransfer_research package.

Run:

/bin/bash ./scripts/research/iatransfer_full_run.sh

or

iatransfer_full_run.sh

if iatransfer_research has been installed.