Skip to content

Latest commit

 

History

History

benchmark

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
opennmt_ende_wmt14
opennmt_ende_wmt14
 
 
opus_mt_ende
opus_mt_ende
 
 
README.md
README.md
 
 
benchmark.py
benchmark.py
 
 
benchmark_all.py
benchmark_all.py
 
 
requirements.txt
requirements.txt
 
 

README.md

Benchmark tools

This directory contains some scripts to benchmark translation systems.

Requirements

  • Python 3
  • Docker
python3 -m pip install -r requirements.txt

Usage

python3 benchmark.py <IMAGE> <SOURCE> <REFERENCE>

The Docker image must contain 3 executable files at its root:

  • /tokenize $input $output
  • /detokenize $input $output
  • /translate $device $input $output, where:
    • $device is "CPU" or "GPU"
    • $input is the path to the tokenized input file
    • $output is the path where the tokenized output should be written

The benchmark script will report multiple metrics. The results can be aggregated over multiple runs using the option --num_samples N. See python3 benchmark.py -h for additional options.

Note: the script focuses on raw decoding performance so the following steps are not included in the translation time:

  • tokenization
  • detokenization
  • model initialization (obtained by translating an empty file)

Reproducing the benchmark numbers from the README

We use the script benchmark_all.py to produce the benchmark numbers in the main README. The script builds all Docker images defined in subdirectories and reports the results as a Markdown table. The execution can take up to 3 hours.

# Run CPU benchmark:
python3 benchmark_all.py cpu

# Run GPU benchmark:
python3 benchmark_all.py gpu