Bicleaner AI (bicleaner-ai-classify) is a tool in Python that aims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0).
Sentence pairs considered very noisy are scored with 0.
Although a training tool (bicleaner-ai-train) is provided, you may want to use the available ready-to-use language packages.
Please, use bicleaner-ai-download to download the latest language packages or visit the Github releases for lite models and Hugging Face Hub for full models since v2.0.
Visit our docs for a detailed example on Bicleaner training.
If you find Bicleaner AI useful, please consider citing us.
New improved multilingual models for zero-shot classification.
Previous news
Model accuracy improvements and HF integration! See CHANGELOG.
Bicleaner AI is a Bicleaner fork that uses neural networks. It comes with two types of models, lite models for fast scoring and full models for high performance. Lite models use A Decomposable Attention Model for Natural Language Inference (Parikh et al.). Full models use fine-tuned XLMRoberta (Unsupervised Cross-lingual Representation Learning at Scale).
The use of XLMRoberta and 1:10 positive to negative ratio were inspired in the winner of WMT20 Parallel Corpus Filtering Task paper (Filtering noisy parallel corpus using transformers with proxy task learning).
- Python >= 3.8
- PIP >= 23.0
- CUDA >=11.2 (for training and inference with full models)
Bicleaner AI is written in Python and can be installed using pip.
It also requires the KenLM Python bindings with support for 7-gram language models.
Hardrules uses FastSpell that requires cyhunspell to be installed manually.
You can easily install all the requirements by running the following commands:
pip install bicleaner-ai git+https://github.com/MSeal/cython_hunspell@2.0.3
pip install --config-settings="--build-option=--max_order=7" https://github.com/kpu/kenlm/archive/master.zipAfter installation, three binary files (bicleaner-ai-train, bicleaner-ai-classify, bicleaner-ai-download) will be located in your python/installation/prefix/bin directory. This is usually $HOME/.local/bin or /usr/local/bin/.
TensorFlow 2 will be installed as a dependency and GPU support is required for training.
pip will install latest TensorFlow supported version, but older versions >=2.6.5 are supported and can be installed if your machine does not meet TensorFlow CUDA requirements.
See this table for the CUDA and TensorFlow versions compatibility.
In case you want a different TensorFlow version, you can downgrade using:
pip install tensorflow==2.6.5TensorFlow logging messages are suppressed by default, in case you want to see them you have to explicitly set TF_CPP_MIN_LOG_LEVEL environment variable.
For example:
TF_CPP_MIN_LOG_LEVEL=0 bicleaner-ai-classifyWARNING: If you are experiencing slow downs because Bicleaner AI is not running in the GPU, you should check those logs to see if TensorFlow is loading all the libraries correctly.
For Serbo-Croatian languages, models work better with transliteration. To be able score transliterated text, install optional dependency:
pip install bicleaner-ai[transliterate]
Note that this won't transliterate the output text, it will be used only for scoring.
bicleaner-ai-classify aims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.
By default, the input file (the parallel corpus to be classified) expects at least four columns, being:
- col1: URL 1
- col2: URL 2
- col3: Source sentence
- col4: Target sentence
but the source and target sentences column index can be customized by using the --scol and --tcol flags. Urls are not mandatory.
The generated output file will contain the same lines and columns that the original input file had, adding an extra column containing the Bicleaner AI classifier score.
Bicleaner AI has two types of models, full and lite models. Full models are recommended, as they provide much higher quality. If speed is a hard constraint to you, lite models could be an option (take a look at the speed comparison).
See available full models here and available lite models here.
You can download the model with:
bicleaner-ai-download en fr full
This will download bitextor/bicleaner-ai-full-en-fr model from HuggingFace and store it at the cache directory.
Or you can download a lite model with:
bicleaner-ai-download en fr lite ./bicleaner-models
This will download and store the en-fr lite model at ./bicleaner-models/en-fr.
Since 2.3.0 version, full models also accept a local path to download, instead of the HF cache directory. In that case, to use the model, provide the local path instead of the HF identifier.
To read more information about how HF cache works, please read the official documentation.
To classify a tab separated file containing English sentences in the first column and French sentences in the second column, use
bicleaner-ai-classify \
--scol 1 --tcol 2
corpus.en-fr.tsv \
corpus.en-fr.classifed.tsv \
bitextor/bicleaner-ai-full-en-frwhere --scol and --tcol indicate the location of source and target sentence,
corpus.en-fr.tsv the input file,
corpus.en-fr.classified.tsv output file and bitextor/bicleaner-ai-en-fr is the HuggingFace model name.
Each line of the new file will contain the same content as the input file, adding a column with the score given by the Bicleaner AI classifier.
Note that, to use a lite model, you need to provide model path in your local file system, instead of HuggingFace model name.
There are multilingual full models available. They can work with, potentially, any language (currently only paired with English) that XLMR supports. To see a further explaination on how to train a multilingual model or how our models perform, take a look here and here.
WARNING: multilingual models will disable hardrules that expect language parameter.
You can, however, overwrite the language code in the model configuration with -s/--source_lang or -t/--target_lang options during classify. For example when scoring English-Icelandic data, use:
bicleaner-ai-classify \
--scol 1 --tcol 2 \
-t is \
corpus.en-is.tsv \
corpus.en-is.classified.tsv \
bitextor/bicleaner-ai-full-en-xx
Full description of the command-line parameters:
usage: bicleaner-ai-classify [-h] [-s SOURCE_LANG] [-t TARGET_LANG] [-S SOURCE_TOKENIZER_COMMAND] [-T TARGET_TOKENIZER_COMMAND] [--header] [--scol SCOL] [--tcol TCOL] [-b BLOCK_SIZE] [-p PROCESSES] [--batch_size BATCH_SIZE]
[--tmp_dir TMP_DIR] [--score_only] [--calibrated] [--raw_output] [--lm_threshold LM_THRESHOLD] [--disable_hardrules] [--disable_lm_filter] [--disable_porn_removal] [--disable_minimal_length]
[--run_all_rules] [--rules_config RULES_CONFIG] [--offline] [--auth_token AUTH_TOKEN] [-q] [--debug] [--logfile LOGFILE] [-v]
input [output] model
positional arguments:
input Tab-separated files to be classified
output Output of the classification (default: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
model Path to model directory or HuggingFace Hub model identifier (such as 'bitextor/bicleaner-ai-full-en-fr')
options:
-h, --help show this help message and exit
Optional:
-s SOURCE_LANG, --source_lang SOURCE_LANG
Overwrite model config source language (default: None)
-t TARGET_LANG, --target_lang TARGET_LANG
Overwrite model config target language (default: None)
-S SOURCE_TOKENIZER_COMMAND, --source_tokenizer_command SOURCE_TOKENIZER_COMMAND
Source language (SL) tokenizer full command (default: None)
-T TARGET_TOKENIZER_COMMAND, --target_tokenizer_command TARGET_TOKENIZER_COMMAND
Target language (TL) tokenizer full command (default: None)
--header Input file will be expected to have a header, and the output will have a header as well (default: False)
--scol SCOL Source sentence column (starting in 1). The name of the field is expected instead of the position if --header is set (default: 3)
--tcol TCOL Target sentence column (starting in 1). The name of the field is expected instead of the position if --header is set (default: 4)
-b BLOCK_SIZE, --block_size BLOCK_SIZE
Sentence pairs per block (default: 10000)
-p PROCESSES, --processes PROCESSES
Option no longer available, please set BICLEANER_AI_THREADS environment variable (default: None)
--batch_size BATCH_SIZE
Sentence pairs per block (default: 32)
--tmp_dir TMP_DIR Temporary directory where creating the temporary files of this program (default: /tmp)
--score_only Only output one column which is the bicleaner score (default: False)
--calibrated Output calibrated scores (default: False)
--raw_output Return raw output without computing positive class probability. (default: False)
--lm_threshold LM_THRESHOLD
Threshold for language model fluency scoring. All TUs whose LM fluency score falls below the threshold will are removed (classifier score set to 0), unless the option --keep_lm_result set. (default: 0.5)
--disable_hardrules Disables the bicleaner_hardrules filtering (only bicleaner_classify is applied) (default: False)
--disable_lm_filter Disables LM filtering (default: False)
--disable_porn_removal
Don't apply porn removal (default: False)
--disable_minimal_length
Don't apply minimal length rule (default: False)
--run_all_rules Run all rules of Hardrules instead of stopping at first discard (default: False)
--rules_config RULES_CONFIG
Hardrules configuration file (default: None)
--offline Don't try to download the model, instead try directly to load from local storage (default: False)
--auth_token AUTH_TOKEN
Auth token for the Hugging Face Hub (default: None)
Logging:
-q, --quiet Silent logging mode (default: False)
--debug Debug logging mode (default: False)
--logfile LOGFILE Store log to a file (default: <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)
-v, --version Show version of the package and exit
Bicleaner AI provides a command-line tool to train your own model, in case available models do not fit your needs. Please go to our training documentation for a quick start and further details.
To set the maximum number of threads/processes to be used during training or classifying, --processes option is no longer available.
You will need to set BICLEANER_AI_THREADS environment variable to the desired value.
For example:
BICLEANER_AI_THREADS=12 bicleaner-ai-classify ...
If the variable is not set, the program will use all the available CPU cores.
A comparison of the speed in number of sentences per second between different types of models and hardware:
| model | speed CPUx1 | speed GPUx1 |
|---|---|---|
| full | 1.78 rows/sec | 200 rows/sec |
| lite | 600 rows/sec | 10,000 rows/sec |
- CPU: Intel Core i9-9960X single core (lite model batch 16, full model batch 1)
- GPU: Nvidia V100 (lite model batch 2048, full model batch 16)
J. Zaragoza-Bernabeu, M. Bañón, G. Ramírez-Sánchez, S. Ortiz-Rojas,
"Bicleaner AI: Bicleaner Goes Neural",
in Proceedings of the 13th Language Resources and Evaluation Conference.
Marseille, France: Language Resources and Evaluation Conference, June 2022
@inproceedings{zaragoza-bernabeu-etal-2022-bicleaner,
title = {"Bicleaner {AI}: Bicleaner Goes Neural"},
author = {"Zaragoza-Bernabeu, Jaume and
Ram{\'\i}rez-S{\'a}nchez, Gema and
Ba{\~n}{\'o}n, Marta and
Ortiz Rojas, Sergio"},
booktitle = {"Proceedings of the Thirteenth Language Resources and Evaluation Conference"},
month = jun,
year = {"2022"},
address = {"Marseille, France"},
publisher = {"European Language Resources Association"},
url = {"https://aclanthology.org/2022.lrec-1.87"},
pages = {"824--831"},
abstract = {"This paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The tool, which now implements a new neural classifier, uses state-of-the-art techniques based on pre-trained transformer-based language models fine-tuned on a binary classification task. After that, parallel corpus filtering is performed, discarding the sentences that have lower probability of being mutual translations. Our experiments, based on the training of neural machine translation (NMT) with corpora filtered using Bicleaner AI for two different scenarios, show significant improvements in translation quality compared to the previous version of the tool which implemented a classifier based on Extremely Randomized Trees."},
}
All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.