diff --git a/examples/README.md b/examples/README.md index c1cddd2e473457..ac2cc048d13cec 100644 --- a/examples/README.md +++ b/examples/README.md @@ -17,9 +17,9 @@ limitations under the License. We host a wide range of example scripts for multiple learning frameworks. Simply choose your favorite: [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow), [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch) or [JAX/Flax](https://github.com/huggingface/transformers/tree/main/examples/flax). -We also have some [research projects](https://github.com/huggingface/transformers/tree/main/examples/research_projects), as well as some [legacy examples](https://github.com/huggingface/transformers/tree/main/examples/legacy). Note that unlike the main examples these are not actively maintained, and may require specific older versions of dependencies in order to run. +We also have some [research projects](https://github.com/huggingface/transformers/tree/main/examples/research_projects), as well as some [legacy examples](https://github.com/huggingface/transformers/tree/main/examples/legacy). Note that unlike the main examples these are not actively maintained, and may require specific older versions of dependencies in order to run. -While we strive to present as many use cases as possible, the example scripts are just that - examples. It is expected that they won't work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data, allowing you to tweak and edit them as required. +While we strive to present as many use cases as possible, the example scripts are just that - examples. It is expected that they won't work out-of-the-box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data, allowing you to tweak and edit them as required. Please discuss on the [forum](https://discuss.huggingface.co/) or in an [issue](https://github.com/huggingface/transformers/issues) a feature you would like to implement in an example before submitting a PR; we welcome bug fixes, but since we want to keep the examples as simple as possible it's unlikely that we will merge a pull request adding more functionality at the cost of readability. @@ -97,16 +97,16 @@ and run the example command as usual afterward. ## Running the Examples on Remote Hardware with Auto-Setup -[run_on_remote.py](./run_on_remote.py) is a script that launches any example on remote self-hosted hardware, -with automatic hardware and environment setup. It uses [Runhouse](https://github.com/run-house/runhouse) to launch -on self-hosted hardware (e.g. in your own cloud account or on-premise cluster) but there are other options -for running remotely as well. You can easily customize the example used, command line arguments, dependencies, +[run_on_remote.py](./run_on_remote.py) is a script that launches any example on remote self-hosted hardware, +with automatic hardware and environment setup. It uses [Runhouse](https://github.com/run-house/runhouse) to launch +on self-hosted hardware (e.g. in your own cloud account or on-premise cluster) but there are other options +for running remotely as well. You can easily customize the example used, command line arguments, dependencies, and type of compute hardware, and then run the script to automatically launch the example. -You can refer to -[hardware setup](https://runhouse-docs.readthedocs-hosted.com/en/main/rh_primitives/cluster.html#hardware-setup) +You can refer to +[hardware setup](https://runhouse-docs.readthedocs-hosted.com/en/latest/api/python/cluster.html#hardware-setup) for more information about hardware and dependency setup with Runhouse, or this -[Colab tutorial](https://colab.research.google.com/drive/1sh_aNQzJX5BKAdNeXthTNGxKz7sM9VPc) for a more in-depth +[Colab tutorial](https://colab.research.google.com/drive/1sh_aNQzJX5BKAdNeXthTNGxKz7sM9VPc) for a more in-depth walkthrough. You can run the script with the following commands: @@ -119,7 +119,7 @@ pip install runhouse python run_on_remote.py \ --example pytorch/text-generation/run_generation.py \ --model_type=gpt2 \ - --model_name_or_path=gpt2 \ + --model_name_or_path=openai-community/gpt2 \ --prompt "I am a language model and" # For byo (bring your own) cluster: @@ -131,4 +131,4 @@ python run_on_remote.py --instance --provider \ --example ``` -You can also adapt the script to your own needs. \ No newline at end of file +You can also adapt the script to your own needs. diff --git a/examples/diff-conversion/README.md b/examples/diff-conversion/README.md new file mode 100644 index 00000000000000..a575a83b015c63 --- /dev/null +++ b/examples/diff-conversion/README.md @@ -0,0 +1,20 @@ +# Using the `diff_converter` linter + +`pip install libcst` is a must! + +# `sh examples/diff-conversion/convert_examples.sh` to get the converted outputs + +The diff converter is a new `linter` specific to `transformers`. It allows us to unpack inheritance in python to convert a modular `diff` file like `diff_gemma.py` into a `single model single file`. + +Examples of possible usage are available in the `examples/diff-conversion`, or `diff_gemma` for a full model usage. + +`python utils/diff_model_converter.py --files_to_parse "/Users/arthurzucker/Work/transformers/examples/diff-conversion/diff_my_new_model2.py"` + +## How it works +We use the `libcst` parser to produce an AST representation of the `diff_xxx.py` file. For any imports that are made from `transformers.models.modeling_xxxx` we parse the source code of that module, and build a class dependency mapping, which allows us to unpack the difference dependencies. + +The code from the `diff` file and the class dependency mapping are "merged" to produce the single model single file. +We use ruff to automatically remove the potential duplicate imports. + +## Why we use libcst instead of the native AST? +AST is super powerful, but it does not keep the `docstring`, `comment` or code formatting. Thus we decided to go with `libcst` \ No newline at end of file diff --git a/examples/diff-conversion/convert_examples.sh b/examples/diff-conversion/convert_examples.sh new file mode 100644 index 00000000000000..1cfdc3e33cdf82 --- /dev/null +++ b/examples/diff-conversion/convert_examples.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +# Iterate over each file in the current directory +for file in examples/diff-conversion/diff_*; do + # Check if it's a regular file + if [ -f "$file" ]; then + # Call the Python script with the file name as an argument + python utils/diff_model_converter.py --files_to_parse "$file" + fi +done \ No newline at end of file diff --git a/examples/diff-conversion/diff_dummy.py b/examples/diff-conversion/diff_dummy.py new file mode 100644 index 00000000000000..c5fd57f9f66eb5 --- /dev/null +++ b/examples/diff-conversion/diff_dummy.py @@ -0,0 +1,44 @@ +from math import log +from typing import List, Optional, Tuple, Union + +import torch + +from transformers import Cache +from transformers.modeling_outputs import CausalLMOutputWithPast +from transformers.models.llama.modeling_llama import LlamaModel + + +def _pre_process_input(input_ids): + print(log(input_ids)) + return input_ids + + +# example where we need some deps and some functions +class DummyModel(LlamaModel): + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + use_cache: Optional[bool] = None, + output_attentions: Optional[bool] = None, + output_hidden_states: Optional[bool] = None, + return_dict: Optional[bool] = None, + cache_position: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + input_ids = _pre_process_input(input_ids) + + return super().forward( + None, + attention_mask, + position_ids, + past_key_values, + inputs_embeds, + use_cache, + output_attentions, + output_hidden_states, + return_dict, + cache_position, + ) diff --git a/examples/diff-conversion/diff_my_new_model.py b/examples/diff-conversion/diff_my_new_model.py new file mode 100644 index 00000000000000..dddcc1d61c11d6 --- /dev/null +++ b/examples/diff-conversion/diff_my_new_model.py @@ -0,0 +1,14 @@ +from transformers.models.llama.configuration_llama import LlamaConfig + + +# Example where we only want to only add a new config argument and new arg doc +# here there is no `ARG` so we are gonna take parent doc +class MyNewModelConfig(LlamaConfig): + r""" + mlp_bias (`bool`, *optional*, defaults to `False`) + """ + + def __init__(self, mlp_bias=True, new_param=0, **super_kwargs): + self.mlp_bias = mlp_bias + self.new_param = new_param + super().__init__(self, **super_kwargs) diff --git a/examples/diff-conversion/diff_my_new_model2.py b/examples/diff-conversion/diff_my_new_model2.py new file mode 100644 index 00000000000000..2e449e06b16225 --- /dev/null +++ b/examples/diff-conversion/diff_my_new_model2.py @@ -0,0 +1,31 @@ +from transformers.models.gemma.modeling_gemma import GemmaForSequenceClassification +from transformers.models.llama.configuration_llama import LlamaConfig + + +# Example where we only want to only modify the docstring +class MyNewModel2Config(LlamaConfig): + r""" + This is the configuration class to store the configuration of a [`GemmaModel`]. It is used to instantiate an Gemma + model according to the specified arguments, defining the model architecture. Instantiating a configuration with the + defaults will yield a similar configuration to that of the Gemma-7B. + e.g. [google/gemma-7b](https://huggingface.co/google/gemma-7b) + Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the + documentation from [`PretrainedConfig`] for more information. + Args: + vocab_size (`int`, *optional*, defaults to 256000): + Vocabulary size of the Gemma model. Defines the number of different tokens that can be represented by the + `inputs_ids` passed when calling [`GemmaModel`] + ```python + >>> from transformers import GemmaModel, GemmaConfig + >>> # Initializing a Gemma gemma-7b style configuration + >>> configuration = GemmaConfig() + >>> # Initializing a model from the gemma-7b style configuration + >>> model = GemmaModel(configuration) + >>> # Accessing the model configuration + >>> configuration = model.config + ```""" + + +# Example where alllllll the dependencies are fetched to just copy the entire class +class MyNewModel2ForSequenceClassification(GemmaForSequenceClassification): + pass diff --git a/examples/diff-conversion/diff_new_model.py b/examples/diff-conversion/diff_new_model.py new file mode 100644 index 00000000000000..1486d40c6cdbd5 --- /dev/null +++ b/examples/diff-conversion/diff_new_model.py @@ -0,0 +1,30 @@ +# Example where we only want to overwrite the defaults of an init + +from transformers.models.gemma.configuration_gemma import GemmaConfig + + +class NewModelConfig(GemmaConfig): + def __init__( + self, + vocab_size=256030, + hidden_size=64, + intermediate_size=90, + num_hidden_layers=28, + num_attention_heads=16, + num_key_value_heads=16, + head_dim=256, + hidden_act="gelu_pytorch_tanh", + hidden_activation=None, + max_position_embeddings=1500, + initializer_range=0.02, + rms_norm_eps=1e-6, + use_cache=True, + pad_token_id=0, + eos_token_id=1, + bos_token_id=2, + tie_word_embeddings=True, + rope_theta=10000.0, + attention_bias=False, + attention_dropout=0.0, + ): + super().__init__(self) diff --git a/examples/diff-conversion/diff_super.py b/examples/diff-conversion/diff_super.py new file mode 100644 index 00000000000000..160f067ee01b85 --- /dev/null +++ b/examples/diff-conversion/diff_super.py @@ -0,0 +1,38 @@ +from typing import List, Optional, Tuple, Union + +import torch + +from transformers import Cache +from transformers.modeling_outputs import CausalLMOutputWithPast +from transformers.models.llama.modeling_llama import LlamaModel + + +# example where we need some deps and some functions +class SuperModel(LlamaModel): + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + use_cache: Optional[bool] = None, + output_attentions: Optional[bool] = None, + output_hidden_states: Optional[bool] = None, + return_dict: Optional[bool] = None, + cache_position: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + out = super().forward( + input_ids, + attention_mask, + position_ids, + past_key_values, + inputs_embeds, + use_cache, + output_attentions, + output_hidden_states, + return_dict, + cache_position, + ) + out.logits *= 2**4 + return out diff --git a/examples/flax/_tests_requirements.txt b/examples/flax/_tests_requirements.txt index f1e0fb2d90712f..2e93a1f2c549ff 100644 --- a/examples/flax/_tests_requirements.txt +++ b/examples/flax/_tests_requirements.txt @@ -1,8 +1,10 @@ -datasets >= 1.1.3 -pytest +datasets >= 1.13.3 +pytest<8.0.1 conllu nltk rouge-score seqeval tensorboard -evaluate >= 0.2.0 \ No newline at end of file +evaluate >= 0.2.0 +torch +accelerate diff --git a/examples/flax/conftest.py b/examples/flax/conftest.py index 131c6af92c44cc..4cf2e46ef07393 100644 --- a/examples/flax/conftest.py +++ b/examples/flax/conftest.py @@ -21,7 +21,7 @@ # allow having multiple repository checkouts and not needing to remember to rerun -# 'pip install -e .[dev]' when switching between checkouts and running tests. +# `pip install -e '.[dev]'` when switching between checkouts and running tests. git_repo_path = abspath(join(dirname(dirname(dirname(__file__))), "src")) sys.path.insert(1, git_repo_path) diff --git a/examples/flax/image-captioning/README.md b/examples/flax/image-captioning/README.md index 0faf56124bc2d0..dd2b420639258f 100644 --- a/examples/flax/image-captioning/README.md +++ b/examples/flax/image-captioning/README.md @@ -1,7 +1,7 @@ # Image Captioning (vision-encoder-text-decoder model) training example The following example showcases how to finetune a vision-encoder-text-decoder model for image captioning -using the JAX/Flax backend, leveraging 🤗 Transformers library's [FlaxVisionEncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/visionencoderdecoder#transformers.FlaxVisionEncoderDecoderModel). +using the JAX/Flax backend, leveraging 🤗 Transformers library's [FlaxVisionEncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder#transformers.FlaxVisionEncoderDecoderModel). JAX/Flax allows you to trace pure functions and compile them into efficient, fused accelerator code on both GPU and TPU. Models written in JAX/Flax are **immutable** and updated in a purely functional @@ -10,7 +10,7 @@ way which enables simple and efficient model parallelism. `run_image_captioning_flax.py` is a lightweight example of how to download and preprocess a dataset from the 🤗 Datasets library or use your own files (jsonlines or csv), then fine-tune one of the architectures above on it. -For custom datasets in `jsonlines` format please see: https://huggingface.co/docs/datasets/loading_datasets.html#json-files and you also will find examples of these below. +For custom datasets in `jsonlines` format please see: https://huggingface.co/docs/datasets/loading_datasets#json-files and you also will find examples of these below. ### Download COCO dataset (2017) This example uses COCO dataset (2017) through a custom dataset script, which requires users to manually download the @@ -34,7 +34,7 @@ Next, we create a [FlaxVisionEncoderDecoderModel](https://huggingface.co/docs/tr python3 create_model_from_encoder_decoder_models.py \ --output_dir model \ --encoder_model_name_or_path google/vit-base-patch16-224-in21k \ - --decoder_model_name_or_path gpt2 + --decoder_model_name_or_path openai-community/gpt2 ``` ### Train the model diff --git a/examples/flax/image-captioning/create_model_from_encoder_decoder_models.py b/examples/flax/image-captioning/create_model_from_encoder_decoder_models.py index c5ce0e4ce133c4..0ebd1464883874 100644 --- a/examples/flax/image-captioning/create_model_from_encoder_decoder_models.py +++ b/examples/flax/image-captioning/create_model_from_encoder_decoder_models.py @@ -37,7 +37,7 @@ class ModelArguments: encoder_model_name_or_path: str = field( metadata={ "help": ( - "The encoder model checkpoint for weights initialization." + "The encoder model checkpoint for weights initialization. " "Don't set if you want to train an encoder model from scratch." ) }, @@ -45,7 +45,7 @@ class ModelArguments: decoder_model_name_or_path: str = field( metadata={ "help": ( - "The decoder model checkpoint for weights initialization." + "The decoder model checkpoint for weights initialization. " "Don't set if you want to train a decoder model from scratch." ) }, diff --git a/examples/flax/image-captioning/run_image_captioning_flax.py b/examples/flax/image-captioning/run_image_captioning_flax.py index ef9c515da450e9..879372a7523823 100644 --- a/examples/flax/image-captioning/run_image_captioning_flax.py +++ b/examples/flax/image-captioning/run_image_captioning_flax.py @@ -41,7 +41,7 @@ from flax.jax_utils import unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from PIL import Image from tqdm import tqdm @@ -53,7 +53,7 @@ HfArgumentParser, is_tensorboard_available, ) -from transformers.utils import get_full_repo_name, is_offline_mode, send_example_telemetry +from transformers.utils import is_offline_mode, send_example_telemetry logger = logging.getLogger(__name__) @@ -182,12 +182,22 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -239,7 +249,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the `max_length` param of `model.generate`, which is used " "during evaluation." ) @@ -364,7 +374,7 @@ def write_metric(summary_writer, metrics, train_time, step, metric_key_prefix="t def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -400,7 +410,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -424,14 +434,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -448,7 +457,8 @@ def main(): cache_dir=model_args.cache_dir, keep_in_memory=False, data_dir=data_args.data_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -465,28 +475,31 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer model = FlaxVisionEncoderDecoderModel.from_pretrained( model_args.model_name_or_path, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) image_processor = AutoImageProcessor.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer.pad_token = tokenizer.convert_ids_to_tokens(model.config.pad_token_id) @@ -659,7 +672,7 @@ def preprocess_fn(examples, max_target_length, check_image=True): eval_batch_size = int(training_args.per_device_eval_batch_size) * jax.device_count() if training_args.block_size % train_batch_size > 0 or training_args.block_size % eval_batch_size > 0: raise ValueError( - "`training_args.block_size` needs to be a multiple of the global train/eval batch size." + "`training_args.block_size` needs to be a multiple of the global train/eval batch size. " f"Got {training_args.block_size}, {train_batch_size} and {eval_batch_size} respectively instead." ) @@ -824,7 +837,7 @@ def blockwise_data_loader( yield batch # Metric - metric = evaluate.load("rouge") + metric = evaluate.load("rouge", cache_dir=model_args.cache_dir) def postprocess_text(preds, labels): preds = [pred.strip() for pred in preds] @@ -1032,7 +1045,13 @@ def save_ckpt(ckpt_dir: str, commit_msg: str = ""): model.save_pretrained(os.path.join(training_args.output_dir, ckpt_dir), params=params) tokenizer.save_pretrained(os.path.join(training_args.output_dir, ckpt_dir)) if training_args.push_to_hub: - repo.push_to_hub(commit_message=commit_msg, blocking=False) + api.upload_folder( + commit_message=commit_msg, + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) def evaluation_loop( rng: jax.random.PRNGKey, diff --git a/examples/flax/language-modeling/README.md b/examples/flax/language-modeling/README.md index 5346904d84c688..9b95d9ec0911bd 100644 --- a/examples/flax/language-modeling/README.md +++ b/examples/flax/language-modeling/README.md @@ -16,7 +16,7 @@ limitations under the License. # Language model training examples -The following example showcases how to train a language model from scratch +The following example showcases how to train a language model from scratch using the JAX/Flax backend. JAX/Flax allows you to trace pure functions and compile them into efficient, fused accelerator code on both GPU and TPU. @@ -25,10 +25,10 @@ way which enables simple and efficient model parallelism. ## Masked language modeling -In the following, we demonstrate how to train a bi-directional transformer model +In the following, we demonstrate how to train a bi-directional transformer model using masked language modeling objective as introduced in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805). -More specifically, we demonstrate how JAX/Flax can be leveraged -to pre-train [**`roberta-base`**](https://huggingface.co/roberta-base) +More specifically, we demonstrate how JAX/Flax can be leveraged +to pre-train [**`FacebookAI/roberta-base`**](https://huggingface.co/FacebookAI/roberta-base) in Norwegian on a single TPUv3-8 pod. The example script uses the 🤗 Datasets library. You can easily customize them to your needs if you need extra processing on your datasets. @@ -75,14 +75,14 @@ tokenizer.save("./norwegian-roberta-base/tokenizer.json") ### Create configuration -Next, we create the model's configuration file. This is as simple -as loading and storing [`**roberta-base**`](https://huggingface.co/roberta-base) +Next, we create the model's configuration file. This is as simple +as loading and storing [`**FacebookAI/roberta-base**`](https://huggingface.co/FacebookAI/roberta-base) in the local model folder: ```python from transformers import RobertaConfig -config = RobertaConfig.from_pretrained("roberta-base", vocab_size=50265) +config = RobertaConfig.from_pretrained("FacebookAI/roberta-base", vocab_size=50265) config.save_pretrained("./norwegian-roberta-base") ``` @@ -117,20 +117,20 @@ python run_mlm_flax.py \ --push_to_hub ``` -Training should converge at a loss and accuracy +Training should converge at a loss and accuracy of 1.78 and 0.64 respectively after 18 epochs on a single TPUv3-8. This should take less than 18 hours. Training statistics can be accessed on [tfhub.dev](https://tensorboard.dev/experiment/GdYmdak2TWeVz0DDRYOrrg). -For a step-by-step walkthrough of how to do masked language modeling in Flax, please have a +For a step-by-step walkthrough of how to do masked language modeling in Flax, please have a look at [this](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb) google colab. ## Causal language modeling -In the following, we demonstrate how to train an auto-regressive causal transformer model +In the following, we demonstrate how to train an auto-regressive causal transformer model in JAX/Flax. -More specifically, we pretrain a randomly initialized [**`gpt2`**](https://huggingface.co/gpt2) model in Norwegian on a single TPUv3-8. -to pre-train 124M [**`gpt2`**](https://huggingface.co/gpt2) +More specifically, we pretrain a randomly initialized [**`openai-community/gpt2`**](https://huggingface.co/openai-community/gpt2) model in Norwegian on a single TPUv3-8. +to pre-train 124M [**`openai-community/gpt2`**](https://huggingface.co/openai-community/gpt2) in Norwegian on a single TPUv3-8 pod. The example script uses the 🤗 Datasets library. You can easily customize them to your needs if you need extra processing on your datasets. @@ -178,14 +178,14 @@ tokenizer.save("./norwegian-gpt2/tokenizer.json") ### Create configuration -Next, we create the model's configuration file. This is as simple -as loading and storing [`**gpt2**`](https://huggingface.co/gpt2) +Next, we create the model's configuration file. This is as simple +as loading and storing [`**openai-community/gpt2**`](https://huggingface.co/openai-community/gpt2) in the local model folder: ```python from transformers import GPT2Config -config = GPT2Config.from_pretrained("gpt2", resid_pdrop=0.0, embd_pdrop=0.0, attn_pdrop=0.0, vocab_size=50257) +config = GPT2Config.from_pretrained("openai-community/gpt2", resid_pdrop=0.0, embd_pdrop=0.0, attn_pdrop=0.0, vocab_size=50257) config.save_pretrained("./norwegian-gpt2") ``` @@ -218,19 +218,19 @@ python run_clm_flax.py \ --push_to_hub ``` -Training should converge at a loss and perplexity +Training should converge at a loss and perplexity of 3.24 and 25.72 respectively after 20 epochs on a single TPUv3-8. This should take less than ~21 hours. Training statistics can be accessed on [tfhub.de](https://tensorboard.dev/experiment/2zEhLwJ0Qp2FAkI3WVH9qA). -For a step-by-step walkthrough of how to do causal language modeling in Flax, please have a +For a step-by-step walkthrough of how to do causal language modeling in Flax, please have a look at [this](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb) google colab. ## T5-like span-masked language modeling -In the following, we demonstrate how to train a T5 model using the span-masked language model +In the following, we demonstrate how to train a T5 model using the span-masked language model objective as proposed in the [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683). -More specifically, we demonstrate how JAX/Flax can be leveraged +More specifically, we demonstrate how JAX/Flax can be leveraged to pre-train [**`google/t5-v1_1-base`**](https://huggingface.co/google/t5-v1_1-base) in Norwegian on a single TPUv3-8 pod. @@ -247,9 +247,9 @@ cd ./norwegian-t5-base ### Train tokenizer -In the first step, we train a tokenizer to efficiently process the text input for the model. -We make use of the [tokenizers](https://github.com/huggingface/tokenizers) library to train -a sentencepiece unigram tokenizer as shown in [t5_tokenizer_model.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling/t5_tokenizer_model.py) +In the first step, we train a tokenizer to efficiently process the text input for the model. +We make use of the [tokenizers](https://github.com/huggingface/tokenizers) library to train +a sentencepiece unigram tokenizer as shown in [t5_tokenizer_model.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling/t5_tokenizer_model.py) which is heavily inspired from [yandex-research/DeDLOC's tokenizer model](https://github.com/yandex-research/DeDLOC/blob/5c994bc64e573702a9a79add3ecd68b38f14b548/sahajbert/tokenizer/tokenizer_model.py) . The tokenizer is trained on the complete Norwegian dataset of OSCAR @@ -293,7 +293,7 @@ tokenizer.save("./norwegian-t5-base/tokenizer.json") ### Create configuration -Next, we create the model's configuration file. This is as simple +Next, we create the model's configuration file. This is as simple as loading and storing [`**google/t5-v1_1-base**`](https://huggingface.co/google/t5-v1_1-base) in the local model folder: @@ -333,16 +333,16 @@ python run_t5_mlm_flax.py \ --push_to_hub ``` -Training should converge at a loss and accuracy +Training should converge at a loss and accuracy of 2.36 and 57.0 respectively after 3 epochs on a single TPUv3-8. This should take around 4.5 hours. Training statistics can be accessed on directly on the 🤗 [hub](https://huggingface.co/patrickvonplaten/t5-base-norwegian/tensorboard) ## BART: Denoising language modeling -In the following, we demonstrate how to train a BART model +In the following, we demonstrate how to train a BART model using denoising language modeling objective as introduced in [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/abs/1910.13461). -More specifically, we demonstrate how JAX/Flax can be leveraged +More specifically, we demonstrate how JAX/Flax can be leveraged to pre-train [**`bart-base`**](https://huggingface.co/facebook/bart-base) in Norwegian on a single TPUv3-8 pod. @@ -389,7 +389,7 @@ tokenizer.save("./norwegian-bart-base/tokenizer.json") ### Create configuration -Next, we create the model's configuration file. This is as simple +Next, we create the model's configuration file. This is as simple as loading and storing [`**facebook/bart-base**`](https://huggingface.co/facebook/bart-base) in the local model folder: @@ -425,7 +425,7 @@ python run_bart_dlm_flax.py \ --push_to_hub ``` -Training should converge at a loss and accuracy +Training should converge at a loss and accuracy of 1.36 and 0.77 respectively after 3 epochs on a single TPUv3-8. This should take less than 6 hours. Training statistics can be accessed on [tfhub.dev](https://tensorboard.dev/experiment/Maw62QlaSXWS0MOf2V2lbg/). @@ -440,16 +440,16 @@ For reproducibility, we state the training commands used for PyTorch/XLA and PyT |-------|-----------|------------|------------| | MLM | 15h32m | 23h46m | 44h14m | -*All experiments are ran on Google Cloud Platform. +*All experiments are ran on Google Cloud Platform. GPU experiments are ran without further optimizations besides JAX transformations. GPU experiments are ran with full precision (fp32). "TPU v3-8" are 8 TPU cores on 4 chips (each chips has 2 cores), while "8 GPU" are 8 GPU chips. ### Script to run MLM with PyTorch/XLA on TPUv3-8 -For comparison one can run the same pre-training with PyTorch/XLA on TPU. To set up PyTorch/XLA on Cloud TPU VMs, please +For comparison one can run the same pre-training with PyTorch/XLA on TPU. To set up PyTorch/XLA on Cloud TPU VMs, please refer to [this](https://cloud.google.com/tpu/docs/pytorch-xla-ug-tpu-vm) guide. -Having created the tokenzier and configuration in `norwegian-roberta-base`, we create the following symbolic links: +Having created the tokenizer and configuration in `norwegian-roberta-base`, we create the following symbolic links: ```bash ln -s ~/transformers/examples/pytorch/language-modeling/run_mlm.py ./ @@ -490,16 +490,16 @@ python3 xla_spawn.py --num_cores ${NUM_TPUS} run_mlm.py --output_dir="./runs" \ --do_train \ --do_eval \ --logging_steps="500" \ - --evaluation_strategy="epoch" \ + --eval_strategy="epoch" \ --report_to="tensorboard" \ --save_strategy="no" ``` ### Script to compare pre-training with PyTorch on 8 GPU V100's -For comparison you can run the same pre-training with PyTorch on GPU. Note that we have to make use of `gradient_accumulation` +For comparison you can run the same pre-training with PyTorch on GPU. Note that we have to make use of `gradient_accumulation` because the maximum batch size that fits on a single V100 GPU is 32 instead of 128. -Having created the tokenzier and configuration in `norwegian-roberta-base`, we create the following symbolic links: +Having created the tokenizer and configuration in `norwegian-roberta-base`, we create the following symbolic links: ```bash ln -s ~/transformers/examples/pytorch/language-modeling/run_mlm.py ./ @@ -538,7 +538,7 @@ python3 -m torch.distributed.launch --nproc_per_node ${NUM_GPUS} run_mlm.py \ --do_train \ --do_eval \ --logging_steps="500" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --report_to="tensorboard" \ --save_strategy="no" ``` diff --git a/examples/flax/language-modeling/run_bart_dlm_flax.py b/examples/flax/language-modeling/run_bart_dlm_flax.py index 4cb862bb37b94c..1f87eedd8a6aea 100644 --- a/examples/flax/language-modeling/run_bart_dlm_flax.py +++ b/examples/flax/language-modeling/run_bart_dlm_flax.py @@ -43,7 +43,7 @@ from flax.jax_utils import pad_shard_unpad from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm from transformers import ( @@ -59,7 +59,7 @@ set_seed, ) from transformers.models.bart.modeling_flax_bart import shift_tokens_right -from transformers.utils import get_full_repo_name, send_example_telemetry +from transformers.utils import send_example_telemetry MODEL_CONFIG_CLASSES = list(FLAX_MODEL_FOR_MASKED_LM_MAPPING.keys()) @@ -138,7 +138,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -168,12 +168,12 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( - default=False, + token: str = field( + default=None, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." ) }, ) @@ -191,6 +191,16 @@ class DataTrainingArguments: dataset_config_name: Optional[str] = field( default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."} ) + trust_remote_code: bool = field( + default=False, + metadata={ + "help": ( + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." + ) + }, + ) train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."}) validation_file: Optional[str] = field( default=None, @@ -242,10 +252,12 @@ def __post_init__(self): else: if self.train_file is not None: extension = self.train_file.split(".")[-1] - assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, a json or a txt file." + if extension not in ["csv", "json", "txt"]: + raise ValueError("train_file` should be a csv, json or text file.") if self.validation_file is not None: extension = self.validation_file.split(".")[-1] - assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, a json or a txt file." + if extension not in ["csv", "json", "txt"]: + raise ValueError("`validation_file` should be a csv, json or text file.") @flax.struct.dataclass @@ -278,7 +290,7 @@ class FlaxDataCollatorForBartDenoisingLM: def __post_init__(self): if self.tokenizer.mask_token is None or self.tokenizer.eos_token is None: raise ValueError( - "This tokenizer does not have a mask token or eos token token which is necessary for denoising" + "This tokenizer does not have a mask token or eos token which is necessary for denoising" " language modeling. " ) @@ -472,7 +484,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -494,14 +506,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -515,7 +526,9 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) if "validation" not in datasets.keys(): @@ -524,29 +537,35 @@ def main(): data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) datasets["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) else: data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" datasets = load_dataset( extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) if "validation" not in datasets.keys(): @@ -555,17 +574,19 @@ def main(): data_files=data_files, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) datasets["train"] = load_dataset( extension, data_files=data_files, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -574,18 +595,18 @@ def main(): model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -594,13 +615,13 @@ def main(): model_args.config_name, cache_dir=model_args.cache_dir, vocab_size=len(tokenizer), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) elif model_args.model_name_or_path: config = BartConfig.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -669,7 +690,7 @@ def group_texts(examples): # might be slower to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map tokenized_datasets = tokenized_datasets.map( group_texts, batched=True, @@ -705,7 +726,7 @@ def group_texts(examples): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: config.vocab_size = len(tokenizer) @@ -924,7 +945,13 @@ def eval_step(params, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) # Eval after training if training_args.do_eval: diff --git a/examples/flax/language-modeling/run_clm_flax.py b/examples/flax/language-modeling/run_clm_flax.py index 952419dc965656..c486aae71f6227 100755 --- a/examples/flax/language-modeling/run_clm_flax.py +++ b/examples/flax/language-modeling/run_clm_flax.py @@ -43,7 +43,7 @@ from flax.jax_utils import pad_shard_unpad, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm import transformers @@ -58,7 +58,7 @@ set_seed, ) from transformers.testing_utils import CaptureLogger -from transformers.utils import get_full_repo_name, send_example_telemetry +from transformers.utils import send_example_telemetry logger = logging.getLogger(__name__) @@ -139,7 +139,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -169,12 +169,22 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -215,9 +225,6 @@ class DataTrainingArguments: ) }, ) - overwrite_cache: bool = field( - default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} - ) validation_split_percentage: Optional[int] = field( default=5, metadata={ @@ -251,10 +258,12 @@ def __post_init__(self): else: if self.train_file is not None: extension = self.train_file.split(".")[-1] - assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, a json or a txt file." + if extension not in ["csv", "json", "txt"]: + raise ValueError("train_file` should be a csv, json or text file.") if self.validation_file is not None: extension = self.validation_file.split(".")[-1] - assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, a json or a txt file." + if extension not in ["csv", "json", "txt"]: + raise ValueError("`validation_file` should be a csv, json or text file.") class TrainState(train_state.TrainState): @@ -307,7 +316,7 @@ def write_eval_metric(summary_writer, eval_metrics, step): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -343,7 +352,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -370,14 +379,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -395,7 +403,9 @@ def main(): data_args.dataset_config_name, cache_dir=model_args.cache_dir, keep_in_memory=False, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) if "validation" not in dataset.keys(): @@ -404,23 +414,28 @@ def main(): data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) dataset["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} dataset_args = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" dataset_args["keep_linebreaks"] = data_args.keep_linebreaks @@ -429,7 +444,8 @@ def main(): data_files=data_files, cache_dir=model_args.cache_dir, **dataset_args, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) if "validation" not in dataset.keys(): @@ -439,7 +455,8 @@ def main(): split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, **dataset_args, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) dataset["train"] = load_dataset( extension, @@ -447,10 +464,11 @@ def main(): split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, **dataset_args, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -461,13 +479,15 @@ def main(): config = AutoConfig.from_pretrained( model_args.config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: config = AutoConfig.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -478,18 +498,20 @@ def main(): model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -499,13 +521,15 @@ def main(): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: model = FlaxAutoModelForCausalLM.from_config( config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), + trust_remote_code=model_args.trust_remote_code, ) # Preprocessing the datasets. @@ -543,13 +567,13 @@ def tokenize_function(examples): if block_size > config.max_position_embeddings: logger.warning( f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " - "Picking 1024 instead. You can change that default value by passing --block_size xxx." + f"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx." ) - block_size = 1024 + block_size = min(1024, config.max_position_embeddings) else: if data_args.block_size > tokenizer.model_max_length: logger.warning( - f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" + f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model " f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." ) block_size = min(data_args.block_size, tokenizer.model_max_length) @@ -576,7 +600,7 @@ def group_texts(examples): # to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map lm_datasets = tokenized_datasets.map( group_texts, @@ -806,8 +830,13 @@ def eval_step(params, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) - + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) # Eval after training if training_args.do_eval: eval_metrics = [] diff --git a/examples/flax/language-modeling/run_mlm_flax.py b/examples/flax/language-modeling/run_mlm_flax.py index ae289b84708f2c..4d837e9c113c3b 100755 --- a/examples/flax/language-modeling/run_mlm_flax.py +++ b/examples/flax/language-modeling/run_mlm_flax.py @@ -20,6 +20,7 @@ Here is the full list of checkpoints on the hub that can be fine-tuned by this script: https://huggingface.co/models?filter=fill-mask """ + import json import logging import math @@ -44,7 +45,7 @@ from flax.jax_utils import pad_shard_unpad from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm from transformers import ( @@ -59,7 +60,7 @@ is_tensorboard_available, set_seed, ) -from transformers.utils import get_full_repo_name, send_example_telemetry +from transformers.utils import send_example_telemetry MODEL_CONFIG_CLASSES = list(FLAX_MODEL_FOR_MASKED_LM_MAPPING.keys()) @@ -144,7 +145,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -174,12 +175,22 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -388,7 +399,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -410,14 +421,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -434,7 +444,9 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) if "validation" not in datasets.keys(): @@ -443,29 +455,35 @@ def main(): data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) datasets["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" datasets = load_dataset( extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) if "validation" not in datasets.keys(): @@ -474,17 +492,19 @@ def main(): data_files=data_files, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) datasets["train"] = load_dataset( extension, data_files=data_files, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -495,13 +515,15 @@ def main(): config = AutoConfig.from_pretrained( model_args.config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: config = AutoConfig.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -512,18 +534,20 @@ def main(): model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -598,7 +622,7 @@ def group_texts(examples): # might be slower to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map tokenized_datasets = tokenized_datasets.map( group_texts, batched=True, @@ -638,13 +662,15 @@ def group_texts(examples): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: model = FlaxAutoModelForMaskedLM.from_config( config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), + trust_remote_code=model_args.trust_remote_code, ) if training_args.gradient_checkpointing: @@ -851,8 +877,13 @@ def eval_step(params, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) - + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) # Eval after training if training_args.do_eval: num_eval_samples = len(tokenized_datasets["validation"]) diff --git a/examples/flax/language-modeling/run_t5_mlm_flax.py b/examples/flax/language-modeling/run_t5_mlm_flax.py index f3cec97b2e82b3..c133824fcc2c18 100755 --- a/examples/flax/language-modeling/run_t5_mlm_flax.py +++ b/examples/flax/language-modeling/run_t5_mlm_flax.py @@ -19,6 +19,7 @@ Here is the full list of checkpoints on the hub that can be pretrained by this script: https://huggingface.co/models?filter=t5 """ + import json import logging import math @@ -43,7 +44,7 @@ from flax.jax_utils import pad_shard_unpad from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm from transformers import ( @@ -59,7 +60,7 @@ set_seed, ) from transformers.models.t5.modeling_flax_t5 import shift_tokens_right -from transformers.utils import get_full_repo_name, send_example_telemetry +from transformers.utils import send_example_telemetry MODEL_CONFIG_CLASSES = list(FLAX_MODEL_FOR_MASKED_LM_MAPPING.keys()) @@ -138,7 +139,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -168,12 +169,12 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( - default=False, + token: str = field( + default=None, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." ) }, ) @@ -191,6 +192,16 @@ class DataTrainingArguments: dataset_config_name: Optional[str] = field( default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."} ) + trust_remote_code: bool = field( + default=False, + metadata={ + "help": ( + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." + ) + }, + ) train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."}) validation_file: Optional[str] = field( default=None, @@ -515,7 +526,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -537,14 +548,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -558,7 +568,9 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) if "validation" not in datasets.keys(): @@ -567,29 +579,35 @@ def main(): data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) datasets["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, + trust_remote_code=data_args.trust_remote_code, ) else: data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" datasets = load_dataset( extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) if "validation" not in datasets.keys(): @@ -598,17 +616,19 @@ def main(): data_files=data_files, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) datasets["train"] = load_dataset( extension, data_files=data_files, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + num_proc=data_args.preprocessing_num_workers, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -617,18 +637,18 @@ def main(): model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -637,13 +657,13 @@ def main(): model_args.config_name, cache_dir=model_args.cache_dir, vocab_size=len(tokenizer), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) elif model_args.model_name_or_path: config = T5Config.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -702,7 +722,7 @@ def group_texts(examples): # might be slower to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map tokenized_datasets = tokenized_datasets.map( group_texts, batched=True, @@ -738,7 +758,7 @@ def group_texts(examples): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: config.vocab_size = len(tokenizer) @@ -954,8 +974,13 @@ def eval_step(params, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) - + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) # Eval after training if training_args.do_eval: num_eval_samples = len(tokenized_datasets["validation"]) diff --git a/examples/flax/language-modeling/t5_tokenizer_model.py b/examples/flax/language-modeling/t5_tokenizer_model.py index fbccd52bd8c726..a2be4afc946284 100755 --- a/examples/flax/language-modeling/t5_tokenizer_model.py +++ b/examples/flax/language-modeling/t5_tokenizer_model.py @@ -46,12 +46,16 @@ def __init__( ) tokenizer.pre_tokenizer = pre_tokenizers.Sequence( [ - pre_tokenizers.Metaspace(replacement=replacement, add_prefix_space=add_prefix_space), + pre_tokenizers.Metaspace( + replacement=replacement, prepend_scheme="always" if add_prefix_space else "never" + ), pre_tokenizers.Digits(individual_digits=True), pre_tokenizers.Punctuation(), ] ) - tokenizer.decoder = decoders.Metaspace(replacement=replacement, add_prefix_space=add_prefix_space) + tokenizer.decoder = decoders.Metaspace( + replacement=replacement, prepend_scheme="always" if add_prefix_space else "never" + ) tokenizer.post_processor = TemplateProcessing( single=f"$A {self.special_tokens['eos']['token']}", diff --git a/examples/flax/question-answering/README.md b/examples/flax/question-answering/README.md index 822342a99e2168..2f6caa984d4bc1 100644 --- a/examples/flax/question-answering/README.md +++ b/examples/flax/question-answering/README.md @@ -29,7 +29,7 @@ The following example fine-tunes BERT on SQuAD: ```bash python run_qa.py \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --dataset_name squad \ --do_train \ --do_eval \ @@ -67,7 +67,7 @@ Here is an example training on 4 TITAN RTX GPUs and Bert Whole Word Masking unca ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 python run_qa.py \ ---model_name_or_path bert-large-uncased-whole-word-masking \ +--model_name_or_path google-bert/bert-large-uncased-whole-word-masking \ --dataset_name squad \ --do_train \ --do_eval \ diff --git a/examples/flax/question-answering/run_qa.py b/examples/flax/question-answering/run_qa.py index 7f18ef0a69c0d7..969ee5e5d5291c 100644 --- a/examples/flax/question-answering/run_qa.py +++ b/examples/flax/question-answering/run_qa.py @@ -41,7 +41,7 @@ from flax.jax_utils import pad_shard_unpad, replicate, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm from utils_qa import postprocess_qa_predictions @@ -55,13 +55,13 @@ PreTrainedTokenizerFast, is_tensorboard_available, ) -from transformers.utils import check_min_version, get_full_repo_name, send_example_telemetry +from transformers.utils import check_min_version, send_example_telemetry logger = logging.getLogger(__name__) # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") Array = Any Dataset = datasets.arrow_dataset.Dataset @@ -155,12 +155,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -372,7 +382,7 @@ def cross_entropy_loss(logits, labels): # region Create learning rate function def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -416,7 +426,8 @@ def eval_data_collator(dataset: Dataset, batch_size: int): for idx in batch_idx: batch = dataset[idx] - batch = {k: np.array(v) for k, v in batch.items()} + # Ignore `offset_mapping` to avoid numpy/JAX array conversion issue. + batch = {k: np.array(v) for k, v in batch.items() if k != "offset_mapping"} yield batch @@ -462,14 +473,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # region Load Data # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) @@ -487,7 +497,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: # Loading the dataset from local csv or json file. @@ -507,10 +518,10 @@ def main(): data_files=data_files, field="data", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Load pretrained model and tokenizer @@ -520,14 +531,16 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=True, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -557,7 +570,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -646,7 +659,7 @@ def prepare_train_features(examples): raise ValueError("--do_train requires a train dataset") train_dataset = raw_datasets["train"] if data_args.max_train_samples is not None: - # We will select sample from whole data if agument is specified + # We will select sample from whole data if argument is specified max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) # Create train feature from dataset @@ -779,7 +792,9 @@ def post_processing_function(examples, features, predictions, stage="eval"): references = [{"id": ex["id"], "answers": ex[answer_column_name]} for ex in examples] return EvalPrediction(predictions=formatted_predictions, label_ids=references) - metric = evaluate.load("squad_v2" if data_args.version_2_with_negative else "squad") + metric = evaluate.load( + "squad_v2" if data_args.version_2_with_negative else "squad", cache_dir=model_args.cache_dir + ) def compute_metrics(p: EvalPrediction): return metric.compute(predictions=p.predictions, references=p.label_ids) @@ -874,7 +889,8 @@ def write_eval_metric(summary_writer, eval_metrics, step): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), ) @@ -986,7 +1002,6 @@ def eval_step(state, batch): position=2, ): _ = batch.pop("example_id") - _ = batch.pop("offset_mapping") predictions = pad_shard_unpad(p_eval_step)( state, batch, min_device_batch=per_device_eval_batch_size ) @@ -1020,7 +1035,13 @@ def eval_step(state, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) epochs.desc = f"Epoch ... {epoch + 1}/{num_epochs}" # endregion @@ -1035,7 +1056,6 @@ def eval_step(state, batch): eval_loader, total=math.ceil(len(eval_dataset) / eval_batch_size), desc="Evaluating ...", position=2 ): _ = batch.pop("example_id") - _ = batch.pop("offset_mapping") predictions = pad_shard_unpad(p_eval_step)(state, batch, min_device_batch=per_device_eval_batch_size) start_logits = np.array(predictions[0]) end_logits = np.array(predictions[1]) diff --git a/examples/flax/question-answering/utils_qa.py b/examples/flax/question-answering/utils_qa.py index 23a46370d17393..79497dbb816ed2 100644 --- a/examples/flax/question-answering/utils_qa.py +++ b/examples/flax/question-answering/utils_qa.py @@ -15,6 +15,7 @@ """ Post-processing utilities for question answering. """ + import collections import json import logging diff --git a/examples/flax/speech-recognition/README.md b/examples/flax/speech-recognition/README.md new file mode 100644 index 00000000000000..943c98761aa660 --- /dev/null +++ b/examples/flax/speech-recognition/README.md @@ -0,0 +1,68 @@ + + +# Automatic Speech Recognition - Flax Examples + +## Sequence to Sequence + +The script [`run_flax_speech_recognition_seq2seq.py`](https://github.com/huggingface/transformers/blob/main/examples/flax/speech-recognition/run_flax_speech_recognition_seq2seq.py) +can be used to fine-tune any [Flax Speech Sequence-to-Sequence Model](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.FlaxAutoModelForSpeechSeq2Seq) +for automatic speech recognition on one of the [official speech recognition datasets](https://huggingface.co/datasets?task_ids=task_ids:automatic-speech-recognition) +or a custom dataset. This includes the Whisper model from OpenAI, or a warm-started Speech-Encoder-Decoder Model, +an example for which is included below. + +### Whisper Model + +We can load all components of the Whisper model directly from the pretrained checkpoint, including the pretrained model +weights, feature extractor and tokenizer. We simply have to specify the id of fine-tuning dataset and the necessary +training hyperparameters. + +The following example shows how to fine-tune the [Whisper small](https://huggingface.co/openai/whisper-small) checkpoint +on the Hindi subset of the [Common Voice 13](https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0) dataset. +Note that before running this script you must accept the dataset's [terms of use](https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0) +and register your Hugging Face Hub token on your device by running `huggingface-hub login`. + +```bash +python run_flax_speech_recognition_seq2seq.py \ + --model_name_or_path="openai/whisper-small" \ + --dataset_name="mozilla-foundation/common_voice_13_0" \ + --dataset_config_name="hi" \ + --language="hindi" \ + --train_split_name="train+validation" \ + --eval_split_name="test" \ + --output_dir="./whisper-small-hi-flax" \ + --per_device_train_batch_size="16" \ + --per_device_eval_batch_size="16" \ + --num_train_epochs="10" \ + --learning_rate="1e-4" \ + --warmup_steps="500" \ + --logging_steps="25" \ + --generation_max_length="40" \ + --preprocessing_num_workers="32" \ + --dataloader_num_workers="32" \ + --max_duration_in_seconds="30" \ + --text_column_name="sentence" \ + --overwrite_output_dir \ + --do_train \ + --do_eval \ + --predict_with_generate \ + --push_to_hub \ + --use_auth_token +``` + +On a TPU v4-8, training should take approximately 25 minutes, with a final cross-entropy loss of 0.02 and word error +rate of **34%**. See the checkpoint [sanchit-gandhi/whisper-small-hi-flax](https://huggingface.co/sanchit-gandhi/whisper-small-hi-flax) +for an example training run. diff --git a/examples/flax/speech-recognition/requirements.txt b/examples/flax/speech-recognition/requirements.txt new file mode 100644 index 00000000000000..b68b236ad76c2b --- /dev/null +++ b/examples/flax/speech-recognition/requirements.txt @@ -0,0 +1,8 @@ +datasets[audio]>=2.14.0 +jax>=0.3.6 +jaxlib>=0.3.6 +flax>=0.4.1 +optax>=0.0.8 +torch>=1.9.0 +jiwer +evaluate diff --git a/examples/flax/speech-recognition/run_flax_speech_recognition_seq2seq.py b/examples/flax/speech-recognition/run_flax_speech_recognition_seq2seq.py new file mode 100644 index 00000000000000..30e66065d8a943 --- /dev/null +++ b/examples/flax/speech-recognition/run_flax_speech_recognition_seq2seq.py @@ -0,0 +1,878 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2023 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Fine-tuning the Flax library models for sequence to sequence speech recognition. +""" +# You can also adapt this script on your own sequence to sequence task. Pointers for this are left as comments. + +import logging +import os +import sys +import time +from dataclasses import field +from functools import partial +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional, Union + +import datasets +import evaluate +import flax +import jax +import jax.numpy as jnp +import numpy as np +import optax +from datasets import DatasetDict, load_dataset +from flax import jax_utils, traverse_util +from flax.jax_utils import pad_shard_unpad, unreplicate +from flax.training import train_state +from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key +from huggingface_hub import HfApi +from torch.utils.data import DataLoader +from tqdm import tqdm + +import transformers +from transformers import ( + AutoConfig, + AutoFeatureExtractor, + AutoProcessor, + AutoTokenizer, + FlaxAutoModelForSpeechSeq2Seq, + HfArgumentParser, + Seq2SeqTrainingArguments, + is_tensorboard_available, +) +from transformers.file_utils import get_full_repo_name +from transformers.utils import check_min_version, send_example_telemetry +from transformers.utils.versions import require_version + + +# Will error if the minimal version of Transformers is not installed. Remove at your own risk. +check_min_version("4.44.0") + +require_version("datasets>=2.14.0", "To fix: pip install -r examples/flax/speech-recognition/requirements.txt") + +logger = logging.getLogger(__name__) + + +@flax.struct.dataclass +class ModelArguments: + """ + Arguments pertaining to which model/config/tokenizer we are going to fine-tune from. + """ + + model_name_or_path: str = field( + metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"} + ) + config_name: Optional[str] = field( + default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"} + ) + tokenizer_name: Optional[str] = field( + default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"} + ) + feature_extractor_name: Optional[str] = field( + default=None, metadata={"help": "feature extractor name or path if not the same as model_name"} + ) + cache_dir: Optional[str] = field( + default=None, + metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"}, + ) + use_fast_tokenizer: bool = field( + default=True, + metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."}, + ) + model_revision: str = field( + default="main", + metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, + ) + use_auth_token: bool = field( + default=False, + metadata={ + "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " + "with private models)." + }, + ) + dtype: Optional[str] = field( + default="float32", + metadata={ + "help": ( + "Floating-point format in which the model weights should be initialized and trained. Choose one of" + " `[float32, float16, bfloat16]`." + ) + }, + ) + num_beams: Optional[int] = field( + default=None, + metadata={ + "help": ( + "Number of beams to use for evaluation. This argument will be passed to `model.generate`, " + "which is used during evaluation." + ) + }, + ) + + +@flax.struct.dataclass +class DataTrainingArguments: + """ + Arguments pertaining to what data we are going to input our model for training and eval. + """ + + dataset_name: str = field( + default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."} + ) + dataset_config_name: Optional[str] = field( + default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."} + ) + trust_remote_code: bool = field( + default=False, + metadata={ + "help": ( + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." + ) + }, + ) + text_column: Optional[str] = field( + default=None, + metadata={"help": "The name of the column in the datasets containing the full texts (for summarization)."}, + ) + dataset_cache_dir: Optional[str] = field( + default=None, metadata={"help": "Path to cache directory for saving and loading datasets"} + ) + overwrite_cache: bool = field( + default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} + ) + preprocessing_num_workers: Optional[int] = field( + default=None, + metadata={"help": "The number of processes to use for the preprocessing."}, + ) + max_train_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of training examples to this " + "value if set." + }, + ) + max_eval_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this " + "value if set." + }, + ) + audio_column_name: str = field( + default="audio", + metadata={"help": "The name of the dataset column containing the audio data. Defaults to 'audio'"}, + ) + text_column_name: str = field( + default="text", + metadata={"help": "The name of the dataset column containing the text data. Defaults to 'text'"}, + ) + max_duration_in_seconds: float = field( + default=20.0, + metadata={"help": "Filter audio files that are longer than `max_duration_in_seconds` seconds"}, + ) + min_duration_in_seconds: float = field( + default=0.0, + metadata={"help": "Filter audio files that are shorter than `min_duration_in_seconds` seconds"}, + ) + max_label_length: float = field( + default=128, + metadata={"help": "Truncate transcriptions that are longer `max_eval_length` tokens."}, + ) + pad_input_to_multiple_of: Optional[int] = field( + default=None, + metadata={ + "help": "If set will pad the input sequence to a multiple of the provided value. " + "This is important to avoid triggering recompilations on TPU. If unspecified, will default to padding the inputs to max length." + }, + ) + pad_target_to_multiple_of: Optional[int] = field( + default=None, + metadata={ + "help": "If set will pad the target sequence to a multiple of the provided value. " + "This is important to avoid triggering recompilations on TPU. If unspecified, will default to padding the targets to max length." + }, + ) + preprocessing_only: bool = field( + default=False, + metadata={ + "help": "Whether to only do data preprocessing and skip training. " + "This is especially useful when data preprocessing errors out in distributed training due to timeout. " + "In this case, one should run the preprocessing in a non-distributed setup with `preprocessing_only=True` " + "so that the cached datasets can consequently be loaded in distributed training" + }, + ) + train_split_name: str = field( + default="train", + metadata={ + "help": "The name of the training data set split to use (via the datasets library). Defaults to 'train'" + }, + ) + eval_split_name: str = field( + default="validation", + metadata={ + "help": "The name of the evaluation data set split to use (via the datasets library). Defaults to 'validation'" + }, + ) + do_lower_case: bool = field( + default=True, + metadata={"help": "Whether the target text should be lower cased."}, + ) + language: str = field( + default=None, + metadata={ + "help": ( + "Language for multilingual fine-tuning. This argument should be set for multilingual fine-tuning " + "only. For English speech recognition, it should be set to `None`." + ) + }, + ) + task: str = field( + default="transcribe", + metadata={"help": "Task, either `transcribe` for speech recognition or `translate` for speech translation."}, + ) + + +def shift_tokens_right(label_ids: np.array, decoder_start_token_id: int) -> np.ndarray: + """ + Shift label ids one token to the right. + """ + shifted_label_ids = np.zeros_like(label_ids) + shifted_label_ids[:, 1:] = label_ids[:, :-1] + shifted_label_ids[:, 0] = decoder_start_token_id + + return shifted_label_ids + + +@flax.struct.dataclass +class FlaxDataCollatorSpeechSeq2SeqWithPadding: + """ + Data collator that will dynamically pad the inputs received. + Args: + processor ([`Wav2Vec2Processor`]) + The processor used for proccessing the data. + decoder_start_token_id (:obj: `int`) + The begin-of-sentence of the decoder. + input_padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`): + Select a strategy to pad the returned input sequences (according to the model's padding side and padding index) + among: + * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single + sequence if provided). + * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the + maximum acceptable input length for the model if that argument is not provided. + * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of + different lengths). + target_padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`): + Select a strategy to pad the returned target sequences (according to the model's padding side and padding index). + See above for details. + max_input_length (:obj:`float`, `optional`): + Maximum length of the ``input_values`` of the returned list and optionally padding length (see above). + max_target_length (:obj:`int`, `optional`): + Maximum length of the ``labels`` of the returned list and optionally padding length (see above). + pad_input_to_multiple_of (:obj:`int`, `optional`): + If set will pad the input sequence to a multiple of the provided value. + This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= + 7.5 (Volta). + pad_target_to_multiple_of (:obj:`int`, `optional`): + If set will pad the target sequence to a multiple of the provided value. + This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= + 7.5 (Volta). + """ + + processor: Any + decoder_start_token_id: int + input_padding: Union[bool, str] = "longest" + target_padding: Union[bool, str] = "max_length" + max_input_length: Optional[float] = None + max_target_length: Optional[int] = None + pad_input_to_multiple_of: Optional[int] = None + pad_target_to_multiple_of: Optional[int] = None + + def __call__(self, features: List[Dict[str, Union[List[int], np.ndarray]]]) -> Dict[str, np.ndarray]: + # split inputs and labels since they have to be of different lengths and need + # different padding methods + model_input_name = self.processor.model_input_names[0] + + # dataloader returns a list of features which we convert to a dict + input_features = {model_input_name: [feature[model_input_name] for feature in features]} + label_features = {"input_ids": [feature["labels"] for feature in features]} + + # reformat list to dict and set to pytorch format + batch = self.processor.feature_extractor.pad( + input_features, + max_length=self.max_input_length, + padding=self.input_padding, + pad_to_multiple_of=self.pad_input_to_multiple_of, + return_tensors="np", + ) + + labels_batch = self.processor.tokenizer.pad( + label_features, + max_length=self.max_target_length, + padding=self.target_padding, + pad_to_multiple_of=self.pad_target_to_multiple_of, + return_tensors="np", + ) + + # if bos token is appended in previous tokenization step, + # cut bos token here as it's append later anyways + labels = labels_batch["input_ids"] + if (labels[:, 0] == self.decoder_start_token_id).all().item(): + labels = labels[:, 1:] + labels_batch.attention_mask = labels_batch.attention_mask[:, 1:] + + decoder_input_ids = shift_tokens_right(labels, self.decoder_start_token_id) + + # replace padding with -100 to ignore correctly when computing the loss + labels = np.ma.array(labels, mask=np.not_equal(labels_batch.attention_mask, 1)) + labels = labels.filled(fill_value=-100) + + batch["labels"] = labels + batch["decoder_input_ids"] = decoder_input_ids + + return batch + + +class TrainState(train_state.TrainState): + dropout_rng: jnp.ndarray + + def replicate(self): + return jax_utils.replicate(self).replace(dropout_rng=shard_prng_key(self.dropout_rng)) + + +def write_metric(summary_writer, train_metrics, eval_metrics, train_time, step): + summary_writer.scalar("train_time", train_time, step) + + train_metrics = get_metrics(train_metrics) + for key, vals in train_metrics.items(): + tag = f"train_{key}" + for i, val in enumerate(vals): + summary_writer.scalar(tag, val, step - len(vals) + i + 1) + + for metric_name, value in eval_metrics.items(): + summary_writer.scalar(f"eval_{metric_name}", value, step) + + +def create_learning_rate_fn( + num_train_steps: int, num_warmup_steps: int, learning_rate: float +) -> Callable[[int], jnp.ndarray]: + """Returns a linear warmup, linear_decay learning rate function.""" + warmup_fn = optax.linear_schedule(init_value=0.0, end_value=learning_rate, transition_steps=num_warmup_steps) + decay_fn = optax.linear_schedule( + init_value=learning_rate, end_value=0, transition_steps=num_train_steps - num_warmup_steps + ) + schedule_fn = optax.join_schedules(schedules=[warmup_fn, decay_fn], boundaries=[num_warmup_steps]) + return schedule_fn + + +def main(): + # 1. Parse input arguments + # See all possible arguments in src/transformers/training_args.py + # or by passing the --help flag to this script. + # We now keep distinct sets of args, for a cleaner separation of concerns. + parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments)) + + if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): + # If we pass only one argument to the script and it's the path to a json file, + # let's parse it to get our arguments. + model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1])) + else: + model_args, data_args, training_args = parser.parse_args_into_dataclasses() + + # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The + # information sent is the one passed as arguments along with your JAX/Flax versions. + send_example_telemetry("run_speech_recognition_seq2seq", model_args, data_args, framework="flax") + + # 2. Setup logging + # Make one log on every process with the configuration for debugging. + logging.basicConfig( + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", + datefmt="%m/%d/%Y %H:%M:%S", + handlers=[logging.StreamHandler(sys.stdout)], + ) + # Set the verbosity to info of the Transformers logger. + # We only want one process per machine to log things on the screen. + logger.setLevel(logging.INFO if jax.process_index() == 0 else logging.ERROR) + if jax.process_index() == 0: + datasets.utils.logging.set_verbosity_warning() + transformers.utils.logging.set_verbosity_info() + else: + datasets.utils.logging.set_verbosity_error() + transformers.utils.logging.set_verbosity_error() + + logger.info("Training/evaluation parameters %s", training_args) + + # Check the output dir is valid + if ( + os.path.exists(training_args.output_dir) + and os.listdir(training_args.output_dir) + and training_args.do_train + and not training_args.overwrite_output_dir + ): + raise ValueError( + f"Output directory ({training_args.output_dir}) already exists and is not empty. " + "Use `--overwrite_output_dir` to overcome." + ) + + # Handle the repository creation + if training_args.push_to_hub: + if training_args.hub_model_id is None: + repo_name = get_full_repo_name( + Path(training_args.output_dir).absolute().name, token=training_args.hub_token + ) + else: + repo_name = training_args.hub_model_id + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id + + # 3. Load dataset + raw_datasets = DatasetDict() + + if training_args.do_train: + raw_datasets["train"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=data_args.train_split_name, + cache_dir=data_args.dataset_cache_dir, + num_proc=data_args.preprocessing_num_workers, + token=True if model_args.use_auth_token else None, + trust_remote_code=data_args.trust_remote_code, + ) + + if training_args.do_eval: + raw_datasets["eval"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=data_args.eval_split_name, + cache_dir=data_args.dataset_cache_dir, + num_proc=data_args.preprocessing_num_workers, + token=True if model_args.use_auth_token else None, + trust_remote_code=data_args.trust_remote_code, + ) + + if not training_args.do_train and not training_args.do_eval: + raise ValueError( + "Cannot not train and not do evaluation. At least one of training or evaluation has to be performed." + ) + + if data_args.audio_column_name not in next(iter(raw_datasets.values())).column_names: + raise ValueError( + f"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'. " + "Make sure to set `--audio_column_name` to the correct audio column - one of " + f"{', '.join(next(iter(raw_datasets.values())).column_names)}." + ) + + if data_args.text_column_name not in next(iter(raw_datasets.values())).column_names: + raise ValueError( + f"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. " + "Make sure to set `--text_column_name` to the correct text column - one of " + f"{', '.join(next(iter(raw_datasets.values())).column_names)}." + ) + + # 5. Load pretrained model, tokenizer, and feature extractor + config = AutoConfig.from_pretrained( + model_args.config_name if model_args.config_name else model_args.model_name_or_path, + cache_dir=model_args.cache_dir, + revision=model_args.model_revision, + token=True if model_args.use_auth_token else None, + ) + feature_extractor = AutoFeatureExtractor.from_pretrained( + model_args.feature_extractor_name if model_args.feature_extractor_name else model_args.model_name_or_path, + cache_dir=model_args.cache_dir, + revision=model_args.model_revision, + token=True if model_args.use_auth_token else None, + ) + tokenizer = AutoTokenizer.from_pretrained( + model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, + cache_dir=model_args.cache_dir, + use_fast=model_args.use_fast_tokenizer, + revision=model_args.model_revision, + token=True if model_args.use_auth_token else None, + ) + + model = FlaxAutoModelForSpeechSeq2Seq.from_pretrained( + model_args.model_name_or_path, + config=config, + dtype=getattr(jnp, model_args.dtype), + cache_dir=model_args.cache_dir, + revision=model_args.model_revision, + token=True if model_args.use_auth_token else None, + ) + + if model.config.decoder_start_token_id is None: + raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined") + + # 6. Resample speech dataset: `datasets` takes care of automatically loading and resampling the audio, + # so we just need to set the correct target sampling rate. + raw_datasets = raw_datasets.cast_column( + data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate) + ) + + # 7. Preprocessing the datasets. + # We need to read the audio files as arrays and tokenize the targets. + max_input_length = int(data_args.max_duration_in_seconds * feature_extractor.sampling_rate) + min_input_length = int(data_args.min_duration_in_seconds * feature_extractor.sampling_rate) + max_label_length = ( + data_args.max_label_length if data_args.max_label_length is not None else model.config.max_length + ) + pad_input_to_multiple_of = data_args.pad_input_to_multiple_of + pad_target_to_multiple_of = data_args.pad_target_to_multiple_of + audio_column_name = data_args.audio_column_name + num_workers = data_args.preprocessing_num_workers + text_column_name = data_args.text_column_name + model_input_name = feature_extractor.model_input_names[0] + do_lower_case = data_args.do_lower_case + + if training_args.do_train and data_args.max_train_samples is not None: + raw_datasets["train"] = raw_datasets["train"].select(range(data_args.max_train_samples)) + + if training_args.do_eval and data_args.max_eval_samples is not None: + raw_datasets["eval"] = raw_datasets["eval"].select(range(data_args.max_eval_samples)) + + if data_args.language is not None: + # We only need to set the task id when the language is specified (i.e. in a multilingual setting) + tokenizer.set_prefix_tokens(language=data_args.language, task=data_args.task) + + def prepare_dataset(batch): + # process audio + sample = batch[audio_column_name] + inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"]) + # process audio length + batch[model_input_name] = inputs.get(model_input_name)[0] + batch["input_length"] = len(sample["array"]) + + # process targets + input_str = batch[text_column_name].lower() if do_lower_case else batch[text_column_name] + batch["labels"] = tokenizer(input_str).input_ids + return batch + + vectorized_datasets = raw_datasets.map( + prepare_dataset, + remove_columns=next(iter(raw_datasets.values())).column_names, + num_proc=num_workers, + desc="preprocess train and eval dataset", + ) + + # filter training data with inputs longer than max_input_length + def is_audio_in_length_range(length): + return min_input_length < length < max_input_length + + vectorized_datasets = vectorized_datasets.filter( + is_audio_in_length_range, + num_proc=num_workers, + input_columns=["input_length"], + ) + + # for large datasets it is advised to run the preprocessing on a + # single machine first with `args.preprocessing_only` since there will mostly likely + # be a timeout when running the script in distributed mode. + # In a second step `args.preprocessing_only` can then be set to `False` to load the + # cached dataset + if data_args.preprocessing_only: + cache = {k: v.cache_files for k, v in vectorized_datasets.items()} + logger.info(f"Data preprocessing finished. Files cached at {cache}.") + return + + # 8. Load Metric + metric = evaluate.load("wer", cache_dir=model_args.cache_dir) + + def compute_metrics(preds, labels): + # replace padded labels by the padding token + for idx in range(len(labels)): + labels[idx][labels[idx] == -100] = tokenizer.pad_token_id + + pred_str = tokenizer.batch_decode(preds, skip_special_tokens=True) + # we do not want to group tokens when computing the metrics + label_str = tokenizer.batch_decode(labels, skip_special_tokens=True) + + wer = metric.compute(predictions=pred_str, references=label_str) + return {"wer": wer} + + # 9. Save feature extractor, tokenizer and config + feature_extractor.save_pretrained(training_args.output_dir) + tokenizer.save_pretrained(training_args.output_dir) + config.save_pretrained(training_args.output_dir) + + processor = AutoProcessor.from_pretrained(training_args.output_dir) + + data_collator = FlaxDataCollatorSpeechSeq2SeqWithPadding( + processor=processor, + decoder_start_token_id=model.config.decoder_start_token_id, + input_padding="longest", + target_padding="longest", + max_target_length=max_label_length, + pad_input_to_multiple_of=pad_input_to_multiple_of, + pad_target_to_multiple_of=pad_target_to_multiple_of if pad_target_to_multiple_of else max_label_length, + ) + + # Enable tensorboard only on the master node + has_tensorboard = is_tensorboard_available() + if has_tensorboard and jax.process_index() == 0: + try: + from flax.metrics.tensorboard import SummaryWriter + + summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir)) + except ImportError as ie: + has_tensorboard = False + logger.warning( + f"Unable to display metrics through TensorBoard because some package are not installed: {ie}" + ) + else: + logger.warning( + "Unable to display metrics through TensorBoard because the package is not installed: " + "Please run pip install tensorboard to enable." + ) + + # Initialize our training + rng = jax.random.PRNGKey(training_args.seed) + rng, dropout_rng = jax.random.split(rng) + + # Store some constant + num_epochs = int(training_args.num_train_epochs) + train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count() + per_device_eval_batch_size = int(training_args.per_device_eval_batch_size) + eval_batch_size = per_device_eval_batch_size * jax.device_count() + steps_per_epoch = len(vectorized_datasets["train"]) // train_batch_size + total_train_steps = steps_per_epoch * num_epochs + + # Create learning rate schedule + linear_decay_lr_schedule_fn = create_learning_rate_fn( + total_train_steps, + training_args.warmup_steps, + training_args.learning_rate, + ) + + # We use Optax's "masking" functionality to not apply weight decay + # to bias and LayerNorm scale parameters. decay_mask_fn returns a + # mask boolean with the same structure as the parameters. + # The mask is True for parameters that should be decayed. + def decay_mask_fn(params): + flat_params = traverse_util.flatten_dict(params) + # find out all LayerNorm parameters + layer_norm_candidates = ["layer_norm", "self_attn_layer_norm", "final_layer_norm", "encoder_attn_layer_norm"] + layer_norm_named_params = { + layer[-2:] + for layer_norm_name in layer_norm_candidates + for layer in flat_params.keys() + if layer_norm_name in "".join(layer).lower() + } + flat_mask = {path: (path[-1] != "bias" and path[-2:] not in layer_norm_named_params) for path in flat_params} + return traverse_util.unflatten_dict(flat_mask) + + # create adam optimizer + adamw = optax.adamw( + learning_rate=linear_decay_lr_schedule_fn, + b1=training_args.adam_beta1, + b2=training_args.adam_beta2, + eps=training_args.adam_epsilon, + weight_decay=training_args.weight_decay, + mask=decay_mask_fn, + ) + + # Setup train state + state = TrainState.create(apply_fn=model.__call__, params=model.params, tx=adamw, dropout_rng=dropout_rng) + + # label smoothed cross entropy + def loss_fn(logits, labels, label_smoothing_factor=0.0): + """ + The label smoothing implementation is adapted from Flax's official example: + https://github.com/google/flax/blob/87a211135c6a377c8f29048a1cac3840e38b9da4/examples/wmt/train.py#L104 + """ + vocab_size = logits.shape[-1] + confidence = 1.0 - label_smoothing_factor + low_confidence = (1.0 - confidence) / (vocab_size - 1) + normalizing_constant = -( + confidence * jnp.log(confidence) + (vocab_size - 1) * low_confidence * jnp.log(low_confidence + 1e-20) + ) + soft_labels = onehot(labels, vocab_size, on_value=confidence, off_value=low_confidence) + + loss = optax.softmax_cross_entropy(logits, soft_labels) + loss = loss - normalizing_constant + + # ignore padded tokens from loss, i.e. where labels are not set to -100 + padding_mask = labels >= 0 + loss = loss * padding_mask + loss = loss.sum() + num_labels = padding_mask.sum() + return loss, num_labels + + # Define gradient update step fn + def train_step(state, batch, label_smoothing_factor=0.0): + dropout_rng, new_dropout_rng = jax.random.split(state.dropout_rng) + + def compute_loss(params): + labels = batch.pop("labels") + logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0] + loss, num_labels = loss_fn(logits, labels, label_smoothing_factor) + return loss, num_labels + + grad_fn = jax.value_and_grad(compute_loss, has_aux=True) + (loss, num_labels), grad = grad_fn(state.params) + num_labels = jax.lax.psum(num_labels, "batch") + + # true loss = total loss / total samples + loss = jax.lax.psum(loss, "batch") + loss = jax.tree_util.tree_map(lambda x: x / num_labels, loss) + + # true grad = total grad / total samples + grad = jax.lax.psum(grad, "batch") + grad = jax.tree_util.tree_map(lambda x: x / num_labels, grad) + new_state = state.apply_gradients(grads=grad, dropout_rng=new_dropout_rng) + + metrics = {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)} + return new_state, metrics + + # Define eval fn + def eval_step(params, batch, label_smoothing_factor=0.0): + labels = batch.pop("labels") + logits = model(**batch, params=params, train=False)[0] + + loss, num_labels = loss_fn(logits, labels, label_smoothing_factor) + num_labels = jax.lax.psum(num_labels, "batch") + + # true loss = total loss / total samples + loss = jax.lax.psum(loss, "batch") + loss = jax.tree_util.tree_map(lambda x: x / num_labels, loss) + + metrics = {"loss": loss} + return metrics + + # Define generation function + num_beams = model_args.num_beams if model_args.num_beams is not None else model.config.num_beams + gen_kwargs = {"max_length": max_label_length, "num_beams": num_beams} + + def generate_step(params, batch): + model.params = params + output_ids = model.generate(batch[model_input_name], attention_mask=batch.get("attention_mask"), **gen_kwargs) + return output_ids.sequences + + # Create parallel version of the train and eval step + p_train_step = jax.pmap( + partial(train_step, label_smoothing_factor=training_args.label_smoothing_factor), "batch", donate_argnums=(0,) + ) + p_eval_step = jax.pmap(partial(eval_step, label_smoothing_factor=training_args.label_smoothing_factor), "batch") + p_generate_step = jax.pmap(generate_step, "batch") + + # Replicate the train state on each device + state = state.replicate() + + logger.info("***** Running training *****") + logger.info(f" Num examples = {len(vectorized_datasets['train'])}") + logger.info(f" Num Epochs = {num_epochs}") + logger.info(f" Instantaneous batch size per device = {training_args.per_device_train_batch_size}") + logger.info(f" Total train batch size (w. parallel & distributed) = {train_batch_size}") + logger.info(f" Total optimization steps = {total_train_steps}") + + train_time = 0 + epochs = tqdm(range(num_epochs), desc=f"Epoch ... (1/{num_epochs})", position=0) + for epoch in epochs: + # ======================== Training ================================ + train_start = time.time() + + train_metrics = [] + + # Generate an epoch by shuffling sampling indices from the train dataset and create a data loader + vectorized_datasets["train"] = vectorized_datasets["train"].shuffle(training_args.seed) + train_loader = DataLoader( + vectorized_datasets["train"], + batch_size=train_batch_size, + drop_last=True, + collate_fn=data_collator, + num_workers=training_args.dataloader_num_workers, + ) + # train + for batch in tqdm(train_loader, desc="Training...", position=1, leave=False): + batch = shard(batch.data) + state, train_metric = p_train_step(state, batch) + train_metrics.append(train_metric) + + train_time += time.time() - train_start + + train_metric = unreplicate(train_metric) + + epochs.write( + f"Epoch... ({epoch + 1}/{num_epochs} | Loss: {train_metric['loss']}, Learning Rate:" + f" {train_metric['learning_rate']})" + ) + + # ======================== Evaluating ============================== + eval_metrics = [] + eval_preds = [] + eval_labels = [] + + eval_loader = DataLoader( + vectorized_datasets["eval"], + batch_size=eval_batch_size, + drop_last=False, + collate_fn=data_collator, + num_workers=training_args.dataloader_num_workers, + ) + for batch in tqdm(eval_loader, desc="Evaluating...", position=2, leave=False): + # Model forward + labels = batch["labels"] + + metrics = pad_shard_unpad(p_eval_step, static_return=True)( + state.params, batch.data, min_device_batch=per_device_eval_batch_size + ) + eval_metrics.append(metrics) + + # generation + if training_args.predict_with_generate: + generated_ids = pad_shard_unpad(p_generate_step)(state.params, batch.data) + eval_preds.extend(jax.device_get(generated_ids.reshape(-1, gen_kwargs["max_length"]))) + eval_labels.extend(labels) + + # normalize eval metrics + eval_metrics = get_metrics(eval_metrics) + eval_metrics = jax.tree_util.tree_map(jnp.mean, eval_metrics) + + # compute WER metric + wer_desc = "" + if training_args.predict_with_generate: + wer_metric = compute_metrics(eval_preds, eval_labels) + eval_metrics.update(wer_metric) + wer_desc = " ".join([f"Eval {key}: {value} |" for key, value in wer_metric.items()]) + + # Print metrics and update progress bar + desc = f"Epoch... ({epoch + 1}/{num_epochs} | Eval Loss: {eval_metrics['loss']} | {wer_desc})" + epochs.write(desc) + epochs.desc = desc + + # Save metrics + if has_tensorboard and jax.process_index() == 0: + cur_step = epoch * (len(vectorized_datasets["train"]) // train_batch_size) + write_metric(summary_writer, train_metrics, eval_metrics, train_time, cur_step) + + # save checkpoint after each epoch and push checkpoint to the hub + if jax.process_index() == 0: + params = jax.device_get(jax.tree_util.tree_map(lambda x: x[0], state.params)) + model.save_pretrained(training_args.output_dir, params=params) + tokenizer.save_pretrained(training_args.output_dir) + if training_args.push_to_hub: + api.upload_folder( + commit_message=f"Saving weights and logs of epoch {epoch}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) + + +if __name__ == "__main__": + main() diff --git a/examples/flax/summarization/README.md b/examples/flax/summarization/README.md index bbe231f31a569f..c94b048ec88b42 100644 --- a/examples/flax/summarization/README.md +++ b/examples/flax/summarization/README.md @@ -9,7 +9,7 @@ way which enables simple and efficient model parallelism. `run_summarization_flax.py` is a lightweight example of how to download and preprocess a dataset from the 🤗 Datasets library or use your own files (jsonlines or csv), then fine-tune one of the architectures above on it. -For custom datasets in `jsonlines` format please see: https://huggingface.co/docs/datasets/loading_datasets.html#json-files and you also will find examples of these below. +For custom datasets in `jsonlines` format please see: https://huggingface.co/docs/datasets/loading_datasets#json-files and you also will find examples of these below. ### Train the model Next we can run the example script to train the model: diff --git a/examples/flax/summarization/run_summarization_flax.py b/examples/flax/summarization/run_summarization_flax.py index 2d7e0acbf5846c..36407df3b41d35 100644 --- a/examples/flax/summarization/run_summarization_flax.py +++ b/examples/flax/summarization/run_summarization_flax.py @@ -43,7 +43,7 @@ from flax.jax_utils import pad_shard_unpad, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm import transformers @@ -56,7 +56,7 @@ HfArgumentParser, is_tensorboard_available, ) -from transformers.utils import get_full_repo_name, is_offline_mode, send_example_telemetry +from transformers.utils import is_offline_mode, send_example_telemetry logger = logging.getLogger(__name__) @@ -158,7 +158,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -188,12 +188,22 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -251,7 +261,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the `max_length` param of `model.generate`, which is used " "during evaluation." ) @@ -295,7 +305,7 @@ class DataTrainingArguments: default=False, metadata={"help": "Whether to use generate to calculate generative metrics (ROUGE, BLEU)."} ) num_beams: Optional[int] = field( - default=None, + default=1, metadata={ "help": ( "Number of beams to use for evaluation. This argument will be passed to `model.generate`, " @@ -392,7 +402,7 @@ def write_metric(summary_writer, train_metrics, eval_metrics, train_time, step): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -428,7 +438,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -452,14 +462,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below) # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ @@ -475,7 +484,8 @@ def main(): data_args.dataset_config_name, cache_dir=model_args.cache_dir, keep_in_memory=False, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -492,10 +502,10 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -503,13 +513,15 @@ def main(): config = AutoConfig.from_pretrained( model_args.config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: config = AutoConfig.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -520,18 +532,20 @@ def main(): model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -541,13 +555,15 @@ def main(): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: model = FlaxAutoModelForSeq2SeqLM.from_config( config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), + trust_remote_code=model_args.trust_remote_code, ) if training_args.gradient_checkpointing: @@ -678,7 +694,7 @@ def preprocess_function(examples): ) # Metric - metric = evaluate.load("rouge") + metric = evaluate.load("rouge", cache_dir=model_args.cache_dir) def postprocess_text(preds, labels): preds = [pred.strip() for pred in preds] @@ -944,7 +960,13 @@ def generate_step(params, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of epoch {epoch}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of epoch {epoch}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) # ======================== Prediction loop ============================== if training_args.do_predict: diff --git a/examples/flax/test_flax_examples.py b/examples/flax/test_flax_examples.py index 2fc2dcc16adc0c..c81d6378185070 100644 --- a/examples/flax/test_flax_examples.py +++ b/examples/flax/test_flax_examples.py @@ -32,6 +32,7 @@ "summarization", "token-classification", "question-answering", + "speech-recognition", ] ] sys.path.extend(SRC_DIRS) @@ -41,6 +42,7 @@ import run_clm_flax import run_flax_glue import run_flax_ner + import run_flax_speech_recognition_seq2seq import run_mlm_flax import run_qa import run_summarization_flax @@ -76,7 +78,7 @@ def test_run_glue(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_glue.py - --model_name_or_path distilbert-base-uncased + --model_name_or_path distilbert/distilbert-base-uncased --output_dir {tmp_dir} --train_file ./tests/fixtures/tests_samples/MRPC/train.csv --validation_file ./tests/fixtures/tests_samples/MRPC/dev.csv @@ -99,7 +101,7 @@ def test_run_clm(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_clm_flax.py - --model_name_or_path distilgpt2 + --model_name_or_path distilbert/distilgpt2 --train_file ./tests/fixtures/sample_text.txt --validation_file ./tests/fixtures/sample_text.txt --do_train @@ -123,7 +125,7 @@ def test_run_summarization(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_summarization.py - --model_name_or_path t5-small + --model_name_or_path google-t5/t5-small --train_file tests/fixtures/tests_samples/xsum/sample.json --validation_file tests/fixtures/tests_samples/xsum/sample.json --test_file tests/fixtures/tests_samples/xsum/sample.json @@ -153,7 +155,7 @@ def test_run_mlm(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_mlm.py - --model_name_or_path distilroberta-base + --model_name_or_path distilbert/distilroberta-base --train_file ./tests/fixtures/sample_text.txt --validation_file ./tests/fixtures/sample_text.txt --output_dir {tmp_dir} @@ -177,7 +179,7 @@ def test_run_t5_mlm(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_t5_mlm_flax.py - --model_name_or_path t5-small + --model_name_or_path google-t5/t5-small --train_file ./tests/fixtures/sample_text.txt --validation_file ./tests/fixtures/sample_text.txt --do_train @@ -204,7 +206,7 @@ def test_run_ner(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_flax_ner.py - --model_name_or_path bert-base-uncased + --model_name_or_path google-bert/bert-base-uncased --train_file tests/fixtures/tests_samples/conll/sample.json --validation_file tests/fixtures/tests_samples/conll/sample.json --output_dir {tmp_dir} @@ -231,7 +233,7 @@ def test_run_qa(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_qa.py - --model_name_or_path bert-base-uncased + --model_name_or_path google-bert/bert-base-uncased --version_2_with_negative --train_file tests/fixtures/tests_samples/SQUAD/sample.json --validation_file tests/fixtures/tests_samples/SQUAD/sample.json @@ -252,3 +254,33 @@ def test_run_qa(self): result = get_results(tmp_dir) self.assertGreaterEqual(result["eval_f1"], 30) self.assertGreaterEqual(result["eval_exact"], 30) + + @slow + def test_run_flax_speech_recognition_seq2seq(self): + tmp_dir = self.get_auto_remove_tmp_dir() + testargs = f""" + run_flax_speech_recognition_seq2seq.py + --model_name_or_path openai/whisper-tiny.en + --dataset_name hf-internal-testing/librispeech_asr_dummy + --dataset_config clean + --train_split_name validation + --eval_split_name validation + --trust_remote_code + --output_dir {tmp_dir} + --overwrite_output_dir + --num_train_epochs=2 + --max_train_samples 10 + --max_eval_samples 10 + --warmup_steps=8 + --do_train + --do_eval + --learning_rate=2e-4 + --per_device_train_batch_size=2 + --per_device_eval_batch_size=1 + --predict_with_generate + """.split() + + with patch.object(sys, "argv", testargs): + run_flax_speech_recognition_seq2seq.main() + result = get_results(tmp_dir, split="eval") + self.assertLessEqual(result["eval_wer"], 0.05) diff --git a/examples/flax/text-classification/README.md b/examples/flax/text-classification/README.md index 8d43ab7725a241..65e50a075b78d5 100644 --- a/examples/flax/text-classification/README.md +++ b/examples/flax/text-classification/README.md @@ -31,7 +31,7 @@ GLUE is made up of a total of 9 different tasks. Here is how to run the script o export TASK_NAME=mrpc python run_flax_glue.py \ - --model_name_or_path bert-base-cased \ + --model_name_or_path google-bert/bert-base-cased \ --task_name ${TASK_NAME} \ --max_seq_length 128 \ --learning_rate 2e-5 \ diff --git a/examples/flax/text-classification/run_flax_glue.py b/examples/flax/text-classification/run_flax_glue.py index c587faf5544671..8e4e75be74b701 100755 --- a/examples/flax/text-classification/run_flax_glue.py +++ b/examples/flax/text-classification/run_flax_glue.py @@ -13,7 +13,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning a 🤗 Flax Transformers model for sequence classification on GLUE.""" +"""Finetuning a 🤗 Flax Transformers model for sequence classification on GLUE.""" + import json import logging import math @@ -21,6 +22,7 @@ import random import sys import time +import warnings from dataclasses import dataclass, field from pathlib import Path from typing import Any, Callable, Dict, Optional, Tuple @@ -36,7 +38,7 @@ from flax.jax_utils import pad_shard_unpad, replicate, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm import transformers @@ -49,12 +51,12 @@ TrainingArguments, is_tensorboard_available, ) -from transformers.utils import check_min_version, get_full_repo_name, send_example_telemetry +from transformers.utils import check_min_version, send_example_telemetry logger = logging.getLogger(__name__) # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") Array = Any Dataset = datasets.arrow_dataset.Dataset @@ -101,12 +103,28 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) use_auth_token: bool = field( + default=None, + metadata={ + "help": "The `use_auth_token` argument is deprecated and will be removed in v4.34. Please use `token` instead." + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option " + "should only be set to `True` for repositories you trust and in which you have read the code, as it will " + "execute code present on the Hub on your local machine." ) }, ) @@ -195,7 +213,7 @@ def __post_init__(self): if self.validation_file is not None: extension = self.validation_file.split(".")[-1] assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file." - self.task_name = self.task_name.lower() if type(self.task_name) == str else self.task_name + self.task_name = self.task_name.lower() if isinstance(self.task_name, str) else self.task_name def create_train_state( @@ -271,7 +289,7 @@ def cross_entropy_loss(logits, labels): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -321,6 +339,15 @@ def main(): else: model_args, data_args, training_args = parser.parse_args_into_dataclasses() + if model_args.use_auth_token is not None: + warnings.warn( + "The `use_auth_token` argument is deprecated and will be removed in v4.34. Please use `token` instead.", + FutureWarning, + ) + if model_args.token is not None: + raise ValueError("`token` and `use_auth_token` are both specified. Please set only the argument `token`.") + model_args.token = model_args.use_auth_token + # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The # information sent is the one passed as arguments along with your Python/PyTorch versions. send_example_telemetry("run_glue", model_args, data_args, framework="flax") @@ -342,14 +369,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below) # or specify a GLUE benchmark task (the dataset will be downloaded automatically from the datasets Hub). @@ -368,7 +394,7 @@ def main(): raw_datasets = load_dataset( "glue", data_args.task_name, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: # Loading the dataset from local csv or json file. @@ -381,10 +407,10 @@ def main(): raw_datasets = load_dataset( extension, data_files=data_files, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Labels if data_args.task_name is not None: @@ -401,7 +427,7 @@ def main(): num_labels = 1 else: # A useful fast method: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.unique + # https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.unique label_list = raw_datasets["train"].unique("label") label_list.sort() # Let's sort it for determinism num_labels = len(label_list) @@ -411,17 +437,20 @@ def main(): model_args.model_name_or_path, num_labels=num_labels, finetuning_task=data_args.task_name, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path, use_fast=not model_args.use_slow_tokenizer, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) model = FlaxAutoModelForSequenceClassification.from_pretrained( model_args.model_name_or_path, config=config, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # Preprocessing the datasets @@ -455,7 +484,7 @@ def main(): label_to_id = {i: label_name_to_id[label_list[i]] for i in range(num_labels)} else: logger.warning( - "Your model seems to have been trained with labels, but they don't match the dataset: ", + "Your model seems to have been trained with labels, but they don't match the dataset: " f"model labels: {sorted(label_name_to_id.keys())}, dataset labels: {sorted(label_list)}." "\nIgnoring the model labels as a result.", ) @@ -570,9 +599,9 @@ def eval_step(state, batch): p_eval_step = jax.pmap(eval_step, axis_name="batch") if data_args.task_name is not None: - metric = evaluate.load("glue", data_args.task_name) + metric = evaluate.load("glue", data_args.task_name, cache_dir=model_args.cache_dir) else: - metric = evaluate.load("accuracy") + metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir) logger.info(f"===== Starting training ({num_epochs} epochs) =====") train_time = 0 @@ -648,7 +677,13 @@ def eval_step(state, batch): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of epoch {epoch}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) epochs.desc = f"Epoch ... {epoch + 1}/{num_epochs}" # save the eval metrics in json diff --git a/examples/flax/token-classification/README.md b/examples/flax/token-classification/README.md index 915cf6ae20ff93..1f8175072148bb 100644 --- a/examples/flax/token-classification/README.md +++ b/examples/flax/token-classification/README.md @@ -25,7 +25,7 @@ The following example fine-tunes BERT on CoNLL-2003: ```bash python run_flax_ner.py \ - --model_name_or_path bert-base-cased \ + --model_name_or_path google-bert/bert-base-cased \ --dataset_name conll2003 \ --max_seq_length 128 \ --learning_rate 2e-5 \ diff --git a/examples/flax/token-classification/run_flax_ner.py b/examples/flax/token-classification/run_flax_ner.py index a58f79d37a7f2b..5e83513d173b8e 100644 --- a/examples/flax/token-classification/run_flax_ner.py +++ b/examples/flax/token-classification/run_flax_ner.py @@ -13,7 +13,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-tuning a 🤗 Flax Transformers model on token classification tasks (NER, POS, CHUNKS)""" +"""Fine-tuning a 🤗 Flax Transformers model on token classification tasks (NER, POS, CHUNKS)""" + import json import logging import math @@ -21,6 +22,7 @@ import random import sys import time +import warnings from dataclasses import asdict, dataclass, field from enum import Enum from itertools import chain @@ -38,7 +40,7 @@ from flax.jax_utils import pad_shard_unpad, replicate, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm import transformers @@ -49,13 +51,13 @@ HfArgumentParser, is_tensorboard_available, ) -from transformers.utils import check_min_version, get_full_repo_name, send_example_telemetry +from transformers.utils import check_min_version, send_example_telemetry from transformers.utils.versions import require_version logger = logging.getLogger(__name__) # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") require_version("datasets>=1.8.0", "To fix: pip install -r examples/pytorch/token-classification/requirements.txt") @@ -149,12 +151,28 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) use_auth_token: bool = field( + default=None, + metadata={ + "help": "The `use_auth_token` argument is deprecated and will be removed in v4.34. Please use `token` instead." + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -323,7 +341,7 @@ def cross_entropy_loss(logits, labels): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -377,6 +395,15 @@ def main(): else: model_args, data_args, training_args = parser.parse_args_into_dataclasses() + if model_args.use_auth_token is not None: + warnings.warn( + "The `use_auth_token` argument is deprecated and will be removed in v4.34. Please use `token` instead.", + FutureWarning, + ) + if model_args.token is not None: + raise ValueError("`token` and `use_auth_token` are both specified. Please set only the argument `token`.") + model_args.token = model_args.use_auth_token + # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The # information sent is the one passed as arguments along with your Python/PyTorch versions. send_example_telemetry("run_ner", model_args, data_args, framework="flax") @@ -398,14 +425,13 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) # or just provide the name of one of the public datasets for token classification task available on the hub at https://huggingface.co/datasets/ @@ -422,7 +448,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: # Loading the dataset from local csv or json file. @@ -436,10 +463,10 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. if raw_datasets["train"] is not None: column_names = raw_datasets["train"].column_names @@ -490,7 +517,8 @@ def get_label_list(labels): finetuning_task=data_args.task_name, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer_name_or_path = model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path if config.model_type in {"gpt2", "roberta"}: @@ -498,7 +526,8 @@ def get_label_list(labels): tokenizer_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, add_prefix_space=True, ) else: @@ -506,14 +535,16 @@ def get_label_list(labels): tokenizer_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) model = FlaxAutoModelForTokenClassification.from_pretrained( model_args.model_name_or_path, config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # Preprocessing the datasets @@ -646,7 +677,7 @@ def eval_step(state, batch): p_eval_step = jax.pmap(eval_step, axis_name="batch") - metric = evaluate.load("seqeval") + metric = evaluate.load("seqeval", cache_dir=model_args.cache_dir) def get_labels(y_pred, y_true): # Transform predictions and references tensos to numpy arrays @@ -768,7 +799,13 @@ def compute_metrics(): model.save_pretrained(training_args.output_dir, params=params) tokenizer.save_pretrained(training_args.output_dir) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of step {cur_step}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of step {cur_step}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) epochs.desc = f"Epoch ... {epoch + 1}/{num_epochs}" # Eval after training diff --git a/examples/flax/vision/requirements.txt b/examples/flax/vision/requirements.txt index cf1859d7549477..f335c1f1c63096 100644 --- a/examples/flax/vision/requirements.txt +++ b/examples/flax/vision/requirements.txt @@ -3,6 +3,6 @@ jaxlib>=0.1.59 flax>=0.3.5 optax>=0.0.8 -f https://download.pytorch.org/whl/torch_stable.html -torch==1.9.0+cpu +torch==1.13.1 -f https://download.pytorch.org/whl/torch_stable.html -torchvision==0.10.0+cpu \ No newline at end of file +torchvision==0.12.0+cpu diff --git a/examples/flax/vision/run_image_classification.py b/examples/flax/vision/run_image_classification.py index 6a88f0f8d67b28..0228a8797b6edd 100644 --- a/examples/flax/vision/run_image_classification.py +++ b/examples/flax/vision/run_image_classification.py @@ -41,7 +41,7 @@ from flax.jax_utils import pad_shard_unpad, unreplicate from flax.training import train_state from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key -from huggingface_hub import Repository, create_repo +from huggingface_hub import HfApi from tqdm import tqdm import transformers @@ -54,7 +54,7 @@ is_tensorboard_available, set_seed, ) -from transformers.utils import get_full_repo_name, send_example_telemetry +from transformers.utils import send_example_telemetry logger = logging.getLogger(__name__) @@ -136,7 +136,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -159,12 +159,22 @@ class ModelArguments: ) }, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option " + "should only be set to `True` for repositories you trust and in which you have read the code, as it will " + "execute code present on the Hub on your local machine." ) }, ) @@ -232,7 +242,7 @@ def write_metric(summary_writer, train_metrics, eval_metrics, train_time, step): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -268,7 +278,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -293,18 +303,17 @@ def main(): # Handle the repository creation if training_args.push_to_hub: - if training_args.hub_model_id is None: - repo_name = get_full_repo_name( - Path(training_args.output_dir).absolute().name, token=training_args.hub_token - ) - else: - repo_name = training_args.hub_model_id - create_repo(repo_name, exist_ok=True, token=training_args.hub_token) - repo = Repository(training_args.output_dir, clone_from=repo_name, token=training_args.hub_token) + # Retrieve of infer repo_name + repo_name = training_args.hub_model_id + if repo_name is None: + repo_name = Path(training_args.output_dir).absolute().name + # Create repo and retrieve repo_id + api = HfApi() + repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id # Initialize datasets and pre-processing transforms # We use torchvision here for faster pre-processing - # Note that here we are using some default pre-processing, for maximum accuray + # Note that here we are using some default pre-processing, for maximum accuracy # one should tune this part and carefully select what transformations to use. normalize = transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]) train_dataset = torchvision.datasets.ImageFolder( @@ -338,7 +347,8 @@ def main(): num_labels=len(train_dataset.classes), image_size=data_args.image_size, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: config = AutoConfig.from_pretrained( @@ -346,7 +356,8 @@ def main(): num_labels=len(train_dataset.classes), image_size=data_args.image_size, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: config = CONFIG_MAPPING[model_args.model_type]() @@ -358,13 +369,15 @@ def main(): config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: model = FlaxAutoModelForImageClassification.from_config( config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype), + trust_remote_code=model_args.trust_remote_code, ) # Store some constant @@ -565,7 +578,13 @@ def eval_step(params, batch): params = jax.device_get(jax.tree_util.tree_map(lambda x: x[0], state.params)) model.save_pretrained(training_args.output_dir, params=params) if training_args.push_to_hub: - repo.push_to_hub(commit_message=f"Saving weights and logs of epoch {epoch}", blocking=False) + api.upload_folder( + commit_message=f"Saving weights and logs of epoch {epoch}", + folder_path=training_args.output_dir, + repo_id=repo_id, + repo_type="model", + token=training_args.hub_token, + ) if __name__ == "__main__": diff --git a/examples/legacy/benchmarking/README.md b/examples/legacy/benchmarking/README.md new file mode 100644 index 00000000000000..03e174770d1077 --- /dev/null +++ b/examples/legacy/benchmarking/README.md @@ -0,0 +1,26 @@ + + +# 🤗 Benchmark results + +Here, you can find a list of the different benchmark results created by the community. + +If you would like to list benchmark results on your favorite models of the [model hub](https://huggingface.co/models) here, please open a Pull Request and add it below. + +| Benchmark description | Results | Environment info | Author | +|:----------|:-------------|:-------------|------:| +| PyTorch Benchmark on inference for `google-bert/bert-base-cased` |[memory](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_memory.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | +| PyTorch Benchmark on inference for `google-bert/bert-base-cased` |[time](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_time.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | diff --git a/examples/legacy/benchmarking/plot_csv_file.py b/examples/legacy/benchmarking/plot_csv_file.py new file mode 100644 index 00000000000000..aa092f5c047d44 --- /dev/null +++ b/examples/legacy/benchmarking/plot_csv_file.py @@ -0,0 +1,178 @@ +# Copyright 2020 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import csv +from collections import defaultdict +from dataclasses import dataclass, field +from typing import List, Optional + +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.ticker import ScalarFormatter + +from transformers import HfArgumentParser + + +def list_field(default=None, metadata=None): + return field(default_factory=lambda: default, metadata=metadata) + + +@dataclass +class PlotArguments: + """ + Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. + """ + + csv_file: str = field( + metadata={"help": "The csv file to plot."}, + ) + plot_along_batch: bool = field( + default=False, + metadata={"help": "Whether to plot along batch size or sequence length. Defaults to sequence length."}, + ) + is_time: bool = field( + default=False, + metadata={"help": "Whether the csv file has time results or memory results. Defaults to memory results."}, + ) + no_log_scale: bool = field( + default=False, + metadata={"help": "Disable logarithmic scale when plotting"}, + ) + is_train: bool = field( + default=False, + metadata={ + "help": "Whether the csv file has training results or inference results. Defaults to inference results." + }, + ) + figure_png_file: Optional[str] = field( + default=None, + metadata={"help": "Filename under which the plot will be saved. If unused no plot is saved."}, + ) + short_model_names: Optional[List[str]] = list_field( + default=None, metadata={"help": "List of model names that are used instead of the ones in the csv file."} + ) + + +def can_convert_to_int(string): + try: + int(string) + return True + except ValueError: + return False + + +def can_convert_to_float(string): + try: + float(string) + return True + except ValueError: + return False + + +class Plot: + def __init__(self, args): + self.args = args + self.result_dict = defaultdict(lambda: {"bsz": [], "seq_len": [], "result": {}}) + + with open(self.args.csv_file, newline="") as csv_file: + reader = csv.DictReader(csv_file) + for row in reader: + model_name = row["model"] + self.result_dict[model_name]["bsz"].append(int(row["batch_size"])) + self.result_dict[model_name]["seq_len"].append(int(row["sequence_length"])) + if can_convert_to_int(row["result"]): + # value is not None + self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = ( + int(row["result"]) + ) + elif can_convert_to_float(row["result"]): + # value is not None + self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = ( + float(row["result"]) + ) + + def plot(self): + fig, ax = plt.subplots() + title_str = "Time usage" if self.args.is_time else "Memory usage" + title_str = title_str + " for training" if self.args.is_train else title_str + " for inference" + + if not self.args.no_log_scale: + # set logarithm scales + ax.set_xscale("log") + ax.set_yscale("log") + + for axis in [ax.xaxis, ax.yaxis]: + axis.set_major_formatter(ScalarFormatter()) + + for model_name_idx, model_name in enumerate(self.result_dict.keys()): + batch_sizes = sorted(set(self.result_dict[model_name]["bsz"])) + sequence_lengths = sorted(set(self.result_dict[model_name]["seq_len"])) + results = self.result_dict[model_name]["result"] + + (x_axis_array, inner_loop_array) = ( + (batch_sizes, sequence_lengths) if self.args.plot_along_batch else (sequence_lengths, batch_sizes) + ) + + label_model_name = ( + model_name if self.args.short_model_names is None else self.args.short_model_names[model_name_idx] + ) + + for inner_loop_value in inner_loop_array: + if self.args.plot_along_batch: + y_axis_array = np.asarray( + [results[(x, inner_loop_value)] for x in x_axis_array if (x, inner_loop_value) in results], + dtype=int, + ) + else: + y_axis_array = np.asarray( + [results[(inner_loop_value, x)] for x in x_axis_array if (inner_loop_value, x) in results], + dtype=np.float32, + ) + + (x_axis_label, inner_loop_label) = ( + ("batch_size", "len") if self.args.plot_along_batch else ("in #tokens", "bsz") + ) + + x_axis_array = np.asarray(x_axis_array, int)[: len(y_axis_array)] + plt.scatter( + x_axis_array, y_axis_array, label=f"{label_model_name} - {inner_loop_label}: {inner_loop_value}" + ) + plt.plot(x_axis_array, y_axis_array, "--") + + title_str += f" {label_model_name} vs." + + title_str = title_str[:-4] + y_axis_label = "Time in s" if self.args.is_time else "Memory in MB" + + # plot + plt.title(title_str) + plt.xlabel(x_axis_label) + plt.ylabel(y_axis_label) + plt.legend() + + if self.args.figure_png_file is not None: + plt.savefig(self.args.figure_png_file) + else: + plt.show() + + +def main(): + parser = HfArgumentParser(PlotArguments) + plot_args = parser.parse_args_into_dataclasses()[0] + plot = Plot(args=plot_args) + plot.plot() + + +if __name__ == "__main__": + main() diff --git a/examples/legacy/benchmarking/requirements.txt b/examples/legacy/benchmarking/requirements.txt new file mode 100644 index 00000000000000..68c56b321909d9 --- /dev/null +++ b/examples/legacy/benchmarking/requirements.txt @@ -0,0 +1 @@ +torch >= 1.3 \ No newline at end of file diff --git a/examples/legacy/benchmarking/run_benchmark.py b/examples/legacy/benchmarking/run_benchmark.py new file mode 100644 index 00000000000000..85f266566bfa4e --- /dev/null +++ b/examples/legacy/benchmarking/run_benchmark.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2020 The HuggingFace Inc. team. +# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Benchmarking the library on inference and training""" + +from transformers import HfArgumentParser, PyTorchBenchmark, PyTorchBenchmarkArguments + + +def main(): + parser = HfArgumentParser(PyTorchBenchmarkArguments) + try: + benchmark_args = parser.parse_args_into_dataclasses()[0] + except ValueError as e: + arg_error_msg = "Arg --no_{0} is no longer used, please use --no-{0} instead." + begin_error_msg = " ".join(str(e).split(" ")[:-1]) + full_error_msg = "" + depreciated_args = eval(str(e).split(" ")[-1]) + wrong_args = [] + for arg in depreciated_args: + # arg[2:] removes '--' + if arg[2:] in PyTorchBenchmarkArguments.deprecated_args: + # arg[5:] removes '--no_' + full_error_msg += arg_error_msg.format(arg[5:]) + else: + wrong_args.append(arg) + if len(wrong_args) > 0: + full_error_msg = full_error_msg + begin_error_msg + str(wrong_args) + raise ValueError(full_error_msg) + + benchmark = PyTorchBenchmark(args=benchmark_args) + benchmark.run() + + +if __name__ == "__main__": + main() diff --git a/examples/legacy/multiple_choice/run_multiple_choice.py b/examples/legacy/multiple_choice/run_multiple_choice.py index 451397042594f7..fece480cad0d4c 100644 --- a/examples/legacy/multiple_choice/run_multiple_choice.py +++ b/examples/legacy/multiple_choice/run_multiple_choice.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for multiple choice (Bert, Roberta, XLNet).""" - +"""Finetuning the library models for multiple choice (Bert, Roberta, XLNet).""" import logging import os diff --git a/examples/legacy/multiple_choice/utils_multiple_choice.py b/examples/legacy/multiple_choice/utils_multiple_choice.py index 9ffaa7971b5624..6b7559c49e530d 100644 --- a/examples/legacy/multiple_choice/utils_multiple_choice.py +++ b/examples/legacy/multiple_choice/utils_multiple_choice.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Multiple choice fine-tuning: utilities to work with multiple choice tasks of reading comprehension """ - +"""Multiple choice fine-tuning: utilities to work with multiple choice tasks of reading comprehension""" import csv import glob @@ -379,7 +378,7 @@ def get_test_examples(self, data_dir): """See base class.""" logger.info("LOOKING AT {} dev".format(data_dir)) raise ValueError( - "For swag testing, the input file does not contain a label column. It can not be tested in current code" + "For swag testing, the input file does not contain a label column. It can not be tested in current code " "setting!" ) return self._create_examples(self._read_csv(os.path.join(data_dir, "test.csv")), "test") @@ -541,7 +540,7 @@ def convert_examples_to_features( if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0: logger.info( "Attention! you are cropping tokens (swag task is ok). " - "If you are training ARC and RACE and you are poping question + options," + "If you are training ARC and RACE and you are poping question + options, " "you need to try to use a bigger max seq length!" ) diff --git a/examples/legacy/pytorch-lightning/lightning_base.py b/examples/legacy/pytorch-lightning/lightning_base.py index f246ecab0dd01b..640828bacd3401 100644 --- a/examples/legacy/pytorch-lightning/lightning_base.py +++ b/examples/legacy/pytorch-lightning/lightning_base.py @@ -313,7 +313,7 @@ def add_generic_args(parser, root_dir) -> None: type=str, default="O2", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/legacy/pytorch-lightning/requirements.txt b/examples/legacy/pytorch-lightning/requirements.txt index b3ed7cbc82ceb1..a6f2d6dce5a9d5 100644 --- a/examples/legacy/pytorch-lightning/requirements.txt +++ b/examples/legacy/pytorch-lightning/requirements.txt @@ -14,7 +14,7 @@ nltk pandas datasets >= 1.1.3 fire -pytest +pytest<8.0.1 conllu sentencepiece != 0.1.92 protobuf diff --git a/examples/legacy/pytorch-lightning/run_glue.py b/examples/legacy/pytorch-lightning/run_glue.py index 5f22e2fc7a1311..681f633fcd6d2b 100644 --- a/examples/legacy/pytorch-lightning/run_glue.py +++ b/examples/legacy/pytorch-lightning/run_glue.py @@ -23,7 +23,7 @@ class GLUETransformer(BaseTransformer): mode = "sequence-classification" def __init__(self, hparams): - if type(hparams) == dict: + if isinstance(hparams, dict): hparams = Namespace(**hparams) hparams.glue_output_mode = glue_output_modes[hparams.task] num_labels = glue_tasks_num_labels[hparams.task] diff --git a/examples/legacy/pytorch-lightning/run_ner.py b/examples/legacy/pytorch-lightning/run_ner.py index 7f6b00854d99cc..fc6f812275ea2c 100644 --- a/examples/legacy/pytorch-lightning/run_ner.py +++ b/examples/legacy/pytorch-lightning/run_ner.py @@ -25,7 +25,7 @@ class NERTransformer(BaseTransformer): mode = "token-classification" def __init__(self, hparams): - if type(hparams) == dict: + if isinstance(hparams, dict): hparams = Namespace(**hparams) module = import_module("tasks") try: diff --git a/examples/legacy/question-answering/README.md b/examples/legacy/question-answering/README.md index 494ae4ffd7eebf..339837c94f5d86 100644 --- a/examples/legacy/question-answering/README.md +++ b/examples/legacy/question-answering/README.md @@ -1,7 +1,7 @@ #### Fine-tuning BERT on SQuAD1.0 with relative position embeddings The following examples show how to fine-tune BERT models with different relative position embeddings. The BERT model -`bert-base-uncased` was pretrained with default absolute position embeddings. We provide the following pretrained +`google-bert/bert-base-uncased` was pretrained with default absolute position embeddings. We provide the following pretrained models which were pre-trained on the same training data (BooksCorpus and English Wikipedia) as in the BERT model training, but with different relative position embeddings. @@ -10,7 +10,7 @@ Shaw et al., [Self-Attention with Relative Position Representations](https://arx * `zhiheng-huang/bert-base-uncased-embedding-relative-key-query`, trained from scratch with relative embedding method 4 in Huang et al. [Improve Transformer Models with Better Relative Position Embeddings](https://arxiv.org/abs/2009.13658) * `zhiheng-huang/bert-large-uncased-whole-word-masking-embedding-relative-key-query`, fine-tuned from model -`bert-large-uncased-whole-word-masking` with 3 additional epochs with relative embedding method 4 in Huang et al. +`google-bert/bert-large-uncased-whole-word-masking` with 3 additional epochs with relative embedding method 4 in Huang et al. [Improve Transformer Models with Better Relative Position Embeddings](https://arxiv.org/abs/2009.13658) @@ -18,7 +18,7 @@ in Huang et al. [Improve Transformer Models with Better Relative Position Embedd ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 -python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \ +torchrun --nproc_per_node=8 ./examples/question-answering/run_squad.py \ --model_name_or_path zhiheng-huang/bert-base-uncased-embedding-relative-key-query \ --dataset_name squad \ --do_train \ @@ -46,7 +46,7 @@ gpu training leads to the f1 score of 90.71. ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 -python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \ +torchrun --nproc_per_node=8 ./examples/question-answering/run_squad.py \ --model_name_or_path zhiheng-huang/bert-large-uncased-whole-word-masking-embedding-relative-key-query \ --dataset_name squad \ --do_train \ @@ -61,15 +61,15 @@ python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answer --gradient_accumulation_steps 3 ``` Training with the above command leads to the f1 score of 93.52, which is slightly better than the f1 score of 93.15 for -`bert-large-uncased-whole-word-masking`. +`google-bert/bert-large-uncased-whole-word-masking`. #### Distributed training Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.1: ```bash -python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \ - --model_name_or_path bert-large-uncased-whole-word-masking \ +torchrun --nproc_per_node=8 ./examples/question-answering/run_squad.py \ + --model_name_or_path google-bert/bert-large-uncased-whole-word-masking \ --dataset_name squad \ --do_train \ --do_eval \ @@ -90,7 +90,7 @@ exact_match = 86.91 ``` This fine-tuned model is available as a checkpoint under the reference -[`bert-large-uncased-whole-word-masking-finetuned-squad`](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad). +[`google-bert/bert-large-uncased-whole-word-masking-finetuned-squad`](https://huggingface.co/google-bert/bert-large-uncased-whole-word-masking-finetuned-squad). ## Results diff --git a/examples/legacy/question-answering/run_squad.py b/examples/legacy/question-answering/run_squad.py index fc9411e95d220a..f5a827c15acac1 100644 --- a/examples/legacy/question-answering/run_squad.py +++ b/examples/legacy/question-answering/run_squad.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet).""" - +"""Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet).""" import argparse import glob @@ -148,7 +147,7 @@ def train(args, train_dataset, model, tokenizer): # Check if continuing training from a checkpoint if os.path.exists(args.model_name_or_path): try: - # set global_step to gobal_step of last saved checkpoint from model path + # set global_step to global_step of last saved checkpoint from model path checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0] global_step = int(checkpoint_suffix) epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps) @@ -166,7 +165,7 @@ def train(args, train_dataset, model, tokenizer): train_iterator = trange( epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0] ) - # Added here for reproductibility + # Added here for reproducibility set_seed(args) for _ in train_iterator: @@ -663,7 +662,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -705,7 +704,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/legacy/question-answering/run_squad_trainer.py b/examples/legacy/question-answering/run_squad_trainer.py index 7e3a6f28e0ba1e..a27cb76295ca94 100644 --- a/examples/legacy/question-answering/run_squad_trainer.py +++ b/examples/legacy/question-answering/run_squad_trainer.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-tuning the library models for question-answering.""" - +"""Fine-tuning the library models for question-answering.""" import logging import os diff --git a/examples/legacy/run_camembert.py b/examples/legacy/run_camembert.py index 9651570b39e1e8..67e04babe1043e 100755 --- a/examples/legacy/run_camembert.py +++ b/examples/legacy/run_camembert.py @@ -39,8 +39,8 @@ def fill_mask(masked_input, model, tokenizer, topk=5): return topk_filled_outputs -tokenizer = CamembertTokenizer.from_pretrained("camembert-base") -model = CamembertForMaskedLM.from_pretrained("camembert-base") +tokenizer = CamembertTokenizer.from_pretrained("almanach/camembert-base") +model = CamembertForMaskedLM.from_pretrained("almanach/camembert-base") model.eval() masked_input = "Le camembert est :)" diff --git a/examples/legacy/run_language_modeling.py b/examples/legacy/run_language_modeling.py index 59490f710e1338..317b2dfb48b562 100755 --- a/examples/legacy/run_language_modeling.py +++ b/examples/legacy/run_language_modeling.py @@ -20,7 +20,6 @@ using a masked language modeling (MLM) loss. XLNet is fine-tuned using a permutation language modeling (PLM) loss. """ - import logging import math import os @@ -149,7 +148,7 @@ class DataTrainingArguments: default=-1, metadata={ "help": ( - "Optional input sequence length after tokenization." + "Optional input sequence length after tokenization. " "The training dataset will be truncated in block of this size for training." "Default to the model max input length for single sentence inputs (take into account special tokens)." ) @@ -283,7 +282,7 @@ def main(): if config.model_type in ["bert", "roberta", "distilbert", "camembert"] and not data_args.mlm: raise ValueError( - "BERT and RoBERTa-like models do not have LM heads but masked LM heads. They must be run using the" + "BERT and RoBERTa-like models do not have LM heads but masked LM heads. They must be run using the " "--mlm flag (masked language modeling)." ) diff --git a/examples/legacy/run_openai_gpt.py b/examples/legacy/run_openai_gpt.py index 03031f205768ff..3831c1bd4401d6 100755 --- a/examples/legacy/run_openai_gpt.py +++ b/examples/legacy/run_openai_gpt.py @@ -20,7 +20,7 @@ This script with default values fine-tunes and evaluate a pretrained OpenAI GPT on the RocStories dataset: python run_openai_gpt.py \ - --model_name openai-gpt \ + --model_name openai-community/openai-gpt \ --do_train \ --do_eval \ --train_dataset "$ROC_STORIES_DIR/cloze_test_val__spring2016 - cloze_test_ALL_val.csv" \ @@ -28,6 +28,7 @@ --output_dir ../log \ --train_batch_size 16 \ """ + import argparse import csv import logging @@ -104,7 +105,7 @@ def pre_process_datasets(encoded_datasets, input_len, cap_length, start_token, d def main(): parser = argparse.ArgumentParser() - parser.add_argument("--model_name", type=str, default="openai-gpt", help="pretrained model name") + parser.add_argument("--model_name", type=str, default="openai-community/openai-gpt", help="pretrained model name") parser.add_argument("--do_train", action="store_true", help="Whether to run training.") parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.") parser.add_argument( diff --git a/examples/legacy/run_swag.py b/examples/legacy/run_swag.py index bde05016875265..dbf712a71ff200 100755 --- a/examples/legacy/run_swag.py +++ b/examples/legacy/run_swag.py @@ -15,10 +15,9 @@ # See the License for the specific language governing permissions and # limitations under the License. """BERT finetuning runner. - Finetuning the library models for multiple choice on SWAG (Bert). +Finetuning the library models for multiple choice on SWAG (Bert). """ - import argparse import csv import glob @@ -338,7 +337,7 @@ def train(args, train_dataset, model, tokenizer): tr_loss, logging_loss = 0.0, 0.0 model.zero_grad() train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]) - set_seed(args) # Added here for reproductibility + set_seed(args) # Added here for reproducibility for _ in train_iterator: epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) for step, batch in enumerate(epoch_iterator): @@ -538,7 +537,7 @@ def main(): default=1, help="Number of updates steps to accumulate before performing a backward/update pass.", ) - parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.") + parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.") parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.") parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.") parser.add_argument( @@ -579,7 +578,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -612,7 +611,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/legacy/run_transfo_xl.py b/examples/legacy/run_transfo_xl.py index 7ee941150852e1..ce24fe13d79231 100755 --- a/examples/legacy/run_transfo_xl.py +++ b/examples/legacy/run_transfo_xl.py @@ -14,14 +14,13 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" PyTorch Transformer XL model evaluation script. - Adapted from https://github.com/kimiyoung/transformer-xl. - In particular https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/eval.py +"""PyTorch Transformer XL model evaluation script. +Adapted from https://github.com/kimiyoung/transformer-xl. +In particular https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/eval.py - This script with default values evaluates a pretrained Transformer-XL on WikiText 103 +This script with default values evaluates a pretrained Transformer-XL on WikiText 103 """ - import argparse import logging import math @@ -40,7 +39,7 @@ def main(): parser = argparse.ArgumentParser(description="PyTorch Transformer Language Model") - parser.add_argument("--model_name", type=str, default="transfo-xl-wt103", help="pretrained model name") + parser.add_argument("--model_name", type=str, default="transfo-xl/transfo-xl-wt103", help="pretrained model name") parser.add_argument( "--split", type=str, default="test", choices=["all", "valid", "test"], help="which split to evaluate" ) diff --git a/examples/legacy/seq2seq/README.md b/examples/legacy/seq2seq/README.md index 5a3c2dbd3506be..f574ccabda2c4a 100644 --- a/examples/legacy/seq2seq/README.md +++ b/examples/legacy/seq2seq/README.md @@ -140,7 +140,7 @@ python finetune_trainer.py --help For multi-gpu training use `torch.distributed.launch`, e.g. with 2 gpus: ```bash -python -m torch.distributed.launch --nproc_per_node=2 finetune_trainer.py ... +torchrun --nproc_per_node=2 finetune_trainer.py ... ``` **At the moment, `Seq2SeqTrainer` does not support *with teacher* distillation.** @@ -170,7 +170,7 @@ If 'translation' is in your task name, the computed metric will be BLEU. Otherwi For t5, you need to specify --task translation_{src}_to_{tgt} as follows: ```bash export DATA_DIR=wmt_en_ro -./run_eval.py t5-base \ +./run_eval.py google-t5/t5-base \ $DATA_DIR/val.source t5_val_generations.txt \ --reference_path $DATA_DIR/val.target \ --score_path enro_bleu.json \ @@ -214,7 +214,7 @@ because it uses SortishSampler to minimize padding. You can also use it on 1 GPU `{type_path}.source` and `{type_path}.target`. Run `./run_distributed_eval.py --help` for all clargs. ```bash -python -m torch.distributed.launch --nproc_per_node=8 run_distributed_eval.py \ +torchrun --nproc_per_node=8 run_distributed_eval.py \ --model_name sshleifer/distilbart-large-xsum-12-3 \ --save_dir xsum_generations \ --data_dir xsum \ @@ -228,7 +228,7 @@ Contributions that implement this command for other distributed hardware setups When using `run_eval.py`, the following features can be useful: * if you running the script multiple times and want to make it easier to track what arguments produced that output, use `--dump-args`. Along with the results it will also dump any custom params that were passed to the script. For example if you used: `--num_beams 8 --early_stopping true`, the output will be: - ``` + ```json {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True} ``` @@ -236,13 +236,13 @@ When using `run_eval.py`, the following features can be useful: If using `--dump-args --info`, the output will be: - ``` + ```json {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': '2020-09-13 18:44:43'} ``` If using `--dump-args --info "pair:en-ru chkpt=best`, the output will be: - ``` + ```json {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': 'pair=en-ru chkpt=best'} ``` @@ -321,7 +321,7 @@ For example, ./save_len_file.py Helsinki-NLP/opus-mt-en-ro wmt_en_ro ./dynamic_bs_example.sh --max_tokens_per_batch=2000 --output_dir benchmark_dynamic_bs ``` -splits `wmt_en_ro/train` into 11,197 uneven lengthed batches and can finish 1 epoch in 8 minutes on a v100. +splits `wmt_en_ro/train` into 11,197 uneven length batches and can finish 1 epoch in 8 minutes on a v100. For comparison, ```bash diff --git a/examples/legacy/seq2seq/finetune.sh b/examples/legacy/seq2seq/finetune.sh index 1f518835d63859..60023df7bad6ae 100644 --- a/examples/legacy/seq2seq/finetune.sh +++ b/examples/legacy/seq2seq/finetune.sh @@ -18,7 +18,7 @@ python finetune_trainer.py \ --learning_rate=3e-5 \ --fp16 \ --do_train --do_eval --do_predict \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate \ --n_val 1000 \ "$@" diff --git a/examples/legacy/seq2seq/finetune_tpu.sh b/examples/legacy/seq2seq/finetune_tpu.sh index 68cf0d77360292..ef72b0953b440b 100644 --- a/examples/legacy/seq2seq/finetune_tpu.sh +++ b/examples/legacy/seq2seq/finetune_tpu.sh @@ -20,7 +20,7 @@ python xla_spawn.py --num_cores $TPU_NUM_CORES \ finetune_trainer.py \ --learning_rate=3e-5 \ --do_train --do_eval \ - --evaluation_strategy steps \ + --eval_strategy steps \ --prediction_loss_only \ --n_val 1000 \ "$@" diff --git a/examples/legacy/seq2seq/finetune_trainer.py b/examples/legacy/seq2seq/finetune_trainer.py index 4e186c96d8c218..e269bc2474eca5 100755 --- a/examples/legacy/seq2seq/finetune_trainer.py +++ b/examples/legacy/seq2seq/finetune_trainer.py @@ -271,7 +271,7 @@ def main(): max_source_length=data_args.max_source_length, prefix=model.config.prefix or "", ) - if training_args.do_eval or training_args.evaluation_strategy != EvaluationStrategy.NO + if training_args.do_eval or training_args.eval_strategy != EvaluationStrategy.NO else None ) test_dataset = ( diff --git a/examples/legacy/seq2seq/old_test_datasets.py b/examples/legacy/seq2seq/old_test_datasets.py index 0b907b1ed9fbb6..be108f7645f8a9 100644 --- a/examples/legacy/seq2seq/old_test_datasets.py +++ b/examples/legacy/seq2seq/old_test_datasets.py @@ -28,7 +28,7 @@ from utils import FAIRSEQ_AVAILABLE, DistributedSortishSampler, LegacySeq2SeqDataset, Seq2SeqDataset -BERT_BASE_CASED = "bert-base-cased" +BERT_BASE_CASED = "google-bert/bert-base-cased" PEGASUS_XSUM = "google/pegasus-xsum" ARTICLES = [" Sam ate lunch today.", "Sams lunch ingredients."] SUMMARIES = ["A very interesting story about what I ate for lunch.", "Avocado, celery, turkey, coffee"] diff --git a/examples/legacy/seq2seq/pack_dataset.py b/examples/legacy/seq2seq/pack_dataset.py index 8b069e452a7177..5c13c74f412df6 100755 --- a/examples/legacy/seq2seq/pack_dataset.py +++ b/examples/legacy/seq2seq/pack_dataset.py @@ -74,7 +74,7 @@ def pack_data_dir(tok, data_dir: Path, max_tokens, save_path): def packer_cli(): parser = argparse.ArgumentParser() - parser.add_argument("--tok_name", type=str, help="like facebook/bart-large-cnn,t5-base, etc.") + parser.add_argument("--tok_name", type=str, help="like facebook/bart-large-cnn,google-t5/t5-base, etc.") parser.add_argument("--max_seq_len", type=int, default=128) parser.add_argument("--data_dir", type=str) parser.add_argument("--save_path", type=str) diff --git a/examples/legacy/seq2seq/requirements.txt b/examples/legacy/seq2seq/requirements.txt index e40aef17932017..434f647adea299 100644 --- a/examples/legacy/seq2seq/requirements.txt +++ b/examples/legacy/seq2seq/requirements.txt @@ -14,7 +14,7 @@ nltk pandas datasets >= 1.1.3 fire -pytest +pytest<8.0.1 conllu sentencepiece != 0.1.92 protobuf diff --git a/examples/legacy/seq2seq/run_distributed_eval.py b/examples/legacy/seq2seq/run_distributed_eval.py index 55f3839d736483..40a946f81c5e15 100755 --- a/examples/legacy/seq2seq/run_distributed_eval.py +++ b/examples/legacy/seq2seq/run_distributed_eval.py @@ -124,7 +124,7 @@ def run_generate(): parser.add_argument( "--model_name", type=str, - help="like facebook/bart-large-cnn,t5-base, etc.", + help="like facebook/bart-large-cnn,google-t5/t5-base, etc.", default="sshleifer/distilbart-xsum-12-3", ) parser.add_argument("--save_dir", type=str, help="where to save", default="tmp_gen") @@ -154,7 +154,7 @@ def run_generate(): parser.add_argument("--src_lang", type=str, default=None, required=False) parser.add_argument("--tgt_lang", type=str, default=None, required=False) parser.add_argument( - "--prefix", type=str, required=False, default=None, help="will be added to the begininng of src examples" + "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples" ) parser.add_argument("--fp16", action="store_true") parser.add_argument("--debug", action="store_true") diff --git a/examples/legacy/seq2seq/run_eval.py b/examples/legacy/seq2seq/run_eval.py index 35e11c86a116bf..f69e5d51264c78 100755 --- a/examples/legacy/seq2seq/run_eval.py +++ b/examples/legacy/seq2seq/run_eval.py @@ -100,14 +100,14 @@ def run_generate(verbose=True): """ parser = argparse.ArgumentParser() - parser.add_argument("model_name", type=str, help="like facebook/bart-large-cnn,t5-base, etc.") + parser.add_argument("model_name", type=str, help="like facebook/bart-large-cnn,google-t5/t5-base, etc.") parser.add_argument("input_path", type=str, help="like cnn_dm/test.source") parser.add_argument("save_path", type=str, help="where to save summaries") parser.add_argument("--reference_path", type=str, required=False, help="like cnn_dm/test.target") parser.add_argument("--score_path", type=str, required=False, default="metrics.json", help="where to save metrics") parser.add_argument("--device", type=str, required=False, default=DEFAULT_DEVICE, help="cuda, cuda:1, cpu etc.") parser.add_argument( - "--prefix", type=str, required=False, default=None, help="will be added to the begininng of src examples" + "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples" ) parser.add_argument("--task", type=str, default="summarization", help="used for task_specific_params + metrics") parser.add_argument("--bs", type=int, default=8, required=False, help="batch size") diff --git a/examples/legacy/seq2seq/seq2seq_trainer.py b/examples/legacy/seq2seq/seq2seq_trainer.py index dbf12725f2db07..0c981a201dd4b1 100644 --- a/examples/legacy/seq2seq/seq2seq_trainer.py +++ b/examples/legacy/seq2seq/seq2seq_trainer.py @@ -19,7 +19,6 @@ from torch.utils.data import DistributedSampler, RandomSampler from transformers import PreTrainedModel, Trainer, logging -from transformers.integrations import is_fairscale_available from transformers.models.fsmt.configuration_fsmt import FSMTConfig from transformers.optimization import ( Adafactor, @@ -33,11 +32,7 @@ ) from transformers.trainer_pt_utils import get_tpu_sampler from transformers.training_args import ParallelMode -from transformers.utils import is_torch_tpu_available - - -if is_fairscale_available(): - from fairscale.optim import OSS +from transformers.utils import is_torch_xla_available logger = logging.get_logger(__name__) @@ -70,7 +65,7 @@ def __init__(self, config=None, data_args=None, *args, **kwargs): if self.args.label_smoothing != 0 or (self.data_args is not None and self.data_args.ignore_pad_token_for_loss): assert self.config.pad_token_id is not None, ( - "Make sure that `config.pad_token_id` is correcly defined when ignoring `pad_token` for loss" + "Make sure that `config.pad_token_id` is correctly defined when ignoring `pad_token` for loss" " calculation or doing label smoothing." ) @@ -118,14 +113,7 @@ def create_optimizer_and_scheduler(self, num_training_steps: int): "eps": self.args.adam_epsilon, } optimizer_kwargs["lr"] = self.args.learning_rate - if self.sharded_ddp: - self.optimizer = OSS( - params=optimizer_grouped_parameters, - optim=optimizer_cls, - **optimizer_kwargs, - ) - else: - self.optimizer = optimizer_cls(optimizer_grouped_parameters, **optimizer_kwargs) + self.optimizer = optimizer_cls(optimizer_grouped_parameters, **optimizer_kwargs) if self.lr_scheduler is None: self.lr_scheduler = self._get_lr_scheduler(num_training_steps) @@ -147,7 +135,7 @@ def _get_lr_scheduler(self, num_training_steps): def _get_train_sampler(self) -> Optional[torch.utils.data.Sampler]: if isinstance(self.train_dataset, torch.utils.data.IterableDataset): return None - elif is_torch_tpu_available(): + elif is_torch_xla_available(): return get_tpu_sampler(self.train_dataset) else: if self.args.sortish_sampler: diff --git a/examples/legacy/seq2seq/seq2seq_training_args.py b/examples/legacy/seq2seq/seq2seq_training_args.py index 1583acd36fc4b7..9da1c69262a8c0 100644 --- a/examples/legacy/seq2seq/seq2seq_training_args.py +++ b/examples/legacy/seq2seq/seq2seq_training_args.py @@ -31,7 +31,7 @@ class Seq2SeqTrainingArguments(TrainingArguments): label_smoothing (:obj:`float`, `optional`, defaults to 0): The label smoothing epsilon to apply (if not zero). sortish_sampler (:obj:`bool`, `optional`, defaults to :obj:`False`): - Whether to SortishSamler or not. It sorts the inputs according to lenghts in-order to minimizing the padding size. + Whether to SortishSampler or not. It sorts the inputs according to lengths in-order to minimizing the padding size. predict_with_generate (:obj:`bool`, `optional`, defaults to :obj:`False`): Whether to use generate to calculate generative metrics (ROUGE, BLEU). """ @@ -39,7 +39,7 @@ class Seq2SeqTrainingArguments(TrainingArguments): label_smoothing: Optional[float] = field( default=0.0, metadata={"help": "The label smoothing epsilon to apply (if not zero)."} ) - sortish_sampler: bool = field(default=False, metadata={"help": "Whether to SortishSamler or not."}) + sortish_sampler: bool = field(default=False, metadata={"help": "Whether to SortishSampler or not."}) predict_with_generate: bool = field( default=False, metadata={"help": "Whether to use generate to calculate generative metrics (ROUGE, BLEU)."} ) diff --git a/examples/legacy/seq2seq/train_distil_marian_enro.sh b/examples/legacy/seq2seq/train_distil_marian_enro.sh index fc1b90595c5e69..5e86a6991c579e 100644 --- a/examples/legacy/seq2seq/train_distil_marian_enro.sh +++ b/examples/legacy/seq2seq/train_distil_marian_enro.sh @@ -32,7 +32,7 @@ python finetune_trainer.py \ --max_source_length $MAX_LEN --max_target_length $MAX_LEN \ --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \ --do_train --do_eval --do_predict \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate --logging_first_step \ --task translation --label_smoothing_factor 0.1 \ "$@" diff --git a/examples/legacy/seq2seq/train_distil_marian_enro_tpu.sh b/examples/legacy/seq2seq/train_distil_marian_enro_tpu.sh index 2fce7684ab449d..00ef672261963b 100644 --- a/examples/legacy/seq2seq/train_distil_marian_enro_tpu.sh +++ b/examples/legacy/seq2seq/train_distil_marian_enro_tpu.sh @@ -33,7 +33,7 @@ python xla_spawn.py --num_cores $TPU_NUM_CORES \ --max_source_length $MAX_LEN --max_target_length $MAX_LEN \ --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \ --do_train --do_eval \ - --evaluation_strategy steps \ + --eval_strategy steps \ --prediction_loss_only \ --task translation --label_smoothing_factor 0.1 \ "$@" diff --git a/examples/legacy/seq2seq/train_distilbart_cnn.sh b/examples/legacy/seq2seq/train_distilbart_cnn.sh index ec0aec8e597fb4..42f34e0cb6e75a 100644 --- a/examples/legacy/seq2seq/train_distilbart_cnn.sh +++ b/examples/legacy/seq2seq/train_distilbart_cnn.sh @@ -34,6 +34,6 @@ python finetune_trainer.py \ --logging_first_step \ --max_target_length 56 --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN\ --do_train --do_eval --do_predict \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate --sortish_sampler \ "$@" diff --git a/examples/legacy/seq2seq/train_mbart_cc25_enro.sh b/examples/legacy/seq2seq/train_mbart_cc25_enro.sh index 2b603eda7c35e6..63c8051b47def1 100644 --- a/examples/legacy/seq2seq/train_mbart_cc25_enro.sh +++ b/examples/legacy/seq2seq/train_mbart_cc25_enro.sh @@ -29,7 +29,7 @@ python finetune_trainer.py \ --num_train_epochs 6 \ --save_steps 25000 --eval_steps 25000 --logging_steps 1000 \ --do_train --do_eval --do_predict \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate --logging_first_step \ --task translation \ "$@" diff --git a/examples/legacy/seq2seq/xla_spawn.py b/examples/legacy/seq2seq/xla_spawn.py index 5df6bfa2d5dc31..f9955acfa201a7 100644 --- a/examples/legacy/seq2seq/xla_spawn.py +++ b/examples/legacy/seq2seq/xla_spawn.py @@ -23,7 +23,6 @@ """ - import importlib import sys from argparse import REMAINDER, ArgumentParser diff --git a/examples/legacy/text-classification/run_tf_text_classification.py b/examples/legacy/text-classification/run_tf_text_classification.py deleted file mode 100755 index 1f845db04c0448..00000000000000 --- a/examples/legacy/text-classification/run_tf_text_classification.py +++ /dev/null @@ -1,313 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -# Copyright 2020 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -""" Fine-tuning the library models for sequence classification.""" - - -import logging -import os -from dataclasses import dataclass, field -from typing import Dict, Optional - -import datasets -import numpy as np -import tensorflow as tf - -from transformers import ( - AutoConfig, - AutoTokenizer, - EvalPrediction, - HfArgumentParser, - PreTrainedTokenizer, - TFAutoModelForSequenceClassification, - TFTrainer, - TFTrainingArguments, -) -from transformers.utils import logging as hf_logging - - -hf_logging.set_verbosity_info() -hf_logging.enable_default_handler() -hf_logging.enable_explicit_format() - - -def get_tfds( - train_file: str, - eval_file: str, - test_file: str, - tokenizer: PreTrainedTokenizer, - label_column_id: int, - max_seq_length: Optional[int] = None, -): - files = {} - - if train_file is not None: - files[datasets.Split.TRAIN] = [train_file] - if eval_file is not None: - files[datasets.Split.VALIDATION] = [eval_file] - if test_file is not None: - files[datasets.Split.TEST] = [test_file] - - ds = datasets.load_dataset("csv", data_files=files) - features_name = list(ds[list(files.keys())[0]].features.keys()) - label_name = features_name.pop(label_column_id) - label_list = list(set(ds[list(files.keys())[0]][label_name])) - label2id = {label: i for i, label in enumerate(label_list)} - input_names = tokenizer.model_input_names - transformed_ds = {} - - if len(features_name) == 1: - for k in files.keys(): - transformed_ds[k] = ds[k].map( - lambda example: tokenizer.batch_encode_plus( - example[features_name[0]], truncation=True, max_length=max_seq_length, padding="max_length" - ), - batched=True, - ) - elif len(features_name) == 2: - for k in files.keys(): - transformed_ds[k] = ds[k].map( - lambda example: tokenizer.batch_encode_plus( - (example[features_name[0]], example[features_name[1]]), - truncation=True, - max_length=max_seq_length, - padding="max_length", - ), - batched=True, - ) - - def gen_train(): - for ex in transformed_ds[datasets.Split.TRAIN]: - d = {k: v for k, v in ex.items() if k in input_names} - label = label2id[ex[label_name]] - yield (d, label) - - def gen_val(): - for ex in transformed_ds[datasets.Split.VALIDATION]: - d = {k: v for k, v in ex.items() if k in input_names} - label = label2id[ex[label_name]] - yield (d, label) - - def gen_test(): - for ex in transformed_ds[datasets.Split.TEST]: - d = {k: v for k, v in ex.items() if k in input_names} - label = label2id[ex[label_name]] - yield (d, label) - - train_ds = ( - tf.data.Dataset.from_generator( - gen_train, - ({k: tf.int32 for k in input_names}, tf.int64), - ({k: tf.TensorShape([None]) for k in input_names}, tf.TensorShape([])), - ) - if datasets.Split.TRAIN in transformed_ds - else None - ) - - if train_ds is not None: - train_ds = train_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.TRAIN]))) - - val_ds = ( - tf.data.Dataset.from_generator( - gen_val, - ({k: tf.int32 for k in input_names}, tf.int64), - ({k: tf.TensorShape([None]) for k in input_names}, tf.TensorShape([])), - ) - if datasets.Split.VALIDATION in transformed_ds - else None - ) - - if val_ds is not None: - val_ds = val_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.VALIDATION]))) - - test_ds = ( - tf.data.Dataset.from_generator( - gen_test, - ({k: tf.int32 for k in input_names}, tf.int64), - ({k: tf.TensorShape([None]) for k in input_names}, tf.TensorShape([])), - ) - if datasets.Split.TEST in transformed_ds - else None - ) - - if test_ds is not None: - test_ds = test_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.TEST]))) - - return train_ds, val_ds, test_ds, label2id - - -logger = logging.getLogger(__name__) - - -@dataclass -class DataTrainingArguments: - """ - Arguments pertaining to what data we are going to input our model for training and eval. - - Using `HfArgumentParser` we can turn this class - into argparse arguments to be able to specify them on - the command line. - """ - - label_column_id: int = field(metadata={"help": "Which column contains the label"}) - train_file: str = field(default=None, metadata={"help": "The path of the training file"}) - dev_file: Optional[str] = field(default=None, metadata={"help": "The path of the development file"}) - test_file: Optional[str] = field(default=None, metadata={"help": "The path of the test file"}) - max_seq_length: int = field( - default=128, - metadata={ - "help": ( - "The maximum total input sequence length after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded." - ) - }, - ) - overwrite_cache: bool = field( - default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} - ) - - -@dataclass -class ModelArguments: - """ - Arguments pertaining to which model/config/tokenizer we are going to fine-tune from. - """ - - model_name_or_path: str = field( - metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"} - ) - config_name: Optional[str] = field( - default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"} - ) - tokenizer_name: Optional[str] = field( - default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"} - ) - use_fast: bool = field(default=False, metadata={"help": "Set this flag to use fast tokenization."}) - # If you want to tweak more attributes on your tokenizer, you should do it in a distinct script, - # or just modify its tokenizer_config.json. - cache_dir: Optional[str] = field( - default=None, - metadata={"help": "Where do you want to store the pretrained models downloaded from huggingface.co"}, - ) - - -def main(): - # See all possible arguments in src/transformers/training_args.py - # or by passing the --help flag to this script. - # We now keep distinct sets of args, for a cleaner separation of concerns. - parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments)) - model_args, data_args, training_args = parser.parse_args_into_dataclasses() - - if ( - os.path.exists(training_args.output_dir) - and os.listdir(training_args.output_dir) - and training_args.do_train - and not training_args.overwrite_output_dir - ): - raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty. Use" - " --overwrite_output_dir to overcome." - ) - - # Setup logging - logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, - ) - logger.info( - f"n_replicas: {training_args.n_replicas}, distributed training: {bool(training_args.n_replicas > 1)}, " - f"16-bits training: {training_args.fp16}" - ) - logger.info(f"Training/evaluation parameters {training_args}") - - # Load pretrained model and tokenizer - # - # Distributed training: - # The .from_pretrained methods guarantee that only one local process can concurrently - # download model & vocab. - - tokenizer = AutoTokenizer.from_pretrained( - model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, - cache_dir=model_args.cache_dir, - ) - - train_dataset, eval_dataset, test_ds, label2id = get_tfds( - train_file=data_args.train_file, - eval_file=data_args.dev_file, - test_file=data_args.test_file, - tokenizer=tokenizer, - label_column_id=data_args.label_column_id, - max_seq_length=data_args.max_seq_length, - ) - - config = AutoConfig.from_pretrained( - model_args.config_name if model_args.config_name else model_args.model_name_or_path, - num_labels=len(label2id), - label2id=label2id, - id2label={id: label for label, id in label2id.items()}, - finetuning_task="text-classification", - cache_dir=model_args.cache_dir, - ) - - with training_args.strategy.scope(): - model = TFAutoModelForSequenceClassification.from_pretrained( - model_args.model_name_or_path, - from_pt=bool(".bin" in model_args.model_name_or_path), - config=config, - cache_dir=model_args.cache_dir, - ) - - def compute_metrics(p: EvalPrediction) -> Dict: - preds = np.argmax(p.predictions, axis=1) - - return {"acc": (preds == p.label_ids).mean()} - - # Initialize our Trainer - trainer = TFTrainer( - model=model, - args=training_args, - train_dataset=train_dataset, - eval_dataset=eval_dataset, - compute_metrics=compute_metrics, - ) - - # Training - if training_args.do_train: - trainer.train() - trainer.save_model() - tokenizer.save_pretrained(training_args.output_dir) - - # Evaluation - results = {} - if training_args.do_eval: - logger.info("*** Evaluate ***") - result = trainer.evaluate() - output_eval_file = os.path.join(training_args.output_dir, "eval_results.txt") - - with open(output_eval_file, "w") as writer: - logger.info("***** Eval results *****") - - for key, value in result.items(): - logger.info(f" {key} = {value}") - writer.write(f"{key} = {value}\n") - - results.update(result) - - return results - - -if __name__ == "__main__": - main() diff --git a/examples/legacy/token-classification/README.md b/examples/legacy/token-classification/README.md index c2fa6eec7282b2..fbf17f84d2d7ee 100644 --- a/examples/legacy/token-classification/README.md +++ b/examples/legacy/token-classification/README.md @@ -34,7 +34,7 @@ Let's define some variables that we need for further pre-processing steps and tr ```bash export MAX_LENGTH=128 -export BERT_MODEL=bert-base-multilingual-cased +export BERT_MODEL=google-bert/bert-base-multilingual-cased ``` Run the pre-processing script on training, dev and test datasets: @@ -92,7 +92,7 @@ Instead of passing all parameters via commandline arguments, the `run_ner.py` sc { "data_dir": ".", "labels": "./labels.txt", - "model_name_or_path": "bert-base-multilingual-cased", + "model_name_or_path": "google-bert/bert-base-multilingual-cased", "output_dir": "germeval-model", "max_seq_length": 128, "num_train_epochs": 3, @@ -222,7 +222,7 @@ Let's define some variables that we need for further pre-processing steps: ```bash export MAX_LENGTH=128 -export BERT_MODEL=bert-large-cased +export BERT_MODEL=google-bert/bert-large-cased ``` Here we use the English BERT large model for fine-tuning. @@ -250,7 +250,7 @@ This configuration file looks like: { "data_dir": "./data_wnut_17", "labels": "./data_wnut_17/labels.txt", - "model_name_or_path": "bert-large-cased", + "model_name_or_path": "google-bert/bert-large-cased", "output_dir": "wnut-17-model-1", "max_seq_length": 128, "num_train_epochs": 3, diff --git a/examples/legacy/token-classification/run_ner.py b/examples/legacy/token-classification/run_ner.py index c571d44a1203c5..4fb74e78ff9fe1 100644 --- a/examples/legacy/token-classification/run_ner.py +++ b/examples/legacy/token-classification/run_ner.py @@ -13,7 +13,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-tuning the library models for named entity recognition on CoNLL-2003. """ +"""Fine-tuning the library models for named entity recognition on CoNLL-2003.""" + import logging import os import sys diff --git a/examples/legacy/token-classification/run_tf_ner.py b/examples/legacy/token-classification/run_tf_ner.py deleted file mode 100755 index a9c41d58183d4a..00000000000000 --- a/examples/legacy/token-classification/run_tf_ner.py +++ /dev/null @@ -1,310 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -# Copyright 2018 The HuggingFace Inc. team. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -""" Fine-tuning the library models for named entity recognition.""" - - -import logging -import os -from dataclasses import dataclass, field -from importlib import import_module -from typing import Dict, List, Optional, Tuple - -import numpy as np -from seqeval.metrics import classification_report, f1_score, precision_score, recall_score -from utils_ner import Split, TFTokenClassificationDataset, TokenClassificationTask - -from transformers import ( - AutoConfig, - AutoTokenizer, - EvalPrediction, - HfArgumentParser, - TFAutoModelForTokenClassification, - TFTrainer, - TFTrainingArguments, -) -from transformers.utils import logging as hf_logging - - -hf_logging.set_verbosity_info() -hf_logging.enable_default_handler() -hf_logging.enable_explicit_format() - - -logger = logging.getLogger(__name__) - - -@dataclass -class ModelArguments: - """ - Arguments pertaining to which model/config/tokenizer we are going to fine-tune from. - """ - - model_name_or_path: str = field( - metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"} - ) - config_name: Optional[str] = field( - default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"} - ) - task_type: Optional[str] = field( - default="NER", metadata={"help": "Task type to fine tune in training (e.g. NER, POS, etc)"} - ) - tokenizer_name: Optional[str] = field( - default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"} - ) - use_fast: bool = field(default=False, metadata={"help": "Set this flag to use fast tokenization."}) - # If you want to tweak more attributes on your tokenizer, you should do it in a distinct script, - # or just modify its tokenizer_config.json. - cache_dir: Optional[str] = field( - default=None, - metadata={"help": "Where do you want to store the pretrained models downloaded from huggingface.co"}, - ) - - -@dataclass -class DataTrainingArguments: - """ - Arguments pertaining to what data we are going to input our model for training and eval. - """ - - data_dir: str = field( - metadata={"help": "The input data dir. Should contain the .txt files for a CoNLL-2003-formatted task."} - ) - labels: Optional[str] = field( - metadata={"help": "Path to a file containing all labels. If not specified, CoNLL-2003 labels are used."} - ) - max_seq_length: int = field( - default=128, - metadata={ - "help": ( - "The maximum total input sequence length after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded." - ) - }, - ) - overwrite_cache: bool = field( - default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} - ) - - -def main(): - # See all possible arguments in src/transformers/training_args.py - # or by passing the --help flag to this script. - # We now keep distinct sets of args, for a cleaner separation of concerns. - parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments)) - model_args, data_args, training_args = parser.parse_args_into_dataclasses() - - if ( - os.path.exists(training_args.output_dir) - and os.listdir(training_args.output_dir) - and training_args.do_train - and not training_args.overwrite_output_dir - ): - raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty. Use" - " --overwrite_output_dir to overcome." - ) - - module = import_module("tasks") - - try: - token_classification_task_clazz = getattr(module, model_args.task_type) - token_classification_task: TokenClassificationTask = token_classification_task_clazz() - except AttributeError: - raise ValueError( - f"Task {model_args.task_type} needs to be defined as a TokenClassificationTask subclass in {module}. " - f"Available tasks classes are: {TokenClassificationTask.__subclasses__()}" - ) - - # Setup logging - logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - level=logging.INFO, - ) - logger.info( - "n_replicas: %s, distributed training: %s, 16-bits training: %s", - training_args.n_replicas, - bool(training_args.n_replicas > 1), - training_args.fp16, - ) - logger.info("Training/evaluation parameters %s", training_args) - - # Prepare Token Classification task - labels = token_classification_task.get_labels(data_args.labels) - label_map: Dict[int, str] = dict(enumerate(labels)) - num_labels = len(labels) - - # Load pretrained model and tokenizer - # - # Distributed training: - # The .from_pretrained methods guarantee that only one local process can concurrently - # download model & vocab. - - config = AutoConfig.from_pretrained( - model_args.config_name if model_args.config_name else model_args.model_name_or_path, - num_labels=num_labels, - id2label=label_map, - label2id={label: i for i, label in enumerate(labels)}, - cache_dir=model_args.cache_dir, - ) - tokenizer = AutoTokenizer.from_pretrained( - model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, - cache_dir=model_args.cache_dir, - use_fast=model_args.use_fast, - ) - - with training_args.strategy.scope(): - model = TFAutoModelForTokenClassification.from_pretrained( - model_args.model_name_or_path, - from_pt=bool(".bin" in model_args.model_name_or_path), - config=config, - cache_dir=model_args.cache_dir, - ) - - # Get datasets - train_dataset = ( - TFTokenClassificationDataset( - token_classification_task=token_classification_task, - data_dir=data_args.data_dir, - tokenizer=tokenizer, - labels=labels, - model_type=config.model_type, - max_seq_length=data_args.max_seq_length, - overwrite_cache=data_args.overwrite_cache, - mode=Split.train, - ) - if training_args.do_train - else None - ) - eval_dataset = ( - TFTokenClassificationDataset( - token_classification_task=token_classification_task, - data_dir=data_args.data_dir, - tokenizer=tokenizer, - labels=labels, - model_type=config.model_type, - max_seq_length=data_args.max_seq_length, - overwrite_cache=data_args.overwrite_cache, - mode=Split.dev, - ) - if training_args.do_eval - else None - ) - - def align_predictions(predictions: np.ndarray, label_ids: np.ndarray) -> Tuple[List[int], List[int]]: - preds = np.argmax(predictions, axis=2) - batch_size, seq_len = preds.shape - out_label_list = [[] for _ in range(batch_size)] - preds_list = [[] for _ in range(batch_size)] - - for i in range(batch_size): - for j in range(seq_len): - if label_ids[i, j] != -100: - out_label_list[i].append(label_map[label_ids[i][j]]) - preds_list[i].append(label_map[preds[i][j]]) - - return preds_list, out_label_list - - def compute_metrics(p: EvalPrediction) -> Dict: - preds_list, out_label_list = align_predictions(p.predictions, p.label_ids) - - return { - "precision": precision_score(out_label_list, preds_list), - "recall": recall_score(out_label_list, preds_list), - "f1": f1_score(out_label_list, preds_list), - } - - # Initialize our Trainer - trainer = TFTrainer( - model=model, - args=training_args, - train_dataset=train_dataset.get_dataset() if train_dataset else None, - eval_dataset=eval_dataset.get_dataset() if eval_dataset else None, - compute_metrics=compute_metrics, - ) - - # Training - if training_args.do_train: - trainer.train() - trainer.save_model() - tokenizer.save_pretrained(training_args.output_dir) - - # Evaluation - results = {} - if training_args.do_eval: - logger.info("*** Evaluate ***") - - result = trainer.evaluate() - output_eval_file = os.path.join(training_args.output_dir, "eval_results.txt") - - with open(output_eval_file, "w") as writer: - logger.info("***** Eval results *****") - - for key, value in result.items(): - logger.info(" %s = %s", key, value) - writer.write("%s = %s\n" % (key, value)) - - results.update(result) - - # Predict - if training_args.do_predict: - test_dataset = TFTokenClassificationDataset( - token_classification_task=token_classification_task, - data_dir=data_args.data_dir, - tokenizer=tokenizer, - labels=labels, - model_type=config.model_type, - max_seq_length=data_args.max_seq_length, - overwrite_cache=data_args.overwrite_cache, - mode=Split.test, - ) - - predictions, label_ids, metrics = trainer.predict(test_dataset.get_dataset()) - preds_list, labels_list = align_predictions(predictions, label_ids) - report = classification_report(labels_list, preds_list) - - logger.info("\n%s", report) - - output_test_results_file = os.path.join(training_args.output_dir, "test_results.txt") - - with open(output_test_results_file, "w") as writer: - writer.write("%s\n" % report) - - # Save predictions - output_test_predictions_file = os.path.join(training_args.output_dir, "test_predictions.txt") - - with open(output_test_predictions_file, "w") as writer: - with open(os.path.join(data_args.data_dir, "test.txt"), "r") as f: - example_id = 0 - - for line in f: - if line.startswith("-DOCSTART-") or line == "" or line == "\n": - writer.write(line) - - if not preds_list[example_id]: - example_id += 1 - elif preds_list[example_id]: - output_line = line.split()[0] + " " + preds_list[example_id].pop(0) + "\n" - - writer.write(output_line) - else: - logger.warning("Maximum sequence length exceeded: No prediction for '%s'.", line.split()[0]) - - return results - - -if __name__ == "__main__": - main() diff --git a/examples/legacy/token-classification/utils_ner.py b/examples/legacy/token-classification/utils_ner.py index 2b54c7c4a49159..da4d8c3b6059fd 100644 --- a/examples/legacy/token-classification/utils_ner.py +++ b/examples/legacy/token-classification/utils_ner.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Named entity recognition fine-tuning: utilities to work with CoNLL-2003 task. """ - +"""Named entity recognition fine-tuning: utilities to work with CoNLL-2003 task.""" import logging import os @@ -113,7 +112,7 @@ def convert_examples_to_features( for word, label in zip(example.words, example.labels): word_tokens = tokenizer.tokenize(word) - # bert-base-multilingual-cased sometimes output "nothing ([]) when calling tokenize with just a space. + # google-bert/bert-base-multilingual-cased sometimes output "nothing ([]) when calling tokenize with just a space. if len(word_tokens) > 0: tokens.extend(word_tokens) # Use the real label id for the first token of the word, and padding ids for the remaining tokens diff --git a/examples/research_projects/README.md b/examples/research_projects/README.md index 32d7fee0453c50..b2f5d431f25b50 100644 --- a/examples/research_projects/README.md +++ b/examples/research_projects/README.md @@ -20,7 +20,7 @@ This folder contains various research projects using 🤗 Transformers. They are version of 🤗 Transformers that is indicated in the requirements file of each folder. Updating them to the most recent version of the library will require some work. To use any of them, just run the command -``` +```bash pip install -r requirements.txt ``` inside the folder of your choice. diff --git a/examples/research_projects/adversarial/requirements.txt b/examples/research_projects/adversarial/requirements.txt index f6332785ea0b31..99636a7fce1b8e 100644 --- a/examples/research_projects/adversarial/requirements.txt +++ b/examples/research_projects/adversarial/requirements.txt @@ -1 +1 @@ -transformers == 3.5.1 +transformers == 4.38.0 diff --git a/examples/research_projects/adversarial/run_hans.py b/examples/research_projects/adversarial/run_hans.py index 3affbb7a69257a..23625dfa7ee43c 100644 --- a/examples/research_projects/adversarial/run_hans.py +++ b/examples/research_projects/adversarial/run_hans.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for sequence classification on HANS.""" +"""Finetuning the library models for sequence classification on HANS.""" import logging import os diff --git a/examples/research_projects/bert-loses-patience/README.md b/examples/research_projects/bert-loses-patience/README.md index d1e5baa92e90bb..b405e8a9488750 100755 --- a/examples/research_projects/bert-loses-patience/README.md +++ b/examples/research_projects/bert-loses-patience/README.md @@ -15,7 +15,7 @@ export TASK_NAME=MRPC python ./run_glue_with_pabee.py \ --model_type albert \ - --model_name_or_path bert-base-uncased/albert-base-v2 \ + --model_name_or_path google-bert/bert-base-uncased/albert/albert-base-v2 \ --task_name $TASK_NAME \ --do_train \ --do_eval \ diff --git a/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_albert.py b/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_albert.py index 5e17352dc19b54..5b30155a736a96 100644 --- a/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_albert.py +++ b/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_albert.py @@ -12,7 +12,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""PyTorch ALBERT model with Patience-based Early Exit. """ +"""PyTorch ALBERT model with Patience-based Early Exit.""" import logging @@ -253,7 +253,7 @@ def forward( Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: - loss: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: + loss (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Classification (or regression if config.num_labels==1) loss. logits ``torch.FloatTensor`` of shape ``(batch_size, config.num_labels)`` Classification (or regression if config.num_labels==1) scores (before SoftMax). @@ -276,8 +276,8 @@ def forward( from torch import nn import torch - tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') - model = AlbertForSequenceClassificationWithPabee.from_pretrained('albert-base-v2') + tokenizer = AlbertTokenizer.from_pretrained('albert/albert-base-v2') + model = AlbertForSequenceClassificationWithPabee.from_pretrained('albert/albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) diff --git a/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_bert.py b/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_bert.py index b32f47d0c30020..c1ce924a57a297 100644 --- a/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_bert.py +++ b/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_bert.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""PyTorch BERT model with Patience-based Early Exit. """ - +"""PyTorch BERT model with Patience-based Early Exit.""" import logging @@ -300,8 +299,8 @@ def forward( from torch import nn import torch - tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') - model = BertForSequenceClassificationWithPabee.from_pretrained('bert-base-uncased') + tokenizer = BertTokenizer.from_pretrained('google-bert/bert-base-uncased') + model = BertForSequenceClassificationWithPabee.from_pretrained('google-bert/bert-base-uncased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 diff --git a/examples/research_projects/bert-loses-patience/requirements.txt b/examples/research_projects/bert-loses-patience/requirements.txt index 3c01e97e7cb2d0..af3b01e0645d79 100644 --- a/examples/research_projects/bert-loses-patience/requirements.txt +++ b/examples/research_projects/bert-loses-patience/requirements.txt @@ -1 +1 @@ -transformers == 3.5.1 \ No newline at end of file +transformers == 4.38.0 \ No newline at end of file diff --git a/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py b/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py index 8a59b46ab5224a..d1ee5ddde3c6ca 100755 --- a/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py +++ b/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Training and inference using the library models for sequence classification on GLUE (Bert, Albert) with PABEE.""" - +"""Training and inference using the library models for sequence classification on GLUE (Bert, Albert) with PABEE.""" import argparse import glob @@ -148,7 +147,7 @@ def train(args, train_dataset, model, tokenizer): steps_trained_in_current_epoch = 0 # Check if continuing training from a checkpoint if os.path.exists(args.model_name_or_path): - # set global_step to gobal_step of last saved checkpoint from model path + # set global_step to global_step of last saved checkpoint from model path global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0]) epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps) steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps) @@ -169,7 +168,7 @@ def train(args, train_dataset, model, tokenizer): desc="Epoch", disable=args.local_rank not in [-1, 0], ) - set_seed(args) # Added here for reproductibility + set_seed(args) # Added here for reproducibility for _ in train_iterator: epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) for step, batch in enumerate(epoch_iterator): @@ -575,7 +574,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -614,7 +613,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/research_projects/bert-loses-patience/test_run_glue_with_pabee.py b/examples/research_projects/bert-loses-patience/test_run_glue_with_pabee.py index 6a084d0741d5f5..5516924f0f2fb7 100644 --- a/examples/research_projects/bert-loses-patience/test_run_glue_with_pabee.py +++ b/examples/research_projects/bert-loses-patience/test_run_glue_with_pabee.py @@ -29,7 +29,7 @@ def test_run_glue(self): testargs = f""" run_glue_with_pabee.py --model_type albert - --model_name_or_path albert-base-v2 + --model_name_or_path albert/albert-base-v2 --data_dir ./tests/fixtures/tests_samples/MRPC/ --output_dir {tmp_dir} --overwrite_output_dir diff --git a/examples/research_projects/bertabs/README.md b/examples/research_projects/bertabs/README.md index d5e6bbbaa28699..7109c0fb72be1b 100644 --- a/examples/research_projects/bertabs/README.md +++ b/examples/research_projects/bertabs/README.md @@ -8,7 +8,7 @@ The model is loaded with the pre-trained weights for the abstractive summarizati ## Setup -``` +```bash git clone https://github.com/huggingface/transformers && cd transformers pip install . pip install nltk py-rouge diff --git a/examples/research_projects/bertabs/configuration_bertabs.py b/examples/research_projects/bertabs/configuration_bertabs.py index 02b8f27cb30a2a..4c65cd3395c2f4 100644 --- a/examples/research_projects/bertabs/configuration_bertabs.py +++ b/examples/research_projects/bertabs/configuration_bertabs.py @@ -13,7 +13,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" BertAbs configuration """ +"""BertAbs configuration""" + import logging from transformers import PretrainedConfig diff --git a/examples/research_projects/bertabs/convert_bertabs_original_pytorch_checkpoint.py b/examples/research_projects/bertabs/convert_bertabs_original_pytorch_checkpoint.py index 53ba3829b15030..338ffd21c99353 100644 --- a/examples/research_projects/bertabs/convert_bertabs_original_pytorch_checkpoint.py +++ b/examples/research_projects/bertabs/convert_bertabs_original_pytorch_checkpoint.py @@ -12,7 +12,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Convert BertExtAbs's checkpoints. +"""Convert BertExtAbs's checkpoints. The script looks like it is doing something trivial but it is not. The "weights" proposed by the authors are actually the entire model pickled. We need to load @@ -107,7 +107,7 @@ def convert_bertabs_checkpoints(path_to_checkpoints, dump_path): # ---------------------------------- logging.info("Make sure that the models' outputs are identical") - tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") + tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased") # prepare the model inputs encoder_input_ids = tokenizer.encode("This is sample éàalj'-.") diff --git a/examples/research_projects/bertabs/modeling_bertabs.py b/examples/research_projects/bertabs/modeling_bertabs.py index 19e62804ef08ea..c2c6a54be75ffa 100644 --- a/examples/research_projects/bertabs/modeling_bertabs.py +++ b/examples/research_projects/bertabs/modeling_bertabs.py @@ -33,10 +33,6 @@ MAX_SIZE = 5000 -BERTABS_FINETUNED_MODEL_ARCHIVE_LIST = [ - "remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization", -] - class BertAbsPreTrainedModel(PreTrainedModel): config_class = BertAbsConfig @@ -128,7 +124,7 @@ class Bert(nn.Module): def __init__(self): super().__init__() - config = BertConfig.from_pretrained("bert-base-uncased") + config = BertConfig.from_pretrained("google-bert/bert-base-uncased") self.model = BertModel(config) def forward(self, input_ids, attention_mask=None, token_type_ids=None, **kwargs): @@ -561,7 +557,7 @@ def unshape(x): return context -class DecoderState(object): +class DecoderState: """Interface for grouping together the current state of a recurrent decoder. In the simplest case just represents the hidden state of the model. But can also be used for implementing various forms of @@ -698,7 +694,7 @@ def build_predictor(args, tokenizer, symbols, model, logger=None): return translator -class GNMTGlobalScorer(object): +class GNMTGlobalScorer: """ NMT re-ranking score from "Google's Neural Machine Translation System" :cite:`wu2016google` @@ -721,7 +717,7 @@ def score(self, beam, logprobs): return normalized_probs -class PenaltyBuilder(object): +class PenaltyBuilder: """ Returns the Length and Coverage Penalty function for Beam Search. @@ -767,7 +763,7 @@ def length_none(self, beam, logprobs, alpha=0.0, beta=0.0): return logprobs -class Translator(object): +class Translator: """ Uses a model to translate a batch of sentences. @@ -1006,7 +1002,7 @@ def tile(x, count, dim=0): # -class BertSumOptimizer(object): +class BertSumOptimizer: """Specific optimizer for BertSum. As described in [1], the authors fine-tune BertSum for abstractive diff --git a/examples/research_projects/bertabs/requirements.txt b/examples/research_projects/bertabs/requirements.txt index cdbfb260c7df86..bc2a3d6a163005 100644 --- a/examples/research_projects/bertabs/requirements.txt +++ b/examples/research_projects/bertabs/requirements.txt @@ -1,4 +1,4 @@ -transformers == 3.5.1 +transformers == 4.38.0 # For ROUGE nltk diff --git a/examples/research_projects/bertabs/run_summarization.py b/examples/research_projects/bertabs/run_summarization.py index 82ef8ab39ea9b7..1f969f117baaf2 100644 --- a/examples/research_projects/bertabs/run_summarization.py +++ b/examples/research_projects/bertabs/run_summarization.py @@ -29,7 +29,7 @@ def evaluate(args): - tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", do_lower_case=True) + tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased", do_lower_case=True) model = BertAbs.from_pretrained("remi/bertabs-finetuned-extractive-abstractive-summarization") model.to(args.device) model.eval() diff --git a/examples/research_projects/bertology/requirements.txt b/examples/research_projects/bertology/requirements.txt index f6332785ea0b31..99636a7fce1b8e 100644 --- a/examples/research_projects/bertology/requirements.txt +++ b/examples/research_projects/bertology/requirements.txt @@ -1 +1 @@ -transformers == 3.5.1 +transformers == 4.38.0 diff --git a/examples/research_projects/bertology/run_bertology.py b/examples/research_projects/bertology/run_bertology.py index 4cb046066c768b..35d096f1649915 100644 --- a/examples/research_projects/bertology/run_bertology.py +++ b/examples/research_projects/bertology/run_bertology.py @@ -12,13 +12,14 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Bertology: this script shows how you can explore the internals of the models in the library to: - - compute the entropy of the head attentions - - compute the importance of each head - - prune (remove) the low importance head. - Some parts of this script are adapted from the code of Michel et al. (http://arxiv.org/abs/1905.10650) - which is available at https://github.com/pmichel31415/are-16-heads-really-better-than-1 +"""Bertology: this script shows how you can explore the internals of the models in the library to: +- compute the entropy of the head attentions +- compute the importance of each head +- prune (remove) the low importance head. +Some parts of this script are adapted from the code of Michel et al. (http://arxiv.org/abs/1905.10650) +which is available at https://github.com/pmichel31415/are-16-heads-really-better-than-1 """ + import argparse import logging import os diff --git a/examples/research_projects/bertology/run_prune_gpt.py b/examples/research_projects/bertology/run_prune_gpt.py index fa7484a787b6c2..d227634c2bf787 100644 --- a/examples/research_projects/bertology/run_prune_gpt.py +++ b/examples/research_projects/bertology/run_prune_gpt.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -""" This script is adapted from the Bertology pruning code (https://github.com/huggingface/transformers/blob/783d7d2629e97c5f0c5f9ef01b8c66410275c204/examples/research_projects/bertology/run_bertology.py) +"""This script is adapted from the Bertology pruning code (https://github.com/huggingface/transformers/blob/783d7d2629e97c5f0c5f9ef01b8c66410275c204/examples/research_projects/bertology/run_bertology.py) to prune GPT-like models. The author is @altsoph. """ diff --git a/examples/research_projects/codeparrot/README.md b/examples/research_projects/codeparrot/README.md index 6c57c4350fbc02..f0af3d144f781a 100644 --- a/examples/research_projects/codeparrot/README.md +++ b/examples/research_projects/codeparrot/README.md @@ -50,7 +50,7 @@ The raw dataset contains many duplicates. We deduplicated and filtered the datas - fraction of alphanumeric characters < 0.25 - containing the word "auto-generated" or similar in the first 5 lines - filtering with a probability of 0.7 of files with a mention of "test file" or "configuration file" or similar in the first 5 lines -- filtering with a probability of 0.7 of files with high occurence of the keywords "test " or "config" +- filtering with a probability of 0.7 of files with high occurrence of the keywords "test " or "config" - filtering with a probability of 0.7 of files without a mention of the keywords `def` , `for`, `while` and `class` - filtering files that use the assignment operator `=` less than 5 times - filtering files with ratio between number of characters and number of tokens after tokenization < 1.5 (the average ratio is 3.6) @@ -79,7 +79,7 @@ python scripts/pretokenizing.py \ Before training a new model for code we create a new tokenizer that is efficient at code tokenization. To train the tokenizer you can run the following command: ```bash python scripts/bpe_training.py \ - --base_tokenizer gpt2 \ + --base_tokenizer openai-community/gpt2 \ --dataset_name codeparrot/codeparrot-clean-train ``` @@ -90,12 +90,12 @@ The models are randomly initialized and trained from scratch. To initialize a ne ```bash python scripts/initialize_model.py \ ---config_name gpt2-large \ +--config_name openai-community/gpt2-large \ --tokenizer_name codeparrot/codeparrot \ --model_name codeparrot \ --push_to_hub True ``` -This will initialize a new model with the architecture and configuration of `gpt2-large` and use the tokenizer to appropriately size the input embeddings. Finally, the initilaized model is pushed the hub. +This will initialize a new model with the architecture and configuration of `openai-community/gpt2-large` and use the tokenizer to appropriately size the input embeddings. Finally, the initilaized model is pushed the hub. We can either pass the name of a text dataset or a pretokenized dataset which speeds up training a bit. Now that the tokenizer and model are also ready we can start training the model. The main training script is built with `accelerate` to scale across a wide range of platforms and infrastructure scales. We train two models with [110M](https://huggingface.co/codeparrot/codeparrot-small/) and [1.5B](https://huggingface.co/codeparrot/codeparrot/) parameters for 25-30B tokens on a 16xA100 (40GB) machine which takes 1 day and 1 week, respectively. diff --git a/examples/research_projects/codeparrot/examples/requirements.txt b/examples/research_projects/codeparrot/examples/requirements.txt index 997334e27e18fc..64ee5b508f77a9 100644 --- a/examples/research_projects/codeparrot/examples/requirements.txt +++ b/examples/research_projects/codeparrot/examples/requirements.txt @@ -1,5 +1,5 @@ datasets==2.3.2 -transformers==4.21.1 +transformers==4.38.0 wandb==0.13.1 evaluate==0.2.2 -scikit-learn==1.1.2 \ No newline at end of file +scikit-learn==1.5.0 \ No newline at end of file diff --git a/examples/research_projects/codeparrot/examples/train_complexity_predictor.py b/examples/research_projects/codeparrot/examples/train_complexity_predictor.py index 927a15f9be679f..de06b988db634c 100644 --- a/examples/research_projects/codeparrot/examples/train_complexity_predictor.py +++ b/examples/research_projects/codeparrot/examples/train_complexity_predictor.py @@ -100,7 +100,7 @@ def tokenize(example): output_dir=args.output_dir, learning_rate=args.learning_rate, lr_scheduler_type=args.lr_scheduler_type, - evaluation_strategy="epoch", + eval_strategy="epoch", save_strategy="epoch", logging_strategy="epoch", per_device_train_batch_size=args.batch_size, diff --git a/examples/research_projects/codeparrot/requirements.txt b/examples/research_projects/codeparrot/requirements.txt index 7eff3ac7f135f0..2dbc65a376142c 100644 --- a/examples/research_projects/codeparrot/requirements.txt +++ b/examples/research_projects/codeparrot/requirements.txt @@ -1,8 +1,8 @@ -transformers==4.19.0 +transformers==4.38.0 datasets==1.16.0 wandb==0.12.0 tensorboard==2.6.0 -torch==1.11.0 +torch==1.13.1 huggingface-hub==0.1.0 git+https://github.com/huggingface/accelerate.git@3c45b6f760ad8745be9ebc9bbb26f5b04dea4abe datasketch==1.5.7 diff --git a/examples/research_projects/codeparrot/scripts/arguments.py b/examples/research_projects/codeparrot/scripts/arguments.py index 4def9ac3b854ec..1540319b3daf65 100644 --- a/examples/research_projects/codeparrot/scripts/arguments.py +++ b/examples/research_projects/codeparrot/scripts/arguments.py @@ -132,7 +132,7 @@ class PreprocessingArguments: default="transformersbook/codeparrot", metadata={"help": "Folder or name of dataset to process."} ) output_dir: Optional[str] = field( - default="codeparrot-clean", metadata={"help": "Folder to save processed processed dataset."} + default="codeparrot-clean", metadata={"help": "Folder to save processed dataset."} ) samples_per_file: Optional[int] = field( default=100_000, metadata={"help": "Number of files to save per JSON output file."} @@ -172,7 +172,7 @@ class TokenizerTrainingArguments: """ base_tokenizer: Optional[str] = field( - default="gpt2", metadata={"help": "Base tokenizer to build new tokenizer from."} + default="openai-community/gpt2", metadata={"help": "Base tokenizer to build new tokenizer from."} ) dataset_name: Optional[str] = field( default="transformersbook/codeparrot-train", metadata={"help": "Dataset to train tokenizer on."} @@ -211,7 +211,7 @@ class InitializationArguments: """ config_name: Optional[str] = field( - default="gpt2-large", metadata={"help": "Configuration to use for model initialization."} + default="openai-community/gpt2-large", metadata={"help": "Configuration to use for model initialization."} ) tokenizer_name: Optional[str] = field( default="codeparrot/codeparrot", metadata={"help": "Tokenizer attached to model."} diff --git a/examples/research_projects/codeparrot/scripts/codeparrot_training.py b/examples/research_projects/codeparrot/scripts/codeparrot_training.py index 2510e02c94700d..16f6077f2415c8 100644 --- a/examples/research_projects/codeparrot/scripts/codeparrot_training.py +++ b/examples/research_projects/codeparrot/scripts/codeparrot_training.py @@ -7,6 +7,7 @@ import datasets import torch from accelerate import Accelerator, DistributedType +from accelerate.utils import ProjectConfiguration from arguments import TrainingArguments from datasets import load_dataset from huggingface_hub import Repository @@ -195,7 +196,8 @@ def evaluate(args): args = parser.parse_args() # Accelerator -accelerator = Accelerator(log_with=["wandb", "tensorboard"], logging_dir=f"{args.save_dir}/log") +config = ProjectConfiguration(project_dir=args.save_dir, logging_dir="log") +accelerator = Accelerator(log_with=["wandb", "tensorboard"], project_config=config) acc_state = {str(k): str(v) for k, v in accelerator.state.__dict__.items()} args = Namespace(**vars(args), **acc_state) diff --git a/examples/research_projects/codeparrot/scripts/human_eval.py b/examples/research_projects/codeparrot/scripts/human_eval.py index 157079881d5f73..ef217a597e3385 100644 --- a/examples/research_projects/codeparrot/scripts/human_eval.py +++ b/examples/research_projects/codeparrot/scripts/human_eval.py @@ -60,7 +60,7 @@ def __call__(self, input_ids, scores, **kwargs): decoded_generations = self.tokenizer.batch_decode(input_ids[:, self.start_length :]) done = [] for decoded_generation in decoded_generations: - done.append(any([stop_string in decoded_generation for stop_string in self.eof_strings])) + done.append(any(stop_string in decoded_generation for stop_string in self.eof_strings)) return all(done) diff --git a/examples/research_projects/codeparrot/scripts/preprocessing.py b/examples/research_projects/codeparrot/scripts/preprocessing.py index aecc37223f0d51..d9cac5abfd8e19 100644 --- a/examples/research_projects/codeparrot/scripts/preprocessing.py +++ b/examples/research_projects/codeparrot/scripts/preprocessing.py @@ -1,5 +1,4 @@ import gzip -import hashlib import json import multiprocessing import os @@ -11,6 +10,7 @@ import numpy as np from arguments import PreprocessingArguments from datasets import load_dataset +from huggingface_hub.utils import insecure_hashlib from minhash_deduplication import deduplicate_dataset from transformers import AutoTokenizer, HfArgumentParser @@ -21,7 +21,7 @@ def get_hash(example): """Get hash of content field.""" - return {"hash": hashlib.md5(re.sub(PATTERN, "", example["content"]).encode("utf-8")).hexdigest()} + return {"hash": insecure_hashlib.md5(re.sub(PATTERN, "", example["content"]).encode("utf-8")).hexdigest()} def line_stats(example): @@ -60,7 +60,7 @@ def is_autogenerated(example, scan_width=5): def is_config_or_test(example, scan_width=5, coeff=0.05): """Check if file is a configuration file or a unit test by : 1- looking for keywords in the first few lines of the file. - 2- counting number of occurence of the words 'config' and 'test' with respect to number of lines. + 2- counting number of occurrence of the words 'config' and 'test' with respect to number of lines. """ keywords = ["unit tests", "test file", "configuration file"] diff --git a/examples/research_projects/decision_transformer/requirements.txt b/examples/research_projects/decision_transformer/requirements.txt index 3cf50951975ca5..732f91a6d9a5af 100644 --- a/examples/research_projects/decision_transformer/requirements.txt +++ b/examples/research_projects/decision_transformer/requirements.txt @@ -1,5 +1,5 @@ absl-py==1.0.0 -aiohttp==3.8.1 +aiohttp==3.9.4 aiosignal==1.2.0 alembic==1.7.7 appdirs==1.4.4 @@ -15,12 +15,12 @@ backcall==0.2.0 backoff==1.11.1 backports.zoneinfo==0.2.1 binaryornot==0.4.4 -black==22.1.0 +black==24.3.0 boto3==1.16.34 botocore==1.19.63 Brotli==1.0.9 cachetools==5.0.0 -certifi==2022.12.7 +certifi==2024.7.4 cffi==1.15.0 chardet==4.0.0 charset-normalizer==2.0.12 @@ -34,11 +34,11 @@ cmd2==2.4.0 codecarbon==1.2.0 colorlog==6.6.0 cookiecutter==2.1.1 -cryptography==39.0.1 +cryptography==42.0.0 csvw==2.0.0 cycler==0.11.0 Cython==0.29.28 -dash==2.3.0 +dash==2.15.0 dash-bootstrap-components==1.0.3 dash-core-components==2.0.0 dash-html-components==2.0.0 @@ -61,25 +61,25 @@ Flask==2.3.2 Flask-Compress==1.11 flatbuffers==2.0 flax==0.4.0 -fonttools==4.31.1 +fonttools==4.43.0 frozenlist==1.3.0 fsspec==2022.2.0 fugashi==1.1.2 gast==0.5.3 gitdb==4.0.9 -GitPython==3.1.30 +GitPython==3.1.41 glfw==2.5.1 google-auth==2.6.2 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 greenlet==1.1.2 -grpcio==1.44.0 +grpcio==1.53.2 gym==0.23.1 gym-notices==0.0.6 h5py==3.6.0 huggingface-hub==0.4.0 hypothesis==6.39.4 -idna==3.3 +idna==3.7 imageio==2.16.1 importlib-metadata==4.11.3 importlib-resources==5.4.0 @@ -92,12 +92,12 @@ itsdangerous==2.1.1 jax==0.3.4 jaxlib==0.3.2 jedi==0.18.1 -Jinja2==2.11.3 +Jinja2==3.1.4 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.2.0 jsonschema==4.4.0 -keras==2.8.0 +keras==2.13.1 Keras-Preprocessing==1.1.2 kiwisolver==1.4.0 kubernetes==12.0.1 @@ -119,7 +119,7 @@ nltk==3.7 numba==0.55.1 numpy==1.22.3 oauthlib==3.2.2 -onnx==1.13.0 +onnx>=1.15.0 onnxconverter-common==1.9.0 opt-einsum==3.3.0 optax==0.1.1 @@ -133,7 +133,7 @@ pbr==5.8.1 pexpect==4.8.0 phonemizer==3.0.1 pickleshare==0.7.5 -Pillow==9.3.0 +Pillow==10.3.0 Pint==0.16.1 plac==1.3.4 platformdirs==2.5.1 @@ -150,14 +150,14 @@ ptyprocess==0.7.0 pure-eval==0.2.2 py==1.11.0 py-cpuinfo==8.0.0 -pyarrow==7.0.0 +pyarrow==15.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycodestyle==2.8.0 pycparser==2.21 pyctcdecode==0.3.0 pyflakes==2.4.0 -Pygments==2.11.2 +Pygments==2.15.0 pygtrie==2.4.2 pynvml==11.4.1 pyOpenSSL==22.0.0 @@ -174,10 +174,10 @@ python-slugify==6.1.1 pytz==2022.1 pytz-deprecation-shim==0.1.0.post0 PyYAML==6.0 -ray==1.11.0 +ray>2.6.3 redis==4.5.4 regex==2022.3.15 -requests==2.27.1 +requests==2.32.0 requests-oauthlib==1.3.1 resampy==0.2.2 responses==0.18.0 @@ -187,7 +187,7 @@ rsa==4.8 s3transfer==0.3.7 sacrebleu==1.5.1 sacremoses==0.0.49 -scikit-learn==1.0.2 +scikit-learn==1.5.0 scipy==1.8.0 segments==2.2.0 sentencepiece==0.1.96 @@ -205,7 +205,7 @@ tensorboard==2.8.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorboardX==2.5 -tensorflow==2.8.1 +tensorflow==2.11.1 tensorflow-io-gcs-filesystem==0.24.0 termcolor==1.1.0 text-unidecode==1.3 @@ -217,10 +217,10 @@ timm==0.5.4 tokenizers==0.11.6 tomli==2.0.1 toolz==0.11.2 -torch==1.11.0 +torch==1.13.1 torchaudio==0.11.0 torchvision==0.12.0 -tqdm==4.63.0 +tqdm==4.66.3 traitlets==5.1.1 -e git+git@github.com:edbeeching/transformers.git@77b90113ca0a0e4058b046796c874bdc98f1da61#egg=transformers typing-extensions==4.1.1 @@ -229,12 +229,12 @@ tzlocal==4.1 unidic==1.1.0 unidic-lite==1.0.8 uritemplate==4.1.1 -urllib3==1.26.9 +urllib3==1.26.19 wasabi==0.9.0 wcwidth==0.2.5 websocket-client==1.3.1 -Werkzeug==2.2.3 +Werkzeug==3.0.3 wrapt==1.14.0 xxhash==3.0.0 yarl==1.7.2 -zipp==3.7.0 \ No newline at end of file +zipp==3.19.1 \ No newline at end of file diff --git a/examples/research_projects/deebert/README.md b/examples/research_projects/deebert/README.md index 30c871e1a594fc..08a087dc03ebaf 100644 --- a/examples/research_projects/deebert/README.md +++ b/examples/research_projects/deebert/README.md @@ -34,7 +34,7 @@ This is for evaluating fine-tuned DeeBERT models, given a number of different ea ## Citation Please cite our paper if you find the resource useful: -``` +```bibtex @inproceedings{xin-etal-2020-deebert, title = "{D}ee{BERT}: Dynamic Early Exiting for Accelerating {BERT} Inference", author = "Xin, Ji and diff --git a/examples/research_projects/deebert/requirements.txt b/examples/research_projects/deebert/requirements.txt index f6332785ea0b31..99636a7fce1b8e 100644 --- a/examples/research_projects/deebert/requirements.txt +++ b/examples/research_projects/deebert/requirements.txt @@ -1 +1 @@ -transformers == 3.5.1 +transformers == 4.38.0 diff --git a/examples/research_projects/deebert/run_glue_deebert.py b/examples/research_projects/deebert/run_glue_deebert.py index 6f7cfe65d0ef67..6ca28ab5bc07bc 100644 --- a/examples/research_projects/deebert/run_glue_deebert.py +++ b/examples/research_projects/deebert/run_glue_deebert.py @@ -162,7 +162,7 @@ def train(args, train_dataset, model, tokenizer, train_highway=False): tr_loss, logging_loss = 0.0, 0.0 model.zero_grad() train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]) - set_seed(args) # Added here for reproductibility (even between python 2 and 3) + set_seed(args) # Added here for reproducibility (even between python 2 and 3) for _ in train_iterator: epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) for step, batch in enumerate(epoch_iterator): @@ -491,7 +491,7 @@ def main(): help="Number of updates steps to accumulate before performing a backward/update pass.", ) parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.") - parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.") + parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.") parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.") parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.") parser.add_argument( @@ -532,7 +532,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -566,7 +566,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/research_projects/deebert/src/modeling_highway_bert.py b/examples/research_projects/deebert/src/modeling_highway_bert.py index 2a881decbbd529..b866ef0869c758 100644 --- a/examples/research_projects/deebert/src/modeling_highway_bert.py +++ b/examples/research_projects/deebert/src/modeling_highway_bert.py @@ -32,7 +32,7 @@ def __init__(self, config): self.early_exit_entropy = [-1 for _ in range(config.num_hidden_layers)] def set_early_exit_entropy(self, x): - if (type(x) is float) or (type(x) is int): + if isinstance(x, (float, int)): for i in range(len(self.early_exit_entropy)): self.early_exit_entropy[i] = x else: @@ -232,9 +232,7 @@ def forward( outputs = ( sequence_output, pooled_output, - ) + encoder_outputs[ - 1: - ] # add hidden_states and attentions if they are here + ) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, pooled_output, (hidden_states), (attentions), highway exits diff --git a/examples/research_projects/deebert/test_glue_deebert.py b/examples/research_projects/deebert/test_glue_deebert.py index 775c4d70b6523e..7a5f059c8cedff 100644 --- a/examples/research_projects/deebert/test_glue_deebert.py +++ b/examples/research_projects/deebert/test_glue_deebert.py @@ -48,7 +48,7 @@ def run_and_check(self, args): def test_glue_deebert_train(self): train_args = """ --model_type roberta - --model_name_or_path roberta-base + --model_name_or_path FacebookAI/roberta-base --task_name MRPC --do_train --do_eval @@ -61,7 +61,7 @@ def test_glue_deebert_train(self): --num_train_epochs 3 --overwrite_output_dir --seed 42 - --output_dir ./examples/deebert/saved_models/roberta-base/MRPC/two_stage + --output_dir ./examples/deebert/saved_models/FacebookAI/roberta-base/MRPC/two_stage --plot_data_dir ./examples/deebert/results/ --save_steps 0 --overwrite_cache @@ -71,12 +71,12 @@ def test_glue_deebert_train(self): eval_args = """ --model_type roberta - --model_name_or_path ./examples/deebert/saved_models/roberta-base/MRPC/two_stage + --model_name_or_path ./examples/deebert/saved_models/FacebookAI/roberta-base/MRPC/two_stage --task_name MRPC --do_eval --do_lower_case --data_dir ./tests/fixtures/tests_samples/MRPC/ - --output_dir ./examples/deebert/saved_models/roberta-base/MRPC/two_stage + --output_dir ./examples/deebert/saved_models/FacebookAI/roberta-base/MRPC/two_stage --plot_data_dir ./examples/deebert/results/ --max_seq_length 128 --eval_each_highway @@ -88,12 +88,12 @@ def test_glue_deebert_train(self): entropy_eval_args = """ --model_type roberta - --model_name_or_path ./examples/deebert/saved_models/roberta-base/MRPC/two_stage + --model_name_or_path ./examples/deebert/saved_models/FacebookAI/roberta-base/MRPC/two_stage --task_name MRPC --do_eval --do_lower_case --data_dir ./tests/fixtures/tests_samples/MRPC/ - --output_dir ./examples/deebert/saved_models/roberta-base/MRPC/two_stage + --output_dir ./examples/deebert/saved_models/FacebookAI/roberta-base/MRPC/two_stage --plot_data_dir ./examples/deebert/results/ --max_seq_length 128 --early_exit_entropy 0.1 diff --git a/examples/research_projects/distillation/README.md b/examples/research_projects/distillation/README.md index 36b45f79889f0f..594e953f99d76d 100644 --- a/examples/research_projects/distillation/README.md +++ b/examples/research_projects/distillation/README.md @@ -183,7 +183,7 @@ Happy distillation! If you find the resource useful, you should cite the following paper: -``` +```bibtex @inproceedings{sanh2019distilbert, title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}, author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas}, diff --git a/examples/research_projects/distillation/distiller.py b/examples/research_projects/distillation/distiller.py index 3ef2ba87b2e211..963af976f5a4a9 100644 --- a/examples/research_projects/distillation/distiller.py +++ b/examples/research_projects/distillation/distiller.py @@ -12,9 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" The distiller to distil the student. - Adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) +"""The distiller to distil the student. +Adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) """ + import math import os import time diff --git a/examples/research_projects/distillation/grouped_batch_sampler.py b/examples/research_projects/distillation/grouped_batch_sampler.py index a068f7e09e6a8e..e25def738a8483 100644 --- a/examples/research_projects/distillation/grouped_batch_sampler.py +++ b/examples/research_projects/distillation/grouped_batch_sampler.py @@ -12,8 +12,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Adapted from PyTorch Vision (https://github.com/pytorch/vision/blob/master/references/detection/group_by_aspect_ratio.py) -""" +"""Adapted from PyTorch Vision (https://github.com/pytorch/vision/blob/master/references/detection/group_by_aspect_ratio.py)""" + import bisect import copy from collections import defaultdict @@ -59,7 +59,7 @@ class GroupedBatchSampler(BatchSampler): def __init__(self, sampler, group_ids, batch_size): if not isinstance(sampler, Sampler): - raise ValueError( + raise TypeError( "sampler should be an instance of torch.utils.data.Sampler, but got sampler={}".format(sampler) ) self.sampler = sampler diff --git a/examples/research_projects/distillation/lm_seqs_dataset.py b/examples/research_projects/distillation/lm_seqs_dataset.py index 8e0a5814abf85c..647c8f464f7eb8 100644 --- a/examples/research_projects/distillation/lm_seqs_dataset.py +++ b/examples/research_projects/distillation/lm_seqs_dataset.py @@ -12,9 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Dataset to distilled models - adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) +"""Dataset to distilled models +adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) """ + import numpy as np import torch from torch.utils.data import Dataset diff --git a/examples/research_projects/distillation/requirements.txt b/examples/research_projects/distillation/requirements.txt index 80ee9335e6f67b..4a2ed783a7c984 100644 --- a/examples/research_projects/distillation/requirements.txt +++ b/examples/research_projects/distillation/requirements.txt @@ -1,6 +1,6 @@ transformers -gitpython==3.1.30 +gitpython==3.1.41 tensorboard>=1.14.0 tensorboardX==1.8 psutil==5.6.6 diff --git a/examples/research_projects/distillation/run_squad_w_distillation.py b/examples/research_projects/distillation/run_squad_w_distillation.py index 4b8b8e542f702d..a1150f6b437e59 100644 --- a/examples/research_projects/distillation/run_squad_w_distillation.py +++ b/examples/research_projects/distillation/run_squad_w_distillation.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" This is the exact same script as `examples/question-answering/run_squad.py` (as of 2020, January 8th) with an additional and optional step of distillation.""" +"""This is the exact same script as `examples/question-answering/run_squad.py` (as of 2020, January 8th) with an additional and optional step of distillation.""" import argparse import glob @@ -165,7 +165,7 @@ def train(args, train_dataset, model, tokenizer, teacher=None): # Check if continuing training from a checkpoint if os.path.exists(args.model_name_or_path): try: - # set global_step to gobal_step of last saved checkpoint from model path + # set global_step to global_step of last saved checkpoint from model path checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0] global_step = int(checkpoint_suffix) epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps) @@ -183,7 +183,7 @@ def train(args, train_dataset, model, tokenizer, teacher=None): train_iterator = trange( epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0] ) - # Added here for reproductibility + # Added here for reproducibility set_seed(args) for _ in train_iterator: @@ -696,7 +696,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -731,7 +731,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/research_projects/distillation/scripts/binarized_data.py b/examples/research_projects/distillation/scripts/binarized_data.py index 951530d5c75aa6..3fc3214acf7ff9 100644 --- a/examples/research_projects/distillation/scripts/binarized_data.py +++ b/examples/research_projects/distillation/scripts/binarized_data.py @@ -15,6 +15,7 @@ """ Preprocessing script before distillation. """ + import argparse import logging import pickle diff --git a/examples/research_projects/distillation/scripts/extract.py b/examples/research_projects/distillation/scripts/extract.py index f60f243dece6c6..c45821d187312a 100644 --- a/examples/research_projects/distillation/scripts/extract.py +++ b/examples/research_projects/distillation/scripts/extract.py @@ -16,6 +16,7 @@ Preprocessing script before training the distilled model. Specific to RoBERTa -> DistilRoBERTa and GPT2 -> DistilGPT2. """ + import argparse import torch diff --git a/examples/research_projects/distillation/scripts/extract_distilbert.py b/examples/research_projects/distillation/scripts/extract_distilbert.py index a58105f999e827..8637970c5117c5 100644 --- a/examples/research_projects/distillation/scripts/extract_distilbert.py +++ b/examples/research_projects/distillation/scripts/extract_distilbert.py @@ -16,6 +16,7 @@ Preprocessing script before training DistilBERT. Specific to BERT -> DistilBERT. """ + import argparse import torch diff --git a/examples/research_projects/distillation/scripts/token_counts.py b/examples/research_projects/distillation/scripts/token_counts.py index 736b564ee76ea4..2f80bf31f47778 100644 --- a/examples/research_projects/distillation/scripts/token_counts.py +++ b/examples/research_projects/distillation/scripts/token_counts.py @@ -15,6 +15,7 @@ """ Preprocessing script before training the distilled model. """ + import argparse import logging import pickle diff --git a/examples/research_projects/distillation/train.py b/examples/research_projects/distillation/train.py index bb35a1df853943..15d98ace09b56c 100644 --- a/examples/research_projects/distillation/train.py +++ b/examples/research_projects/distillation/train.py @@ -16,6 +16,7 @@ Training the distilled model. Supported architectures include: BERT -> DistilBERT, RoBERTa -> DistilRoBERTa, GPT2 -> DistilGPT2. """ + import argparse import json import os @@ -208,7 +209,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/research_projects/distillation/utils.py b/examples/research_projects/distillation/utils.py index 6d439453fe08de..e86d2593bbd991 100644 --- a/examples/research_projects/distillation/utils.py +++ b/examples/research_projects/distillation/utils.py @@ -12,9 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Utils to train DistilBERT - adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) +"""Utils to train DistilBERT +adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) """ + import json import logging import os diff --git a/examples/research_projects/fsner/src/fsner/tokenizer_utils.py b/examples/research_projects/fsner/src/fsner/tokenizer_utils.py index bc5f6650ccd9f5..7169e23dbe490d 100644 --- a/examples/research_projects/fsner/src/fsner/tokenizer_utils.py +++ b/examples/research_projects/fsner/src/fsner/tokenizer_utils.py @@ -3,7 +3,7 @@ from transformers import AutoTokenizer -class FSNERTokenizerUtils(object): +class FSNERTokenizerUtils: def __init__(self, pretrained_model_name_or_path): self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path) @@ -17,7 +17,7 @@ def tokenize(self, x): `transformers.tokenization_utils_base.BatchEncoding` dict with additional keys and values for start_token_id, end_token_id and sizes of example lists for each entity type """ - if isinstance(x, list) and all([isinstance(_x, list) for _x in x]): + if isinstance(x, list) and all(isinstance(_x, list) for _x in x): d = None for l in x: t = self.tokenizer( @@ -37,7 +37,7 @@ def tokenize(self, x): d["start_token_id"] = torch.tensor(self.tokenizer.convert_tokens_to_ids("[E]")) d["end_token_id"] = torch.tensor(self.tokenizer.convert_tokens_to_ids("[/E]")) - elif isinstance(x, list) and all([isinstance(_x, str) for _x in x]): + elif isinstance(x, list) and all(isinstance(_x, str) for _x in x): d = self.tokenizer( x, padding="max_length", diff --git a/examples/research_projects/information-gain-filtration/README.md b/examples/research_projects/information-gain-filtration/README.md index bf95cb8ea81423..f685a512509f0d 100644 --- a/examples/research_projects/information-gain-filtration/README.md +++ b/examples/research_projects/information-gain-filtration/README.md @@ -64,7 +64,7 @@ To fine-tune a transformer model with IGF on a language modeling task, use the f ```python python run_clm_igf.py\ ---model_name_or_path "gpt2" \ +--model_name_or_path "openai-community/gpt2" \ --data_file="data/tokenized_stories_train_wikitext103" \ --igf_data_file="data/IGF_values" \ --context_len 32 \ @@ -84,7 +84,7 @@ python run_clm_igf.py\ If you find the resource useful, please cite the following paper -``` +```bibtex @inproceedings{antonello-etal-2021-selecting, title = "Selecting Informative Contexts Improves Language Model Fine-tuning", author = "Antonello, Richard and Beckage, Nicole and Turek, Javier and Huth, Alexander", diff --git a/examples/research_projects/information-gain-filtration/igf/igf.py b/examples/research_projects/information-gain-filtration/igf/igf.py index 6861467a33592a..4c5aefd9584e16 100644 --- a/examples/research_projects/information-gain-filtration/igf/igf.py +++ b/examples/research_projects/information-gain-filtration/igf/igf.py @@ -69,9 +69,9 @@ def compute_perplexity(model, test_data, context_len): return perplexity -def load_gpt2(model_name="gpt2"): +def load_gpt2(model_name="openai-community/gpt2"): """ - load original gpt2 and save off for quicker loading + load original openai-community/gpt2 and save off for quicker loading Args: model_name: GPT-2 diff --git a/examples/research_projects/information-gain-filtration/run_clm_igf.py b/examples/research_projects/information-gain-filtration/run_clm_igf.py index c1584a2f89adc1..74973309c4e16b 100644 --- a/examples/research_projects/information-gain-filtration/run_clm_igf.py +++ b/examples/research_projects/information-gain-filtration/run_clm_igf.py @@ -84,7 +84,7 @@ def generate_n_pairs( device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") # load pretrained model - model = load_gpt2("gpt2").to(device) + model = load_gpt2("openai-community/gpt2").to(device) print("computing perplexity on objective set") orig_perp = compute_perplexity(model, objective_set, context_len).item() print("perplexity on objective set:", orig_perp) @@ -121,7 +121,7 @@ def training_secondary_learner( set_seed(42) # Load pre-trained model - model = GPT2LMHeadModel.from_pretrained("gpt2") + model = GPT2LMHeadModel.from_pretrained("openai-community/gpt2") # Initialize secondary learner to use embedding weights of model secondary_learner = SecondaryLearner(model) @@ -153,7 +153,7 @@ def finetune( recopy_model=recopy_gpt2, secondary_learner=None, eval_interval=10, - finetuned_model_name="gpt2_finetuned.pt", + finetuned_model_name="openai-community/gpt2_finetuned.pt", ): """ fine-tune with IGF if secondary_learner is not None, else standard fine-tuning @@ -346,7 +346,10 @@ def main(): ) parser.add_argument( - "--batch_size", default=16, type=int, help="batch size of training data of language model(gpt2) " + "--batch_size", + default=16, + type=int, + help="batch size of training data of language model(openai-community/gpt2) ", ) parser.add_argument( @@ -354,7 +357,7 @@ def main(): default=10, type=int, help=( - "decay the selectivity of our secondary learner filter from" + "decay the selectivity of our secondary learner filter from " "1 standard deviation above average to 1 below average after 10 batches" ), ) @@ -383,7 +386,9 @@ def main(): ), ) - parser.add_argument("--finetuned_model_name", default="gpt2_finetuned.pt", type=str, help="finetuned_model_name") + parser.add_argument( + "--finetuned_model_name", default="openai-community/gpt2_finetuned.pt", type=str, help="finetuned_model_name" + ) parser.add_argument( "--recopy_model", @@ -416,16 +421,16 @@ def main(): igf_model_path="igf_model.pt", ) - # load pretrained gpt2 model - model = GPT2LMHeadModel.from_pretrained("gpt2") + # load pretrained openai-community/gpt2 model + model = GPT2LMHeadModel.from_pretrained("openai-community/gpt2") set_seed(42) - # Generate train and test data to train and evaluate gpt2 model + # Generate train and test data to train and evaluate openai-community/gpt2 model train_dataset, test_dataset = generate_datasets( context_len=32, file="data/tokenized_stories_train_wikitext103.jbl", number=100, min_len=1026, trim=True ) - # fine-tuning of the gpt2 model using igf (Information Gain Filtration) + # fine-tuning of the openai-community/gpt2 model using igf (Information Gain Filtration) finetune( model, train_dataset, @@ -437,7 +442,7 @@ def main(): recopy_model=recopy_gpt2, secondary_learner=secondary_learner, eval_interval=10, - finetuned_model_name="gpt2_finetuned.pt", + finetuned_model_name="openai-community/gpt2_finetuned.pt", ) diff --git a/examples/research_projects/jax-projects/README.md b/examples/research_projects/jax-projects/README.md index 66bb6c61a376e6..88d8d7f9eba926 100644 --- a/examples/research_projects/jax-projects/README.md +++ b/examples/research_projects/jax-projects/README.md @@ -159,13 +159,13 @@ to be used, but that everybody in team is on the same page on what type of model To give an example, a well-defined project would be the following: - task: summarization -- model: [t5-small](https://huggingface.co/t5-small) +- model: [google-t5/t5-small](https://huggingface.co/google-t5/t5-small) - dataset: [CNN/Daily mail](https://huggingface.co/datasets/cnn_dailymail) - training script: [run_summarization_flax.py](https://github.com/huggingface/transformers/blob/main/examples/flax/summarization/run_summarization_flax.py) - outcome: t5 model that can summarize news -- work flow: adapt `run_summarization_flax.py` to work with `t5-small`. +- work flow: adapt `run_summarization_flax.py` to work with `google-t5/t5-small`. -This example is a very easy and not the most interesting project since a `t5-small` +This example is a very easy and not the most interesting project since a `google-t5/t5-small` summarization model exists already for CNN/Daily mail and pretty much no code has to be written. A well-defined project does not need to have the dataset be part of @@ -227,7 +227,7 @@ the forum and making use of the [🤗 hub](http://huggingface.co/) to have a ver control for your models and training logs. - When debugging, it is important that the debugging cycle is kept as short as possible to be able to effectively debug. *E.g.* if there is a problem with your training script, -you should run it with just a couple of hundreds of examples and not the whole dataset script. This can be done by either making use of [datasets streaming](https://huggingface.co/docs/datasets/master/dataset_streaming.html?highlight=streaming) or by selecting just the first +you should run it with just a couple of hundreds of examples and not the whole dataset script. This can be done by either making use of [datasets streaming](https://huggingface.co/docs/datasets/master/dataset_streaming?highlight=streaming) or by selecting just the first X number of data samples after loading: ```python @@ -311,7 +311,7 @@ library from source to profit from the most current additions during the communi Simply run the following steps: -``` +```bash $ cd ~/ $ git clone https://github.com/huggingface/datasets.git $ cd datasets @@ -335,7 +335,7 @@ dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', str dummy_input = next(iter(dataset))["text"] -tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base") +tokenizer = RobertaTokenizerFast.from_pretrained("FacebookAI/roberta-base") input_ids = tokenizer(dummy_input, return_tensors="np").input_ids[:, :10] model = FlaxRobertaModel.from_pretrained("julien-c/dummy-unknown") @@ -389,13 +389,13 @@ source ~//bin/activate Next you should install JAX's TPU version on TPU by running the following command: -``` +```bash $ pip install requests ``` and then: -``` +```bash $ pip install "jax[tpu]>=0.2.16" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html ``` @@ -468,7 +468,7 @@ library from source to profit from the most current additions during the communi Simply run the following steps: -``` +```bash $ cd ~/ $ git clone https://github.com/huggingface/datasets.git $ cd datasets @@ -492,7 +492,7 @@ dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', str dummy_input = next(iter(dataset))["text"] -tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base") +tokenizer = RobertaTokenizerFast.from_pretrained("FacebookAI/roberta-base") input_ids = tokenizer(dummy_input, return_tensors="np").input_ids[:, :10] model = FlaxRobertaModel.from_pretrained("julien-c/dummy-unknown") @@ -518,7 +518,7 @@ be available in a couple of days. - [BigBird](https://github.com/huggingface/transformers/blob/main/src/transformers/models/big_bird/modeling_flax_big_bird.py) - [CLIP](https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_flax_clip.py) - [ELECTRA](https://github.com/huggingface/transformers/blob/main/src/transformers/models/electra/modeling_flax_electra.py) -- [GPT2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_flax_gpt2.py) +- [GPT2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/openai-community/gpt2/modeling_flax_gpt2.py) - [(TODO) MBART](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mbart/modeling_flax_mbart.py) - [RoBERTa](https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/modeling_flax_roberta.py) - [T5](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_flax_t5.py) @@ -568,7 +568,7 @@ class ModelPyTorch: Instantiating an object `model_pytorch` of the class `ModelPyTorch` would actually allocate memory for the model weights and attach them to the attributes `self.key_proj`, `self.value_proj`, `self.query_proj`, and `self.logits.proj`. We could access the weights via: -``` +```python key_projection_matrix = model_pytorch.key_proj.weight.data ``` @@ -729,7 +729,7 @@ Let's use the base `FlaxRobertaModel` without any heads as an example. from transformers import FlaxRobertaModel, RobertaTokenizerFast import jax -tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base") +tokenizer = RobertaTokenizerFast.from_pretrained("FacebookAI/roberta-base") inputs = tokenizer("JAX/Flax is amazing ", padding="max_length", max_length=128, return_tensors="np") model = FlaxRobertaModel.from_pretrained("julien-c/dummy-unknown") @@ -1011,7 +1011,7 @@ and run the following commands in a Python shell to save a config. ```python from transformers import RobertaConfig -config = RobertaConfig.from_pretrained("roberta-base") +config = RobertaConfig.from_pretrained("FacebookAI/roberta-base") config.save_pretrained("./") ``` @@ -1117,7 +1117,7 @@ params = model.init(key2, x) bytes_output = serialization.to_bytes(params) -repo = Repository("flax-model", clone_from="flax-community/flax-model-dummy", use_auth_token=True) +repo = Repository("flax-model", clone_from="flax-community/flax-model-dummy", token=True) with repo.commit("My cool Flax model :)"): with open("flax_model.msgpack", "wb") as f: f.write(bytes_output) @@ -1153,7 +1153,7 @@ In the following, we will describe how to do so using a standard console, but yo 2. Once you've installed the google cloud sdk, you should set your account by running the following command. Make sure that `` corresponds to the gmail address you used to sign up for this event. ```bash -$ gcloud config set account +$ gcloud config set account ``` 3. Let's also make sure the correct project is set in case your email is used for multiple gcloud projects: @@ -1193,12 +1193,12 @@ All the widgets are open sourced in the `huggingface_hub` [repo](https://github. **NLP** * **Conversational:** To have the best conversations!. [Example](https://huggingface.co/microsoft/DialoGPT-large?). * **Feature Extraction:** Retrieve the input embeddings. [Example](https://huggingface.co/sentence-transformers/distilbert-base-nli-mean-tokens?text=test). -* **Fill Mask:** Predict potential words for a mask token. [Example](https://huggingface.co/bert-base-uncased?). -* **Question Answering:** Given a context and a question, predict the answer. [Example](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad). +* **Fill Mask:** Predict potential words for a mask token. [Example](https://huggingface.co/google-bert/bert-base-uncased?). +* **Question Answering:** Given a context and a question, predict the answer. [Example](https://huggingface.co/google-bert/bert-large-uncased-whole-word-masking-finetuned-squad). * **Sentence Simmilarity:** Predict how similar a set of sentences are. Useful for Sentence Transformers. * **Summarization:** Given a text, output a summary of it. [Example](https://huggingface.co/sshleifer/distilbart-cnn-12-6). * **Table Question Answering:** Given a table and a question, predict the answer. [Example](https://huggingface.co/google/tapas-base-finetuned-wtq). -* **Text Generation:** Generate text based on a prompt. [Example](https://huggingface.co/gpt2) +* **Text Generation:** Generate text based on a prompt. [Example](https://huggingface.co/openai-community/gpt2) * **Token Classification:** Useful for tasks such as Named Entity Recognition and Part of Speech. [Example](https://huggingface.co/dslim/bert-base-NER). * **Zero-Shot Classification:** Too cool to explain with words. Here is an [example](https://huggingface.co/typeform/distilbert-base-uncased-mnli) * ([WIP](https://github.com/huggingface/huggingface_hub/issues/99)) **Table to Text Generation**. @@ -1224,25 +1224,25 @@ Sometimes you might be using different libraries or a very specific application A common use case is how to load files you have in your model repository in the Hub from the Streamlit demo. The `huggingface_hub` library is here to help you! -``` +```bash pip install huggingface_hub ``` Here is an example downloading (and caching!) a specific file directly from the Hub -``` +```python from huggingface_hub import hf_hub_download filepath = hf_hub_download("flax-community/roberta-base-als", "flax_model.msgpack"); ``` In many cases you will want to download the full repository. Here is an example downloading all the files from a repo. You can even specify specific revisions! -``` +```python from huggingface_hub import snapshot_download local_path = snapshot_download("flax-community/roberta-base-als"); ``` Note that if you're using 🤗 Transformers library, you can quickly load the model and tokenizer as follows -``` +```python from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("REPO_ID") diff --git a/examples/research_projects/jax-projects/big_bird/README.md b/examples/research_projects/jax-projects/big_bird/README.md index e8ef274bbe07cd..42586e49580ebb 100644 --- a/examples/research_projects/jax-projects/big_bird/README.md +++ b/examples/research_projects/jax-projects/big_bird/README.md @@ -57,4 +57,4 @@ wget https://huggingface.co/datasets/vasudevgupta/natural-questions-validation/r python3 evaluate.py ``` -You can find our checkpoint on HuggingFace Hub ([see this](https://huggingface.co/vasudevgupta/flax-bigbird-natural-questions)). In case you are interested in PyTorch BigBird fine-tuning, you can refer to [this repositary](https://github.com/thevasudevgupta/bigbird). +You can find our checkpoint on HuggingFace Hub ([see this](https://huggingface.co/vasudevgupta/flax-bigbird-natural-questions)). In case you are interested in PyTorch BigBird fine-tuning, you can refer to [this repository](https://github.com/thevasudevgupta/bigbird). diff --git a/examples/research_projects/jax-projects/big_bird/prepare_natural_questions.py b/examples/research_projects/jax-projects/big_bird/prepare_natural_questions.py index 6a202ba77522a6..ebbb184ccb6b6b 100644 --- a/examples/research_projects/jax-projects/big_bird/prepare_natural_questions.py +++ b/examples/research_projects/jax-projects/big_bird/prepare_natural_questions.py @@ -50,7 +50,7 @@ def choose_first(answer, is_long_answer=False): answer["remove_it"] = False cols = ["start_token", "end_token", "start_byte", "end_byte", "text"] - if not all([isinstance(answer[k], list) for k in cols]): + if not all(isinstance(answer[k], list) for k in cols): raise ValueError("Issue in ID", example["id"]) return answer diff --git a/examples/research_projects/jax-projects/dataset-streaming/README.md b/examples/research_projects/jax-projects/dataset-streaming/README.md index 416eee06af33d6..bdb6629e509c6f 100644 --- a/examples/research_projects/jax-projects/dataset-streaming/README.md +++ b/examples/research_projects/jax-projects/dataset-streaming/README.md @@ -23,7 +23,7 @@ JAX/Flax allows you to trace pure functions and compile them into efficient, fus Models written in JAX/Flax are **immutable** and updated in a purely functional way which enables simple and efficient model parallelism. -All of the following examples make use of [dataset streaming](https://huggingface.co/docs/datasets/master/dataset_streaming.html), therefore allowing to train models on massive datasets\ +All of the following examples make use of [dataset streaming](https://huggingface.co/docs/datasets/master/dataset_streaming), therefore allowing to train models on massive datasets\ without ever having to download the full dataset. ## Masked language modeling @@ -31,7 +31,7 @@ without ever having to download the full dataset. In the following, we demonstrate how to train a bi-directional transformer model using masked language modeling objective as introduced in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805). More specifically, we demonstrate how JAX/Flax and dataset streaming can be leveraged -to pre-train [**`roberta-base`**](https://huggingface.co/roberta-base) +to pre-train [**`FacebookAI/roberta-base`**](https://huggingface.co/FacebookAI/roberta-base) in English on a single TPUv3-8 pod for 10000 update steps. The example script uses the 🤗 Datasets library. You can easily customize them to your needs if you need extra processing on your datasets. @@ -42,20 +42,20 @@ Here we call the model `"english-roberta-base-dummy"`, but you can change the mo You can do this either directly on [huggingface.co](https://huggingface.co/new) (assuming that you are logged in) or via the command line: -``` +```bash huggingface-cli repo create english-roberta-base-dummy ``` Next we clone the model repository to add the tokenizer and model files. -``` +```bash git clone https://huggingface.co//english-roberta-base-dummy ``` To ensure that all tensorboard traces will be uploaded correctly, we need to track them. You can run the following command inside your model repo to do so. -``` +```bash cd english-roberta-base-dummy git lfs track "*tfevents*" ``` @@ -80,8 +80,8 @@ from transformers import RobertaTokenizerFast, RobertaConfig model_dir = "./english-roberta-base-dummy" -tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base") -config = RobertaConfig.from_pretrained("roberta-base") +tokenizer = RobertaTokenizerFast.from_pretrained("FacebookAI/roberta-base") +config = RobertaConfig.from_pretrained("FacebookAI/roberta-base") tokenizer.save_pretrained(model_dir) config.save_pretrained(model_dir) diff --git a/examples/research_projects/jax-projects/dataset-streaming/run_mlm_flax_stream.py b/examples/research_projects/jax-projects/dataset-streaming/run_mlm_flax_stream.py index 3c5bdb7b44507c..8940fab5bda34b 100755 --- a/examples/research_projects/jax-projects/dataset-streaming/run_mlm_flax_stream.py +++ b/examples/research_projects/jax-projects/dataset-streaming/run_mlm_flax_stream.py @@ -20,6 +20,7 @@ Here is the full list of checkpoints on the hub that can be fine-tuned by this script: https://huggingface.co/models?filter=fill-mask """ + import logging import os import sys @@ -76,7 +77,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -341,7 +342,7 @@ def write_eval_metric(summary_writer, eval_metrics, step): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -399,7 +400,7 @@ def write_eval_metric(summary_writer, eval_metrics, step): ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) diff --git a/examples/research_projects/jax-projects/hybrid_clip/README.md b/examples/research_projects/jax-projects/hybrid_clip/README.md index 282d5c813b7da4..72d3db1935895f 100644 --- a/examples/research_projects/jax-projects/hybrid_clip/README.md +++ b/examples/research_projects/jax-projects/hybrid_clip/README.md @@ -32,7 +32,7 @@ Models written in JAX/Flax are **immutable** and updated in a purely functional way which enables simple and efficient model parallelism. In this example we will use the vision model from [CLIP](https://huggingface.co/models?filter=clip) -as the image encoder and [`roberta-base`](https://huggingface.co/roberta-base) as the text encoder. +as the image encoder and [`FacebookAI/roberta-base`](https://huggingface.co/FacebookAI/roberta-base) as the text encoder. Note that one can also use the [ViT](https://huggingface.co/models?filter=vit) model as image encoder and any other BERT or ROBERTa model as text encoder. To train the model on languages other than English one should choose a text encoder trained on the desired language and a image-text dataset in that language. One such dataset is [WIT](https://github.com/google-research-datasets/wit). @@ -43,17 +43,17 @@ Here we call the model `"clip-roberta-base"`, but you can change the model name You can do this either directly on [huggingface.co](https://huggingface.co/new) (assuming that you are logged in) or via the command line: -``` +```bash huggingface-cli repo create clip-roberta-base ``` Next we clone the model repository to add the tokenizer and model files. -``` +```bash git clone https://huggingface.co//clip-roberta-base ``` To ensure that all tensorboard traces will be uploaded correctly, we need to track them. You can run the following command inside your model repo to do so. -``` +```bash cd clip-roberta-base git lfs track "*tfevents*" ``` @@ -76,7 +76,7 @@ Here is an example of how to load the model using pre-trained text and vision mo ```python from modeling_hybrid_clip import FlaxHybridCLIP -model = FlaxHybridCLIP.from_text_vision_pretrained("bert-base-uncased", "openai/clip-vit-base-patch32") +model = FlaxHybridCLIP.from_text_vision_pretrained("google-bert/bert-base-uncased", "openai/clip-vit-base-patch32") # save the model model.save_pretrained("bert-clip") @@ -89,7 +89,7 @@ If the checkpoints are in PyTorch then one could pass `text_from_pt=True` and `v PyTorch checkpoints convert them to flax and load the model. ```python -model = FlaxHybridCLIP.from_text_vision_pretrained("bert-base-uncased", "openai/clip-vit-base-patch32", text_from_pt=True, vision_from_pt=True) +model = FlaxHybridCLIP.from_text_vision_pretrained("google-bert/bert-base-uncased", "openai/clip-vit-base-patch32", text_from_pt=True, vision_from_pt=True) ``` This loads both the text and vision encoders using pre-trained weights, the projection layers are randomly @@ -154,9 +154,9 @@ Next we can run the example script to train the model: ```bash python run_hybrid_clip.py \ --output_dir ${MODEL_DIR} \ - --text_model_name_or_path="roberta-base" \ + --text_model_name_or_path="FacebookAI/roberta-base" \ --vision_model_name_or_path="openai/clip-vit-base-patch32" \ - --tokenizer_name="roberta-base" \ + --tokenizer_name="FacebookAI/roberta-base" \ --train_file="coco_dataset/train_dataset.json" \ --validation_file="coco_dataset/validation_dataset.json" \ --do_train --do_eval \ diff --git a/examples/research_projects/jax-projects/hybrid_clip/modeling_hybrid_clip.py b/examples/research_projects/jax-projects/hybrid_clip/modeling_hybrid_clip.py index e60f07bdd06325..08cb3bd0b3412e 100644 --- a/examples/research_projects/jax-projects/hybrid_clip/modeling_hybrid_clip.py +++ b/examples/research_projects/jax-projects/hybrid_clip/modeling_hybrid_clip.py @@ -314,8 +314,6 @@ def from_text_vision_pretrained( Information necessary to initiate the text model. Can be either: - A string, the `model id` of a pretrained model hosted inside a model repo on huggingface.co. - Valid model ids can be located at the root-level, like ``bert-base-uncased``, or namespaced under - a user or organization name, like ``dbmdz/bert-base-german-cased``. - A path to a `directory` containing model weights saved using :func:`~transformers.FlaxPreTrainedModel.save_pretrained`, e.g., ``./my_model_directory/``. - A path or url to a `PyTorch checkpoint folder` (e.g, ``./pt_model``). In @@ -327,8 +325,6 @@ def from_text_vision_pretrained( Information necessary to initiate the vision model. Can be either: - A string, the `model id` of a pretrained model hosted inside a model repo on huggingface.co. - Valid model ids can be located at the root-level, like ``bert-base-uncased``, or namespaced under - a user or organization name, like ``dbmdz/bert-base-german-cased``. - A path to a `directory` containing model weights saved using :func:`~transformers.FlaxPreTrainedModel.save_pretrained`, e.g., ``./my_model_directory/``. - A path or url to a `PyTorch checkpoint folder` (e.g, ``./pt_model``). In @@ -354,7 +350,7 @@ def from_text_vision_pretrained( >>> from transformers import FlaxHybridCLIP >>> # initialize a model from pretrained BERT and CLIP models. Note that the projection layers will be randomly initialized. >>> # If using CLIP's vision model the vision projection layer will be initialized using pre-trained weights - >>> model = FlaxHybridCLIP.from_text_vision_pretrained('bert-base-uncased', 'openai/clip-vit-base-patch32') + >>> model = FlaxHybridCLIP.from_text_vision_pretrained('google-bert/bert-base-uncased', 'openai/clip-vit-base-patch32') >>> # saving model after fine-tuning >>> model.save_pretrained("./bert-clip") >>> # load fine-tuned model diff --git a/examples/research_projects/jax-projects/hybrid_clip/requirements.txt b/examples/research_projects/jax-projects/hybrid_clip/requirements.txt index cf1859d7549477..912a362af88aa3 100644 --- a/examples/research_projects/jax-projects/hybrid_clip/requirements.txt +++ b/examples/research_projects/jax-projects/hybrid_clip/requirements.txt @@ -3,6 +3,6 @@ jaxlib>=0.1.59 flax>=0.3.5 optax>=0.0.8 -f https://download.pytorch.org/whl/torch_stable.html -torch==1.9.0+cpu +torch==1.13.1 -f https://download.pytorch.org/whl/torch_stable.html torchvision==0.10.0+cpu \ No newline at end of file diff --git a/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py b/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py index f54641408f80a2..2020f0a35c40a4 100644 --- a/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py +++ b/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py @@ -78,7 +78,7 @@ class ModelArguments: text_model_name_or_path: str = field( metadata={ "help": ( - "The text model checkpoint for weights initialization." + "The text model checkpoint for weights initialization. " "Don't set if you want to train a model from scratch." ) }, @@ -86,7 +86,7 @@ class ModelArguments: vision_model_name_or_path: str = field( metadata={ "help": ( - "The vision model checkpoint for weights initialization." + "The vision model checkpoint for weights initialization. " "Don't set if you want to train a model from scratch." ) }, @@ -163,9 +163,6 @@ class DataTrainingArguments: overwrite_cache: bool = field( default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} ) - overwrite_cache: bool = field( - default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} - ) preprocessing_num_workers: Optional[int] = field( default=None, metadata={"help": "The number of processes to use for the preprocessing."}, @@ -283,7 +280,7 @@ def write_metric(summary_writer, train_metrics, eval_metrics, train_time, step): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -311,7 +308,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -341,7 +338,7 @@ def main(): ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) diff --git a/examples/research_projects/jax-projects/model_parallel/README.md b/examples/research_projects/jax-projects/model_parallel/README.md index b63b93862db06f..393c9e89375085 100644 --- a/examples/research_projects/jax-projects/model_parallel/README.md +++ b/examples/research_projects/jax-projects/model_parallel/README.md @@ -27,7 +27,7 @@ To adapt the script for other models, we need to also change the `ParitionSpec` TODO: Add more explantion. -Before training, let's prepare our model first. To be able to shard the model, the sharded dimention needs to be a multiple of devices it'll be sharded on. But GPTNeo's vocab size is 50257, so we need to resize the embeddings accordingly. +Before training, let's prepare our model first. To be able to shard the model, the sharded dimension needs to be a multiple of devices it'll be sharded on. But GPTNeo's vocab size is 50257, so we need to resize the embeddings accordingly. ```python from transformers import FlaxGPTNeoForCausalLM, GPTNeoConfig @@ -54,7 +54,7 @@ model.save_pretrained("gpt-neo-1.3B") ```bash python run_clm_mp.py \ --model_name_or_path gpt-neo-1.3B \ - --tokenizer_name gpt2 \ + --tokenizer_name openai-community/gpt2 \ --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \ --do_train --do_eval \ --block_size 1024 \ diff --git a/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py b/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py index 7103b5a28111ff..067f7cb2b1854c 100644 --- a/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py +++ b/examples/research_projects/jax-projects/model_parallel/run_clm_mp.py @@ -70,7 +70,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -156,9 +156,6 @@ class DataTrainingArguments: ) }, ) - overwrite_cache: bool = field( - default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} - ) preprocessing_num_workers: Optional[int] = field( default=None, metadata={"help": "The number of processes to use for the preprocessing."}, @@ -214,7 +211,7 @@ def write_eval_metric(summary_writer, eval_metrics, step): def create_learning_rate_fn( train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float -) -> Callable[[int], jnp.array]: +) -> Callable[[int], jnp.ndarray]: """Returns a linear warmup, linear_decay learning rate function.""" steps_per_epoch = train_ds_size // train_batch_size num_train_steps = steps_per_epoch * num_train_epochs @@ -246,7 +243,7 @@ def main(): and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -297,14 +294,15 @@ def main(): data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" dataset = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained config and tokenizer if model_args.config_name: @@ -325,7 +323,7 @@ def main(): ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -362,13 +360,13 @@ def tokenize_function(examples): if block_size > config.max_position_embeddings: logger.warning( f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " - "Picking 1024 instead. You can change that default value by passing --block_size xxx." + f"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx." ) - block_size = 1024 + block_size = min(1024, config.max_position_embeddings) else: if data_args.block_size > tokenizer.model_max_length: logger.warning( - f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" + f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model " f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." ) block_size = min(data_args.block_size, tokenizer.model_max_length) @@ -395,7 +393,7 @@ def group_texts(examples): # to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map lm_datasets = tokenized_datasets.map( group_texts, diff --git a/examples/research_projects/jax-projects/wav2vec2/README.md b/examples/research_projects/jax-projects/wav2vec2/README.md index 3b1b74743085a2..5f8e14f47c590c 100644 --- a/examples/research_projects/jax-projects/wav2vec2/README.md +++ b/examples/research_projects/jax-projects/wav2vec2/README.md @@ -10,7 +10,7 @@ way which enables simple and efficient model parallelism. `run_wav2vec2_pretrain_flax.py` is a lightweight example of how to download and preprocess a dataset from the 🤗 Datasets library or use your own files (jsonlines or csv), then pretrain the wav2vec2 architectures above on it. -For custom datasets in `jsonlines` format please see: [the Datasets documentation](https://huggingface.co/docs/datasets/loading_datasets.html#json-files) and you also will find examples of these below. +For custom datasets in `jsonlines` format please see: [the Datasets documentation](https://huggingface.co/docs/datasets/loading_datasets#json-files) and you also will find examples of these below. Let's start by creating a model repository to save the trained model and logs. Here we call the model `"wav2vec2-base-robust"`, but you can change the model name as you like. @@ -18,20 +18,20 @@ Here we call the model `"wav2vec2-base-robust"`, but you can change the model na You can do this either directly on [huggingface.co](https://huggingface.co/new) (assuming that you are logged in) or via the command line: -``` +```bash huggingface-cli repo create wav2vec2-base-robust ``` Next we clone the model repository to add the tokenizer and model files. -``` +```bash git clone https://huggingface.co//wav2vec2-base-robust ``` To ensure that all tensorboard traces will be uploaded correctly, we need to track them. You can run the following command inside your model repo to do so. -``` +```bash cd wav2vec2-base-robust git lfs track "*tfevents*" ``` diff --git a/examples/research_projects/jax-projects/wav2vec2/run_wav2vec2_pretrain_flax.py b/examples/research_projects/jax-projects/wav2vec2/run_wav2vec2_pretrain_flax.py index 5034e1ee9137a2..017e910db0a3c6 100755 --- a/examples/research_projects/jax-projects/wav2vec2/run_wav2vec2_pretrain_flax.py +++ b/examples/research_projects/jax-projects/wav2vec2/run_wav2vec2_pretrain_flax.py @@ -144,7 +144,7 @@ class FlaxDataCollatorForWav2Vec2Pretraining: The Wav2Vec2 model used for pretraining. The data collator needs to have access to config and ``_get_feat_extract_output_lengths`` function for correct padding. feature_extractor (:class:`~transformers.Wav2Vec2FeatureExtractor`): - The processor used for proccessing the data. + The processor used for processing the data. padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: diff --git a/examples/research_projects/layoutlmv3/README.md b/examples/research_projects/layoutlmv3/README.md index 17bf4bb67cd90f..2cc0fb75bd2c16 100644 --- a/examples/research_projects/layoutlmv3/README.md +++ b/examples/research_projects/layoutlmv3/README.md @@ -32,7 +32,7 @@ python run_funsd_cord.py \ --do_train \ --do_eval \ --max_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --eval_steps 100 \ --learning_rate 1e-5 \ --load_best_model_at_end \ @@ -57,7 +57,7 @@ python run_funsd_cord.py \ --do_train \ --do_eval \ --max_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --eval_steps 100 \ --learning_rate 5e-5 \ --load_best_model_at_end \ diff --git a/examples/research_projects/layoutlmv3/run_funsd_cord.py b/examples/research_projects/layoutlmv3/run_funsd_cord.py index e826fd997424ed..ad83fbdef9dec4 100644 --- a/examples/research_projects/layoutlmv3/run_funsd_cord.py +++ b/examples/research_projects/layoutlmv3/run_funsd_cord.py @@ -250,7 +250,7 @@ def main(): "nielsr/funsd-layoutlmv3", data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) elif data_args.dataset_name == "cord": # Downloading and loading a dataset from the hub. @@ -258,7 +258,7 @@ def main(): "nielsr/cord-layoutlmv3", data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) else: raise ValueError("This script only supports either FUNSD or CORD out-of-the-box.") @@ -313,7 +313,7 @@ def get_label_list(labels): finetuning_task=data_args.task_name, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) processor = AutoProcessor.from_pretrained( @@ -321,7 +321,7 @@ def get_label_list(labels): cache_dir=model_args.cache_dir, use_fast=True, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, add_prefix_space=True, apply_ocr=False, ) @@ -332,7 +332,7 @@ def get_label_list(labels): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # Set the correspondences label/ID inside the model config diff --git a/examples/research_projects/longform-qa/eli5_app.py b/examples/research_projects/longform-qa/eli5_app.py index 1bcb6fd20d25fc..6b1b15cc9cbba3 100644 --- a/examples/research_projects/longform-qa/eli5_app.py +++ b/examples/research_projects/longform-qa/eli5_app.py @@ -36,7 +36,7 @@ def load_models(): _ = s2s_model.eval() else: s2s_tokenizer, s2s_model = make_qa_s2s_model( - model_name="t5-small", from_file="seq2seq_models/eli5_t5_model_1024_4.pth", device="cuda:0" + model_name="google-t5/t5-small", from_file="seq2seq_models/eli5_t5_model_1024_4.pth", device="cuda:0" ) return (qar_tokenizer, qar_model, s2s_tokenizer, s2s_model) @@ -158,9 +158,7 @@ def answer_question( -""" % ( - header_html, -) +""" % (header_html,) st.sidebar.markdown( header_full, unsafe_allow_html=True, diff --git a/examples/research_projects/luke/run_luke_ner_no_trainer.py b/examples/research_projects/luke/run_luke_ner_no_trainer.py index 4c5227d2c7e011..cac487b059d71f 100644 --- a/examples/research_projects/luke/run_luke_ner_no_trainer.py +++ b/examples/research_projects/luke/run_luke_ner_no_trainer.py @@ -29,7 +29,7 @@ import torch from accelerate import Accelerator, DistributedDataParallelKwargs from datasets import ClassLabel, load_dataset, load_metric -from huggingface_hub import Repository +from huggingface_hub import Repository, create_repo from luke_utils import DataCollatorForLukeTokenClassification, is_punctuation, padding_tensor from torch.utils.data import DataLoader from tqdm.auto import tqdm @@ -45,7 +45,6 @@ get_scheduler, set_seed, ) -from transformers.file_utils import get_full_repo_name from transformers.utils.versions import require_version @@ -258,11 +257,14 @@ def main(): # Handle the repository creation if accelerator.is_main_process: if args.push_to_hub: - if args.hub_model_id is None: - repo_name = get_full_repo_name(Path(args.output_dir).name, token=args.hub_token) - else: - repo_name = args.hub_model_id - repo = Repository(args.output_dir, clone_from=repo_name) + # Retrieve of infer repo_name + repo_name = args.hub_model_id + if repo_name is None: + repo_name = Path(args.output_dir).absolute().name + # Create repo and retrieve repo_id + repo_id = create_repo(repo_name, exist_ok=True, token=args.hub_token).repo_id + # Clone repo locally + repo = Repository(args.output_dir, clone_from=repo_id, token=args.hub_token) elif args.output_dir is not None: os.makedirs(args.output_dir, exist_ok=True) accelerator.wait_for_everyone() @@ -283,16 +285,17 @@ def main(): data_files = {} if args.train_file is not None: data_files["train"] = args.train_file + extension = args.train_file.split(".")[-1] if args.validation_file is not None: data_files["validation"] = args.validation_file - extension = args.train_file.split(".")[-1] + extension = args.validation_file.split(".")[-1] raw_datasets = load_dataset(extension, data_files=data_files) # Trim a number of training examples if args.debug: for split in raw_datasets.keys(): raw_datasets[split] = raw_datasets[split].select(range(100)) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. if raw_datasets["train"] is not None: column_names = raw_datasets["train"].column_names @@ -355,7 +358,7 @@ def get_label_list(labels): tokenizer_name_or_path = args.tokenizer_name if args.tokenizer_name else args.model_name_or_path if not tokenizer_name_or_path: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -610,7 +613,7 @@ def get_luke_labels(outputs, ner_tags, original_entity_spans): predicted_sequence = [label_list[0]] * len(true_tags) for _, span, label in sorted(predictions, key=lambda o: o[0], reverse=True): - if all([o == label_list[0] for o in predicted_sequence[span[0] : span[1]]]): + if all(o == label_list[0] for o in predicted_sequence[span[0] : span[1]]): predicted_sequence[span[0]] = label if span[1] - span[0] > 1: predicted_sequence[span[0] + 1 : span[1]] = [label] * (span[1] - span[0] - 1) diff --git a/examples/research_projects/lxmert/modeling_frcnn.py b/examples/research_projects/lxmert/modeling_frcnn.py index edbd224cbe08d7..c7c3bf376ce382 100644 --- a/examples/research_projects/lxmert/modeling_frcnn.py +++ b/examples/research_projects/lxmert/modeling_frcnn.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 && Huggingface Co. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 && Huggingface Co. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" + import itertools import math import os @@ -416,7 +417,7 @@ def __new__(cls, *, channels=None, height=None, width=None, stride=None): return super().__new__(cls, channels, height, width, stride) -class Box2BoxTransform(object): +class Box2BoxTransform: """ This R-CNN transformation scales the box's width and height by exp(dw), exp(dh) and shifts a box's center by the offset @@ -518,7 +519,7 @@ def apply_deltas(self, deltas, boxes): return pred_boxes -class Matcher(object): +class Matcher: """ This class assigns to each predicted "element" (e.g., a box) a ground-truth element. Each predicted element will have exactly zero or one matches; each @@ -554,8 +555,8 @@ def __init__( assert thresholds[0] > 0 thresholds.insert(0, -float("inf")) thresholds.append(float("inf")) - assert all([low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:])]) - assert all([label_i in [-1, 0, 1] for label_i in labels]) + assert all(low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:])) + assert all(label_i in [-1, 0, 1] for label_i in labels) assert len(labels) == len(thresholds) - 1 self.thresholds = thresholds self.labels = labels @@ -621,7 +622,7 @@ def set_low_quality_matches_(self, match_labels, match_quality_matrix): match_labels[pred_inds_with_highest_quality] = 1 -class RPNOutputs(object): +class RPNOutputs: def __init__( self, box2box_transform, @@ -1131,7 +1132,7 @@ def forward(self, feature_maps, boxes): return output -class ROIOutputs(object): +class ROIOutputs: def __init__(self, cfg, training=False): self.smooth_l1_beta = cfg.ROI_BOX_HEAD.SMOOTH_L1_BETA self.box2box_transform = Box2BoxTransform(weights=cfg.ROI_BOX_HEAD.BBOX_REG_WEIGHTS) @@ -1706,9 +1707,7 @@ def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs): elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path): archive_file = pretrained_model_name_or_path elif os.path.isfile(pretrained_model_name_or_path + ".index"): - assert ( - from_tf - ), "We found a TensorFlow checkpoint at {}, please set from_tf to True to load from this checkpoint".format( + assert from_tf, "We found a TensorFlow checkpoint at {}, please set from_tf to True to load from this checkpoint".format( pretrained_model_name_or_path + ".index" ) archive_file = pretrained_model_name_or_path + ".index" diff --git a/examples/research_projects/lxmert/processing_image.py b/examples/research_projects/lxmert/processing_image.py index 4343cfdbce846e..65f8f6cd377c9f 100644 --- a/examples/research_projects/lxmert/processing_image.py +++ b/examples/research_projects/lxmert/processing_image.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" + import sys from typing import Tuple diff --git a/examples/research_projects/lxmert/requirements.txt b/examples/research_projects/lxmert/requirements.txt index 0d483b6d18923d..5e2076be9ddf1f 100644 --- a/examples/research_projects/lxmert/requirements.txt +++ b/examples/research_projects/lxmert/requirements.txt @@ -4,7 +4,7 @@ async-generator==1.10 attrs==20.2.0 backcall==0.2.0 CacheControl==0.12.6 -certifi==2022.12.7 +certifi==2024.7.4 cffi==1.14.2 chardet==3.0.4 click==7.1.2 @@ -21,7 +21,7 @@ entrypoints==0.3 filelock==3.0.12 future==0.18.3 html5lib==1.0.1 -idna==2.8 +idna==3.7 ipaddr==2.2.0 ipykernel==5.3.4 ipython @@ -34,7 +34,7 @@ jsonschema==3.2.0 jupyter==1.0.0 jupyter-client==6.1.7 jupyter-console==6.2.0 -jupyter-core==4.6.3 +jupyter-core==4.11.2 jupyterlab-pygments==0.1.1 kiwisolver==1.2.0 lockfile==0.12.2 @@ -62,7 +62,7 @@ prometheus-client==0.8.0 prompt-toolkit==3.0.7 ptyprocess==0.6.0 pyaml==20.4.0 -pyarrow==1.0.1 +pyarrow==15.0.0 pycparser==2.20 Pygments>=2.7.4 pyparsing==2.4.6 @@ -75,7 +75,7 @@ pyzmq==19.0.2 qtconsole==4.7.7 QtPy==1.9.0 regex==2020.7.14 -requests==2.22.0 +requests==2.32.2 retrying==1.3.3 sacremoses==0.0.43 Send2Trash==1.5.0 @@ -84,13 +84,13 @@ six==1.14.0 terminado==0.8.3 testpath==0.4.4 tokenizers==0.8.1rc2 -torch==1.6.0 +torch==1.13.1 torchvision==0.7.0 -tornado==6.0.4 -tqdm==4.48.2 +tornado==6.4.1 +tqdm==4.66.3 traitlets git+https://github.com/huggingface/transformers.git -urllib3==1.26.5 +urllib3==1.26.19 wcwidth==0.2.5 webencodings==0.5.1 wget==3.2 diff --git a/examples/research_projects/lxmert/utils.py b/examples/research_projects/lxmert/utils.py index 2fc6ea2062efd2..995fbd2c19aecf 100644 --- a/examples/research_projects/lxmert/utils.py +++ b/examples/research_projects/lxmert/utils.py @@ -1,20 +1,20 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal, Huggingface team :) - Adapted From Facebook Inc, Detectron2 +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal, Huggingface team :) +Adapted From Facebook Inc, Detectron2 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 + http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" import copy import fnmatch @@ -28,7 +28,6 @@ from collections import OrderedDict from contextlib import contextmanager from functools import partial -from hashlib import sha256 from io import BytesIO from pathlib import Path from urllib.parse import urlparse @@ -39,6 +38,7 @@ import requests import wget from filelock import FileLock +from huggingface_hub.utils import insecure_hashlib from PIL import Image from tqdm.auto import tqdm from yaml import Loader, dump, load @@ -402,12 +402,12 @@ def _resumable_file_manager(): def url_to_filename(url, etag=None): url_bytes = url.encode("utf-8") - url_hash = sha256(url_bytes) + url_hash = insecure_hashlib.sha256(url_bytes) filename = url_hash.hexdigest() if etag: etag_bytes = etag.encode("utf-8") - etag_hash = sha256(etag_bytes) + etag_hash = insecure_hashlib.sha256(etag_bytes) filename += "." + etag_hash.hexdigest() if url.endswith(".h5"): diff --git a/examples/research_projects/lxmert/visualizing_image.py b/examples/research_projects/lxmert/visualizing_image.py index 163d661e873ec3..dcfd8426ff4f36 100644 --- a/examples/research_projects/lxmert/visualizing_image.py +++ b/examples/research_projects/lxmert/visualizing_image.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ import colorsys import io diff --git a/examples/research_projects/mlm_wwm/README.md b/examples/research_projects/mlm_wwm/README.md index 9426be7c27be1f..bf5aa9410826ed 100644 --- a/examples/research_projects/mlm_wwm/README.md +++ b/examples/research_projects/mlm_wwm/README.md @@ -32,7 +32,7 @@ to that word). This technique has been refined for Chinese in [this paper](https To fine-tune a model using whole word masking, use the following script: ```bash python run_mlm_wwm.py \ - --model_name_or_path roberta-base \ + --model_name_or_path FacebookAI/roberta-base \ --dataset_name wikitext \ --dataset_config_name wikitext-2-raw-v1 \ --do_train \ @@ -83,7 +83,7 @@ export VALIDATION_REF_FILE=/path/to/validation/chinese_ref/file export OUTPUT_DIR=/tmp/test-mlm-wwm python run_mlm_wwm.py \ - --model_name_or_path roberta-base \ + --model_name_or_path FacebookAI/roberta-base \ --train_file $TRAIN_FILE \ --validation_file $VALIDATION_FILE \ --train_ref_file $TRAIN_REF_FILE \ @@ -95,4 +95,4 @@ python run_mlm_wwm.py \ **Note1:** On TPU, you should the flag `--pad_to_max_length` to make sure all your batches have the same length. -**Note2:** And if you have any questions or something goes wrong when runing this code, don't hesitate to pin @wlhgtc. +**Note2:** And if you have any questions or something goes wrong when running this code, don't hesitate to pin @wlhgtc. diff --git a/examples/research_projects/mlm_wwm/run_mlm_wwm.py b/examples/research_projects/mlm_wwm/run_mlm_wwm.py index f14ad5adfeff16..629026bdb20a63 100644 --- a/examples/research_projects/mlm_wwm/run_mlm_wwm.py +++ b/examples/research_projects/mlm_wwm/run_mlm_wwm.py @@ -62,7 +62,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -271,14 +271,15 @@ def main(): data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" datasets = load_dataset(extension, data_files=data_files) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer # @@ -314,7 +315,7 @@ def main(): tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -325,7 +326,7 @@ def main(): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) else: logger.info("Training new model from scratch") diff --git a/examples/research_projects/mm-imdb/README.md b/examples/research_projects/mm-imdb/README.md index 7cfc2a7487ba71..68b2f15159ec23 100644 --- a/examples/research_projects/mm-imdb/README.md +++ b/examples/research_projects/mm-imdb/README.md @@ -6,11 +6,11 @@ Based on the script [`run_mmimdb.py`](https://github.com/huggingface/transformer ### Training on MM-IMDb -``` +```bash python run_mmimdb.py \ --data_dir /path/to/mmimdb/dataset/ \ --model_type bert \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --output_dir /path/to/save/dir/ \ --do_train \ --do_eval \ diff --git a/examples/research_projects/mm-imdb/run_mmimdb.py b/examples/research_projects/mm-imdb/run_mmimdb.py index 2cc3bc3a0c73cc..686691e0b9ced2 100644 --- a/examples/research_projects/mm-imdb/run_mmimdb.py +++ b/examples/research_projects/mm-imdb/run_mmimdb.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for multimodal multiclass prediction on MM-IMDB dataset.""" - +"""Finetuning the library models for multimodal multiclass prediction on MM-IMDB dataset.""" import argparse import glob @@ -134,7 +133,7 @@ def train(args, train_dataset, model, tokenizer, criterion): best_f1, n_no_improve = 0, 0 model.zero_grad() train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]) - set_seed(args) # Added here for reproductibility + set_seed(args) # Added here for reproducibility for _ in train_iterator: epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0]) for step, batch in enumerate(epoch_iterator): @@ -384,7 +383,7 @@ def main(): help="Number of updates steps to accumulate before performing a backward/update pass.", ) parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.") - parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.") + parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.") parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.") parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.") parser.add_argument( @@ -426,7 +425,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -460,7 +459,7 @@ def main(): if args.local_rank == -1 or args.no_cuda: device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count() - else: # Initializes the distributed backend which will take care of sychronizing nodes/GPUs + else: # Initializes the distributed backend which will take care of synchronizing nodes/GPUs torch.cuda.set_device(args.local_rank) device = torch.device("cuda", args.local_rank) torch.distributed.init_process_group(backend="nccl") diff --git a/examples/research_projects/movement-pruning/README.md b/examples/research_projects/movement-pruning/README.md index 76c660187472a3..575ec1a9b49287 100644 --- a/examples/research_projects/movement-pruning/README.md +++ b/examples/research_projects/movement-pruning/README.md @@ -61,7 +61,7 @@ python examples/movement-pruning/masked_run_squad.py \ --predict_file dev-v1.1.json \ --do_train --do_eval --do_lower_case \ --model_type masked_bert \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --per_gpu_train_batch_size 16 \ --warmup_steps 5400 \ --num_train_epochs 10 \ @@ -84,7 +84,7 @@ python examples/movement-pruning/masked_run_squad.py \ --predict_file dev-v1.1.json \ --do_train --do_eval --do_lower_case \ --model_type masked_bert \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --per_gpu_train_batch_size 16 \ --warmup_steps 5400 \ --num_train_epochs 10 \ @@ -104,7 +104,7 @@ python examples/movement-pruning/masked_run_squad.py \ --predict_file dev-v1.1.json \ --do_train --do_eval --do_lower_case \ --model_type masked_bert \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --per_gpu_train_batch_size 16 \ --warmup_steps 5400 \ --num_train_epochs 10 \ @@ -124,7 +124,7 @@ python examples/movement-pruning/masked_run_squad.py \ --predict_file dev-v1.1.json \ --do_train --do_eval --do_lower_case \ --model_type masked_bert \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --per_gpu_train_batch_size 16 \ --warmup_steps 5400 \ --num_train_epochs 10 \ @@ -173,7 +173,7 @@ In particular, hardware manufacturers are announcing devices that will speedup i If you find this resource useful, please consider citing the following paper: -``` +```bibtex @article{sanh2020movement, title={Movement Pruning: Adaptive Sparsity by Fine-Tuning}, author={Victor Sanh and Thomas Wolf and Alexander M. Rush}, diff --git a/examples/research_projects/movement-pruning/bertarize.py b/examples/research_projects/movement-pruning/bertarize.py index 0c9cc63571d7c1..da7534f4a6f985 100644 --- a/examples/research_projects/movement-pruning/bertarize.py +++ b/examples/research_projects/movement-pruning/bertarize.py @@ -112,8 +112,8 @@ def main(args): type=float, required=False, help=( - "For `magnitude` and `topK`, it is the level of remaining weights (in %) in the fine-pruned model." - "For `sigmoied_threshold`, it is the threshold \tau against which the (sigmoied) scores are compared." + "For `magnitude` and `topK`, it is the level of remaining weights (in %) in the fine-pruned model. " + "For `sigmoied_threshold`, it is the threshold \tau against which the (sigmoied) scores are compared. " "Not needed for `l0`" ), ) diff --git a/examples/research_projects/movement-pruning/counts_parameters.py b/examples/research_projects/movement-pruning/counts_parameters.py index 17ddb029f89780..c0ac53fb7859aa 100644 --- a/examples/research_projects/movement-pruning/counts_parameters.py +++ b/examples/research_projects/movement-pruning/counts_parameters.py @@ -15,6 +15,7 @@ Count remaining (non-zero) weights in the encoder (i.e. the transformer layers). Sparsity and remaining weights levels are equivalent: sparsity % = 100 - remaining weights %. """ + import argparse import os @@ -79,8 +80,8 @@ def main(args): type=float, required=False, help=( - "For `topK`, it is the level of remaining weights (in %) in the fine-pruned model." - "For `sigmoied_threshold`, it is the threshold \tau against which the (sigmoied) scores are compared." + "For `topK`, it is the level of remaining weights (in %) in the fine-pruned model. " + "For `sigmoied_threshold`, it is the threshold \tau against which the (sigmoied) scores are compared. " "Not needed for `l0`" ), ) diff --git a/examples/research_projects/movement-pruning/emmental/configuration_bert_masked.py b/examples/research_projects/movement-pruning/emmental/configuration_bert_masked.py index 2a3bd763a2de36..9c7459f27a7b66 100644 --- a/examples/research_projects/movement-pruning/emmental/configuration_bert_masked.py +++ b/examples/research_projects/movement-pruning/emmental/configuration_bert_masked.py @@ -13,10 +13,9 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Masked BERT model configuration. It replicates the class `~transformers.BertConfig` +"""Masked BERT model configuration. It replicates the class `~transformers.BertConfig` and adapts it to the specificities of MaskedBert (`pruning_method`, `mask_init` and `mask_scale`.""" - import logging from transformers.configuration_utils import PretrainedConfig diff --git a/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py b/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py index d404bf49aaa62d..8c0b091c7de7e7 100644 --- a/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py +++ b/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py @@ -18,7 +18,6 @@ compute the adaptive mask. Built on top of `transformers.models.bert.modeling_bert`""" - import logging import math @@ -652,9 +651,7 @@ def forward( outputs = ( sequence_output, pooled_output, - ) + encoder_outputs[ - 1: - ] # add hidden_states and attentions if they are here + ) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, pooled_output, (hidden_states), (attentions) diff --git a/examples/research_projects/movement-pruning/emmental/modules/binarizer.py b/examples/research_projects/movement-pruning/emmental/modules/binarizer.py index b4a801d56d9de2..c96975e3b37509 100644 --- a/examples/research_projects/movement-pruning/emmental/modules/binarizer.py +++ b/examples/research_projects/movement-pruning/emmental/modules/binarizer.py @@ -108,7 +108,7 @@ def backward(ctx, gradOutput): return gradOutput, None -class MagnitudeBinarizer(object): +class MagnitudeBinarizer: """ Magnitude Binarizer. Computes a binary mask M from a real value matrix S such that `M_{i,j} = 1` if and only if `S_{i,j}` diff --git a/examples/research_projects/movement-pruning/masked_run_glue.py b/examples/research_projects/movement-pruning/masked_run_glue.py index a28cdcc583b620..4ddb4248357518 100644 --- a/examples/research_projects/movement-pruning/masked_run_glue.py +++ b/examples/research_projects/movement-pruning/masked_run_glue.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-pruning Masked BERT on sequence classification on GLUE.""" +"""Fine-pruning Masked BERT on sequence classification on GLUE.""" import argparse import glob @@ -98,7 +98,7 @@ def regularization(model: nn.Module, mode: str): elif mode == "l0": regu += torch.sigmoid(param - 2 / 3 * np.log(0.1 / 1.1)).sum() / param.numel() else: - ValueError("Don't know this mode.") + raise ValueError("Don't know this mode.") counter += 1 return regu / counter @@ -311,8 +311,7 @@ def train(args, train_dataset, model, tokenizer, teacher=None): tr_loss += loss.item() if (step + 1) % args.gradient_accumulation_steps == 0 or ( # last step in epoch but step is always smaller than gradient_accumulation_steps - len(epoch_iterator) <= args.gradient_accumulation_steps - and (step + 1) == len(epoch_iterator) + len(epoch_iterator) <= args.gradient_accumulation_steps and (step + 1) == len(epoch_iterator) ): if args.fp16: nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm) @@ -671,7 +670,7 @@ def main(): default=1, type=int, help=( - "Run `initial_warmup` * `warmup_steps` steps of threshold warmup during which threshold stays" + "Run `initial_warmup` * `warmup_steps` steps of threshold warmup during which threshold stays " "at its `initial_threshold` value (sparsity schedule)." ), ) @@ -680,7 +679,7 @@ def main(): default=2, type=int, help=( - "Run `final_warmup` * `warmup_steps` steps of threshold cool-down during which threshold stays" + "Run `final_warmup` * `warmup_steps` steps of threshold cool-down during which threshold stays " "at its final_threshold value (sparsity schedule)." ), ) @@ -799,7 +798,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/research_projects/movement-pruning/masked_run_squad.py b/examples/research_projects/movement-pruning/masked_run_squad.py index 189ed5be6707b3..7b1c2b322097a4 100644 --- a/examples/research_projects/movement-pruning/masked_run_squad.py +++ b/examples/research_projects/movement-pruning/masked_run_squad.py @@ -13,8 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-pruning Masked BERT for question-answering on SQuAD.""" - +"""Fine-pruning Masked BERT for question-answering on SQuAD.""" import argparse import glob @@ -102,7 +101,7 @@ def regularization(model: nn.Module, mode: str): elif mode == "l0": regu += torch.sigmoid(param - 2 / 3 * np.log(0.1 / 1.1)).sum() / param.numel() else: - ValueError("Don't know this mode.") + raise ValueError("Don't know this mode.") counter += 1 return regu / counter @@ -789,7 +788,7 @@ def main(): default=1, type=int, help=( - "Run `initial_warmup` * `warmup_steps` steps of threshold warmup during which threshold stays" + "Run `initial_warmup` * `warmup_steps` steps of threshold warmup during which threshold stays " "at its `initial_threshold` value (sparsity schedule)." ), ) @@ -798,7 +797,7 @@ def main(): default=2, type=int, help=( - "Run `final_warmup` * `warmup_steps` steps of threshold cool-down during which threshold stays" + "Run `final_warmup` * `warmup_steps` steps of threshold cool-down during which threshold stays " "at its final_threshold value (sparsity schedule)." ), ) @@ -946,7 +945,7 @@ def main(): type=str, default="O1", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/research_projects/onnx/summarization/run_onnx_exporter.py b/examples/research_projects/onnx/summarization/run_onnx_exporter.py index 889eefb4e74b56..fa826732701f1a 100644 --- a/examples/research_projects/onnx/summarization/run_onnx_exporter.py +++ b/examples/research_projects/onnx/summarization/run_onnx_exporter.py @@ -13,9 +13,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" +""" """ -""" import argparse import logging import os diff --git a/examples/research_projects/performer/README.md b/examples/research_projects/performer/README.md index 42cb6fa358f95f..fa847268b0c8b3 100644 --- a/examples/research_projects/performer/README.md +++ b/examples/research_projects/performer/README.md @@ -10,8 +10,8 @@ Paper authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyo ## Examples -`sanity_script.sh` will launch performer fine-tuning from the bert-base-cased checkpoint on the Simple Wikipedia dataset (a small, easy-language English Wikipedia) from `datasets`. -`full_script.sh` will launch performer fine-tuning from the bert-large-cased checkpoint on the English Wikipedia dataset from `datasets`. +`sanity_script.sh` will launch performer fine-tuning from the google-bert/bert-base-cased checkpoint on the Simple Wikipedia dataset (a small, easy-language English Wikipedia) from `datasets`. +`full_script.sh` will launch performer fine-tuning from the google-bert/bert-large-cased checkpoint on the English Wikipedia dataset from `datasets`. Here are a few key arguments: - Remove the `--performer` argument to use a standard Bert model. diff --git a/examples/research_projects/performer/modeling_flax_performer_utils.py b/examples/research_projects/performer/modeling_flax_performer_utils.py index 6e6173729cc348..24c5e4d7c7fcec 100644 --- a/examples/research_projects/performer/modeling_flax_performer_utils.py +++ b/examples/research_projects/performer/modeling_flax_performer_utils.py @@ -284,7 +284,7 @@ def kernel_feature_creator( return attention_fn -class RandomMatrix(object): +class RandomMatrix: r""" Abstract class providing a method for constructing 2D random arrays. Class is responsible for constructing 2D random arrays. @@ -348,7 +348,7 @@ def get_2d_array(self): return jnp.matmul(jnp.diag(multiplier), final_matrix) -class FastAttention(object): +class FastAttention: r""" Abstract class providing a method for fast attention. Class is responsible for providing a method for fast approximate attention. diff --git a/examples/research_projects/performer/run_mlm_performer.py b/examples/research_projects/performer/run_mlm_performer.py index 1547ead421fd6f..0332fe1575ffe8 100644 --- a/examples/research_projects/performer/run_mlm_performer.py +++ b/examples/research_projects/performer/run_mlm_performer.py @@ -19,6 +19,7 @@ Here is the full list of checkpoints on the hub that can be fine-tuned by this script: https://huggingface.co/models?filter=fill-mask """ + import logging import os import sys @@ -99,7 +100,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -466,7 +467,7 @@ def generate_batch_splits(samples_idx: np.ndarray, batch_size: int) -> np.ndarra and not training_args.overwrite_output_dir ): raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty." + f"Output directory ({training_args.output_dir}) already exists and is not empty. " "Use --overwrite_output_dir to overcome." ) @@ -517,14 +518,15 @@ def generate_batch_splits(samples_idx: np.ndarray, batch_size: int) -> np.ndarra data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" datasets = load_dataset(extension, data_files=data_files) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer @@ -558,7 +560,7 @@ def generate_batch_splits(samples_idx: np.ndarray, batch_size: int) -> np.ndarra ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) diff --git a/examples/research_projects/pplm/requirements.txt b/examples/research_projects/pplm/requirements.txt index 70530cd79983a7..f93fde0f78f6e0 100644 --- a/examples/research_projects/pplm/requirements.txt +++ b/examples/research_projects/pplm/requirements.txt @@ -19,4 +19,4 @@ pytest conllu sentencepiece != 0.1.92 protobuf -transformers==3.5.1 +transformers==4.38.0 diff --git a/examples/research_projects/pplm/run_pplm.py b/examples/research_projects/pplm/run_pplm.py index 54008d56c14cba..cc49b7fa83c4c3 100644 --- a/examples/research_projects/pplm/run_pplm.py +++ b/examples/research_projects/pplm/run_pplm.py @@ -61,7 +61,7 @@ "embed_size": 1024, "class_vocab": {"non_clickbait": 0, "clickbait": 1}, "default_class": 1, - "pretrained_model": "gpt2-medium", + "pretrained_model": "openai-community/gpt2-medium", }, "sentiment": { "url": "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/discriminators/SST_classifier_head.pt", @@ -69,7 +69,7 @@ "embed_size": 1024, "class_vocab": {"very_positive": 2, "very_negative": 3}, "default_class": 3, - "pretrained_model": "gpt2-medium", + "pretrained_model": "openai-community/gpt2-medium", }, } @@ -585,7 +585,7 @@ def set_generic_model_params(discrim_weights, discrim_meta): def run_pplm_example( - pretrained_model="gpt2-medium", + pretrained_model="openai-community/gpt2-medium", cond_text="", uncond=False, num_samples=1, @@ -738,7 +738,7 @@ def run_pplm_example( "--pretrained_model", "-M", type=str, - default="gpt2-medium", + default="openai-community/gpt2-medium", help="pretrained model name or path to local checkpoint", ) parser.add_argument("--cond_text", type=str, default="The lake", help="Prefix texts to condition on") diff --git a/examples/research_projects/pplm/run_pplm_discrim_train.py b/examples/research_projects/pplm/run_pplm_discrim_train.py index d53b557d1af031..43ec5823e37764 100644 --- a/examples/research_projects/pplm/run_pplm_discrim_train.py +++ b/examples/research_projects/pplm/run_pplm_discrim_train.py @@ -45,7 +45,7 @@ class Discriminator(nn.Module): """Transformer encoder followed by a Classification Head""" - def __init__(self, class_size, pretrained_model="gpt2-medium", cached_mode=False, device="cpu"): + def __init__(self, class_size, pretrained_model="openai-community/gpt2-medium", cached_mode=False, device="cpu"): super().__init__() self.tokenizer = GPT2Tokenizer.from_pretrained(pretrained_model) self.encoder = GPT2LMHeadModel.from_pretrained(pretrained_model) @@ -218,7 +218,7 @@ def get_cached_data_loader(dataset, batch_size, discriminator, shuffle=False, de def train_discriminator( dataset, dataset_fp=None, - pretrained_model="gpt2-medium", + pretrained_model="openai-community/gpt2-medium", epochs=10, batch_size=64, log_interval=10, @@ -490,8 +490,8 @@ def train_discriminator( default="SST", choices=("SST", "clickbait", "toxic", "generic"), help=( - "dataset to train the discriminator on." - "In case of generic, the dataset is expected" + "dataset to train the discriminator on. " + "In case of generic, the dataset is expected " "to be a TSBV file with structure: class \\t text" ), ) @@ -502,7 +502,10 @@ def train_discriminator( help="File path of the dataset to use. Needed only in case of generic datadset", ) parser.add_argument( - "--pretrained_model", type=str, default="gpt2-medium", help="Pretrained model to use as encoder" + "--pretrained_model", + type=str, + default="openai-community/gpt2-medium", + help="Pretrained model to use as encoder", ) parser.add_argument("--epochs", type=int, default=10, metavar="N", help="Number of training epochs") parser.add_argument( diff --git a/examples/research_projects/quantization-qdqbert/README.md b/examples/research_projects/quantization-qdqbert/README.md index fe69819cc5be80..2cc2d5e5f98c71 100644 --- a/examples/research_projects/quantization-qdqbert/README.md +++ b/examples/research_projects/quantization-qdqbert/README.md @@ -30,17 +30,17 @@ Required: ## Setup the environment with Dockerfile Under the directory of `transformers/`, build the docker image: -``` +```bash docker build . -f examples/research_projects/quantization-qdqbert/Dockerfile -t bert_quantization:latest ``` Run the docker: -``` +```bash docker run --gpus all --privileged --rm -it --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 bert_quantization:latest ``` In the container: -``` +```bash cd transformers/examples/research_projects/quantization-qdqbert/ ``` @@ -48,21 +48,21 @@ cd transformers/examples/research_projects/quantization-qdqbert/ Calibrate the pretrained model and finetune with quantization awared: -``` +```bash python3 run_quant_qa.py \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --dataset_name squad \ --max_seq_length 128 \ --doc_stride 32 \ - --output_dir calib/bert-base-uncased \ + --output_dir calib/google-bert/bert-base-uncased \ --do_calib \ --calibrator percentile \ --percentile 99.99 ``` -``` +```bash python3 run_quant_qa.py \ - --model_name_or_path calib/bert-base-uncased \ + --model_name_or_path calib/google-bert/bert-base-uncased \ --dataset_name squad \ --do_train \ --do_eval \ @@ -71,8 +71,8 @@ python3 run_quant_qa.py \ --num_train_epochs 2 \ --max_seq_length 128 \ --doc_stride 32 \ - --output_dir finetuned_int8/bert-base-uncased \ - --tokenizer_name bert-base-uncased \ + --output_dir finetuned_int8/google-bert/bert-base-uncased \ + --tokenizer_name google-bert/bert-base-uncased \ --save_steps 0 ``` @@ -80,16 +80,16 @@ python3 run_quant_qa.py \ To export the QAT model finetuned above: -``` +```bash python3 run_quant_qa.py \ - --model_name_or_path finetuned_int8/bert-base-uncased \ + --model_name_or_path finetuned_int8/google-bert/bert-base-uncased \ --output_dir ./ \ --save_onnx \ --per_device_eval_batch_size 1 \ --max_seq_length 128 \ --doc_stride 32 \ --dataset_name squad \ - --tokenizer_name bert-base-uncased + --tokenizer_name google-bert/bert-base-uncased ``` Use `--recalibrate-weights` to calibrate the weight ranges according to the quantizer axis. Use `--quant-per-tensor` for per tensor quantization (default is per channel). @@ -97,19 +97,19 @@ Recalibrating will affect the accuracy of the model, but the change should be mi ### Benchmark the INT8 QAT ONNX model inference with TensorRT using dummy input -``` +```bash trtexec --onnx=model.onnx --explicitBatch --workspace=16384 --int8 --shapes=input_ids:64x128,attention_mask:64x128,token_type_ids:64x128 --verbose ``` ### Benchmark the INT8 QAT ONNX model inference with [ONNX Runtime-TRT](https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html) using dummy input -``` +```bash python3 ort-infer-benchmark.py ``` ### Evaluate the INT8 QAT ONNX model inference with TensorRT -``` +```bash python3 evaluate-hf-trt-qa.py \ --onnx_model_path=./model.onnx \ --output_dir ./ \ @@ -117,7 +117,7 @@ python3 evaluate-hf-trt-qa.py \ --max_seq_length 128 \ --doc_stride 32 \ --dataset_name squad \ - --tokenizer_name bert-base-uncased \ + --tokenizer_name google-bert/bert-base-uncased \ --int8 \ --seed 42 ``` @@ -126,16 +126,16 @@ python3 evaluate-hf-trt-qa.py \ Finetune a fp32 precision model with [transformers/examples/pytorch/question-answering/](../../pytorch/question-answering/): -``` +```bash python3 ../../pytorch/question-answering/run_qa.py \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --dataset_name squad \ --per_device_train_batch_size 12 \ --learning_rate 3e-5 \ --num_train_epochs 2 \ --max_seq_length 128 \ --doc_stride 32 \ - --output_dir ./finetuned_fp32/bert-base-uncased \ + --output_dir ./finetuned_fp32/google-bert/bert-base-uncased \ --save_steps 0 \ --do_train \ --do_eval @@ -145,15 +145,15 @@ python3 ../../pytorch/question-answering/run_qa.py \ ### PTQ by calibrating and evaluating the finetuned FP32 model above: -``` +```bash python3 run_quant_qa.py \ - --model_name_or_path ./finetuned_fp32/bert-base-uncased \ + --model_name_or_path ./finetuned_fp32/google-bert/bert-base-uncased \ --dataset_name squad \ --calibrator percentile \ --percentile 99.99 \ --max_seq_length 128 \ --doc_stride 32 \ - --output_dir ./calib/bert-base-uncased \ + --output_dir ./calib/google-bert/bert-base-uncased \ --save_steps 0 \ --do_calib \ --do_eval @@ -161,21 +161,21 @@ python3 run_quant_qa.py \ ### Export the INT8 PTQ model to ONNX -``` +```bash python3 run_quant_qa.py \ - --model_name_or_path ./calib/bert-base-uncased \ + --model_name_or_path ./calib/google-bert/bert-base-uncased \ --output_dir ./ \ --save_onnx \ --per_device_eval_batch_size 1 \ --max_seq_length 128 \ --doc_stride 32 \ --dataset_name squad \ - --tokenizer_name bert-base-uncased + --tokenizer_name google-bert/bert-base-uncased ``` ### Evaluate the INT8 PTQ ONNX model inference with TensorRT -``` +```bash python3 evaluate-hf-trt-qa.py \ --onnx_model_path=./model.onnx \ --output_dir ./ \ @@ -183,7 +183,7 @@ python3 evaluate-hf-trt-qa.py \ --max_seq_length 128 \ --doc_stride 32 \ --dataset_name squad \ - --tokenizer_name bert-base-uncased \ + --tokenizer_name google-bert/bert-base-uncased \ --int8 \ --seed 42 ``` diff --git a/examples/research_projects/quantization-qdqbert/evaluate-hf-trt-qa.py b/examples/research_projects/quantization-qdqbert/evaluate-hf-trt-qa.py index 814f95d0ab8f79..7a8ea2109bc53b 100755 --- a/examples/research_projects/quantization-qdqbert/evaluate-hf-trt-qa.py +++ b/examples/research_projects/quantization-qdqbert/evaluate-hf-trt-qa.py @@ -12,7 +12,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet).""" +"""Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet).""" + import argparse import logging import os @@ -153,7 +154,7 @@ tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_name, use_fast=True) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -272,10 +273,10 @@ def model_infer(inputs, context, d_inputs, h_output0, h_output1, d_output0, d_ou else: raise ValueError("Evaluation requires a dataset name") # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at -# https://huggingface.co/docs/datasets/loading_datasets.html. +# https://huggingface.co/docs/datasets/loading_datasets. # Preprocessing the datasets. -# Preprocessing is slighlty different for training and evaluation. +# Preprocessing is slightly different for training and evaluation. column_names = raw_datasets["validation"].column_names @@ -288,7 +289,7 @@ def model_infer(inputs, context, d_inputs, h_output0, h_output1, d_output0, d_ou if args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) diff --git a/examples/research_projects/quantization-qdqbert/quant_trainer.py b/examples/research_projects/quantization-qdqbert/quant_trainer.py index 73e880ebf67636..132aa284905875 100755 --- a/examples/research_projects/quantization-qdqbert/quant_trainer.py +++ b/examples/research_projects/quantization-qdqbert/quant_trainer.py @@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. """Helper functions for training models with pytorch-quantization""" + import logging import re @@ -239,7 +240,7 @@ def print_model_summary(model, name_width=25, line_width=180, ignore=None): continue if type(mod) in ignore: continue - if [True for s in ignore if type(s) is str and s in name]: + if [True for s in ignore if isinstance(s, str) and s in name]: continue act_str = f"Act:{input_q.extra_repr()}" wgt_str = f"Wgt:{weight_q.extra_repr()}" diff --git a/examples/research_projects/quantization-qdqbert/run_quant_qa.py b/examples/research_projects/quantization-qdqbert/run_quant_qa.py index ba5dfe4c090736..770a36525b5caa 100755 --- a/examples/research_projects/quantization-qdqbert/run_quant_qa.py +++ b/examples/research_projects/quantization-qdqbert/run_quant_qa.py @@ -308,7 +308,7 @@ def main(): extension = data_args.test_file.split(".")[-1] raw_datasets = load_dataset(extension, data_files=data_files, field="data", cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # set default quantization parameters before building model quant_trainer.set_default_quantizers(quant_trainer_args) @@ -322,14 +322,14 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=True, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) model = QDQBertForQuestionAnswering.from_pretrained( model_args.model_name_or_path, @@ -337,7 +337,7 @@ def main(): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # Tokenizer check: this script requires a fast tokenizer. @@ -349,7 +349,7 @@ def main(): ) # Preprocessing the datasets. - # Preprocessing is slighlty different for training and evaluation. + # Preprocessing is slightly different for training and evaluation. if training_args.do_train or model_args.do_calib: column_names = raw_datasets["train"].column_names elif training_args.do_eval or model_args.save_onnx: @@ -365,7 +365,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -448,7 +448,7 @@ def prepare_train_features(examples): raise ValueError("--do_train requires a train dataset") train_dataset = raw_datasets["train"] if data_args.max_train_samples is not None: - # We will select sample from whole data if agument is specified + # We will select sample from whole data if argument is specified max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) # Create train feature from dataset diff --git a/examples/research_projects/quantization-qdqbert/trainer_quant_qa.py b/examples/research_projects/quantization-qdqbert/trainer_quant_qa.py index 9b8c53b272b11b..a56d875354ddb0 100644 --- a/examples/research_projects/quantization-qdqbert/trainer_quant_qa.py +++ b/examples/research_projects/quantization-qdqbert/trainer_quant_qa.py @@ -24,13 +24,13 @@ import torch from torch.utils.data import DataLoader -from transformers import Trainer, is_torch_tpu_available +from transformers import Trainer, is_torch_xla_available from transformers.trainer_utils import PredictionOutput logger = logging.getLogger(__name__) -if is_torch_tpu_available(check_device=False): +if is_torch_xla_available(): import torch_xla.core.xla_model as xm import torch_xla.debug.metrics as met diff --git a/examples/research_projects/quantization-qdqbert/utils_qa.py b/examples/research_projects/quantization-qdqbert/utils_qa.py index fd0bc16f7e44cf..e90d6c4747c97c 100644 --- a/examples/research_projects/quantization-qdqbert/utils_qa.py +++ b/examples/research_projects/quantization-qdqbert/utils_qa.py @@ -15,6 +15,7 @@ """ Post-processing utilities for question answering. """ + import collections import json import logging diff --git a/examples/research_projects/rag-end2end-retriever/eval_rag.py b/examples/research_projects/rag-end2end-retriever/eval_rag.py index a8e7abbca6ce29..55f4da56571d3a 100644 --- a/examples/research_projects/rag-end2end-retriever/eval_rag.py +++ b/examples/research_projects/rag-end2end-retriever/eval_rag.py @@ -1,4 +1,4 @@ -""" Evaluation script for RAG models.""" +"""Evaluation script for RAG models.""" import argparse import ast diff --git a/examples/research_projects/rag-end2end-retriever/finetune_rag.py b/examples/research_projects/rag-end2end-retriever/finetune_rag.py index b0a6c1831907a0..9bc2e5db6d5d10 100644 --- a/examples/research_projects/rag-end2end-retriever/finetune_rag.py +++ b/examples/research_projects/rag-end2end-retriever/finetune_rag.py @@ -680,7 +680,7 @@ def add_ray_specific_args(parser): type=int, default=1, help=( - "The number of retrieval actors to use when Ray is selected" + "The number of retrieval actors to use when Ray is selected " "for the distributed retriever. Has no effect when " "distributed_retriever is set to pytorch." ), @@ -719,7 +719,7 @@ def main(args=None, model=None) -> GenerativeQAModule: ray.init(address=args.ray_address, namespace="rag") except (ConnectionError, ValueError): logger.warning( - "Connection to Ray cluster failed. Make sure a Ray" + "Connection to Ray cluster failed. Make sure a Ray " "cluster is running by either using Ray's cluster " "launcher (`ray up`) or by manually starting Ray on " "each node via `ray start --head` for the head node " diff --git a/examples/research_projects/rag-end2end-retriever/lightning_base.py b/examples/research_projects/rag-end2end-retriever/lightning_base.py index b9f8c6e3d7b5c0..9c918eea47b618 100644 --- a/examples/research_projects/rag-end2end-retriever/lightning_base.py +++ b/examples/research_projects/rag-end2end-retriever/lightning_base.py @@ -333,7 +333,7 @@ def add_generic_args(parser, root_dir) -> None: type=str, default="O2", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) @@ -410,5 +410,5 @@ def generic_train( trainer.fit(model) else: - print("RAG modeling tests with new set functions successfuly executed!") + print("RAG modeling tests with new set functions successfully executed!") return trainer diff --git a/examples/research_projects/rag-end2end-retriever/use_own_knowledge_dataset.py b/examples/research_projects/rag-end2end-retriever/use_own_knowledge_dataset.py index e0aa86a3a65ba9..20e0ea2d3cc2a2 100644 --- a/examples/research_projects/rag-end2end-retriever/use_own_knowledge_dataset.py +++ b/examples/research_projects/rag-end2end-retriever/use_own_knowledge_dataset.py @@ -65,7 +65,7 @@ def main( "csv", data_files=[rag_example_args.csv_path], split="train", delimiter="\t", column_names=["title", "text"] ) - # More info about loading csv files in the documentation: https://huggingface.co/docs/datasets/loading_datasets.html?highlight=csv#csv-files + # More info about loading csv files in the documentation: https://huggingface.co/docs/datasets/loading_datasets?highlight=csv#csv-files # Then split the documents into passages of 100 words dataset = dataset.map(split_documents, batched=True, num_proc=processing_args.num_proc) diff --git a/examples/research_projects/rag/README.md b/examples/research_projects/rag/README.md index 36c4a47841e560..7fbaea84b93782 100644 --- a/examples/research_projects/rag/README.md +++ b/examples/research_projects/rag/README.md @@ -17,7 +17,7 @@ Read more about RAG at https://arxiv.org/abs/2005.11401. # Finetuning -Our finetuning logic is based on scripts from [`examples/seq2seq`](https://github.com/huggingface/transformers/tree/main/examples/seq2seq). We accept training data in the same format as specified there - we expect a directory consisting of 6 text files: +Our finetuning logic is based on scripts from [`examples/legacy/seq2seq`](https://github.com/huggingface/transformers/tree/main/examples/legacy/seq2seq). We accept training data in the same format as specified there - we expect a directory consisting of 6 text files: ```bash train.source train.target @@ -45,7 +45,7 @@ We publish two `base` models which can serve as a starting point for finetuning The `base` models initialize the question encoder with [`facebook/dpr-question_encoder-single-nq-base`](https://huggingface.co/facebook/dpr-question_encoder-single-nq-base) and the generator with [`facebook/bart-large`](https://huggingface.co/facebook/bart-large). If you would like to initialize finetuning with a base model using different question encoder and generator architectures, you can build it with a consolidation script, e.g.: -``` +```bash python examples/research_projects/rag/consolidate_rag_checkpoint.py \ --model_type rag_sequence \ --generator_name_or_path facebook/bart-large-cnn \ diff --git a/examples/research_projects/rag/eval_rag.py b/examples/research_projects/rag/eval_rag.py index a8e7abbca6ce29..55f4da56571d3a 100644 --- a/examples/research_projects/rag/eval_rag.py +++ b/examples/research_projects/rag/eval_rag.py @@ -1,4 +1,4 @@ -""" Evaluation script for RAG models.""" +"""Evaluation script for RAG models.""" import argparse import ast diff --git a/examples/research_projects/rag/finetune_rag.py b/examples/research_projects/rag/finetune_rag.py index 64116a1d53d328..7f4778d7d71eeb 100644 --- a/examples/research_projects/rag/finetune_rag.py +++ b/examples/research_projects/rag/finetune_rag.py @@ -525,7 +525,7 @@ def add_ray_specific_args(parser): type=int, default=1, help=( - "The number of retrieval actors to use when Ray is selected" + "The number of retrieval actors to use when Ray is selected " "for the distributed retriever. Has no effect when " "distributed_retriever is set to pytorch." ), @@ -552,7 +552,7 @@ def main(args=None, model=None) -> GenerativeQAModule: ray.init(address=args.ray_address, namespace="rag") except (ConnectionError, ValueError): logger.warning( - "Connection to Ray cluster failed. Make sure a Ray" + "Connection to Ray cluster failed. Make sure a Ray " "cluster is running by either using Ray's cluster " "launcher (`ray up`) or by manually starting Ray on " "each node via `ray start --head` for the head node " diff --git a/examples/research_projects/rag/lightning_base.py b/examples/research_projects/rag/lightning_base.py index e78a7582395875..12099bc3aa106e 100644 --- a/examples/research_projects/rag/lightning_base.py +++ b/examples/research_projects/rag/lightning_base.py @@ -322,7 +322,7 @@ def add_generic_args(parser, root_dir) -> None: type=str, default="O2", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/research_projects/rag/use_own_knowledge_dataset.py b/examples/research_projects/rag/use_own_knowledge_dataset.py index 84d7c854975f11..d2ab6d07d5cc36 100644 --- a/examples/research_projects/rag/use_own_knowledge_dataset.py +++ b/examples/research_projects/rag/use_own_knowledge_dataset.py @@ -73,7 +73,7 @@ def main( "csv", data_files=[rag_example_args.csv_path], split="train", delimiter="\t", column_names=["title", "text"] ) - # More info about loading csv files in the documentation: https://huggingface.co/docs/datasets/loading_datasets.html?highlight=csv#csv-files + # More info about loading csv files in the documentation: https://huggingface.co/docs/datasets/loading_datasets?highlight=csv#csv-files # Then split the documents into passages of 100 words dataset = dataset.map(split_documents, batched=True, num_proc=processing_args.num_proc) diff --git a/examples/research_projects/robust-speech-event/README.md b/examples/research_projects/robust-speech-event/README.md index fd1a42c7d4bb58..ca3c5cdecdecea 100644 --- a/examples/research_projects/robust-speech-event/README.md +++ b/examples/research_projects/robust-speech-event/README.md @@ -3,7 +3,7 @@ Welcome to the robust speech recognition challenge 🎙️ ! The goal of this event is to build **robust**, **real-world** speech recognition (ASR) systems in as many languages as possible 🌏🌍🌎. -If necessary and available, free access to a V100S 32 GB GPU will kindly be provided by the [OVHcloud team]( https://www.ovhcloud.com/) 🚀. +If necessary and available, free access to a V100S 32 GB GPU will kindly be provided by the [OVHcloud team](https://www.ovhcloud.com/) 🚀. This document summarizes all the relevant information required for the speech community event 📋. To sign-up, please see [this forum post](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) 🤗. Please make sure to: @@ -112,7 +112,7 @@ Hugging Face Hub for additional audio data, for example by selecting the categor ["speech-processing"](https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads). All datasets that are available on the Hub can be downloaded via the 🤗 Datasets library in the same way Common Voice is downloaded. If one wants to combine multiple datasets for training, it might make sense to take a look at -the [`interleave_datasets`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=interleave#datasets.interleave_datasets) function. +the [`interleave_datasets`](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=interleave#datasets.interleave_datasets) function. In addition, participants can also make use of their audio data. Here, please make sure that you **are allowed to use the audio data**. E.g., if audio data is taken from media platforms, such as YouTube, it should be verified that the media platform and the owner of the data have given her/his approval to use the audio @@ -216,7 +216,7 @@ library from source to profit from the most current additions during the communi Simply run the following steps: -``` +```bash $ cd ~/ $ git clone https://github.com/huggingface/datasets.git $ cd datasets @@ -362,7 +362,7 @@ echo '''python run_speech_recognition_ctc.py \ --per_device_train_batch_size="2" \ --learning_rate="3e-4" \ --save_total_limit="1" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --text_column_name="sentence" \ --length_column_name="input_length" \ --save_steps="5" \ @@ -438,7 +438,7 @@ echo '''python run_speech_recognition_ctc.py \ --learning_rate="7.5e-5" \ --warmup_steps="2000" \ --length_column_name="input_length" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --text_column_name="sentence" \ --chars_to_ignore , ? . ! \- \; \: \" “ % ‘ ” � — ’ … – \ --save_steps="500" \ diff --git a/examples/research_projects/robust-speech-event/eval.py b/examples/research_projects/robust-speech-event/eval.py index a8acca1825d7da..b6c89a6d49fac9 100755 --- a/examples/research_projects/robust-speech-event/eval.py +++ b/examples/research_projects/robust-speech-event/eval.py @@ -65,7 +65,7 @@ def normalize_text(text: str) -> str: def main(args): # load dataset - dataset = load_dataset(args.dataset, args.config, split=args.split, use_auth_token=True) + dataset = load_dataset(args.dataset, args.config, split=args.split, token=True) # for testing: only process the first two examples as a test # dataset = dataset.select(range(10)) diff --git a/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_bnb.py b/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_bnb.py index abbe9a9982990b..0f7abde37ed66b 100755 --- a/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_bnb.py +++ b/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_bnb.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and -""" Fine-tuning a 🤗 Transformers CTC model for automatic speech recognition""" +"""Fine-tuning a 🤗 Transformers CTC model for automatic speech recognition""" import functools import json @@ -104,8 +104,8 @@ class ModelArguments: default=0.05, metadata={ "help": ( - "Probability of each feature vector along the time axis to be chosen as the start of the vector" - "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature" + "Probability of each feature vector along the time axis to be chosen as the start of the vector " + "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature " "vectors will be masked along the time axis." ) }, @@ -292,7 +292,7 @@ class DataCollatorCTCWithPadding: pad_to_multiple_of_labels: Optional[int] = None def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: - # split inputs and labels since they have to be of different lenghts and need + # split inputs and labels since they have to be of different lengths and need # different padding methods input_features = [{"input_values": feature["input_values"]} for feature in features] label_features = [{"input_ids": feature["labels"]} for feature in features] @@ -399,7 +399,7 @@ def main(): # Log on each process the small summary: logger.warning( - f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" + f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, " f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" ) # Set the verbosity to info of the Transformers logger (on main process only): @@ -418,7 +418,7 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, split=data_args.train_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) if data_args.audio_column_name not in raw_datasets["train"].column_names: @@ -443,7 +443,7 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, split=data_args.eval_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) if data_args.max_eval_samples is not None: @@ -481,7 +481,7 @@ def remove_special_characters(batch): # the tokenizer # load config config = AutoConfig.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) # 4. Next, if no tokenizer file is defined, @@ -532,11 +532,11 @@ def remove_special_characters(batch): # load feature_extractor and tokenizer tokenizer = AutoTokenizer.from_pretrained( tokenizer_name_or_path, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, **tokenizer_kwargs, ) feature_extractor = AutoFeatureExtractor.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) # adapt config @@ -564,7 +564,7 @@ def remove_special_characters(batch): model_args.model_name_or_path, cache_dir=model_args.cache_dir, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) # freeze encoder diff --git a/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py b/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py index 54338f15988154..0fb567aba053d4 100644 --- a/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py +++ b/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and -""" Fine-tuning a 🤗 Transformers CTC model for automatic speech recognition in streaming mode""" +"""Fine-tuning a 🤗 Transformers CTC model for automatic speech recognition in streaming mode""" import logging import os @@ -103,8 +103,8 @@ class ModelArguments: default=0.05, metadata={ "help": ( - "Probability of each feature vector along the time axis to be chosen as the start of the vector" - "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature" + "Probability of each feature vector along the time axis to be chosen as the start of the vector " + "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature " "vectors will be masked along the time axis." ) }, @@ -284,7 +284,7 @@ class DataCollatorCTCWithPadding: pad_to_multiple_of_labels: Optional[int] = None def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: - # split inputs and labels since they have to be of different lenghts and need + # split inputs and labels since they have to be of different lengths and need # different padding methods input_features = [] label_features = [] @@ -354,7 +354,7 @@ def main(): # Log on each process the small summary: logger.warning( - f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" + f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, " f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" ) # Set the verbosity to info of the Transformers logger (on main process only): @@ -395,7 +395,7 @@ def load_streaming_dataset(split, sampling_rate, **kwargs): # so we just need to set the correct target sampling rate and normalize the input # via the `feature_extractor` feature_extractor = AutoFeatureExtractor.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) if training_args.do_train: @@ -403,7 +403,7 @@ def load_streaming_dataset(split, sampling_rate, **kwargs): path=data_args.dataset_name, name=data_args.dataset_config_name, split=data_args.train_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, streaming=True, sampling_rate=feature_extractor.sampling_rate, ) @@ -431,7 +431,7 @@ def load_streaming_dataset(split, sampling_rate, **kwargs): path=data_args.dataset_name, name=data_args.dataset_config_name, split=data_args.eval_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, streaming=True, sampling_rate=feature_extractor.sampling_rate, ) @@ -465,7 +465,7 @@ def remove_special_characters(batch): # 3. Next, let's load the config as we might need it to create # the tokenizer config = AutoConfig.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) # 4. Now we can instantiate the tokenizer and model @@ -481,7 +481,7 @@ def remove_special_characters(batch): tokenizer = AutoTokenizer.from_pretrained( tokenizer_name_or_path, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) # adapt config @@ -509,7 +509,7 @@ def remove_special_characters(batch): model_args.model_name_or_path, cache_dir=model_args.cache_dir, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) # freeze encoder diff --git a/examples/research_projects/self-training-text-classification/README.md b/examples/research_projects/self-training-text-classification/README.md index 7e0f3f97148ee6..062d5de7afd057 100644 --- a/examples/research_projects/self-training-text-classification/README.md +++ b/examples/research_projects/self-training-text-classification/README.md @@ -51,7 +51,7 @@ parameters_dict = { 'train_file': os.path.join(data_dir, 'train.csv'), 'infer_file': os.path.join(data_dir, 'infer.csv'), 'eval_file': os.path.join(data_dir, 'eval.csv'), - 'evaluation_strategy': 'steps', + 'eval_strategy': 'steps', 'task_name': 'scitail', 'label_list': ['entails', 'neutral'], 'per_device_train_batch_size': 32, diff --git a/examples/research_projects/self-training-text-classification/finetuning.py b/examples/research_projects/self-training-text-classification/finetuning.py index eeb0a285dff987..0afff6a91eadca 100644 --- a/examples/research_projects/self-training-text-classification/finetuning.py +++ b/examples/research_projects/self-training-text-classification/finetuning.py @@ -190,7 +190,7 @@ class FTTrainingArguments: ) }, ) - evaluation_strategy: Optional[str] = dataclasses.field( + eval_strategy: Optional[str] = dataclasses.field( default="no", metadata={ "help": 'The evaluation strategy to adopt during training. Possible values are: ["no", "step", "epoch]' @@ -198,7 +198,7 @@ class FTTrainingArguments: ) eval_steps: Optional[int] = dataclasses.field( default=1, - metadata={"help": 'Number of update steps between two evaluations if `evaluation_strategy="steps"`.'}, + metadata={"help": 'Number of update steps between two evaluations if `eval_strategy="steps"`.'}, ) eval_metric: Optional[str] = dataclasses.field( default="accuracy", metadata={"help": "The evaluation metric used for the task."} @@ -265,7 +265,7 @@ def train(args, accelerator, model, tokenizer, train_dataloader, optimizer, lr_s # Evaluate during training if ( eval_dataloader is not None - and args.evaluation_strategy == IntervalStrategy.STEPS.value + and args.eval_strategy == IntervalStrategy.STEPS.value and args.eval_steps > 0 and completed_steps % args.eval_steps == 0 ): @@ -331,7 +331,7 @@ def train(args, accelerator, model, tokenizer, train_dataloader, optimizer, lr_s break # Evaluate during training - if eval_dataloader is not None and args.evaluation_strategy == IntervalStrategy.EPOCH.value: + if eval_dataloader is not None and args.eval_strategy == IntervalStrategy.EPOCH.value: accelerator.wait_for_everyone() new_checkpoint = f"checkpoint-{IntervalStrategy.EPOCH.value}-{epoch}" new_eval_result = evaluate(args, accelerator, eval_dataloader, "eval", model, new_checkpoint)[ @@ -571,7 +571,7 @@ def finetune(accelerator, model_name_or_path, train_file, output_dir, **kwargs): assert args.train_file is not None data_files[Split.TRAIN.value] = args.train_file - if args.do_eval or args.evaluation_strategy != IntervalStrategy.NO.value: + if args.do_eval or args.eval_strategy != IntervalStrategy.NO.value: assert args.eval_file is not None data_files[Split.EVAL.value] = args.eval_file diff --git a/examples/research_projects/self-training-text-classification/run.sh b/examples/research_projects/self-training-text-classification/run.sh index 435a41461801e6..34e91d7c127c89 100755 --- a/examples/research_projects/self-training-text-classification/run.sh +++ b/examples/research_projects/self-training-text-classification/run.sh @@ -60,7 +60,7 @@ parameters_dict = { 'train_file': os.path.join(data_dir, '${TRAIN_FILE}'), 'infer_file': os.path.join(data_dir, '${INFER_FILE}'), 'eval_file': os.path.join(data_dir, '${EVAL_FILE}'), - 'evaluation_strategy': 'steps', + 'eval_strategy': 'steps', 'task_name': 'scitail', 'label_list': ['entails', 'neutral'], 'per_device_train_batch_size': 32, diff --git a/examples/research_projects/self-training-text-classification/selftraining.py b/examples/research_projects/self-training-text-classification/selftraining.py index 70a6c2f319e0cb..d741225b061e88 100644 --- a/examples/research_projects/self-training-text-classification/selftraining.py +++ b/examples/research_projects/self-training-text-classification/selftraining.py @@ -79,7 +79,7 @@ class STTrainingArguments: eval_metric: Optional[str] = dataclasses.field( default="accuracy", metadata={"help": "The evaluation metric used for the task."} ) - evaluation_strategy: Optional[str] = dataclasses.field( + eval_strategy: Optional[str] = dataclasses.field( default="no", metadata={ "help": 'The evaluation strategy to adopt during training. Possible values are: ["no", "step", "epoch]' @@ -208,7 +208,7 @@ def selftrain(model_name_or_path, train_file, infer_file, output_dir, **kwargs): data_files["train"] = args.train_file data_files["infer"] = args.infer_file - if args.evaluation_strategy != IntervalStrategy.NO.value: + if args.eval_strategy != IntervalStrategy.NO.value: assert args.eval_file is not None data_files["eval"] = args.eval_file @@ -267,7 +267,7 @@ def selftrain(model_name_or_path, train_file, infer_file, output_dir, **kwargs): "label_list": args.label_list, "output_dir": current_output_dir, "eval_metric": args.eval_metric, - "evaluation_strategy": args.evaluation_strategy, + "eval_strategy": args.eval_strategy, "early_stopping_patience": args.early_stopping_patience, "early_stopping_threshold": args.early_stopping_threshold, "seed": args.seed, @@ -341,7 +341,7 @@ def selftrain(model_name_or_path, train_file, infer_file, output_dir, **kwargs): data_files["train_pseudo"] = os.path.join(next_data_dir, f"train_pseudo.{args.data_file_extension}") - if args.evaluation_strategy != IntervalStrategy.NO.value: + if args.eval_strategy != IntervalStrategy.NO.value: new_eval_result = eval_result if best_iteration is None: diff --git a/examples/research_projects/seq2seq-distillation/README.md b/examples/research_projects/seq2seq-distillation/README.md index 930e5b8fc98398..ab79a652ed38c3 100644 --- a/examples/research_projects/seq2seq-distillation/README.md +++ b/examples/research_projects/seq2seq-distillation/README.md @@ -239,7 +239,7 @@ For example, ./save_len_file.py Helsinki-NLP/opus-mt-en-ro wmt_en_ro ./dynamic_bs_example.sh --max_tokens_per_batch=2000 --output_dir benchmark_dynamic_bs ``` -splits `wmt_en_ro/train` into 11,197 uneven lengthed batches and can finish 1 epoch in 8 minutes on a v100. +splits `wmt_en_ro/train` into 11,197 uneven length batches and can finish 1 epoch in 8 minutes on a v100. For comparison, ```bash diff --git a/examples/research_projects/seq2seq-distillation/_test_seq2seq_examples.py b/examples/research_projects/seq2seq-distillation/_test_seq2seq_examples.py index 454951ed3888a0..0ee4dd8afe1d5e 100644 --- a/examples/research_projects/seq2seq-distillation/_test_seq2seq_examples.py +++ b/examples/research_projects/seq2seq-distillation/_test_seq2seq_examples.py @@ -418,7 +418,7 @@ def test_finetune_lr_schedulers(self): with CaptureStdout() as cs: args = parser.parse_args(args) assert False, "--help is expected to sys.exit" - assert excinfo.type == SystemExit + assert excinfo.type is SystemExit expected = lightning_base.arg_to_scheduler_metavar assert expected in cs.out, "--help is expected to list the supported schedulers" @@ -429,7 +429,7 @@ def test_finetune_lr_schedulers(self): with CaptureStderr() as cs: args = parser.parse_args(args) assert False, "invalid argument is expected to sys.exit" - assert excinfo.type == SystemExit + assert excinfo.type is SystemExit expected = f"invalid choice: '{unsupported_param}'" assert expected in cs.err, f"should have bailed on invalid choice of scheduler {unsupported_param}" diff --git a/examples/research_projects/seq2seq-distillation/lightning_base.py b/examples/research_projects/seq2seq-distillation/lightning_base.py index f246ecab0dd01b..640828bacd3401 100644 --- a/examples/research_projects/seq2seq-distillation/lightning_base.py +++ b/examples/research_projects/seq2seq-distillation/lightning_base.py @@ -313,7 +313,7 @@ def add_generic_args(parser, root_dir) -> None: type=str, default="O2", help=( - "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']." + "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. " "See details at https://nvidia.github.io/apex/amp.html" ), ) diff --git a/examples/research_projects/seq2seq-distillation/run_eval.py b/examples/research_projects/seq2seq-distillation/run_eval.py index 98c9786d2c95cd..54ad6c6fb6b637 100755 --- a/examples/research_projects/seq2seq-distillation/run_eval.py +++ b/examples/research_projects/seq2seq-distillation/run_eval.py @@ -94,7 +94,7 @@ def run_generate(verbose=True): parser.add_argument("--score_path", type=str, required=False, default="metrics.json", help="where to save metrics") parser.add_argument("--device", type=str, required=False, default=DEFAULT_DEVICE, help="cuda, cuda:1, cpu etc.") parser.add_argument( - "--prefix", type=str, required=False, default=None, help="will be added to the begininng of src examples" + "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples" ) parser.add_argument("--task", type=str, default="summarization", help="used for task_specific_params + metrics") parser.add_argument("--bs", type=int, default=8, required=False, help="batch size") diff --git a/examples/research_projects/tapex/README.md b/examples/research_projects/tapex/README.md index 7d98901e281e65..b98eb9b428d01c 100644 --- a/examples/research_projects/tapex/README.md +++ b/examples/research_projects/tapex/README.md @@ -71,7 +71,7 @@ python run_wikisql_with_tapex.py \ --eval_steps 1000 \ --save_steps 1000 \ --warmup_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate \ --num_beams 5 \ --weight_decay 1e-2 \ @@ -101,7 +101,7 @@ python run_wikisql_with_tapex.py \ --eval_steps 1000 \ --save_steps 1000 \ --warmup_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate \ --num_beams 5 \ --weight_decay 1e-2 \ @@ -132,7 +132,7 @@ python run_wikitablequestions_with_tapex.py \ --eval_steps 1000 \ --save_steps 1000 \ --warmup_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate \ --num_beams 5 \ --weight_decay 1e-2 \ @@ -162,7 +162,7 @@ python run_wikitablequestions_with_tapex.py \ --eval_steps 1000 \ --save_steps 1000 \ --warmup_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --predict_with_generate \ --num_beams 5 \ --weight_decay 1e-2 \ @@ -223,7 +223,7 @@ python run_tabfact_with_tapex.py \ --learning_rate 3e-5 \ --eval_steps 1000 \ --save_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --weight_decay 1e-2 \ --max_steps 30000 \ --max_grad_norm 0.1 @@ -252,7 +252,7 @@ python run_tabfact_with_tapex.py \ --learning_rate 3e-5 \ --eval_steps 1000 \ --save_steps 1000 \ - --evaluation_strategy steps \ + --eval_strategy steps \ --weight_decay 1e-2 \ --max_steps 30000 \ --max_grad_norm 0.1 diff --git a/examples/research_projects/tapex/run_tabfact_with_tapex.py b/examples/research_projects/tapex/run_tabfact_with_tapex.py index 23d094f8992a63..5dcec10a084c5f 100644 --- a/examples/research_projects/tapex/run_tabfact_with_tapex.py +++ b/examples/research_projects/tapex/run_tabfact_with_tapex.py @@ -277,7 +277,7 @@ def main(): # Loading a dataset from local json files raw_datasets = load_dataset("json", data_files=data_files, cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Labels label_list = raw_datasets["train"].features["label"].names @@ -292,7 +292,7 @@ def main(): num_labels=num_labels, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # load tapex tokenizer tokenizer = TapexTokenizer.from_pretrained( @@ -300,7 +300,7 @@ def main(): cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, add_prefix_space=True, ) model = BartForSequenceClassification.from_pretrained( @@ -309,7 +309,7 @@ def main(): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # Padding strategy @@ -325,7 +325,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) diff --git a/examples/research_projects/tapex/run_wikisql_with_tapex.py b/examples/research_projects/tapex/run_wikisql_with_tapex.py index a5717d245cb6c9..81e940a77c882c 100644 --- a/examples/research_projects/tapex/run_wikisql_with_tapex.py +++ b/examples/research_projects/tapex/run_wikisql_with_tapex.py @@ -170,7 +170,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used " "during ``evaluate`` and ``predict``." ) @@ -317,7 +317,7 @@ def main(): datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer # @@ -329,7 +329,7 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # IMPORTANT: the initial BART model's decoding is penalized by no_repeat_ngram_size, and thus @@ -344,7 +344,7 @@ def main(): cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, add_prefix_space=True, ) @@ -355,7 +355,7 @@ def main(): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) if model.config.decoder_start_token_id is None: @@ -379,7 +379,7 @@ def main(): if training_args.label_smoothing_factor > 0 and not hasattr(model, "prepare_decoder_input_ids_from_labels"): logger.warning( - "label_smoothing is enabled but the `prepare_decoder_input_ids_from_labels` method is not defined for" + "label_smoothing is enabled but the `prepare_decoder_input_ids_from_labels` method is not defined for " f"`{model.__class__.__name__}`. This will lead to loss being calculated twice and will take up more memory" ) diff --git a/examples/research_projects/tapex/run_wikitablequestions_with_tapex.py b/examples/research_projects/tapex/run_wikitablequestions_with_tapex.py index 901e921f26a694..55350025cb3bb4 100644 --- a/examples/research_projects/tapex/run_wikitablequestions_with_tapex.py +++ b/examples/research_projects/tapex/run_wikitablequestions_with_tapex.py @@ -168,7 +168,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used " "during ``evaluate`` and ``predict``." ) @@ -315,7 +315,7 @@ def main(): datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Load pretrained model and tokenizer # @@ -327,7 +327,7 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) # IMPORTANT: the initial BART model's decoding is penalized by no_repeat_ngram_size, and thus @@ -342,7 +342,7 @@ def main(): cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, add_prefix_space=True, ) @@ -353,7 +353,7 @@ def main(): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=True if model_args.use_auth_token else None, ) if model.config.decoder_start_token_id is None: @@ -377,7 +377,7 @@ def main(): if training_args.label_smoothing_factor > 0 and not hasattr(model, "prepare_decoder_input_ids_from_labels"): logger.warning( - "label_smoothing is enabled but the `prepare_decoder_input_ids_from_labels` method is not defined for" + "label_smoothing is enabled but the `prepare_decoder_input_ids_from_labels` method is not defined for " f"`{model.__class__.__name__}`. This will lead to loss being calculated twice and will take up more memory" ) diff --git a/examples/research_projects/tapex/wikisql_utils.py b/examples/research_projects/tapex/wikisql_utils.py index 110b14e02fb8e0..13d10e091a10c1 100644 --- a/examples/research_projects/tapex/wikisql_utils.py +++ b/examples/research_projects/tapex/wikisql_utils.py @@ -21,7 +21,7 @@ # The following script is adapted from the script of TaPas. # Original: https://github.com/google-research/tapas/master/wikisql_utils.py -from typing import Any, List, Text +from typing import Any, List EMPTY_ANSWER = "none" @@ -48,7 +48,7 @@ def convert_to_float(value): if isinstance(value, int): return float(value) if not isinstance(value, str): - raise ValueError("Argument value is not a string. Can't parse it as float") + raise TypeError("Argument value is not a string. Can't parse it as float") sanitized = value try: @@ -114,7 +114,7 @@ class _Operator(enum.Enum): class _Condition: """Represents an SQL where clauses (e.g A = "a" or B > 5).""" - column: Text + column: str operator: _Operator cmp_value: Any @@ -158,7 +158,7 @@ def _respect_conditions(table, row, conditions): cmp_value = _normalize_for_match(cmp_value) if not isinstance(table_value, type(cmp_value)): - raise ValueError("Type difference {} != {}".format(type(table_value), type(cmp_value))) + raise TypeError("Type difference {} != {}".format(type(table_value), type(cmp_value))) if not _compare(cond.operator, table_value, cmp_value): return False diff --git a/examples/research_projects/token-healing/README.md b/examples/research_projects/token-healing/README.md new file mode 100644 index 00000000000000..f3594f32dc7ad4 --- /dev/null +++ b/examples/research_projects/token-healing/README.md @@ -0,0 +1,40 @@ + + + + +## What is token healing? + +Token healing rectifies the token boundary bias in greedy tokenization. It does this by trimming and regrowing the prompt to better align with the model's tokenizer, thus enhancing generation quality. The improvement is clearest with completion models. + +Example: given a completion prompt with a partial url ending with `:`, the model might have seen the expected completion `://` as a _single_ token in training. However, the prompt's tail token `:` tells it that the next token is not `//`, and so it looks for wrong completions. Such errors compound in auto-regressive language models. + +Debiasing token boundaries also addresses output sensitivity to prompts ending with whitespace. + +A more thorough explanation can be found on [The Art of Prompt Design: Prompt Boundaries and Token Healing | by Scott Lundberg](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38). + +## Usage + +```py +prompt = 'The link is (back to top)

\ No newline at end of file diff --git a/examples/research_projects/token-healing/run_token_healing.py b/examples/research_projects/token-healing/run_token_healing.py new file mode 100644 index 00000000000000..2dd9148c1bcc58 --- /dev/null +++ b/examples/research_projects/token-healing/run_token_healing.py @@ -0,0 +1,62 @@ +import argparse + +from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig + + +def generate(inputs, model, tokenizer, token_healing): + input_ids = tokenizer(inputs, return_tensors="pt", padding=True, device_map="auto").input_ids + generation_config = GenerationConfig( + max_new_tokens=8, + token_healing=token_healing, + pad_token_id=model.config.pad_token_id, + repetition_penalty=1.1, + ) + output = model.generate(inputs=input_ids, generation_config=generation_config) + return tokenizer.batch_decode(output, skip_special_tokens=True) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--prompt", type=str) + parser.add_argument("--model_name_or_path", type=str, default="TheBloke/deepseek-llm-7B-base-GPTQ") + args = parser.parse_args() + + prompts = ( + [args.prompt] + if args.prompt + else [ + 'An example ["like this"] and another example [', + 'The link is https + "I read a book about ", # test trailing whitespace + "I read a book about", # test nothing to heal + ] + ) + + model_name_or_path = args.model_name_or_path + completion_model = AutoModelForCausalLM.from_pretrained( + model_name_or_path, + device_map="auto", + use_cache=True, + ) + tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) + + raw_output = generate(prompts, completion_model, tokenizer, token_healing=False) + healed_output = generate(prompts, completion_model, tokenizer, token_healing=True) + + for p, a, b in zip(prompts, raw_output, healed_output): + print(f"\nPrompt: {p}\nWithout healing:\n{a}\nWith healing:\n{b}") + + # You can also use token healing in isolation + # This can be useful if you have other work to do before the generation + # Or if you want to delegate generation to another process + input_ids = tokenizer(prompts, return_tensors="pt", padding=True).input_ids.cuda() + healed_ids = completion_model.heal_tokens(input_ids) + healed_prompts = tokenizer.batch_decode(healed_ids, skip_special_tokens=True) + print("\nhealed prompts:") + for p in healed_prompts: + print(p) + + +if __name__ == "__main__": + main() diff --git a/examples/research_projects/visual_bert/modeling_frcnn.py b/examples/research_projects/visual_bert/modeling_frcnn.py index edbd224cbe08d7..c7c3bf376ce382 100644 --- a/examples/research_projects/visual_bert/modeling_frcnn.py +++ b/examples/research_projects/visual_bert/modeling_frcnn.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 && Huggingface Co. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 && Huggingface Co. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" + import itertools import math import os @@ -416,7 +417,7 @@ def __new__(cls, *, channels=None, height=None, width=None, stride=None): return super().__new__(cls, channels, height, width, stride) -class Box2BoxTransform(object): +class Box2BoxTransform: """ This R-CNN transformation scales the box's width and height by exp(dw), exp(dh) and shifts a box's center by the offset @@ -518,7 +519,7 @@ def apply_deltas(self, deltas, boxes): return pred_boxes -class Matcher(object): +class Matcher: """ This class assigns to each predicted "element" (e.g., a box) a ground-truth element. Each predicted element will have exactly zero or one matches; each @@ -554,8 +555,8 @@ def __init__( assert thresholds[0] > 0 thresholds.insert(0, -float("inf")) thresholds.append(float("inf")) - assert all([low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:])]) - assert all([label_i in [-1, 0, 1] for label_i in labels]) + assert all(low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:])) + assert all(label_i in [-1, 0, 1] for label_i in labels) assert len(labels) == len(thresholds) - 1 self.thresholds = thresholds self.labels = labels @@ -621,7 +622,7 @@ def set_low_quality_matches_(self, match_labels, match_quality_matrix): match_labels[pred_inds_with_highest_quality] = 1 -class RPNOutputs(object): +class RPNOutputs: def __init__( self, box2box_transform, @@ -1131,7 +1132,7 @@ def forward(self, feature_maps, boxes): return output -class ROIOutputs(object): +class ROIOutputs: def __init__(self, cfg, training=False): self.smooth_l1_beta = cfg.ROI_BOX_HEAD.SMOOTH_L1_BETA self.box2box_transform = Box2BoxTransform(weights=cfg.ROI_BOX_HEAD.BBOX_REG_WEIGHTS) @@ -1706,9 +1707,7 @@ def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs): elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path): archive_file = pretrained_model_name_or_path elif os.path.isfile(pretrained_model_name_or_path + ".index"): - assert ( - from_tf - ), "We found a TensorFlow checkpoint at {}, please set from_tf to True to load from this checkpoint".format( + assert from_tf, "We found a TensorFlow checkpoint at {}, please set from_tf to True to load from this checkpoint".format( pretrained_model_name_or_path + ".index" ) archive_file = pretrained_model_name_or_path + ".index" diff --git a/examples/research_projects/visual_bert/processing_image.py b/examples/research_projects/visual_bert/processing_image.py index 4343cfdbce846e..65f8f6cd377c9f 100644 --- a/examples/research_projects/visual_bert/processing_image.py +++ b/examples/research_projects/visual_bert/processing_image.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" + import sys from typing import Tuple diff --git a/examples/research_projects/visual_bert/requirements.txt b/examples/research_projects/visual_bert/requirements.txt index 0d483b6d18923d..5e2076be9ddf1f 100644 --- a/examples/research_projects/visual_bert/requirements.txt +++ b/examples/research_projects/visual_bert/requirements.txt @@ -4,7 +4,7 @@ async-generator==1.10 attrs==20.2.0 backcall==0.2.0 CacheControl==0.12.6 -certifi==2022.12.7 +certifi==2024.7.4 cffi==1.14.2 chardet==3.0.4 click==7.1.2 @@ -21,7 +21,7 @@ entrypoints==0.3 filelock==3.0.12 future==0.18.3 html5lib==1.0.1 -idna==2.8 +idna==3.7 ipaddr==2.2.0 ipykernel==5.3.4 ipython @@ -34,7 +34,7 @@ jsonschema==3.2.0 jupyter==1.0.0 jupyter-client==6.1.7 jupyter-console==6.2.0 -jupyter-core==4.6.3 +jupyter-core==4.11.2 jupyterlab-pygments==0.1.1 kiwisolver==1.2.0 lockfile==0.12.2 @@ -62,7 +62,7 @@ prometheus-client==0.8.0 prompt-toolkit==3.0.7 ptyprocess==0.6.0 pyaml==20.4.0 -pyarrow==1.0.1 +pyarrow==15.0.0 pycparser==2.20 Pygments>=2.7.4 pyparsing==2.4.6 @@ -75,7 +75,7 @@ pyzmq==19.0.2 qtconsole==4.7.7 QtPy==1.9.0 regex==2020.7.14 -requests==2.22.0 +requests==2.32.2 retrying==1.3.3 sacremoses==0.0.43 Send2Trash==1.5.0 @@ -84,13 +84,13 @@ six==1.14.0 terminado==0.8.3 testpath==0.4.4 tokenizers==0.8.1rc2 -torch==1.6.0 +torch==1.13.1 torchvision==0.7.0 -tornado==6.0.4 -tqdm==4.48.2 +tornado==6.4.1 +tqdm==4.66.3 traitlets git+https://github.com/huggingface/transformers.git -urllib3==1.26.5 +urllib3==1.26.19 wcwidth==0.2.5 webencodings==0.5.1 wget==3.2 diff --git a/examples/research_projects/visual_bert/utils.py b/examples/research_projects/visual_bert/utils.py index 2fc6ea2062efd2..995fbd2c19aecf 100644 --- a/examples/research_projects/visual_bert/utils.py +++ b/examples/research_projects/visual_bert/utils.py @@ -1,20 +1,20 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal, Huggingface team :) - Adapted From Facebook Inc, Detectron2 +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal, Huggingface team :) +Adapted From Facebook Inc, Detectron2 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 + http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" import copy import fnmatch @@ -28,7 +28,6 @@ from collections import OrderedDict from contextlib import contextmanager from functools import partial -from hashlib import sha256 from io import BytesIO from pathlib import Path from urllib.parse import urlparse @@ -39,6 +38,7 @@ import requests import wget from filelock import FileLock +from huggingface_hub.utils import insecure_hashlib from PIL import Image from tqdm.auto import tqdm from yaml import Loader, dump, load @@ -402,12 +402,12 @@ def _resumable_file_manager(): def url_to_filename(url, etag=None): url_bytes = url.encode("utf-8") - url_hash = sha256(url_bytes) + url_hash = insecure_hashlib.sha256(url_bytes) filename = url_hash.hexdigest() if etag: etag_bytes = etag.encode("utf-8") - etag_hash = sha256(etag_bytes) + etag_hash = insecure_hashlib.sha256(etag_bytes) filename += "." + etag_hash.hexdigest() if url.endswith(".h5"): diff --git a/examples/research_projects/visual_bert/visualizing_image.py b/examples/research_projects/visual_bert/visualizing_image.py index 163d661e873ec3..dcfd8426ff4f36 100644 --- a/examples/research_projects/visual_bert/visualizing_image.py +++ b/examples/research_projects/visual_bert/visualizing_image.py @@ -1,20 +1,21 @@ """ - coding=utf-8 - Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal - Adapted From Facebook Inc, Detectron2 +coding=utf-8 +Copyright 2018, Antonio Mendoza Hao Tan, Mohit Bansal +Adapted From Facebook Inc, Detectron2 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.import copy +""" - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License.import copy - """ import colorsys import io diff --git a/examples/research_projects/vqgan-clip/README.md b/examples/research_projects/vqgan-clip/README.md index aef95093542208..a74bf9209b0a9a 100644 --- a/examples/research_projects/vqgan-clip/README.md +++ b/examples/research_projects/vqgan-clip/README.md @@ -21,7 +21,7 @@ To install locally: In the root of the repo run: -``` +```bash conda create -n vqganclip python=3.8 conda activate vqganclip git-lfs install @@ -30,7 +30,7 @@ pip install -r requirements.txt ``` ### Generate new images -``` +```python from VQGAN_CLIP import VQGAN_CLIP vqgan_clip = VQGAN_CLIP() vqgan_clip.generate("a picture of a smiling woman") @@ -41,7 +41,7 @@ To get a test image, run `git clone https://huggingface.co/datasets/erwann/vqgan-clip-pic test_images` To edit: -``` +```python from VQGAN_CLIP import VQGAN_CLIP vqgan_clip = VQGAN_CLIP() diff --git a/examples/research_projects/vqgan-clip/requirements.txt b/examples/research_projects/vqgan-clip/requirements.txt index 540bac904f29db..b97adf4140d3c9 100644 --- a/examples/research_projects/vqgan-clip/requirements.txt +++ b/examples/research_projects/vqgan-clip/requirements.txt @@ -21,7 +21,7 @@ taming-transformers torch torchvision tqdm -transformers==4.26.0 +transformers==4.38.0 tokenizers==0.13.2 typing_extensions wandb diff --git a/examples/research_projects/wav2vec2/FINE_TUNE_XLSR_WAV2VEC2.md b/examples/research_projects/wav2vec2/FINE_TUNE_XLSR_WAV2VEC2.md index d8a4e110873015..7a580a36132441 100644 --- a/examples/research_projects/wav2vec2/FINE_TUNE_XLSR_WAV2VEC2.md +++ b/examples/research_projects/wav2vec2/FINE_TUNE_XLSR_WAV2VEC2.md @@ -138,20 +138,20 @@ For bigger datasets, we recommend to train Wav2Vec2 locally instead of in a goog First, you need to clone the `transformers` repo with: -``` +```bash $ git clone https://github.com/huggingface/transformers.git ``` Second, head over to the `examples/research_projects/wav2vec2` directory, where the `run_common_voice.py` script is located. -``` +```bash $ cd transformers/examples/research_projects/wav2vec2 ``` Third, install the required packages. The packages are listed in the `requirements.txt` file and can be installed with -``` +```bash $ pip install -r requirements.txt ``` @@ -182,7 +182,7 @@ Here we will run the script on the *Turkish* Common Voice dataset for demonstrat --per_device_train_batch_size="16" \ --learning_rate="3e-4" \ --warmup_steps="500" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --save_steps="400" \ --eval_steps="400" \ --logging_steps="400" \ @@ -209,7 +209,7 @@ Here we will run the script on the *Turkish* Common Voice dataset for demonstrat --per_device_train_batch_size="16" \ --learning_rate="3e-4" \ --warmup_steps="500" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --save_steps="400" \ --eval_steps="400" \ --logging_steps="400" \ @@ -259,7 +259,7 @@ Then and add the following files that fully define a XLSR-Wav2Vec2 checkpoint in - `pytorch_model.bin` Having added the above files, you should run the following to push files to your model repository. -``` +```bash git add . && git commit -m "Add model files" && git push ``` diff --git a/examples/research_projects/wav2vec2/README.md b/examples/research_projects/wav2vec2/README.md index 1dcd8dcc283538..88f62778a3add9 100644 --- a/examples/research_projects/wav2vec2/README.md +++ b/examples/research_projects/wav2vec2/README.md @@ -18,7 +18,7 @@ python run_asr.py \ --num_train_epochs="30" \ --per_device_train_batch_size="20" \ --per_device_eval_batch_size="20" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_steps="500" \ --eval_steps="100" \ --logging_steps="50" \ @@ -73,7 +73,7 @@ python run_asr.py \ --per_device_train_batch_size="1" \ --per_device_eval_batch_size="1" \ --gradient_accumulation_steps="8" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_steps="500" \ --eval_steps="100" \ --logging_steps="50" \ @@ -134,7 +134,7 @@ which helps with capping GPU memory usage. To learn how to deploy Deepspeed Integration please refer to [this guide](https://huggingface.co/transformers/main/main_classes/deepspeed.html#deepspeed-trainer-integration). But to get started quickly all you need is to install: -``` +```bash pip install deepspeed ``` and then use the default configuration files in this directory: @@ -148,11 +148,11 @@ Here are examples of how you can use DeepSpeed: ZeRO-2: -``` +```bash PYTHONPATH=../../../src deepspeed --num_gpus 2 \ run_asr.py \ --output_dir=output_dir --num_train_epochs=2 --per_device_train_batch_size=2 \ ---per_device_eval_batch_size=2 --evaluation_strategy=steps --save_steps=500 --eval_steps=100 \ +--per_device_eval_batch_size=2 --eval_strategy=steps --save_steps=500 --eval_steps=100 \ --logging_steps=5 --learning_rate=5e-4 --warmup_steps=3000 \ --model_name_or_path=patrickvonplaten/wav2vec2_tiny_random_robust \ --dataset_name=hf-internal-testing/librispeech_asr_dummy --dataset_config_name=clean \ @@ -162,7 +162,7 @@ run_asr.py \ ``` For ZeRO-2 with more than 1 gpu you need to use (which is already in the example configuration file): -``` +```json "zero_optimization": { ... "find_unused_parameters": true, @@ -172,11 +172,11 @@ For ZeRO-2 with more than 1 gpu you need to use (which is already in the example ZeRO-3: -``` +```bash PYTHONPATH=../../../src deepspeed --num_gpus 2 \ run_asr.py \ --output_dir=output_dir --num_train_epochs=2 --per_device_train_batch_size=2 \ ---per_device_eval_batch_size=2 --evaluation_strategy=steps --save_steps=500 --eval_steps=100 \ +--per_device_eval_batch_size=2 --eval_strategy=steps --save_steps=500 --eval_steps=100 \ --logging_steps=5 --learning_rate=5e-4 --warmup_steps=3000 \ --model_name_or_path=patrickvonplaten/wav2vec2_tiny_random_robust \ --dataset_name=hf-internal-testing/librispeech_asr_dummy --dataset_config_name=clean \ @@ -192,7 +192,7 @@ It is recommended to pre-train Wav2Vec2 with Trainer + Deepspeed (please refer t Here is an example of how you can use DeepSpeed ZeRO-2 to pretrain a small Wav2Vec2 model: -``` +```bash PYTHONPATH=../../../src deepspeed --num_gpus 4 run_pretrain.py \ --output_dir="./wav2vec2-base-libri-100h" \ --num_train_epochs="3" \ @@ -238,7 +238,7 @@ Output directory will contain 0000.txt and 0001.txt. Each file will have format #### Run command -``` +```bash python alignment.py \ --model_name="arijitx/wav2vec2-xls-r-300m-bengali" \ --wav_dir="./wavs" diff --git a/examples/research_projects/wav2vec2/finetune_base_100.sh b/examples/research_projects/wav2vec2/finetune_base_100.sh index 8002dd81235f9e..254b0afef3d62e 100755 --- a/examples/research_projects/wav2vec2/finetune_base_100.sh +++ b/examples/research_projects/wav2vec2/finetune_base_100.sh @@ -4,7 +4,7 @@ python run_asr.py \ --num_train_epochs="30" \ --per_device_train_batch_size="32" \ --per_device_eval_batch_size="32" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_total_limit="3" \ --save_steps="500" \ --eval_steps="100" \ diff --git a/examples/research_projects/wav2vec2/finetune_base_timit_asr.sh b/examples/research_projects/wav2vec2/finetune_base_timit_asr.sh index 6219e26b642f63..508cb532b0f08d 100755 --- a/examples/research_projects/wav2vec2/finetune_base_timit_asr.sh +++ b/examples/research_projects/wav2vec2/finetune_base_timit_asr.sh @@ -4,7 +4,7 @@ python run_asr.py \ --num_train_epochs="30" \ --per_device_train_batch_size="20" \ --per_device_eval_batch_size="20" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_steps="500" \ --eval_steps="100" \ --logging_steps="50" \ diff --git a/examples/research_projects/wav2vec2/finetune_large_lv60_100.sh b/examples/research_projects/wav2vec2/finetune_large_lv60_100.sh index 3d2423df970c8e..6956b093e72530 100755 --- a/examples/research_projects/wav2vec2/finetune_large_lv60_100.sh +++ b/examples/research_projects/wav2vec2/finetune_large_lv60_100.sh @@ -4,7 +4,7 @@ python run_asr.py \ --num_train_epochs="30" \ --per_device_train_batch_size="16" \ --per_device_eval_batch_size="16" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_total_limit="3" \ --save_steps="500" \ --eval_steps="100" \ diff --git a/examples/research_projects/wav2vec2/finetune_large_lv60_timit_asr.sh b/examples/research_projects/wav2vec2/finetune_large_lv60_timit_asr.sh index eb9671d015271e..fa02e71ea82c68 100755 --- a/examples/research_projects/wav2vec2/finetune_large_lv60_timit_asr.sh +++ b/examples/research_projects/wav2vec2/finetune_large_lv60_timit_asr.sh @@ -5,7 +5,7 @@ python run_asr.py \ --per_device_train_batch_size="2" \ --per_device_eval_batch_size="2" \ --gradient_accumulation_steps="4" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_steps="500" \ --eval_steps="100" \ --logging_steps="50" \ diff --git a/examples/research_projects/wav2vec2/finetune_large_xlsr_53_arabic_speech_corpus.sh b/examples/research_projects/wav2vec2/finetune_large_xlsr_53_arabic_speech_corpus.sh index 9b325c42771e64..e90bc8caa6c001 100755 --- a/examples/research_projects/wav2vec2/finetune_large_xlsr_53_arabic_speech_corpus.sh +++ b/examples/research_projects/wav2vec2/finetune_large_xlsr_53_arabic_speech_corpus.sh @@ -5,7 +5,7 @@ python run_asr.py \ --per_device_train_batch_size="1" \ --per_device_eval_batch_size="1" \ --gradient_accumulation_steps="8" \ ---evaluation_strategy="steps" \ +--eval_strategy="steps" \ --save_steps="500" \ --eval_steps="100" \ --logging_steps="50" \ diff --git a/examples/research_projects/wav2vec2/finetune_wav2vec2_xlsr_turkish.sh b/examples/research_projects/wav2vec2/finetune_wav2vec2_xlsr_turkish.sh index 0726bb09eb51e2..70da0e0a0d1219 100644 --- a/examples/research_projects/wav2vec2/finetune_wav2vec2_xlsr_turkish.sh +++ b/examples/research_projects/wav2vec2/finetune_wav2vec2_xlsr_turkish.sh @@ -6,7 +6,7 @@ python run_common_voice.py \ --overwrite_output_dir \ --num_train_epochs="5" \ --per_device_train_batch_size="16" \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --learning_rate="3e-4" \ --warmup_steps="500" \ --fp16 \ diff --git a/examples/research_projects/wav2vec2/run_asr.py b/examples/research_projects/wav2vec2/run_asr.py index bfb9134943aca2..6535e3485d177e 100755 --- a/examples/research_projects/wav2vec2/run_asr.py +++ b/examples/research_projects/wav2vec2/run_asr.py @@ -254,7 +254,7 @@ class DataCollatorCTCWithPadding: pad_to_multiple_of_labels: Optional[int] = None def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: - # split inputs and labels since they have to be of different lenghts and need + # split inputs and labels since they have to be of different lengths and need # different padding methods input_features = [{"input_values": feature["input_values"]} for feature in features] label_features = [{"input_ids": feature["labels"]} for feature in features] diff --git a/examples/research_projects/wav2vec2/run_common_voice.py b/examples/research_projects/wav2vec2/run_common_voice.py index 01a877a8092ecf..a7f57960d89f2c 100644 --- a/examples/research_projects/wav2vec2/run_common_voice.py +++ b/examples/research_projects/wav2vec2/run_common_voice.py @@ -69,19 +69,19 @@ class ModelArguments: hidden_dropout: Optional[float] = field( default=0.1, metadata={ - "help": "The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler." + "help": "The dropout probability for all fully connected layers in the embeddings, encoder, and pooler." }, ) feat_proj_dropout: Optional[float] = field( default=0.1, - metadata={"help": "The dropout probabilitiy for all 1D convolutional layers in feature extractor."}, + metadata={"help": "The dropout probability for all 1D convolutional layers in feature extractor."}, ) mask_time_prob: Optional[float] = field( default=0.05, metadata={ "help": ( - "Propability of each feature vector along the time axis to be chosen as the start of the vector" - "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature" + "Propability of each feature vector along the time axis to be chosen as the start of the vector " + "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature " "vectors will be masked along the time axis. This is only relevant if ``apply_spec_augment is True``." ) }, @@ -173,7 +173,7 @@ class DataCollatorCTCWithPadding: pad_to_multiple_of_labels: Optional[int] = None def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: - # split inputs and labels since they have to be of different lenghts and need + # split inputs and labels since they have to be of different lengths and need # different padding methods input_features = [{"input_values": feature["input_values"]} for feature in features] label_features = [{"input_ids": feature["labels"]} for feature in features] diff --git a/examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py b/examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py index 0f3e239df6d205..8fb2df71112594 100644 --- a/examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py +++ b/examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py @@ -32,7 +32,7 @@ from parameterized import parameterized # noqa from transformers import TrainingArguments, is_torch_available # noqa -from transformers.deepspeed import is_deepspeed_available # noqa +from transformers.integrations.deepspeed import is_deepspeed_available # noqa from transformers.file_utils import WEIGHTS_NAME # noqa from transformers.testing_utils import ( # noqa CaptureLogger, @@ -161,7 +161,7 @@ def run_trainer( --num_train_epochs {str(num_train_epochs)} --per_device_train_batch_size 2 --per_device_eval_batch_size 2 - --evaluation_strategy steps + --eval_strategy steps --learning_rate 5e-4 --warmup_steps 8 --orthography timit diff --git a/examples/research_projects/xtreme-s/README.md b/examples/research_projects/xtreme-s/README.md index dc7e783c75d124..5314ba9880ad35 100644 --- a/examples/research_projects/xtreme-s/README.md +++ b/examples/research_projects/xtreme-s/README.md @@ -90,7 +90,7 @@ python -m torch.distributed.launch \ --gradient_accumulation_steps=2 \ --learning_rate="3e-4" \ --warmup_steps=3000 \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --max_duration_in_seconds=20 \ --save_steps=500 \ --eval_steps=500 \ @@ -134,7 +134,7 @@ python -m torch.distributed.launch \ --gradient_accumulation_steps=1 \ --learning_rate="3e-4" \ --warmup_steps=1500 \ - --evaluation_strategy="steps" \ + --eval_strategy="steps" \ --max_duration_in_seconds=30 \ --save_steps=200 \ --eval_steps=200 \ diff --git a/examples/research_projects/xtreme-s/run_xtreme_s.py b/examples/research_projects/xtreme-s/run_xtreme_s.py index 6c5b4bde892da1..a467b3c6eb8de6 100644 --- a/examples/research_projects/xtreme-s/run_xtreme_s.py +++ b/examples/research_projects/xtreme-s/run_xtreme_s.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and -""" Fine-tuning a 🤗 Transformers pretrained speech model on the XTREME-S benchmark tasks""" +"""Fine-tuning a 🤗 Transformers pretrained speech model on the XTREME-S benchmark tasks""" import json import logging @@ -116,8 +116,8 @@ class ModelArguments: default=0.05, metadata={ "help": ( - "Probability of each feature vector along the time axis to be chosen as the start of the vector" - "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature" + "Probability of each feature vector along the time axis to be chosen as the start of the vector " + "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature " "vectors will be masked along the time axis." ) }, @@ -335,7 +335,7 @@ class SpeechDataCollatorWithPadding: pad_to_multiple_of_labels: Optional[int] = None def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: - # split inputs and labels since they have to be of different lenghts and need + # split inputs and labels since they have to be of different lengths and need # different padding methods input_features = [{"input_values": feature["input_values"]} for feature in features] @@ -455,7 +455,7 @@ def main(): # Log on each process the small summary: logger.warning( - f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" + f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, " f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" ) # Set the verbosity to info of the Transformers logger (on main process only): @@ -502,7 +502,7 @@ def main(): data_args.dataset_name, config_name, split=data_args.train_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, cache_dir=model_args.cache_dir, ) @@ -528,7 +528,7 @@ def main(): data_args.dataset_name, config_name, split=data_args.eval_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, cache_dir=model_args.cache_dir, ) @@ -540,7 +540,7 @@ def main(): data_args.dataset_name, config_name, split=data_args.predict_split_name, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, cache_dir=model_args.cache_dir, ) @@ -595,7 +595,7 @@ def remove_special_characters(batch): # 3. Next, let's load the config as we might need it to create # the tokenizer config = AutoConfig.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) if is_text_target: @@ -651,11 +651,11 @@ def remove_special_characters(batch): if is_text_target: tokenizer = AutoTokenizer.from_pretrained( tokenizer_name_or_path, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, **tokenizer_kwargs, ) feature_extractor = AutoFeatureExtractor.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_auth_token=data_args.use_auth_token + model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token ) # adapt config @@ -694,14 +694,14 @@ def remove_special_characters(batch): model_args.model_name_or_path, cache_dir=model_args.cache_dir, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) elif config.is_encoder_decoder: model = AutoModelForSpeechSeq2Seq.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) if model.config.decoder_start_token_id is None: raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined") @@ -710,7 +710,7 @@ def remove_special_characters(batch): model_args.model_name_or_path, cache_dir=model_args.cache_dir, config=config, - use_auth_token=data_args.use_auth_token, + token=data_args.use_auth_token, ) # freeze encoder diff --git a/examples/research_projects/zero-shot-distillation/README.md b/examples/research_projects/zero-shot-distillation/README.md index cbc33071f0c9b4..14b6a8ea07f7ae 100644 --- a/examples/research_projects/zero-shot-distillation/README.md +++ b/examples/research_projects/zero-shot-distillation/README.md @@ -21,7 +21,7 @@ classification performance to the original zero-shot model A teacher NLI model can be distilled to a more efficient student model by running [`distill_classifier.py`](https://github.com/huggingface/transformers/blob/main/examples/research_projects/zero-shot-distillation/distill_classifier.py): -``` +```bash python distill_classifier.py \ --data_file \ --class_names_file \ diff --git a/examples/research_projects/zero-shot-distillation/distill_classifier.py b/examples/research_projects/zero-shot-distillation/distill_classifier.py index 3325c3aae0504c..56181208477767 100644 --- a/examples/research_projects/zero-shot-distillation/distill_classifier.py +++ b/examples/research_projects/zero-shot-distillation/distill_classifier.py @@ -41,7 +41,7 @@ class TeacherModelArguments: default="This example is {}.", metadata={ "help": ( - "Template used to turn class names into mock hypotheses for teacher NLI model. Must include {{}}" + "Template used to turn class names into mock hypotheses for teacher NLI model. Must include {{}} " "where class name is inserted." ) }, @@ -53,7 +53,7 @@ class TeacherModelArguments: default=False, metadata={ "help": ( - "Allow multiple classes to be true rather than forcing them to sum to 1 (sometimes called" + "Allow multiple classes to be true rather than forcing them to sum to 1 (sometimes called " "multi-class multi-label classification)." ) }, @@ -98,7 +98,7 @@ class DistillTrainingArguments(TrainingArguments): default=True, metadata={ "help": ( - "Whether to evaluate the agreement of the final student predictions and the teacher predictions" + "Whether to evaluate the agreement of the final student predictions and the teacher predictions " "after training." ) }, @@ -107,7 +107,7 @@ class DistillTrainingArguments(TrainingArguments): default=0, metadata={ "help": ( - "Limit the total amount of checkpoints." + "Limit the total amount of checkpoints. " "Deletes the older checkpoints in the output_dir. Default is 0 (no checkpoints)." ) }, diff --git a/examples/tensorflow/README.md b/examples/tensorflow/README.md index 7936e3d4650950..2c4115b369f75a 100644 --- a/examples/tensorflow/README.md +++ b/examples/tensorflow/README.md @@ -15,7 +15,7 @@ limitations under the License. # Examples -This folder contains actively maintained examples of use of 🤗 Transformers organized into different ML tasks. All examples in this folder are **TensorFlow** examples, and are written using native Keras rather than classes like `TFTrainer`, which we now consider deprecated. If you've previously only used 🤗 Transformers via `TFTrainer`, we highly recommend taking a look at the new style - we think it's a big improvement! +This folder contains actively maintained examples of the use of 🤗 Transformers organized into different ML tasks. All examples in this folder are **TensorFlow** examples and are written using native Keras. If you've previously only used 🤗 Transformers via `TFTrainer`, we highly recommend taking a look at the new style - we think it's a big improvement! In addition, all scripts here now support the [🤗 Datasets](https://github.com/huggingface/datasets) library - you can grab entire datasets just by changing one command-line argument! @@ -32,13 +32,13 @@ Here is the list of all our examples: | Task | Example datasets | |---|---| | [**`language-modeling`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling) | WikiText-2 -| [**`multiple-choice`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) | SWAG +| [**`multiple-choice`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) | SWAG | [**`question-answering`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) | SQuAD -| [**`summarization`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) | XSum +| [**`summarization`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) | XSum | [**`text-classification`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) | GLUE | [**`token-classification`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) | CoNLL NER | [**`translation`**](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) | WMT ## Coming soon -- **Colab notebooks** to easily run through these scripts! +- **Colab notebooks** to easily run through these scripts! diff --git a/examples/tensorflow/_tests_requirements.txt b/examples/tensorflow/_tests_requirements.txt index d44a342b4f328c..6971795ce4ea19 100644 --- a/examples/tensorflow/_tests_requirements.txt +++ b/examples/tensorflow/_tests_requirements.txt @@ -1,4 +1,5 @@ -tensorflow<2.13 +tensorflow<2.16 +keras<2.16 tensorboard scikit-learn seqeval @@ -15,7 +16,7 @@ nltk pandas datasets >= 1.13.3 fire -pytest +pytest<8.0.1 conllu sentencepiece != 0.1.92 protobuf diff --git a/examples/tensorflow/benchmarking/README.md b/examples/tensorflow/benchmarking/README.md index 7099ed9f6b3d3d..03e174770d1077 100644 --- a/examples/tensorflow/benchmarking/README.md +++ b/examples/tensorflow/benchmarking/README.md @@ -22,5 +22,5 @@ If you would like to list benchmark results on your favorite models of the [mode | Benchmark description | Results | Environment info | Author | |:----------|:-------------|:-------------|------:| -| PyTorch Benchmark on inference for `bert-base-cased` |[memory](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_memory.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | -| PyTorch Benchmark on inference for `bert-base-cased` |[time](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_time.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | +| PyTorch Benchmark on inference for `google-bert/bert-base-cased` |[memory](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_memory.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | +| PyTorch Benchmark on inference for `google-bert/bert-base-cased` |[time](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/inference_time.csv) | [env](https://github.com/patrickvonplaten/files_to_link_to/blob/master/bert_benchmark/env.csv) | [Partick von Platen](https://github.com/patrickvonplaten) | diff --git a/examples/tensorflow/benchmarking/plot_csv_file.py b/examples/tensorflow/benchmarking/plot_csv_file.py index 9a9ad9c670470e..aa092f5c047d44 100644 --- a/examples/tensorflow/benchmarking/plot_csv_file.py +++ b/examples/tensorflow/benchmarking/plot_csv_file.py @@ -93,14 +93,14 @@ def __init__(self, args): self.result_dict[model_name]["seq_len"].append(int(row["sequence_length"])) if can_convert_to_int(row["result"]): # value is not None - self.result_dict[model_name]["result"][ - (int(row["batch_size"]), int(row["sequence_length"])) - ] = int(row["result"]) + self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = ( + int(row["result"]) + ) elif can_convert_to_float(row["result"]): # value is not None - self.result_dict[model_name]["result"][ - (int(row["batch_size"]), int(row["sequence_length"])) - ] = float(row["result"]) + self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = ( + float(row["result"]) + ) def plot(self): fig, ax = plt.subplots() diff --git a/examples/tensorflow/benchmarking/run_benchmark_tf.py b/examples/tensorflow/benchmarking/run_benchmark_tf.py index 25aabc5f51c669..4a43cf7d91696e 100755 --- a/examples/tensorflow/benchmarking/run_benchmark_tf.py +++ b/examples/tensorflow/benchmarking/run_benchmark_tf.py @@ -14,7 +14,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Benchmarking the library on inference and training in TensorFlow""" +"""Benchmarking the library on inference and training in TensorFlow""" from transformers import HfArgumentParser, TensorFlowBenchmark, TensorFlowBenchmarkArguments diff --git a/examples/tensorflow/contrastive-image-text/README.md b/examples/tensorflow/contrastive-image-text/README.md index 9e3a011fcb33c4..29d9b897734cb2 100644 --- a/examples/tensorflow/contrastive-image-text/README.md +++ b/examples/tensorflow/contrastive-image-text/README.md @@ -65,7 +65,7 @@ Finally, we can run the example script to train the model: python examples/tensorflow/contrastive-image-text/run_clip.py \ --output_dir ./clip-roberta-finetuned \ --vision_model_name_or_path openai/clip-vit-base-patch32 \ - --text_model_name_or_path roberta-base \ + --text_model_name_or_path FacebookAI/roberta-base \ --data_dir $PWD/data \ --dataset_name ydshieh/coco_dataset_script \ --dataset_config_name=2017 \ diff --git a/examples/tensorflow/contrastive-image-text/run_clip.py b/examples/tensorflow/contrastive-image-text/run_clip.py index f8c44df15f9438..d629419950615b 100644 --- a/examples/tensorflow/contrastive-image-text/run_clip.py +++ b/examples/tensorflow/contrastive-image-text/run_clip.py @@ -51,7 +51,7 @@ logger = logging.getLogger(__name__) # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") require_version( "datasets>=1.8.0", "To fix: pip install -r examples/tensorflow/contrastive-image-text/requirements.txt" @@ -92,12 +92,22 @@ class ModelArguments: default=True, metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -186,9 +196,9 @@ def __post_init__(self): if self.validation_file is not None: extension = self.validation_file.split(".")[-1] assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file." - if self.validation_file is not None: - extension = self.validation_file.split(".")[-1] - assert extension == "json", "`validation_file` should be a json file." + if self.test_file is not None: + extension = self.test_file.split(".")[-1] + assert extension in ["csv", "json"], "`test_file` should be a csv or a json file." dataset_name_mapping = { @@ -285,7 +295,7 @@ def main(): # Log on each process the small summary: logger.info(f"Training/evaluation parameters {training_args}") - # 3. Detecting last checkpoint and eventualy continue from last checkpoint + # 3. Detecting last checkpoint and eventually continue from last checkpoint last_checkpoint = None if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir: if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: @@ -315,7 +325,8 @@ def main(): cache_dir=model_args.cache_dir, keep_in_memory=False, data_dir=data_args.data_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -332,27 +343,39 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # 5. Load pretrained model, tokenizer, and image processor if model_args.tokenizer_name: tokenizer = AutoTokenizer.from_pretrained( - model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer + model_args.tokenizer_name, + cache_dir=model_args.cache_dir, + use_fast=model_args.use_fast_tokenizer, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( - model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer + model_args.model_name_or_path, + cache_dir=model_args.cache_dir, + use_fast=model_args.use_fast_tokenizer, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) elif model_args.text_model_name_or_path: tokenizer = AutoTokenizer.from_pretrained( - model_args.text_model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer + model_args.text_model_name_or_path, + cache_dir=model_args.cache_dir, + use_fast=model_args.use_fast_tokenizer, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) @@ -362,14 +385,16 @@ def main(): model_args.image_processor_name or model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) with training_args.strategy.scope(): model = TFAutoModel.from_pretrained( model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: # Load image_processor, in this script we only use this to get the mean and std for normalization. @@ -377,14 +402,16 @@ def main(): model_args.image_processor_name or model_args.vision_model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) with training_args.strategy.scope(): model = TFVisionTextDualEncoderModel.from_vision_text_pretrained( vision_model_name_or_path=model_args.vision_model_name_or_path, text_model_name_or_path=model_args.text_model_name_or_path, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) config = model.config @@ -561,6 +588,8 @@ def filter_corrupt_images(examples): weight_decay_rate=training_args.weight_decay, adam_global_clipnorm=training_args.max_grad_norm, ) + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla) if not training_args.do_eval: diff --git a/examples/tensorflow/image-classification/README.md b/examples/tensorflow/image-classification/README.md index 28da5e894e1782..a343b443ef1ae5 100644 --- a/examples/tensorflow/image-classification/README.md +++ b/examples/tensorflow/image-classification/README.md @@ -45,7 +45,7 @@ python run_image_classification.py \ --per_device_eval_batch_size 8 \ --logging_strategy steps \ --logging_steps 10 \ - --evaluation_strategy epoch \ + --eval_strategy epoch \ --save_strategy epoch \ --load_best_model_at_end True \ --save_total_limit 3 \ @@ -107,10 +107,10 @@ from datasets import load_dataset # example 1: local folder dataset = load_dataset("imagefolder", data_dir="path_to_your_folder") -# example 2: local files (suppoted formats are tar, gzip, zip, xz, rar, zstd) +# example 2: local files (supported formats are tar, gzip, zip, xz, rar, zstd) dataset = load_dataset("imagefolder", data_files="path_to_zip_file") -# example 3: remote files (suppoted formats are tar, gzip, zip, xz, rar, zstd) +# example 3: remote files (supported formats are tar, gzip, zip, xz, rar, zstd) dataset = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip") # example 4: providing several splits diff --git a/examples/tensorflow/image-classification/run_image_classification.py b/examples/tensorflow/image-classification/run_image_classification.py index 74cbd71e6f5ccc..599b40c856f830 100644 --- a/examples/tensorflow/image-classification/run_image_classification.py +++ b/examples/tensorflow/image-classification/run_image_classification.py @@ -46,6 +46,7 @@ set_seed, ) from transformers.keras_callbacks import KerasMetricCallback +from transformers.modeling_tf_utils import keras from transformers.trainer_utils import get_last_checkpoint, is_main_process from transformers.utils import check_min_version, send_example_telemetry from transformers.utils.versions import require_version @@ -54,7 +55,7 @@ logger = logging.getLogger(__name__) # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") require_version("datasets>=1.8.0", "To fix: pip install -r examples/pytorch/image-classification/requirements.txt") @@ -157,12 +158,22 @@ class ModelArguments: metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) image_processor_name: str = field(default=None, metadata={"help": "Name or path of preprocessor config."}) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -226,9 +237,6 @@ def main(): else: model_args, data_args, training_args = parser.parse_args_into_dataclasses() - if not (training_args.do_train or training_args.do_eval or training_args.do_predict): - exit("Must specify at least one of --do_train, --do_eval or --do_predict!") - # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The # information sent is the one passed as arguments along with your Python/TensorFlow versions. send_example_telemetry("run_image_classification", model_args, data_args, framework="tensorflow") @@ -275,7 +283,8 @@ def main(): data_args.dataset_config_name, cache_dir=model_args.cache_dir, task="image-classification", - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -290,7 +299,7 @@ def main(): task="image-classification", ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # Prepare label mappings. # We'll include these in the model's config to get human readable labels in the Inference API. @@ -309,13 +318,15 @@ def main(): finetuning_task="image-classification", cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) image_processor = AutoImageProcessor.from_pretrained( model_args.image_processor_name or model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # If we don't have a validation split, split off a percentage of train as validation. @@ -335,7 +346,7 @@ def main(): def _train_transforms(image): img_size = image_size - image = tf.keras.utils.img_to_array(image) + image = keras.utils.img_to_array(image) image = random_resized_crop(image, size=img_size) image = tf.image.random_flip_left_right(image) image /= 255.0 @@ -344,7 +355,7 @@ def _train_transforms(image): return image def _val_transforms(image): - image = tf.keras.utils.img_to_array(image) + image = keras.utils.img_to_array(image) image = tf.image.resize(image, size=image_size) # image = np.array(image) # FIXME - use tf.image function image = center_crop(image, size=image_size) @@ -412,7 +423,7 @@ def val_transforms(example_batch): collate_fn = DefaultDataCollator(return_tensors="np") # Load the accuracy metric from the datasets package - metric = evaluate.load("accuracy") + metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir) # Define our compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a # predictions and label_ids field) and has to return a dictionary string to float. @@ -435,7 +446,8 @@ def compute_metrics(p): from_pt=bool(".bin" in model_path), cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ignore_mismatched_sizes=model_args.ignore_mismatched_sizes, ) num_replicas = training_args.strategy.num_replicas_in_sync @@ -479,7 +491,7 @@ def compute_metrics(p): collate_fn=collate_fn, ).with_options(dataset_options) else: - optimizer = None + optimizer = "sgd" # Just write anything because we won't be using it if training_args.do_eval: eval_dataset = model.prepare_tf_dataset( @@ -497,6 +509,8 @@ def compute_metrics(p): collate_fn=collate_fn, ).with_options(dataset_options) + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=["accuracy"]) push_to_hub_model_id = training_args.push_to_hub_model_id @@ -543,6 +557,7 @@ def compute_metrics(p): logging.info(f"{metric_name}: {value:.3f}") if training_args.output_dir is not None: + os.makedirs(training_args.output_dir, exist_ok=True) with open(os.path.join(training_args.output_dir, "all_results.json"), "w") as f: f.write(json.dumps(eval_metrics)) diff --git a/examples/tensorflow/language-modeling-tpu/prepare_tfrecord_shards.py b/examples/tensorflow/language-modeling-tpu/prepare_tfrecord_shards.py index a8bb7d37929f61..260f77226b1a30 100644 --- a/examples/tensorflow/language-modeling-tpu/prepare_tfrecord_shards.py +++ b/examples/tensorflow/language-modeling-tpu/prepare_tfrecord_shards.py @@ -42,6 +42,15 @@ def parse_args(): parser.add_argument( "--dataset_config", type=str, default="wikitext-103-raw-v1", help="Configuration name of the dataset." ) + parser.add_argument( + "--trust_remote_code", + action="store_true", + help=( + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." + ), + ) parser.add_argument( "--tokenizer_name_or_path", type=str, @@ -105,7 +114,9 @@ def get_serialized_examples(tokenized_data): def main(args): - dataset = datasets.load_dataset(args.dataset_name, args.dataset_config, split=args.split) + dataset = datasets.load_dataset( + args.dataset_name, args.dataset_config, split=args.split, trust_remote_code=args.trust_remote_code + ) if args.limit is not None: max_samples = min(len(dataset), args.limit) diff --git a/examples/tensorflow/language-modeling-tpu/requirements.txt b/examples/tensorflow/language-modeling-tpu/requirements.txt index 60bbe767a21427..47ec780c02def9 100644 --- a/examples/tensorflow/language-modeling-tpu/requirements.txt +++ b/examples/tensorflow/language-modeling-tpu/requirements.txt @@ -1,3 +1,3 @@ -transformers==4.26.1 +transformers==4.38.0 datasets==2.9.0 tokenizers==0.13.2 diff --git a/examples/tensorflow/language-modeling-tpu/run_mlm.py b/examples/tensorflow/language-modeling-tpu/run_mlm.py index 30923b982e1ed7..7ed111ab12712b 100644 --- a/examples/tensorflow/language-modeling-tpu/run_mlm.py +++ b/examples/tensorflow/language-modeling-tpu/run_mlm.py @@ -22,6 +22,7 @@ import re import tensorflow as tf +from packaging.version import parse from transformers import ( AutoConfig, @@ -33,6 +34,19 @@ ) +try: + import tf_keras as keras +except (ModuleNotFoundError, ImportError): + import keras + + if parse(keras.__version__).major > 2: + raise ValueError( + "Your currently installed version of Keras is Keras 3, but this is not yet supported in " + "Transformers. Please install the backwards-compatible tf-keras package with " + "`pip install tf-keras`." + ) + + logger = logging.getLogger(__name__) AUTO = tf.data.AUTOTUNE @@ -43,7 +57,7 @@ def parse_args(): parser.add_argument( "--pretrained_model_config", type=str, - default="roberta-base", + default="FacebookAI/roberta-base", help="The model config to use. Note that we don't copy the model's weights, only the config!", ) parser.add_argument( @@ -209,7 +223,7 @@ def main(args): strategy = tf.distribute.OneDeviceStrategy(device="/gpu:0") if args.bfloat16: - tf.keras.mixed_precision.set_global_policy("mixed_bfloat16") + keras.mixed_precision.set_global_policy("mixed_bfloat16") tokenizer = AutoTokenizer.from_pretrained(args.tokenizer) config = AutoConfig.from_pretrained(args.pretrained_model_config) @@ -235,8 +249,10 @@ def main(args): num_warmup_steps=total_train_steps // 20, init_lr=args.learning_rate, weight_decay_rate=args.weight_decay_rate, - # TODO Add the other Adam parameters? ) + + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, metrics=["accuracy"]) def decode_fn(example): diff --git a/examples/tensorflow/language-modeling-tpu/train_unigram.py b/examples/tensorflow/language-modeling-tpu/train_unigram.py index ea8246a99f3b08..615f93bc1bfb0c 100644 --- a/examples/tensorflow/language-modeling-tpu/train_unigram.py +++ b/examples/tensorflow/language-modeling-tpu/train_unigram.py @@ -41,6 +41,15 @@ def parse_args(): parser.add_argument( "--dataset_config", type=str, default="wikitext-103-raw-v1", help="Configuration name of the dataset." ) + parser.add_argument( + "--trust_remote_code", + action="store_true", + help=( + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." + ), + ) parser.add_argument( "--batch_size", type=int, @@ -69,7 +78,9 @@ def parse_args(): def main(args): - dataset = datasets.load_dataset(args.dataset_name, args.dataset_config, split="train") + dataset = datasets.load_dataset( + args.dataset_name, args.dataset_config, split="train", trust_remote_code=args.trust_remote_code + ) if args.limit is not None: max_train_samples = min(len(dataset), args.limit) @@ -109,7 +120,7 @@ def batch_iterator(): tokenizer.decoder = decoders.Metaspace() if args.export_to_hub: - logger.info("Exporting the trained tokenzier to Hub.") + logger.info("Exporting the trained tokenizer to Hub.") new_tokenizer = AlbertTokenizerFast(tokenizer_object=tokenizer) new_tokenizer.push_to_hub("unigram-tokenizer-dataset") diff --git a/examples/tensorflow/language-modeling/README.md b/examples/tensorflow/language-modeling/README.md index b96217c1f5da6d..ed4f507d4e82ce 100644 --- a/examples/tensorflow/language-modeling/README.md +++ b/examples/tensorflow/language-modeling/README.md @@ -41,18 +41,18 @@ can also be used by passing the name of the TPU resource with the `--tpu` argume This script trains a masked language model. ### Example command -``` +```bash python run_mlm.py \ ---model_name_or_path distilbert-base-cased \ +--model_name_or_path distilbert/distilbert-base-cased \ --output_dir output \ --dataset_name wikitext \ --dataset_config_name wikitext-103-raw-v1 ``` When using a custom dataset, the validation file can be separately passed as an input argument. Otherwise some split (customizable) of training data is used as validation. -``` +```bash python run_mlm.py \ ---model_name_or_path distilbert-base-cased \ +--model_name_or_path distilbert/distilbert-base-cased \ --output_dir output \ --train_file train_file_path ``` @@ -62,9 +62,9 @@ python run_mlm.py \ This script trains a causal language model. ### Example command -``` +```bash python run_clm.py \ ---model_name_or_path distilgpt2 \ +--model_name_or_path distilbert/distilgpt2 \ --output_dir output \ --dataset_name wikitext \ --dataset_config_name wikitext-103-raw-v1 @@ -72,9 +72,9 @@ python run_clm.py \ When using a custom dataset, the validation file can be separately passed as an input argument. Otherwise some split (customizable) of training data is used as validation. -``` +```bash python run_clm.py \ ---model_name_or_path distilgpt2 \ +--model_name_or_path distilbert/distilgpt2 \ --output_dir output \ --train_file train_file_path ``` diff --git a/examples/tensorflow/language-modeling/run_clm.py b/examples/tensorflow/language-modeling/run_clm.py index 645dae55be8691..00cfa6f7d245b4 100755 --- a/examples/tensorflow/language-modeling/run_clm.py +++ b/examples/tensorflow/language-modeling/run_clm.py @@ -77,7 +77,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -112,12 +112,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -287,7 +297,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) if "validation" not in raw_datasets.keys(): raw_datasets["validation"] = load_dataset( @@ -295,14 +306,16 @@ def main(): data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) raw_datasets["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -323,7 +336,7 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, **dataset_args, ) # If no validation data is there, validation_split_percentage will be used to divide the dataset. @@ -333,7 +346,7 @@ def main(): data_files=data_files, split=f"train[:{data_args.validation_split_percentage}%]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, **dataset_args, ) raw_datasets["train"] = load_dataset( @@ -341,11 +354,11 @@ def main(): data_files=data_files, split=f"train[{data_args.validation_split_percentage}%:]", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, **dataset_args, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Load pretrained model and tokenizer @@ -353,20 +366,30 @@ def main(): # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently # download model & vocab. if model_args.config_name: - config = AutoConfig.from_pretrained(model_args.config_name) + config = AutoConfig.from_pretrained( + model_args.config_name, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) elif model_args.model_name_or_path: - config = AutoConfig.from_pretrained(model_args.model_name_or_path) + config = AutoConfig.from_pretrained( + model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) else: config = CONFIG_MAPPING[model_args.model_type]() logger.warning("You are instantiating a new config instance from scratch.") if model_args.tokenizer_name: - tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name) + tokenizer = AutoTokenizer.from_pretrained( + model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.model_name_or_path: - tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) + tokenizer = AutoTokenizer.from_pretrained( + model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) # endregion @@ -390,16 +413,16 @@ def tokenize_function(examples): if data_args.block_size is None: block_size = tokenizer.model_max_length - if block_size > 1024: + if block_size > config.max_position_embeddings: logger.warning( f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " - "Picking 1024 instead. You can change that default value by passing --block_size xxx." + f"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx." ) - block_size = 1024 + block_size = min(1024, config.max_position_embeddings) else: if data_args.block_size > tokenizer.model_max_length: logger.warning( - f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" + f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model " f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." ) block_size = min(data_args.block_size, tokenizer.model_max_length) @@ -426,7 +449,7 @@ def group_texts(examples): # to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map lm_datasets = tokenized_datasets.map( group_texts, @@ -466,12 +489,21 @@ def group_texts(examples): with training_args.strategy.scope(): # region Prepare model if checkpoint is not None: - model = TFAutoModelForCausalLM.from_pretrained(checkpoint, config=config) + model = TFAutoModelForCausalLM.from_pretrained( + checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.model_name_or_path: - model = TFAutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, config=config) + model = TFAutoModelForCausalLM.from_pretrained( + model_args.model_name_or_path, + config=config, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) else: logger.info("Training new model from scratch") - model = TFAutoModelForCausalLM.from_config(config) + model = TFAutoModelForCausalLM.from_config( + config, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch # on a small vocab and want a smaller embedding size, remove this test. @@ -537,7 +569,8 @@ def group_texts(examples): adam_global_clipnorm=training_args.max_grad_norm, ) - # no user-specified loss = will use the model internal loss + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla) # endregion diff --git a/examples/tensorflow/language-modeling/run_mlm.py b/examples/tensorflow/language-modeling/run_mlm.py index cff0f51df09e1c..9e1cded9a31b77 100755 --- a/examples/tensorflow/language-modeling/run_mlm.py +++ b/examples/tensorflow/language-modeling/run_mlm.py @@ -75,7 +75,7 @@ class ModelArguments: default=None, metadata={ "help": ( - "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch." + "The model checkpoint for weights initialization. Don't set if you want to train a model from scratch." ) }, ) @@ -110,12 +110,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -296,38 +306,42 @@ def main(): raw_datasets = load_dataset( data_args.dataset_name, data_args.dataset_config_name, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) if "validation" not in raw_datasets.keys(): raw_datasets["validation"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[:{data_args.validation_split_percentage}%]", - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) raw_datasets["train"] = load_dataset( data_args.dataset_name, data_args.dataset_config_name, split=f"train[{data_args.validation_split_percentage}%:]", - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] if extension == "txt": extension = "text" raw_datasets = load_dataset( extension, data_files=data_files, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Load pretrained model and tokenizer @@ -335,22 +349,32 @@ def main(): # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently # download model & vocab. if checkpoint is not None: - config = AutoConfig.from_pretrained(checkpoint) + config = AutoConfig.from_pretrained( + checkpoint, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.config_name: - config = AutoConfig.from_pretrained(model_args.config_name) + config = AutoConfig.from_pretrained( + model_args.config_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.model_name_or_path: - config = AutoConfig.from_pretrained(model_args.model_name_or_path) + config = AutoConfig.from_pretrained( + model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) else: config = CONFIG_MAPPING[model_args.model_type]() logger.warning("You are instantiating a new config instance from scratch.") if model_args.tokenizer_name: - tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name) + tokenizer = AutoTokenizer.from_pretrained( + model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.model_name_or_path: - tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) + tokenizer = AutoTokenizer.from_pretrained( + model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) else: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) # endregion @@ -371,7 +395,7 @@ def main(): else: if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -441,7 +465,7 @@ def group_texts(examples): # might be slower to preprocess. # # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + # https://huggingface.co/docs/datasets/process#map tokenized_datasets = tokenized_datasets.map( group_texts, @@ -482,12 +506,21 @@ def group_texts(examples): with training_args.strategy.scope(): # region Prepare model if checkpoint is not None: - model = TFAutoModelForMaskedLM.from_pretrained(checkpoint, config=config) + model = TFAutoModelForMaskedLM.from_pretrained( + checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) elif model_args.model_name_or_path: - model = TFAutoModelForMaskedLM.from_pretrained(model_args.model_name_or_path, config=config) + model = TFAutoModelForMaskedLM.from_pretrained( + model_args.model_name_or_path, + config=config, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) else: logger.info("Training new model from scratch") - model = TFAutoModelForMaskedLM.from_config(config) + model = TFAutoModelForMaskedLM.from_config( + config, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch # on a small vocab and want a smaller embedding size, remove this test. @@ -559,8 +592,9 @@ def group_texts(examples): adam_global_clipnorm=training_args.max_grad_norm, ) - # no user-specified loss = will use the model internal loss - model.compile(optimizer=optimizer, jit_compile=training_args.xla, run_eagerly=True) + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). + model.compile(optimizer=optimizer, jit_compile=training_args.xla) # endregion # region Preparing push_to_hub and model card diff --git a/examples/tensorflow/multiple-choice/README.md b/examples/tensorflow/multiple-choice/README.md index 01e33fb62dbe23..a7f499963ec678 100644 --- a/examples/tensorflow/multiple-choice/README.md +++ b/examples/tensorflow/multiple-choice/README.md @@ -36,7 +36,7 @@ README, but for more information you can see the 'Input Datasets' section of ### Example command ```bash python run_swag.py \ - --model_name_or_path distilbert-base-cased \ + --model_name_or_path distilbert/distilbert-base-cased \ --output_dir output \ --do_eval \ --do_train diff --git a/examples/tensorflow/multiple-choice/run_swag.py b/examples/tensorflow/multiple-choice/run_swag.py index cd55ea7d651499..2bf164386cc50e 100644 --- a/examples/tensorflow/multiple-choice/run_swag.py +++ b/examples/tensorflow/multiple-choice/run_swag.py @@ -50,7 +50,7 @@ # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") logger = logging.getLogger(__name__) @@ -146,12 +146,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option " + "should only be set to `True` for repositories you trust and in which you have read the code, as it will " + "execute code present on the Hub on your local machine." ) }, ) @@ -294,14 +304,15 @@ def main(): data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] raw_datasets = load_dataset( extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: # Downloading and loading the swag dataset from the hub. @@ -309,10 +320,10 @@ def main(): "swag", "regular", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # When using your own dataset or a different dataset from swag, you will probably need to change this. ending_names = [f"ending{i}" for i in range(4)] @@ -335,14 +346,16 @@ def main(): config_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -358,7 +371,7 @@ def main(): else: if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -387,13 +400,12 @@ def preprocess_function(examples): if data_args.max_train_samples is not None: max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) - with training_args.main_process_first(desc="train dataset map pre-processing"): - train_dataset = train_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - load_from_cache_file=not data_args.overwrite_cache, - ) + train_dataset = train_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + load_from_cache_file=not data_args.overwrite_cache, + ) if training_args.do_eval: if "validation" not in raw_datasets: @@ -402,13 +414,12 @@ def preprocess_function(examples): if data_args.max_eval_samples is not None: max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples) eval_dataset = eval_dataset.select(range(max_eval_samples)) - with training_args.main_process_first(desc="validation dataset map pre-processing"): - eval_dataset = eval_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - load_from_cache_file=not data_args.overwrite_cache, - ) + eval_dataset = eval_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + load_from_cache_file=not data_args.overwrite_cache, + ) if data_args.pad_to_max_length: data_collator = DefaultDataCollator(return_tensors="np") @@ -428,7 +439,8 @@ def preprocess_function(examples): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) num_replicas = training_args.strategy.num_replicas_in_sync @@ -454,7 +466,9 @@ def preprocess_function(examples): adam_global_clipnorm=training_args.max_grad_norm, ) else: - optimizer = None + optimizer = "sgd" # Just write anything because we won't be using it + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, metrics=["accuracy"], jit_compile=training_args.xla) # endregion diff --git a/examples/tensorflow/question-answering/README.md b/examples/tensorflow/question-answering/README.md index b7c0443b1b079e..c7e85623199fbe 100644 --- a/examples/tensorflow/question-answering/README.md +++ b/examples/tensorflow/question-answering/README.md @@ -18,11 +18,12 @@ limitations under the License. This folder contains the `run_qa.py` script, demonstrating *question answering* with the 🤗 Transformers library. For straightforward use-cases you may be able to use this script without modification, although we have also -included comments in the code to indicate areas that you may need to adapt to your own projects. +included comments in the code to indicate areas that you may need to adapt to your own projects. ### Usage notes + Note that when contexts are long they may be split into multiple training cases, not all of which may contain -the answer span. +the answer span. As-is, the example script will train on SQuAD or any other question-answering dataset formatted the same way, and can handle user inputs as well. @@ -32,7 +33,7 @@ inputs as well. By default, the script uses a `MirroredStrategy` and will use multiple GPUs effectively if they are available. TPUs can also be used by passing the name of the TPU resource with the `--tpu` argument. There are some issues surrounding these strategies and our models right now, which are most likely to appear in the evaluation/prediction steps. We're -actively working on better support for multi-GPU and TPU training in TF, but if you encounter problems a quick +actively working on better support for multi-GPU and TPU training in TF, but if you encounter problems a quick workaround is to train in the multi-GPU or TPU context and then perform predictions outside of it. ### Memory usage and data loading @@ -40,16 +41,17 @@ workaround is to train in the multi-GPU or TPU context and then perform predicti One thing to note is that all data is loaded into memory in this script. Most question answering datasets are small enough that this is not an issue, but if you have a very large dataset you will need to modify the script to handle data streaming. This is particularly challenging for TPUs, given the stricter requirements and the sheer volume of data -required to keep them fed. A full explanation of all the possible pitfalls is a bit beyond this example script and -README, but for more information you can see the 'Input Datasets' section of +required to keep them fed. A full explanation of all the possible pitfalls is a bit beyond this example script and +README, but for more information you can see the 'Input Datasets' section of [this document](https://www.tensorflow.org/guide/tpu). ### Example command -``` + +```bash python run_qa.py \ ---model_name_or_path distilbert-base-cased \ +--model_name_or_path distilbert/distilbert-base-cased \ --output_dir output \ --dataset_name squad \ --do_train \ ---do_eval \ +--do_eval ``` diff --git a/examples/tensorflow/question-answering/run_qa.py b/examples/tensorflow/question-answering/run_qa.py index c3b7386132867b..14b50434e2dcfb 100755 --- a/examples/tensorflow/question-answering/run_qa.py +++ b/examples/tensorflow/question-answering/run_qa.py @@ -29,6 +29,7 @@ import evaluate import tensorflow as tf from datasets import load_dataset +from packaging.version import parse from utils_qa import postprocess_qa_predictions import transformers @@ -47,8 +48,21 @@ from transformers.utils import CONFIG_NAME, TF2_WEIGHTS_NAME, check_min_version, send_example_telemetry +try: + import tf_keras as keras +except (ModuleNotFoundError, ImportError): + import keras + + if parse(keras.__version__).major > 2: + raise ValueError( + "Your currently installed version of Keras is Keras 3, but this is not yet supported in " + "Transformers. Please install the backwards-compatible tf-keras package with " + "`pip install tf-keras`." + ) + + # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") logger = logging.getLogger(__name__) @@ -77,12 +91,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -216,7 +240,7 @@ def __post_init__(self): # region Helper classes -class SavePretrainedCallback(tf.keras.callbacks.Callback): +class SavePretrainedCallback(keras.callbacks.Callback): # Hugging Face models have a save_pretrained() method that saves both the weights and the necessary # metadata to allow them to be loaded as a pretrained model in future. This is a simple Keras callback # that saves the model with this method after each epoch. @@ -304,7 +328,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -323,10 +348,10 @@ def main(): data_files=data_files, field="data", cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Load pretrained model and tokenizer @@ -338,14 +363,16 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=True, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -375,7 +402,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -470,7 +497,7 @@ def prepare_train_features(examples): raise ValueError("--do_train requires a train dataset") train_dataset = datasets["train"] if data_args.max_train_samples is not None: - # We will select sample from whole data if agument is specified + # We will select sample from whole data if argument is specified max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) # Create train feature from dataset @@ -603,7 +630,9 @@ def post_processing_function(examples, features, predictions, stage="eval"): references = [{"id": ex["id"], "answers": ex[answer_column_name]} for ex in examples] return EvalPrediction(predictions=formatted_predictions, label_ids=references) - metric = evaluate.load("squad_v2" if data_args.version_2_with_negative else "squad") + metric = evaluate.load( + "squad_v2" if data_args.version_2_with_negative else "squad", cache_dir=model_args.cache_dir + ) def compute_metrics(p: EvalPrediction): return metric.compute(predictions=p.predictions, references=p.label_ids) @@ -625,7 +654,8 @@ def compute_metrics(p: EvalPrediction): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) if training_args.do_train: training_dataset = model.prepare_tf_dataset( @@ -656,11 +686,13 @@ def compute_metrics(p: EvalPrediction): adam_global_clipnorm=training_args.max_grad_norm, ) - # no user-specified loss = will use the model internal loss + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=["accuracy"]) else: - model.compile(optimizer=None, jit_compile=training_args.xla, metrics=["accuracy"]) + # Optimizer doesn't matter as it won't be used anyway + model.compile(optimizer="sgd", jit_compile=training_args.xla, metrics=["accuracy"]) training_dataset = None if training_args.do_eval: diff --git a/examples/tensorflow/question-answering/utils_qa.py b/examples/tensorflow/question-answering/utils_qa.py index 23a46370d17393..79497dbb816ed2 100644 --- a/examples/tensorflow/question-answering/utils_qa.py +++ b/examples/tensorflow/question-answering/utils_qa.py @@ -15,6 +15,7 @@ """ Post-processing utilities for question answering. """ + import collections import json import logging diff --git a/examples/tensorflow/summarization/run_summarization.py b/examples/tensorflow/summarization/run_summarization.py index 905a20ae312f62..c0d7ddfcd249b0 100644 --- a/examples/tensorflow/summarization/run_summarization.py +++ b/examples/tensorflow/summarization/run_summarization.py @@ -53,7 +53,7 @@ # region Checking dependencies # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") require_version("datasets>=1.8.0", "To fix: pip install -r examples/pytorch/summarization/requirements.txt") @@ -99,12 +99,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -177,7 +187,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used " "during ``evaluate`` and ``predict``." ) @@ -221,7 +231,7 @@ class DataTrainingArguments: }, ) num_beams: Optional[int] = field( - default=None, + default=1, metadata={ "help": ( "Number of beams to use for evaluation. This argument will be passed to ``model.generate``, " @@ -308,11 +318,11 @@ def main(): # region T5 special-casing if data_args.source_prefix is None and model_args.model_name_or_path in [ - "t5-small", - "t5-base", - "t5-large", - "t5-3b", - "t5-11b", + "google-t5/t5-small", + "google-t5/t5-base", + "google-t5/t5-large", + "google-t5/t5-3b", + "google-t5/t5-11b", ]: logger.warning( "You're running a t5 model but didn't provide a source prefix, which is the expected, e.g. with " @@ -355,7 +365,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -372,10 +383,10 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Load model config and tokenizer @@ -388,14 +399,16 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) prefix = data_args.source_prefix if data_args.source_prefix is not None else "" @@ -460,15 +473,14 @@ def preprocess_function(examples): if data_args.max_train_samples is not None: max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) - with training_args.main_process_first(desc="train dataset map pre-processing"): - train_dataset = train_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on train dataset", - ) + train_dataset = train_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on train dataset", + ) else: train_dataset = None @@ -480,15 +492,14 @@ def preprocess_function(examples): if data_args.max_eval_samples is not None: max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples) eval_dataset = eval_dataset.select(range(max_eval_samples)) - with training_args.main_process_first(desc="validation dataset map pre-processing"): - eval_dataset = eval_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on validation dataset", - ) + eval_dataset = eval_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on validation dataset", + ) else: eval_dataset = None # endregion @@ -513,7 +524,8 @@ def postprocess_text(preds, labels): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch @@ -594,13 +606,13 @@ def postprocess_text(preds, labels): adam_global_clipnorm=training_args.max_grad_norm, ) else: - optimizer = None + optimizer = "sgd" # Just write anything because we won't be using it # endregion # region Metric and KerasMetricCallback if training_args.do_eval: - metric = evaluate.load("rouge") + metric = evaluate.load("rouge", cache_dir=model_args.cache_dir) if data_args.val_max_target_length is None: data_args.val_max_target_length = data_args.max_target_length @@ -674,6 +686,8 @@ def compute_metrics(preds): # endregion # region Training + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla) eval_metrics = None if training_args.do_train: diff --git a/examples/tensorflow/test_tensorflow_examples.py b/examples/tensorflow/test_tensorflow_examples.py index 956209baade456..bbb8bfa3891206 100644 --- a/examples/tensorflow/test_tensorflow_examples.py +++ b/examples/tensorflow/test_tensorflow_examples.py @@ -23,6 +23,20 @@ from unittest.mock import patch import tensorflow as tf +from packaging.version import parse + + +try: + import tf_keras as keras +except (ModuleNotFoundError, ImportError): + import keras + + if parse(keras.__version__).major > 2: + raise ValueError( + "Your currently installed version of Keras is Keras 3, but this is not yet supported in " + "Transformers. Please install the backwards-compatible tf-keras package with " + "`pip install tf-keras`." + ) from transformers.testing_utils import TestCasePlus, get_gpu_count, slow @@ -93,7 +107,7 @@ def test_run_text_classification(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_text_classification.py - --model_name_or_path distilbert-base-uncased + --model_name_or_path distilbert/distilbert-base-uncased --output_dir {tmp_dir} --overwrite_output_dir --train_file ./tests/fixtures/tests_samples/MRPC/train.csv @@ -115,7 +129,7 @@ def test_run_text_classification(self): with patch.object(sys, "argv", testargs): run_text_classification.main() # Reset the mixed precision policy so we don't break other tests - tf.keras.mixed_precision.set_global_policy("float32") + keras.mixed_precision.set_global_policy("float32") result = get_results(tmp_dir) self.assertGreaterEqual(result["eval_accuracy"], 0.75) @@ -123,7 +137,7 @@ def test_run_clm(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_clm.py - --model_name_or_path distilgpt2 + --model_name_or_path distilbert/distilgpt2 --train_file ./tests/fixtures/sample_text.txt --validation_file ./tests/fixtures/sample_text.txt --do_train @@ -149,7 +163,7 @@ def test_run_mlm(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_mlm.py - --model_name_or_path distilroberta-base + --model_name_or_path distilbert/distilroberta-base --train_file ./tests/fixtures/sample_text.txt --validation_file ./tests/fixtures/sample_text.txt --max_seq_length 64 @@ -174,7 +188,7 @@ def test_run_ner(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_ner.py - --model_name_or_path bert-base-uncased + --model_name_or_path google-bert/bert-base-uncased --train_file tests/fixtures/tests_samples/conll/sample.json --validation_file tests/fixtures/tests_samples/conll/sample.json --output_dir {tmp_dir} @@ -198,7 +212,7 @@ def test_run_squad(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_qa.py - --model_name_or_path bert-base-uncased + --model_name_or_path google-bert/bert-base-uncased --version_2_with_negative --train_file tests/fixtures/tests_samples/SQUAD/sample.json --validation_file tests/fixtures/tests_samples/SQUAD/sample.json @@ -223,7 +237,7 @@ def test_run_swag(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_swag.py - --model_name_or_path bert-base-uncased + --model_name_or_path google-bert/bert-base-uncased --train_file tests/fixtures/tests_samples/swag/sample.json --validation_file tests/fixtures/tests_samples/swag/sample.json --output_dir {tmp_dir} @@ -247,7 +261,7 @@ def test_run_summarization(self): tmp_dir = self.get_auto_remove_tmp_dir() testargs = f""" run_summarization.py - --model_name_or_path t5-small + --model_name_or_path google-t5/t5-small --train_file tests/fixtures/tests_samples/xsum/sample.json --validation_file tests/fixtures/tests_samples/xsum/sample.json --output_dir {tmp_dir} @@ -302,6 +316,7 @@ def test_run_image_classification(self): testargs = f""" run_image_classification.py --dataset_name hf-internal-testing/cats_vs_dogs_sample + --trust_remote_code --model_name_or_path microsoft/resnet-18 --do_train --do_eval diff --git a/examples/tensorflow/text-classification/README.md b/examples/tensorflow/text-classification/README.md index 898cfa70145b26..08d0324b51dd94 100644 --- a/examples/tensorflow/text-classification/README.md +++ b/examples/tensorflow/text-classification/README.md @@ -36,7 +36,7 @@ may not always be what you want, especially if you have more than two fields! Here is a snippet of a valid input JSON file, though note that your texts can be much longer than these, and are not constrained (despite the field name) to being single grammatical sentences: -``` +```json {"sentence1": "COVID-19 vaccine updates: How is the rollout proceeding?", "label": "news"} {"sentence1": "Manchester United celebrates Europa League success", "label": "sports"} ``` @@ -69,13 +69,16 @@ README, but for more information you can see the 'Input Datasets' section of [this document](https://www.tensorflow.org/guide/tpu). ### Example command -``` +```bash python run_text_classification.py \ ---model_name_or_path distilbert-base-cased \ +--model_name_or_path distilbert/distilbert-base-cased \ --train_file training_data.json \ --validation_file validation_data.json \ --output_dir output/ \ ---test_file data_to_predict.json +--test_file data_to_predict.json \ +--do_train \ +--do_eval \ +--do_predict ``` ## run_glue.py @@ -101,9 +104,9 @@ README, but for more information you can see the 'Input Datasets' section of [this document](https://www.tensorflow.org/guide/tpu). ### Example command -``` +```bash python run_glue.py \ ---model_name_or_path distilbert-base-cased \ +--model_name_or_path distilbert/distilbert-base-cased \ --task_name mnli \ --do_train \ --do_eval \ diff --git a/examples/tensorflow/text-classification/run_glue.py b/examples/tensorflow/text-classification/run_glue.py index 45a0b3a9f1ac4c..73ce2ef2fec322 100644 --- a/examples/tensorflow/text-classification/run_glue.py +++ b/examples/tensorflow/text-classification/run_glue.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Finetuning the library models for sequence classification on GLUE.""" +"""Finetuning the library models for sequence classification on GLUE.""" # You can also adapt this script on your own text classification task. Pointers for this are left as comments. import json @@ -47,7 +47,7 @@ # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") task_to_keys = { "cola": ("sentence", None), @@ -164,12 +164,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option " + "should only be set to `True` for repositories you trust and in which you have read the code, as it will " + "execute code present on the Hub on your local machine." ) }, ) @@ -239,13 +249,13 @@ def main(): # Downloading and loading a dataset from the hub. In distributed training, the load_dataset function guarantee # that only one local process can concurrently download the dataset. datasets = load_dataset( - "glue", + "nyu-mll/glue", data_args.task_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. is_regression = data_args.task_name == "stsb" if not is_regression: @@ -284,14 +294,16 @@ def main(): finetuning_task=data_args.task_name, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -314,7 +326,7 @@ def main(): label_to_id = {i: int(label_name_to_id[label_list[i]]) for i in range(num_labels)} else: logger.warning( - "Your model seems to have been trained with labels, but they don't match the dataset: ", + "Your model seems to have been trained with labels, but they don't match the dataset: " f"model labels: {sorted(label_name_to_id.keys())}, dataset labels: {sorted(label_list)}." "\nIgnoring the model labels as a result.", ) @@ -328,7 +340,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -351,7 +363,7 @@ def preprocess_function(examples): # endregion # region Metric function - metric = evaluate.load("glue", data_args.task_name) + metric = evaluate.load("glue", data_args.task_name, cache_dir=model_args.cache_dir) def compute_metrics(preds, label_ids): preds = preds["logits"] @@ -374,7 +386,8 @@ def compute_metrics(preds, label_ids): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -448,11 +461,13 @@ def compute_metrics(preds, label_ids): adam_global_clipnorm=training_args.max_grad_norm, ) else: - optimizer = "adam" # Just write anything because we won't be using it + optimizer = "sgd" # Just write anything because we won't be using it if is_regression: metrics = [] else: metrics = ["accuracy"] + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, metrics=metrics, jit_compile=training_args.xla) # endregion diff --git a/examples/tensorflow/text-classification/run_text_classification.py b/examples/tensorflow/text-classification/run_text_classification.py index 64799eda3c0283..1aaa632cd78803 100644 --- a/examples/tensorflow/text-classification/run_text_classification.py +++ b/examples/tensorflow/text-classification/run_text_classification.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -""" Fine-tuning the library models for sequence classification.""" +"""Fine-tuning the library models for sequence classification.""" # You can also adapt this script on your own text classification task. Pointers for this are left as comments. import json @@ -26,6 +26,7 @@ import numpy as np from datasets import load_dataset +from packaging.version import parse from transformers import ( AutoConfig, @@ -45,11 +46,24 @@ import tensorflow as tf # noqa: E402 +try: + import tf_keras as keras +except (ModuleNotFoundError, ImportError): + import keras + + if parse(keras.__version__).major > 2: + raise ValueError( + "Your currently installed version of Keras is Keras 3, but this is not yet supported in " + "Transformers. Please install the backwards-compatible tf-keras package with " + "`pip install tf-keras`." + ) + + logger = logging.getLogger(__name__) # region Helper classes -class SavePretrainedCallback(tf.keras.callbacks.Callback): +class SavePretrainedCallback(keras.callbacks.Callback): # Hugging Face models have a save_pretrained() method that saves both the weights and the necessary # metadata to allow them to be loaded as a pretrained model in future. This is a simple Keras callback # that saves the model with this method after each epoch. @@ -100,7 +114,7 @@ class DataTrainingArguments: metadata={ "help": ( "Whether to pad all samples to `max_seq_length`. " - "If False, will pad the samples dynamically when batching to the maximum length in the batch." + "If False, will pad the samples dynamically when batching to the maximum length in the batch. " "Data will always be padded when using TPUs." ) }, @@ -170,12 +184,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option " + "should only be set to `True` for repositories you trust and in which you have read the code, as it will " + "execute code present on the Hub on your local machine." ) }, ) @@ -258,13 +282,13 @@ def main(): "csv", data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) else: # Loading a dataset from local json files datasets = load_dataset("json", data_files=data_files, cache_dir=model_args.cache_dir) # See more about loading any type of standard or custom dataset at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. # endregion # region Label preprocessing @@ -301,20 +325,23 @@ def main(): num_labels=num_labels, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: config = AutoConfig.from_pretrained( config_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -334,7 +361,7 @@ def main(): if data_args.max_seq_length > tokenizer.model_max_length: logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the " f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." ) max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) @@ -347,7 +374,7 @@ def main(): label_to_id = label_name_to_id # Use the model's labels else: logger.warning( - "Your model seems to have been trained with labels, but they don't match the dataset: ", + "Your model seems to have been trained with labels, but they don't match the dataset: " f"model labels: {sorted(label_name_to_id.keys())}, dataset labels:" f" {sorted(label_list)}.\nIgnoring the model labels as a result.", ) @@ -402,7 +429,8 @@ def preprocess_function(examples): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # endregion @@ -482,11 +510,13 @@ def preprocess_function(examples): adam_global_clipnorm=training_args.max_grad_norm, ) else: - optimizer = None + optimizer = "sgd" # Just use any default if is_regression: metrics = [] else: metrics = ["accuracy"] + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, metrics=metrics) # endregion diff --git a/examples/tensorflow/token-classification/README.md b/examples/tensorflow/token-classification/README.md index 0e5ec84528f8f2..6c8a15c00e813a 100644 --- a/examples/tensorflow/token-classification/README.md +++ b/examples/tensorflow/token-classification/README.md @@ -27,7 +27,7 @@ The following example fine-tunes BERT on CoNLL-2003: ```bash python run_ner.py \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --dataset_name conll2003 \ --output_dir /tmp/test-ner ``` @@ -36,7 +36,7 @@ To run on your own training and validation files, use the following command: ```bash python run_ner.py \ - --model_name_or_path bert-base-uncased \ + --model_name_or_path google-bert/bert-base-uncased \ --train_file path_to_train_file \ --validation_file path_to_validation_file \ --output_dir /tmp/test-ner diff --git a/examples/tensorflow/token-classification/run_ner.py b/examples/tensorflow/token-classification/run_ner.py index a358cd12dd63e7..19d153108b1d1f 100644 --- a/examples/tensorflow/token-classification/run_ner.py +++ b/examples/tensorflow/token-classification/run_ner.py @@ -75,12 +75,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -228,22 +238,24 @@ def main(): raw_datasets = load_dataset( data_args.dataset_name, data_args.dataset_config_name, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} if data_args.train_file is not None: data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] if data_args.validation_file is not None: data_files["validation"] = data_args.validation_file - extension = data_args.train_file.split(".")[-1] + extension = data_args.validation_file.split(".")[-1] raw_datasets = load_dataset( extension, data_files=data_files, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading_datasets. if raw_datasets["train"] is not None: column_names = raw_datasets["train"].column_names @@ -291,9 +303,19 @@ def get_label_list(labels): # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently # download model & vocab. if model_args.config_name: - config = AutoConfig.from_pretrained(model_args.config_name, num_labels=num_labels) + config = AutoConfig.from_pretrained( + model_args.config_name, + num_labels=num_labels, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) elif model_args.model_name_or_path: - config = AutoConfig.from_pretrained(model_args.model_name_or_path, num_labels=num_labels) + config = AutoConfig.from_pretrained( + model_args.model_name_or_path, + num_labels=num_labels, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) else: config = CONFIG_MAPPING[model_args.model_type]() logger.warning("You are instantiating a new config instance from scratch.") @@ -301,14 +323,25 @@ def get_label_list(labels): tokenizer_name_or_path = model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path if not tokenizer_name_or_path: raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You are instantiating a new tokenizer from scratch. This is not supported by this script. " "You can do it from another script, save it, and load it from here, using --tokenizer_name." ) if config.model_type in {"gpt2", "roberta"}: - tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path, use_fast=True, add_prefix_space=True) + tokenizer = AutoTokenizer.from_pretrained( + tokenizer_name_or_path, + use_fast=True, + add_prefix_space=True, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) else: - tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path, use_fast=True) + tokenizer = AutoTokenizer.from_pretrained( + tokenizer_name_or_path, + use_fast=True, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, + ) # endregion # region Preprocessing the raw datasets @@ -379,10 +412,14 @@ def tokenize_and_align_labels(examples): model = TFAutoModelForTokenClassification.from_pretrained( model_args.model_name_or_path, config=config, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: logger.info("Training new model from scratch") - model = TFAutoModelForTokenClassification.from_config(config) + model = TFAutoModelForTokenClassification.from_config( + config, token=model_args.token, trust_remote_code=model_args.trust_remote_code + ) # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch # on a small vocab and want a smaller embedding size, remove this test. @@ -454,12 +491,13 @@ def tokenize_and_align_labels(examples): weight_decay_rate=training_args.weight_decay, adam_global_clipnorm=training_args.max_grad_norm, ) - + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla) # endregion # Metrics - metric = evaluate.load("seqeval") + metric = evaluate.load("seqeval", cache_dir=model_args.cache_dir) def get_labels(y_pred, y_true): # Transform predictions and references tensos to numpy arrays diff --git a/examples/tensorflow/translation/README.md b/examples/tensorflow/translation/README.md index df5ee9c1ae36ba..bbe6e27e9c78a4 100644 --- a/examples/tensorflow/translation/README.md +++ b/examples/tensorflow/translation/README.md @@ -29,11 +29,11 @@ can also be used by passing the name of the TPU resource with the `--tpu` argume MBart and some T5 models require special handling. -T5 models `t5-small`, `t5-base`, `t5-large`, `t5-3b` and `t5-11b` must use an additional argument: `--source_prefix "translate {source_lang} to {target_lang}"`. For example: +T5 models `google-t5/t5-small`, `google-t5/t5-base`, `google-t5/t5-large`, `google-t5/t5-3b` and `google-t5/t5-11b` must use an additional argument: `--source_prefix "translate {source_lang} to {target_lang}"`. For example: ```bash python run_translation.py \ - --model_name_or_path t5-small \ + --model_name_or_path google-t5/t5-small \ --do_train \ --do_eval \ --source_lang en \ diff --git a/examples/tensorflow/translation/run_translation.py b/examples/tensorflow/translation/run_translation.py index 10112232f793a5..9804686e831301 100644 --- a/examples/tensorflow/translation/run_translation.py +++ b/examples/tensorflow/translation/run_translation.py @@ -56,7 +56,7 @@ # region Dependencies and constants # Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.29.0") +check_min_version("4.44.0") require_version("datasets>=1.8.0", "To fix: pip install -r examples/pytorch/summarization/requirements.txt") @@ -93,12 +93,22 @@ class ModelArguments: default="main", metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."}, ) - use_auth_token: bool = field( + token: str = field( + default=None, + metadata={ + "help": ( + "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token " + "generated when running `huggingface-cli login` (stored in `~/.huggingface`)." + ) + }, + ) + trust_remote_code: bool = field( default=False, metadata={ "help": ( - "Will use the token generated when running `huggingface-cli login` (necessary to use this script " - "with private models)." + "Whether to trust the execution of code from datasets/models defined on the Hub." + " This option should only be set to `True` for repositories you trust and in which you have read the" + " code, as it will execute code present on the Hub on your local machine." ) }, ) @@ -165,7 +175,7 @@ class DataTrainingArguments: metadata={ "help": ( "The maximum total sequence length for validation target text after tokenization. Sequences longer " - "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`." + "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`. " "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used " "during ``evaluate`` and ``predict``." ) @@ -209,7 +219,7 @@ class DataTrainingArguments: }, ) num_beams: Optional[int] = field( - default=None, + default=1, metadata={ "help": ( "Number of beams to use for evaluation. This argument will be passed to ``model.generate``, " @@ -322,7 +332,8 @@ def main(): data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) else: data_files = {} @@ -336,10 +347,10 @@ def main(): extension, data_files=data_files, cache_dir=model_args.cache_dir, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, ) # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. + # https://huggingface.co/docs/datasets/loading # endregion # region Load model config and tokenizer @@ -352,14 +363,16 @@ def main(): model_args.config_name if model_args.config_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) tokenizer = AutoTokenizer.from_pretrained( model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) prefix = data_args.source_prefix if data_args.source_prefix is not None else "" @@ -426,15 +439,14 @@ def preprocess_function(examples): if data_args.max_train_samples is not None: max_train_samples = min(len(train_dataset), data_args.max_train_samples) train_dataset = train_dataset.select(range(max_train_samples)) - with training_args.main_process_first(desc="train dataset map pre-processing"): - train_dataset = train_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on train dataset", - ) + train_dataset = train_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on train dataset", + ) else: train_dataset = None @@ -446,15 +458,14 @@ def preprocess_function(examples): if data_args.max_eval_samples is not None: max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples) eval_dataset = eval_dataset.select(range(max_eval_samples)) - with training_args.main_process_first(desc="validation dataset map pre-processing"): - eval_dataset = eval_dataset.map( - preprocess_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on validation dataset", - ) + eval_dataset = eval_dataset.map( + preprocess_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on validation dataset", + ) else: eval_dataset = None # endregion @@ -466,7 +477,8 @@ def preprocess_function(examples): config=config, cache_dir=model_args.cache_dir, revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, + token=model_args.token, + trust_remote_code=model_args.trust_remote_code, ) # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch @@ -557,12 +569,12 @@ def preprocess_function(examples): adam_global_clipnorm=training_args.max_grad_norm, ) else: - optimizer = None + optimizer = "sgd" # Just write anything because we won't be using it # endregion # region Metric and postprocessing if training_args.do_eval: - metric = evaluate.load("sacrebleu") + metric = evaluate.load("sacrebleu", cache_dir=model_args.cache_dir) if data_args.val_max_target_length is None: data_args.val_max_target_length = data_args.max_target_length @@ -643,6 +655,8 @@ def compute_metrics(preds): # region Training eval_metrics = None + # Transformers models compute the right loss for their task by default when labels are passed, and will + # use this for training unless you specify your own loss function in compile(). model.compile(optimizer=optimizer, jit_compile=training_args.xla) if training_args.do_train: