> For the complete documentation index, see [llms.txt](https://remi-calizzano.gitbook.io/benchmark-for-transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://remi-calizzano.gitbook.io/benchmark-for-transformers/get-started.md).

# Get started

## Benchmark parameters

### Scenario

A scenario is composed of several parameters:

* `name` - name of the scenario
* `model_class` - name of the class to use (a class corresponds to a task) or path to a customized python script containing a model class
* `model_name` - name of the [Huggingface transformers](https://huggingface.co/transformers/) model to use. Every model from the [hub](https://huggingface.co/models) can be used
* `tokenizer_name` - name of the [Huggingface transformers](https://huggingface.co/transformers/) tokenizer to use
* `init_kwarg` - parameters to pass to the init function of the model
* `batch_size` - batch size to use during the evaluation
* `quantization` - if true, the model will be dynamically quantized
* `onnx` - if true, the model will be accelerate with [ONNX](https://github.com/microsoft/onnxruntime)

### Dataset

A dataset is composed of several parameters:

* `dataset_name` - name of a `nlp` dataset (from the [dataset hub](https://huggingface.co/datasets)) or path to a customized dataset script 
* `split` - split of the dataset. See the [`nlp` documentation](https://huggingface.co/nlp/loading_datasets.html#selecting-a-split) for more details
* `x_column_name` - list of column name of the dataset that contains the input
* `y_column_name` - column name of the dataset that contains the reference

### Metric

A metric is composed of several parameters:

* `metric_name` - name of a `nlp` metric (from the [dataset hub](https://huggingface.co/metrics)) or path to a customized metric script 
* `values` - values of the metric to extract
* `run_kwargs` - parameters to pass to the run function of the metric

## Create a benchmark

To create a benchmark, you can either define all the parameters in a json file and then load it using the API or directly use the API to define all the benchmark parameters.

### Using json file

You first have to create a json file containing all the parameters. For example, the following json file is a benchmark for summarization.

```
{
    "scenarios": [
        {
            "name": "distilbert",
            "model_class": "classification",
            "model_name": "distilbert-base-uncased-finetuned-sst-2-english",
            "tokenizer_name": "distilbert-base-uncased",
            "batch_size": 1,
            "device": "cuda"
        },
        {
            "name": "albert-base",
            "model_class": "classification",
            "model_name": "textattack/albert-base-v2-SST-2",
            "tokenizer_name": "textattack/albert-base-v2-SST-2",
            "batch_size": 1,
            "device": "cuda"
        },
        {
            "name": "bert base",
            "model_class": "classification",
            "model_name": "textattack/bert-base-uncased-SST-2",
            "batch_size": 1,
            "device": "cuda"
        }
    ],
    "dataset": {
        "dataset_name": "glue",
        "split": "validation",
        "x_column_name": ["sentence"],
        "y_column_name": "label",
        "init_kwargs": {"name": "sst2"}
    },
    "metrics": [
        {
            "metric_name": "glue",
            "values": ["accuracy"],
            "init_kwargs": {"config_name": "sst2"}
        }
    ]
}
```

Then, you can simply load the benchmark using the API and run it.

```python
from benchmark.benchmark import Benchmark

torch.set_num_threads(1)
benchmark = Benchmark.from_json("path/to/json/file")
df = benchmark.run()
print(df)
#  	# of parameters 	latency (mean) 	latency (90th percentile) 	glue_accuracy
# distilbert 	66955010 	0.006111 	0.007480 	0.910550
# albert-base 	11685122 	0.012642 	0.014657 	0.925459
# bert base 	109483778 	0.010371 	0.012245 	0.924312
```

{% hint style="info" %}
It is better to set the number of threads of PyTorch in order to be sure to always have the same.
{% endhint %}

### Using the API

First, you have to init the `Benchmark`by defining the dataset and the metric to use using `Benchmark.from_args`. Then, you can add several scenarios to the benchmark using `add_scenario`. Finally, you just have to `run`the benchmark.

```python
from benchmark_for_transformers import Benchmark
import torch

torch.set_num_threads(1)

# Set the dataset and the metric to use for the Benchmark
benchmark = Benchmark.from_args(
    dataset_name="xsum",
    dataset_split="test[:10]",
    x_column_name=["document"],
    y_column_name="summary",
    metric_name="rouge",
    metric_values=["rouge1", "rouge2", "rougeL"],
    metric_run_kwargs={"rouge_types": ["rouge1", "rouge2", "rougeL"]},
)

# Add a scenario
benchmark.reset_scenarios()
benchmark.add_scenario(
    name="Bart Xsum on cuda",
    model_class="summarization",
    model_name="facebook/bart-large-xsum",
    tokenizer_name="facebook/bart-large",
    init_kwargs={
        "generation_parameters": {
            "num_beams": 4,
            "length_penalty": 0.5,
            "min_length": 11,
            "max_length": 62
        }
    },
    batch_size=1,
    device="cuda"
)

df = benchmark.run()
print(df)
#  	# of parameters 	latency (mean) 	latency (90th percentile) 	rouge_rouge1 	rouge_rouge2 	rouge_rougeL
# Bart Xsum on cuda 	406290432 	0.850256 	0.941304 	0.376018 	0.118984 	0.274553
```