> For the complete documentation index, see [llms.txt](https://remi-calizzano.gitbook.io/benchmark-for-transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://remi-calizzano.gitbook.io/benchmark-for-transformers/master.md).

# Introduction

benchmark-for-transformers is a tool for easily comparing and evaluating the performance of transformers under different scenarios.

## Overview

The overall idea of benchmark-for-transformers is to simplify the evaluation and optimization process of transformers after their training.

You just have to define your benchmark and everything is then done automaticly. A benchmark is composed of scenarios, a dataset and metics. The different scenarios are evaluate on the dataset using the metrics. A scenario is a trained model with optimization features (batch size, quantization, onnx or specific parameters).

![Overview of how benchmark-for-transformers works](https://3181481886-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MG2KSPiIkeq16q8ISp0%2F-MG2LU7fop5qxnu_oL69%2F-MG2O9QBnKPNKVvVFvkV%2Ftestbed_overview.png?alt=media\&token=2c8b1771-dcd5-4f8f-965e-bdcf6934a94f)

## Available features

### Tasks

* Summarization
* NER
* Classification
* Ordering (can not be used yet)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#task) *to add a new task*

### Optimization features

* Batch size
* Quantization
* [ONNX](https://github.com/microsoft/onnxruntime) acceleration
* Task specific parameters

### Inference metrics

* Mean latency
* 90th pourcentile latency

### Datasets

* All datasets from [datasets](https://huggingface.co/docs/datasets/)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#dataset) *to add a new dataset*

### Metrics

* All metrics from [datasets](https://huggingface.co/docs/datasets/)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#metric) *to add a new metric*