> For the complete documentation index, see [llms.txt](https://remi-calizzano.gitbook.io/benchmark-for-transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://remi-calizzano.gitbook.io/benchmark-for-transformers/master.md).

# Introduction

## Overview

The overall idea of benchmark-for-transformers is to simplify the evaluation and optimization process of transformers after their training.

You just have to define your benchmark and everything is then done automaticly. A benchmark is composed of scenarios, a dataset and metics. The different scenarios are evaluate on the dataset using the metrics. A scenario is a trained model with optimization features (batch size, quantization, onnx or specific parameters).

![Overview of how benchmark-for-transformers works](/files/-MG2O9QBnKPNKVvVFvkV)

## Available features

### Tasks

* Summarization
* NER
* Classification
* Ordering (can not be used yet)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#task) *to add a new task*

### Optimization features

* Batch size
* Quantization
* [ONNX](https://github.com/microsoft/onnxruntime) acceleration
* Task specific parameters

### Inference metrics

* Mean latency
* 90th pourcentile latency

### Datasets

* All datasets from [datasets](https://huggingface.co/docs/datasets/)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#dataset) *to add a new dataset*

### Metrics

* All metrics from [datasets](https://huggingface.co/docs/datasets/)
* *See section* [*Add custom features*](/benchmark-for-transformers/add-custom-features.md#metric) *to add a new metric*