Performance of LLM backends and models in Curnagl

Introduction

This page shows performance of Llama and mistral models on Curnagl hardware. We have measured the token throughput which should help you to have an idea of what is possible using Curnagl resources. Training time and inference time for different task could be estimated using these results.

Models and backends tested

Tested Models

Llama3

• Official access to Meta Llama3 models: Meta Llama3 models on Hugging Face
• Meta-Llama-3.1-8B-Instruct
• Meta-Llama-3.1-70B-Instruct

Mistral

• Official access to Mistral models: Mistral models on MistralAI website
• Access to Mistral models on Hugging Face: Mistral models on Hugging Face
• mistral-7B-Instruct-v0.3
• Mixtral-8x7B-v0.1-Instruct

Tested Backends

• vLLM

vLLM backend provides efficient memory usage and fast token sampling. This backend is ideal for testing Llama3 and Mistral models in environments that require high-speed responses and low latency.

• llama.cpp

llama.cpp was primarily used for llama but it can be applied to other LLM models. This optimized backend provides efficient inference on GPUs.

• Transformers

If not the most widely used LLM black box, it is one of them. Easy to use, the Hugging Face Transformers library supports a wide range of models and backends. One of its main advantages is its quick set up, which enables quick experimentation across architectures.

• mistral-inference

This is the official inference backend for Mistral. It is (supposed to be) optimized for Mistral's architecture, thus increasing the model performance. However, our benchmarks results do not demonstrate any specificities to Mistral model as llama.cpp seems to perform better.

Hardware description

Three different types of GPUs have been used to benchmark LLM models:

• A100 which are available on Curnagl, official documentation,
• GH200 which will be available soon on Curnagl, official documentation,
• L40 which will be available soon on Curnagl, official documentation and specifications.

Here are their specifications

Depending on the code you are running, one GPU may better suit your requirements and expectations.

Note: These architectures are not powerful enough to train Large Language Models.

Note: Our benchmarks aim to determine which GPU types should be provided to researchers. If you require new GPUs for your research, feel free to reach out to us through the Help Desk. In case, you and other researchers agree on the same GPU request, we will do our best to provide new resources that meet your needs.

Inference latency results

This chat dataset from GPT3 has been used to benchmark models.

In order to guarantee reproduciblity of resultst and be able to perform a comparison between different bechmarks we set the following parameters:

• The maximum number of tokens to generate, is set to 400
• The temperature, which controls the output randomness, is set to 0
• The context size, which is the number of tokens the model can process within a single input, is set to default. This means the maximum context size of the model (e.g 131072 for Llama3.1)
• Use of GPU exclusively
• All models are loaded in F16 (no quantization)

Mistral models

mistral-7B-Instruct-v0.3

Mixtral-8x7B-v0.1-Instruct

Llama models

8B Instruct

70B Instruct

Conclusions

• Mixtral 8x7B and Llama 70B Instruct are composed of several billions of parameters. Therefore the resulting memory consumption for inference can only be supported by multiple GPUs using the same machine or by using a combination of VRAM and RAM host memory. This of course will degrade the performance because we need to transfert data between two types of memory which could be slow. GH200 has a large bus memory which offers a good performance on this types of cases.

• The use of distributed setup adds a lot of latency.

• Transformers backend offers a good trade-off between learning curve and performance.

• Banckends offer the possibility to configure a context size. The parameter has no impact on peformance (token throughput) but it is correlated to the amount of VRAM consumed. Therefore, if you want to optimize memory consumption you should set the context size to an appropiate value.

• GH200 offers the best inference speed but it could be difficult to set up and install libraries on.

• The results shown here were obtained without any optimization. There are optimization than can be applied like quantization and flash attention.