Guide for Running Llama 2 Using LLAMA.CPP on AWS Fargate

Image generated with Stable Diffusion

Llama 2 is a new family of open-source large language models released by Meta (more on that here https://ai.meta.com/llama/) and which became a standard in the industry for using in cases with self-hosted LLM.

LLAMA.CPP is an open-source framework that is focused on running Llama models on CPU hardware (but can run on GPU as well). It democratized access to LLMs by enabling people to run large language models on their local computers.

One of the challenges with using LLM in production is finding the right way to host the models in the cloud. GPUs are usually expensive so a lot of developers are looking for ways to host the model using CPU hardware.

In this blog post, I will guide you through a quick and efficient deployment of the Llama 2 model on AWS with LLAMA.CPP framework utilizing a powerful tool from AWS, known as AWS Copilot. This is essentially a command-line interface designed specifically for containers, thereby simplifying the deployment and management of containerized applications including LLM ones.

The code for the demo is available on github. I have a similar blog post on the topic which covers using TGI framework by hugging face on Fargate.

Step-by-Step Deployment

1. Clone the Repository

First, clone the GitHub repository to your local machine:

git clone https://github.com/ryfeus/aws-inference-benchmark.git
cd copilot/llamacpp/aws-copilot-inference-service

This repository contains all the necessary code to deploy the deep learning model for Llama 2 inference.

2. Clone the model from HuggingFace

Next, we need to clone the HuggingFace repo with the model. I chose TheBloke/Llama-2–7B-GGML for this example since it has a good collection of quantized Llama 2 models, but other models could be used as well. Keep in mind the CPU limitation though. Repo has more than 10 different quantizations. In my case, I used llama-2-7b.ggmlv3.q5_K_M.bin based on the blog post which compares different quantized models

git lfs install
git clone https://huggingface.co/TheBloke/Llama-2-7B-GGML
mv Llama-2-7B-GGML/llama-2-7b.ggmlv3.q5_K_M.bin <path-to-repo>

3. Code in the repo

The main files in the code repo are app.py with business logic, Dockerfile with docker image implementation and manifest.yml which is used for configuring cloud infrastructure as well as requirements.txt file with python dependencies.

app.py — contains code for the transformers inference, health check, and web request management.

from aiohttp import web
import json
from llama_cpp import Llama

llm = Llama(model_path="model/llama-2-7b.ggmlv3.q5_K_M.bin")
routes = web.RouteTableDef()

@routes.get("/")
async def healthcheck(request):
    return web.Response()

@routes.post("/predict")
async def predict(request):
    try:
        req = await request.json()
        instruction = req.get("instruction", "What is the capital of Spain?")
        response = llm(instruction, max_tokens=32, stop=["Q:", "\n"], echo=True)
        resp = {"result": response}
        return web.Response(text=json.dumps(resp))
    except Exception as e:
        return web.Response(text=str(e), status=500)

def create_app():
    app = web.Application()
    app.add_routes(routes)
    return app

async def create_gunicorn_app():
    return create_app()

if __name__ == "__main__":
    web.run_app(create_app(), port=8080)

Dockerfile — is used for building a docker image which will be running on ECS cluster deployed by Copilot. We will be installing LLAMA.CPP framework with python wrapper llama-cpp-python so that we can easily use it in our python code.

FROM --platform=linux/amd64 python:3.11
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8080
CMD [ "python", "./app.py" ]

Copilot manifest — configuration file that defines different parameters of the cluster like CPU/Memory allocation, number of VMs, and available paths. When defining CPU/RAM combination for Fargate keep in mind capacity considerations aka which pairs are permitted. There is a good blog post on this topic which I recommend to check out. In short, CPU ranges from 0.25 vCPU to 4 vCPU and RAM ranges from 0.5 GB to 30 GB.

name: demo
type: Load Balanced Web Service
http:
  path: '/'
# Configuration for your containers and service.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
  build: copilot/aws-copilot-sample-service/Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 8080
cpu: 4096       # Number of CPU units for the task.
memory: 10240    # Amount of memory in MiB used by the task.
platform: linux/x86_64  # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1       # Number of tasks that should be running in your service.
exec: true     # Enable running commands in your container.
network:
  connect: true # Enable Service Connect for intra-environment traffic between services.

Next, initialize your AWS Copilot environment:

copilot env init

This command sets up the environment where your service will reside.

Get Rustem Feyzkhanov’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Now, you can deploy the application using:

copilot deploy

This command will take care of building the Docker image, pushing it to Amazon ECR, and deploying it to Amazon ECS/Fargate.

5. Test the Endpoint

After successful deployment, you can make a single prediction by making a POST request to your service. Replace <prefix> with the prefix of your endpoint:

curl -X POST -H "Content-Type: application/json" -d '{"instruction":"What are the main tour attractions in Rome?"}' http://<prefix>.us-east-1.elb.amazonaws.com/predict

The response should be:

{"result": "1. Colosseum 2. Pantheon 3. Roman Forum 4. Trevi Fountain 5. Vatican City 6. Roman Forum 7. Roman Forum 8. Roman Forum 9. Castel Sant'Angelo 10. Roman Forum"}

Cost

The cost of hosting the application would be ~170$ per month (us-west-2 region), which is still a lot for a pet project, but significantly cheaper than using GPU instances. The cost would come from two places:

• AWS Fargate cost — $0.04048 per vCPU-hour and $0.004445 per GB-hour. So with 4 vCPUs and 10 GB RAM that becomes: 4 vCPUs x $0.04048 x 24 hours x 30 days + 10 GB x $0.004445 x 24 hours x 30 days = $148.48
• ALB (Application Load Balancer) cost — hourly charge $0.0225 per hour + LCU cost — $0.008 LCU hours. Assuming we don’t have a lot of concurrent requests with 1 LCU the cost will be $0.0225 x 24 hours x 30 days + $0.008 x 24 hours x 30 days = $21.96

Resources

• Making LLMs Scalable: Cloud Inference with AWS Fargate and Copilot
• Introducing Llama 2
• 7 Frameworks for Serving LLMs
• Quantize Llama models with GGML and llama.cpp
• AWS ECS Fargate- Compute Capacity Planning

Conclusion

In this blog post, I covered how to deploy Llama 2 model on AWS. AWS Copilot simplifies the process of deploying your services, and AWS Fargate ensures that they run smoothly in a serverless environment. LLAMA.CPP makes it possible to use CPU for LLM and Llama 2 is the current open source standard. This stack is flexible and easy to manage, so we hope that this tutorial will help you in your journey of deploying scalable and robust LLMs to the cloud.