Shell Completion

LocalAI provides shell completion support for bash, zsh, and fish shells. Once installed, tab completion works for all CLI commands, subcommands, and flags.

Generating Completion Scripts

Use the completion subcommand to generate a completion script for your shell:

local-ai completion bash
local-ai completion zsh
local-ai completion fish

Installation

Bash

Add the following to your ~/.bashrc:

source <(local-ai completion bash)

Or install it system-wide:

local-ai completion bash > /etc/bash_completion.d/local-ai

Zsh

Add the following to your ~/.zshrc:

source <(local-ai completion zsh)

Or install it to a completions directory:

local-ai completion zsh > "${fpath[1]}/_local-ai"

If shell completions are not already enabled in your zsh environment, add the following to the beginning of your ~/.zshrc:

autoload -Uz compinit
compinit

Fish

local-ai completion fish | source

Or install it permanently:

local-ai completion fish > ~/.config/fish/completions/local-ai.fish

Usage

After installation, restart your shell or source your shell configuration file. Then type local-ai followed by a tab to see available commands:

$ local-ai <TAB>
run              backends         completion       explorer         models
federated        sound-generation transcript       tts              util

Tab completion also works for subcommands and flags:

$ local-ai models <TAB>
install  list

$ local-ai run --<TAB>
--address          --backends-path    --context-size     --debug            ...

System Info and Version

LocalAI provides endpoints to inspect the running instance, including available backends, loaded models, and version information.

System Information

• Method: GET
• Endpoint: /system

Returns available backends and currently loaded models.

Response

Usage

curl http://localhost:8080/system

Example response

{
  "backends": [
    "llama-cpp",
    "huggingface",
    "diffusers",
    "whisper"
  ],
  "loaded_models": [
    {
      "id": "my-llama-model"
    },
    {
      "id": "whisper-1"
    }
  ]
}

Version

• Method: GET
• Endpoint: /version

Returns the LocalAI version and build commit.

Response

Usage

curl http://localhost:8080/version

Example response

{
  "version": "2.26.0 (a1b2c3d4)"
}

Error Responses

Model compatibility table

Besides llama based models, LocalAI is compatible also with other architectures. The table below lists all the backends, compatible models families and the associated repository.

Text Generation & Language Models

Audio & Speech Processing

Image & Video Generation

Specialized AI Tasks

Acceleration Support Summary

GPU Acceleration

• NVIDIA CUDA: CUDA 12.0, CUDA 13.0 support across most backends
• AMD ROCm: HIP-based acceleration for AMD GPUs
• Intel oneAPI: SYCL-based acceleration for Intel GPUs (F16/F32 precision)
• Vulkan: Cross-platform GPU acceleration
• Metal: Apple Silicon GPU acceleration (M1/M2/M3+)

Specialized Hardware

• NVIDIA Jetson (L4T CUDA 12): ARM64 support for embedded AI (AGX Orin, Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
• NVIDIA Jetson (L4T CUDA 13): ARM64 support for embedded AI (DGX Spark)
• Apple Silicon: Native Metal acceleration for Mac M1/M2/M3+
• Darwin x86: Intel Mac support

CPU Optimization

• AVX/AVX2/AVX512: Advanced vector extensions for x86
• Quantization: 4-bit, 5-bit, 8-bit integer quantization support
• Mixed Precision: F16/F32 mixed precision support

Note: any backend name listed above can be used in the backend field of the model configuration file (See the advanced section).

• * Only for CUDA and OpenVINO CPU/XPU acceleration.

Architecture

LocalAI is an API written in Go that serves as an OpenAI shim, enabling software already developed with OpenAI SDKs to seamlessly integrate with LocalAI. It can be effortlessly implemented as a substitute, even on consumer-grade hardware. This capability is achieved by employing various C++ backends, including ggml, to perform inference on LLMs using both CPU and, if desired, GPU. Internally LocalAI backends are just gRPC server, indeed you can specify and build your own gRPC server and extend LocalAI in runtime as well. It is possible to specify external gRPC server and/or binaries that LocalAI will manage internally.

LocalAI uses a mixture of backends written in various languages (C++, Golang, Python, …). You can check the model compatibility table to learn about all the components of LocalAI.

Backstory

As much as typical open source projects starts, I, mudler, was fiddling around with llama.cpp over my long nights and wanted to have a way to call it from go, as I am a Golang developer and use it extensively. So I’ve created LocalAI (or what was initially known as llama-cli) and added an API to it.

But guess what? The more I dived into this rabbit hole, the more I realized that I had stumbled upon something big. With all the fantastic C++ projects floating around the community, it dawned on me that I could piece them together to create a full-fledged OpenAI replacement. So, ta-da! LocalAI was born, and it quickly overshadowed its humble origins.

Now, why did I choose to go with C++ bindings, you ask? Well, I wanted to keep LocalAI snappy and lightweight, allowing it to run like a champ on any system and avoid any Golang penalties of the GC, and, most importantly built on shoulders of giants like llama.cpp. Go is good at backends and API and is easy to maintain. And hey, don’t forget that I’m all about sharing the love. That’s why I made LocalAI MIT licensed, so everyone can hop on board and benefit from it.

As if that wasn’t exciting enough, as the project gained traction, mkellerman and Aisuko jumped in to lend a hand. mkellerman helped set up some killer examples, while Aisuko is becoming our community maestro. The community now is growing even more with new contributors and users, and I couldn’t be happier about it!

Oh, and let’s not forget the real MVP here—llama.cpp. Without this extraordinary piece of software, LocalAI wouldn’t even exist. So, a big shoutout to the community for making this magic happen!

CLI Reference

Complete reference for all LocalAI command-line interface (CLI) parameters and environment variables.

Note: All CLI flags can also be set via environment variables. Environment variables take precedence over CLI flags. See .env files for configuration file support.

Global Flags

Storage Flags

Backend Flags

For more information on VRAM management, see VRAM and Memory Management.

Models Flags

Note: You can also pass model configuration URLs as positional arguments: local-ai run MODEL_URL1 MODEL_URL2 ...

Performance Flags

API Flags

Hardening Flags

P2P Flags

Other Commands

LocalAI supports several subcommands beyond run:

• local-ai models - Manage LocalAI models and definitions
• local-ai backends - Manage LocalAI backends and definitions
• local-ai tts - Convert text to speech
• local-ai sound-generation - Generate audio files from text or audio
• local-ai transcript - Convert audio to text
• local-ai worker - Run workers to distribute workload (llama.cpp-only)
• local-ai util - Utility commands
• local-ai explorer - Run P2P explorer
• local-ai federated - Run LocalAI in federated mode

Use local-ai <command> --help for more information on each command.

Examples

Basic Usage

./local-ai run

./local-ai run --models-path /path/to/models --address :9090

./local-ai run --f16

Environment Variables

export LOCALAI_MODELS_PATH=/path/to/models
export LOCALAI_ADDRESS=:9090
export LOCALAI_F16=true
./local-ai run

Advanced Configuration

./local-ai run \
  --models model1.yaml model2.yaml \
  --enable-watchdog-idle \
  --watchdog-idle-timeout=10m \
  --p2p \
  --federated

Related Documentation

• See Advanced Usage for configuration examples
• See VRAM and Memory Management for memory management options

API Error Reference

This page documents the error responses returned by the LocalAI API. LocalAI supports multiple API formats (OpenAI, Anthropic, Open Responses), each with its own error structure.

Error Response Formats

OpenAI-Compatible Format

Most endpoints return errors using the OpenAI-compatible format:

{
  "error": {
    "code": 400,
    "message": "A human-readable description of the error",
    "type": "invalid_request_error",
    "param": null
  }
}

This format is used by: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions, /models, and other OpenAI-compatible endpoints.

Anthropic Format

The /v1/messages endpoint returns errors in Anthropic’s format:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "A human-readable description of the error"
  }
}

Open Responses Format

The /v1/responses endpoint returns errors with this structure:

{
  "error": {
    "type": "invalid_request",
    "message": "A human-readable description of the error",
    "code": "",
    "param": "parameter_name"
  }
}

HTTP Status Codes

Global Error Handling

Authentication Errors (401)

When API keys are configured (via LOCALAI_API_KEY or --api-keys), all requests must include a valid key. Keys can be provided through:

• Authorization: Bearer <key> header
• x-api-key: <key> header
• xi-api-key: <key> header
• token cookie

Example request without a key:

curl http://localhost:8080/v1/models \
  -H "Content-Type: application/json"

Error response:

{
  "error": {
    "code": 401,
    "message": "An authentication key is required",
    "type": "invalid_request_error"
  }
}

The response also includes the header WWW-Authenticate: Bearer.

Request Parsing Errors (400)

All endpoints return a 400 error if the request body cannot be parsed:

{
  "error": {
    "code": 400,
    "message": "failed parsing request body: <details>",
    "type": ""
  }
}

Not Found (404)

Requests to undefined routes return:

{
  "error": {
    "code": 404,
    "message": "Resource not found"
  }
}

Opaque Errors Mode

When LOCALAI_OPAQUE_ERRORS=true is set, all error responses return an empty body with only the HTTP status code. This is a security hardening option that prevents information leaks.

Per-Endpoint Error Scenarios

Chat Completions — POST /v1/chat/completions

# Missing model field
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "hello"}]}'

See also: Text Generation

Completions — POST /v1/completions

Embeddings — POST /v1/embeddings

See also: Embeddings

Image Generation — POST /v1/images/generations

See also: Image Generation

Image Editing (Inpainting) — POST /v1/images/edits

Audio Transcription — POST /v1/audio/transcriptions

See also: Audio to Text

Text to Speech — POST /v1/audio/speech, POST /tts

See also: Text to Audio

ElevenLabs TTS — POST /v1/text-to-speech/:voice-id

ElevenLabs Sound Generation — POST /v1/sound-generation

Reranking — POST /v1/rerank, POST /jina/v1/rerank

See also: Reranker

Anthropic Messages — POST /v1/messages

# Missing model field
curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "hello"}], "max_tokens": 100}'

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "model is required"
  }
}

Open Responses — POST /v1/responses

# Missing model field
curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{"input": "hello"}'

{
  "error": {
    "type": "invalid_request",
    "message": "model is required",
    "code": "",
    "param": ""
  }
}

Open Responses — GET /v1/responses/:id

Open Responses Events — GET /v1/responses/:id/events

Object Detection — POST /v1/detection

See also: Object Detection

Video Generation — POST /v1/video/generations

Voice Activity Detection — POST /v1/audio/vad

Tokenize — POST /v1/tokenize

Models — GET /v1/models, GET /models

See also: Model Gallery

Handling Errors in Client Code

Python (OpenAI SDK)

from openai import OpenAI, APIError

client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-key")

try:
    response = client.chat.completions.create(
        model="my-model",
        messages=[{"role": "user", "content": "hello"}],
    )
except APIError as e:
    print(f"Status: {e.status_code}, Message: {e.message}")

curl

# Check HTTP status code
response=$(curl -s -w "\n%{http_code}" http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "nonexistent", "messages": [{"role": "user", "content": "hi"}]}')

http_code=$(echo "$response" | tail -1)
body=$(echo "$response" | head -1)

if [ "$http_code" -ne 200 ]; then
  echo "Error $http_code: $body"
fi

Related Configuration

LocalAI binaries

LocalAI binaries are available for both Linux and MacOS platforms and can be executed directly from your command line. These binaries are continuously updated and hosted on our GitHub Releases page. This method also supports Windows users via the Windows Subsystem for Linux (WSL).

macOS Download

You can download the DMG and install the application:

Note: the DMGs are not signed by Apple as quarantined. See https://github.com/mudler/LocalAI/issues/6268 for a workaround, fix is tracked here: https://github.com/mudler/LocalAI/issues/6244

Otherwise, use the following one-liner command in your terminal to download and run LocalAI on Linux or MacOS:

curl -Lo local-ai "https://github.com/mudler/LocalAI/releases/download/v3.12.1/local-ai-$(uname -s)-$(uname -m)" && chmod +x local-ai && ./local-ai

Otherwise, here are the links to the binaries:

Running on Nvidia ARM64

LocalAI can be run on Nvidia ARM64 devices, such as the Jetson Nano, Jetson Xavier NX, Jetson AGX Orin, and Nvidia DGX Spark. The following instructions will guide you through building and using the LocalAI container for Nvidia ARM64 devices.

Platform Compatibility

• CUDA 12 L4T images: Compatible with Nvidia AGX Orin and similar platforms (Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
• CUDA 13 L4T images: Compatible with Nvidia DGX Spark

Prerequisites

• Docker engine installed (https://docs.docker.com/engine/install/ubuntu/)
• Nvidia container toolkit installed (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-ap)

Pre-built Images

Pre-built images are available on quay.io and dockerhub:

CUDA 12 (for AGX Orin and similar platforms)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64
# or
docker pull localai/localai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13
# or
docker pull localai/localai:latest-nvidia-l4t-arm64-cuda-13

Build the container

If you need to build the container yourself, use the following commands:

CUDA 12 (for AGX Orin and similar platforms)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=true --build-arg BUILD_TYPE=cublas --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core .

CUDA 13 (for DGX Spark)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=false --build-arg BUILD_TYPE=cublas --build-arg CUDA_MAJOR_VERSION=13 --build-arg CUDA_MINOR_VERSION=0 --build-arg BASE_IMAGE=ubuntu:24.04 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-cuda-13-core .

Usage

Run the LocalAI container on Nvidia ARM64 devices using the following commands, where /data/models is the directory containing the models:

CUDA 12 (for AGX Orin and similar platforms)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13

Note: /data/models is the directory containing the models. You can replace it with the directory containing your models.