Model and metadata cache root (default ~/.buttress/models)
huggingface_token
string
Hugging Face auth token; falls back to $HUGGINGFACE_TOKEN. Applied to all backends regardless of variable name
http_headers
table
Extra headers attached to Hugging Face / HTTP downloads
context_release_delay_ms
number
Idle time before unloading a context (default 10000; 0 = immediate)
prefer_variants
string[]
Override backend variant probe order (ggml backends)
n_threads
number
CPU thread count
n_ctx
number
Context window (per-model value wins; auto-capped at training context)
n_gpu_layers
number or "auto"
Layers offloaded to GPU (default "auto")
n_batch
number
Prompt batch size. Note: the model layer defaults n_batch to 512, which shadows the runtime value unless n_batch is set explicitly under [generators.model]
n_ubatch
number
Prompt micro-batch size
n_parallel
number
Parallel sequences (default 4)
n_cpu_moe
number
MoE expert layers offloaded to CPU
flash_attn_type
string
"on", "off", or "auto". Default is GPU-conditional: "auto" when a GPU backend is selected, "off" on CPU
cache_type_k, cache_type_v
string
KV-cache dtype (f16, f32, q8_0, q4_0, …)
kv_unified
boolean
Use a unified KV cache across sequences
swa_full
boolean
Materialize full attention even for sliding-window layers
For ggml-llm generators, the server can persist KV cache state between requests so that a follow-up completion sharing a prompt prefix skips prompt processing.
Total disk budget; accepts "500MB", "50GB", or a number
max_entries
1000
Max number of cached states (LRU eviction)
Cache files are stored under {cache_dir}/.session-state-cache/.mlx-llm keeps a separate session cache under {cache_dir}/mlx-session-cache/, configured independently per generator.
Each [[generators]] block declares one model the server can host. Repeat the block to host multiple. Every block has a type, an optional [generators.backend] table, and a [generators.model] table.
[[generators]]type = "ggml-llm"[generators.backend]# backend selection and resource planning[generators.model]repo_id = "..."# model identity and runtime overrides
[generators.backend] only controls backend selection and resource planning. Runtime overrides (n_ctx, n_gpu_layers, flash_attn_type, etc.) go under [generators.model].[generators.backend]
Key
Type
Default
Description
variant
string
auto
Force cuda, vulkan, snapdragon, or default
variant_preference
string[]
["cuda", "vulkan", "snapdragon", "default"]
Probe order when variant is unset
gpu_memory_fraction
number
0.85
Max GPU fraction the hardware guardrails may plan against
cpu_memory_fraction
number
0.5
Max RAM fraction for CPU-side buffers
[generators.model] — in addition to the common ggml keys above, every [runtime] key can be overridden per-generator: n_ctx, n_gpu_layers, n_batch, n_ubatch, n_threads, n_parallel, n_cpu_moe, flash_attn_type, cache_type_k, cache_type_v, kv_unified, swa_full, ctx_shift, use_mmap, use_mlock, no_extra_bufts, cpu_mask, cpu_strict, devices.Multimodal (mtmd) — auto-downloads the matching mmproj-*.gguf from the same repo:
Key
Type
Description
enable_mtmd
boolean
Default false
mmproj_filename
string
Pin a specific projector file
mmproj_url
string
Direct URL override
mmproj_local_path
string
Local projector file (requires allow_local_file = true)
mmproj_use_gpu
boolean
Unset = auto (true when n_gpu_layers > 0)
mmproj_image_min_tokens
number
Min visual tokens (dynamic-resolution models; -1 = unset)
There is no [generators.backend] section for mlx-llm.On first use, the backend creates a Python virtualenv at {cache_dir}/mlx-env and installs mlx_lm_package, mlx_vlm_package, plus torch and torchvision (required by some VLM processors). If an existing venv already has mlx_vlm and torch importable, the install step is skipped.[generators.model] — common repo_id / revision / download plus:
Key
Type
Default
Description
adapter_path
string
—
Local LoRA adapter directory
vlm
"auto" or boolean
"auto"
Force VLM (true) vs text-only (false); "auto" infers from the repo
tokenizer_config
table
—
Forwarded to mlx_lm.load(..., tokenizer_config=...)
model_config
table
—
Forwarded to mlx_lm.load(..., model_config=...)
quantization, filename, and preferred_quantizations are not used — the MLX repo itself determines the quantization.Runtime extras — under [runtime] for mlx-llm:
Environment variables applied at startup, but only if they are not already set in the system environment. System variables and command-line exports take precedence.
The ggml backends read HUGGINGFACE_TOKEN (not HF_TOKEN). For a single token that applies to every backend regardless of variable name, set [runtime] huggingface_token instead.
