batch-infer-lora-sglang
Batch chat completions with your LoRA adapter on SGLang
batch-infer-lora on SGLang: runs an OpenAI batch file of /v1/chat/completions (or /v1/completions) requests with a LoRA adapter applied (for example the adapter.tar.gz a finetune-lora job produced) and writes one result line per request, model = LORA_NAME. Shared prompt prefixes are prefilled once, as in batch-infer-sglang. Default base model Qwen3-4B-Instruct-2507 (Apache-2.0); the adapter must have been trained on the same base.
- Status
- Available
- GPU
- NVIDIA A10G · 24 GB
a10g-24gb-x1 - Default runtime
- 1 h maximum
- Publisher
- Unify Compute
What it reads and writes
Input /mnt/in | requests.jsonl: one OpenAI batch line per request with url /v1/chat/completions or /v1/completions; the model field is ignored, the adapter applies to every line |
|---|---|
Input /mnt/adapter | adapter.tar.gz: a PEFT LoRA adapter directory (adapter_config.json, adapter_model.safetensors), as finetune-lora writes it |
Output /mnt/out | results.jsonl: one OpenAI batch result line per request, in input order, matched by custom_id, model = LORA_NAME; a line SGLang could not answer carries error instead of response |
What you can change
A submit may set: GPU (SKU) (sku), Cloud placement (providers), Environment variables (spec.env), Maximum runtime (spec.max_runtime_seconds). The image and the declared inputs and outputs are fixed: a different image is a different template.
MODEL | Qwen/Qwen3-4B-Instruct-2507 |
|---|---|
MODEL_REVISION | cdbee75f17c01a7cc42f958dc650907174af0554 |
MAX_MODEL_LEN | 8192 |
MAX_LORA_RANK | 64 |
LORA_NAME | lora |
CONCURRENCY | 64 |
Run it
With a key in $UNIFY_KEY (see the quick start), give each input an https:// URL, public or presigned, that ends in the file name above:
curl -s https://api.compute.unifyai.us/compute/v1/jobs \
-H "Authorization: Bearer $UNIFY_KEY" \
-H "Content-Type: application/json" \
-d '{
"template": "batch-infer-lora-sglang",
"spec": {
"inputs": [
{
"source": "https://YOUR-BUCKET.s3.amazonaws.com/requests.jsonl",
"mount": "/mnt/in"
},
{
"source": "https://YOUR-BUCKET.s3.amazonaws.com/adapter.tar.gz",
"mount": "/mnt/adapter"
}
]
}
}'The job is billed for the GPU time it uses at the SKU's rate, the same as a job you write yourself.
GPU job execution is turned on per account while it rolls out. Create an account, then ask us to enable it as the quick start describes.
Where the image comes from
| Image | lmsysorg/sglang:v0.5.19-cu129-runtime |
|---|---|
| Digest | sha256:fe08f6f52bea380415792849a703d92279c86320f8c39283bcd04043f8b63c41 |
| Licence | Apache-2.0 |
| Source | https://hub.docker.com/r/lmsysorg/sglang (tag v0.5.19-cu129-runtime, pushed 2026-09-04, the newest CUDA 12.9 build; multi-arch index read from registry-1.docker.io on 2026-11-06; LICENSE at github.com/sgl-project/sglang/blob/v0.5.19/LICENSE) |
The digest is what runs. A tag can be re-pushed; a digest cannot, so what was reviewed is what your job pulls.