batch-infer-vllm

Batch chat completions with vLLM

Runs an OpenAI batch file of /v1/chat/completions requests through vLLM's batch runner and writes one result line per request. Default model Qwen3-4B-Instruct-2507 (Apache-2.0), downloaded from Hugging Face at start.

Status

Available

GPU

NVIDIA A10G · 24 GB a10g-24gb-x1

Default runtime

1 h maximum

Publisher

Unify Compute

What it reads and writes

What you can change

A submit may set: GPU (SKU) (sku), Cloud placement (providers), Environment variables (spec.env), Maximum runtime (spec.max_runtime_seconds). The image and the declared inputs and outputs are fixed: a different image is a different template.

Run it

With a key in $UNIFY_KEY (see the quick start), give each input an https:// URL, public or presigned, that ends in the file name above:

curl -s https://api.compute.unifyai.us/compute/v1/jobs \
  -H "Authorization: Bearer $UNIFY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "template": "batch-infer-vllm",
  "spec": {
    "inputs": [
      {
        "source": "https://YOUR-BUCKET.s3.amazonaws.com/requests.jsonl",
        "mount": "/mnt/in"
      }
    ]
  }
}'

The job is billed for the GPU time it uses at the SKU's rate, the same as a job you write yourself.

Where the image comes from

The digest is what runs. A tag can be re-pushed; a digest cannot, so what was reviewed is what your job pulls.