Version

Qwen/Qwen2.5-0.5B on CPU

A 0.5B dense chat model served on CPUs alone, on a Civo CPU pool.

On this page

This document is for an unreleased version of Modelplane.

This document applies to the Modelplane main branch and not to the latest release v0.5.

A 0.5B dense chat model served on CPUs alone, on a Civo g4p.kube.medium pool: one Standalone engine, no cache, weights pulled straight from Hugging Face. A CPU pool is the cheapest way to exercise a Modelplane deployment end to end - no GPU to find, no driver to wait for - and a 0.5B model is small enough that CPU inference answers in sensible time.

The class’s provisioning block omits accelerator, which is what makes the pool a CPU pool: the cluster composition builds it with a standard node image and the modelplane.ai/cpu label and taint, and the serving stack installs dra-driver-cpu - and no GPU machinery - because the class claims dra.cpu. The claim pins the engine to its NUMA node’s cpuset. The engine is vLLM’s CPU build, whose image serves the same OpenAI API as the CUDA one. The same class without its provisioning block describes a bring-your-own CPU pool, on any conformant cluster: label its nodes modelplane.ai/pool=cpu-pool and modelplane.ai/cpu=true instead.

Validated deployments

Dense 0.5B 4,096 ctx vLLM

Cloud

AWS Google Cloud Azure Nebius Vultr B Bring your own

GPU

no GPU

Serving mode

Standalone LeaderWorker PrefillDecode

Precision

Engine

vLLM SGLang llama.cpp

Image

vllm/vllm-openai-cpu:v0.32.0-x86_64

Manifests

docs/manifests/recipes/qwen2.5-0.5b-cpu

Platform

# An InferenceClass describing a Civo CPU node pool - no GPU at all.
#
# The provisioning block has no accelerator, so the cluster composition
# builds the pool with the CPU role: a standard node image and the
# modelplane.ai/cpu label and taint, nothing GPU. The serving stack installs
# dra-driver-cpu when a class claims dra.cpu, which publishes each node's
# cores as claimable devices (one device per NUMA node) and registers the
# dra.cpu DeviceClass.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: civo-cpu-8vcpu
spec:
  description: "Civo g4p.kube.medium, 8 vCPUs, no GPU"
  provisioning:
    provider: Civo
    civo:
      size: g4p.kube.medium
  devices:
  - name: cpu
    claim: DRA
    driver: dra.cpu
    deviceClassName: dra.cpu
    count: 1
    capacity:
      # The node's 8 vCPUs less the two dra-driver-cpu reserves for the OS
      # and kubelet (reservedCPUs 0-1), which is what the driver's
      # ResourceSlices report as claimable.
      cpu: { value: "6" }

# A Civo cluster in NYC1 with a single CPU node offering the class above.
# The composed ServingStack gets draDrivers: [dra.cpu], so dra-driver-cpu
# installs and the GPU device stacks stay out - a cluster with no GPU runs
# no GPU machinery.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: civo-cpu
  labels:
    modelplane.ai/region: nyc1
spec:
  cluster:
    source: Civo
    civo:
      region: NYC1
  nodePools:
  - name: cpu-pool
    className: civo-cpu-8vcpu
    nodeCount: 1

Deployment

# A small model served on CPUs alone. The device selector matches against
# the claimable cores the cpu-8vcpu class declares, so >= 4 lands it on the
# CPU pool; the claim takes the whole NUMA-node device and pins the engine
# container to its cpuset. The engine is vLLM's CPU build - the default
# vllm/vllm-openai image is CUDA-only. --block-size=16 matches the block
# size the endpoint picker assumes when an engine doesn't state one, so
# prefix-cache routing keeps working; VLLM_CPU_KVCACHE_SPACE sets the KV
# cache's share of the node's RAM in GiB. No ModelCache: the engine pulls
# weights straight from Hugging Face.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-cpu-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: cpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["dra.cpu"].cpu.compareTo(quantity("4")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai-cpu:v0.32.0-x86_64
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)
                - --max-model-len=4096
                - --block-size=16
                env:
                - name: VLLM_CPU_KVCACHE_SPACE
                  value: "4"

# Publishes the deployment's endpoints as one model a caller can name:
# <namespace>/<service>, so ml-team/qwen-cpu here. Modelplane composes one
# ModelEndpoint per ModelReplica, labeled with the deployment name, so
# this selector reaches every replica.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-cpu
  namespace: ml-team
spec:
  endpoints:
  - name: qwen-cpu-demo
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-cpu-demo