Qwen/Qwen2.5-0.5B on AMD

# An InferenceClass describing a BYO 8x AMD Instinct MI325X node pool.
#
# No provisioning block: this class describes hardware that already exists on
# a bring-your-own cluster whose node images carry the amdgpu kernel driver.
# When a cluster references it, the serving stack installs the AMD GPU
# operator in DRA mode (driver management off), whose DRA driver publishes
# the GPUs as gpu.amd.com ResourceSlices. Label the pool's nodes
# modelplane.ai/pool=<pool name>; the amd.com/gpu=true:NoSchedule taint is
# the convention that keeps system workloads off them, and engine pods
# tolerate it.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: amd-mi325x-8x
spec:
  description: "BYO 8x AMD Instinct MI325X 256GB"
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.amd.com
    deviceClassName: gpu.amd.com
    count: 8
    attributes:
      productName: { string: AMD_Instinct_MI325_OAM }
    capacity:
      # The MI325X's usable HBM3e as the AMD DRA driver's ResourceSlices
      # report it, not its nominal 256GB. A nodeSelector asking for
      # >= 256Gi would never bind.
      memory: { value: "262128Mi" }

# Registers the bring-your-own cluster and installs the serving stack onto
# it through the kubeconfig Secret. The composed ServingStack gets
# draDrivers: [gpu.amd.com], so the AMD GPU operator (DRA mode) installs and
# the NVIDIA and CPU device stacks stay out. Existing is the one cloud that
# installs the AMD stack today.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: amd-byo
spec:
  cluster:
    source: Existing
    existing:
      secretRef:
        name: byo-cluster-kubeconfig
        key: kubeconfig
  nodePools:
  - name: gpu-mi325x
    className: amd-mi325x-8x
    nodeCount: 1

# A small model to prove placement and serving on the AMD pool. The device
# selector matches against the capacity the amd-mi325x-8x class declares
# (262128Mi), not the pod's resource requests, so >= 200Gi lands it on the
# MI325X pool; the claim takes 1 of the node's 8 GPUs. The engine is AMD's
# ROCm vLLM build (the CDNA variant covers MI325X/gfx942) - the upstream
# vllm/vllm-openai image is CUDA-only. Unlike that image, rocm/vllm's
# entrypoint is not the OpenAI server, so the command invokes it explicitly.
# No ModelCache: the engine pulls weights straight from Hugging Face.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.amd.com"].memory.compareTo(quantity("200Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: rocm/vllm:rocm7.14.1_cdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0
                command: ["vllm", "serve", "Qwen/Qwen2.5-0.5B-Instruct"]
                args:
                - --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)

# Publishes the deployment's endpoints as one model a caller can name:
# <namespace>/<service>, so ml-team/qwen here. Modelplane composes one
# ModelEndpoint per ModelReplica, labeled with the deployment name, so
# this selector reaches every replica.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen
  namespace: ml-team
spec:
  endpoints:
  - name: qwen-demo
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-demo