Project-HAMi/HAMi

English version | 中文版 | 日本語版

Kubernetes GPU virtualization and heterogeneous accelerator scheduling for AI infrastructure.

HAMi stands for Heterogeneous AI Computing Virtualization Middleware. Formerly known as k8s-vGPU-scheduler, HAMi helps platform teams share expensive GPUs and other AI accelerators across Kubernetes workloads, isolate device memory and compute, and schedule pods with device-aware policies without changing application code.

HAMi is a CNCF Incubating and CNCF Landscape project. It is also listed in the CNAI Landscape.

AI infrastructure teams often run into the same Kubernetes accelerator problems: whole GPUs are allocated to small jobs, teams compete for scarce devices, different accelerator vendors expose different operational models, and schedulers lack enough device context to place workloads efficiently.

HAMi provides a Kubernetes-native layer for:

• Device sharing: allocate a fraction of a physical accelerator by memory, core, or device count.
• Resource isolation: enforce per-workload accelerator memory and compute limits where the device backend supports it.
• Device-aware scheduling: place pods with topology-aware, binpack, spread, and device-specific scheduling policies.
• Heterogeneous AI clusters: manage NVIDIA GPUs, NPUs, HCUs, MLUs, and other accelerator types through one scheduling and allocation workflow.
• Zero application changes: keep using standard Kubernetes resource requests and limits.
• Production operations: expose metrics, dashboards, WebUI, Helm installation, and community-supported deployment guidance.

• Increase GPU utilization in shared Kubernetes AI clusters.
• Run multi-tenant notebook, training, and inference workloads on the same accelerator pool.
• Build private cloud AI platforms with fair device allocation and quota control.
• Operate heterogeneous accelerator clusters across NVIDIA, Ascend, Cambricon, Hygon, Iluvatar, Kunlunxin, MetaX, Moore Threads, and other vendors.
• Combine HAMi with Kubernetes schedulers such as kube-scheduler and Volcano for batch AI workloads.

HAMi is composed of a mutating webhook, scheduler extender, device plugins, and device-specific in-container virtualization components.

Pod submission
  -> HAMi mutating webhook
  -> HAMi scheduler filter / score / bind
  -> device allocation written to pod annotations
  -> device plugin Allocate()
  -> container runtime environment
  -> HAMi monitor and metrics

HAMi lets workloads request only the accelerator resources they need. For example, the following pod asks for one physical NVIDIA GPU with 3 GiB of GPU memory:

resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 3000

The workload sees the allocated device resources inside the container, while HAMi coordinates scheduling, allocation, and isolation.

Notes:

1. After installing HAMi, the value of nvidia.com/gpu registered on the node defaults to the number of vGPUs.
2. When requesting resources in a pod, nvidia.com/gpu refers to the number of physical GPUs required by the current pod.

HAMi supports multiple heterogeneous accelerator backends, including GPUs, NPUs, HCUs, MLUs, GCUs, XPUs, and more. Device capabilities vary by vendor, model, driver, and hardware generation.

Device backends in this repository: NVIDIA GPU (including MIG), AMD GPU, Huawei Ascend NPU, Cambricon MLU, Hygon HCU, Iluvatar GPU, Kunlunxin XPU, MetaX GPU, Moore Threads GPU, Enflame GCU, Biren GPU, VastAI, AWS Neuron, and remote NVIDIA GPUs served by Lupine. Pod examples for each vendor are under examples/.

See the current HAMi supported devices page for the maintained support matrix.

For the NVIDIA device plugin path, prepare:

• NVIDIA driver >= 440
• NVIDIA Container Toolkit (the successor to nvidia-docker2)
• NVIDIA configured as the default runtime for containerd, Docker, or CRI-O
• Kubernetes >= 1.23
• glibc >= 2.17
• Linux kernel >= 3.10
• Helm > 3.0

Label GPU nodes so HAMi can manage them:

kubectl label nodes <node-name> gpu=on

Add the HAMi Helm repository:

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update

Install HAMi:

helm install hami hami-charts/hami -n kube-system

Verify that the scheduler and device plugin are running:

kubectl get pods -n kube-system

When hami-device-plugin and hami-scheduler are both Running, submit an example workload:

kubectl apply -f examples/nvidia/default_use.yaml

Vendor settings live under devices.<vendor> in values.yaml. The chart README documents the configurable values and how to move from the previous layout.

For the complete installation guide and configuration options, see the HAMi documentation.

• High availability: set scheduler.leaderElect=true and scheduler.replicas above 1 to run several scheduler replicas behind one leader. A PodDisruptionBudget is created for them.
• OpenShift and SELinux: set platform.openshift=true for OpenShift, and selinux.enabled=true on SELinux-enabled nodes.
• NetworkPolicy: scheduler.networkPolicy.enabled=true restricts access to the scheduler's webhook and /refit port. Some CNIs do not match host-networked kube-apiserver traffic against the namespaceSelector, which can make the webhook unreachable; with scheduler.admissionWebhook.failurePolicy=Ignore, Pods are then admitted without HAMi's changes.
• Prometheus Operator: prometheus.enabled=true creates ServiceMonitors for the scheduler and the device plugin when the ServiceMonitor API is available; the device-plugin one also requires devicePlugin.enabled.

HAMi supports multiple scheduling modes for AI workloads:

• binpack: pack workloads onto fewer nodes or devices to improve consolidation.
• spread: distribute workloads across nodes or devices to reduce contention.
• topology-aware scheduling: choose device combinations based on GPU topology when supported.
• mutex: allocate only idle GPUs, so a workload does not share a device.
• numa: a sort key that groups devices by NUMA node, chained with binpack or spread. A pod can ask for GPU-to-GPU NUMA co-location with the nvidia.com/numa-bind annotation.

Set the cluster defaults with scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy and gpuSchedulerPolicy. A pod overrides them with the hami.io/node-scheduler-policy and hami.io/gpu-scheduler-policy annotations, and the GPU policy accepts a comma-separated combination such as binpack,numa.

• dynamic MIG: create, cache, reuse, and reclaim NVIDIA MIG instances dynamically for supported cards and modes. See the Dynamic MIG instance lifecycle.

HAMi works with the default Kubernetes scheduler path and can also be used with Volcano for batch-oriented AI workloads. See the HAMi website for current scheduler integration guides.

HAMi exposes metrics for monitoring cluster accelerator usage. After installation, metrics are available through the scheduler monitor endpoint:

http://<scheduler-ip>:<monitor-port>/metrics

The default monitor port is 31993. You can change it with Helm values such as --set scheduler.service.monitorPort=<port>.

HAMi also provides:

• HAMi-WebUI for visual cluster and device management.
• Agent skills under skill/ for debugging GPU pods and summarizing vGPU metrics.
• Grafana dashboards for accelerator monitoring.
• Benchmark material for evaluating workload behavior and scheduling effects.

HAMi is governed by maintainers and contributors. Governance is described in the HAMi community repository.

To contribute code, documentation, tests, or device backend improvements, read CONTRIBUTING.md.

The HAMi community is open to users, contributors, hardware vendors, and platform teams building Kubernetes-based AI infrastructure.

• Website: project-hami.io
• Discord: Join the HAMi Discord (recommended)
• Slack: #hami-dev on CNCF Slack
• Mailing list: hami-project
• Meeting notes and agenda
• Chinese community meeting: Friday 16:00 UTC+8, weekly, Meeting link
• English community meeting: Wednesday 16:00 UTC+8, biweekly, Meeting link

HAMi is licensed under the Apache License 2.0. See LICENSE for details.

Copyright Contributors to HAMi, established as HAMi a Series of LF Projects, LLC.