vSphere CSI Driver - Deployment with Topology

Note: Volume Topology and Availability Zone feature is beta in vSphere CSI Driver.

When you deploy CPI and CSI in a vSphere environment that includes multiple data centers or host clusters, you can use zoning.

Zoning enables orchestration systems, like Kubernetes, to integrate with vSphere storage resources that are not equally available to all nodes. As a result, the orchestration system can make intelligent decisions when dynamically provisioning volumes, and avoid situations such as those where a pod cannot start because the storage resource it needs is not accessible.

Set Up Zones in the vSphere CNS Environment

Depending on your vSphere storage environment, you can use different deployment scenarios for zones. For example, you can have zones per host cluster, per data center, or have a combination of both.

In the following example, the vCenter Server environment includes three clusters with node VMs located on all three clusters.


The sample workflow creates zones per cluster and per data center.


  • Create Zones Using vSphere Tags
    • You can use vSphere tags to label zones in your vSphere environment.
  • Enable Zones for the CCM and CSI Driver
    • Install the CCM and the CSI driver using the zone and region entries.

Create Zones Using vSphere Tags

You can use vSphere tags to label zones in your vSphere environment.

The task assumes that your vCenter Server environment includes three clusters, cluster1, cluster2, and cluster3, with the node VMs on all three clusters. In the task, you create two tag categories, k8s-zone and k8s-region. You tag the clusters as three zones, zone-a, zone-b, and zone-c, and mark the data center as a region, region-1.


Make sure that you have appropriate tagging privileges that control your ability to work with tags. See vSphere Tagging Privileges in the vSphere Security documentation.

Note: Ancestors of node VMs, such as host, cluster, and data center, must have the ReadOnly role set for the vSphere user configured to use the CSI driver and CCM. This is required to allow reading tags and categories to prepare nodes' topology.

  1. In the vSphere Client, create two tag categories, k8s-zone and k8s-region.

    For information, see Create, Edit, or Delete a Tag Category in the vCenter Server and Host Management documentation.

  2. In each category, create appropriate zone tags.

    For information on creating tags, see Create, Edit, or Delete a Tag in the vCenter Server and Host Management documentation.

    Categories Tags
    k8s-zone zone-a
    k8s-region region-1
  3. Apply corresponding tags to the data center and clusters as indicated in the table.

    For information, see Assign or Remove a Tag in the vCenter Server and Host Management documentation.

    vSphere Objects Tags
    datacenter region-1
    cluster1 zone-a
    cluster2 zone-b
    cluster3 zone-c

Enable Zones for the vSphere CPI and CSI Driver

Install the vSphere CPI and the CSI driver using the zone and region entries.

  1. Create a config secret for vSphere CPI:

    In the value fields of the configmap cloud-config file, specify region and zone. Make sure to add the names of categories you defined in vSphere, such as k8s-region and k8s-zone.

    insecure-flag = "true"
    [VirtualCenter "vCenter Server IP address"]
    user = "user"
    password = "password"
    port = "443"
    datacenters = "datacenter"
    public-network = "VM Network"
    region = k8s-region
    zone = k8s-zone
    cd /etc/kubernetes
    kubectl create configmap cloud-config --from-file=vsphere.conf --namespace=kube-system
  2. Create Roles, Roles Bindings, Service Account, Service and cloud-controller-manager Pod

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
    kubectl apply -f https://github.com/kubernetes/cloud-provider-vsphere/raw/master/manifests/controller-manager/vsphere-cloud-controller-manager-ds.yaml
  3. Verify that your CCM installation is successful.

    After installation, labels failure-domain.beta.kubernetes.io/region and failure-domain.beta.kubernetes.io/zone are applied to all nodes.

    kubectl get nodes -L failure-domain.beta.kubernetes.io/zone -L failure-domain.beta.kubernetes.io/region
    k8s-master   Ready    master   32m   v1.14.2   zone-a   region-1
    k8s-node1    Ready    <none>   18m   v1.14.2   zone-a   region-1
    k8s-node2    Ready    <none>   18m   v1.14.2   zone-b   region-1
    k8s-node3    Ready    <none>   18m   v1.14.2   zone-b   region-1
    k8s-node4    Ready    <none>   18m   v1.14.2   zone-c   region-1
    k8s-node5    Ready    <none>   18m   v1.14.2   zone-c   region-1
  4. Install the CSI driver.

    Make sure external-provisioner is deployed with the arguments --feature-gates=Topology=true and --strict-topology.

    In the credential secret file, add entries for region and zone.

    region = k8s-region
    zone = k8s-zone

    Make sure secret is mounted on all workload nodes as well. This is required to help node discover its topology.

      - name: vsphere-csi-node
        - name: VSPHERE_CSI_CONFIG
          value: "/etc/cloud/csi-vsphere.conf"
        - name: vsphere-config-volume
          mountPath: /etc/cloud
          readOnly: true
      - name: vsphere-config-volume
          secretName: vsphere-config-secret
  5. Verify that your CSI driver installation is successful.

    kubectl get csinodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec}{"\n"}{end}'
    k8s-node1 map[drivers:[map[name:csi.vsphere.vmware.com nodeID:k8s-node1 topologyKeys:[failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]]]]
    k8s-node2 map[drivers:[map[name:csi.vsphere.vmware.com nodeID:k8s-node2 topologyKeys:[failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]]]]
    k8s-node3 map[drivers:[map[name:csi.vsphere.vmware.com nodeID:k8s-node3 topologyKeys:[failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]]]]
    k8s-node4 map[drivers:[map[name:csi.vsphere.vmware.com nodeID:k8s-node4 topologyKeys:[failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]]]]
    k8s-node5 map[drivers:[map[name:csi.vsphere.vmware.com nodeID:k8s-node5 topologyKeys:[failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]]]]

results matching ""

    No results matching ""