vSphere CSI Driver - Volume Topology

Prerequisite : Enable topology in the Kubernetes Cluster. Follow steps mentioned in Deployment with Topology.

Deploy workloads using topology with immediate volume binding mode

When topology is enabled in the cluster, you can deploy a Kubernetes workload to a specific region or zone defined in the topology.

Use the sample workflow to provision and verify your workloads.

Procedure

  1. Create a StorageClass that defines zone and region mapping.

    To the StorageClass YAML file, add zone-a and region-1 in the allowedTopologies field.

    tee example-zone-sc.yaml >/dev/null <<'EOF'
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: example-vanilla-block-zone-sc
    provisioner: csi.vsphere.vmware.com
    allowedTopologies:
      - matchLabelExpressions:
          - key: failure-domain.beta.kubernetes.io/zone
            values:
             - zone-a
          - key: failure-domain.beta.kubernetes.io/region
            values:
              - region-1
    EOF
    

    Note: Here volumeBindingMode will be Immediate, as it is default when not specified.

    kubectl create -f example-zone-sc.yaml
    storageclass.storage.k8s.io/example-vanilla-block-zone-sc created
    
  2. Create a PersistenceVolumeClaim.

    tee example-zone-pvc.yaml >/dev/null <<'EOF'
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: example-vanilla-block-zone-pvc
    spec:
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 5Gi
       storageClassName: example-vanilla-block-zone-sc
     EOF
    
     kubectl create -f example-zone-pvc.yaml
     persistentvolumeclaim/example-vanilla-block-zone-pvc created
    
  3. Verify that a volume is created for the PersistentVolumeClaim.

     kubectl get pvc example-vanilla-block-zone-pvc
     NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
     example-vanilla-block-zone-pvc   Bound    pvc-5b340a9b-a990-11e9-b26e-005056a04307   5Gi        RWO            example-vanilla-block-zone-sc   58s
    
     kubectl get pvc example-vanilla-block-zone-pvc
     NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
     example-vanilla-block-zone-pvc   Bound    pvc-5b340a9b-a990-11e9-b26e-005056a04307   5Gi        RWO            example-vanilla-block-zone-sc   91s
    
  4. Verify that the persistent volume is provisioned with the Node Affinity rules containing zone and region specified in the StorageClass.

     kubectl describe pv pvc-5b340a9b-a990-11e9-b26e-005056a04307
     Name:              pvc-5b340a9b-a990-11e9-b26e-005056a04307
     Labels:            <none>
     Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
     Finalizers:        [kubernetes.io/pv-protection]
     StorageClass:      example-vanilla-block-zone-sc
     Status:            Bound
     Claim:             default/example-vanilla-block-zone-pvc
     Reclaim Policy:    Delete
     Access Modes:      RWO
     VolumeMode:        Filesystem
     Capacity:          5Gi
     Node Affinity:
       Required Terms:
         Term 0:        failure-domain.beta.kubernetes.io/zone in [zone-a]
                        failure-domain.beta.kubernetes.io/region in [region-1]
     Message:  
     Source:
         Type:              CSI (a Container Storage Interface (CSI) volume source)
         Driver:            csi.vsphere.vmware.com
         VolumeHandle:      8f1f5e44-fafa-4404-91f7-d7a9bfd30e16
         ReadOnly:          false
         VolumeAttributes:      fstype=
                                storage.kubernetes.io/csiProvisionerIdentity=1563472725085-8081-csi.vsphere.vmware.com
                                type=vSphere CNS Block Volume
     Events:                <none>
    
  5. Create a pod.

     tee example-zone-pod.yaml >/dev/null <<'EOF'
     apiVersion: v1
     kind: Pod
     metadata:
       name: example-vanilla-block-zone-pod
     spec:
       containers:
       - name: test-container
         image: gcr.io/google_containers/busybox:1.24
         command: ["/bin/sh", "-c", "echo
      'hello' > /mnt/volume1/index.html  && chmod o+rX /mnt
     /mnt/volume1/index.html && while true ; do sleep 2 ; done"]
         volumeMounts:
         - name: test-volume
           mountPath: /mnt/volume1
       restartPolicy: Never
       volumes:
       - name: test-volume
         persistentVolumeClaim:
           claimName: example-vanilla-block-zone-pvc
     EOF
    
     kubectl create -f example-zone-pod.yaml
     pod/example-vanilla-block-zone-pod created
    

    Pod is scheduled on the node k8s-node1 which belongs to zone: "zone-a" and region: "region-1"

     kubectl describe pod example-vanilla-block-zone-pod | egrep "Node:"
     Node:               k8s-node1/10.160.78.255
    

Deploy workloads using topology with WaitForFirstConsumer volume binding mode

The vSphere CSI driver supports topology-aware volume provisioning with WaitForFirstConsumer

Topology-aware provisioning allows Kubernetes to make intelligent decisions and find the best place to dynamically provision a volume for a pod. In multi-zone clusters, volumes are provisioned in an appropriate zone that can run your pod, allowing you to easily deploy and scale your stateful workloads across failure domains to provide high availability and fault tolerance.

external-provisioner must be deployed with the --strict-topology arguments.

This argument controls which topology information is passed to CreateVolumeRequest.AccessibilityRequirements in case of a delayed binding.

For information on how this option changes the result, see the table at https://github.com/kubernetes-csi/external-provisioner#topology-support. This option has no effect if the Topology feature is disabled or the Immediate volume binding mode is used.

Procedure

  1. Enable the Topology feature.

    Create a StorageClass with the volumeBindingMode parameter set to WaitForFirstConsumer.

     tee topology-aware-standard.yaml >/dev/null <<'EOF'
     apiVersion: v1
     kind: StorageClass
     apiVersion: storage.k8s.io/v1
     metadata:
       name: topology-aware-standard
     provisioner: csi.vsphere.vmware.com
     volumeBindingMode: WaitForFirstConsumer
     EOF
    
     kubectl create -f topology-aware-standard.yaml
     storageclass.storage.k8s.io/topology-aware-standard created
    

    This new setting instructs the volume provisioner, instead of creating a volume immediately, to wait until a pod using an associated PVC runs through scheduling. Note that in the previous StorageClass, failure-domain.beta.kubernetes.io/zone and failure-domain.beta.kubernetes.io/region were specified in the allowedTopologies entry. You do not need to specify them again, as pod policies now drive the decision of which zone to use for a volume provisioning.

  2. Create a pod and PVC using the StorageClass created previously.

    The following example demonstrates multiple pod constraints and scheduling policies.

     tee topology-aware-statefulset.yaml >/dev/null <<'EOF'
     ---
     apiVersion: apps/v1
     kind: StatefulSet
     metadata:
       name: web
     spec:
       replicas: 2
       selector:
         matchLabels:
           app: nginx
       serviceName: nginx
       template:
         metadata:
           labels:
             app: nginx
         spec:
           affinity:
             nodeAffinity:
               requiredDuringSchedulingIgnoredDuringExecution:
                 nodeSelectorTerms:
                   -
                     matchExpressions:
                       -
                         key: failure-domain.beta.kubernetes.io/zone
                         operator: In
                         values:
                           - zone-a
                           - zone-b
             podAntiAffinity:
               requiredDuringSchedulingIgnoredDuringExecution:
                 -
                   labelSelector:
                     matchExpressions:
                       -
                         key: app
                         operator: In
                         values:
                           - nginx
                   topologyKey: failure-domain.beta.kubernetes.io/zone
           containers:
             - name: nginx
               image: gcr.io/google_containers/nginx-slim:0.8
               ports:
                 - containerPort: 80
                   name: web
               volumeMounts:
                 - name: www
                   mountPath: /usr/share/nginx/html
                 - name: logs
                   mountPath: /logs
       volumeClaimTemplates:
         - metadata:
             name: www
           spec:
             accessModes: [ "ReadWriteOnce" ]
             storageClassName: topology-aware-standard
             resources:
               requests:
                 storage: 5Gi
         - metadata:
             name: logs
           spec:
             accessModes: [ "ReadWriteOnce" ]
             storageClassName: topology-aware-standard
             resources:
               requests:
                 storage: 1Gi
     EOF
    
     kubectl create -f topology-aware-statefulset.yaml
     statefulset.apps/web created
    
  3. Verify statefulset is up and running.

     kubectl get statefulset
     NAME   READY   AGE
     web    2/2     5m1s
    
  4. Review your pods and your nods.

    Pods are created in the zone-a and zone-b specified in the nodeAffinity entry. web-0 is scheduled on the node k8s-node3, which belongs to zone-b. web-1 is scheduled on the node k8s-node1, which belongs to zone-a.

     kubectl get pods -o json | egrep "hostname|nodeName|claimName"
                     "hostname": "web-0",
                     "nodeName": "k8s-node3",
                                 "claimName": "www-web-0"
                                 "claimName": "logs-web-0"
                     "hostname": "web-1",
                     "nodeName": "k8s-node1",
                                 "claimName": "www-web-1"
                                 "claimName": "logs-web-1"
    
     kubectl get nodes k8s-node3 k8s-node1 -L failure-domain.beta.kubernetes.io/zone -L failure-domain.beta.kubernetes.io/region --no-headers
     k8s-node3   Ready   <none>   25h   v1.14.2   zone-b   region-1
     k8s-node1   Ready   <none>   25h   v1.14.2   zone-a   region-1
    
  5. Verify volumes are provisioned in zones according to the policies set by the pod.

     $ kubectl describe pvc www-web-0 | egrep "volume.kubernetes.io/selected-node"
                    volume.kubernetes.io/selected-node: k8s-node3
    
     $ kubectl describe pvc logs-web-0 | egrep "volume.kubernetes.io/selected-node"
                    volume.kubernetes.io/selected-node: k8s-node3
    
     $ kubectl describe pvc www-web-1 | egrep "volume.kubernetes.io/selected-node"
                    volume.kubernetes.io/selected-node: k8s-node1
    
     $ kubectl describe pvc logs-web-1 | egrep "volume.kubernetes.io/selected-node"
                    volume.kubernetes.io/selected-node: k8s-node1
    
     $ kubectl get pv -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.claimRef.name}{"\t"}{.spec.nodeAffinity}{"\n"}{end}'
     pvc-2253dc52-a9ed-11e9-b26e-005056a04307    www-web-0    map[required:map[nodeSelectorTerms:[map[matchExpressions:[map[key:failure-domain.beta.kubernetes.io/region operator:In values:[region-1]] map[operator:In values:[zone-b] key:failure-domain.beta.kubernetes.io/zone]]]]]]
     pvc-22575240-a9ed-11e9-b26e-005056a04307    logs-web-0    map[required:map[nodeSelectorTerms:[map[matchExpressions:[map[key:failure-domain.beta.kubernetes.io/zone operator:In values:[zone-b]] map[key:failure-domain.beta.kubernetes.io/region operator:In values:[region-1]]]]]]]
     pvc-3c963150-a9ed-11e9-b26e-005056a04307    www-web-1    map[required:map[nodeSelectorTerms:[map[matchExpressions:[map[key:failure-domain.beta.kubernetes.io/zone operator:In values:[zone-a]] map[operator:In values:[region-1] key:failure-domain.beta.kubernetes.io/region]]]]]]
     pvc-3c98978f-a9ed-11e9-b26e-005056a04307    logs-web-1    map[required:map[nodeSelectorTerms:[map[matchExpressions:[map[key:failure-domain.beta.kubernetes.io/zone operator:In values:[zone-a]] map[key:failure-domain.beta.kubernetes.io/region operator:In values:[region-1]]]]]]]
    
  6. If required, specify allowedTopologies.

    When a cluster operator specifies the WaitForFirstConsumer volume binding mode, it is no longer necessary to restrict provisioning to specific topologies in most situations. However, if required, you can specify allowedTopologies.

    The following example demonstrates how to restrict the topology to specific zone.

     kind: StorageClass
     apiVersion: storage.k8s.io/v1
     metadata:
       name: example-vanilla-block-sc
     provisioner: csi.vsphere.vmware.com
     volumeBindingMode: WaitForFirstConsumer
     allowedTopologies:
       - matchLabelExpressions:
           - key: failure-domain.beta.kubernetes.io/zone
             values:
               - zone-b
           - key: failure-domain.beta.kubernetes.io/region
             values:
               - region-1
    

results matching ""

    No results matching ""