vSphere CSI Driver - Prerequisites

Compatible vSphere and ESXi versions

VMware's Cloud Native Storage (CNS) solution is delivered on vSphere version 6.7U3 and above. The volume driver interacts with CNS and hence only connects to vSphere 6.7U3 and above. When upgrading an older version of vSphere to 6.7U3 or above, make sure to upgrade the individual ESXi hosts that are part of the cluster to 6.7U3 or above as well. Note that the vSphere versions and ESXi versions should match.

vSphere Roles and Privileges

The vSphere user for CSI driver requires a set of privileges to perform Cloud Native Storage operations.

To know how to create and assign a role, refer the vSphere Security documentation.

The following roles need to be created with sets of privileges.

Role Privileges for the role Required on
CNS-DATASTORE ROLE-CNS-DATASTORE
govc role.ls CNS-DATASTORE
Datastore.FileManagement
System.Anonymous
System.Read
System.View
Shared datastores where persistent volumes need to be provisioned.
CNS-HOST-CONFIG-STORAGE ROLE-CNS-HOST-CONFIG-STORAGE
% govc role.ls CNS-HOST-CONFIG-STORAGE
Host.Config.Storage
System.Anonymous
System.Read
System.View
Required on vSAN file service enabled vSAN cluster. Required for file volume only.
CNS-VM ROLE-CNS-VM
% govc role.ls CNS-VM
System.Anonymous
System.Read
System.View
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddRemoveDevice
All node VMs.
CNS-SEARCH-AND-SPBM ROLE-CNS-SEARCH-AND-SPBM
% govc role.ls CNS-SEARCH-AND-SPBM
Cns.Searchable
StorageProfile.View
System.Anonymous
System.Read
System.View
Root vCenter Server.
ReadOnly This role is already available in the vCenter.
% govc role.ls ReadOnly
System.Anonymous
System.Read
Users with the ReadOnly role for an object are allowed to view the state of the object and details about the object.

For example, users with this role can find the shared datastores accessible to all node VMs.

For zone and topology-aware environments, all ancestors of node VMs such as a host, cluster, and datacenter must have the ReadOnly role set for the vSphere user configured to use the CSI driver and CPI.
This is required to allow reading tags and categories to prepare the nodes' topology.

Roles need to be assigned to the vSphere objects participating in the Cloud Native Storage environment.

To understand roles assignment to vSphere objects, consider we have following vSphere inventory.

sc2-rdops-vm06-dhcp-215-129.eng.vmware.com (vCenter Server)
|
|- datacenter (Data Center)
    |
    |-vSAN-cluster (cluster)
      |
      |-10.192.209.1 (ESXi Host)
      | |
      | |-k8s-master (node-vm)
      |
      |-10.192.211.250 (ESXi Host)
      | |
      | |-k8s-node1 (node-vm)
      |
      |-10.192.217.166 (ESXi Host)
      | |
      | |-k8s-node2 (node-vm)
      | |
      |-10.192.218.26 (ESXi Host)
      | |
      | |-k8s-node3 (node-vm)

Consider each host has the following shared datastores along with some local VMFS datastores.

  • shared-vmfs
  • shared-nfs
  • vsanDatastore

Considering the above inventory, roles should be assigned as specified below:

Role Usage
ReadOnly READ-ONLY-USAGE
CNS-HOST-CONFIG-STORAGE HOST-CONFIG-STORAGE-USAGE
CNS-DATASTORE CNS-DATASTORE-USAGE
CNS-VM CNS-VM-USAGE
CNS-SEARCH-AND-SPBM CNS-SEARCH-AND-SPBM-USAGE

Setting up the management network

By default, CPI and CSI Pods are scheduled on k8s master nodes. In this case, for non-topology aware Kubernetes clusters, it is sufficient to provide the k8s master node(s) credentials to the vCenter that this cluster runs on.

For topology-aware clusters, every k8s node needs to discover its topology by communicating with the vCenter. This is needed to utilize the topology-aware provisioning and late binding feature.

Refer to the Deployment with Zones to understand how to provide vCenter credentials access to Kubernetes nodes.

Virtual Machine Configuration

Make sure to configure all the VMs that form the Kubernetes cluster with the following:

  • We recommend using the VMware Paravirtual SCSI controller for Primary Disk on the Node VMs.
  • Set the disk.EnableUUID parameter to TRUE for each node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.
    • This can be done on the VirtualCenter User Interface by right-clicking on the VM → Edit Settings → VM Options → Advanced → Edit Configuration.
  • VM Hardware version must be 15 or higher.
    • This can be done on the VirtualCenter User Interface by right-clicking on the VM → Compatibility → Upgrade VM Compatibility.

The VMs can also be configured by using the govc command-line tool.

  • Install govc on your devbox/workstation.
  • Get the VM Paths

      $ export GOVC_INSECURE=1
      $ export GOVC_URL='https://<VC_Admin_User>:<VC_Admin_Passwd>@<VC_IP>'
    
      $ govc ls
      /<datacenter-name>/vm
      /<datacenter-name>/network
      /<datacenter-name>/host
      /<datacenter-name>/datastore
    
      // To retrieve all Node VMs
      $ govc ls /<datacenter-name>/vm
      /<datacenter-name>/vm/<vm-name1>
      /<datacenter-name>/vm/<vm-name2>
      /<datacenter-name>/vm/<vm-name3>
      /<datacenter-name>/vm/<vm-name4>
      /<datacenter-name>/vm/<vm-name5>
    
  • Enable disk UUID

    Run the below command for all Node VMs that are part of the Kubernetes cluster.

      govc vm.change -vm '/<datacenter-name>/vm/<vm-name1>' -e="disk.enableUUID=1"
    
  • Upgrade VM hardware version of node VMs to 15 or higher.

    Run the below command for all Node VMs that are part of the Kubernetes cluster.

      govc vm.upgrade -version=15 -vm '/<datacenter-name>/vm/<vm-name1>'
    

vSphere Cloud Provider Interface (CPI)

vSphere CSI driver needs the ProviderID field to be set for all nodes.

This can be done by installing vSphere Cloud Provider Interface (CPI) on your k8s cluster.

Before installing CPI, verify that all nodes are tainted with "node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule".

$ kubectl describe nodes | egrep "Taints:|Name:"
Name:               k8s-master
Taints:             node-role.kubernetes.io/master:NoSchedule
Name:               k8s-node1
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Name:               k8s-node2
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Name:               k8s-node3
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Name:               k8s-node4
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

When the kubelet is started with an “external” cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

Follow the steps described under “Install the vSphere Cloud Provider Interface” in https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md to deploy CPI.

Verify ProviderID is set for all nodes.

$ kubectl describe nodes | grep "ProviderID"
ProviderID: vsphere://<provider-id1>
ProviderID: vsphere://<provider-id2>
ProviderID: vsphere://<provider-id3>
ProviderID: vsphere://<provider-id4>
ProviderID: vsphere://<provider-id5>

results matching ""

    No results matching ""