vSphere CSI Driver - Prerequisites

Compatible vSphere and ESXi versions

VMware's Cloud Native Storage (CNS) solution is delivered on vSphere version 6.7U3 and above. The volume driver interacts with CNS and hence only connects to vSphere 6.7U3 and above. When upgrading an older version of vSphere to 6.7U3 or above, make sure to upgrade the individual ESXi hosts that are part of the cluster to 6.7U3 or above as well. Note that the vSphere versions and ESXi versions should match.

vSphere Roles and Privileges

The vSphere user for CSI driver requires a set of privileges to perform Cloud Native Storage operations.

To know how to create and assign a role, refer the vSphere Security documentation.

The following roles need to be created with sets of privileges.

Role Privileges for the role Required on
govc role.ls CNS-DATASTORE
Shared datastores where persistent volumes need to be provisioned.
Required on vSAN file service enabled vSAN cluster. Required for file volume only.
% govc role.ls CNS-VM
All node VMs.
% govc role.ls CNS-SEARCH-AND-SPBM
Root vCenter Server.
ReadOnly This role is already available in the vCenter.
% govc role.ls ReadOnly
Users with the ReadOnly role for an object are allowed to view the state of the object and details about the object.

For example, users with this role can find the shared datastores accessible to all node VMs.

For zone and topology-aware environments, all ancestors of node VMs such as a host, cluster, and datacenter must have the ReadOnly role set for the vSphere user configured to use the CSI driver and CPI.
This is required to allow reading tags and categories to prepare the nodes' topology.

Roles need to be assigned to the vSphere objects participating in the Cloud Native Storage environment.

To understand roles assignment to vSphere objects, consider we have following vSphere inventory.

sc2-rdops-vm06-dhcp-215-129.eng.vmware.com (vCenter Server)
|- datacenter (Data Center)
    |-vSAN-cluster (cluster)
      |- (ESXi Host)
      | |
      | |-k8s-master (node-vm)
      |- (ESXi Host)
      | |
      | |-k8s-node1 (node-vm)
      |- (ESXi Host)
      | |
      | |-k8s-node2 (node-vm)
      | |
      |- (ESXi Host)
      | |
      | |-k8s-node3 (node-vm)

Consider each host has the following shared datastores along with some local VMFS datastores.

  • shared-vmfs
  • shared-nfs
  • vsanDatastore

Considering the above inventory, roles should be assigned as specified below:

Role Usage

Setting up the management network

By default, CPI and CSI Pods are scheduled on k8s master nodes. In this case, for non-topology aware Kubernetes clusters, it is sufficient to provide the k8s master node(s) credentials to the vCenter that this cluster runs on.

For topology-aware clusters, every k8s node needs to discover its topology by communicating with the vCenter. This is needed to utilize the topology-aware provisioning and late binding feature.

Refer to the Deployment with Zones to understand how to provide vCenter credentials access to Kubernetes nodes.

Virtual Machine Configuration

Make sure to configure all the VMs that form the Kubernetes cluster with the following:

  • We recommend using the VMware Paravirtual SCSI controller for Primary Disk on the Node VMs.
  • Set the disk.EnableUUID parameter to TRUE for each node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.
    • This can be done on the VirtualCenter User Interface by right-clicking on the VM → Edit Settings → VM Options → Advanced → Edit Configuration.
  • VM Hardware version must be 15 or higher.
    • This can be done on the VirtualCenter User Interface by right-clicking on the VM → Compatibility → Upgrade VM Compatibility.

The VMs can also be configured by using the govc command-line tool.

  • Install govc on your devbox/workstation.
  • Get the VM Paths

      $ export GOVC_INSECURE=1
      $ export GOVC_URL='https://<VC_Admin_User>:<VC_Admin_Passwd>@<VC_IP>'
      $ govc ls
      // To retrieve all Node VMs
      $ govc ls /<datacenter-name>/vm
  • Enable disk UUID

    Run the below command for all Node VMs that are part of the Kubernetes cluster.

      govc vm.change -vm '/<datacenter-name>/vm/<vm-name1>' -e="disk.enableUUID=1"
  • Upgrade VM hardware version of node VMs to 15 or higher.

    Run the below command for all Node VMs that are part of the Kubernetes cluster.

      govc vm.upgrade -version=15 -vm '/<datacenter-name>/vm/<vm-name1>'

vSphere Cloud Provider Interface (CPI)

Follow the steps described under “Install the vSphere Cloud Provider Interface” in https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md to deploy CPI.

Installation steps for vSphere CPI is briefly described here

Step-1: Taint nodes.

Before installing CPI, verify all nodes (including master nodes) are tainted with "node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule".

To taint nodes use following command

kubectl taint node <node-name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

When the kubelet is started with an “external” cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

Step-2: Create a cloud-config configmap of vSphere configuration. Note: This is used for CPI. There is separate secret required for vSphere CSI driver. Here is an example configuration file with dummy values:

tee /etc/kubernetes/vsphere.conf >/dev/null <<EOF
insecure-flag = "true"

[VirtualCenter "IP or FQDN"]
user = "username@vsphere.local"
password = "password"
port = "port"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."


Here in vsphere.conf file,

  • insecure-flag - should be set to true to use self-signed certificate for login.
  • VirtualCenter - section defines vCenter IP address / FQDN.
  • user - vCenter username.
  • password - password for vCenter user specified with user.
  • port - vCenter Server Port.
  • datacenters - list of all comma separated datacenter paths where kubernetes node VMs are present. When datacenter is located at the root, the name of datacenter is enough but when datacenter is placed in the folder, path needs to be specified as folder/datacenter-name. Please note since comma is used as a delimiter, the datacenter name itself must not contain a comma.
cd /etc/kubernetes
kubectl create configmap cloud-config --from-file=vsphere.conf --namespace=kube-system
kubectl get configmap cloud-config --namespace=kube-system
NAME           DATA   AGE
cloud-config   1      82s

Remove vsphere.conf file created at /etc/kubernetes/

Step-3: Create Roles, Roles Bindings, Service Account, Service and cloud-controller-manager Pod.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
kubectl apply -f https://github.com/kubernetes/cloud-provider-vsphere/raw/master/manifests/controller-manager/vsphere-cloud-controller-manager-ds.yaml

Step-4: Verify ProviderID is set for all nodes.

kubectl describe nodes | grep "ProviderID"
ProviderID: vsphere://<provider-id1>
ProviderID: vsphere://<provider-id2>
ProviderID: vsphere://<provider-id3>
ProviderID: vsphere://<provider-id4>
ProviderID: vsphere://<provider-id5>

vSphere CSI driver needs the ProviderID field to be set for all nodes.

results matching ""

    No results matching ""