TKG-S Setup with vCenter Server Network - Part 1

Mon, Nov 23, 2020 5 min read kubernetes, storage, vcenter, vsphere, vSAN

With the new vCenter 7.0.1 it is possible to deploy the Workload Management feature using the vCenter Server Network instead of NSX-T. I wanted to try it out immediately and this is my experience.

Prerequisites

A VMware vCenter 7.0.1 installed
A VMware ESXi 7.0.1 installed
Distributed Switch with the Distributed Port Group configured for:
- Management Network
- Workload Network
- Frontend Network
Every Network MUST have its Gateway
vSAN configured

Network Configurations

Management:
- subnet: 192.168.1.0/24
- gateway: 192.168.1.254
- dns: 192.168.1.254
- ntp: 192.168.1.254
- HAproxy management ip: 192.168.1.245
- dataplane API port: 5556
- SupervisorControlPlanes starting IP: 192.168.1.140
Workload:
- subnet: 192.168.140.0/24
- gateway: 192.168.140.254
- dns: 192.168.1.254
- workload IPs range: 192.168.140.2-192.168.140.253
- HAproxy workload IP: 192.168.140.1
Frontend
- subnet: 192.168.150.0/24
- gateway: 192.168.150.254
- dns: 192.168.1.254
- frontend IPs range: 192.168.150.1-192.168.150.253
- frontend IPs ranges with CIDR: 192.168.150.0/25,192.168.150.128/26,192.168.150.192/27,192.168.150.224/28,192.168.150.240/29,192.168.150.248/30,192.168.150.252/31
- HAproxy frontend IP: 192.168.150.1

Procedures

Content Library

Go to the Content Libraries menu

and create a new one

Select Subscribed content library and select when you prefer to download the content.
Use https://wp-content.vmware.com/v2/latest/lib.json as subscription url.

Select the storage and click NEXT

Check if all the data are correct and then click FINISH

Storage Policy

Go to the Policies and Profiles menu

Click on VM Storage Policies

and click CREATE. Select the right vCenter Server and choose a Name for the policy. Add a Description if needed. Then click NEXT

Click *Enable rules for “vSAN” storage and click NEXT

Select the vSAN policies. My setup is a single vSAN node.

Click on Advanced Policy Rules and configure what it’s needed. Then click NEXT

Check in the Storage compatibility page if the vsanDatastore is compatible with the choseb policies, then click NEXT

Review all the configurations, then click FINISH

VMware-HAproxy OVA as LoadBalancer

Without NSX-T there is the need to use something which manages the LoadBalancing configuration. We’ll use HAproxy. The HAproxy OVA is going to be deployed with the 3 networks: the management network, the workload network which is used to let the SupervisorControlPlanes communicate with the nodes of the guest clusters and also be used as real server and the frontend network used for the VIPs created by the kubernetes service type LoadBalancer. The HAproxy istance is configured by its dataplane api using the management IP. The frontend IPs ranges with CIDR are used for the anyIP kernel feature. The HAproxy will respond to the ARP requests for any IP address in those subnets.

Go to the Hosts and Clusters menu.

Right click on the cluster and then click Deploy OVF Template

Add https://cdn.haproxy.com/download/haproxy/vsphere/ova/vmware-haproxy-v0.1.8.ova as URL and then click NEXT

Select the Virtual machine name and the location for the virtual machine then click NEXT

Select a compute resource then click NEXT

Review the details then click NEXT

Accept all the license agreements and then click NEXT

Select Frontend Network and then click NEXT

Select the storage and then click NEXT

Select the source network and then click NEXT

Set the root password and add your CA. If you don’t have one, leave blank and it will be generated

Configure the network

Add the Load Balancing settings and click NEXT

Review the configurations and then click FINISH

Power On the haproxy VM

Workload Management

Go to the Workload Management menu

Click GET STARTED

Select vCenter Server Network and then click NEXT

Select the compatible cluster and then click NEXT

Select the size for control plane VM and then click NEXT

Select the storage policy to the Control Plane VMs

Configure load balancer for workloads created on the cluster

If the Server Certificate Authority used is a self-signed cert, retrieve that data from the VM with the follow powershell snippet

  $vc = "10.174.71.163"
  $vc_user = "[email protected]"
  $vc_password = "Admin!23"
  Connect-VIServer -User $vc_user -Password $vc_password -Server $vc
  $VMname = "haproxy-demo"
  $AdvancedSettingName = "guestinfo.dataplaneapi.cacert"
  $Base64cert = get-vm $VMname |Get-AdvancedSetting -Name $AdvancedSettingName
  while ([string]::IsNullOrEmpty($Base64cert.Value)) {
  Write-Host "Waiting for CA Cert Generation... This may take a under 5-10
  minutes as the VM needs to boot and generate the CA Cert (if you haven't provided one already)."
  $Base64cert = get-vm $VMname |Get-AdvancedSetting -Name
  $AdvancedSettingName
  Start-sleep -seconds 2
  }
  Write-Host "CA Cert Found... Converting from BASE64"
  $cert = [Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($Base64cert.Value))
  Write-Host $cert