Kubernetes Cluster API Provider IBM Cloud

IBM Cloud


Kubernetes-native declarative infrastructure for IBM Cloud.

What is the Cluster API Provider IBM Cloud

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true IBM Cloud hybrid deployments of Kubernetes. It is built atop the lessons learned from previous cluster managers such as kops and kubicorn.

Quick Start

Check out the getting started section to create your first Kubernetes cluster on IBM Cloud using Cluster API.

Tilt-based development environment

See developer guide section for details.

Documentation

Please see our Book for in-depth user documentation.

Additional docs can be found in the /docs directory, and the index is here.

Getting involved and contributing

Are you interested in contributing to cluster-api-provider-ibmcloud? We, the maintainers and community, would love your suggestions, contributions, and help! Also, the maintainers can be contacted at any time to learn more about how to get involved.

In the interest of getting more new people involved, we tag issues with good first issue. These are typically issues that have smaller scope but are good ways to start to get acquainted with the codebase.

We also encourage all active community participants to act as if they are maintainers, even if you don’t have “official” write permissions. This is a community effort, we are here to serve the Kubernetes community. If you have an active interest and you want to get involved, you have real power! Don’t assume that the only people who can get things done around here are the “maintainers”.

We also would love to add more “official” maintainers, so show us what you can do!

This repository uses the Kubernetes bots. See a full list of the commands here.

Join us

The community holds a weekly meeting every Friday at 09:00 (IST) / 03:30 (UTC) on Zoom. Subscribe to the SIG Cluster Lifecycle Google Group for access to documents and calendars

Other ways to communicate with the contributors

Please check in with us in the #cluster-api-ibmcloud channel on Slack.

Github issues

Bugs

If you think you have found a bug please follow the instructions below.

  • Please spend a small amount of time giving due diligence to the issue tracker. Your issue might be a duplicate.
  • Get the logs from the cluster controllers. Please paste this into your issue.
  • Open a bug report.
  • Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.

Tracking new features

We also use the issue tracker to track features. If you have an idea for a feature, or think you can help Cluster API Provider IBMCloud become even more awesome, then follow the steps below.

  • Open a feature request.
  • Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.
  • Clearly define the use case, using concrete examples. EG: I type this and cluster-api-provider-ibmcloud does that.
  • Some of our larger features will require some design. If you would like to include a technical design for your feature please include it in the issue.
  • After the new feature is well understood, and the design agreed upon we can start coding the feature. We would love for you to code it. So please open up a WIP (work in progress) pull request, and happy coding.

Getting Started

For prerequisites, check the respective sections for VPC and PowerVS

Now that we’ve got all the prerequisites in place, let’s create a Kubernetes cluster and transform it into a management cluster using clusterctl.

Provision local boostrap management cluster:

  1. Create simple, local bootstrap cluster with a control-plane and worker node

    Using kind:

    ~ kind create cluster --name my-bootstrap --config bootstrap.yaml
    

    Example bootstrap.yaml:

    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    nodes:
       - role: control-plane
       - role: worker
    

    Make sure the nodes are in Ready state before moving on.

    ~ kubectl get nodes
    NAME                         STATUS   ROLES                  AGE   VERSION
    my-bootstrap-control-plane   Ready    control-plane,master   46h   v1.20.2
    my-bootstrap-worker          Ready    <none>                 46h   v1.20.2
    
  2. Set workload cluster environment variables

    Make sure these value reflects your API Key for your target VPC environment or PowerVS environment in IBM Cloud.

    export IBMCLOUD_API_KEY=<YOUR_API_KEY>
    

    For enabling debug level logs for the controller, set the LOGLEVEL environment variable(defaults to 0).

    export LOGLEVEL=5
    

    Note: Refer Regions-Zones Mapping for more information.

    Note: To deploy workload cluster with Power VS cloud controller manager which is currently in experimental stage. Set the POWERVS_PROVIDER_ID_FORMAT environmental variable

      export POWERVS_PROVIDER_ID_FORMAT=v2
    

    Note: To deploy workload cluster with Custom Service Endpoint, Set SERVICE_ENDPOINT environmental variable in semi-colon separated format:

    `${ServiceRegion1}:${ServiceID1}=${URL1},${ServiceID2}=${URL2};${ServiceRegion2}:${ServiceID1}=${URL1...}`.
    

    Supported ServiceIDs include - vpc, powervs, rc

      export SERVICE_ENDPOINT=us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com
    
  3. Initialize local bootstrap cluster as a management cluster

    When executed for the first time, the following command accepts the infrastructure provider as an input to install. clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers, thereby converting it into a management cluster which will be used to provision a workload cluster in IBM Cloud.

    ~ clusterctl init --infrastructure ibmcloud:<TAG>
    

    Note: If the latest release version of the provider is available, specifying TAG can be avoided. In other cases, you can specify any prerelease version compatible with the supported API contract as the TAG.
    Example: clusterctl init --infrastructure ibmcloud:v0.2.0-alpha.5

    Output:

    Fetching providers
    Installing cert-manager Version="v1.5.3"
    Waiting for cert-manager to be available...
    Installing Provider="cluster-api" Version="v0.4.4" TargetNamespace="capi-system"
    Installing Provider="bootstrap-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-bootstrap-system"
    Installing Provider="control-plane-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-control-plane-system"
    Installing Provider="infrastructure-ibmcloud" Version="v0.1.0-alpha.2" TargetNamespace="capi-ibmcloud-system"
    
    Your management cluster has been initialized successfully!
    
    You can now create your first workload cluster by running the following:
    
    clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
    
  4. Once the management cluster is ready with the required providers up and running, proceed to provisioning the workload cluster. Check the respective sections for VPC and PowerVS to deploy the cluster.

IBM Cloud Machine Images for CAPIBM Clusters

CAPIBM requires a “machine image” containing pre-installed, matching versions of kubeadm and kubelet. Machine image is required during the cluster creation in the AWSMachineTemplate spec.

Pre-built public Images are published by the maintainers regularly for each new Kubernetes version.

Note: These images are only for the test purpose

VPC Images

RegionBucketObjectKubernetes Version
us-southpower-oss-bucketcapibm-vpc-ubuntu-2004-kube-v1-25-2.qcow21.25.2

Note: These images are built using the image-builder tool and more information can be found here

Power VS Images

RegionBucketObjectKubernetes Version
us-southpower-oss-bucketcapibm-powervs-centos-streams8-1-25-1.ova.gz1.25.1
us-southpower-oss-bucketcapibm-powervs-centos-streams8-1-24-2.ova.gz1.24.2
us-southpower-oss-bucketcapibm-powervs-centos-streams8-1-23-5.ova.gz1.23.5
us-southpower-oss-bucketcapibm-powervs-centos-streams8-1-22-4.ova.gz1.22.4

Note: These images are built using the image-builder tool and more information can be found here

Topics

This section contains information about using IBM Cloud features with Cluster API Provider IBM Cloud.

VPC Cluster

Contents

Prerequisites

  1. Install kubectl (see here). Because kustomize was included into kubectl and it’s used by cluster-api-provider-ibmcloud in generating yaml files, so version 1.14.0+ of kubectl is required, see integrate kustomize into kubectl for more info.
  2. You can use either VM, container or existing Kubernetes cluster act as the bootstrap cluster.
    • If you want to use container, install kind. This is preferred.
    • If you want to use VM, install minikube, version 0.30.0 or greater.
    • If you want to use existing Kubernetes cluster, prepare your kubeconfig.
  3. Install a driver if you are using minikube. For Linux, we recommend kvm2. For MacOS, we recommend VirtualBox.
  4. An appropriately configured Go development environment
  5. Install clusterctl tool (see here)

Build workload cluster image:

  1. Build a qcow2 image suitable for use as a Kubernetes cluster machine as detailed in the image builder book.

    Note: Rename the output image to add the .qcow2 extension. This is required by the next step.

  2. Upload the VPC Gen2 custom image to IBM Cloud following this section or the detailed explainations in the VPC documentation.

Uploading an image to the IBM Cloud

Build the Ubuntu image as described in the previous VPC section. Make sure to build the qcow2 version by following the instructions for raw image build.

Since the IBM Cloud does not support dots before the qcow2 extension, rename the file as follows:

ubuntu-2004-ibmcloud-kube-v1-23-4.qcow2

Upload VM image:

  1. Create an IBM COS instance
  2. Create a bucket in the COS instance.
  3. Upload the image
    1. Upload via aspera
      • Install the browser extension for Aspera
      • Downloading the Aspera tool
      • Selecting the image via Aspera dialog
      • Upload the image via aspera
    2. Using minio cli
      • Install minio cli
      • Creating a service credential with hmac=true for the bucket
      • Example upload for eu-de:
        mc alias set uploadcos https://s3.eu-de.cloud-object-storage.appdomain.cloud <hmac access id> <hmac secret key>
        
        mc cp <image-name>.qcow2 uploadcos/<my-bucket-name>
        

Add VM image to VPC

  1. Make sure you have editor rights for all/most VPC services
  2. Add additional read rights for:
src: service VPC Infrastructure Services resourceType equals image
target: serviceInstance string equals <your-Cloud-Object Storage-VM-plain-name>

Add write rights for:

Service VPC Infrastructure Services in Resource_group <your_resource_group_or_account> resourceType equals image
target: service Cloud object storage in resource_group <your_resource_group_or_account>
  1. Go to https://cloud.ibm.com/vpc-ext/provision/customImage
  • Fill in imagename, resource group or account
  • Choice box: Cloud Object Storage
  • Set Filter: <your_cos_plain_name> <eu-de_or_other> <your_vm_bucket>
  • Choice box: Select your image
  • Select base os (ubuntu-20-04-amd64 for example)
  • Click Create Image

Now you can provision a VM with your own VM image. Then please continue with creating a cluster.

Make sure you take the ImageID from your VM image. The ImageID can be determined using ibmcloud cli. In addition, the Kubernetes version must be set to match the image. In this example:

v1.23.4

Provision workload Cluster in IBM Cloud VPC

Now that we have a management cluster ready, you can create your workload cluster by following the steps below.

  1. Using clusterctl, render the yaml through templates and deploy the cluster

    Note: the IBMVPC_IMAGE_ID value below should reflect the ID of the custom qcow2 image

    IBMVPC_REGION=us-south \
    IBMVPC_ZONE=us-south-1 \
    IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
    IBMVPC_NAME=ibm-vpc-0 \
    IBMVPC_IMAGE_ID=r134-ea84bbec-7986-4ff5-8489-d9ec34611dd4 \
    IBMVPC_PROFILE=bx2-4x16 \
    IBMVPC_SSHKEY_ID=r134-2a82b725-e570-43d3-8b23-9539e8641944 \
    clusterctl generate cluster ibm-vpc-0 --kubernetes-version v1.19.9 \
    --target-namespace default \
    --control-plane-machine-count=1 \
    --worker-machine-count=2 | kubectl apply -f -
    

    Output:

    cluster.cluster.x-k8s.io/ibm-vpc-5 created
    ibmvpccluster.infrastructure.cluster.x-k8s.io/ibm-vpc-5 created
    kubeadmcontrolplane.controlplane.cluster.x-k8s.io/ibm-vpc-5-control-plane created
    ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-5-control-plane created
    machinedeployment.cluster.x-k8s.io/ibm-vpc-5-md-0 created
    ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-5-md-0 created
    kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/ibm-vpc-5-md-0 created
    

    Note: Refer below for more detailed information on VPC variables.

  2. Check the state of the provisioned cluster and machine objects within the local management cluster

    Clusters

    ~ kubectl get clusters
    NAME         PHASE
    ibm-vpc-0    Provisioned
    

    Kubeadm Control Plane

    ~ kubectl get kubeadmcontrolplane
    NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
    ibm-vpc-0-control-plane    true          true                   v1.19.9   1          1       1
    

    Machines

    ~ kubectl get machines
    ibm-vpc-0-control-plane-vzz47     ibmvpc://ibm-vpc-0/ibm-vpc-0-control-plane-rg6xv   Running        v1.19.9
    ibm-vpc-0-md-0-5444cfcbcd-6gg5z   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-dbxb7            Running        v1.19.9
    ibm-vpc-0-md-0-5444cfcbcd-7kr9x   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-k7blr            Running        v1.19.9
    
  3. Deploy Container Network Interface (CNI)

    Example: calico

    ~ clusterctl get kubeconfig ibm-vpc-0 > ~/.kube/ibm-vpc-0
    ~ export KUBECONFIG=~/.kube/ibm-vpc-0
    ~ kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
    
  4. Check the state of the newly provisioned cluster within IBM Cloud

    ~ kubectl get nodes
    NAME                             STATUS   ROLES    AGE   VERSION
    ibm-vpc-0-control-plane-rg6xv    Ready    master   41h   v1.18.15
    ibm-vpc-0-md-0-4dc5c             Ready    <none>   41h   v1.18.15
    ibm-vpc-0-md-0-dbxb7             Ready    <none>   20h   v1.18.15
    

Provision workload Cluster with Load Balancer in IBM Cloud VPC

This feature is currently in experimental stage

Deploy VPC cluster with Load Balancer

    IBMVPC_REGION=us-south \
    IBMVPC_ZONE=us-south-1 \
    IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
    IBMVPC_NAME=ibm-vpc-0 \
    IBMVPC_IMAGE_ID=r134-ea84bbec-7986-4ff5-8489-d9ec34611dd4 \
    IBMVPC_PROFILE=bx2-4x16 \
    IBMVPC_SSHKEY_ID=r134-2a82b725-e570-43d3-8b23-9539e8641944 \
    clusterctl generate cluster ibm-vpc-0 --kubernetes-version v1.22.0 \
    --target-namespace default \
    --control-plane-machine-count=3 \
    --worker-machine-count=1 \
    --flavor=load-balancer | kubectl apply -f -

Power VS Cluster

Contents

Prerequisites

  1. Install kubectl (see here). Because kustomize was included into kubectl and it’s used by cluster-api-provider-ibmcloud in generating yaml files, so version 1.14.0+ of kubectl is required, see integrate kustomize into kubectl for more info.
  2. You can use either VM, container or existing Kubernetes cluster act as the bootstrap cluster.
    • If you want to use container, install kind. This is preferred.
    • If you want to use VM, install minikube, version 0.30.0 or greater.
    • If you want to use existing Kubernetes cluster, prepare your kubeconfig.
  3. Install a driver if you are using minikube. For Linux, we recommend kvm2. For MacOS, we recommend VirtualBox.
  4. An appropriately configured Go development environment
  5. Install clusterctl tool (see here)
  6. Install pvsadm tool (see here)
  7. Install ibmcloud tool (see here)

PowerVS Prerequisites

Create an IBM Cloud account.

If you don’t already have one, you need a paid IBM Cloud account to create your Power Systems Virtual Server instance. To create an account, go to: cloud.ibm.com.

Create an IBM Cloud account API key

Please refer to the following documentation to create an API key.

Create Power Systems Virtual Server Service Instance

After you have an active IBM Cloud account, you can create a Power Systems Virtual Server service. To do so, perform the following steps:

  1. TO-DO

Create Network

A public network is required for your kubernetes cluster. Perform the following steps to create a public network for the Power Systems Virtual Server service instance created in the previous step.

  1. Create Public Network

    ~ ibmcloud pi network-create-public capi-test --dns-servers "8.8.8.8 9.9.9.9"
    

    Output:

    Network capi-test created.
                        
    ID                fea9ac26-693d-402b-b22f-aa3d90ed0a31   
    Name              capi-test  
    Type              pub-vlan   
    VLAN              2008   
    CIDR Block        192.168.150.96/29   
    IP Range          [192.168.150.98 192.168.150.102]   
    Public IP Range   [158.175.161.98 158.175.161.102]   
    Gateway           192.168.150.97   
    DNS               8.8.8.8, 9.9.9.9
    

Import the machine boot image:

$ export IBMCLOUD_API_KEY=<API_KEY>
$ pvsadm image import --pvs-instance-id <SERVICE_INSTANCE_ID> -b <BUCKETNAME> --object <OBJECT> --pvs-image-name <POWERVS_IMAGE_NAME> --bucket-region <REGION> --public-bucket

e.g:

$ pvsadm image import --pvs-instance-id 6d892c30-5387-4685-85d0-4999d9c22a8c -b power-oss-bucket --object capibm-powervs-centos-streams8-1-24-2.ova.gz --pvs-image-name capibm-powervs-centos-streams8-1-24-2 --bucket-region us-south --public-bucket

For more information about the images can be found at machine-images section

Provision workload cluster in IBM Cloud PowerVS

Now that we have a management cluster ready, you can create your workload cluster by following the steps below.

  1. Create PowerVS network port

    ~ pvsadm create port --description "capi-port" --network <NETWORK_NAME> --instance-id <SERVICE_INSTANCE_ID>
    

    Output:

    I1125 15:24:20.581757 1548881 port.go:89] Successfully created a port, id: ac18ef17-8517-40e3-889d-4f246e9bd17e
    +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
    |     DESCRIPTION     | EXTERNALIP | HREF |    IPADDRESS    |    MACADDRESS     |                PORTID                | PVMINSTANCE | STATUS |
    +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
    | capi-port |            |      | 192.168.151.125 | fa:16:3e:34:2c:ef | 7eff02b5-040c-4934-957a-18209e65eca4 |             | DOWN   |
    +---------------------+------------+------+-----------------+-------------------+--------------------------------------+-------------+--------+
    
    ~ pvsadm get ports --instance-id <SERVICE_INSTANCE_ID> --network <NETWORK_NAME>
    

    Output:

    +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
    |    DESCRIPTION    |   EXTERNALIP    |    IPADDRESS    |    MACADDRESS     |                PORTID                | STATUS |
    +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
    | capi-port         | 158.175.162.125 | 192.168.151.125 | fa:16:3e:34:2c:ef | 7eff02b5-040c-4934-957a-18209e65eca4 | DOWN   |
    +-------------------+-----------------+-----------------+-------------------+--------------------------------------+--------+
    
  2. Use clusterctl to render the yaml through templates and deploy the cluster

    Note: To deploy workload cluster with Power VS cloud controller manager which is currently in experimental stage follow these steps.

    Note: the IBMPOWERVS_IMAGE_ID value below should reflect the ID of the custom qcow2 image, the kubernetes-version value below should reflect the kubernetes version of the custom qcow2 image.

    IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
    IBMPOWERVS_VIP="192.168.151.22" \
    IBMPOWERVS_VIP_EXTERNAL="158.175.162.22" \
    IBMPOWERVS_VIP_CIDR="29" \
    IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-8-1-22-4" \
    IBMPOWERVS_SERVICE_INSTANCE_ID="7845d372-d4e1-46b8-91fc-41051c984601" \
    IBMPOWERVS_NETWORK_NAME="capi-test-3" \
    clusterctl generate cluster ibm-powervs-1 --kubernetes-version v1.22.4 \
    --target-namespace default \
    --control-plane-machine-count=3 \
    --worker-machine-count=1 \
    --flavor=powervs | kubectl apply -f -
    

    Output:

    cluster.cluster.x-k8s.io/ibm-powervs-1 created
    ibmpowervscluster.infrastructure.cluster.x-k8s.io/ibm-powervs-1 created
    kubeadmcontrolplane.controlplane.cluster.x-k8s.io/ibm-powervs-1-control-plane created
    ibmpowervsmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-powervs-1-control-plane created
    machinedeployment.cluster.x-k8s.io/ibm-powervs-1-md-0 created
    ibmpowervsmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-powervs-1-md-0 created
    kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/ibm-powervs-1-md-0 created
    

    Additional parameters for modifying PowerVS Control-Plane spec

    IBMPOWERVS_CONTROL_PLANE_MEMORY
    IBMPOWERVS_CONTROL_PLANE_PROCESSORS
    IBMPOWERVS_CONTROL_PLANE_SYSTYPE
    IBMPOWERVS_CONTROL_PLANE_PROCTYPE
    

    Additional parameters for modifying PowerVS Compute node spec

    IBMPOWERVS_COMPUTE_MEMORY
    IBMPOWERVS_COMPUTE_PROCESSORS
    IBMPOWERVS_COMPUTE_SYSTYPE
    IBMPOWERVS_COMPUTE_PROCTYPE
    
  3. Check the state of the provisioned cluster and machine objects within the local management cluster

    Clusters

    ~ kubectl get clusters
    NAME         PHASE
    ibm-powervs-1    Provisioned
    

    Kubeadm Control Plane

    ~ kubectl get kubeadmcontrolplane
    NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
    ibm-powervs-1-control-plane    true          true                   v1.22.4   1          1       1
    

    Machines

    ~ kubectl get machines
    ibm-powervs-1-control-plane-vzz47     ibmpowervs://ibm-powervs-1/ibm-powervs-1-control-plane-rg6xv   Running        v1.22.4
    ibm-powervs-1-md-0-5444cfcbcd-6gg5z   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-dbxb7            Running        v1.22.4
    ibm-powervs-1-md-0-5444cfcbcd-7kr9x   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-k7blr            Running        v1.22.4
    
  4. Deploy Container Network Interface (CNI)

    Example: calico

    ~ clusterctl get kubeconfig ibm-powervs-1 > ~/.kube/ibm-powervs-1
    ~ export KUBECONFIG=~/.kube/ibm-powervs-1
    ~ kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
    
  5. Check the state of the newly provisioned cluster within IBM Cloud

    ~ kubectl get nodes
    NAME                             STATUS   ROLES    AGE   VERSION
    ibm-powervs-1-control-plane-rg6xv    Ready    master   41h   v1.22.4
    ibm-powervs-1-md-0-4dc5c             Ready    <none>   41h   v1.22.4
    ibm-powervs-1-md-0-dbxb7             Ready    <none>   20h   v1.22.4
    

IBM Power VS External Cloud Provider

This feature currently in experimental stage

Steps

Deploy Power VS cluster with IBM Power VS cloud provider

IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
IBMPOWERVS_VIP="192.168.151.22" \
IBMPOWERVS_VIP_EXTERNAL="158.175.162.22" \
IBMPOWERVS_VIP_CIDR="29" \
IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-8-1-22-4" \
IBMPOWERVS_SERVICE_INSTANCE_ID="7845d372-d4e1-46b8-91fc-41051c984601" \
IBMPOWERVS_NETWORK_NAME="capi-test-3" \
IBMACCOUNT_ID="ibm-accountid" \
IBMPOWERVS_REGION="powervs-region" \
IBMPOWERVS_ZONE="powervs-zone" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-powervs-1 --kubernetes-version v1.22.4 \
--target-namespace default \
--control-plane-machine-count=3 \
--worker-machine-count=1 \
--flavor=powervs-cloud-provider | kubectl apply -f -

When the cluster is created with above parameters, The IBM Power VS cloud provider will

  1. Initialize the node by fetching appropriate VM information such as IP, zone, region from Power Cloud.

Using Autoscaler to scale machines from 0 machine

The autoscaler project supports cluster-api. With this enhancement now the user can use cluster-api feature to scaling from 0 machine.

Settinng up the workload cluster

While creating a workload cluster, We need to set the below annotations to machinedeployment inorder to enable the autoscaling, This is one of the prerequisites for autoscaler.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: "${CLUSTER_NAME}-md-0"
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"

Setting up the cluster-autoscaler

  1. Clone the autoscaler repository
git clone https://github.com/kubernetes/autoscaler.git
  1. Build the autoscaler binary
cd cluster-autoscaler 
go build .
  1. Start the autoscaler
./cluster-autoscaler \
--cloud-provider=clusterapi \
--v=2 \
--namespace=default \
--max-nodes-total=30 \
--scale-down-delay-after-add=10s \
--scale-down-delay-after-delete=10s \
--scale-down-delay-after-failure=10s \
--scale-down-unneeded-time=5m \
--max-node-provision-time=30m \
--balance-similar-node-groups \
--expander=random \
--kubeconfig=<workload_cluster_kubeconfig> \
--cloud-config=<management_cluster_kubeconfig>

Note:

  1. autoscaler can be run in different ways the possible ways are described here.
  2. autoscaler supports various command line flags and more details about it can be found here.

Use case of cluster-autoscaler

  1. Create a workload cluster with 0 worker machines
  2. Create a sample workload
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: busybox
  name: busybox-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - command:
            - sh
            - -c
            - echo Container 1 is Running ; sleep 3600
          image: busybox
          imagePullPolicy: IfNotPresent
          name: busybox
          resources:
            requests:
              cpu: "0.2"
              memory: 3G
  1. Scale the deployment to create addition pods
kubectl scale --replicas=2 deployment/busybox-deployment 
  1. Obeserve the status of new pods
kubectl get pods                                        
NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-7c87788568-qhqdb   1/1     Running   0          48s
busybox-deployment-7c87788568-t26bb   0/1     Pending   0          5s
  1. On the management cluster verify that the new machine creation is being triggered by autoscaler
NAME                                        CLUSTER               NODENAME                                  PROVIDERID                                                                                      PHASE          AGE     VERSION
karthik-ibm-powervs-control-plane-smvf7     karthik-ibm-powervs   karthik-ibm-powervs-control-plane-pgwmz   ibmpowervs://osa/osa21/3229a-af54-4212-bf60-6202b6fd0a07/809cd0f2-7502-4112-bf44-84d178020d8a   Running        82m     v1.24.2
karthik-ibm-powervs-md-0-6b4d67ccf4-npdbm   karthik-ibm-powervs   karthik-ibm-powervs-md-0-qch8f            ibmpowervs://osa/osa21/3229a-af54-4212-bf60-6202b6fd0a07/50f841e5-f58c-4569-894d-b40ba0d2696e   Running        76m     v1.24.2
karthik-ibm-powervs-md-0-6b4d67ccf4-v7xv9   karthik-ibm-powervs                                                                                                                                             Provisioning   3m19s   v1.24.2
  1. After sometime verify that the new node being added to the cluster and pod is in running state
kubectl get nodes
NAME                                      STATUS   ROLES           AGE   VERSION
karthik-ibm-powervs-control-plane-pgwmz   Ready    control-plane   92m   v1.24.2
karthik-ibm-powervs-md-0-n8c6d            Ready    <none>          42s   v1.24.2
karthik-ibm-powervs-md-0-qch8f            Ready    <none>          85m   v1.24.2

kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-7c87788568-qhqdb   1/1     Running   0          19m
busybox-deployment-7c87788568-t26bb   1/1     Running   0          18m
  1. Delete the deployment to observe the scale down of nodes by autoscaler
kubectl delete deployment/busybox-deployment

kubectl get nodes
NAME                                      STATUS   ROLES           AGE    VERSION
karthik-ibm-powervs-control-plane-pgwmz   Ready    control-plane   105m   v1.24.2
karthik-ibm-powervs-md-0-qch8f            Ready    <none>          98m    v1.24.2

Developer Guide

Rapid iterative development with Tilt

Overview

This document describes how to use kind and Tilt for a simplified workflow that offers easy deployments and rapid iterative builds.

Prerequisites

  1. Docker v19.03 or newer
  2. kind v0.9 or newer (other clusters can be used if preload_images_for_kind is set to false)
  3. kustomize
  4. Tilt v0.22.2 or newer
  5. envsubst or similar to handle clusterctl var replacement
  6. Clone the Cluster API repository locally
  7. Clone the cluster-api-provider-ibmcloud repository you want to deploy locally as well

Create a kind cluster

First, make sure you have a kind cluster and that your KUBECONFIG is set up correctly:

kind create cluster

This local cluster will be running all the cluster api controllers and become the management cluster which then can be used to spin up workload clusters on IBM Cloud.

Create a tilt-settings.yaml file

Next, create a tilt-settings.yaml file and place it in your local copy of cluster-api. Here is an example:

Example tilt-settings.yaml for CAPI-IBM clusters:

Make sure to replace the parameter IBMCLOUD_API_KEY with a valid API key.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"

Add following extra_args to log Power VS REST API Requests/Responses

extra_args:
  ibmcloud:
    - '-v=5'

To deploy workload cluster with Power VS cloud controller manager which is currently in experimental stage, Set POWERVS_PROVIDER_ID_FORMAT to v2 under kustomize_substitutions.

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  POWERVS_PROVIDER_ID_FORMAT: "v2"

To deploy workload cluster with Custom Service Endpoint, Set SERVICE_ENDPOINT environmental variable in semi-colon separated format: ${ServiceRegion}:${ServiceID1}=${URL1},${ServiceID2}=${URL2...}

default_registry: "gcr.io/you-project-name-here"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "XXXXXXXXXXXXXXXXXX"
  SERVICE_ENDPOINT: "us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com"
  IBMCLOUD_AUTH_URL: "https://iam.test.cloud.ibm.com"

NOTE: For information about all the fields that can be used in the tilt-settings.yaml file, check them here.

Run Tilt

To launch your development environment, run:

tilt up

Kind cluster becomes a management cluster after this point, check the pods running on the kind cluster by running kubectl get pods -A.

Create workload clusters

To provision your workload cluster, check the Creating a cluster section for VPC and PowerVS.

After deploying it, check the tilt logs and wait for the clusters to be created.

Clean up

Before deleting the kind cluster, make sure you delete all the workload clusters.

kubectl delete cluster <clustername>
tilt up (ctrl-c)
kind delete cluster

Guide for API conversions

Introduction

The purpose of this document is to help/assist contributors with future API conversions using conversion-gen tool.

Prerequisites

  1. Create a new API version.
kubebuilder create api --group <group> --version <version> --kind <kind>
  1. Copy over existing types, and make the required changes.
  2. Mark a storage version, add marker +kubebuilder:storageversion to concerned version package.

NOTE: Refer for more detailed information about prerequisites.

Conversion flow

  1. In each “spoke” version package, add marker +k8s:conversion-gen directive pointing to the “hub” version package. It must be in doc.go. Refer
  2. In “hub” version package, create doc.go file without any marker. Refer
  3. In “spoke” version package, add a var localSchemeBuilder = &SchemeBuilder.SchemeBuilder in groupversion_info.go so the auto-generated code would compile. Refer
  4. In “hub” version package, create a conversion.go to implement the “hub” methods. Refer
  5. Run target make generate-go-conversions-core, this will generate zz_generated.conversion.go in the spoke version package.
  6. In “spoke” version package, update {kind}_conversion.go to implement Convertible for each type. When conversion-gen stops generating methods because of incompatibilities or we need to override the behavior, we stick them in this source file. Our “spoke” versions need to implement the Convertible interface. Namely, they’ll need ConvertTo and ConvertFrom methods to convert to/from the hub version. Refer

References

Release Process

Alpha/Beta releases

  • Create a tag and push
    git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
    git tag -s -m "v0.2.0-alpha.3" v0.2.0-alpha.3
    git push origin v0.2.0-alpha.3
    
  • Wait for the google cloud build to be finished
  • Create a draft release with release notes for the tag
  • Tick the prerelease checkbox
  • Download the artifacts once cloud build is finished
    gsutil -m cp \
     "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template-powervs.yaml" \
     "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template.yaml" \
     "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/infrastructure-components.yaml" \
     "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/metadata.yaml" \
     .
    
  • Upload the downloaded artifacts into the release asset
  • Publish the drafted release

Note: In the above instructions, v0.2.0-alpha.3 is the version/tag is being released

GA Releases

  • Create a tag and push
    git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
    git tag -s -m "v0.1.0" v0.1.0
    git push origin v0.1.0
    
  • Wait for the google cloud build to be finished
  • Create a draft release with release notes for the tag
  • Perform the image promotion process:
    • Clone and pull down the latest from kubernetes/k8s.io
    • Create a new branch in your fork of kubernetes/k8s.io.
    • The staging repository is here.
    • Once image is present in the above staging repository, find the sha256 tag for the image by following instructions
    $ manifest-tool inspect --raw gcr.io/k8s-staging-capi-ibmcloud/cluster-api-ibmcloud-controller:v0.1.0 | jq '.[0].Digest'
    "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363"
    
    • In your kubernetes/k8s.io branch edit k8s.gcr.io/images/k8s-staging-capi-ibmcloud/images.yaml and add an entry for the version using the sha256 value got from the above command. For example: "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363": ["v0.1.0"]
    • You can use this PR as example
    • Wait for the PR to be approved and merged
    • Run make release command
    • Copy the content from out directory to release asset
    • Publish the drafted release

Note: In the above instructions, v0.1.0 is the version/tag is being released

How to build the machine boot images

VPC

TO-DO

Power VS

Compose the user-varibales.json file contains the information for the Power VS

{
  "account_id": "",
  "apikey": "",
  "capture_cos_access_key": "",
  "capture_cos_bucket": "",
  "capture_cos_region": "",
  "capture_cos_secret_key": "",
  "key_pair_name": "",
  "kubernetes_deb_version": "",
  "kubernetes_rpm_version": "",
  "kubernetes_semver": "",
  "kubernetes_series": "",
  "region": "",
  "service_instance_id": "",
  "ssh_private_key_file": "",
  "zone": ""
}
  • account_id: IBM Cloud account ID
  • apikey: IBM Cloud API Key
  • capture_cos_access_key: Access key for the IBM Cloud Object Storage(COS) to export the image to
  • capture_cos_bucket: IBM Cloud Object Storage(COS) bucket name
  • capture_cos_region: IBM Cloud Object Storage(COS) bucket region
  • capture_cos_secret_key: IBM Cloud Object Storage(COS) secret key
  • key_pair_name: SSH key name present in the Power VS
  • kubernetes_deb_version: Kubernetes deb version, e.g: 1.24.2-00
  • kubernetes_rpm_version: Kubernetes RPM package version, e.g: 1.24.2-0
  • kubernetes_semver: e.g: v1.24.2
  • kubernetes_series: e.g: v1.24
  • region: Power VS region, e.g: osa
  • service_instance_id: Power VS service instance ID
  • ssh_private_key_file: Path to the SSH private key file used to connect to the vm while image preparation, e.g: /Users/manjunath/.ssh/id_rsa
  • zone: Power VS zone, e.g: osa21
# Clone the image-builder repository
$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi
$ PACKER_VAR_FILES=user-variables.json make build-powervs-centos-8

Troubleshooting

Reference

Regions-Zones Mapping

GeographyLocationpowervs regionspowervs zonespowervs service endpoint URL (public)ibmcloud_vpc regionsibmcloud_vpc zonesvpc service endpoint URL (public)
AmericaDallas, USAdaldal12dal.power-iaas.cloud.ibm.comus-southus-south-1
us-south-2
us-south-3
us-south.iaas.cloud.ibm.com
AmericaDallas, USAus-southus-southus-south.power-iaas.cloud.ibm.comus-southus-south-1
us-south-2
us-south-3
us-south.iaas.cloud.ibm.com
AmericaWashington DC, USAwdcwdc06wdc.power-iaas.cloud.ibm.comus-eastus-east-1
us-east-2
us-east-3
us-east.iaas.cloud.ibm.com
AmericaWashington DC, USAus-eastus-eastus-east.power-iaas.cloud.ibm.comus-eastus-east-1
us-east-2
us-east-3
us-east.iaas.cloud.ibm.com
AmericaSão Paulo, Brazilsaosao01sao.power-iaas.cloud.ibm.combr-saobr-sao-1
br-sao-2
br-sao-3
br-sao.iaas.cloud.ibm.com
AmericaToronto, Canadatortor01tor.power-iaas.cloud.ibm.comca-torca-tor-1
ca-tor-2
ca-tor-3
ca-tor.iaas.cloud.ibm.com
AmericaMontreal, Canadamonmon01mon.power-iaas.cloud.ibm.com---
EuropeFrankfurt, Germanyeu-deeu-de-1
eu-de-2
eu-de.power-iaas.cloud.ibm.comeu-deeu-de-1
eu-de-2
eu-de-3
eu-de.iaas.cloud.ibm.com
EuropeLondon, UKlonlon04
lon06
lon.power-iaas.cloud.ibm.comeu-gbeu-gb-1
eu-gb-2
eu-gb-3
eu-gb.iaas.cloud.ibm.com
Asia PacificSydney, Australiasydsyd04
syd05
syd.power-iaas.cloud.ibm.comau-sydau-syd-1
au-syd-2
au-syd-3
au-syd.iaas.cloud.ibm.com
Asia PacificTokyo, Japantoktok04tok.power-iaas.cloud.ibm.comjp-tokjp-tok-1
jp-tok-2
jp-tok-3
jp-tok.iaas.cloud.ibm.com
Asia PacificOsaka, Japanosaosa21osa.power-iaas.cloud.ibm.comjp-osajp-osa-1
jp-osa-2
jp-osa-3
jp-osa.iaas.cloud.ibm.com

References:

  1. IBM Cloud doc for the PowerVS locations
  2. IBM Cloud doc for the VPC Regions