353 lines
18 KiB
Markdown
353 lines
18 KiB
Markdown
Autoscaling Kubernetes Cluster Resources on 3Engines Cloud OpenStack Magnum[🔗](#autoscaling-kubernetes-cluster-resources-on-brand-name-openstack-magnum "Permalink to this headline")
|
||
=======================================================================================================================================================================================
|
||
|
||
When **autoscaling of Kubernetes clusters** is turned on, the system can
|
||
|
||
> * Add resources when the demand is high, or
|
||
> * Remove unneeded resources when the demand is low and thus keep the costs down.
|
||
> * The whole process can be automatic, helping the administrator concentrate on more important tasks at hand.
|
||
|
||
This article explains various commands to resize or scale the cluster and will lead to a command to automatically create an autoscalable Kubernetes cluster for OpenStack Magnum.
|
||
|
||
What We Are Going To Cover[🔗](#what-we-are-going-to-cover "Permalink to this headline")
|
||
---------------------------------------------------------------------------------------
|
||
|
||
> * Definitions of horizontal, vertical and nodes scaling
|
||
> * Define autoscaling when creating the cluster in Horizon interface
|
||
> * Define autoscaling when creating the cluster using the CLI
|
||
> * Get cluster template labels from Horizon interface
|
||
> * Get cluster template labels from the CLI
|
||
|
||
Prerequisites[🔗](#prerequisites "Permalink to this headline")
|
||
-------------------------------------------------------------
|
||
|
||
No. 1 **Hosting**
|
||
|
||
You need a 3Engines Cloud hosting account with Horizon interface <https://horizon.3Engines.com>.
|
||
|
||
No. 2 **Creating clusters with CLI**
|
||
|
||
The article [How To Use Command Line Interface for Kubernetes Clusters On 3Engines Cloud OpenStack Magnum](How-To-Use-Command-Line-Interface-for-Kubernetes-Clusters-On-3Engines-Cloud-OpenStack-Magnum.html.md) will introduce you to creation of clusters using a command line interface.
|
||
|
||
No. 3 **Connect openstack client to the cloud**
|
||
|
||
Prepare **openstack** and **magnum** clients by executing *Step 2 Connect OpenStack and Magnum Clients to Horizon Cloud* from article [How To Install OpenStack and Magnum Clients for Command Line Interface to 3Engines Cloud Horizon](How-To-Install-OpenStack-and-Magnum-Clients-for-Command-Line-Interface-to-3Engines-Cloud-Horizon.html.md)
|
||
|
||
No. 4. **Resizing Nodegroups**
|
||
|
||
Step 7 of article [Creating Additional Nodegroups in Kubernetes Cluster on 3Engines Cloud OpenStack Magnum](Creating-Additional-Nodegroups-in-Kubernetes-Cluster-on-3Engines-Cloud-OpenStack-Magnum.html.md) shows example of resizing the nodegroups for autoscaling.
|
||
|
||
No. 5 **Creating Clusters**
|
||
|
||
Step 2 of article [How to Create a Kubernetes Cluster Using 3Engines Cloud OpenStack Magnum](How-to-Create-a-Kubernetes-Cluster-Using-3Engines-Cloud-OpenStack-Magnum.html.md) shows how to define master and worker nodes for autoscaling.
|
||
|
||
There are three different autoscaling features that a Kubernetes cloud can offer:
|
||
|
||
Horizontal Pod Autoscaler[🔗](#horizontal-pod-autoscaler "Permalink to this headline")
|
||
-------------------------------------------------------------------------------------
|
||
|
||
Scaling Kubernetes cluster horizontally means increasing or decreasing the number of running pods, depending on the actual demands at run time. Parameters to take into account are the usage of CPU and memory, as well as the desired minimum and maximum numbers of pod replicas.
|
||
|
||
Horizontal scaling is also known as “scaling out” and is shorthened as HPA.
|
||
|
||
Vertical Pod Autoscaler[🔗](#vertical-pod-autoscaler "Permalink to this headline")
|
||
---------------------------------------------------------------------------------
|
||
|
||
Vertical scaling (or “scaling up”, VPA) is adding or subtracting resources to and from an existing machine. If more CPUs are needed, add them. When they are not needed, shut some of them down.
|
||
|
||
Cluster Autoscaler[🔗](#cluster-autoscaler "Permalink to this headline")
|
||
-----------------------------------------------------------------------
|
||
|
||
HPA and VPA reorganize the usage of resources and the number of pods, however, there may come a time when the size of the system itself prevents from satisfying the demand. The solution is to autoscale the cluster itself, to increase or decrease the number of nodes on which the pods will run on.
|
||
|
||
Once the number of nodes is adjusted, the pods and other resources need to rebalance themselves across the cluster, also automatically. The number of nodes acts as a physical barrier to the autoscaling of pods.
|
||
|
||
All three models of autoscaling can be combined together.
|
||
|
||
Define Autoscaling When Creating a Cluster[🔗](#define-autoscaling-when-creating-a-cluster "Permalink to this headline")
|
||
-----------------------------------------------------------------------------------------------------------------------
|
||
|
||
You can define autoscaling parameters while defining a new cluster, using window called **Size** in the cluster creation wizard:
|
||
|
||
|
||
|
||
Specify a minimum and maximum number of worker nodes. If these values are 2 and 4 respectively, the cluster will have not less that 2 nodes and not more than 4 nodes at any time. If there is no traffic to the cluster, it will be automatically scaled to 2 nodes. In this example, the cluster can have 2, 3, or 4 nodes depending on the traffic.
|
||
|
||
For the entire process of creating a Kubernetes cluster in Horizon, see Prerequisites No. 5.
|
||
|
||
Warning
|
||
|
||
If you decide to use NGINX Ingress option while defining a cluster, NGINX ingress will run as 3 replicas on 3 separate nodes. This will override the minimum number of nodes in Magnum autoscaler.
|
||
|
||
Autoscaling Node Groups at Run Time[🔗](#autoscaling-node-groups-at-run-time "Permalink to this headline")
|
||
---------------------------------------------------------------------------------------------------------
|
||
|
||
The autoscaler in Magnum uses Node Groups. Node groups can be used to create workers with different flavors. The default-worker node group is automatically created when cluster is provisioned. Node groups have lower and upper limits of node count. This is the command to print them out for a given cluster:
|
||
|
||
```
|
||
openstack coe nodegroup show NoLoadBalancer default-worker -f json -c max_node_count -c node_count -c min_node_count
|
||
|
||
```
|
||
|
||
The result would be:
|
||
|
||
```
|
||
{
|
||
"node_count": 1,
|
||
"max_node_count": 2,
|
||
"min_node_count": 1
|
||
}
|
||
|
||
```
|
||
|
||
This works fine until you try to resize cluster beyond the limit set in node group. If you try to resize the above cluster to 12 nodes, like this:
|
||
|
||
```
|
||
openstack coe cluster resize NoLoadBalancer --nodegroup default-worker 12
|
||
|
||
```
|
||
|
||
you will get the following error:
|
||
|
||
```
|
||
Resizing default-worker outside the allowed range: min_node_count = 1, max_node_count = 2 (HTTP 400) (Request-ID: req-bbb09fc3-7df4-45c3-8b9b-fbf78d202ffd)
|
||
|
||
```
|
||
|
||
To resolve this error, change *node\_group max\_node\_count* manually:
|
||
|
||
```
|
||
openstack coe nodegroup update NoLoadBalancer default-worker replace max_node_count=15
|
||
|
||
```
|
||
|
||
and then resize cluster to the desired value which was less that 15 in this example:
|
||
|
||
openstack coe cluster resize NoLoadBalancer –nodegroup default-worker 12
|
||
|
||
If you repeat the first statement:
|
||
|
||
```
|
||
openstack coe nodegroup show NoLoadBalancer default-worker -f json -c max_node_count -c node_count -c min_node_count
|
||
|
||
```
|
||
|
||
the result will now be with a corrected value:
|
||
|
||
```
|
||
{
|
||
"node_count": 12,
|
||
"max_node_count": 15,
|
||
"min_node_count": 1
|
||
}
|
||
|
||
```
|
||
|
||
How Autoscaling Detects Upper Limit[🔗](#how-autoscaling-detects-upper-limit "Permalink to this headline")
|
||
---------------------------------------------------------------------------------------------------------
|
||
|
||
The first version of Autoscaling would take the current upper limit of autoscaling in variable *node\_count* and add 1 to it. If the command to create a cluster were
|
||
|
||
```
|
||
openstack coe cluster create mycluster --cluster-template mytemplate --node-count 8 --master-count 3
|
||
|
||
```
|
||
|
||
that version of Autoscaler would take the value of **9** (counting as **8 + 1**). However, that procedure was limited to the default-worker node group only.
|
||
|
||
The current Autoscaler can support multiple node groups by detecting the role of the node group:
|
||
|
||
```
|
||
openstack coe nodegroup show NoLoadBalancer default-worker -f json -c role
|
||
|
||
```
|
||
|
||
and the result is
|
||
|
||
```
|
||
{
|
||
"role": "worker"
|
||
}
|
||
|
||
```
|
||
|
||
As long as the role is *worker* and *max\_node\_count* is greater than 0, the Autoscaler will try to scale the *default-worker* node group by adding **1** to *max\_node\_count*.
|
||
|
||
Attention
|
||
|
||
Any additional node group must include concrete *max\_node\_count* attribute.
|
||
|
||
See Prerequisites No. 4 for detailed examples of using the **openstack coe nodegroup** family of commands.
|
||
|
||
Autoscaling Labels for Clusters[🔗](#autoscaling-labels-for-clusters "Permalink to this headline")
|
||
-------------------------------------------------------------------------------------------------
|
||
|
||
There are three labels for clusters that influence autoscaling:
|
||
|
||
> * **auto\_scaling\_enabled** – if true, it is enabled
|
||
> * **min\_node\_count** – the minimal number of nodes
|
||
> * **max\_node\_count** – the maximal number of nodes, at any time.
|
||
|
||
When defining cluster through the Horizon interface, you are actually setting up these cluster labels.
|
||
|
||
|
||
|
||
List clusters with **Container Infra** => **Cluster** and click on the name of the cluster. Under *Labels*, you will find the current value for **auto\_scaling\_enabled**.
|
||
|
||
|
||
|
||
If true, it is enabled, the cluster will autoscale.
|
||
|
||
Create New Cluster Using CLI With Autoscaling On[🔗](#create-new-cluster-using-cli-with-autoscaling-on "Permalink to this headline")
|
||
-----------------------------------------------------------------------------------------------------------------------------------
|
||
|
||
The command to create a cluster with CLI must encompass all of the usual parameters as well as **all of the labels** needed for the cluster to function. The peculiarity of the syntax is that label parameters must be one single string, without any blanks inbetween.
|
||
|
||
This is what one such command could look like:
|
||
|
||
```
|
||
openstack coe cluster create mycluster
|
||
--cluster-template k8s-stable-1.23.5
|
||
--keypair sshkey
|
||
--master-count 1
|
||
--node-count 3
|
||
--labels auto_scaling_enabled=true,autoscaler_tag=v1.22.0,calico_ipv4pool_ipip=Always,cinder_csi_plugin_tag=v1.21.0,cloud_provider_enabled=true,cloud_provider_tag=v1.21.0,container_infra_prefix=registry-public.3Engines.com/magnum/,eodata_access_enabled=false,etcd_volume_size=8,etcd_volume_type=ssd,hyperkube_prefix=registry-public.3Engines.com/magnum/,k8s_keystone_auth_tag=v1.21.0,kube_tag=v1.21.5-rancher1,master_lb_floating_ip_enabled=true
|
||
|
||
```
|
||
|
||
If you just tried to copy and paste it into the terminal, you would get syntax errors. The end of the line is not allowed, the entire command must be one long string. To make your life easier, here is a version of the command that you *can* copy with success.
|
||
|
||
Warning
|
||
|
||
The line containing labels will be only partially visible on the screen, but once you paste it into the command line, the terminal software will execute it without problems.
|
||
|
||
The command is:
|
||
|
||
> **openstack coe cluster create mycluster –cluster-template k8s-stable-1.23.5 –keypair sshkey –master-count 1 –node-count 3 –labels auto\_scaling\_enabled=true,autoscaler\_tag=v1.22.0,calico\_ipv4pool\_ipip=Always,cinder\_csi\_plugin\_tag=v1.21.0/,cloud\_provider\_enabled=true,cloud\_provider\_tag=v1.21.0,container\_infra\_prefix=registry-public.3Engines.com/magnum/,eodata\_access\_enabled=false,etcd\_volume\_size=8,etcd\_volume\_type=ssd,hyperkube\_prefix=registry-public.3Engines.com/magnum/,k8s\_keystone\_auth\_tag=v1.21.0,kube\_tag=v1.21.5-rancher1,master\_lb\_floating\_ip\_enabled=true,min\_node\_count=2,max\_node\_count=4**
|
||
|
||
The name will be *mycluster*, one master node and three worker nodes in the beginning.
|
||
|
||
Note
|
||
|
||
It is mandatory to set up the maximal number of nodes in autoscaling. If not specified, the **max\_node\_count** will default to 0, and there will be no autoscaling at all for the particular nodegroup.
|
||
|
||
This is the result after the creation:
|
||
|
||
|
||
|
||
Three worker node addresses are active: **10.0.0.102**, **10.0.0.27**, and **10.0.0.194**.
|
||
|
||
There is no traffic to the cluster so the autoscaling immediately kicked in. A minute or two after the creation was finished, the number of worker nodes fell down by one, to addresses **10.0.0.27** and **10.0.0.194** – that is autoscaling at work.
|
||
|
||
Nodegroups With Worker Role Will Be Automatically Autoscalled[🔗](#nodegroups-with-worker-role-will-be-automatically-autoscalled "Permalink to this headline")
|
||
-------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||
|
||
Autoscaler automaticaly detects all new nodegroups with “worker” role assigned.
|
||
The “worker” role is assigned by default if not specified. The maximum number of nodes must be specified as well.
|
||
|
||
First see which nodegroups are present for cluster *k8s-cluster*. The command is
|
||
|
||
```
|
||
openstack coe nodegroup list k8s-cluster -c name -c node_count -c status -c role
|
||
|
||
```
|
||
|
||
Switch **-c** denotes which column to show, disregarding all other columns that are not listed in the command. You will see a table with columns *name*, *node\_count*, *status* and *role*, which means that columns such as *uuid*, *flavor\_id* and *image\_id* will not take valueable space onscreen. The result is table with only the four columns that are relevant to adding nodegroupes with roles:
|
||
|
||
|
||
|
||
Now add and print a nodegroup without role:
|
||
|
||
```
|
||
openstack coe nodegroup create k8s-cluster nodegroup-without-role --node-count 1 --min-nodes 1 --max-nodes 5
|
||
|
||
openstack coe nodegroup list k8s-cluster -c name -c node_count -c status -c role
|
||
|
||
```
|
||
|
||
Since the role was not specified, a default value of “worker” was assigned to node group *nodegroup-without-role*. Since the system is set up to automatically autoscale nodegroups with *worker* role, if you add nodegroup without a role, it will autoscale.
|
||
|
||
|
||
|
||
Now add a node group called *nodegroup-with-role* and the name of the role will be *custom*:
|
||
|
||
```
|
||
openstack coe nodegroup create k8s-cluster nodegroup-with-role --node-count 1 --min-nodes 1 --max-nodes 5 --role custom
|
||
|
||
openstack coe nodegroup list k8s-cluster -c name -c node_count -c status -c role
|
||
|
||
```
|
||
|
||
|
||
|
||
That will add a nodegroup but will not autoscale it on its own, as there is no *worker* role specified for the nodegroup.
|
||
|
||
Finally, add a nodegroup called *nodegroup-with-role-2* which will have two roles defined in one statement, that is, both *custom* and *worker*. Since at least one of the roles is *worker*, it will autoscale automatically.
|
||
|
||
```
|
||
openstack coe nodegroup create k8s-cluster nodegroup-with-role-2 --node-count 1 --min-nodes 1 --max-nodes 5 --role custom,worker
|
||
|
||
openstack coe nodegroup list k8s-cluster -c name -c node_count -c status -c role
|
||
|
||
```
|
||
|
||
|
||
|
||
Cluster **k8s-cluster** now has **8** nodes:
|
||
|
||
|
||
|
||
You can delete these three clusters with the following set of commands:
|
||
|
||
```
|
||
openstack coe nodegroup delete k8s-cluster nodegroup-with-role
|
||
|
||
openstack coe nodegroup delete k8s-cluster nodegroup-with-role-2
|
||
|
||
openstack coe nodegroup delete k8s-cluster nodegroup-without-role
|
||
|
||
```
|
||
|
||
Once again, see the result:
|
||
|
||
```
|
||
openstack coe nodegroup list k8s-cluster -c name -c node_count -c status -c role
|
||
|
||
```
|
||
|
||
|
||
|
||
How to Obtain All Labels From Horizon Interface[🔗](#how-to-obtain-all-labels-from-horizon-interface "Permalink to this headline")
|
||
---------------------------------------------------------------------------------------------------------------------------------
|
||
|
||
Use **Container Infra** => **Clusters** and click on the cluster name. You will get plain text in browser, just copy the rows under **Labels** and paste them to the text editor of your choice.
|
||
|
||
|
||
|
||
In text editor, manually remove line ends and make one string without breaks and carriage returns, then paste it back to the command.
|
||
|
||
How To Obtain All Labels From the CLI[🔗](#how-to-obtain-all-labels-from-the-cli "Permalink to this headline")
|
||
-------------------------------------------------------------------------------------------------------------
|
||
|
||
There is a special command which will produce labels from a cluster:
|
||
|
||
```
|
||
openstack coe cluster template show k8s-stable-1.23.5 -c labels -f yaml
|
||
|
||
```
|
||
|
||
This is the result:
|
||
|
||
|
||
|
||
That is *yaml* format, as specified by the **-f** parameter. The rows represent label values and your next action is to create one long string without line breaks as in the previous example, then form the CLI command.
|
||
|
||
Use Labels String When Creating Cluster in Horizon[🔗](#use-labels-string-when-creating-cluster-in-horizon "Permalink to this headline")
|
||
---------------------------------------------------------------------------------------------------------------------------------------
|
||
|
||
The long labels string can also be used when creating the cluster manually, i.e. from the Horizon interface. The place to insert those labels is described in *Step 4 Define Labels* in Prerequisites No. 2.
|
||
|
||
What To Do Next[🔗](#what-to-do-next "Permalink to this headline")
|
||
-----------------------------------------------------------------
|
||
|
||
Autoscaling is similar to autohealing of Kubernetes clusters and both bring automation to the table. They also guarantee that the system will autocorrect as long as it is within its basic parameters. Use autoscaling of cluster resources as much as you can! |