HTTP Request-based Autoscaling on K8S using Prometheus and Keda on 3Engines Cloud[🔗](#http-request-based-autoscaling-on-k8s-using-prometheus-and-keda-on-brand-name "Permalink to this headline")
===================================================================================================================================================================================================

Kubernetes pod autoscaler (HPA) natively utilizes CPU and RAM metrics as the default triggers for increasing or decreasing number of pods. While this is often sufficient, there can be use cases where scaling on custom metrics is preferred.

[KEDA](https://keda.sh/) is a tool for autoscaling based on events/metrics provided from popular sources/technologies such as Prometheus, Kafka, Postgres and multiple others.

With this article we will deploy a sample app on 3Engines Cloud cloud. We will collect HTTP requests from NGINX Ingress on our Kubernetes cluster and, using Keda with Prometheus scaler, apply custom HTTP request-based scaling.

Note

We will use *NGINX web server* to demonstrate the app, and *NGINX ingress* to deploy it and collect metrics. Note that *NGINX web server* and *NGINX ingress* are two separate pieces of software, with two different purposes.

What We Are Going To Cover[🔗](#what-we-are-going-to-cover "Permalink to this headline")
---------------------------------------------------------------------------------------

> * Install NGINX ingress on Magnum cluster
> * Install Prometheus
> * Install Keda
> * Deploy a sample app
> * Deploy our app ingress
> * Access Prometheus dashboard
> * Deploy KEDA ScaledObject
> * Test with Locust

Prerequisites[🔗](#prerequisites "Permalink to this headline")
-------------------------------------------------------------

No. 1 **Account**
:   You need a 3Engines Cloud hosting account with access to the Horizon interface: <https://horizon.3Engines.com>.

No. 2 **Create a new Kubernetes cluster without Magnum NGINX preinstalled from Horizon UI**

The default NGINX ingress deployed from Magnum from Horizon UI does not yet implement Prometheus metrics export. Instead of trying to configure Magnum ingress for this use case, we will rather install a new NGINX ingress. To avoid conflicts, best to follow the below instruction on a Kubernetes cluster **without** Magnum NGINX preinstalled from Horizon UI.

No. 3 **kubectl pointed to the Kubernetes cluster**

The following article gives options for creating a new cluster and activating the **kubectl** command:

[How To Access Kubernetes Cluster Post Deployment Using Kubectl On 3Engines Cloud OpenStack Magnum](How-To-Access-Kubernetes-Cluster-Post-Deployment-Using-Kubectl-On-3Engines-Cloud-OpenStack-Magnum.html.md).

As mentioned, create the cluster **without** installing the NGINX ingress option.

No. 4 **Familiarity with deploying Helm charts**

This article will introduce you to Helm charts on Kubernetes:

[Deploying Helm Charts on Magnum Kubernetes Clusters on 3Engines Cloud Cloud](Deploying-Helm-Charts-on-Magnum-Kubernetes-Clusters-on-3Engines-Cloud-Cloud.html.md)

Install NGINX ingress on Magnum cluster[🔗](#install-nginx-ingress-on-magnum-cluster "Permalink to this headline")
-----------------------------------------------------------------------------------------------------------------

Please type in the following commands to download the *ingress-nginx* Helm repo and then install the chart. Note we are using a custom namespace *ingress-nginx* as well as setting the options to enable Prometheus metrics.

```
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

kubectl create namespace ingress-nginx

helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set-string controller.podAnnotations."prometheus\.io/scrape"="true" \
--set-string controller.podAnnotations."prometheus\.io/port"="10254"

```

Now run the following command to get the external IP address of the ingress controller, which will be used by ingress resources created in the further steps of this article.

```
$ kubectl get services -n ingress-nginx
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller  LoadBalancer   10.254.118.18    64.225.135.67   80:31573/TCP,443:30786/TCP   26h

```

We get **64.225.135.67**. Instead of that value, use the EXTERNAL-IP value you get in your terminal after running the above command.

Install Prometheus[🔗](#install-prometheus "Permalink to this headline")
-----------------------------------------------------------------------

In order to install Prometheus, please apply the following command on your cluster:

```
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/

```

Note that this is Prometheus installation customized for NGINX Ingress and already installs to the *ingress-nginx* namespace by default, so no need to provide the namespace flag or create one.

Install Keda[🔗](#install-keda "Permalink to this headline")
-----------------------------------------------------------

With below steps, create a separate namespace for Keda artifacts, download the repo and install the Keda-Core chart:

```
kubectl create namespace keda

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

helm install keda kedacore/keda --version 2.3.0 --namespace keda

```

Deploy a sample app[🔗](#deploy-a-sample-app "Permalink to this headline")
-------------------------------------------------------------------------

With the above steps completed, we can deploy a simple application. It will be an NGINX web server, serving a simple “Welcome to nginx!” page. Note, we create a deployment and then expose this deployment as a service of type ClusterIP. Create a file *app-deployment.yaml* in your favorite editor:

**app-deployment.yaml**

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

```

Then apply with the below command:

```
kubectl apply -f app-deployment.yaml -n ingress-nginx

```

We are deploying this application into the *ingress-nginx* namespace where also the ingress installation and Prometheus is hosted. For production scenarios, you might want to have better isolation of application vs. infrastructure, this is however beyond the scope of this article.

Deploy our app ingress[🔗](#deploy-our-app-ingress "Permalink to this headline")
-------------------------------------------------------------------------------

Our application is already running and exposed in our cluster, but we want to also expose it publicly. For this purpose we will use NGINX ingress, which will also act as a proxy to register the request metrics. Create a file *app-ingress.yaml* with the following contents:

**app-ingress.yaml**

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: "64.225.135.67.nip.io"
    http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /app
        pathType: Prefix

```

Then apply with:

```
kubectl apply -f app-ingress.yaml -n ingress-nginx

```

After a while, you can get a public IP address where the app is available:

```
$ kubectl get ingress -n ingress-nginx
NAME           CLASS   HOSTS                  ADDRESS         PORTS   AGE
app-ingress    nginx   64.225.135.67.nip.io   64.225.135.67   80      18h

```

After typing the IP address with the prefix (replace with your own floating IP with /app suffix), we can see the app exposed. We are using the *nip.io* service, which works as a DNS resolver, so there is no need to set up DNS records for the purpose of the demo.

![welcome_nginx.png](../_images/welcome_nginx.png)

Access Prometheus dashboard[🔗](#access-prometheus-dashboard "Permalink to this headline")
-----------------------------------------------------------------------------------------

To access Prometheus dashboard we can port-forward the running prometheus-server to our localhost. This could be useful for troubleshooting. We have the *prometheus-server* running as a *NodePort* service, which can be verified per below:

```
$ kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.254.3.172     64.225.135.67   80:30881/TCP,443:30942/TCP   26h
ingress-nginx-controller-admission   ClusterIP      10.254.51.201    <none>          443/TCP                      26h
ingress-nginx-controller-metrics     ClusterIP      10.254.15.196    <none>          10254/TCP                    26h
nginx                                ClusterIP      10.254.160.207   <none>          80/TCP                       25h
prometheus-server                    NodePort       10.254.24.85     <none>          9090:32051/TCP               26h

```

We will port-forward to the localhost in the following command:

```
kubectl port-forward deployment/prometheus-server 9090:9090 -n ingress-nginx

```

Then enter *localhost:9090* in your browser, you will see the Prometheus dashboard. In this view we will be able to see various metrics exposed by nginx-ingress. This can be verified by starting to type “nginx-ingress” to search bar, then various related metrics will start to show up.

![prometheus-dashboard_9090.png](../_images/prometheus-dashboard_9090.png)

Deploy KEDA ScaledObject[🔗](#deploy-keda-scaledobject "Permalink to this headline")
-----------------------------------------------------------------------------------

Keda ScaledObject is a custom resource which will enable scaling our application based on custom metrics. In the YAML manifest we define what will be scaled (the nginx deployment), what are the conditions for scaling, and the definition and configuration of the trigger, in this case Prometheus. Prepare a file *scaled-object.yaml* with the following contents:

**scaled-object.yaml**

```
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: ingress-nginx
  labels:
    deploymentName: nginx
spec:
  scaleTargetRef:
    kind: Deployment
    name: nginx # name of the deployment, must be in the same namespace as ScaledObject
  minReplicaCount: 1
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.ingress-nginx.svc.cluster.local:9090
      metricName: nginx_ingress_controller_requests
      threshold: '100'
      query: sum(rate(nginx_ingress_controller_requests[1m]))

```

For detailed definition of *ScaledObject*, refer to Keda documentation. In this example, we are leaving out a lot of default settings, most notable of which is called *coolDownPeriod*. Being not explicitly assigned a value, its default value of 300 seconds will be in effect, however, see below how you can change that value to something else.

We are using here the *nginx-ingress-controller-requests* metric for scaling. This metric will only populate in the Prometheus dashboard once the requests start hitting our app service. We are setting the threshold for **100** and the time to **1** minute, so in case there is more requests than **100** per pod in a minute, this will trigger scale up.

```
kubectl apply -f scaled-object.yaml -n ingress-nginx

```

Test with Locust[🔗](#test-with-locust "Permalink to this headline")
-------------------------------------------------------------------

We can now test whether the scaling works as expected. We will use *Locust* for this, which is a load testing tool. To quickly deploy *Locust* as LoadBalancer service type, enter the following commands:

```
kubectl create deployment locust --image paultur/locustproject:latest
kubectl expose deployment locust --type LoadBalancer --port 80 --target-port 8089

```

After a couple of minutes the LoadBalancer is created and Locust is exposed:

```
$ kubectl get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
kubernetes   ClusterIP      10.254.0.1     <none>           443/TCP        28h
locust       LoadBalancer   10.254.88.89   64.225.132.243   80:31287/TCP   4m19s

```

Enter Locust UI in the browser using the EXTERNAL-IP. It can be only **64.225.132.243** or **64.225.132.243.nip.io**, one of these values is sure to work. Then hit “Start Swarming” to initiate mock requests on our app’s public endpoint:

![locust_test.png](../_images/locust_test.png)

With the default setting and even single user, *Locust* will start swarming hundreds of requests immediately. Tuning Locust is not in scope of this article, but we can quickly see the effect. The additional pod replicas are generated:

```
$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx-controller-557bf68967-h9zf5   1/1     Running             0          27h
nginx-85b98978db-2kjx6                      1/1     Running             0          30s
nginx-85b98978db-2kxzz                      1/1     Running             0          61s
nginx-85b98978db-2t42c                      1/1     Running             0          31s
nginx-85b98978db-2xdzw                      0/1     ContainerCreating   0          16s
nginx-85b98978db-2zdjm                      1/1     Running             0          30s
nginx-85b98978db-4btfm                      1/1     Running             0          30s
nginx-85b98978db-4mmlz                      0/1     ContainerCreating   0          16s
nginx-85b98978db-4n5bk                      1/1     Running             0          46s
nginx-85b98978db-525mq                      1/1     Running             0          30s
nginx-85b98978db-5czdf                      1/1     Running             0          46s
nginx-85b98978db-5kkgq                      0/1     ContainerCreating   0          16s
nginx-85b98978db-5rt54                      1/1     Running             0          30s
nginx-85b98978db-5wmdk                      1/1     Running             0          46s
nginx-85b98978db-6tc6p                      1/1     Running             0          77s
nginx-85b98978db-6zcdw                      1/1     Running             0          61s
...

```

Cooling down[🔗](#cooling-down "Permalink to this headline")
-----------------------------------------------------------

After hitting “Stop” in Locust, the pods will scale down to one replica, in line with the value of *coolDownPeriod* parameter, which is defined in the Keda ScaledObject. Its default value is 300 seconds. If you want to change it, use command

```
kubectl edit scaledobject prometheus-scaledobject -n ingress-nginx

```