Setup Grafana and prometheus monitoring in kubernetes with flask inside and deploy using circle ci

Prerequities

64-bit chip in your system
minikube
docker
docker-compose
python
helm
nodejs
make

Setup directories

mkdir -p app/gulp/assets/{css,images,js}
mkdir -p app/gulp/assets/js/modules
mkdir -p docker/{flask,nginx}
mkdir -p kube
mkdir -p templates

Setup grafana and promethues

Run the following to setup helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create the custom-values touch ./kube/custom-values.yaml and paste the following

# custom-values.yaml
prometheus:
  service:
    type: NodePort
grafana:
  service:
    type: NodePort

Then, install the kube-prometheus-stack using Helm run the following:

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack -f ./kube/custom-values.yaml

Verify kubectl get services

Download images

Run the following command to download an image that you'll display in the website

wget https://blog.adobe.com/en/publish/2021/04/07/media_1460789842033a3aab3da4086a5abfd2326d59789.png -O app/gulp/assets/images/landscape.png

Create Flask application

As a first step, we'll create the flask application and once it is running we'll add all other functionalities

`./requirements.txt`

This is the file that stores the dependencies necessary for our small application

click==8.0.3
Flask==2.0.2
gunicorn==20.1.0
itsdangerous==2.0.1
Jinja2==3.0.3
MarkupSafe==2.0.1
Werkzeug==2.0.2

`./app.py`

This is the main flask application that holds the rules for our website.

#!/bin/env python3
import os
from flask import Flask, render_template
app = Flask(name)
@app.route("/")
def homepage():
return render_template("homepage.html", content="Hello world")
if name == 'main':
app.run(
host=os.getenv('FLASK_IP', '0.0.0.0'),
port=os.getenv('FLASK_PORT', 5000),
debug=bool(os.getenv('FLASK_DEBUG', True))
)

`./templates/homepage.html`

Let's just write a dummy html code that will be displayed by flask application. Don't worry about the missing css and js files yet. They'll be added later.

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport"
          content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
    <link rel="stylesheet" href="/css/external.css">
    <link rel="stylesheet" href="/css/app.css">
</head>
<body>
<h1>Hello world</h1>
<h3>welcome to my page</h3>
<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />
<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />
<img src="/images/bmw-r-1250-gs.jpg" width="500" alt="BMW R1250GS" />
<script src="/js/external.js"></script>
<script src="/js/app.js"></script>
</body>
</html>

Before we run our application, we'll have to create a virtual environment:

python3 -m venv myv
source myv/bin/activate

Then, let's install the dependencies for our little project:

pip install -r requirements.txt

Now, let's start our project and view it in the browser:

python app.py

Open the browser at url localhost:5000 and you should see the text from our html page, but you won't be able to see the image displayed three times.

The reason why you don't see the colored text and images is that they are not existing in the public directory and you don't have any server configured to serve those files. We'll configure nginx inside a docker container a little bit later.

Create nodejs assets

To compile the assets files (css, js and images) we'll use gulp and a small npm package that I wrote to make life easier when it comes to frontend dependencies: kisphp-assets

If you haven't used gulp before, have a look at the gulp documentation.

Create the following files:

`./app/gulp/gulpfile.js`

This will be the main gulpfile.js file where you configure what tasks you want to run by gulp for your project.

const { task, series } = require('gulp');
const config = require('../../gulp-config');
function requireUncached(module) {
delete require.cache[require.resolve(module)];
return require(module);
}
const js = require('kisphp-assets/tasks/javascripts')(config.js.external);
const bsrf = requireUncached('kisphp-assets/tasks/browserify')(config.js.project);
const css = require('kisphp-assets/tasks/css')(config.css.external);
const incss = requireUncached('kisphp-assets/tasks/css')(config.css.project);
const files = require('kisphp-assets/tasks/copy_files')(config.files);
task('default', series(
files.copy_files,
css.css,
incss.css,
js.javascripts,
bsrf.browserify,
));

`./app/gulp/assets/css/main.css`

Just a small styling for your page so you can see how it interacts with your application

h1 {
    color: #9C1A1C;
}
h3 {
color: #3A7734;
}

`./app/gulp/assets/js/modules/demo.js`

This js file will not do much, but it will show you how to add custom js code. All you have to do, is to create a file per use case and have init function for the exported object. You also can create functions, classes or everything you need in that file.

The files will be compiled by gulp with browserify plugin.

module.exports = {
    init: function() {
        console.log('file loaded');
    }
}

`./app/gulp/assets/js/app.js`

Here is the main javascript file for your application and will load all modules inside a document.ready jquery object.

$(document).ready(function(){
    require('./modules/demo').init();
    // add here more files that do one thing (Single Responsibility Principle)
});

`./package.json`

Let's create the dependencies list for our assets

{
  "scripts": {
    "build": "gulp --gulpfile app/gulp/gulpfile.js --cwd ."
  },
  "dependencies": {
    "bootstrap": "^5.1.3",
    "jquery": "^3.6.0",
    "kisphp-assets": "^0.6.0"
  }
}

`./gulp-config.js`

The purpose of this file is to have the list for which files will be compiled by gulp tasks. You'll have the following configurations:

js.external -> combine external javascript dependencies and combine them all into one external.js file
js.project -> local javascript files written in require.js format and compiled by browserify into one app.js file
css.external -> combine external css dependencies and save them into one external.css file
css.project -> build local css files and save them into app.css file. Here you can use css, stylus or scss sources.
files.xxxx -> here is the definition of the files you want to copy from dependencies to public directory. Usually you will copy images and fonts.

const settings = function(){
  this.root_path = __dirname;
  this.project_assets = __dirname + "/app/gulp/";
this.settings = {
"name": "kisphp demo",
"root_path": this.root_path,
"project_assets": this.project_assets,
"js": {
  "external": {
    "sources": [
      'node_modules/jquery/dist/jquery.min.js',
      'node_modules/bootstrap/dist/js/bootstrap.min.js',
    ],
    "output_filename": "external.js",
    "output_dirname": "public/js/",
  },
  "project": {
    "sources": [
      this.project_assets + '/assets/js/app.js',
    ],
    "output_filename": "app.js",
    "output_dirname": "public/js/",
  }
},
"css": {
  "external": {
    "sources": [
      'node_modules/bootstrap/dist/css/bootstrap.min.css',
    ],
    "output_filename": "external.css",
    "output_dirname": "public/css/",
  },
  "project": {
    "sources": [
      this.project_assets + '/assets/css/main.css'
    ],
    "output_filename": "app.css",
    "output_dirname": "public/css/",
  }
},
"files": {
  "fonts": {
    "sources": [
      'node_modules/bootstrap/fonts/*.*',
    ],
    "output_dirname": "public/fonts"
  },
  "images": {
    "sources": [
      this.project_assets + '/assets/images/**/*.*'
    ],
    "output_dirname": "public/images"
  }
}

};
return this.settings;
};
module.exports = settings();

Now that you have all these files created, let's install the dependencies:

npm install

Then let's generate our public directory with all required files in it:

npm run build

At this point, you should have a flask application and a generated public directory with two css files, two javascript files and one image

Again, if you run python app.py you still won't be able to load the assets files and the image. We'll do this in the next step.

Create Docker configuration

This file is optional but it will not add the generated directories into the docker context while you build the docker images

`.dockerignore`

myv
public

As you will see, I like to follow the convention of keeping the docker files inside the docker directory, even if I have one or mode docker images per project. This helps me to have the projects a little bit more structured and clean.

`./docker/flask/Dockerfile`

Let's create the dockerfile for the flask application. The installation of nodejs here is necessary only for the kubernetes use case which will be later.

FROM python:3.12 as base COPY requirements.txt /requirements.txt RUN pip install --upgrade pip && pip install -r /requirements.txt FROM base COPY . /app/ RUN cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime && apt-get update && apt-get install -y curl gcc g++ make && curl -fsSL https://deb.nodesource.com/setup_16.x | bash - && apt-get install -y nodejs WORKDIR /app CMD ["gunicorn", "--workers=2", "--chdir=.", "--bind", "0.0.0.0:5000", "--access-logfile=-", "--error-logfile=-", "app:app"]

`./docker/nginx/Dockerfile`

This is the dockerfile for the nginx container

FROM node:20 as base
COPY . /app
WORKDIR /app
RUN npm install --no-interaction
RUN npm run build
FROM nginx
COPY --from=base /app/public /app/public
COPY docker/nginx/proxy.conf /etc/nginx/conf.d/default.conf

`./docker/nginx/proxy.conf`

This is the nginx configuration for our application

server {
    listen 80;
server_name _;

location ~ \.(css|js|jpg|png|jpeg|webp|gif|svg) {
    root /app/public;
}

location / {
    proxy_set_header Host $host ;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto: http;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection &quot;upgrade&quot;;

    proxy_pass http://flask-app:5000;
    proxy_read_timeout 10;
}

}

`./docker-compose.yml`

Here you configure your flask and nginx containers to work together and serve your application in the browser

version: "3" services: flask-app: build: dockerfile: docker/flask/Dockerfile context: . ports: - 5000 volumes: - ./:/app - ./public:/app/public

flask-nginx: image: nginx volumes: - ./docker/nginx/proxy.conf:/etc/nginx/conf.d/default.conf - ./public:/app/public ports: - 80:80

Well, in this point, if you still have flask application running, press CTRL + C to stop it.

Create kubernetes configuration

Now, that we have the application running on local, let's configure kubernetes.

For tests, we'll use minikube

Start minikube

minikube start

Make sure you have the following plugins installed and enabled:

dns
ingress
registry
storage-provisioner
metrics-server

List addons:

minikube addons list

Enable plugins if they are not enabled already

minikube addons enable registry 
minikube addons enable metrics-server 
minikube addons enable ingress 
minikube addons enable ingress-dns
minikube addons enable storage-provisioner

Let's go further and create our kubernetes manifests:

`./kube/deployment.yaml`

This is the deployment manifest where you configure the pods that will run the application. In this setup, we'll create a pod with two containers (flask and nginx) and one initcontainer that will generate the fils in the public directory which will be defined as a shared volume between the containers.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask
  labels:
    app: flask
spec:
  replicas: 2
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      name: flask
      labels:
        app: flask
    spec:
      restartPolicy: Always
      containers:
        - name: flask
          image: localhost:5000/flask:__VERSION__
          imagePullPolicy: Always
          ports:
            - containerPort: 5000
          envFrom:
            - configMapRef:
                name: flask
          volumeMounts:
            - mountPath: /app/public
              name: public-dir
          livenessProbe:
            exec:
              command:
                - cat
                - /app/public/ready
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            exec:
              command:
                - cat
                - /app/public/ready
            initialDelaySeconds: 5
            periodSeconds: 5
        - name: nginx
          image: nginx
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /etc/nginx/conf.d
              name: nginx-config
            - mountPath: /app/public
              name: public-dir
          livenessProbe:
            exec:
              command:
                - cat
                - /app/public/ready
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            exec:
              command:
                - cat
                - /app/public/ready
            initialDelaySeconds: 5
            periodSeconds: 5
      initContainers:
        - name: npm
          image: localhost:5000/flask:__VERSION__
          imagePullPolicy: Always
          workingDir: /app
          command:
            - bash
          args:
            - build.sh
          volumeMounts:
            - mountPath: /app/public
              name: public-dir
      volumes:
        - name: nginx-config
          configMap:
            name: flask-nginx
        - name: public-dir
          emptyDir: {}

`./kube/config-map.yaml`

We create two configurations maps. One for the flask application and one for the nginx container. I think you have already noticed that in the deployment file, we don't use the a custom nginx container. We could and that would have been easier, but let's do it like this so we don't create a docker image with static files.

apiVersion: v1
kind: ConfigMap
metadata:
  name: flask
  namespace: default
data:
  FLASK_PORT: "5000"
  FLASK_DEBUG: "0"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: flask-nginx
  namespace: default
data:
  default.conf: |
    server {
        listen 80;
    server_name _;

    location ~ \.(css|js|jpg|png|jpeg|webp|gif|svg) {
        root /app/public;
    }

    location / {
        proxy_set_header Host $host ;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto: http;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection &quot;upgrade&quot;;

        proxy_pass http://127.0.0.1:5000;
        proxy_read_timeout 10;
    }
}

`./kube/ingress.yaml`

The ingress configuration will be used to access our application in the browser under the http://dev.k8s/ url

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: flask namespace: default spec: rules: - host: dev.k8s http: paths: - path: / pathType: ImplementationSpecific backend: service: name: flask port: number: 80 - host: grafana.k8s http: paths: - path: / pathType: Prefix backend: service: name: kube-prometheus-stack-grafana port: number: 80

Postgresql

`./kube/ps-claim.yaml`

Copy and paste the following to create postgres volume claim


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-volume-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

To create a configmap

`./kube/ps-configmap.yaml`


apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-secret
  labels:
    app: postgres
data:
  POSTGRES_DB: ps_db
  POSTGRES_USER: admin
  POSTGRES_PASSWORD: admin

To create deployment

`./kube/ps-deployment`


apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: 'postgres:16'
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-secret
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgresdata
      volumes:
        - name: postgresdata
          persistentVolumeClaim:
            claimName: postgres-volume-claim

To create a presistent volume use the following code

`./kube/ps-pv.yaml`


apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /data/postgresql

To create postgres service use the following code and file

`./kube/ps-service.yaml`


apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  ports:
    - port: 5432
  selector:
    app: postgres

For this run the following command in your terminal to add the minikube ip to your /etc/hosts:

sudo bash -c "echo \"$(minikube ip) dev.k8s\" >> /etc/hosts"

Also

sudo bash -c "echo \"$(minikube ip) grafana.k8s\" >> /etc/hosts"

Then, if you run cat /etc/hosts you should see on the last line your minikube ip and dev.k8s

`./kube/service.yaml`

We'll create a service of type ClusterIP for our application that will connect to the port 80 on the nginx container in the deployment pod.

apiVersion: v1
kind: Service
metadata:
  name: flask
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: flask
  ports:
    - port: 80
      targetPort: 80

For a better understanding of what happens here, when you make a request to the url http://dev.k8s/, the browser will make a request to the minikube instance and will match the ingress with the url defined earlier which will connect to the flask service which will connect to the nginx container in the running pod of the flask application.

For the requests to the css, js or images files, nginx will directly deliver them, but for other types of requests, the nginx will proxy to the python application.

`./Makefile`

We'll use this makefile to simulate a real deployment to a real kubernetes cluster

.PHONY: run up svc
version = $(shell date +%H%M%S)
run:
python3 app.py
up: dependencies
up:
docker build -f docker/flask/Dockerfile -t localhost:5000/flask:$(version) .
docker push localhost:5000/flask:$(version)
cat kube/deployment.yaml | sed "s/VERSION/$(version)/g" | kubectl apply -f -
svc: dependencies
svc:
cat kube/deployment.yaml | sed "s/VERSION/214714/g" | kubectl apply -f -
dependencies:
kubectl apply -f kube/config-map.yaml
kubectl apply -f kube/service.yaml
kubectl apply -f kube/ingress.yaml
kubectl apply -f kube/ps-claim.yaml
kubectl apply -f kube/ps-configmap.yaml
kubectl apply -f kube/ps-deployment.yaml
kubectl apply -f kube/ps-service.yaml
clean:
docker images | grep localhost | awk '{print $$3}' | uniq | xargs docker rmi -f

Please note that in makefiles, you MUST use tabs for the commands bellow every stage and not spaces

`./build.sh`

This file is used by the init container to generate the content of the public directory

#!/bin/bash npm install --no-interaction npm run build

touch public/ready

At this point, if you open the url http://dev.k8s/ in your browser, you should see a 404 Page Not Found error, which is fine.

Our setup, will use a private/local registry for the docker images and that is running on the minikube virtual machine.

Let's stop and delete the local docker containers that we used earlier:

docker stop $(docker ps -q)
docker rm $(docker ps -q)

Run the following command to connect to the minikube docker

eval $(minikube docker-env)

And then

docker run -d -p 5000:5000 --name myregistry registry:2

again

eval $(minikube docker-env)

To make sure you are using the docker from minikube, run docker images and you should see k8s docker images listed.

Run the following command to build the docker image, push it to the registry and deploy all resources to minikube kubernetes:

make up

After all makefile is get the kubernetes service Also verify the pods are running or not

kubectl get pods And verify the service kubectl get services

Setup CI-CD

We will use CIRCLE-CI for this

Go to https://circleci.com
Signin or Signup if you have not created a circleci account yet.
Now Click on Create Project then Click on Build, test, and deploy your software application
Then give it a name like grafana-app-nginx-kubernetes then click on Next:set up a pipeline
Leave to default click on Next: Choose a repo
On Let's choose a repo for your pipeline select for repository for this project or search for it then click on Next:Set up Config
Then click on Prepare config file after sometime
Click on Next
Again click on Next
Now click on Commit config and run
Let it run for sometime
Run the following

git add .
git commit -m "Adding the files"
git push

Now do git pull and checkout to git checkout circleci-project-setup and copy and paste the following contents in ./.circleci/config.yml

version: 2.1

jobs:
  initialize:
    machine:
      image: ubuntu-2204:current
    steps:
      - checkout

      # Install Minikube
      - run:
          name: Install Minikube
          command: |
            curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
            sudo install minikube-linux-amd64 /usr/local/bin/minikube

      # Start Minikube
      - run:
          name: Start Minikube
          command: minikube start --driver=docker

      - run:
          name: Install kubectl
          command: |
            curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
            sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
      # Build Docker image
      - run:
          name: Install make file
          command: sudo apt install make -y

      # Load Docker image into Minikube
      - run:
          name: Set alias for kubectl
          command: alias kubectl="minikube kubectl --"
      - run:
          name: Install Helm
          command: |
            curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
            chmod 700 get_helm.sh
            ./get_helm.sh
      # Install promethues and grafana
      - run:
          name: Install promethues and grafana
          command: |
            helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
            helm repo update
            helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack -f ./kube/custom-values.yaml
      
      - run: 
          name: Install minikube addons
          command: |
            minikube addons enable registry 
            minikube addons enable metrics-server 
            minikube addons enable ingress 
            minikube addons enable ingress-dns
            minikube addons enable storage-provisioner
      
      - run: 
          name: Install Minikube to hostfile
          command: |
            sudo bash -c "echo \"$(minikube ip) dev.k8s\" >> /etc/hosts"
            sudo bash -c "echo \"$(minikube ip) grafana.k8s\" >> /etc/hosts"
      
      - run: 
          name: Install Nodejs v20
          command: |
            curl -fsSL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
            sudo -E bash nodesource_setup.sh
            sudo apt update
            sudo apt install nodejs -y
      
      - run:
          name: Run npm Build
          command: |
            npm install --no-interaction
            npm run build
            touch public/ready
      
      - run: 
          name: Execute the deployment 
          command: | 
            eval $(minikube docker-env)
            docker run -d -p 5000:5000 --name myregistry registry:2
            eval $(minikube docker-env)  
            make up

workflows:
  version: 2
  deploy:
    jobs:
      - initialize

Now run the following

git add .
git commit -m "Adding the files"
git push

Now go to your github repository click on Pull requests then click on New pull request in compare section select circleci-project-setup then click on create pull request again click on create pull request wait until Merge pull request button appears to be green then click Merge pull request then confirm merge now your are done with the steps.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.circleci		.circleci
app/gulp		app/gulp
docker		docker
kube		kube
templates		templates
.dockerignore		.dockerignore
Makefile		Makefile
README.md		README.md
app.py		app.py
build.sh		build.sh
docker-compose.yml		docker-compose.yml
gulp-config.js		gulp-config.js
package-lock.json		package-lock.json
package.json		package.json
requirements.txt		requirements.txt

roeeelnekave/kubernetes-ingress-circle-ci

Folders and files

Latest commit

History

Repository files navigation

Setup Grafana and prometheus monitoring in kubernetes with flask inside and deploy using circle ci

Prerequities

Setup directories

Setup grafana and promethues

Download images

Create Flask application

./requirements.txt

./app.py

./templates/homepage.html

Create nodejs assets

./app/gulp/gulpfile.js

./app/gulp/assets/css/main.css

./app/gulp/assets/js/modules/demo.js

./app/gulp/assets/js/app.js

./package.json

./gulp-config.js

Create Docker configuration

.dockerignore

./docker/flask/Dockerfile

./docker/nginx/Dockerfile

./docker/nginx/proxy.conf

./docker-compose.yml

Create kubernetes configuration

./kube/deployment.yaml

./kube/config-map.yaml

./kube/ingress.yaml

Postgresql

./kube/ps-claim.yaml

./kube/ps-configmap.yaml

./kube/ps-deployment

./kube/ps-pv.yaml

./kube/ps-service.yaml

./kube/service.yaml

./Makefile

./build.sh