DevOps?

Environment Settings

The container runs in a Docker container on a Tencent Lightweight Cloud Server (Shanghai, 4C4G8M) and controls another Tencent Lightweight Server (Shanghai, 4C4G8M) via Agent to perform tasks such as code pulling, automatic building, packaging, testing, and deployment.

Jenkins receives WebHook push notifications for code from the image repository on Nanjing University Git, and all related operations are conducted according to the documentation. Both the front-end, back-end, and Python services run in containers in the production environment. Currently, the master branch is set as the production environment, and only pushes to this branch will trigger builds. The build status is pushed back to Nanjing University Git using GitLab Connection. The main branch for daily development is develop.

The deployment diagram is as follows.

image-20220401202845923

The list of repositories and descriptions come as follows.

Repository NJU Git Mirror Comments
http://172.29.4.49/MXYZyyds/backend-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_backend_mxyzyyds 后端项目
http://172.29.4.49/MXYZyyds/frontend-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds 前端项目
http://172.29.4.49/MXYZyyds/python-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_python_mxyzyyds/ Python服务项目

Setup Process

This section is omitted as much as possible. In summary, the steps are as follows:

  1. Install the basic environment, pull the image, and configure the image repository.

  2. Start the Jenkins Docker container and ensure it can SSH into the business logic server using RSA key authentication.

  3. Configure the business logic server to pull images from Nanjing University Git via SSH.

  4. Write the pipeline script, configure the WebHook, GitLab access token, GitLab Connection, etc., and gradually debug to optimize the entire CI/CD process for use in daily development.

Deployment Instructions

Trigger Methods

Trigger via WebHook

Trigger a build by actively sending a push event type WebHook when there are new commits on the master branch.

image-20220330193722422

Manually Trigger Build in Jenkins

After logging into the Jenkins backend, navigate to the Jenkins pipeline page and click Build Now to manually trigger the build.

image-20220330193606894

image-20220304112124935

Currently, the Java backend repository can collect JUnit test results and JaCoCo-based test coverage reports, which can be displayed on the Jenkins web page.

Additionally, the build results have been successfully pushed back to Nanjing University Git.

image-20220304112045381

Notes

Front-end

The front-end is generally ready for use immediately after building, but it is strongly recommended to use incognito mode or press Ctrl+F5 to force refresh the webpage. Although cnpm has been adopted, the packaging speed is still not very stable and may exhibit some fluctuations.

image-20220330213026354

Back-end

The back-end packaging is not done within the container; it is merely deployed in the container. This approach effectively utilizes caching, resulting in good speed, with most builds completing within one minute (after upgrading the server to 4C4G).

image-20220330213058741

Python

Due to the need to load pre-trained models, the network may not always be stable (sometimes it loads quickly, while other times it can inexplicably hang, with an estimated 20% probability). As a result, the time required to start the service after a successful build can vary significantly (the Docker background startup does not affect the execution flow).

The Python service can only start normally after all four model files are loaded. It is set to restart the Gunicorn worker to load these four files if they fail to load within ten minutes. If necessary, you can attempt multiple times.

image-20220330213155071

Regarding the pip source, the source has already been changed during the Docker build.

1
2
3
4
5
6
7
8
9
10
11
[2022-03-30 11:28:53 +0000] [7] [INFO] Starting gunicorn 20.1.0
[2022-03-30 11:28:53 +0000] [7] [DEBUG] Arbiter booted
[2022-03-30 11:28:53 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2022-03-30 11:28:53 +0000] [7] [INFO] Using worker: sync
[2022-03-30 11:28:53 +0000] [8] [INFO] Booting worker with pid: 8
[2022-03-30 11:28:53 +0000] [7] [DEBUG] 1 workers
Downloading: 100%|██████████| 319/319 [00:00<00:00, 383kB/s]
Downloading: 100%|██████████| 107k/107k [00:01<00:00, 73.7kB/s]
Downloading: 100%|██████████| 112/112 [00:00<00:00, 133kB/s]
Downloading: 100%|██████████| 856/856 [00:00<00:00, 1.04MB/s]
Downloading: 100%|██████████| 390M/390M [02:51<00:00, 2.38MB/s]

Deployment Script

Currently, the Jenkinsfile has been moved into the code repository. Having the code and pipeline together is preferable.

Back-end (Java Maven Project) Pipeline Script

Before Revision

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
pipeline { 
    agent any  
    stages { 
        stage('SCM from Mirror') { 
            steps { 
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_backend_mxyzyyds.git'
               sh "ls -al"
            }
        }
        stage('Build') { 
            steps { 
               sh "chmod +x mvnw"
               sh "./mvnw install"
               junit 'target/surefire-reports/*.xml'
               step( [ $class: 'JacocoPublisher' ] )
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Launch') {
            steps { 
               sh "/usr/local/shell/start_collect_backend.sh"
            }
        }
    }
}

The /usr/local/shell/start_collect_backend.sh come as follows.

1
2
3
4
5
6
7
8
9
cd /home/webroot/workspace/COLLECT-Backend
ls -al
docker stop collect-backend
docker rm collect-backend
docker system prune -f
docker build . -t collect-backend --no-cache
docker run -d --name collect-backend -p 8888:8888 --restart unless-stopped  collect-backend:latest  /bin/bash -c "java -jar target/collect*.jar; tail -f /dev/null"
exit
exit

Dockerfile come as follows：

1
2
FROM openjdk:8 as production-stage
COPY ./target/ /target/

After Revision

Only the Jenkinsfile has changed, while the Dockerfile remains unchanged.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
pipeline {
    agent {
        label 'XY'
    }
    stages {
        stage('SCM from Mirror') {
            steps {
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_backend_mxyzyyds.git'
               sh "ls -al"
               sh "pwd"
            }
        }
        stage('Compile') {
            steps {
               sh "chmod +x mvnw"
               sh "./mvnw install -Dmaven.test.skip=true"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'compile', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'compile', state: 'success'
                }
            }
        }
        stage('Test') {
            steps {
               sh "./mvnw test"
               junit 'target/surefire-reports/*.xml'
               step( [ $class: 'JacocoPublisher' ] )
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'test', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'test', state: 'success'
                }
            }
        }
        stage('Build') {
            steps {
               sh "docker stop collect-backend | true"
               sh "docker rm collect-backend | true"
               sh "docker build . -t collect-backend --no-cache"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Start') {
            steps {
               sh 'docker run -d --name collect-backend -p 8888:8888 --restart unless-stopped  collect-backend:latest  /bin/bash -c "java -jar target/collect*.jar; tail -f /dev/null"'
               sh "docker ps -a | grep collect-backend"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'start', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'start', state: 'success'
                }
            }
        }
    }
}

Front-end (Node.js-based Vue Project) Pipeline Script

Before Revision

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
pipeline { 
    agent any  
    stages { 
        stage('SCM from Mirror') { 
            steps { 
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds.git'
               sh "ls -al"
               sh "pwd"
            }
        }
        stage('Build') { 
            steps { 
               sh "/usr/local/shell/start_collect_frontend.sh"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Post') {
            steps { 
               sh "docker ps -a | grep collect-frontend"
            }
        }
    }
}

The /usr/local/shell/start_collect_frontend.sh come as follows.

1
2
3
4
5
6
7
8
9
10
cd /home/webroot/workspace/COLLECT-Frontend
ls -al
pwd
docker stop collect-frontend
docker rm collect-frontend
docker build --no-cache . -t collect-frontend
docker run -d --name collect-frontend --restart unless-stopped -p 80:80 -p 443:443 collect-frontend:latest /bin/bash -c "nginx; tail -f /dev/null"
exit
exit

Dockerfile come as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
FROM node:14.18-stretch as build-stage
WORKDIR /app
COPY package*.json ./
RUN npm install -g cnpm --registry=https://registry.npm.taobao.org
RUN cnpm install
COPY ./ .
RUN cnpm run build

FROM nginx as production-stage
RUN mkdir /app
COPY --from=build-stage /app/dist /app
COPY nginx.conf /etc/nginx/nginx.conf
COPY cert/se3.ydjsir.com.cn.pem /etc/nginx/se3.ydjsir.com.cn.pem
COPY cert/se3.ydjsir.com.cn.key /etc/nginx/se3.ydjsir.com.cn.key

nginx.conf come as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
  worker_connections  1024;
}
http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;
  sendfile        on;
  keepalive_timeout  65;
  client_max_body_size 100m;
  server {
    listen       80;
    listen       443 ssl;
    server_name  se3.ydjsir.com.cn;
    ssl_certificate     /etc/nginx/se3.ydjsir.com.cn.pem;
    ssl_certificate_key /etc/nginx/se3.ydjsir.com.cn.key;
    if ($server_port !~ 443){
        rewrite ^(/.*)$ https://$host$1 permanent;
    }

    location / {
      root   /app;
      index  index.html;
      try_files $uri $uri/ /index.html;
    }

    # This is the proxy to backend, obviously the address will be modified accordingly after backend is also in the docker
    location ^~ /api/ {
        proxy_pass http://172.17.0.1:8888/;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
      root   /usr/share/nginx/html;
    }
  }
}

After Revision

Only the Jenkinsfile has changed, while the Dockerfile and nginx.conf remain unchanged.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
pipeline {
    agent {
        label 'XY'
    }
    stages {
        stage('SCM from Mirror') {
            steps {
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds.git'
               sh "ls -al"
               sh "pwd"
            }
        }
        stage('Build') {
            steps {
                sh "docker stop collect-frontend | true"
                sh "docker rm collect-frontend | true"
                sh "docker build --no-cache . -t collect-frontend"
                
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Start') {
            steps {
                sh 'docker run -d --name collect-frontend --restart unless-stopped -p 80:80 -p 443:443 collect-frontend:latest /bin/bash -c "nginx; tail -f /dev/null"'
                sh "docker ps -a | grep collect-frontend"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'start', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'start', state: 'success'
                }
            }
        }
    }
}

Python Service (Flask + Gunicorn Project) Pipeline Script

Before Revision

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
pipeline { 
    agent any  
    stages { 
        stage('SCM from Mirror') { 
            steps { 
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_python_mxyzyyds.git'
               sh "ls -al"
            }
        }
        stage('Build') { 
            steps { 
                sh "docker stop sim"
                sh "docker build -t se3python ."
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Launch') {
            steps { 
               sh "docker run --rm -d -p 5000:5000 --name sim se3python"
               sh "docker system prune -f"
            }
        }
    }
}

Dockerfile come as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM python:3.9

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install -i https://mirrors.cloud.tencent.com/pypi/simple --no-cache-dir -r requirements.txt
RUN pip install -i https://mirrors.cloud.tencent.com/pypi/simple torch

COPY . .
RUN python3 setup.py install

CMD gunicorn -b 0.0.0.0:5000 app:app --timeout 600 --log-level debug
#CMD flask run -p 5000 --host=0.0.0.0

After Revision

Only the Jenkinsfile has changed, while the Dockerfile remains unchanged.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
pipeline {
    agent {
        label 'XY'
    }
    stages {
        stage('SCM from Mirror') {
            steps {
               git url: '[email protected]:YDJSIR/191250186_mxyzyyds_python_mxyzyyds.git'
               sh "ls -al"
            }
        }
        stage('Build') {
            steps {
                sh "docker stop sim | true"
                sh "docker build -t se3python ."
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'build', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'build', state: 'success'
                }
            }
        }
        stage('Start') {
            steps {
               sh "docker run --rm -d -p 5000:5000 --name sim se3python"
               sh "docker system prune -f"
            }
            post {
                failure {
                    updateGitlabCommitStatus name: 'start', state: 'failed'
                }
                success {
                    updateGitlabCommitStatus name:'start', state: 'success'
                }
            }
        }
    }
}