We have a series that covers the full practice of microservices from requirements to live, from code to K8S deployment, from logging to monitoring, and more.

The whole project uses the micro-services developed by GO-Zero, and basically includes some middleware developed by Go-Zero and related GO-Zero authors. The technology stack used is basically self-developed components of go-Zero project team, which is basically the whole Family of Go-Zero.

Actual combat project address: github.com/Mikaelemmmm…

1, an overview of the

In the last section, we have built GitLab, Jenkins, Harbor and K8S. In this section, we will write Jenkins pipline and release our service into K8S completely through Jenkins.

2. Deploy middleware

Deploy mysql, Redis, ES, etc. outside of K8S, simulate as an online independent environment (as for online you want to deploy some middleware inside K8S itself, this time focus on how to deploy the micro-services developed by Go-Zero inside K8S cluster), I used the docker-comemy-env.yaml for my project and installed all dependent third-party middleware environments directly on srv-data.com(192.168.1.181). Docker and Docker-compose are already installed on this server.

Log in to the 192.168.1.181

$ mkdir data && cd data && vim docker-compose.yml
$ docker-compose up -d
$ docker-compose ps # check to confirm
Copy the code

3. Independent configuration

If the online configuration is changed, the files of the repository will be directly modified. When Jenkins makes the CD, the code will be pulled first and then the configuration of the corresponding service will be automatically constructed. For details, please refer to the pipline below.

[Q] Why not configure the center?

1) Db, redis and other modifications need to restart the service, but some configurations do not need to restart the service, and the operation and maintenance need to remember, so it is easy to cause online accidents

2) Easy rollback. We posted the new version online and changed the new version configuration. At this time, there is a problem with the online user feedback, and the online needs to be quickly rolled back. If we use the file to build the image, we can directly use the K8S one-line command to roll back the previous version of the code and configuration directly. If the configuration center is used, the code is rolled back, and the previous version of the configuration center to change back, very troublesome.

The independent online warehouse directory structure is as follows (this structure is related to the pipline writing)

Warehouse address: github.com/Mikaelemmmm… , just download it

1. Modify the middleware in the configuration, such as database and redis, to the machine 192.168.1.181, which we regard as the middleware of the online environment.

2. The other is our service discovery. Online we deploy in K8S, go-Zero directly supports K8S service discovery, so there is no need for ETCD.

4. Write Jenkins’ Pipline

4.1 Setting Parameters

Jenkins, visit http://192.168.1.180:8989/ and goes into the Jenkins home page, click on the left side of the new menu Item

Let’s start by creating the pipeline for the Identity authorization service

Then click “General”, select “This project is parameterized”, “Add Parameter”, “Choice Parameter”, as shown below

Then write something like this

Save it directly.

4.2 write pipline

Scroll down to the Pipeline script and fill in the script contents

pipeline { agent any parameters { gitParameter name: 'branch', type: 'PT_BRANCH', branchFilter: 'origin/(.*)', defaultValue: 'master', selectedValue: 'DEFAULT', sortMode: 'ASCENDING_SMART', description: 'Select branch to build'} stages {stage(' service information ') {steps {sh 'echo Branch: $branch' sh 'echo Build service type: ${JOB_NAME}-$type'}} stage(' pull code ') {steps {checkout([$class: 'GitSCM', Branches: [[name: branches: '$branch']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'SSH: / / [email protected]:2222 / root/go - zero - looklook. Git']]])}} stage (' obtaining commit_id ') {steps {echo 'obtaining commit_id' git credentialsId: 'gitlab-cert', url: 'SSH: / / [email protected]:2222 / root/go - zero - looklook. Git' script mit_id = {env.com sh (returnStdout: true, the script: 'git rev-parse --short HEAD'). Trim ()}}} stage(' Pull config file ') {steps {checkout([$class: 'GitSCM', Branches: [[name: branches: '$branch']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'conf']], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'SSH: / / [email protected]:2222 / root/go - zero - looklook - pro - the conf. Git']]])}} stage (' goctl version test ') {steps {sh '/usr/local/bin/goctl -v' } } stage('Dockerfile Build') { steps{ sh 'yes | cp -rf The conf / ${JOB_NAME} / ${type} / ${JOB_NAME}. The yaml app / ${JOB_NAME} / CMD / ${type} / etc 'online configuration file / / sh' CD app / ${JOB_NAME} / CMD / ${type} && /usr/local/bin/goctl docker -go ${JOB_NAME}.go && ls -l' script{ env.image = sh(returnStdout: true, script: "Echo ${JOB_NAME} - ${type} : ${commit_id} '). The trim ()} sh 'echo image name: ${image} && cp app / ${JOB_NAME} / CMD / ${type} / Dockerfile / && ls -l && docker build - t ${image}. '}} stage (' uploaded to mirror the warehouse ') { Steps {// Docker login will output the account password to the Jenkins page. Sh 'docker login --username=${docker_username} --password=${docker_pwd} http://${docker_repo}' sh 'docker  tag ${image} ${docker_repo}/go-zero-looklook/${image}' sh 'docker push ${docker_repo}/go-zero-looklook/${image}' } } Stage (' deploys to K8s ') {steps{script{env.deployyaml = sh(returnStdout: true, script: 'echo ${JOB_NAME}-${type}-deploy.yaml').trim() env.port=sh(returnStdout: true, script: '/root/port.sh ${JOB_NAME}-${type}').trim() } sh 'echo ${port}' sh 'rm -f ${deployYaml}' sh '/usr/local/bin/goctl kube deploy -secret docker-login -replicas 2 -nodePort 3${port} -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name ${JOB_NAME}-${type} -namespace go-zero-looklook -image ${docker_repo}/${image} -o ${deployYaml} -port ${port} --home /root/template' sh '/usr/local/bin/kubectl apply -f ${deployYaml}' } } stage('Clean') { steps{ sh 'docker rmi -f ${image}' sh 'docker rmi -f ${docker_repo}/${image}' cleanWs notFailBuild: true } } } }Copy the code

Very important!!

  1. Construction optimization: When dockerfile is generated in pipline, we use K8S deployment mode without ETCD, but this deployment mode needs to specify the account (have the permission to get in K8S endpoints, use the default is good. K8s will automatically create a default for us every time we create a new namespace), but the k8s yML generated with goctl does not add the option to specify the account. This has been reported, and may be added in a later version. Here we also use the template to do so. Also the template is in the project directory github.com/Mikaelemmmm…

  2. {credentialsId} replaces your credentialsId with your credentialsId, which is a string in the Add Credential module. We configured gitlab−cert so gitlab−cert. {credentialsId} replaces your credentialsId with your credentialsId if you do not replace it yourself. That’s a string in the Add Credential module. We configured gitlab-cert so gitlab-cert. If you don’t replace the credentialsId, replace it with your credentialsId, a string in the Add Credential module, {gitUrl} should be replaced with the git repository address of your code. Other variables in the form of ${XXX} do not need to be changed. You can leave them as they are.

  3. This is a little bit different from the official document, because my project folder directory is different, the dockerfile generated by goctl I made a little bit of manual adjustment, in a dockerfile that I didn’t create at build time, I put the dockerfile in the directory when I created the project, This eliminates the need for goCTL when building the image

5. Configure K8S to pull private warehouse image

By default, k8S can only pull public images from the Harbor repository. If it pulls private images from the harbor repository, ErrImagePull and ImagePullBackOff errors are reported

1. Launch the machine in Harbor on Jenkins first

$Docker login 192.168.1.180:8077
$ Username: admin
$ Password:
Login Succeeded
Copy the code

2. Generate the harbor login configuration file in K8S

#View the credentials generated by logging into Harbor in the previous step
$ cat /root/.docker/config.json{" authS ": {"192.168.1.180:8077": {"auth": "YWRtaW46SGFyYm9yMTIzNDU="}}Copy the code

3. Base64 encryption for the secret key file

$cat /root/.docker/config.json | base64 -w 0

ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
Copy the code

Create docker-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: docker-login
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
Copy the code
$ kubectl create -f docker-secret.yaml -n go-zero-looklook

secret "docker-login" created
Copy the code

6, building,

We go to the home page and click idenity to enter the details page

Then you can see the identity service configured above, as shown in the picture below. Click “Build with Parameters”, then select RPC, and click “Start Building”.

The first build will fail when pulling code, should be initializing something, just click on it again.

The deployment was successful

In the same way, to build identity-API, to configure userCenter service, to build usercenter RPC, to build usercenter API, to configure other services, build. This time we will build only identity-API, identity-rpc, usercenter-rpc, usercenter-API for demonstration.

6. Add a gateway

Because our API services are exposed in k8S via goctl, we can index the nodePort service in k8S under the Go-zero-looklook namespace, and then configure the nodeport in Nignx.

The k8S backend API service is used to expose the port to Nginx through nodePort. Then nginx configures this API service in the configuration, so that nginx acts as a gateway.

Nginx installation will not be discussed here, remember to have the auth_request module, if not to install their own.

Nginx configuration

server{ listen 8081; access_log /var/log/nginx/looklook.com_access.log; error_log /var/log/nginx//looklook.com_error.log; location /auth { internal; proxy_set_header X-Original-URI $request_uri; proxy_pass_request_body off; proxy_set_header Content-Length ""; Proxy_pass http://192.168.1.182:31001/identity/v1/verify/token; } location ~ /usercenter/ { auth_request /auth; auth_request_set $user $upstream_http_x_user; proxy_set_header x-user $user; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Proxy_pass http://192.168.1.182:31002; } location ~ /travel/ { auth_request /auth; auth_request_set $user $upstream_http_x_user; proxy_set_header x-user $user; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Proxy_pass http://192.168.1.182:31003; } location ~ /order/ { auth_request /auth; auth_request_set $user $upstream_http_x_user; proxy_set_header x-user $user; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Proxy_pass http://192.168.1.182:31004; } location ~ /payment/ { auth_request /auth; auth_request_set $user $upstream_http_x_user; proxy_set_header x-user $user; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Proxy_pass http://192.168.1.182:31005; }}Copy the code

If it is online, there should be multiple NignX to maintain high availability, there will be an SLB in front of Nignx, your domain name including HTTPS configuration should be resolved to SLB, there will be firewalls in front of SLB, etc.

8. Conclusion

At this point, the series is over. The overall architecture diagram should be as shown in the first article, and this series is intended to help you.

The project address

Github.com/zeromicro/g…

Welcome to Go-Zero and star support us!

Wechat communication group

Pay attention to the public account of “micro-service Practice” and click on the exchange group to obtain the QR code of the community group.