background
With the increasing development and popularity of cloud native technology, multi-language microservice architecture (Python, Go, PHP….) Multiple systems communicate with each other through RPC. Once the interface needs to be upgraded, coordinating the release and upgrade of the server side, the change of IDL file warehouse, and the change release of the client side become a very troublesome thing. We need a convenient and transparent specification to coordinate the ends, and a more automated tool to coordinate. Take the mainstream GRPC as an example to discuss, where is the proto and code in the end? How do you automate version control?
Plan 1: Put in the respective code repository
As shown below, all proto files and client code that the project depends on as well as its own are stored directly in the protobuf/ directory without external tools.
Disadvantages:
-
1 all proTO dependent on ️ project is stored under the code warehouse of their respective projects, so all dependent proTO needs to be manually “needed” from other business groups and then put into the directory of Protobuf /, which is extremely troublesome for manual intervention. Too much reliance on copy can become a burden.
-
2 there is no way to maintain the control and update of ️ version and the communication cost is too high.
-
3 The reusability of ️ code is poor.
Advantages:
- Simple to use, each project’s PROto dependencies at a glance. No need to look here or there.
Plan 2: Independent Proto warehouse
Each project has its own proto repository. Dependencies are pulled from the Proto repository (including the service itself).
Disadvantages:
-
1️ each service needs to pay attention to its own service and proTO warehouse at the same time. Proto and client code should be defined and generated in proTO warehouse first during development, so switch between service warehouse and ProTO warehouse is needed during development.
-
2️ retail if the service depends too much on, it may also cross business groups. For example, the figure below shows that after crossing groups, the principal of the business group may need to open multiple PROTO warehouse permissions. You need to rely on them one by one.
- 3️ every time there is a new service, it is troublesome to apply for a PROto warehouse.
Advantages:
-
1️ each service has its own PROTO warehouse, which is convenient for version maintenance and upgrade.
-
2️ dependence can be drawn on demand.
Plan three: centralized warehouse
Manage the PROTO repository by business group dimension, so that if multiple dependencies are dependent on a business group, only one PROTO repository needs to be pulled. At the same time, each time a new service is created, the proOTO warehouse of the respective business group only needs to add its own service, and there is no need to apply for a proTO warehouse of its own service separately.
Advantages:
-
1️ Dependence on proTO of multiple services only needs to rely on central warehouse.
-
2️ new services do not need to apply for PROTO warehouse separately, but only need to be added in the PROTO warehouse of their respective business groups.
Disadvantages:
-
1️ retail service development or need to pay attention to two warehouses, need to cut.
-
2️ version management of proTO of various services cannot be carried out independently under each central warehouse.
-
3️ one may only rely on the PROto of a CERTAIN SVC, while introducing some other unnecessary ones.
Solution 4: Mirror repository + Git Branch
In order to solve some of the pain points of the above solutions, we combined the advantages of each solution with the way of mirror warehouse and Git Branch based on the central warehouse.
- 1 Master readme of central warehouse of ️ retail maintains the corresponding relationship between branches and services, as shown in the figure below.
-
2️ discount is mainly used to obtain the branch name of the next branch when new SVC is added (due to the limitation of go.mod branch version management, it can only be named V2 and V3), and at the same time it is convenient for people to identify which module SVC they depend on. Common represents the set branch of all SVC proTos. A description is added to this REame during cicD each time a new service is added.
-
3 The readme branches of ️ central warehouse maintain their own versions of the situation.
-
4️ can be used according to their own needs to rely on the corresponding branch version.
-
5️ Python CLEint version and GO version are one-to-one corresponding.
-
6️ each SVC only needs to rely on a. Gitlab. yml file to realize. Specific Gitlab CICD jobs are as follows:
.push_tmpl: &push_proto
script:
- echo "push test"
- echo $CI_PROJECT_NAME
- |
userMail=$GITLAB_USER_EMAIL
git config --global user.email "$GITLAB_USER_EMAIL"
userName=${userMail%@*}
echo "$userMail"
git config --global user.name "$userName"
git clone -v https://xxxxx/proto-center.git
cd proto-center
if [ `grep -c $CI_PROJECT_NAME README.md` -eq '0' ]; then
echo "- $CI_PROJECT_NAME-->v$(sed -n '$p' README.md | awk -F "-->v" '{print $2+1}' | head)" >>README.md
export branchName=$(sed -n '$p' README.md | awk -F "-->" '{print $2}')
echo $branchName
git add . && git commit -m "add $CI_PROJECT_NAME "
git push https://xxxxxxxx/proto-center.git master
git checkout -b $branchName
mkdir -p $CI_PROJECT_NAME && cp -r .. /protobuf/* $CI_PROJECT_NAME/
go mod init xxxxxxxx/proto-center/$branchName
go mod tidy
git add .
git commit -m "add $CI_PROJECT_NAME proto"
git push --set-upstream https:/xxxxxxxx/proto-center.git $branchName
git checkout v2
mkdir -p $CI_PROJECT_NAME && cp -r .. /protobuf/* $CI_PROJECT_NAME/
git add .
git commit -m "add $CI_PROJECT_NAME proto"
git push https://xxxxxxxx/proto-center.git v2
else
export branchName=$(grep $CI_PROJECT_NAME README.md | awk -F "-->" '{print $2}' | head)
git checkout $branchName
rm -rf $CI_PROJECT_NAME/
mkdir -p $CI_PROJECT_NAME
cp -r .. /protobuf/* $CI_PROJECT_NAME/ && git add .
git commit -m "update $CI_PROJECT_NAME proto"
git push https://xxxxxxxx/proto-center.git $branchName;
git checkout v2
rm -rf $CI_PROJECT_NAME/
mkdir -p $CI_PROJECT_NAME
cp -r .. /protobuf/* $CI_PROJECT_NAME/ && git add .
git commit -m "update $CI_PROJECT_NAME proto"
git push https://xxxxxxxx/proto-center.git v2;
fi
Copy the code -
7 The jobs of Gitlab AND CICD of ️ central warehouse are as follows:
.buld_tmpl: &tag_proto
script:
- echo "tag proto "
- |
userMail=$GITLAB_USER_EMAIL
userName=${userMail%@*}
echo "$userMail"
git config --global user.email $userMail
git config --global user.name $userName
branch=$CI_BUILD_REF_NAME
isnottag="false"
git describe --tag || isnottag="true"
git clone -b $branch -v https://xxxxx/proto-center.git
cd proto-center
userMail=$( git log --pretty=format:%ae ${CI_COMMIT_SHA} -1)
userName=${userMail%@*}
echo "$userMail"
git config --global user.email $userMail
git config --global user.name $userName
version=""
if [ $isnottag = "true" ]; then
echo $isnottag
Version = $branch. 0.1
echo $version
git tag $version
git push https://xxxxx/proto-center.git --tags
echo "- $version" >> README.md
git add .
git commit -m "add $version"
git push https://xxxxx/proto-center.git $branch
else
echo $isnottag
version_ref=$(git describe --tags | awk -F "-" '{print $1}' | head)
echo $version_ref
version=`echo $version_ref | awk -F. -v OFS=. 'NF==1{print ++$NF}; NF>1{if(length($NF+1)>length($NF))$(NF-1)++; $NF=sprintf("%0*d", length($NF), ($NF+1)%(10^length($NF))); print}'`
echo $version
git tag $version
git push https://xxxxx/proto-center.git --tags
echo "- $version" >> README.md
git add .
git commit -m "add $version"
git push https://xxxxx/proto-center.git $branch
fi
mkdir -p python
cp -r **/*.proto python
pkgName=$( grep "\\-->" README.md | tail -1 | awk -F "-->v" '{print $1}' | awk -F " " '{print $2}')
pkgName=${pkgName//-/_}
echo $pkgName
export pip_pkg_name=$pkgName
echo $version
export pip_tag_name=$version
PIP install grpcio - tools = = 1.4.0
PIP install protobuf = = 3.3.0
echo "#! /bin/env python
# -*- encoding=utf8 -*-
import os
from setuptools import (setup, find_packages)
version = os.getenv('pip_tag_name')
name = os.getenv('pip_pkg_name')
setup(
name=name,
version=version,
description='LLS grpc protocol',
packages=find_packages(exclude=[]),
include_package_data=True,
author='LLS DEV Team',
author_email='',
package_data={'': ['*.*']},
install_requires=[
'grpcio = = 1.18.0',
'protobuf = = 3.3.0'
].
zip_safe=False,
classifiers=[
'Programming Language :: python :: 2.7',
].
)" > python/setup.py
echo "
[distutils]
index-servers = internal
[internal]
repository: https://xxxxx.com/
username: xxxxx
password: xxxxx
" > ~/.pypirc
mkdir -p python/$pkgName
echo "" >> python/$pkgName/__init__.py
python -m grpc_tools.protoc -I python/ --python_out=python/$pkgName --grpc_python_out=python/$pkgName/ python/*.proto
cd python
ls
python setup.py bdist_wheel upload -r internal
cd ..
mkdir -p python3
cp -r -n **/*.proto python3
echo "#! /bin/env python
# -*- encoding=utf8 -*-
import os
from setuptools import (setup, find_packages)
version = os.getenv('pip_tag_name')
name = os.getenv('pip_pkg_name')
setup(
name=name,
version=version,
description='LLS grpc protocol',
packages=find_packages(exclude=[]),
include_package_data=True,
author='LLS DEV Team',
author_email='',
package_data={'': '*. *'},
install_requires=[
'grpcio = = 1.18.0'.
'protobuf = = 3.12.4'
].
zip_safe=False,
classifiers=[
'Programming Language: python :: 3.7'.
].
)" > python3/setup.py
python3 -m pip install Grpcio - tools = = 1.4.0
python3 -m pip install protobuf
mkdir -p python3/$pkgName
echo "" >> python3/$pkgName/__init__.py
python3 -m grpc_tools.protoc -I python3/ --python_out=python3/$pkgName --grpc_python_out=python3/$pkgName/ python3/*.proto
cd python3/$pkgName
ls
pb_files=`ls | grep -v '__init__' | grep -v 'grpc.py'`
echo $pb_files
need_replace_strs=()
for pb_file in ${pb_files[@]}
do
prefix=${pb_file/.py/}
after_handle_package_name=${prefix//_/__}
need_replace_str="import $prefix as $after_handle_package_name"
echo $need_replace_str
need_replace_strs[${#need_replace_strs[@]}]="$need_replace_str"
#echo ${need_replace_strs[0]}
done
need_replace_str_num=${#need_replace_strs[@]}
all_files=`ls | grep -v '__init__'`
echo $all_files
for file in ${all_files[@]}
do
for ((i=0; i<$need_replace_str_num; i++))
do
need_replace_str=${need_replace_strs[${i}]}
echo $need_replace_str
sed -i "s/^$need_replace_str$/from . $need_replace_str/" $file
done
done
echo "finished"
cd .
python3 setup.py bdist_wheel upload -r internal
echo ${CI_COMMIT_SHA}
noticeMail=$userMail
content="\nversion: $version\nproject: $pkgName\ncommit:\nhttps://xxxxx/proto-center/commit/${CI_COMMIT_SHA}\n"
curl --location --request POST 'https://xxxxx/webhookurl' \
--header 'Content-Type: application/json' \
--data-raw '{
"msg": {
"content":"'"${content}"'"
} '
Copy the code -
8️ one after constructing notification, similar robots such as Dingding, wechat and Slack 🤖 notification can be added to the job of the central warehouse, which is convenient for us to know the situation of proto CICD process, and we can bring our COMMIT information and corresponding version.
conclusion
In general, there are many other solutions, each of which has its own advantages and disadvantages. The best solution is the one that suits the business system architecture and company organization. However, try to reduce the human process and improve efficiency by using tools.
Refer to the link
- https://eddycjy.com/posts/where-is-proto
- https://segmentfault.com/a/1190000022532645