Abstract

Unexpectedly, the most recent task was not solved in a week, so that I was busy at home all day on Saturday and didn’t finish it until Sunday morning when I had a flash of inspiration. To be honest, I haven’t experienced this in a long time, being stuck with a technical problem for a few days and feeling so confused, but it worked out. Although this task may not be particularly similar to yours, I would like to share the process of solving the problem with you and talk about how to solve the technical problems.

The problem background

You need to plug an open source Python project into your company’s microservices architecture (mainly Java). Open source projects are collectively referred to as OPA. The underlying opa project relies on c++ libraries such as openImageIO.

Problem analysis

When I received this task, IN fact, I was quite panic, time is relatively tight, almost a week to finish. There are some challenges to assessing the task in context. Lies mainly in

  • I haven’t written python code, c++ code in years.
  • Lack of experience in building complex Docker images
  • No experience with plugging python into microservices
  • Fighting alone, with no one to consult

First of all, I did some disassembling of this task and evaluated its difficulty with a full score of 5 stars

  • To set up the Python development environment, you need to run it locallyopaProject, development, debugging. * * *
  • willopaThe project is transformed into a Web project that provides restful interfaces. **
  • Register the service with the service center *
  • Provide a Java client for external calls *
  • Build docker Image to deploy the service on K8S. * * *

Unexpectedly, this task encountered so many problems in the process of solving

To solve the process

willopaThe project was transformed into a Web project

This task was evaluated with difficulty index of 3 stars at the beginning, but it was actually found to be somewhat difficult, but it could be completed with the help of the search engine. The comprehensive index was 2 stars, but it actually took 1 day

IDE: PyCharm Environment: Py2 / Py3 package build: PIPCopy the code

The main problems encountered are

Python2,python3 problem development environment is installed py3,opa project is a Py2 project, so with the help of tools to convert Python to Python3

Python3 is not compatible with python2 syntax, such as print’s. But python3’s syntax can run in python2, which is a bit of a trap, but 2->3 is a big version, and you can do that, so it doesn’t break down, it doesn’t throw out old stuff, it increases maintenance costs

Package Version issues

For me writing Java, python package management is bullshit, shocking is the best way to describe it. Trough point

  • There are no files for project dependency packages
  • Packages do not specify a version

However, since there are not many packages introduced in this project, I didn’t worry too much about this and installed them individually. If it’s a serious maintenance project, Python now has the necessary package management tools

willopaThe project was transformed into a Web project

The task was completed faster than expected, using the Flask package and the Flask tutorial in a day

Register the service to the service center && provide Java clients for external calls

This content problem is not big, pure physical work, easy road, a day to solve

Build docker Image to deploy the service on K8S

This step evaluated 3 stars at the beginning, but it really tormented me. It took me two working days to solve it. Not until the end of the week. Problems encountered 10+, with some notes to record some

Python3: python2 xbuba.com/questions/5

Finally, solve the problem by specifying the version

PIP install numpy -i https://mirrors.aliyun.com/pypi/simple = = 1.16.4Copy the code

2. Network problems

The problem tormented me so much that I wanted to cry without tears. Because we need to verify that the build is correct, we need to repeatedly build image, install various packages each time, PIP install Python package, which takes half an hour to complete. At home, even with VPN, the Internet is slower and the time is longer, which makes me question myself for many times. Even in 2019, I am still troubled by the Internet bandwidth, what is the meaning of human life on earth, and why I waste my beautiful life here. This problem can be solved through 3 aspects - Docker multi stage build builds multiple Docker base images and then from base images, which can reduce image construction time - change the address sources of downloaded software such as APT,yum and PIP.  - Download other wGET packets and throw them to the Intranet to access the Intranet addressCopy the code

3. The from constructed by docker image needs to build the Java base image based on the company’s internal, as well as the Base image of python project. I took it for granted at first

FROM opa.image
FROM company.image
Copy the code

In fact docker can only inherit FROM one base, the first FROM is ignored. Therefore, you need to use a multi-stage build, and copy the content from the previous layer to the new layer for details on the multi-stage build

4. The problem of c++ dynamic link library

opa

5. Ubuntu,centos Problems The company’s images are based on centos system, open source based on Ubuntu, syntax is somewhat incompatible.

The ultimate solution to

The biggest problem that bothers me is the construction problem of docker image. How to build a docker container that allows c++,python and Java based on two base images. The final solution to the problem came in the wake of a light bulb. The environment required by the project is roughly shown below

I struggled with all kinds of problems in the early days

  • ubuntu,centos
  • Various package install error issues
  • Package version problems
  • C + + library make install
  • Network download problem

So many problems mixed together, the mind is not clear. The biggest problem that bothers me is the construction problem of docker image. How to build a docker container that allows c++,python and Java based on two base images. The final solution to the problem came in the wake of a light bulb.

For me, the main difficulty lies in the docker image construction of the original project, while the image in the company and the newly added code are mainly Java runtime environment, JAR package run and several Python packages PIP install, all of which are relatively simple.

Therefore, another idea is changed, which is based on OPA image instead of centos environment of company image, so that the original image can not be changed. It’s like blocking out all the stuff I’m not familiar with.

Build a base image

FROM company.image AS builder // Copy Java environment to image copy -- FROM =builder The/usr/lib/JVM/jdk1.8.0 _172 / usr/lib/JVM/jdk1.8.0 _172Copy the code

Then build the Python environment on top of the Base Image. Finally the problem is solved.

conclusion

In the process of solving this problem, I have summarized some basic principles that may help you solve the problem

  1. Know your skill level and grasp the pace

    At the beginning of the task, assess the difficulties, and constantly compare the estimated time during the process to exceed your expectations. You need to adjust in time to keep a firm grip on the pace of the task and make sure everything is under control.

  2. Prioritize problems by sorting out which ones need to be addressed and which ones can be ignored.

    You may encounter a lot of problems, and these problems will lead to a lot of problems if you go through them in such depth, which will lead to delays and deep frustration. Only solve the problems you need to solve, not the ones you need to solve but the ones you’re interested in, write them down and follow up.

  3. Divide and conquer. You may be solving tasks that no one else has done before, and the answers are not available online. It is necessary to separate tasks, abstract out and simplify complex problems with many backgrounds, and consult people with relevant experience, which can make communication more efficient and solve problems faster

Follow the public account [Abbot’s Temple], receive the update of the article in the first time, and start the road of technical practice with Abbot

reference

Xbuba.com/questions/5…

Docs.docker.com/develop/dev…

Docs.python.org/zh-cn/3.7/l…