“Live up to the time, the creation of non-stop, this article is participating in 2021 year-end summary essay competition”
Pay attention to the public account [high-performance architecture exploration], share the daily influence, reply [PDF], free access to classic computer books
Hello, I’m Yule.
More than ten years after graduation, I have been engaged in back-end work and network transmission projects. I used to be in charge of recommendation engine, and now I am in charge of advertising delivery engine and advertising settlement engine. The main languages used are C/C ++, Python and shell as auxiliary development.
Welcome communication and progress together.
Writing in the front
One day at a certain time, I suddenly realized that I had graduated and worked for ten years. It is time to write an article to record my ten years, which can be a psychological comfort to myself and also a summary of the first ten years since I entered the workplace.
The college life
I went to the university in 2003. Before going to the university, I had no contact with computers at all. At that time, the two most popular majors were computer and biological engineering. It is often said on TV that the 21st century is the century of biological science. And I, because I have never touched a computer, so more curious, in the application of the time, write for now the most volume of an industry major: computer science and technology.
So one day in September, I came to a city in northern China by train nearly half way across China and began my so-called college life. Still remember, the first computer lesson, in the room turned on the computer, double click on the neighbor PM on the net, input the website, the first song, listen to is the “love you” Rene liu, so much so that many years later, when I hear this song familiar melody, will recall their freshman year computer class for the first time.
At that time, just out of the repression of high school life, just like a bird with wings, flying freely in the air, holding the learning mentality of “sixty thousand years, one more waste”, muddled along, basic courses and professional courses are evenly divided, it is not to leave a little regret.
In this way, I spent the ignorant freshman year, the hesitation of the sophomore year, and the sinking of the junior year. Until the junior year, I began to consider my future, followed my heart, and chose to take the postgraduate entrance examination.
Graduate life
Mu fame came to a school in east China famous for studying hard, began his own volume life. I ate a lot of professional courses, which was also a punishment during my undergraduate course. Daily life, dormitory, canteen, library, began to learn their own professional skills. The first programming book I bought was c++ primer. I borrowed c++ university courses from the library and began to learn programming. With extensive exposure to professional books, I began to understand the underlying principles more deeply. A lot of books are at that time began to contact, but also to their own later into the industry, laid a solid foundation.
Practice life
At that time, I began to feel a sense of crisis about my employment. All my classmates got internship offers from Aliyun, IFlytek and other companies. Finally, when I was about to collapse, I got an internship offer from a certain institute of Chinese Academy of Sciences. I remember the first word of the dormitory buddies is: congratulations to you, the daughter-in-law of many years boils into a woman.
Came to Beijing, lived in the dormitory arranged by the institute, followed the mentor to do a national 863 project, ddos attack related. Because I am an intern, I am in contact with some optimization classes, such as memory pool.
During my internship in the institute, I began to contact CSDN website. Except for the project, I spent all day in the c++ section answering questions. At the beginning, I got a high score and a good ranking, but due to other reasons, I stopped logging in this forum. Still remember inside have a technology very cow group (transliteration cake child hall), oneself fortunately also joined, behind still quit.
With the end of the project, I chose to have an internship in another big foreign company, which is located in Xi Erqi (you can guess which one it is). The project team is to build a whole set of office system under Linux, similar to Word under Windows, based on OpenOffice. It took me two or three hours to compile the whole project with one million lines. I was in charge of solving memory leaks. Here, IT should be mentioned that mentor was a great leader at that time, and basically the whole company would go to him for problems, which deepened my understanding of technology.
Entering the workplace
Enter traditional industries
The first company, a traditional industry, had the honor to enter the company’s most powerful technology research institute, do network related RESEARCH and development work, that is, in this company, began to study TCP, the TCP/IP detailed look again and again, do not understand to check information, even download the source code down for analysis and research. It was there that I first learned about Linux.
As the company’s business is oriented to state-owned enterprises and mainly runs on Windows system, the first requirement is stability, so the update iteration is slow, and sometimes a function has not been optimized for several years. Later, as Linux system was used more and more in China, the business of the department also needed to run on Linux system. At that time, there were two choices: one was to write a code again on Linux system; the other was that the existing code could be completely transplanted to Linux system, that is, the same code could be used in Windows and Linux two systems. Finally, for the sake of efficiency, I chose the second plan.
As you all know, although c++ is a cross-platform language, the API name and underlying implementation are system dependent. For example, WaitForSingleObject, which is STD ::lock under Linux GCC, requires the same function, For example, WaitForSingleObject, which is a built-in API for Windows, is a redeveloped function for Linux that has similar functionality to Windows. I was responsible for the first time coredump appeared and the first time GDB was used for debugging.
Internet entry
In those years, when the formal Internet industry began to be hot, I was also eager to move. Therefore, AFTER several setbacks, I entered an Internet industry, which was also my first contact with the Internet.
When I just joined the company, I was really not used to the fast pace inside. It often took only a few days for a demand to be put forward and put online, which was absolutely impossible in the traditional industry before. The first time I went online led to the online service Coredump. I learned a lot in this company and became more and more proficient in the business. I began to take charge of the whole recommendation engine, which laid a foundation for contacting advertising in the future.
Contact the advertising
Because the company’s business development encountered a bottleneck, plus some other reasons, and began to see opportunities. Recommendation engines and advertising engines are similar in both architecture and function (index-> Filter ->rank, etc.). So I was lucky to enter the advertising industry.
In the company, a great influence on me, if the company gave me an opportunity to enter the Internet, so in this company, I started thinking, how to make better service performance, how more proactive in terms of system design, also be gradually contact architecture, from architecture to understand the whole business, and the rationality of working backward from the business to architecture.
As we all know, advertising has a high requirement for RT (Response Time). Every Time RT increases by a few milliseconds, the conversion rate of advertising display will increase. Therefore, for the service that requests tens of billions of times every day, the performance of a sub-service will affect the whole advertising link. Such as:
- The set of AD candidates is too large, resulting in a longer processing time
- A block of code consumes too much CPU
- Too many useless parameters in the transport chain
- other
A DMP service is implemented. For performance reasons, all operations are carried out in memory. Inside used a lot of memory allocation and release, after the launch, found a phenomenon, is the memory utilization has been very high. If delete or clear is called, the memory usage is still not very high. In order to solve this problem, I used a variety of memory detection tools and added logs (adding both memory application and memory release), but still found no problem, so I went to investigate the underlying library, to study the principle of memory allocation, by studying the source code, investigating the cause of new and or malloc, and finally found that in the underlying API, Even if delete or free is called, memory is not immediately released to the operating system, but is stored in the buffer (this is why memory usage is high), and is reclaimed by the operating system after a certain amount of time. To be reclaimed immediately, malloc_trim(0) is required.
It is the same DMP service, but the performance is not very high due to its large memory usage, and a lot of Misscache is caused. Finally, this problem is solved by changing the page size of the system from the default 4K to 32K (this is because too small page will lead to a lot of missing page interrupts, see < In-depth Understanding of operating system > for details).
Programming ability
As programmers, the ability to program is the foundation of our existence. From my own experience, I feel that the growth of programming ability mainly has the following parts.
primary
Basic programming skills, that is, “can use”. That is to say, using a variety of system apis to get what we want.
Programming begins with learning the syntax of the language, such as if/else while, etc. As a beginner programmer, it is recommended to use the introductory level books instead of reading some advanced books when getting started. (for c++, <c++ primer> is not for entry-level readers, but for those with some basic knowledge of c++, its title is seductively set.)
In addition to reading documents, programming is super practical work, so be sure to write more code, only in this way to really become proficient, there is a saying “no other, only hand familiar with the”, very suitable for entry-level programmers. As an entry-level programmer, try not to use IDE, this will make IDE dependence, resulting in handwriting is difficult, I hope to try to do all API names can be typed out, over time, you will become more and more skilled.
In the primary stage, the core is to write more, study hard practice, this is very important, only in this way to lay a solid foundation for the future.
The intermediate
The primary goal is to be proficient in using programming languages to achieve functionality, while the intermediate goal is to improve “analysis and problem solving skills.”
As a programmer, it is very normal to have problems in the process of writing code, but how to effectively and efficiently troubleshoot problems is the biggest gap between individual programmers in programming ability. If they have strong problem-solving ability, they will easily be recognized by superiors and even colleagues in other departments.
In the ability to check the problem, the first thing to master is some basic debugging skills, easy to use debugging tools, such as commonly used GDB, GperF, btrace and so on. Mastering debugging tools is very helpful to solve problems. Sometimes people in the ability to check the difference, may just because other people than you know a tool, in addition to debugging techniques and tools, will check the problem of higher level and the advanced stage of programming ability have the very big relations, is to understand the principle, a programmer who understand the principle of level in question is a significant gap. Try to write some procedures that will cause problems for yourself, actively look at how others check problems, and actively participate in troubleshooting problems. Many people who have strong ability to check problems in the end are just because of “no other, but familiar with them”.
In addition to checking for problems, better programmers are better at avoiding problems in the process of writing code. They know how to avoid problems in this area, what problems can occur in this area, what conditions or logs need to be added, etc.
At the intermediate level, it is recommended that you take the initiative to analyze and solve problems and become a good programmer who can write high-quality code and troubleshoot problems effectively.
senior
For programmers with advanced programming ability, a standard is to “understand the implementation of the underlying API”, know how it is, know why.
In the process of learning the underlying implementation and troubleshooting problems, it is very important to understand how the programming language works. Learning the underlying implementation of the most direct to see the source code, but this is difficult for many people, so you can try to start from the document or some books (<STL source code analysis >), while reading, while finding the corresponding source code to analyze, so that you can achieve twice the result with half the effort.
Someone once said that language is just a tool to solve and analyze problems, don’t limit yourself to language. I quite agree with this statement. For example, although c++ performance is very good, but its implementation process is more troublesome, there will be many pits. For quick operations, such as parsing online logs, python or shell can be used.
Architecture ability
As you write more and more code, you’re inadvertently starting to think about whether it’s optimal for performance, whether it’s architecturally sound, whether it’s scalable in the future.
I completely set up a system five years ago, that is, five years ago, when I joined the current company. When you start commercializing, there’s nothing, everything has to be built from scratch. For example, if a flow comes, how to request each line of business, an advertising order comes, if it can quickly enter the advertising system. Then you need to allocate modules reasonably.
As more and more code is written, more and more problems are solved, more and more difficult, more and more modules are designed, and in a subtle way, the ability of architecture is developed. When designing a system, you start to think ahead, think about decoupling, think about expansion.
Architecture ability is not something that can be learned at a time. More needs to be learned from practice, to practice, and over time, this ability will be improved.
tips
Team spirit
In a company with perfect processes, requirement review, development, testing and launching are indispensable, and the absence of each link may lead to online failure. Therefore, the completion of a requirement requires the cooperation of all departments.
There’s a problem online, we work together to solve it, and that’s the process of growth.
Don’t show off
Code is something that needs to be passed down. You don’t want your code to be yelled at by someone who takes over from you, so make sure your code is concise, clear, and unambiguous. Don’t use arcane programming techniques just to show off some technical prowess. Soon, not only others, but even you, may not understand the code.
Reasonable structure
There are thousands of architectural options, and the most important thing is to choose the right one. Don’t just do architecture for architecture’s sake, but consider efficiency and functional requirements.
Achieve a balance between architecture and development efficiency, and then look backward to see if the existing architecture design can meet the requirements.
Proving inconclusive
The so-called attack, that is to attack, can also be understood as a siege. That is to achieve product objectives and meet functional requirements; All guarding, that is, guarding, can also be understood as guarding a city. That is, for online problems, can guard, quickly solve.
It is easier to attack a city than to defend it. Online bugs are never solved, it takes patience and the ability to attack and defend.
Code review
Codereview is essential, which requires team spirit and can solve many problems in the early stage of launch.
However, it also has disadvantages, that is, it may lead to conflicts among people. For example, when A function is implemented, A says it will be implemented this way, and B says it will have better performance. As time goes by, there will be conflicts between teams. This needs to find a balance, for the opinion of dispensable, can not mention…
Code comments
❝
Code is the best comment. That’s the ultimate goal, but a lot of people don’t get it, including me. Make as many comments as possible in your code for the following reasons:
❞
- For their own future maintenance
- For others to take over
Especially for functions with complex functions, you can write them in points 1, 2, 3, etc. If necessary, write them in CF, and add url links in the function comments.
The code structure
The structure of the code must be clear and concise. You can read the function description to know what it does, and you can read the class name to know what it does.
What the code structure shows is actually a modular logic idea of the program. Simple and clear structure, also can explain the program to achieve simple logic.
Code style.
Programmers work as a team, each with his or her own different programming style, and without a unified style, the entire code would be messy.
The Google code specification, which is now used in the industry, is also recommended.
Development efficiency
C++ programmers have a characteristic that they don’t like to use IDE, it is a kind of their own pride, prefer to use vim(I am), all code development using vim, it is a bit of a b-grid.
Besides the technology
❝
Learning is like rowing upstream; not to advance is to drop back. Programmer, is a continuous self-learning, self-progress process, need to keep learning, learning new knowledge, can not be eliminated by technology.
❞
Stay true to your original aspiration and stick to it
I interviewed too many people, many of whom graduated from famous universities and had a good resume, but they all had a characteristic that they changed careers. Some of them changed from technology to products, and some from technology to testing, and gradually lost their competitiveness in the industry.
Technology is our foundation, this must not give up, even if you have made it to the management level, please contact the code as much as possible.
Goals, ideals
Ideal is dream, can not be realized, but it is to support the faith of the heart. Only with ideal, can you set goals, otherwise the goals are always changing, there is also a lot of anxiety in the heart.
With your mutual encouragement.