I hesitated about this question for a long time and put it in the draft box three months ago. The reason why I haven’t answered this question is that it seems that there is no instant that makes me feel that my level has improved by leaps and bounds. I feel that my progress has been made slowly over the years, as if there is no “leaps and bounds”. Yesterday, my colleague who brought me in left the company. At the farewell ceremony of the department, we recalled how I was so “little white” on my first day in the company, and how I became independent. So I wondered if I had to sum up one key “concept” or “technique” that improved my game, what it would be.

Author: Ju Aozai, Nanyang Technological University, Datawhale

So I looked back over the years. To be honest, when I started working on algorithms, I wrote far less code per day than WHEN I was working on games, and sometimes I didn’t write a single line of code for days. Of course, I can’t speak for all algorithm engineers. I do work in data science, but I need to keep in touch with customers and understand their needs. But there’s no denying that these days with less code, I do things more reliably and feel more fulfilled.

So I concluded a word: don’t stick to technology to solve problems.

Of course, you may be confused by just one sentence, but let me expand on that.

Don’t get stuck with technology There are three layers:

1. Don’t be obsessed with using the latest technology

Some people want to take the technocratic route, and that’s great. It’s a good habit to check arxiv every day and see the latest SOTA. But don’t change technical solutions too often. I can understand your thinking that you want to make the product to the utmost, and I have also had the thought. However, when you actually implement the product, you will find that the product is a system, and the change of your technical solution may affect other modules.

For example, our model is XGBoost, which isn’t very complicated, is it? Since we have deployed systems to multiple customers, we know what the uptime is on what hardware. For example, in the case of 10,000 pieces of data per day, we use AWS T2. xlarge instance, and the running time is 1~2 hours, which meets customer requirements. But if you see a new model in the latest paper and decide to adopt it, in addition to verifying the feasibility of your technical solution, the impact of your technical solution on the system uptime should also be re-evaluated. If your solution does work better, but the server costs several times more, how do we convince the customer? It’s all one ring at a time, and the actual situation may be much more complicated than I said. This is the world of systems.

Statistical modeling is not about obtaining perfect predictive power, but using the smallest necessary model to achieve the maximum predictive power.

Moreover, in many cases, using a more complex solution may not be the most appropriate solution. I wrote an answer earlier about how deep learning doesn’t work as well as traditional methods.

In this answer, I quoted the famous “electric fan blowing soap box” joke:

A large company introduced a soap packaging line, only to find that the line had a defect: often there would be no soap in the box. Instead of selling empty soap boxes to customers, they hired a postdoc in automation to devise a scheme for sorting them. The postdoctoral pulled up a scientific research team of more than ten people, integrated the use of machinery, microelectronics, automation, X-ray detection and other technologies, spent 900,000 yuan, successfully solved the problem. Whenever an empty soap box passes on the production line, detectors on both sides detect it and drive a robotic arm to push it away. A township company in southern China bought the same production line. When the owner found out about the problem, he was so angry that he hired a small worker and said, “Get this shit done or you’ll crawl out of here.” The young worker soon figured out a way. He spent 190 yuan to put a high-power electric fan beside the production line and blew it violently, so all the empty soap boxes were blown away.

But to be clear, not being attached to the latest technology does not mean not paying attention to the latest technology. Sometimes, the need of the company is one thing, but our technological progress is another. If we don’t care about the latest technology, we will be at a disadvantage when we change jobs.

2. Don’t just use technology

Technology is not the end of a career, especially as an algorithm engineer working on a commercial product. As an algorithm engineer, especially in commercial products, there are a lot of times when technology can’t solve all the problems; Sometimes, although technology can solve, but the medium, or even the worst, and the best policy does not necessarily need to use technology. There are four alternatives to technology, each higher than the last.

Replace technology with money. For example, when your product is launched, you find that online model training is always unsuccessful, and you find that the server performance is not enough, so your first thing is to optimize your algorithm or spend money to expand the server? If the cost of a one-day outage on a high business value project is likely to far exceed the cost of the server, you should buy the server first and save the optimization for later.

Replace technology with communication. Many people (especially designers) often have to start over in the middle of a project. For algorithm engineers, it’s easy to get wasted if you don’t have enough communication and go straight to work. For example, a bank asked you to develop an algorithm to identify which customers might be overdue. However, due to the sensitive data of the bank, you could not directly give it to you. Instead, the bank only gave you data schema so that you could simulate the data by yourself. As soon as the result is online at the customer’s site, the accuracy rate is 99%, and you are happier. But soon you find that the ratio of 0:1 of the customer’s data label is 1:99, and your accuracy rate is 99%, which is the same as a blind guess. In fact, if you communicate with the customer properly and understand the data early on, you won’t use accuracy as a metric.

Replace technology with management. This is an important ability that distinguishes advanced engineers from ordinary engineers. A project, if managed well, can make it easier for front-line developers to get more done at the same time.

Some work is A basis for another work, and the total time from A to B may be less than from B to A.

Work is never done, and even if it’s technical work, there’s a difference in value. Good technical managers should let people do the high value first.

Replace technology with vision. At this level, it is basically a CTO level, where you have a good understanding of the strengths and weaknesses of existing technologies, where you should invest in new technologies, where you should adopt old technologies, and where you don’t even need technology. But I haven’t made it to this level yet, so I don’t have much experience to share.

What’s more, for algorithm engineers, in addition to technical ability, there are many other abilities are required, as shown in the following figure, which constitute the “landing ability” of algorithm engineers.

The picture is from an answer I wrote earlier which was included in Zhihu Daily. I won’t repeat them here.

3. Don’t just focus on the technology in your field

Let’s get rid of non-technical capabilities and focus on technology. As for technical ability, do you think you pay attention to the latest papers, can tune in, and that’s it? It’s not that simple. Every time I talk about what an algorithm engineer does, I post this picture:

The small, almost invisible black block in the middle, if you zoom in, says ML Code, which is the “algorithm” part. Don’t be alarmed by this figure, of course, this is not necessarily all your work, here is the task of a team, the team may be two people, also may be dozens of men’s team, but what is certain is that no matter which company you are, an algorithm engineer is unlikely to do only pure “algorithm”, don’t forget to “engineer” three words.

As an algorithm engineer, you think I only write Python, with the occasional dash of Scala? No, you might be surprised, BUT I also write VBA. Our client is a bank, and they have an internal model verification team. We need to generate model performance reports according to the template provided by them, and the report template cannot be sent to the Internet. What should we do? I can only use VBA to write an Excel macro script to automatically generate reports. After all, the only programming environment they offer is Excel, and as for Python, PIP Install doesn’t work because it’s not connected.

So I suggest, as an algorithm engineer, to learn more about the surrounding technology, not just tuning and reading papers. Peripheral technologies are technologies that you can quickly validate and deploy. Not all technologies need to be learned. As an algorithmic engineer, you learn Linux, which makes it easy for you to deploy models; Learn Spark to quickly process big data. In this way, any new idea you have can be verified quickly without asking data engineers to do ETL for you repeatedly.

However, life is limited, and knowledge is also limited. With the end with the end, dangerous already! But if you’re an algorithmic engineer, learning Android app development seems a little unnecessary, unless you’re sure it’s useful or you’re the only programmer in your company (in which case you’re advised to jump ship).

This is a summary of my journey to self-improvement that I hope will be helpful to you.

The best time to plant a tree was ten years ago. The second best time is now. ** For those of you who are learning to program or are working, if you want to improve your programming skills, help you improve! ** I may be able to help you here ~

C language C++ programming learning exchange circle, **QQ group [951258402] ** wechat public number: C language programming learning base

Share (source code, project actual combat video, project notes, basic introduction tutorial)

Welcome to change careers and learn programming partners, use more information to learn and grow faster than their own thinking oh!

** Programming learning video sharing: **