A few days ago in the group of old colleagues catching shrimp chat some topics, only to find that the day has been catching shrimp in the past ten years. Shrimping is still mentioned today, but often as a lesson from one of Mr. Xu’s entrepreneurial failures. But for those involved, the experience was not just “a lesson in failure.”

The fate of shrimp net and I began in 2006. At the time, I was working for Donews, a thriving IT news site. At that time, Donews was like juxianzhuang of Jianghu, with regular gatherings every Friday night. Although I was only a programmer, I had the opportunity to observe li Xiang, Wang Zhidong, Li Kaifu and other IT tycoons up close. This opportunity refreshed a lot of my knowledge, but also let me master a lot of “gossip”, I first heard of catching shrimp and Xu Yirong, it was at a party.

Since about late 2006, the conversation at parties has been less about IT and more about housing and stocks (Keso was there, and he wrote about this earlier in an article). Donews itself was going through some changes, and I wasn’t doing too badly, but I was thinking about my next job.

One day, the boss sent me an email asking me to contact him. The original is Xu Yirong hope to develop content cooperation, with RSS output customized content to catch shrimp. I’m really good at this because I’ve done RSS development before.

Nowadays, many Internet users probably don’t know what RSS is, and RSS has really fallen by the way. But TEN years ago, RSS was a revolutionary technology, and even today it has many implications. So LET me explain a little bit more here.

Simply put, RSS is an “open API” for content distribution on your site, and it’s “stripped clean” pure content. It is common practice for content providers to follow the RSS specification and export the most pure and core content (excluding irrelevant content such as advertising) in XML format for others to grab.

This technology seems incredible now, but it was valuable then. In my opinion, there are several reasons.

First, since scraping technology was far less advanced than it is today, it was difficult to extract text accurately from a complex web page, even though many websites stored jumbled content in their databases, and RSS promoted a complete separation of data and presentation. Second, everyone was trying to get traffic, and by providing “pure” RSS output, it was easy to push your content to various platforms and build a lot of access points. Third, perhaps most important, RSS provides close “burn” function, because each site provides classify RSS output, you can easily to reorganize the information aggregation, “simple spell spell XML” to sina, netease, sohu sports sports content together, this can be as pioneer of nowadays popular personalized reading; Fourth, you no longer need to refresh the web page frequently to see updates, RSS readers can periodically check for updates to alert readers.

Because of the many advantages of RSS, people quickly realized that with RSS, you don’t have to go through a lot of trouble to visit a website. Perhaps this is the way forward?

The first time I remember receiving a text from Yi Rong was on a Saturday afternoon. At that time, the weather was very good, I was repairing the bike by the second ring road, suddenly received a polite and polite message, so make an appointment to meet the next morning.

The next day I set out early because I had heard that Xu Yirong was very punctual. After catching shrimp to see two people, enthusiastic and cheerful is Xu Yirong, also enthusiastic partner Chen Zhenyu (or my hunan fellow), he seems to have long hair in the impression. At that time, we mainly talked about technical cooperation, and we soon decided on the cooperation plan, which channels to output and how to output. Because we were all engaged in technology, the communication was quite pleasant.

When I went to work on Monday, I began to plan the development. Soon the development was completed and the docking was successful. Yirong even sent a respectful email to express his gratitude — grabbing shrimp was a small site at that time, and the content cooperation was the light of Donews. That’s the end of the matter.

Not long after, Yi Rong sent me another email, saying that they wanted to meet again for dinner and chat. I feel a little strange though — the collaboration is done. However, Yi Rong said, “I feel that we are not familiar with each other, and we will be familiar with each other if we talk more.” So I readily agreed.

This time, yi Rong and Zhenyu played host and talked about more topics. Along the way, they also introduced what they are doing, what they are going to do in the future. I was interested, too, so we just kept talking.

However, yi Rong and Zhenyu looked at each other, stood up and reached over to me and said, “We think you are quite suitable. We would like to invite you to join us. What’s your opinion?”

Although I was considering looking at other jobs, the move came as a surprise to me (and never again in my career). But then I thought of their directness and sincerity, and promised to go back and think it over. Within a few days I had made up my mind to join. At that time, everyone was very cheerful. Before I asked for resignation, Zhenyu sent me a text message saying, “Your workstation and computer are ready for you when you come.

When I first joined shrimp catching, there were more than ten people in the whole team. In the technical team, in addition to the two founders, there are several colleagues who have worked for five or six years: Changle and Mingzhu, respectively responsible for the back-end and front-end. Then there are those like me who have just worked for three or four years, Such as Chongli, Wang Xi, Yonghui and Yi-tao. Next are all the interns in school: Diao, Chen Kai, Lu Ming, Guodong, Yong Chao and so on. At that time is responsible for the UI xiaoxiao, and later is Sina to wang Da. A little later than me, hai Peng, Xi Xi (Jiang Zhixi), Zhu Rong and others came. Later, there comes intern Chen Tuo. Overall, the team is young, energetic and gets along well with each other.

I was impressed by the intern. I had a hard time finding a job when I got to my senior year. Like Guodong, Yongchao are computer department of Peking University, the foundation is quite good, work very hard. Only then did I find out that internships are not the “cheap labor force” of the legend outside. For interns, they can experience the career and workplace in advance, and for the company, they can lock outstanding students in advance. Originally, good students have been divided up in sophomore and junior years.

Judging from my later experience, perhaps Beijing has a unique advantage in this regard. No matter in the Yangtze river Delta or the Pearl River Delta, it is difficult to have the general atmosphere that “top students from prestigious universities are willing to intern at startups with uncertain future”. There are plenty of places for such children to go everywhere, but only in Beijing do a significant number of people ignore the pressures of life and devote themselves to doing interesting things with an uncertain future. But then again, “changing the world” is often accomplished not by calculation, but by complexes and beliefs.

Of course, it may also have something to do with the atmosphere in Beijing during those years. I have discussed with many friends in Beijing when Beijing is the most lovely. A common answer is the period from 2006 to 2008. Although we didn’t have much money (I rode a bicycle every time I went from Chaoyang Gate to Baofu Temple), There was a great sense of tolerance in Beijing at that time. Moreover, the regulations in various aspects were gradually relaxed, and everyone felt a sense of “striving for a bright future”.

I feel the same way myself. I really miss those days. I remember that shortly after I started working, I heard that The Germans had developed a new Web server called Lighttpd, so I went to study it and wrote something on the Internet. Not long after, someone contacted me hoping to talk about Lighttpd. When we met, I found out that this person was A Bei, the founder of Douban. It seems inconceivable today, but at the time it was “unexpected and reasonable”.

There are many such experiences. One night in 2006, I was drinking with Han Lei, Huo Ju and Zhu Huilong in GUI Jie until it was very late. At this time, two wandering singers came and we sang along. Singing, the shopkeeper gave more beer. We were all surprised when we asked and found out that our friend at the other table across the hall didn’t know us but thought our singing was good, so he ordered a dozen beers for us. So, we bring a cup to the table of friends do not know, while saying “thank you”, while drinking a few, so that even if do all the courtesy.

I had a dinner party in the summer of 2007, and I had no idea who was coming, just friends asking me by ear. As a result, at the beginning came a very gentle young man, who introduced himself and said, “I am the Queen of the Chinese pine Club”… Later, another guy came along and asked Feng Dahui, formerly of DBA Notes. After a while, Li Xiaolai teacher, blog view zhou Jun teacher also came in succession. Many of them had never met before, only heard about them on the Internet. In person, take turns to introduce themselves, no one is arrogant, we eat while chatting. I have been away from Beijing for many years now,

Back to the shrimp team. The team was made up of enthusiastic young people in that era, so it was happy, if not professional. On the second floor of the office, there was a room and crib for overtime work, which (and mouthwash) kept us through many a sleepless night, even though the hygiene was terrible and the air always stinked.

Lunch also speaks to the team. At that time, shrimp was packed for lunch, and we were in a restaurant near Huaqing Jiayuan, about 20 minutes’ walk from Baofu Temple. The daily meal routine is asynchronous communication to ensure efficiency: first call to order, then wait for the phone to announce the service time, assuming it is almost time, group to eat. When you go to a restaurant, all the dishes are just put on the table, and you sit down and eat them all. It usually takes less than 10 minutes after the table turns two times. On one occasion, Yi Rong arrived five or six minutes late for an errand, only to be told that “your people have left and the tables are all cleared up.” This style has profound sequelae, AFTER I left shrimp, many people have been talking about “eating fast”.

Also interesting is team building. In 2007, I organized a self-drive to Hebei Laiyuan and the grassland in the sky. That night, the tent crowded by boys was filled with stench. When I got up in the morning, several people could not stand the smell and went to see the sunrise. Yi Jong didn’t look under the damp mat that night, either. He said there was a hunk of horse manure under the spot where he slept, causing him to get cold in his stomach for days. On the return trip, we met a huge traffic jam on badaling Expressway and barely moved from 12 o ‘clock at noon to 8 o ‘clock at night. This was the first time I spent the night in a car on the freeway, and I took a lot of nighttime high-speed shots with my tripod and SLR. Several times during the night I would wake up in the passenger seat after a kilometre or two had been cleared, and he would start the car a bit further and then fall asleep again. We returned to Beijing after 8am the next morning, and everyone was happy: we had an extra day off. I asked Yi Rong if he remembered that I had asked him to drive the night before, but he didn’t remember at all.

Of course, for the company, the team is young, the atmosphere is good, but only one aspect. Being a young team means making mistakes, requiring discipline and repetition. Not least because there is no set of rules and regulations for a large company, there is more effort involved. Later, every time I think about it, I think Yirong and Zhenyu must have put a lot of thought into this aspect. They could keep the technical pursuit and trust of the whole team at a very high level without relying on hard work or brute force. I don’t see many people who can actually do this.

I remember once seeing haipeng’s code in the air, I was surprised to find that the internal management system was also written so beautifully, which made me feel ashamed. Then I believed that IBM, where he worked before, would not only have clumsy and rigid processes, but also develop a good sense of norms and professional qualities. I really knew too little before. Of course, I can only admire others, once I spent some effort to write a small module around, was also praised by colleagues “write really beautiful”, was really elated. Nurturing and affirming technological pursuits contributes to a healthy technological climate. On his first day, Chen Tuolai, an intern, surprised me by saying he would install Linux on his computer instead of Windows. Many years later, when I met him in Changsha and talked about this matter, he smiled and said, “Actually, I am not familiar with Linux, but at that time, everyone around me was using the command line, and it would be difficult to use Windows.”

Perhaps it was because of such a team and such an atmosphere that I later looked back on my shrimp-catching experience and felt that I benefited a lot from the technology.

There are a few examples that stand out to me. The first is the task dispatch system of the whole fetching. Because there are so many RSS feeds to grab, how to efficiently schedule them to catch updates as quickly as possible without doing too much wasted work becomes a big problem. News updates are predictable, but a lot of RSS comes from blogs, where the timing of updates depends on the mood of the writer, and there’s no algorithm that works. The problem is solved in a very clever way: by starting with the reader. Whenever a reader login event occurs, the RSS Feed to which the reader subscribed is checked, and if the corresponding Feed was last captured half an hour ago, it will trigger another capture immediately.

In other words, it’s not about the technology being omnipotent, it’s about giving the reader a sense of “grasping enough, fast enough.” The real issue is getting the best results with limited resources. Later in many work, I found that technical personnel often ignored this point, and insisted on pursuing the effect that could be achieved at a very low cost. Perhaps this is not because they are stupid, but because they have not seen examples of enlightenment.

The second is the storage of shrimp. In order to be readable, all the captured content needs to be stored locally, and millions of feeds generate a huge amount of data every day. Things like HDFS didn’t take off at that time, let alone cloud computing and cloud storage. If you have a large amount of data, at best you can use a library like The Tokyo Cabinet, but this kind of thing itself is not very reliable, and industrial use is always problematic.

Shrimp catching is also a simple and crude way to solve this problem. Gzip compressed binary contents to the MySQL table, only provide ID and GUID query. This allows a single table to hold 300 million pieces of data, and the rules for dividing tables are fairly simple.

This may sound cheesy and far-fetched, but it solved the problem, and data consistency is guaranteed thanks to MySQL’s existing master-slave replication. In the same year, the Chinese version of The Art of Capacity Planning came out. I read it and thought about MySQL’s storage solution, and realized a profound truth: There is no absolute superiority or superiority in the program itself. It is the true meaning of engineering to extract key indicators according to the actual situation, take various variables into comprehensive consideration, and obtain an appropriate program through analysis, estimation, design and verification. In the end, various constraints, conditions and concerns can be thoroughly understood.

This was probably the “common fault” of the technical staff at that time. Because there aren’t that many libraries and frameworks out there, everyone has to stick to the basics. In a recent discussion of the technical proposal, I proposed an approach that seemed simple and direct, and had nothing to do with “high and noble”. After a close look, we found it feasible. I said, “That’s the kind of thinking that happens to people who were growing up in tech in my generation.”

That was just my first impression of catching shrimps, but as I worked more hours, I saw and learned more impressive things.

Before I went to catch shrimp, my focus was mainly on development. Write the code, run it, and the job is done. But that’s not how industrial-grade programs work, as I discovered when I went shrimp fishing. First of all, all service programs have a special startup program, to ensure that the real program can not die, can be restarted quickly. At the same time, all the procedures running state, also must be clear at a glance, not only to have real-time view, but also daily, weekly reports and so on all kinds of statements.

This is the equivalent of writing a program and asking yourself: What key pieces of information should my program provide to identify the state of the program and how that information can be accessed — a process of meta-design that forces us to think more deeply about our work. I also learned a lot about what output to a program should look like, and the difference between it and human information.

Unfortunately, to this day, I find that there are a large number of programmers who never think about such a problem, and their programs are running, neither seeing real-time status, nor providing daily statistics. Problems can only be corrected by the naked eye, not only unaware of the inefficiency, but willing to play the role of “savior”. These, in fact, are the lack of strict training performance. Recall that after we all quit, the server “streaking” for half a year unexpectedly did not have any problems, relying on all the efforts made in the project before.

Also worth mentioning is the “spirit of responsibility”. At that time, some basic modules were written by Zhenyu in the early stage. There were ten famous lies at that time. The first lie was that “Zhenyu’s program has no bug”. As a result, one day when we talked about the program at lunch, it seemed that zhenyu’s program logic really had a bug. He looked solemn and immediately put down his chopsticks and went back. It was only when the bug was fixed that he was relieved. In fact, the bug did not immediately have a significant impact, and Zhenyu was already very busy, but he still maintained the persistent pursuit of quality and technology, which made me ashamed and admired.

The shame and admiration are warranted. Once there was a bug on the line and I didn’t solve it for two or three days. Every time I was asked about it, I would say “I’m working on it”. It was not until zhenyu asked me one day that I admitted, “I can’t move it out” and put it off until tomorrow. I remember his expression changed at that time, awe-faced said: “impossible, I do not believe that there is a bug can not tune out.” And take me with you. Multi-machine, multi-thread system, the original debugging is very complex, but he leaned on GDB, stepping on the swivel chair in front of several computers, from dinner has been adjusted to late at night, we finally positioning, solve the problem, out of the company home has been the stars all over the sky, a look at the watch is already three o ‘clock. I always felt that the IT industry, young as IT was, could learn a lot from the traditional mentoring system. At least later, when I was leading the technical team myself, I said “no way, I don’t believe there are bugs that can’t be tuned out” several times, and then pulled up the chair and sat down to prove that I was right.

Shrimp-catching is technically strict, but not rigid. A lot of times it’s “task-management,” where you’re given an abstract task that doesn’t limit the solution, but requires the end result. It looks liberating, it’s really painful to solve, but it’s also empowering.

Not long after I started, some bloggers reported that shrimp capture was slow. They published a blog of their own, while other online readers might catch shrimp in half an hour, we took an hour. At that time, it seemed to me that this was a small probability event (a few cases of slow grasping among hundreds of thousands of seeds), which was of little significance. However, yi Rong emphasized this matter in several weekly meetings, and I had to face great pressure to solve the problem of “how to comprehensively surpass the competitors in grasping speed”. I racked my brains and thought of various methods, and finally solved the problem. After that, I began to understand what “solving complex problems” is.

Another time, because the frequency adjustment mechanism of the crawl program was out of control, Sina Weibo blocked the IP address of our crawl server. I still remember that Zhenyu sent me a text message saying “I have to finish the Sina thing today” without any instructions on how to do it. I find the friend that had worked in sina before, toss and turn dozen more than 20 phone, contact the technical personnel that executes screen finally. After I confirmed the commitment to fix the procedure, after not overload grab, the blockade problem was finally solved.

It also gave me a lot of inspiration: one suggestion at the time was to get lots of proxy servers to catch, but I didn’t do that. Of course technology can solve the problems that arise from technology, but it is not the only way. We can and should solve them through communication. Knowing this, I benefited a lot in my later work and even in my job search. After I quit from catching shrimp, THE HR asked me to illustrate my professional qualities in an interview, and I gave this example. From the end result, the answer should be good.

I thought it would be easy to write the story ten years ago, but when I actually sat down to write it today, I found it impossible to record all of them. Although it’s been ten years, those memories are still vivid. Every evening ten years ago, I watched the lights in the surrounding residential areas turn on gradually from the high office building of Baofu Temple, and then turn off gradually at night before I left work. It seems like only yesterday. No matter what people say today, that experience has been woven into one’s own, and there is no way to separate it, nor to view it coldly as “objective.”

When we are young, we are often inspired by success stories, moved by talk of being “exceptional”, “different” or “exceptional”, only to discover that we are most likely just part of the “kind”, “crowd” or “crowd”. But who’s to say that’s not necessarily a good thing?