During my one-year internship, I became a big data developer from a beginner in Java.

First internship offer

At the end of the first year of my freshman year, I got my first internship offer in Java development by learning Java by myself. I knew very well that my Java level was limited at that time, and I also knew that I got the offer not because I was excellent, but because the company recruited more than 20 people.

In July, 2017, I packed my luggage and experienced the process of finding, cleaning and checking in with my roommates for a whole day. The next morning, I entered the company for the first time. Eight people in the development position were arranged to wait for the training of new recruits. Training is mainly divided into: data security training, database training, Linux training and so on. Before the training, it was said that there was one person who could intern in the big data group, so I was full of desire and enthusiasm at that time, so that I didn’t learn Linux in college for a year, and learned the basic commands thoroughly after three days of self-study.

After the training, the other seven people were divided into different groups and left the vacant station one after another. In the following month, I remained unclaimed, and no one mentioned the internship quota of big data. During the most comfortable days of my internship, I decided to teach myself big data. At that time, online big data courses were not as abundant as now, so I bought a set of big data courses from a treasure for 25RMB and began to study by myself.

The process of self-study was as bumpy as could be expected. I remember when I started building Hadoop with videos, it never worked. Or a version problem, or an operating system problem, or a personal ability problem. At that time, there was no one to guide me, so I had to explore by myself. It took me 20 minutes to build Hadoop in the video, but IT took me five days to collect information and keep trying, and finally I built it on my VIRTUAL machine.

One day after a month, I was finally taken away, the person who entered the big data group turned out to be me!

First introduction to big data

When I first entered the big data group, I still remember that my mood at that time was cautious. At that time, there were not so many young people in the group. They felt that they were all masters and dared not speak much. They could only sit on the station silently. The first job was about Hive. It was easy. As simple as a list and a HivesQL, I just paste each item in the list into the SQL where condition blank, execute it separately and save the screenshot. In my spare time, I was also assigned to write various engineering documents.

So between the ideal and the reality of the work for a month. When I thought internship career will be spent in this light, life turned a corner. The young operation and maintenance guy in the group decided to quit, and I was called into the office by the leader. With all kinds of chicken soup, I bravely said “I can” for the first time, and I began to transfer my job to become a big data operation and maintenance guy.

Operation and maintenance work is to write some monitoring scheduling to ensure normal production, sometimes also need manual intervention, so go out to play also have to carry the computer in case of need. Although the technical content of my operation and maintenance work is not high, BECAUSE of this job, I am closely related to my colleagues in the group and gradually integrated into the group.

Life, there will always be a reason for you to try, for others, or for yourself.

About ideals and learning

How old are you this year

Although I am an operation and maintenance, I always have a dream of development. After coming home from work, I will continue to watch my courses to learn big data. In the daily work of the company, I began to study after finishing the work arranged by the leader to sort out documents. I love listening to the big guys talk about technology at work, even though I don’t understand a word of it. So at that time, I gave myself the goal is, must understand the technology that the leaders discussed, so I began to use the morning bus hour to read the public number of articles, a day to ask myself to read at least five articles, when one day turned back, found that I had paid attention to nearly 300 computer industry public number.

That was before Flink took off, and the company was using a commercial version of IBM Streams, a platform that struck me as stable and niche. Its development Language was SPL (Stream Processing Language), a development Language that was only available on this platform, so there was almost no takers for the platform at the time. In the process of operation and maintenance of this platform application, I began to learn SPL by myself. At the beginning, I studied other people’s programs, later I learned grammar from the few materials on the official website, and finally I began to take charge of the development of SPL. At present, I am the only one who can develop SPL in the company.

Popularizing a simple SPL example:

composite Main {
 graph
 stream<rstring sourceFile, rstring movedFile, uint64 fileSize> Dir3 = Di {
  param
  directory : "/tmp/work";
  moveToDirectory : "/tmp/active";
  output Dir3 : sourceFile = FilePath(), movedFile = DestinationFilePa
}

stream<rstring name, uint32 age, uint64 salary> Beat = Beacon() {} stream<rstring name, uint32 age, rstring login,
tuple<boolean young, boolean rich> info> Annotated = Punctor(Beat){
  param punctuate : age >= 18u;
  position : after; 
  output Annotated : login = lower(name),
  info = { young = (age<=30u), rich = (salary>100000
 } 
}
Copy the code

The internship was the fastest growth time in my entire computer career, which enabled me to improve my skills, expand my horizon and establish the direction of my career. At that time, I was full of endless thirst for knowledge.

About overtime

Although many people, including me, are very resistant to overtime work and 996, it is undeniable that the days of overtime work as an intern make me grow up quickly.

Seventeen was the year when the company started big data, and also a turbulent year for big data platform. In the big data industry, when data is the core, real-time access of data is the fundamental and top priority of big data. One night in November, the access program was about to be cutoxed as the data specification was updated. As an unknown bystander, chose to stay behind as a supporting actor.

There was no problem in the test before the upgrade by the colleague in charge, but there was a garbled code problem when cutover was performed at ten o ‘clock. At that time, I did self-study of Java, so sitting beside followed, followed them in the tens of thousands of lines of code line by line to find problems, see at five o ‘clock in the morning from ten o ‘clock at night, dazzling, when I see a global variable and a local variable in a method named the same time, weak pointed out, modify the problem after the data is no longer the code, Code level issues are resolved.

At 6 a.m., the new program started, but after running for an hour, the OOM ran out of memory. That’s when I started learning about the JVM for the second time, checking memory and GC through Jstat, Jmap, etc., tweaking JVM parameters until midnight, and finally adding processing machines to keep things running for a while. I went home at 2 p.m., slept until 6 p.m., went back to work until 4 a.m. That week I was number one in the company with 65 hours.

In the days that followed, everyone got together Saturday and Sunday to look for problems. The effort paid off. After a long period of testing, it was found that Kafka0.8 had a throughput bottleneck that caused the performance problem, which was later resolved after the upgrade to v1.0. However, this time, I really integrated into the collective of big data. It was also at this time that I became a participant and understood the whole process from data access to storage, which laid a foundation for me to be fully responsible for the development of big data programs in the future.

The days of internship, thanks to the nights of working overtime that winter, brought me closer to development. Thank you for the 101 bus that I take home, which runs until 1am and makes me feel safe on many nights after work.

After the internship, the most important thing to thank is not the person who laughed with you, but the person who pointed out your mistakes and helped you.

About friends

I would like to thank everyone for their help, especially brother Gang, who took me to the door as a teacher and friend, and Brother Chao, who helped me selflessly and often rode his car.

It is very important to meet a good teacher during the internship. Perhaps a lot of people yearn for the internship life is easy, can laugh every day, these are important, but more important is how many people are willing to teach you their knowledge, what you have learned.

During my internship, I was undoubtedly lucky. There were people who taught me everything, guided me along the way, pointed out my shortcomings, talked about technical life, and gave me a full explanation when I made mistakes. When I grow up and look back, how many more people are willing to help you patiently? Although they often said that they were willing to teach me because I had worked hard to learn, I knew that they were working hard to train me.

On days when I don’t work overtime, I usually study from the office until 8:30, which is sometimes the same time When Gongo goes home to study technology or watch football. When I have problems in study, I turn to him for help. I always remember that every time at this time, he would come and sit down, and then operate on the computer, and explain the solution to the problem to me.

On the way home from work, we only had a short distance of 600 meters and three junctions, but he always told me to work hard and occasionally answered my questions. Whenever the last intersection parted, he went north, I east, he would wave to me, I said I went back, he replied pay attention to safety. Now the road is still, but after all can only become a memorable youth.

Things are not everything. Today, two and a half years after graduation, I can say that I have realized my dream and become a big data developer, and I often boast that I am the second most qualified person in the current big data group. There are more and more young people in the group, and sometimes I really expect to be the patient person at the beginning, but eventually I fail, because no one is willing to get to the bottom of the matter, no one wants to know the architecture of this big data platform, no one cares about the flow of data, and more people are still interested in laughing and talking.

The friends you meet will eventually be separated, but the knowledge you learn and the habits you develop will stay with you for life.

About me

I grew up inarticulate. It was awkward to talk to strangers. They let me express myself.

In the first month of my big data internship, I did some document writing. Later, I worked in operation and maintenance for half a year, sacrificing a lot of rest time to work overtime. One year later, I finished my internship and became a big data developer. When I was in operations, I fantasized about, but never dreamed of, one day doing what I love in development. I was just learning big data at work, at home, on the bus, as I had just taught myself Java.

When you are still complaining that life determines your fate, think more about whether you first give up the ideal.

In June, 2018, just before graduation, I introduced Scrapy to build a crawler program with millions of levels of data, studied the configuration of thousands of lines written with Flume, and landed the data of 800 billion/day to HDFS.

Then I graduated, then they left the company, and then one day, the leader said you can open your own salary. That concludes my story.

Future may be

At the end of 2020, I had dinner with Brother Gang and brother Chao. I also recalled and lamented the previous days. I raised my glass and drank it all in the wine. I marveled that back then there was 100 billion a day of data, and I was just a participant, and now there’s 1 trillion a day of data, and I’m the leader. At that time, I thought we would work together all the time.

The wind was cold that night, and I was walking home when a 101 passed by, but this time in the opposite direction.


Writing is daily work in personal practice, in their own point of view from 0 to 1, to ensure that we can really understand.

This article will be published in the public account [entry to the road to give up], looking forward to your attention.