“Prayer with earnest, hope with joy, suffering with perseverance, will yield.”

preface

Was not updated for some time, blogs and articles, it is mainly a few months in learning, machine learning knowledge of this part of the mathematical requirement is a little high, so eating more difficult, a lot of time in mathematics and technology ability is limited, there is no way to this part of knowledge output as the article notes, explain that he haven’t digest.

Unfortunately, in the first half of the epidemic, I did not grasp the rare time to study at home. Here is a summary of this year’s study and work

Summary in 2020

Study in 2020

  • Learn Spark source code
  • The Authoritative Guide to Spark
  • Kafka In action (Geek Time)
  • Kafka source code (Geek Time)
  • Introduction to Python
  • Machine learning
  • Data Structures and Algorithms (Geek Time)
  • Deep learning
  • Ali Tianchi match
  • Flink
  • B stand live

Spark

The spark source code is mainly about the source code of the three modules, and the basic operation logic of Spark is deeply understood

  • Spark Learn RPC module source code in depth
  • Spark Learn the source code of storage modules in depth
  • Spark learn the source code of the computing module in depth

The Authoritative Guide to Spark is a good primer for beginners to read and understand, and is explained using the latest version of Spark2.

Kafka

On Kafka, both geek Time classes are pretty good. I recommend anyone who uses Kafka to check it out. The notes for this column are 80% complete and will be posted in nuggets when they are finished. Kafka source code to be honest a little difficult, look is also relatively slow, back and forth to read.

python

Python learning is mainly for the use of machine learning, because I have Java background, it is quite fast to learn, a little understanding of the syntax, not too deep.

Machine learning & Deep learning

This is almost the highlight of the second half of the year, from mathematics to basic machine learning algorithms, to deep learning neural networks, and recommendation systems related algorithms. Can say is one step at a time, gnaw very hard.

  • ML
    • LightGBM
    • Collaborative filtering
    • Lagrange multiplier method
    • Kernel function (SVM, PCA, LDA)
    • PCA and LDA dimension reduction
    • Xgboost/GBDT/decision tree
    • KNN
    • K-means
    • LR
    • Random forests
    • FM sorting
    • SVM
  • DL
    • word2vec/transformer/bert/embedding
    • wide&deep
    • FFM/DeepFM
    • dnn/cnn/rnn

Ali tianchi

The project business of the company has a little bit of data mining need, but not for machine learning. So AFTER I learned the theory and code of machine learning, I found a novice on Ali Pool to practice, from the beginning of linear regression, to the baseline of XGBoost, to the parameter tuning of XGboost, to the neural network, to the neural network after feature engineering tuning. From clueless copy east copy west copy, to data processing, feature engineering processing has a certain understanding. It was quite a sense of achievement to see my grades improved bit by bit and finally made it into the top 100.

As of this writing, it’s 105th

Flink

Flink belongs to one of the most popular framework of big data, because the work is using Spark, so another stream processing Flink framework is also quite interested in, the learning video is on the B website is still silicon Valley 2021 latest Java version of Flink, this teaching video after watching belongs to use, but the underlying source logic or lack of.

As a big data developer, not only spark is enough, but flink is also good to know. It is good for personal promotion and job-hopping interview.

Data structures and algorithms

This is geek time on the “data structure and algorithm” column, or very good, follow to learn a more systematic study of data structure and algorithm, and learn a time is not enough, it is best to follow the class examples, after class thinking questions also write their own code again. I have taken some notes on and off, and the rest is in Evernote, which has not been sorted into blog notes.

  • Data Structures and Algorithms
  • Data Structure and Algorithm
  • Data Structure and Algorithm
  • Data Structure and Algorithm

It is a pity that LeetCode did not stick to the daily question, and only did dozens of questions intermittently.

B stand live

At the end of the year, I started to work as a humble little anchor in the learning area on B website, mainly broadcasting my own learning pictures, because I found that this live learning can supervise my better learning and reduce my cheating. After all, the temptation of learning with a computer is still too much, you can see B station, brush variety, galloping call teacher canyon. If I live stream my study, I will be able to see my desktop display, so IT will be difficult to touch fish, and I will concentrate a lot on my learning efficiency.

Live time in December

Compare that to the 2019 target

Look back at 2019’s goals for 2020

  • Python’s learning
  • Swipe through LeetCode’s Easy and Medium
  • Study design patterns
  • Learn spark source code
  • Have a more comprehensive understanding of the company’s projects and try to understand the business and technology stack of each module
  • Principle of computer composition + operating system + computer network
  • More extensive and more sophisticated, not limited to Docker, Netty, Socket, Linux shell, Kafka, ZK, ELK, database, nginx, etc

Let’s just say it wasn’t as good as expected. Personal reflection: During the first half of the year of the epidemic, I was too comfortable working at home. I had too much fishing, and I didn’t take the time to study. As a result, many original goals were not achieved. 2021 come on!

The 2021 target

As a matter of fact, the company’s projects are relatively stable at present, so many new demands are mostly business additions, deletations, changes and checks, which are not very helpful to my personal technical level growth. I feel that the ceiling of this company is almost touched. In 2021, ON the premise of keeping learning, I will take a look at the outside opportunities to see if there are any suitable ones. The most important thing is to improve my position. And want to try to transfer the algorithm, mainly recommended algorithm. If a small partner has a suitable post can be pushed, you can comment DD about me!

Here are your goals for 2021:

  • BI data warehouse learning
  • Zk source code, Redis source code learning
  • An Ali Tianchi match every 2-3 months
  • Machine learning algorithms are reproduced in Python
  • Finish brushing LeetCode’s Easy and Medium!
  • Can get a satisfactory offer

Denver annual essay | 2020 technical way with me The campaign is under way…