What is Kafka?
Kafka is commonly used for real-time streaming data architectures to provide real-time analysis.
Kafka is exploding. More than a third of Fortune 500 companies use Kafka. These companies include the top 10 travel companies, seven of the top 10 banks, eight of the top 10 insurance companies, nine of the top 10 telecommunications companies, and more. LinkedIn, Microsoft, and Netflix use Kafka (1,000,000,000,000) to process trillions of messages every day. Kafka is used for real-time data streaming, collecting big data or performing real-time analysis (or both). Kafka is used with in-memory microservices to provide reliability and can be used to provide events to both CEP (Complex Event Flow System) and IoT/IFTTT-style automated systems.
Why Kafka? Kafka is commonly used for real-time streaming data architectures to provide real-time analysis. Because Kafka is a fast, extensible, durable, and fault-tolerant publish and subscribe messaging system, Kafka is used for JMS, where RabbitMQ and AMQP may not be considered due to volume and response speed. Kafka has higher throughput, reliability, and replication features, making it suitable for tracking service calls (tracking every call) or tracking IoT sensor data that traditional MOM might not consider.
Who is using Kafka? Many large companies that deal with large amounts of data use Kafka.
LinkedIn originated there and used it to track activity data and operational metrics.
Twitter uses it as part of Storm to provide streaming infrastructure.
Square uses Kafka as a message bus to transfer all system events to various Square data centers (logs, custom events, metrics, etc.) for output to Splunk, Graphite (dashboards), and Esper-like/CEP alert systems.
It’s also used by Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box, Cisco, CloudFlare, and Netflix.
Since the whole note is quite comprehensive and contains a lot of content, only some screenshots of the materials are shown here. If you need to get the “Kafka Technical Manual” mentioned in the article, there is a direct access address at the end of the text.
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Kafka knowledge map
Part 1: Getting to know Kafka In this part, you can learn the following:
Know the basic principles of Kafka and understand the key terms and concepts
You can use Kafka for messaging system development
Use the Java language to send and receive messages using Kafka
Contents cover :(features + usage scenarios + technical advantages + suitable groups + detailed concept + installation and configuration + the first Java program + configuration of common server parameters + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
The second part: producer detail
By studying this part, you can learn the content:
Learn the general process of Kafka data production in depth
How do I create and use a Kafka producer
Kafka producers often use configurations
Contents covered :(message sending + sending principle analysis + other producer parameters + summary)
Super detailed: the full network of the first Kafka technical manual, from the foundation to the actual combat is complete Part 3: Consumer details through this part of the study, you can learn the content:
Learn the general process of Kafka data consumption in depth
How do I create and use a Kafka consumer
Kafka consumer common configuration
Contents covered :(concept introduction + message reception + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part IV: Theme
By studying this part, you can learn the content:
Learn more about Kafka topics in management
KafkaAdminClient application
Contents covered :(admin + add partition + partition copy allocation + other topic parameter configuration + KafkaAdminClient application + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part V: Partitioning
By studying this part, you can learn the content:
Learn more about the management of Kafka partitions
These include: election of priority copies, partition reallocation, etc
Contents covered :(replica mechanism + partition Leader election + partition reallocation + modify replica factor + partition allocation strategy + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part 6: Kafka storage
By studying this part, you can learn the content:
After completing the development of Kafka application, I know the file storage mechanism
Why does Kafka use disks for storage
Fast retrieval of messages
Contents covered :(Overview of storage structure + log index + log cleanup + disk storage advantages + summary)
! [Uppass…] (a)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part 7: Stability
By studying this part, you can learn the content:
Learn more about Kafka’s various mechanisms to ensure high availability while ensuring high performance and throughput
Contents covered :(idempotency + transaction + controller + reliability assurance + consistency assurance + message repetition scenarios and solutions + __consumer_offsets + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part 8: Advanced Applications
By studying this part, you can learn the content:
Master command line tools as operations personnel
Use Connect for stream information processing
Master latency message, streaming, etc
Kafka and SpringBoot integration
Contents covered :(Command Line Tools + Connect + Streaming Spark + SpringBoot Kafka + Messaging Middleware Selection Comparison + Summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part 9: Cluster management
By studying this part, you can learn the content:
Familiar with Kafka cluster management
Configure and tune
Contents cover :(the characteristics of the cluster + the capacity of the cluster + cluster usage scenarios + cluster construction + multi-cluster synchronization + summary)
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Part X: Monitoring
By studying this part, you can learn the content:
Know Kafka’s surveillance system
Master JMX monitoring indicators
Real-time reminder of data changes
Monitoring metrics + Broker metrics + Subject partition metrics + Producer metrics + Consumer metrics + Kafka Eagle + Summary
Super detailed: this network first Kafka technical manual, from the foundation to the actual combat should be complete
Write in the last
In the current trend, Kafka is valued by more and more Internet unicorns and is increasingly used in their business! Well, we mean not necessarily skillful, but at least learn how to use it. In this way, I will continue to improve my ability in the business, so that I can have the confidence to talk with my boss and HR about salary increase. Otherwise, just blindly do, keep 996, business ability did not improve, then will be abandoned by the company, not to mention what raise salary, you ask yourself, do you deserve?
Finally, what I want to say is that learning is not difficult, but you should insist, especially after we participate in the work, to continue to insist on learning is not easy. But for programmers, learning is the foundation of career, if give up learning, the market will be eliminated sooner or later. Therefore, learning more and more new knowledge is a greater investment in yourself.
Access method: need this “Kafka technical manual” friends please thumb up, forward, after the collection click here to get the full download path way!