Mapreduce MapTask Hadoop mapReduce MapTask
This article focuses on the workflow of MapTask, including reading data through split listings, how to write to buffers, and how to overwrite.
Read More
Tag: hadoop
Hadoop delete file process – source code
December 21, 2023
by Edward Oliver
No Comments
Hadoop fs -rm -r hadoop fs -rm -r hadoop fs -rm -r hadoop fs -rm -r hadoop fs -rm -r hadoop fs -rm -r
Read More
Hadoop Learning Notes – 01 Big Data Enlightenment
December 21, 2023
by Hamish Richards
No Comments
Write ahead: Before learning big data, you must first have the idea of divide and conquer. And before you can understand distribution, you need to...
Read More
MapReduce hadoop source Code Analysis
December 20, 2023
by Joshua Collins
No Comments
😄 $\color{red}{~}$Hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop See and forget, so I decide
Read More
Map the MapReduce workflow in detail
December 20, 2023
by Anahita Varma
No Comments
This is probably the most detailed version of the MapReduce workflow on the entire network, and it's easy to handle situations like interviews
Read More
MapReduce on YARN
December 20, 2023
by Rhea Sachar
No Comments
What is the MapReduce workflow on YARN clusters? This article provides a variety of versions of the process detailed, absolutely full of dry goods
Read More
How do I solve the MapReduce data skewness problem?
December 20, 2023
by Charlotte Fulton
No Comments
I hate data skew. How do I solve MapReduce data skew? Check it out. Here's the complete solution
Read More
Hello Spark! | Spark, from entry to the master
December 19, 2023
by Mishti Sibal
No Comments
Spark is an open source Hadoop MapReduce-like general parallel framework developed by UC Berkeley AMP Lab. It is a fast and universal big data processing...
Read More
Functions of each component of the Hadoop ecosystem
December 19, 2023
by Katie Kelly
No Comments
Hadoop Distributed File System HDFS is an open source implementation of Google Distributed File System (GFS). It is one of the two core components of...
Read More
Hadoop‖ Over one trillion NameNode performance faults troubleshooting process
December 19, 2023
by Annette Holland
No Comments
With the development of the Domestic Internet industry, although the trillion-scale super cluster is not as rare as a few years ago, it is also...
Read More
Meitu Big Data Platform architecture practice
December 19, 2023
by Dana Mullen
No Comments
This article is the 11th session of Meitu Internet Technology Salon shared by guests. The public account replied to "Meitu Big Data Platform" to get...
Read More
Hadoop-hdfs
December 19, 2023
by Georgina Martin-Smart
No Comments
Why develop an HDFS file system in a Hadoop project? Better support for distributed computing. Storage model Architecture Design Role Function Metadata persistence security mode...
Read More
Uploading hadoop Files
December 19, 2023
by Victoria Johnson
No Comments
The HDSF is essentially a distributed file system, but it scales out relatively easily compared to ordinary computers and comes with its own high availability...
Read More
Hadoop-3.2.2 Hive-3.1.2 hbase-2.3.5 Installation
December 19, 2023
by Luis Bishop
No Comments
1.1 Campus NETWORK NAT Configuring static IP Address Note that: The hotspot cannot be enabled during NIC sharing. You can cancel NIC sharing when the...
Read More
Hadoop Installation Manual
December 19, 2023
by 蔡馨儀
No Comments
1 Installation environment Kernel: 3.10.0-229.el7. X86_64 OPERATING system: CentOS 7 SSH and SSHD are configured by default.
Read More
Will Hadoop be replaced by Spark? This section describes Hadoop ecological components
December 19, 2023
by Dr. Nathan Davis
No Comments
Both Hadoop and Spark are mainstream big data frameworks. However, with the advantages of Spark in terms of speed and ease of use, some experts...
Read More
Does Snowflake’s rise mean Hadoop is dead? What is the big data system?
December 18, 2023
by Emma Freeth-Stewart
No Comments
Guan Tao is a senior expert in big data systems, having spent the last 15 years of his 20-year big data development at Microsoft (Internet...
Read More
To install and configure Hadoop | August more challenges
December 17, 2023
by 徐家瑜
No Comments
Hadoop is an open source big data framework written in Java that provides reliable, scalable distributed computing and is now hosted by Apache. The Hadoop's...
Read More
Build e-commerce search platform with Elasticsearch
December 17, 2023
by Mr. Sean Lane
No Comments
A medium-sized e-commerce platform generates millions of pieces of raw data and hundreds of millions of user behavior data every day. Generally speaking, there are...
Read More
Hadoop Series (vii) — HDFS Java API
December 16, 2023
by Karen Roman
No Comments
To use the HDFS API, import the hadoop-client dependency. For the CDH version of Hadoop, you also need to specify the repository address: FileSystem is...
Read More
Differences between Hadoop MapReduce and Spark
December 16, 2023
by Trevor Clark
No Comments
1. When it comes to big data processing, I believe that many people's first thought is Hadoop MapReduce. Yes, Hadoop MapReduce laid the groundwork for...
Read More
Hadoop Cluster Installation (Note)
December 16, 2023
by Kay Waters
No Comments
8.k8s-master: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Read More
Hadoop Learning 03 — High Availability HA setup
December 16, 2023
by Phillip Carter
No Comments
Single point of failure: There is only one Namenode single point of bottleneck: one Namenod
Read More
Document some hadoop-Mapreduc related issues encountered
December 16, 2023
by Kay Waters
No Comments
1. Inconsistent use of class types (Mapper and Reduce) This is because the inheritance classes of Mapper and Reduce use inconsistent T parameters. For example,...
Read More
Why is the HDFS block size 128MB? block.size
December 16, 2023
by Aaina Agrawal
No Comments
Blocksize depends on dfs.blocksize, 2.x defaults to 128M and 1.x to 64M (128M refers to the maximum size of the block! Each block can store...
Read More
Bytedance 100,000-node HDFS cluster multi-machine room architecture evolution
December 16, 2023
by Julie Moran
No Comments
With the rapid development of bytedance's services, the HDFS service scale has reached the "double 10" level: 100,000 nodes in a single cluster and 10EB...
Read More
How to tune Hadoop? It’s enough to read this one
December 16, 2023
by 金瑋婷
No Comments
How do you tune a Hadoop cluster? Comprehensive and detailed tuning scheme (including Linux, HDFS, MapReduce, YARN)
Read More
SecondaryNameNode and NameNode have to tell the story
December 15, 2023
by Kimberly Houston
No Comments
Mechanism of SecondaryNameNode 1) Stage 1: Starting the Namenode 1: Starting the Namenode for the first time, formatting the Namenode, creating fsimage and Ed
Read More
Hadoop study notes – 07MapReduce ReduceTask source parsing
December 15, 2023
by Alisha Chaudhari
No Comments
This paper briefly introduces the running steps of ReduceTask, and focuses on how ReduceTask uses iterator mode to read data, avoiding the problem of OOM...
Read More