Bytedance EB HDFS practice
Since HDFS is a system that has existed for a very long time and its application scenarios are very mature, we will briefly introduce this...
Read More
Tag: hdfs
Introduction to HDFS (1)
December 19, 2023
by Yasmin Tiwari
No Comments
HDFS is a file system used to store and manage files through a unified namespace (similar to the directory tree of a local file system)....
Read More
Meitu Offline ETL practice
December 19, 2023
by Samantha Cragg
No Comments
Logs collected by Meitu need to be cleaned and organized by the ETL program and continuously landed in the HDFS or Hive to facilitate subsequent...
Read More
【 Fine 】 Thoroughly familiar with Hadoop RPC framework
December 19, 2023
by Elizabeth Crawford
No Comments
This paper introduces Hadoop RPC framework in detail through flow chart + text. Starting with DFSClient, this article traces the propagation path of an RPC...
Read More
HDFS read and write process
December 19, 2023
by Eva Bansal
No Comments
HDFS read and write process is enough for you to read and write this article, absolutely original pictures and details of HDFS read and write...
Read More
Meitu Distributed Bitmap practice: Naix
December 19, 2023
by Ashley Johnson
No Comments
Big data technologies and application systems have been playing a huge role in various industries, and a variety of open source technologies have brought great...
Read More
Meitu Big Data Platform architecture practice
December 19, 2023
by Dana Mullen
No Comments
This article is the 11th session of Meitu Internet Technology Salon shared by guests. The public account replied to "Meitu Big Data Platform" to get...
Read More
HDFS upload read file principle – this article is enough!!
December 17, 2023
by 楊雅涵
No Comments
Upload file 1: first, there is a file, called 1.txt, let's say this file is 300MB. 2: If the client wants to upload the 1.txt...
Read More
Kudu + Impala introduce scientific team blog | micro store data
December 15, 2023
by Dr. Andrew Palmer
No Comments
Kudu + Impala introduce scientific team blog | micro store data
Read More
Flink’s practice in VIPshop
December 15, 2023
by Mr. Denis Woodward
No Comments
At present, the real-time platform of VIPshop is not a unified computing framework, but three major computing frameworks including Storm, Spark and Flink. For historical...
Read More
HDFS3. X new feature – erasure code
December 14, 2023
by Tanya Krishnamurthy
No Comments
EC (erasure code) is a coding technique that, prior to HDFS, was most widely used in inexpensive redundant array of disks (RAID) (RAID Introduction: RAID...
Read More
【Hadoop Learning 】HDFS basic principles
December 14, 2023
by Casey Bowater-Parkes
No Comments
HDFS Introduction HDSF {code... } features {code... } System structure Master and Slave structure. There are three roles: NameNode, SecondarynaNode and DataNode. NameNode {code... }...
Read More
Hadoop Tutorial – HDFS Client Development
December 14, 2023
by Samantha Rice
No Comments
The Hadoop environment Hadoop version is {code... } Client development introduces dependencies (using Maven) {code... } write code {code... If you want to run it...
Read More
Hadoop-5-HDFS for big data
December 14, 2023
by 張婷婷
No Comments
HDFS (Hadoop Distributed File System) is the core sub-project of Hadoop project. In the development of big data, massive data are stored and managed through...
Read More
Technology dry! HDFS reading and writing principle and code simple implementation
December 14, 2023
by Renee Rose
No Comments
Users can manipulate files and directories in HDFS through terminal commands just as they would files on a local file system, such as Linux. Users...
Read More
HDFS path traceback
December 14, 2023
by 何宗翰
No Comments
SegmentFault is a leading developer technology community in China. We take technical questions and answers, technical column, technical courses, technical information as the core product...
Read More
Use SQOOP to import massive MySQL test data into HDFS and HBase
December 14, 2023
by Gerard Sharp
No Comments
Disclaimer: original by the author, reproduced with credit. SQOOP/SQOOP/SQOOP/SQOOP/SQOOP/SQOOP/SQOOP/SQOOP/SQOOP/SQOOP /etc/profile.vim = /etc/profile.vim If SQOOP version information is displayed normally, the installation and configuration is successful:...
Read More
HDFS – What is Metadata
December 14, 2023
by Dana Mullen
No Comments
When a file is uploaded to an HDFS server, it is divided into multiple blocks and stored as multiple copies on the server. How do...
Read More
HDFS – What about more and more files
December 14, 2023
by Hrishita Choudhary
No Comments
With the development of the business, the server will store more and more files, in the initial stage, we will directly through the expansion to...
Read More
SQOOP incremental import, data duplication problem
December 14, 2023
by Shelley Chapman-Williams
No Comments
Original link: [Link] Duplicates when importing data according to auto-increment ID, Incremental importsSQOOP provides an Incremental import mode which can be used to. Incremental importsSQOOP...
Read More
How is the HDFS-Checkpoint mechanism implemented
December 14, 2023
by Sarah Glenn
No Comments
We mentioned the CheckPoint mechanism, which basically merges multiple Edits files. NameNode is already under a lot of pressure, so it is not the NameNode...
Read More
The HDFS command thing
December 14, 2023
by 李詩婷
No Comments
Background Time: Feb 26, 2021 13:58:26 Brief Step: Determine what the command is? What is the location of the command? What is the main class...
Read More
HDFS – the stuff that writes data
December 14, 2023
by Bethan Connor
No Comments
As mentioned earlier, when a client needs to write data, it will tell the NameNode that it is uploading the file to a directory, and...
Read More
Cloud big data distributed file system cost optimization tips
December 14, 2023
by Taimur Chandra
No Comments
CHDFS (Cloud HDFS) is a high-performance distributed file system of Tencent Cloud, which provides standard HDFS access protocol and hierarchical namespace. It mainly solves the...
Read More
Evolution of the HDFS architecture
December 14, 2023
by Angela Allen
No Comments
Hadoop has been developed for more than 10 years, and the versions have gone through numerous updates and iterations. At present, the major versions of...
Read More
This article lowers the threshold of big data learning to the horizon
December 14, 2023
by Graham Curtis
No Comments
The idea is to chop up large chunks of data, make three copies of each piece and put them on three cheap machines, so that...
Read More
HDFS – How does the dual caching mechanism guarantee high concurrent requests for metadata
December 14, 2023
by Jorge Coffey
No Comments
As we mentioned in HDFS - What is Metadata, metadata is stored in memory and disk. Memory is used to improve response time, disk is...
Read More
Cloud small classes | MRS introductory HDFS components is introduced
December 14, 2023
by Dale Cooke-Bishop
No Comments
Abstract: HDFS is the basic File System in MapReduce service, which is called Hadoop Distributed File System. It can support the realization of large-scale data...
Read More
HDFS – What is the security mode
December 14, 2023
by 梁筱涵
No Comments
High availability of HDFS-NameNode mentions that NameNode has active and standby states, and it also has another state, which is SafeMode, which is safemode. In...
Read More
HDFS distributed file system
December 13, 2023
by Sarah Carter
No Comments
The Client provides commands for managing and accessing the hdfsnameNode, which is a master. The Client provides commands for managing and accessing the hdfsnameNode, which...
Read More