This article lowers the threshold of big data learning to the horizon
The idea is to chop up large chunks of data, make three copies of each piece and put them on three cheap machines, so that...
Read More
Tag: hadoop
HDFS – How does the dual caching mechanism guarantee high concurrent requests for metadata
December 14, 2023
by Jorge Coffey
No Comments
As we mentioned in HDFS - What is Metadata, metadata is stored in memory and disk. Memory is used to improve response time, disk is...
Read More
Data handling components: manage data import and export based on SQOOP
December 14, 2023
by Patricia Parsons
No Comments
SQOOP is an open source big data component that is used to transfer data between Hadoop(Hive, HBase, etc.) and traditional databases (MySQL, PostgreSQL, Oracle, etc.).
Read More
Cloud small classes | MRS introductory HDFS components is introduced
December 14, 2023
by Dale Cooke-Bishop
No Comments
Abstract: HDFS is the basic File System in MapReduce service, which is called Hadoop Distributed File System. It can support the realization of large-scale data...
Read More
How to improve the efficiency of Hadoop accessing Object Storage US3? We did these technical practices
December 14, 2023
by Jasmine Simmons
No Comments
In the era of big data with information explosion, how to solve the storage problem of massive data with lower cost has become an important...
Read More
HDFS – What is the security mode
December 14, 2023
by 梁筱涵
No Comments
High availability of HDFS-NameNode mentions that NameNode has active and standby states, and it also has another state, which is SafeMode, which is safemode. In...
Read More
CDH off-line setup
December 14, 2023
by Samantha Cragg
No Comments
CDH Offline Construction Actually, we have been engaged in CDH for a long time since 3 years ago. At that time, we also planned to...
Read More
Ambari environment setup (offline setup)
December 14, 2023
by Megan Miles
No Comments
Ambari is a top-tier open source project from the Apache Software Foundation. It is a tool for centrally deploying, managing, and monitoring Hadoop distributed clusters....
Read More
Interview Questions for Big Data Engineer (1)
December 14, 2023
by Kathy Owens
No Comments
Optimized compression of shuffle in Hive reduces the amount of data stored on disk and improves query speed by reducing I/O. Enable compression for a...
Read More
Kerberos-based Hadoop security cluster construction
December 14, 2023
by Jessica Elliott
No Comments
In my last job, I was mainly responsible for the construction of big data platform, during which I accumulated some notes on the construction and...
Read More
The hadoop tutorial – graphs
December 14, 2023
by Rhys Bruce-Griffin
No Comments
MapReduce is a computing model adopted by Hadoop for multi-node computing, and to put it more simply, it is a set of methodology for Hadoop...
Read More
IDEA runs the WordCount program
December 14, 2023
by Madison Johnson
No Comments
Submit tasks to a Hadoop pseudo-distributed cluster in a virtual machine using IDEA. Run WordCount V1.0, the official example of MapReduce. Windows 10IDEA 2020.2.2CentOS 7.6Hadoop...
Read More
Ambari2.6.0+HDP2.6.3 deploys the Hadoop cluster
December 14, 2023
by Gabriel Washington
No Comments
1. Modify hostname(all nodes) to take effect temporarily {code... } in perpetuity {code... } 2. Configure SSH free secret (all nodes) to generate a secret...
Read More
Hologres Reveals: Deep parsing and efficient distributed query engine
December 14, 2023
by Samarth Sant
No Comments
Hologres (Chinese name) interactive analysis is ali cloud from the research on the number of one-stop real-time warehouse, the cloud native system combines real-time service...
Read More
The Hadoop Framework: MapReduce Rationals and Primer Cases
December 13, 2023
by 李淑娟
No Comments
One of the core components of Hadoop: MapReduce, a distributed computing scheme, is a programming model for parallel operations of large-scale data sets, including Map...
Read More
HDFS distributed file system
December 13, 2023
by Sarah Carter
No Comments
The Client provides commands for managing and accessing the hdfsnameNode, which is a master. The Client provides commands for managing and accessing the hdfsnameNode, which...
Read More
On Windows installation guide | avoid spark encountered some pit pit
December 13, 2023
by Jose Martinez
No Comments
You keep getting errors and basic logic doesn't work. After a long time of debugging, I concluded that every time IDEA did not clearly indicate...
Read More
EMR cluster security authentication and authorization management
December 13, 2023
by Neelofar Roy
No Comments
AliCloud EMR service Kafka and Hadoop security cluster use Kerberos for user security authentication, and access authorization management through Apacheranger service. Using ApacheWeb server logs...
Read More
High availability of HDFS-NameNode
December 13, 2023
by Aniruddh Sule
No Comments
We already know that when a file is read or uploaded, it needs to go through the NameNode. If the NameNode is down, the client...
Read More
Apache Kylin installation and deployment
December 13, 2023
by Nancy Hall
No Comments
Special binary package is a Kylin snapshot binary package compiled on the HBase 1.1+ environment. It requires HBase version 1.1.3 or higher to install it,...
Read More
Hadoop MapReduce Job Stuck for a Long Time
December 13, 2023
by Lori Lopez
No Comments
At present, we preprocess source data files from upstream by writing Hadoop MapReduce program. After the source data file is sent to the Hadoop cluster,...
Read More
How does HDFS-NameNode perceive the DataNode
December 13, 2023
by Sheila Davis
No Comments
In HDFS-HDFS, when the NameNode is started, it will open the RPC service, called ServicerPCServer. There are many protocols in this Server, one of which...
Read More
HDFS – The four roles of HDFS
December 13, 2023
by Rati Bajaj
No Comments
In the last article we have known that HDFS is to cut the file into blocks, and then stored in a number of servers, and...
Read More
Hive Modeling Analysis
December 13, 2023
by Brooke Miller
No Comments
The modeling analysis is based on the case of the visitor system to discuss the modeling analysis. The purpose of the modeling analysis is to...
Read More
Hadoop series
December 13, 2023
by Miss Hilary Rogers
No Comments
Hadoop - Cluster Install Hadoop - Cluster Install (High Availability) HDFSHDFS - More and More Files How to Do HDFS-Four Roles of HDFS-NameNode High Availability...
Read More
Install the Hadoop cluster using Docker and access HDFS through Java API
December 13, 2023
by 張惠婷
No Comments
Recently, I happened to learn Hadoop, and encountered some minor problems in the installation process. I just recorded and shared this process, hoping to help...
Read More
Python Data Mining and Machine Learning — Communication Credit Risk Assessment (3) — Feature Engineering
December 13, 2023
by 金淑貞
No Comments
There is a common saying in the industry that data and features determine the upper limit of machine learning, and models and algorithms only approximate...
Read More
Make the machine understand the user — the user portrait in big data
December 13, 2023
by 張怡安
No Comments
Welcome to visit netease Cloud Community to learn more about Netease's technical product operation experience. The concept of persona was first proposed by Alan Cooper,...
Read More
Big data architecture r&d engineer of station B
December 13, 2023
by Juan Barber Jr.
No Comments
Main responsibilities: Responsible for the architecture design, development and maintenance of the company's data platform; continuously optimize the big data technology architecture; responsible for the...
Read More
Actual combat: Ha-Hadoop cluster construction based on Docker
December 12, 2023
by James Ball
No Comments
Hadoop master and slave run in different Docker containers. NameNode and ResourceManager run in the Hadoop-master container, and D runs in the Hadoop-slave container
Read More