Tag: Big data

The back-end

How to Design real-time Data Platform (Design Part)

January 8, 2024

by Mrs. Gemma Holt

No Comments

This article will discuss an important and common big data infrastructure platform, namely "real-time data platform". In the last design, we first introduce real-time data...

Read More

The back-end

Meituan distribution data governance practices

January 8, 2024

by Samantha Cragg

No Comments

With the advent of the era of big data, more and more enterprises see the value of data assets. It has become a consensus in...

Read More

The back-end

MinIO is Strong – 8 reasons WHY I gave up FastDFS and embraced MinIO

January 8, 2024

by 王雅玲

No Comments

At present, there are many network services available for file storage, such as Ali Cloud OSS, Qiuniuyun, Tencent cloud and so on, but the fees...

Read More

The back-end

What is the difference between Big data hadoop/Hive/hbase? What are the application scenarios?

January 8, 2024

by Todd Wright

No Comments

1. Hadoop is a distributed computing + distributed file system. The former is MapReduce and the latter is HDFS. Hive is a data warehouse. The...

Read More

The back-end

Introduction to big data technology

January 8, 2024

by 嶽靜宜

No Comments

On September 30, 2018, Martin Lau, president of Tencent, sent a letter to all employees, officially launching the third major organizational restructuring in the history...

Read More

The back-end

Take enterprise real-time data platform as an example to understand what agile big data is

January 8, 2024

by Jackson Thomas

No Comments

Agile Big Data is to build a series of universal platform tools and a whole set of big data application life-cycle methodology under the guidance...

Read More

The code of life

Hive -Hive high frequency function set

January 8, 2024

by Anthony Thompson

No Comments

Data Preparation Window function row_number: frequency ★★★★rank: frequency ★★★dense_rank: frequency ★★★★rank/dense_rank/row_number Comparison FIR

Read More

The back-end

Spring uses ThreadPoolTaskExecutor to customize thread pools and implement asynchronous calls

January 8, 2024

by Kanav Sen

No Comments

ThreadPoolTaskExecutor uses the factory method of Executors to configure. 1. Define the parameters used by the thread pool in the configuration file and create executor.propert...

Read More

reading

If you also want to do real-time counting positions…

January 8, 2024

by Sophie Ward-Best

No Comments

Data warehouse is a subject-oriented, Integrate, non-volatile and Time Variant data set, which is used to support management decision. Data warehouse is accompanied by the...

Read More

Artificial intelligence (ai)

Introduction to Spark (III) — Classic Spark word statistics

January 8, 2024

by 解琬婷

No Comments

Since we need a text containing a certain number of words to make statistics, we choose the text of GoneWithTheWind, the English original work, to...

Read More

The back-end

Spark Architecture Overview

January 8, 2024

by Nitara Kalla

No Comments

This section describes the Spark ecology. This section describes the basic concepts and common terms of Spark. This section describes the execution principle and architecture...

Read More

The back-end

How to Design real-time Data Platform (Technical Part)

January 8, 2024

by Melissa Miranda

No Comments

Real-time Data Platform (RTDP) is an important and common big Data infrastructure Platform. In the previous part (design), we introduced RTDP from the perspective of...

Read More

The back-end

In this paper, we thoroughly understand the various time complexity, application scenarios and implementation principles of skip tables

January 8, 2024

by Kevin Smith

No Comments

Skip table is a kind of magical data structure, because almost all versions of undergraduate textbooks do not have this kind of data structure, and...

Read More

reading

The DataV private deployment feature

January 8, 2024

by Eric Kelly

No Comments

DataV developers, go into your user center today and you'll find a little surprise waiting for you. That's it -- private deployment is live! There...

Read More

The back-end

MapReduce/HDFS/YARN Interview question 70

January 8, 2024

by Alison McGregor

No Comments

Warm-up question 30 1. Describe the HDFS write process 2. Describe the HDFS read process 3. Explain the HDFS architecture in detail. 4. If a...

Read More

The back-end

Flink Interview Clearance Manual

January 8, 2024

by Jason Larsen

No Comments

The year 2019 is the most extraordinary year in the field of real-time computing of big data. Alibaba Blink (internal Flink branch version) opened source...

Read More

reading

I work in ali as a backoffice developer

January 8, 2024

by Donna Morgan

No Comments

"Online development of the whole process" has been gaining popularity in recent years, such as the practice of AWS in C9, the well-known TheiaJS in...

Read More

The back-end

This section describes the architecture and working principles of HDFS, MapReduce, and Yarn

January 8, 2024

by Bradley Norman

No Comments

In fact, big data technology is an innovative application of distributed technology in the field of data processing. Its essence is to use more computers...

Read More

The back-end

Hadoop and its commands

January 8, 2024

by Carly Phillips

No Comments

A Hadoop cluster contains multiple nodes. HDFS focuses on distributed storage. A machine can not be stored, so there is the concept of cluster. Clusters...

Read More

reading

Hive command: Hive Command: Hive Command: Hive command

January 8, 2024

by Jeffery Chavez

No Comments

Recently, I am reading the "Massive Data Processing and Big Data Technology Combat" written by The Ice River Big Guy. The book covers a number...

Read More

The back-end

Big data combat -Hive- Skills combat

January 8, 2024

by Badal Kashyap

No Comments

SQL distributeby,sortby, distributeby, distributeby,sortby Bucket list clusteredby (sno) sortedby (agedesc) into

Read More

The back-end

Datax3.0 installation and MySQL To MySQL example

January 8, 2024

by Toby Chambers-Elliott

No Comments

Recently, I joined a bank and was in charge of data intelligent platform. I used Hadoop related technology stack. I had taught myself related technology...

Read More

The back-end

Kafka consumer group delay widget

January 8, 2024

by Madison Cairns-Stevenson

No Comments

Kafka consumer group details and message lag small tool, simple installation and configuration, HTTP API request interface, return Kafka related JSON data

Read More

The back-end

Flink1.12 new KafkaSource source code based on the Flip 27 (2) — Reader and Enumerator custom communication

January 8, 2024

by Amanda Harrell

No Comments

In the KafkaSource of Flink1.12, a running Source has 1+ N threads, enumerators run in a separate thread for split topics and partitions, and readers...

Read More

The back-end

Hive Quick Start

January 8, 2024

by 李佩珊

No Comments

I am a student majoring in data science of Renmin University of China. This document was my notes for this course in school at the...

Read More

reading

Big data interview kill recruit — Hive high frequency test point, afraid you will!

January 8, 2024

by Amber Maldonado

No Comments

Last article introduced 3 common SQL pen test questions, the response is still good. Brother bacteria as a result, over the next few days, will...

Read More

The code of life

Flume Learning 09- Event Deserializers

January 7, 2024

by Sean Smith

No Comments

The following event deserializers ship with Flume. This deserializer generates an event for each line of the input file. Property NameDefaultDescriptiondeserializer.maxLineLength2048Maximum number of characters to...

Read More

The back-end

Yarn-high concurrency RPC source code implementation

January 7, 2024

by John Gibb

No Comments

YARN high concurrency RPC source code implementation, the last article described some of the basic knowledge of RPC, this article for YARN to initialize its...

Read More

The back-end

Number of real-time warehouse |, speaking, reading and writing in the form of the upsert for Flink1.12 Kafka data, for example

January 7, 2024

by Brooke Clark

No Comments

In some scenarios, such as the GROUP BY result, it is necessary to update the previous result value. At this point, the key of the...

Read More

Artificial intelligence (ai)

Real-time sample feature backtracking in large-scale machine learning

January 7, 2024

by Emma Booth

No Comments

The author is engaged in the development of content understanding server, mainly processing text, video and audio, producing corresponding features through NLP and CV technologies,...

Read More