Kylin on Parquet introduction and quick start
Reprinted from the Apache Kylin official account (also by me) : link to the original article. This article is a summary of the introduction by...
Read More
Tag: spark
Kylin 4.0 TOPN implementation principle introduction
December 14, 2023
by Leanne Cooksley
No Comments
Apache Kylin is an open source distributed analysis engine that provides SQL query interfaces on top of Hadoop and multi-dimensional analysis (OLAP) capabilities to support...
Read More
Netease Sails, Cloud Music, Intel, Youzan’s latest big data practice (PPT download + video playback)
December 14, 2023
by Grace Allen
No Comments
At the recent netease Spark Technology Salon hosted by netease Spark and Intel, Yao Qin, a netease Spark Committer, and Chen Qi, head of OLAP...
Read More
Hologres Reveals the Core Principles of High Performance Native Acceleration MaxCompute
December 14, 2023
by Alex Hussain
No Comments
Hologres (Chinese name) interactive analysis is ali cloud from the research on the number of one-stop real-time warehouse, the cloud native system combines real-time service...
Read More
Apache Kylin 4.0 precise de-duplication of the global dictionary principle
December 14, 2023
by Tiya Wali
No Comments
With large data sets, accurate de-duplication and fast query response can be challenging. We know that the most commonly used processing method for accurate reduplication...
Read More
How Carbondata can help Apache Spark in four ways
December 14, 2023
by Christopher Marsh
No Comments
Spark is undoubtedly a powerful processing engine and a distributed clustered computing framework for faster processing. Unfortunately, Spark also falls short in several areas. If...
Read More
The fifth project in the interview series involves the technology Spark
December 14, 2023
by 劉懿
No Comments
2) Standalone: build a resource scheduling cluster based on Mster+Slaves, and Spark tasks are submitted to Master for operation. Spark is a scheduling system of...
Read More
SPARKSQL performs UPDATE operations to modify MySQL data
December 14, 2023
by Chelsea Willis
No Comments
This is the way you don't need to change the source code. } garbled code '?? 'of the situation, there is the big guy who...
Read More
Interview Questions for Big Data Engineer (1)
December 14, 2023
by Kathy Owens
No Comments
Optimized compression of shuffle in Hive reduces the amount of data stored on disk and improves query speed by reducing I/O. Enable compression for a...
Read More
Spark – Distributed high availability cluster installation
December 14, 2023
by Shlok Chandra
No Comments
Spark Release Archives is available on the download page of the official website. I chose Spark-2.4.5, but I chose version 2.7.7 for Hadoop. So the...
Read More
Alibaba big data cloud original biochemical practice, EMR Spark on ACK product introduction
December 13, 2023
by Anna Pedersen
No Comments
Open Source Big Data Community & AMP; Theme: EMR Spark on ACK Product Demo and Best Practices Trainer: Shi Lei, Technical Expert of Aliyun EMR...
Read More
Spark – Use of the Spark Shell
December 13, 2023
by Anya Ratti
No Comments
$SPARK_HOME/bin/spark-shell = $SPARK_HOME/bin/spark-shell = $SPARK_HOME/bin/spark-shell = $SPARK_HOME/bin/spark-shell = $SPARK_HOME/bin/spark-shell = $SPARK_HOME/bin/spark-shell Once in, you can see that SC and Spark have been initialized.
Read More
On Windows installation guide | avoid spark encountered some pit pit
December 13, 2023
by Jose Martinez
No Comments
You keep getting errors and basic logic doesn't work. After a long time of debugging, I concluded that every time IDEA did not clearly indicate...
Read More
How to analyze SAP Spartacus routing problem CheckOutAuthGuard single-step debugging
December 13, 2023
by Miss Hilary Rogers
No Comments
The standard Storefront is used which is generated by Spartacus Schematics.
Read More
Spark – Spark Submit used
December 13, 2023
by 朱惠婷
No Comments
Spark Submit is used to start the applications in the cluster, and it runs the same commands as the Spark Shell. {code... } --class: the...
Read More
Top project committers and contributors are gathered together, and the big data technology salon of Shufan Xintel is waiting for you
December 13, 2023
by Lauren Johnson
No Comments
Under the background of digitalization and intelligent transformation, data, as the core means of production of enterprises, is expected to play a greater value. From...
Read More
SparkSQL queries Iceberg have a lot of small task problems
December 13, 2023
by Leon Gallagher
No Comments
In the test environment, using Spark SQL3.1.1 to query the Iceberg tables stored on Hive Metastore and OSS, many tasks with very small data volumes...
Read More
SparkStreaming project real combat, real-time calculation of PV and UV (hard liver)
December 13, 2023
by Maia Stone
No Comments
Recently there is a requirement, real-time statistics pv, UV, results according to date,hour, PV, UV display, statistics by day, re-statistics the next day, of course,...
Read More
2018 Latest big data learning route from entry to mastery
December 12, 2023
by Michelle Ramos
No Comments
Recently, many people ask xiaobian how they learn big data so much. Many beginners in the initiation of the direction of big data development ideas,...
Read More
Kaiyuan Big Data Weekly – Issue 6
December 12, 2023
by James Knight-Boon
No Comments
Big data is like panning for gold in a pile of sand. When you use data 10,000 times larger than today, a jump from quantitative...
Read More
Flink Table/SQL API planning — Dynamic Table
December 12, 2023
by Lagan Datta
No Comments
The concept of dynamic table was put forward by the community for a long time, but not all of them have been realized. All the...
Read More
Spark Large-scale project (7): User access Session analysis (7) — Database connection pool principle
December 12, 2023
by Umang Dar
No Comments
This time we take the way of technology evolution to talk about the emergence of database connection pool technology and its principle, as well as...
Read More
The difference between Spark terms 01-application, Job, Stage, and Task
December 12, 2023
by Benjamin Stevens
No Comments
SparkContext This article uses spark source code version 2.3.4. SparkContext Note Let's look at a comment of spark source code. When entering SparkContext, you can...
Read More
Spark source Code Parsing 04-Submit process and SparkContext preparation process
December 11, 2023
by 黃婷婷
No Comments
We already know Spark's important role in starting drivers, namely DriverWrapper. Let's go back to cluster
Read More
A set of advanced big data development interview questions (brush up!!)
December 11, 2023
by Dale Cooke-Bishop
No Comments
There are a thousand Hamlets in the eyes of a thousand readers, and a thousand big data programmers in the minds of a thousand big...
Read More
Big Data Processing with Apache Spark – Part 5: Spark Machine Learning Data Pipelining
December 11, 2023
by 林宜君
No Comments
In this article, our other Spark machine learning API, called Spark ML, is the recommended solution for developing big data applications using data pipelining.
Read More
Open Source Big Data Weekly – Issue 20
December 11, 2023
by Dennis Gray
No Comments
Big data and the traditional BI is the product of social development in different stages, big data for traditional BI, both the inheritance, also have...
Read More
Spark Distinct Deduplication principle (Distinct causes shuffle)
December 10, 2023
by Craig Monk
No Comments
Distinct operator principle: Contain reduceByKey will have Shuffle paste Spark source code: example code: end
Read More
Spark implements row and column conversion Pivot and unpivot
December 10, 2023
by Hayley Owen
No Comments
As anyone who has done ETL work with data cleansing knows, row and column transformation is a common data collation requirement. There are different implementations...
Read More
Protobuf combined with Spark Structured Streaming
December 10, 2023
by Melissa Pratt
No Comments
Background Spark Structured Streaming is used to process Streaming data in project development. The processing flow is as follows: Message Middleware (Source) -> Spark Structured...
Read More