Tag: Big data

The back-end

Big Data Development -Spark- Introduction to Spark-Graph && Quick Start

January 3, 2024

by Prisha Kapadia

No Comments

GraphX is a component of Spark that is used to represent graphs and perform parallel computation of graphs. GraphX extends RDD by redefining the abstraction...

Read More

The back-end

Big Data Development – In-depth understanding of the Big data Lambda architecture

January 3, 2024

by Jasmine Simmons

No Comments

In the development of IT, the first stage is the emergence of major systems and platforms, which solves the efficiency problem of moving from offline...

Read More

reading

SmartNews: Practice of accelerating Hive calendar production based on Flink

January 3, 2024

by Kiara Manda

No Comments

Technical challenges and solutions for seamlessly integrating Flink into our massive Airflow and Hive batch systems.

Read More

reading

ClickHouse Data life cycle

January 3, 2024

by Francesca Savage-Hope

No Comments

This morning, I saw the mailbox alarm pushed by CDH cluster and found that the disk was not enough. The production line itself doesn't write...

Read More

reading

Demo of openLooKeng+Ranger+LDAP authentication capability

January 3, 2024

by Dr. Kelly Bailey

No Comments

OpenLooKeng can be connected to LDAP for authentication and Ranger for permission control. This demo uses our experimental version of openLooKeng

Read More

Artificial intelligence (ai)

The revival of ali’s canceled weekly? Three minutes of efficient reporting is the last resort

January 3, 2024

by Ojas Garg

No Comments

A few months ago, ali once had internal employees broke the news that the weekly newspaper system was cancelled, and the workers who were oppressed...

Read More

The back-end

Big Data development -Spark- Developing Streaming processing data && writing Kafka

January 3, 2024

by Bruce Moreno

No Comments

1.SparkStreaming Introduction SparkStreaming reads data from a variety of input sources and groups the data into small batches. New batches are created at even intervals....

Read More

The back-end

Big Data Learning Note 0: Basic Framework of big Data

January 3, 2024

by Renee Hayre

No Comments

Starting from data sources, big data generally needs to go through six major steps, including data collection, data storage, resource management and service coordination, computing...

Read More

reading

Writing Book Notes on Digital Transformation – Day 22

January 3, 2024

by Krystal Mason-Duncan

No Comments

today Writing is really not satisfactory... I think I need to change. Today, I will analyze the three sections of Internet enterprises for you. A...

Read More

The back-end

Companion fish: Complete the upgrade of machine learning feature system with Flink

January 3, 2024

by Hrishita Koshy

No Comments

Flink is used in machine learning feature engineering to solve the problem of feature on-line difficulty. And how the SQL + Python UDF is used...

Read More

The back-end

PyFlink development environment: Zeppelin Notebook

January 3, 2024

by Sienna Hart

No Comments

Use Conda in Zeppelin Notebook to create Python env to automatically deploy to Yarn cluster.

Read More

reading

SmartNews: Practice of accelerating Hive calendar production based on Flink

January 3, 2024

by Leslie Hansen-O'Neill

No Comments

Technical challenges and solutions for seamlessly integrating Flink into our massive Airflow and Hive batch systems.

Read More

The back-end

Application practice of Flink in SF Express

January 3, 2024

by Gokul Chacko

No Comments

Based On Flink's idea of building real-time data warehouse, SF Express introduced Hudi On Flink to accelerate data warehouse width table and the practice of...

Read More

The back-end

Preview of new Flink 1.14 features

January 3, 2024

by Jivika Grewal

No Comments

This post was shared by Flink Meetup on August 7th, and the latest developments in version 1.14 are explained with notes at the end of...

Read More

The code of life

What is the HTTP protocol? What are the commonly used status codes?

January 3, 2024

by Laura Lowery

No Comments

HTTP is short for Hyper Text Transfer Protocol. It is used to Transfer hypertext from the World Wide Web server to the local browser.

Read More

The code of life

Hadoop2.5.2 learning 13-mr sina weibo -DF implementation

January 3, 2024

by 黃淑華

No Comments

This paper is followed by hadoop2.5. In the last microblog, the first MapPreduce is realized, and the word frequency TF and the total number of...

Read More

The back-end

Cloud small classes | large data fusion analysis: GaussDW (DWS) easily import MRS hybrid data source

January 3, 2024

by Kashvi Sane

No Comments

Abstract: By establishing the connection between GaussDB(DWS) and MRS, the database supports data warehouse service SQL on Hadoop, and implements the quick import of Hive...

Read More

The back-end

Data centers consume too much power. Here’s an energy saver

January 2, 2024

by Anya Ratti

No Comments

Abstract: In fact, for energy saving, traditional technology is also made "twelve" efforts. However, in the case of continuous technological evolution, the traditional energy-saving technology...

Read More

The back-end

Big Data Development -Scala- Type checking and pattern matching details

January 2, 2024

by Christopher Decker

No Comments

Type checking and conversions are implemented in every language, such as Instanceof and isInstance in Java, as well as Scala, but Scala has powerful pattern...

Read More

Artificial intelligence (ai)

Hive Data Skew

January 2, 2024

by Mohanlal Brar

No Comments

The causes of data skew are as follows: Uneven distribution of keys, features of service data, data skew of SQL statements, and non-standard table construction....

Read More

The code of life

Apache Griffin+Flink+Kafka data quality monitoring practice

January 2, 2024

by Albert Hardy

No Comments

Components and versions Components used in this article include the following, is a reference to the official case, version can refer to Github and the...

Read More

The code of life

Functions for Scala basics

January 2, 2024

by Rebecca O'Brien

No Comments

2.package com.chb.package com.chb. If it is a recursive function, you also need to specify the return type. Package com.chb. Sometimes there is a requirement to...

Read More

The back-end

Build a streaming data lake using Flink Hudi

January 2, 2024

by Dr. Guy Harrison

No Comments

This paper introduces Flink Hudi's continuous optimization and evolution of the original mini-batch based incremental computing model through flow computation.

Read More

The back-end

Big data development -Spark-Streaming processing data to mysql

January 2, 2024

by Penelope McManaway

No Comments

Streamin reads kafka data processing and writes it to Kafka data. Big data development -Spark- Developing Streaming processing data && writing kafka data is for...

Read More

The back-end

Distributed message flow platforms: Don’t just think Kafka, but Pulsar

January 2, 2024

by Cody Todd

No Comments

Abstract: Pulsar, as a cloud-native distributed message flow platform, has appeared more and more frequently in people's eyes, and has a tendency to replace Kafka.

Read More

The code of life

A collection of Scala basics

January 2, 2024

by Donna Morgan

No Comments

1. _ is a wildcard character, similar to * in Java. Such as import Scala.math. _ 2, :_* as a whole, tells the compiler that...

Read More

The code of life

ANDROID Memory Optimization

January 2, 2024

by Amanda Harrell

No Comments

OOM: Memory leaks can cause a number of problems: 1. Application delays and slow response times (JVMS frequently trigger GC when memory usage is high)...

Read More

The code of life

The basic Linux command 00

January 2, 2024

by Michael Stevens

No Comments

Delete files and directories rm -r Delete directories -f Mandatory Note: In the working environment, back up important files before deleting them. Otherwise, important files...

Read More

The development tools

The pit you stepped in while synchronizing data from Dameng to MemFireDB with Datax

January 2, 2024

by 成雅慧

No Comments

Background During the construction of a data warehouse, some business systems use Dameng database to store original data. Now, data needs to be synchronized to...

Read More

The back-end

Practice and optimization of Apache Flink in JINGdong

January 2, 2024

by Grace Allen

No Comments

This article is compiled from the topic "Practice and Optimization of Apache Flink in JD" shared by Fu Haitao, senior technical expert of JD in...

Read More