The author | | Vineyard team source alibaba cloud native public number

Vineyard is a distributed engine designed to provide in-memory data sharing for end-to-end workflows in big data analytics scenarios in the cloud native environment, and we are pleased to announce that Vineyard has been accepted as a Sandbox project by THE CNCF TOC on April 27, 2021.

Vineyard project open source address: https://github.com/alibaba/v6d

Project introduction

Scenario, the existing large data analysis for end-to-end tasks, often use such as between different subtasks HDFS, S3, OSS such a distributed file system or object storage system, to share between tasks in the middle of the data, this approach has many problems on efficiency and productivity, as shown in the following figure of a risk control operation workflow, for example:

  1. In order to share intermediate data between different tasks in the workflow, the first task writes the results to the file system, and the second task reads the file as input. This process incurs additional serialization and deserialization, memory copy, and network and IO overhead. We observed from historical missions that more than 60% of missions took more than 40% of their execution time.
  2. For production environments, a new system (such as distributed graph computing) is often introduced to efficiently solve a particular paradigm problem, but such a system is often difficult to seamlessly integrate directly with other systems in the workflow, requiring a lot of repetitive IO, data format conversion and adaptation research and development.
  3. Sharing data using an external file system introduces additional disruptions to the workflow because it is often only when one task has written all the results that the next task can begin reading and computing, making pipeline parallelism across tasks impossible to apply.
  4. When sharing intermediate data, the existing distributed file system, especially in the cloud native environment, does not deal with the location of distributed data well, resulting in the waste of network overhead and reducing the end-to-end execution efficiency.

To address these issues in existing big data analytics workflows, we designed and implemented Vineyard, a distributed memory data sharing engine.

Vineyard addressed these issues from three perspectives:

  1. To make data sharing between tasks more efficient in an end-to-end workflow, Vineyard allows zero-copy data sharing between systems through memory mapping, eliminating additional IO overhead.
  2. To simplify the adaptation and development needed for new computing engines to connect to existing systems, Vineyard provides out-of-the-box abstractions for common data types like Tensor, DataFrame, Graph, and so on, eliminating the need for additional serialization and antisequence to share intermediate results between different computing engines. Vineyard also implemented reusable components such as IO, data migration, and snapshots as plug-ins, giving them the flexibility to register on demand with a computing engine, reducing development costs unrelated to the engine itself.
  3. Vineyard provides a series of Operators for more efficient and flexible data sharing. For example, the Pipeline operator realizes Pipeline parallelism across tasks so that subsequent tasks can be calculated simultaneously with the output of the preceding tasks, improving the overall end-to-end efficiency.
  4. Vineyard integrates with Kubernetes and, through the Scheduler Plugin, allows task scheduling to be aware of the locality of the required data, while Kubernetes allows pods for individual tasks to be dispatched as closely as possible to the machines that the PODS need input data. To reduce the network overhead required by data migration and improve end-to-end performance.

In preliminary comparative experiments, Vineyard was able to significantly reduce the overhead introduced by exchanging intermediate results for profiling tasks compared to sharing intermediate data with HDFS, improving end-to-end time by a factor of 1.34 for the overall workflow.

The core function

Next, we’ll look at the core features of Vineyard in terms of its design and implementation, and how Vineyard contributes to the task of big data analysis in the cloud native environment.

1. Distributed memory data sharing

Vineyard represents in-memory data as objects. Object can be Local or Global. Taking Mars and Dask as examples, a DataFrame is often divided into many chunks to utilize the computing power of multiple machines. Each machine has multiple chunks. These chunks are localobjects in Vineyard, and together they form a global view, a GlobalDataFrame. This GlobalDataFrame can be shared directly with other computing engines, such as GraphScope, as input to graph data. With the abstraction of these data types, intermediate results can be seamlessly shared between different computing engines on Vineyard, using the output of one task directly as the output of the next.

More specifically, what about expressing a particular type of Object in Vineyard that can be easily adapted to different computing engines? This is thanks to the flexibility Vineyard offers in the representation of objects. In Vineyard, an Object has two parts, Metadata, and a set of bloBs. The actual data is stored in bloBs, and Metadata is used to interpret the semantics of those BLOBs. For example, for the Tensor, the Blob is a continuum of memory that stores all the elements of the Tensor, and the Metadata records the type of the Tensor, the shape of the Tensor, whether the sequence is row main or column main. In Python, this Object can be interpreted as a NDArray for Numpy, and in C++, this Object can be interpreted as a tensor in xtensor. Sharing that Tensor between the SDKS of these two different programming languages doesn’t have the overhead of IO, copy, serialization/deserialization, conversion.

At the same time, Metadata in Vineyard is nested, which makes it easy to describe any complex data type as an Object in Vineyard without limiting the expressive power of the computing engine. Using GlobalDataFrame as an example, the Metadata structure is shown in the following figure.

2. Collaborative scheduling of data and tasks in the cloud native environment

For a truly deployed big data analysis pipeline, data sharing between tasks is not enough. In the cloud environment, multiple sub-tasks contained in an end-to-end pipeline are scheduled by Kubernetes only considering required resource constraints. Co-locate of two consecutive tasks cannot be guaranteed, and there is still network overhead introduced by data migration when intermediate results are shared between two tasks, as shown in the figure below. When Task B is run, data fragments A3 and A4 need to be migrated to the Vineyard instance where the Pod is located because the pods of the two tasks are not aligned.

Vineyard used CRD to represent data in a cluster as an observable resource, and designed and implemented a Scheduler plug-in based on the Kubernetes Scheduler Framework that considers data locality. After Task A completes, the scheduler plug-in can know the location of all fragments from the Metadata of the resulting object. When starting the next Task, the scheduler gives higher priority to the Node where the data resides (Node 1 and Node 2 in the figure). Task B can also be scheduled to the corresponding node as much as possible, thus eliminating the extra overhead introduced by data migration and improving end-to-end performance.

Quick learning

Vineyard integrates Helm for easy installation and deployment:

helm repo add vineyard https://vineyard.oss-ap-southeast-1.aliyuncs.com/charts/
helm install vineyard vineyard/vineyard
Copy the code

After installation, a Vineyard DaemonSet is deployed in the system and a UNIX Domain socket is exposed for shared memory and IPC communication with the application’s task pods.

In addition, you can refer to the Vineyard of the demo video: https://www.youtube.com/watch?v=vPbF1l5nwwQ&list=PLj6h78yzYM2NoiNaLVZxr-ERc1ifKP7n6&t=585

future

Already serving as a storage engine for Mars, a distributed scientific computing engine, and GraphScope, a one-stop computing system, Vineyard helps big data analysis tasks by closely interacting with the cloud-native community. In the future, Vineyard will further improve its integration with other community projects such as Kubeflow and Fluid to facilitate more big data analysis tasks in the cloud.

Vineyard will continue to work with the community, support community feedback, and promote the ecosystem and application of cloud native technologies in big data analytics. You’re welcome to follow the Vineyard project, join the Vineyard community, and participate in building and launching the Vineyard project!

2021 Ali Cloud Developer Conference opens!

How to better harness the power of the cloud in the digital age? What is the new and convenient development mode? How can developers build applications more efficiently? Technology empowers society, technology drives change, expands developers’ energy boundaries, and everything is different in the cloud. Click now to sign up for the event, and the answer will be given to you at the 2021 Ali Cloud Developer Conference.