preface

This article isSpark knowledge articles directory, easy to view, continuously updated…

0. Spark terms

  • The difference between Spark terms 01-application, Job, Stage, and Task

1. Spark basics

  • Spark Base 01-RDD and width dependency
  • Spark Basic 02-RDD data set operations
  • Spark Basics 03- Associate external data sources
  • Spark Basic 04-RDD Repartition Operations Repartition and coalesce
  • Spark Basics 05-Map and mapValue
  • Spark Basics 06-Spark Client and Cluster Submission Process
  • Spark Basics 07- WorkCount source code and RDD core methods

2. Spark source-resource layer is started

  • Spark source code parsing 01-Master Startup process
  • 02-Worker startup process and heartbeat communication with Master

Spark source code – computing layer

  • Spark source code parsing 03-Submit submission process and Driver start process
  • Spark source Code Parsing 04-Submit process and SparkContext preparation process
  • Submit submission process and Master resource allocation and Executor startup