Scala: Development environment setup, variables, judgments, loops, functions, collections

1. Introduction of scala

Scala is a multi-paradigm programming language that runs on the JVM and supports both object-oriented and functional programming

In its early days, Scala didn’t attract much attention, but with the rise of Scala-based big data frameworks like Kafka and Spark, Scala has come to the attention of big data developers. Scala’s main advantage is its expressiveness.

Next, we are going to learn:

  • Why use Scala?
  • Two examples compare the Java language and scala language

1.1 Why Scala

  • Develop big data applications (Spark program, Flink program)

  • Strong expression ability, a line of code equal to Java lines, fast development

  • Java compatibility, access to a huge Java class library, and integration into the Hadoop ecosystem

1.2 Scala vs. Java

Here are two examples of how much code is implemented using Java and Scala

case

Define three entity classes (user, order, goods)

Java code

/** * User entity class */
public class User {
    private String name;
    private List<Order> orders;

    public String getName(a) {
    	return name;
    }

    public void setName(String name) {
    	this.name = name;
    }

    public List<Order> getOrders(a) {
    	return orders;
    }

    public void setOrders(List<Order> orders) {
    	this.orders = orders; }}Copy the code
/** * Order entity class */
public class Order {
    private int id;
    private List<Product> products;

    public int getId(a) {
    	return id;
    }

    public void setId(int id) {
    	this.id = id;
    }

    public List<Product> getProducts(a) {
    	return products;
    }

    public void setProducts(List<Product> products) {
    	this.products = products; }}Copy the code
/** * commodity entity */
public class Product {
    private int id;
    private String category;

    public int getId(a) {
    	return id;
    }

    public void setId(int id) {
    	this.id = id;
    }

    public String getCategory(a) {
    	return category;
    }

    public void setCategory(String category) {
    	this.category = category; }}Copy the code

Scala code

case class User(var name:String, var orders:List[Order])	// User entity class
case class Order(var id:Int, var products:List[Product])	// Order entity class
case class Product(var id:Int, var category:String)  		// Commodity entity class
Copy the code

2. Install the development environment

Do you need to install the Scala compiler and development tools before learning how to write Scala code

Java programs compile and execute the flow

Scala programs compile and execute the flow

Scala programs rely on Java class libraries to run and must have a Java runtime environment for Scala to run correctly

According to the above flowchart, to compile and run a Scala program, you need

  • The JDK (JVM)
  • Scala Compiler (Scala SDK)

Next, you need to install the following in sequence:

  • Install the JDK
  • Install scala SDK
  • Installing the IDEA plug-in

2.1 to install the JDK

Install the JDK 1.8 64-bit version and configure the environment variables

2.2 Installing the Scala SDK

The Scala SDK is the compiler of scala language. To develop Scala programs, you must install the SDK first

The version of this installation is 2.11.12

steps

  1. Download and install the SDK
  2. Check whether the installation is successful

Specific operation

The Windows platform

  1. Unzip Scala-2.11.12.zip and install Scala in a directory without Chinese and Spaces

  2. Configuring environment Variables

    SCALA_HOME = C: \ Work \ soft \ scala - 2.11.12 path = % SCALA_HOME % \ bin;Copy the code

  3. Open the console and type Scala-version

Linux platform

  1. Unzip Scala-2.11.12.tgz and install Scala in a directory with no Chinese and no Spaces

  2. Configuring environment Variables

    Tar -zxvf scala-2.11.12.tgz vi /etc/profile export SCALA_HOME=/export/ Servers /scala-2.11.12 export PATH=$PATH:$SCALA_HOME/bin source /etc/profileCopy the code

2.3 Installing the IDEA Scala Plug-in

IDEA does not support Scala program development by default, so you need to install scala plug-ins to support the Scala language.

steps

  1. Download the IDEA Scala plug-in of the specified version
  2. IDEA Configures the Scala plug-in
  3. Restart IDEA

Specific operation

Operation 1: View the version of IDEA

Operation 2: Download the CORRESPONDING IDEA Scala plug-in from the OFFICIAL website of IDEA

[!DANGER]

Be sure to download the Scala plug-in of the same IDEA version

Operation 3: Choose Configuration > Select Plug-ins

Operation 4: Click on pinion > Select Install plug-in from local

Action 5: Locate the downloaded plug-in and click OK

Operation 6: Restart IDEA

Action 7: View the Scala plug-in

3. Scala interpreter

Later, we will use the Scala interpreter to learn the basic syntax of Scala. The Scala interpreter executes a piece of code like a Linux command and immediately shows us the result of execution, which is convenient for testing.

We are going to learn:

  • Start the Scala interpreter
  • Executes Scala code in the Scala interpreter
  • Exit the Scala interpreter

3.1 Starting the Scala interpreter

To start the Scala interpreter, there are only a few steps:

  • Hold down theThe Windows key + r
  • The inputscalaCan be

3.2 Executing Scala code

Type println(“hello, world”) in scala’s command prompt window and press Enter to execute

3.3 Exiting the interpreter

To exit the interpreter, run the quit command in the Scala command prompt window

4. Declare variables

We’ll define variables every day we write scala programs. How does the Scala language define variables?

4.1 Syntax Format

Java variable Definition

int a = 0;
Copy the code

In Scala, variables can be defined using val or var in the following syntax:

val/varVariable identifier: Variable type = initial valueCopy the code

Among them

  • valDefines variables that cannot be reassigned
  • varDefines variables that can be reassigned

[!NOTE]

  • In Scala, the defined variable type is written after the variable name
  • Scala statements do not require a semicolon at the end

4.2 Define a variable in the interpreter

Example: Define a variable to hold a person’s name “Tom”

steps

  1. Open the Scala interpreter
  2. Defines a variable of type string to hold the name

Reference code

scala> val name:String = "tom"
name: String = tom
Copy the code

4.3 Val and var variables

The sample

Reassign the name variable to Jim and observe the result

Reference code

scala> name = "Jim"
<console>:12: error: reassignment to val
       name = "Jim"
Copy the code

The sample

Redefine the variable with var to hold the name “Tom”, try reassigning to Jim, and see what happens

Reference code

scala> var name:String = "tom"
name: String = tom

scala> name = "Jim"
name: String = Jim
Copy the code

[!TIP]

Variables are defined using val first, and var only if they need to be reassigned

4.4 Use type inference to define variables

Scala’s syntax is much cleaner than Java’s, and we can define variables in a much cleaner way.

The sample

Let’s use a cleaner syntax to define a variable that holds a person’s name “Tom”

Reference code

scala> val name = "tom"
name: String = tom
Copy the code

Scala can automatically infer the type of a variable from its value, which makes writing code much cleaner.

4.5 Lazy assignment

In enterprise big data development, you can sometimes write very complex SQL statements that may have hundreds or even thousands of rows. These SQL statements, if loaded directly into the JVM, have a significant memory overhead. How to solve it?

When some variables hold a large amount of data but do not need to be loaded into JVM memory immediately. Lazy assignment can be used to improve efficiency.

Syntax format:

lazy val/varVariable name = expressionCopy the code

The sample

The following complex SQL statement needs to be executed in the program, and we want to load it only when the SQL statement is used.

"""insert overwrite table adm.itcast_adm_personas select a.user_id, a.user_name, a.user_sex, a.user_birthday, a.user_age, a.constellation, a.province, a.city, a.city_level, a.hex_mail, a.op_mail, a.hex_phone, a.fore_phone, a.figure_model, a.stature_model, b.first_order_time, b.last_order_time, ... d.month1_hour025_cnt, d.month1_hour627_cnt, d.month1_hour829_cnt, d.month1_hour10212_cnt, d.month1_hour13214_cnt, d.month1_hour15217_cnt, d.month1_hour18219_cnt, d.month1_hour20221_cnt, d.month1_hour22223_cnt from gdm.itcast_gdm_user_basic a left join gdm.itcast_gdm_user_consume_order b on a.user_id=b.user_id left join gdm.itcast_gdm_user_buy_category c on a.user_id=c.user_id left join gdm.itcast_gdm_user_visit d on a.user_id=d.user_id;" ""
Copy the code

Reference code

scala> lazy val sql = """insert overwrite table adm.itcast_adm_personas | select | a.user_id, .... | left join gdm.itcast_gdm_user_buy_category c on a.user_id=c.user_id | left join gdm.itcast_gdm_user_visit d on a.user_id=d.user_id;" ""
sql: String = <lazy>
Copy the code

5. The string

Scala provides a variety of ways to define strings, and you can choose the most convenient way to define strings in the future.

  • Use double quotation marks
  • Use interpolation
  • Use triple quotation marks

5.1 Using Double Quotation marks

grammar

val/varVariable name = "string"Copy the code

The sample

There is a man named “Hadoop”, please print his name and the length of his name.

Reference code

scala> println(name + name.length)
hadoop6
Copy the code

5.2 Use interpolation

In Scala, strings can be defined using interpolation expressions, effectively avoiding concatenation of large numbers of strings.

grammar

val/varThe variable name =s"${variable/expression}The string"
Copy the code

[!TIP]

  • Before the string is defineds
  • In a string, you can useThe ${}To reference a variable or write an expression

The sample

Please define several variables, respectively: “zhangsan”, 30, “male”, define a string, save these information.

Print: name=zhangsan, age=30, sex=male

Reference code

scala> val name = "zhangsan"
name: String = zhangsan

scala> val age = 30
age: Int = 30

scala> val sex = "male"
sex: String = male

scala> val info = s"name=${name}, age=${age}, sex=${sex}"
info: String = name=zhangsan, age=30, sex=male

scala> println(info)
name=zhangsan, age=30, sex=male
Copy the code

5.3 Using Triple Quotation marks

If you have a large chunk of text to save, you can use triple quotes to define strings. For example, save a large section of SQL statements. Any string between the three quotes will be the value of the string.

grammar

val/varThe variable name =String 1 String 2""
Copy the code

The sample

Define a string that holds the following SQL statement

select
	*
from
    t_user
where
    name = "zhangsan"
Copy the code

Print the SQL statement

Reference code

val sql = """select | * | from | t_user | where | name = "zhangsan"""" println(sql)Copy the code

6. Data types and operators

Types and operators in Scala are mostly the same as in Java, which we’ll focus on

  • Some usage differences from Java
  • Scala type inheritance system

6.1 Data Types

The base type Type specification
Byte An 8-bit signed integer
Short 16 – bit signed integer
Int 32 – bit signed integer
Long 64 – bit signed integer
Char A 16-bit unsigned Unicode character
String Char sequence (string)
Float 32-BIT single-precision floating point number
Double 64 – bit double – precision floating point number
Boolean True or false

Notice the difference between Scala types and Java

[!NOTE]

  1. All types in Scala begin with a capital letter
  2. Use plasticIntInstead of an Integer
  3. Variables defined in Scala can be typed without being written, allowing the Scala compiler to infer automatically

6.2 the operator

category The operator
Arithmetic operator +, -, *, /
Relational operator >, <, ==! =, >=, <=
Logical operator , &&, | |,!
An operator &, | |, ^, < <, > >

  • Scala does not have the ++ and — operators

  • Unlike Java, in Scala, you can use ==,! =, which correspond to the equals method. To compare the reference values of two objects, use eq

The sample

Given the string “ABC”, create a second string with the value: concatenate an empty string after the first string.

We then compare the two strings to see if their reference values are equal.

Reference code

val str1 = "abc"
val str2 = str1 + ""
str1 == str2
str1.eq(str2)
Copy the code

6.3 Scala Type Hierarchy

type instructions
Any All types of, which has two subclasses AnyRef and AnyVal
AnyVal All numeric typesThe parent class
AnyRef The parent class of all object types (reference types)
Unit Null, Unit is a subclass of AnyVal and has only one instance {% em %}() {% endem %}

It is similar to void in Java, but Scala is much more object-oriented than Java
Null Null is a subclass of AnyRef, which means it is a subclass of all reference types. Its instance is {% em %}null{% endem %}

Null can be assigned to any object type
Nothing All typesA subclass

You cannot create an instance of this type directly. When a method throws an exception, it returns Nothing, because Nothing is a subclass of all classes, so it can be assigned to any type

The problem

Is there a problem with the following code?

val b:Int = null
Copy the code

Scala interprets the error:

Null cannot be converted to Int, indicating that Null is not a subclass of Int

Conditional expressions

A conditional expression is an if expression, which can perform the corresponding operation according to the result of the condition (true or false) according to whether the given condition is satisfied. Scala conditional expressions have the same syntax as Java.

7.1 If with a return value

Unlike Java,

[!NOTE]

  • In Scala, conditional expressions also return values
  • In Scala, there are no ternary expressions and you can use if expressions instead

The sample

If sex is equal to “male”, result is equal to 1, otherwise result is equal to 0

Reference code

scala> val sex = "male"
sex: String = male

scala> val result = if(sex == "male") 1 else 0
result: Int = 1
Copy the code

7.2 Block Expressions

  • In Scala, {} is used to represent a block expression
  • Like if expressions, block expressions have values
  • The value is the value of the last expression

The problem

In the following code, what is the value of variable A?

scala> val a = {
     | println("1 + 1")
     | 1 + 1|}Copy the code

8. Cycle

In Scala, you can use for and while, but for expressions are generally recommended because of their cleaner syntax

8.1 For Expressions

grammar

for(I <- expression/array/set) {/ / expression
}
Copy the code

8.1.1 Simple Loops

Print the numbers 1-10 using the for expression

steps

  1. Generate numbers 1-10 (tip: Use the to method)
  2. Iterate over each number using a for expression

Reference Code 1

scala> val nums = 1.to(10)                                                              
nums: scala.collection.immutable.Range.Inclusive = Range(1.2.3.4.5.6.7.8.9.10) 
                                                                                        
scala> for(i <- nums) println(i)                                                                                                                                          
Copy the code

shorthand

Reference Code 2

// infix call method
scala> for(i <- 1 to 10) println(i)
Copy the code

8.1.2 Nested loops

Using the for expression, print the following characters

* * * * * * * * * * * * * * *Copy the code

steps

  1. Print 3 rows and 5 columns of stars using the for expression
  2. For every 5 stars printed, newline

Reference code

for(i <- 1 to 3; j <- 1 to 5) {print("*");if(j == 5) println("")}
Copy the code

8.1.3 guard

In a for expression, you can add an if statement, which is called a guard. We can use guards to make the for expression more concise.

grammar

for(I <- expression/array/setifExpression) {/ / expression
}
Copy the code

The sample

Use the for expression to print numbers that are divisible by 3 between 1 and 10

Reference code

// Add a guard to print a number divisible by 3
for(i <- 1 to 10 if i % 3= =0) println(i)
Copy the code

8.1.4 Derivation for

  • In the future, you can use the for derivation to generate a new set (a set of data)

  • In the body of a for loop, we can use yield expressions to build a collection. We call for-expressions that use yield expressions derivations

The sample

Generates a 10, 20, 30… A collection of 100

Reference code

// For: the for expression starts with yield and builds a collection
val v = for(i <- 1 to 10) yield i * 10
Copy the code

8.2 the while loop

The while loop in Scala is the same as in Java

The sample

Print the numbers 1-10

Reference code

scala> var i = 1
i: Int = 1

scala> while(i <= 10) {
     | println(i)
     | i = i+1|}Copy the code

8.3 break and continue

  • In scala, the break/continue keyword similar to Java and C++ has been removed
  • If you must use break/continue, you need to use the breable and break methods of the break class of the Scala.util. control package.

8.3.1 implementation break

usage

  • Import Breaks packageimport scala.util.control.Breaks._
  • Wrap the for expression with breakable
  • Where the for expression needs to exit the loop, addbreak()The method call

The sample

Print numbers from 1 to 100 using the for expression. If the number reaches 50, exit the for expression

Reference code

// Import Break under scala.util. Control
import scala.util.control.Breaks._

breakable{
    for(i <- 1 to 100) {
        if(i >= 50) break(a)else println(i)
    }
}
Copy the code

8.3.2 realize the continue

usage

The implementation of continue is similar to that of break, but with one difference:

[!NOTE]

Breakable {} wraps the entire for expression, and breakable{} wraps the body of the for expression

The sample

Print numbers from 1 to 100, using the for expression to iterate over them. If the number divisible by 10, do not print

// Import Break under scala.util. Control
import scala.util.control.Breaks. _for(i <- 1 to 100 ) {
    breakable{
        if(i % 10= =0) break(a)else println(i)
    }
}
Copy the code

Method of 9.

A class can have its own methods, and methods in Scala are similar to Java methods. But Scala has a different syntax for defining methods than Java does.

9.1 Defining Methods

grammar

def methodName Parameter name: parameter type, parameter name: parameter type) : [return type] = {
    // Method body: a series of code
}
Copy the code

[!NOTE]

  • Parameter types in the parameter list cannot be omitted
  • The return value type can be omitted and inferred automatically by the Scala compiler
  • The default value is the value of the {} block expression

The sample

  1. Define a method that adds two integer values and returns the result
  2. Calling this method

Reference code

scala> def add(a:Int, b:Int) = a + b
m1: (x: Int, y: Int)Int

scala> add(1.2)
res10: Int = 3
Copy the code

9.2 Inference of Return Value Type

Scala definition methods can omit the return value and Scala automatically infer the return value type. This way the method is defined more succinctly.

[!DANGER]

Define recursive methods without omitting return value types

The sample

Define recursive methods (factorial)

10 * 9 * 8 * 7 * 6 *… * 1

Reference code

scala> def m2(x:Int) = {|if(x<=1) 1
     | else m2(x- 1) * x
     | }
<console>:13: error: recursive method m2 needs result type
       else m2(x- 1) * x
Copy the code

9.3 Method Parameters

Method parameters in Scala are flexible to use. It supports the following types of parameters:

  • The default parameters
  • With name parameter
  • Variable-length argument

9.3.1 Default Parameters

You can define a default value for a parameter when you define a method.

The sample

  1. Defines a method that evaluates the addition of two values, which default to 0
  2. This method is called without passing any arguments

Reference code

// x, y with default value 0
def add(x:Int = 0, y:Int = 0) = x + y
add()
Copy the code

9.3.2 Named Parameters

When calling a method, you can specify the name of the argument to make the call.

The sample

  1. Defines a method that evaluates the addition of two values, which default to 0
  2. Call this method, setting only the value of the first parameter

Reference code

def add(x:Int = 0, y:Int = 0) = x + y
add(x=1)
Copy the code

9.3.3 Variable Length Parameters

If the method’s arguments are not fixed, you can define a method whose arguments are variable-length arguments.

Syntax format:

def The method name(Parameter name: Parameter type *): Return value type = {method body}Copy the code

[!NOTE]

The parameter type is followed by an asterisk (*) to indicate that there can be zero or more parameters

The sample

  1. Defines a method that computes the sum of several values
  2. Call the method, passing in the following data: 1,2,3,4,5

Reference code

scala> def add(num:Int*) = num.sum
add: (num: Int*)Int

scala> add(1.2.3.4.5)
res1: Int = 15
Copy the code

9.4 Method Invocation Mode

In Scala, there are several method calls,

  • Postfix method
  • Infix call method
  • Curly bracket call method
  • Call method without parentheses

We will use these method calls when we write spark and Flink programs.

9.4.1 Suffix call method

This approach is no different from Java.

grammar

Object name. Method name (parameter)Copy the code

The sample

Use the suffix math.abs to find the absolute value

Reference code

scala> Math.abs(- 1)
res3: Int = 1
Copy the code

9.4.2 Infix invocation method

grammar

Object name method name parameterCopy the code

For example, 1 to 10

[!TIP]

If there are multiple arguments, enclose them in parentheses

The sample

Use infix method math. abs to find absolute value

scala> Math abs - 1
res4: Int = 1
Copy the code

Operators are methods

Let’s look at an expression

1 + 1
Copy the code

Does the above expression look like a method call to you?

In Scala, operators such as + – * / % are the same as in Java, but in Scala,

  • All operators are methods
  • An operator is a method whose name is a symbol

9.4.3 Curly bracket Call Method

grammar

Math.abs{ 
    // Expression 1
    // Expression 2
}
Copy the code

[!DANGER]

Method takes only one argument to use the curly brace call method

The sample

Use curly braces to call math.abs to find the absolute value

Reference code

scala> Math.abs{- 10}
res13: Int = 10
Copy the code

9.4.4 Invocation without parentheses

If a method has no arguments, you can omit the parentheses following the method name

The sample

  • Define a method with no arguments, print “hello”
  • The method is called using the parenthesless invocation method

Reference code

def m3()=println("hello")
m3()
Copy the code

Function of 10.

Scala supports functional programming, which will be heavily used in Spark/Flink programs in the future

10.1 Defining a Function

grammar

valFunction variable name = (Parameter name: parameter type, parameter name: parameter type....) = > the function bodyCopy the code

[!TIP]

  • A function is an object (variable)
  • Like methods, functions have input parameters and return values
  • Function definitions are not neededdefdefine
  • You do not need to specify a return value type

The sample

  1. Define a function that adds two values
  2. Calling this function

Reference code

scala> val add = (x:Int, y:Int) => x + y
add: (Int.Int) = >Int = <function2>

scala> add(1.2)
res3: Int = 3
Copy the code


10.2 The difference between methods and functions

  • Methods belong to a class or object and are loaded into the JVM’s method section at run time
  • You can assign a function object to a variable that is loaded into the JVM’s heap memory at run time
  • A function is an object that inherits from FunctionN, which has the apply, Curried, toString, tupled methods. Methods don’t

The sample

Method cannot be assigned to a variable

scala> def add(x:Int,y:Int)=x+y
add: (x: Int, y: Int)Int

scala> val a = add
<console>:12: error: missing argument list for method add
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `add _` or `add(_,_)` instead of `add`.
       val a = add
Copy the code

10.3 Conversion of methods to functions

  • Sometimes you need to convert a method to a function, and when you pass it as a variable, you need to convert a method to a function

  • Use _ to convert a method to a function

The sample

  1. Defines a method for adding two numbers
  2. Convert the method to a function and assign a value to the variable

Reference code

scala> def add(x:Int,y:Int)=x+y
add: (x: Int, y: Int)Int

scala> val a = add _
a: (Int.Int) = >Int = <function2>
Copy the code

An array of 11.

The concept of arrays in Scala is similar to Java in that you can use arrays to hold a set of data. In Scala, there are two types of arrays, fixed-length and variable-length

11.2 Fixed-length arrays

  • A fixed-length array means that the length of the array is not allowed to change
  • The elements of an array can be changed

grammar

// Define the array by specifying the length
val/varThe variable name =new Array[Element type](Array length)// Initialize arrays directly with elements
val/varThe variable name =Array(element1Elements,2Elements,3...)
Copy the code

[!NOTE]

  • In Scala, array generics are specified using []

  • Use () to get elements

The sample a

  1. Define an integer array of length 100
  2. Set the first element to 110
  3. Print the first element

Reference code

scala> val a = new Array[Int] (100)
a: Array[Int] = Array(0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0)

scala> a(0) = 110

scala> println(a(0))
110
Copy the code

Example 2

  1. Define an array containing the following elements

    "java"."scala"."python"
Copy the code

  2. Get the array length

Reference code

// Define an array containing jave, Scala, and Python elements
scala> val a = Array("java"."scala"."python")
a: Array[String] = Array(java, scala, python)

scala> a.length
res17: Int = 3
Copy the code

11.3 Variable length Arrays

Variable-length arrays are arrays of variable length that can be added or removed

11.3.1 Defining variable-length Arrays

Create an array of variable length, need to advance import ArrayBuffer class import scala. Collection. The mutable. ArrayBuffer

grammar

  • Create an empty ArrayBuffer variable length array.

    val/var a = ArrayBuffer[Element type]()Copy the code

  • Creates an ArrayBuffer with initial elements

    val/var a = ArrayBuffer(element1Elements,2Elements,3....).Copy the code

The sample a

Defines an integer variable length array of length 0

Reference code

val a = ArrayBuffer[Int] ()Copy the code

Example 2

Defines a variable-length array containing the following elements

"hadoop"."storm"."spark"
Copy the code

Reference code

scala> val a = ArrayBuffer("hadoop"."storm"."spark")
a: scala.collection.mutable.ArrayBuffer[String] = ArrayBuffer(hadoop, storm, spark)
Copy the code

11.3.2 Adding/Modifying/Deleting Elements

  • use+ =Add elements
  • use- =Remove elements
  • use+ + =Appends an array to a variable-length array

The sample

  1. Define a variable length array containing the following elements: “Hadoop “,” Spark “, “flink”
  2. Add a “flume” element to the variable-length array
  3. Remove the “Hadoop” element from the variable-length array
  4. Append an array containing “hive” and “sqoop” to the variable length array

Reference code

// Define variable-length arrays
scala> val a = ArrayBuffer("hadoop"."spark"."flink")
a: scala.collection.mutable.ArrayBuffer[String] = ArrayBuffer(hadoop, spark, flink)

// Appends an element
scala> a += "flume"
res10: a.type = ArrayBuffer(hadoop, spark, flink, flume)

// Delete an element
scala> a -= "hadoop"
res11: a.type = ArrayBuffer(spark, flink, flume)

// Appends an array
scala> a ++= Array("hive"."sqoop")
res12: a.type = ArrayBuffer(spark, flink, flume, hive, sqoop)
Copy the code

11.4 Go through the number group

The array can be traversed in two ways:

  • Use the for expression to iterate directly over the elements in the array

  • Iterate over the elements in a group using an index

The sample a

  1. Define an array containing the following elements: 1,2,3,4,5
  2. Use the for expression to iterate directly over and print the elements of the array

Reference code

scala> val a = Array(1.2.3.4.5)
a: Array[Int] = Array(1.2.3.4.5)

scala> for(i<-a) println(i)
1
2
3
4
5
Copy the code

Example 2

  1. Define an array containing the following elements: 1,2,3,4,5
  2. Use a for expression to iterate over the index subscript and print the elements of the array

Reference code

scala> val a = Array(1.2.3.4.5)
a: Array[Int] = Array(1.2.3.4.5)

scala> for(i <- 0 to a.length - 1) println(a(i))
1
2
3
4
5

scala> for(i <- 0 until a.length) println(a(i))
1
2
3
4
5
Copy the code

[!NOTE]

0 until N — Generates a series of numbers, including 0 but not n

0 to n — contains both 0 and n

11.5 Common Algorithms for Arrays

Arrays in Scala encapsulate common computation operations that we don’t need to re-implement ourselves when processing data in the future. Here are some commonly used algorithms:

  • Sum — sum method
  • Find the maximum — Max method
  • Find the minimum — min method
  • Sort — The sorted method

11.5.1 sum

The sum method in the array adds up all the elements to get the result

The sample

  1. Define an array containing the following elements (1,2,3,4)
  2. Calculate the sum of the array

Reference code

scala> val a = Array(1.2.3.4)
a: Array[Int] = Array(1.2.3.4)

scala> a.sum
res49: Int = 10
Copy the code

11.5.2 maximum

The Max method in an array retrieves the value of the largest element in the array

The sample

  1. Define an array containing the following elements (4,1,2,4,10)
  2. Gets the maximum value of the array

Reference code

scala> val a = Array(4.1.2.4.10)
a: Array[Int] = Array(4.1.2.4.10)

scala> a.max
res50: Int = 10
Copy the code

11.5.3 minimum

The min method of the array can get the smallest element value in the array

The sample

  1. Define an array containing the following elements (4,1,2,4,10)
  2. Gets the minimum value of the array

Reference code

scala> val a = Array(4.1.2.4.10)
a: Array[Int] = Array(4.1.2.4.10)

scala> a.min
res51: Int = 1
Copy the code

11.5.4 sorting

The sorted array method, which sorts an array in ascending order. The Reverse method, on the other hand, reverses the array to sort in descending order

The sample

  1. Define an array containing the following elements (4,1,2,4,10)
  2. Sort an array in ascending or descending order

Reference code

// Ascending sort
scala> a.sorted
res53: Array[Int] = Array(1.2.4.4.10)

/ / descending
scala> a.sorted.reverse
res56: Array[Int] = Array(10.4.4.2.1)
Copy the code

12. A tuple

Tuples can be used to contain a set of values of different types. For example: name, age, sex, date of birth. The elements of a tuple are immutable.

12.1 Defining tuples

grammar

Use parentheses to define tuples

val/varTuples = (elements1Elements,2Elements,3....).Copy the code

Use arrows to define tuples (tuples have only two elements)

val/varTuples = elements1- > element2
Copy the code

The sample

Define a tuple containing the following data for a student

id The name age address
1 zhangsan 20 beijing

Reference code

scala> val a = (1."zhangsan".20."beijing")
a: (Int.String.Int.String) = (1,zhangsan,20,beijing)
Copy the code

The sample

  • Define a tuple containing the name and age of the student (Zhangsan, 20)
  • Use parentheses, and arrows to define tuples, respectively

Reference code

scala> val a = ("zhangsan".20)
a: (String.Int) = (zhangsan,20)

scala> val a = "zhangsan" -> 20
a: (String.Int) = (zhangsan,20)
Copy the code

12.2 Accessing Tuples

Use _1, _2, _3…. To access elements in a tuple, _1 means access to the first element, and so on

The sample

  • Define a tuple containing the name and gender of a student, “zhangsan”, “male”
  • Get the student’s name and gender, respectively

Reference code

scala> val a = "zhangsan" -> "male"
a: (String.String) = (zhangsan,male)

// Get the first element
scala> a._1
res41: String = zhangsan

// Get the second element
scala> a._2
res42: String = male
Copy the code

A list of 13.

Lists are the most important and commonly used data structure in Scala. List has the following properties:

  • Duplicate values can be saved
  • There’s a sequence

In Scala, there are also two types of lists, immutable and mutable

13.1 Immutable list

define

An immutable list is a list whose elements and lengths are immutable.

grammar

Use List(element 1, element 2, element 3…) To create an immutable list, syntax:

val/varThe variable name =List(element1Elements,2Elements,3...)
Copy the code

Use Nil to create an immutable empty list

val/varThe variable name =Nil
Copy the code

Create an immutable list using the :: method

val/varVariable name = element1Elements: :2: :Nil
Copy the code

[!TIP]

To create a list using **:: concatenation, you must add a Nil** at the end

The sample a

Create an immutable list of the following elements (1,2,3,4)

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)
Copy the code

Example 2

Use Nil to create an immutable empty list

Reference code

scala> val a = Nil
a: scala.collection.immutable.Nil.type = List(a)Copy the code

Example 3

Use the :: method to create a list of elements -2 and -1

Reference code

scala> val a = 2 -: :- 1: :Nil
a: List[Int] = List(2 -.- 1)
Copy the code

13.2 Mutable List

A mutable list is a list whose elements and lengths are mutable.

To use the variable list, first to import the import scala. Collection. The mutable. ListBuffer

[!NOTE]

  • The variable set is inmutableIn the package
  • Immutable sets are all inimmutablePackage (imported by default)

define

Create an empty mutable list using ListBuffer[element type]().

val/varThe variable name =ListBuffer[Int] ()Copy the code

Use the ListBuffer(element 1, element 2, element 3…) Create a mutable list.

val/varThe variable name =ListBuffer(element1Elements,2Elements,3...)
Copy the code

The sample a

Creates an empty integer mutable list

Reference code

  scala> val a = ListBuffer[Int]()
  a: scala.collection.mutable.ListBuffer[Int] = ListBuffer(a)Copy the code

Example 2

Create a mutable list containing the following elements: 1,2,3,4

Reference code

scala> val a = ListBuffer(1.2.3.4)
a: scala.collection.mutable.ListBuffer[Int] = ListBuffer(1.2.3.4)
Copy the code

13.3 Mutable list Operations

  • Gets the element (accessed using parentheses(Index value))
  • Add elements (+ =)
  • Append a list (+ + =)
  • Change elements (Use parentheses to get the element and then assign it)
  • Delete elements (- =)
  • Convert to List (toList)
  • Convert to Array (toArray)

The sample

  1. Define a mutable list containing the following elements: 1,2,3
  2. Gets the first element
  3. Add a new element: 4
  4. Appends a list containing the following elements: 5,6,7
  5. Delete element 7
  6. Convert a mutable list to an immutable list
  7. Convert a mutable list to an array

Reference code

// Import immutable list
scala> import scala.collection.mutable.ListBuffer
import scala.collection.mutable.ListBuffer

// Create immutable lists
scala> val a = ListBuffer(1.2.3)
a: scala.collection.mutable.ListBuffer[Int] = ListBuffer(1.2.3)

// Get the first element
scala> a(0)
res19: Int = 1

// Appends an element
scala> a += 4
res20: a.type = ListBuffer(1.2.3.4)

// Append a list
scala> a ++= List(5.6.7)
res21: a.type = ListBuffer(1.2.3.4.5.6.7)

// Delete elements
scala> a -= 7
res22: a.type = ListBuffer(1.2.3.4.5.6)

// Convert to immutable list
scala> a.toList
res23: List[Int] = List(1.2.3.4.5.6)

// Convert to an array
scala> a.toArray
res24: Array[Int] = Array(1.2.3.4.5.6)
Copy the code

13.4 Common Operations in the List

The following are common operations for lists

  • Determine if the list is empty (isEmpty)
  • Concatenate two lists (++)
  • Gets the first element of the list (head) and the remainder (tail)
  • Reverse the list (reverse)
  • Get prefix (take), get the suffix (drop)
  • Flattening (flaten)
  • Zipper (zip) and pull apart (unzip)
  • Conversion string (toString)
  • Generate string (mkString)
  • And set (union)
  • Intersection (intersect)
  • Difference set (diff)

13.4.1 Checking whether the List is empty

The sample

  • Define a list containing the following elements: 1,2,3,4
  • Use isEmpty to determine if the list isEmpty

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

scala> a.isEmpty
res51: Boolean = false
Copy the code

13.4.2 Splicing two Lists

The sample

  • There are two lists containing the following elements 1,2,3, and 4,5, and 6 respectively
  • use++Concatenate the two lists

Reference code

scala> val a = List(1.2.3)
a: List[Int] = List(1.2.3)

scala> val b = List(4.5.6)
b: List[Int] = List(4.5.6)

scala> a ++ b
res52: List[Int] = List(1.2.3.4.5.6)
Copy the code

13.4.3 Getting the first element and the rest of the list

The sample

  • Define a list containing the following elements: 1,2,3
  • Get the first element of the list using the head method
  • Use the tail method to get an element other than the first element, which is also a list

Reference code

scala> val a = List(1.2.3)
a: List[Int] = List(1.2.3)

scala> a.head
res4: Int = 1

scala> a.tail
res5: List[Int] = List(2.3)
Copy the code

13.4.4 Reversing the list

The sample

  • Make a list of the following elements: 1,2,3

  • Use the reverse method to reverse the elements of the list

scala> val a = List(1.2.3)
a: List[Int] = List(1.2.3)

scala> a.reverse
res6: List[Int] = List(3.2.1)
Copy the code

13.4.5 Obtaining the List prefix and suffix

The sample

  • Define a list containing the following elements: 1,2,3,4,5
  • Use the take method to get the prefixes (first three elements) : 1,2, 3
  • Use the drop method to get the suffix (elements other than the first three) : 4,5

Reference code

scala> val a = List(1.2.3.4.5)
a: List[Int] = List(1.2.3.4.5)

scala> a.take(3)
res56: List[Int] = List(1.2.3)

scala> a.drop(3)
res60: List[Int] = List(4.5)
Copy the code

13.4.6 Flattening (Flattening)

Flattening means putting all the elements of a list within a list into one list.

The sample

  • List(1,2), List(3), List(4,5)
  • Convert this List to List(1,2,3,4,5) using flatten

Reference code

scala> val a = List(List(1.2), List(3), List(4.5))
a: List[List[Int]] = List(List(1.2), List(3), List(4.5))

scala> a.flatten
res0: List[Int] = List(1.2.3.4.5)
Copy the code

13.4.7 Zipper and unzipper

  • Zip: Use zip to combine two lists into a tuple list
  • Unpack: Unpack a list containing tuples into a tuple containing two lists

The sample

  • There are two lists
    • The first list holds the names of three students: Zhangsan, Lisi, wangwu
    • The second list holds the ages of three students: 19, 20, 21
  • Use the ZIP operation to “pull” the data from the two lists together to form Zhangsan ->19, Lisi ->20, wangwu->21

Reference code

scala> val a = List("zhangsan"."lisi"."wangwu")
a: List[String] = List(zhangsan, lisi, wangwu)

scala> val b = List(19.20.21)
b: List[Int] = List(19.20.21)

scala> a.zip(b)
res1: List[(String.Int)] = List((zhangsan,19), (lisi,20), (wangwu,21))
Copy the code

The sample

  • Unpack the above tuple list containing students’ names and ages into two lists

Reference code

scala> res1.unzip
res2: (List[String].List[Int]) = (List(zhangsan, lisi, wangwu),List(19.20.21))
Copy the code

13.4.8 Converting Strings

The toString method can return all elements in a List

The sample

  • Define a list containing the following elements: 1,2,3,4
  • Use toString to print the elements of the list

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

scala> println(a.toString)
List(1.2.3.4)
Copy the code

13.4.9 Generating A String

The mkString method, which concatenates elements with delimiters. There are no delimiters by default

The sample

  • Define a list containing the following elements: 1,2,3,4
  • Using mkString, concatenate the elements with a colon

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

scala> a.mkString
res7: String = 1234

scala> a.mkString(":")
res8: String = 1:2:3:4
Copy the code

13.4.10 and set

Union means to take the union of two lists without deduplicating them

The sample

  • Define the first list with the following elements: 1,2,3,4
  • Define a second list with the following elements: 3,4,5,6
  • Use the union operation to get the union of the two lists
  • Use the DISTINCT operation to remove duplicate elements

Reference code

scala> val a1 = List(1.2.3.4)
a1: List[Int] = List(1.2.3.4)

scala> val a2 = List(3.4.5.6)
a2: List[Int] = List(3.4.5.6)

// Merge operation
scala> a1.union(a2)
res17: List[Int] = List(1.2.3.4.3.4.5.6)

// Call distinct
scala> a1.union(a2).distinct
res18: List[Int] = List(1.2.3.4.5.6)
Copy the code

13.4.11 intersection

Intersect denotes the intersection of two lists

The sample

  • Define the first list with the following elements: 1,2,3,4
  • Define a second list with the following elements: 3,4,5,6
  • Get the intersection of the two lists using the INTERSECT operation
scala> val a1 = List(1.2.3.4)
a1: List[Int] = List(1.2.3.4)

scala> val a2 = List(3.4.5.6)
a2: List[Int] = List(3.4.5.6)

scala> a1.intersect(a2)
res19: List[Int] = List(3.4)
Copy the code

13.4.12 difference set

Diff: takes the difference set of two lists, for example, a1.diff(a2), takes the elements of A1 that do not exist in A2

The sample

  • Define the first list with the following elements: 1,2,3,4
  • Define a second list with the following elements: 3,4,5,6
  • Use diff to get the difference set of the two lists
scala> val a1 = List(1.2.3.4)
a1: List[Int] = List(1.2.3.4)

scala> val a2 = List(3.4.5.6)
a2: List[Int] = List(3.4.5.6)

scala> a1.diff(a2)
res24: List[Int] = List(1.2)
Copy the code

14. Set

A Set is a Set that represents no repeating elements. Set has the following properties:

  1. Elements do not repeat
  2. Insertion order is not guaranteed

There are also two types of sets in Scala, immutable and mutable.

14.1 Immutable set

14.1.1 definition

grammar

Create an empty immutable set with the syntax:

val/varThe variable name =Set[type] ()Copy the code

Given an element to create an immutable set, the syntax is:

val/varThe variable name =Set(element1Elements,2Elements,3...)
Copy the code

The sample a

Define an empty immutable set

Reference code

scala> val a = Set[Int]()
a: scala.collection.immutable.Set[Int] = Set(a)Copy the code

Example 2

Define an immutable set that holds the following elements: 1,1,3,2,4,8

Reference code

scala> val a = Set(1.1.3.2.4.8)
a: scala.collection.immutable.Set[Int] = Set(1.2.3.8.4)
Copy the code

14.1.2 Basic Operations

  • Gets the size of the set (size)
  • Traverse the set (Consistent with the traversal number group)
  • Add an element and generate a Set (+)
  • Concatenate two sets to generate a Set (++)
  • Concatenate sets and lists to generate a Set (++)

The sample

  1. Create a set containing the following elements: 1,1,2,3,4,5
  2. Gets the size of the set
  3. Iterate through the set, printing each element
  4. Delete element 1 to generate a new set
  5. Concatenate another set (6, 7, 8)
  6. Concatenate a list (6,7,8, 9)

Reference code

/ / create a set
scala> val a = Set(1.1.2.3.4.5)
a: scala.collection.immutable.Set[Int] = Set(5.1.2.3.4)

// Get the set size
scala> a.size
res0: Int = 5

/ / traverse the set
scala> for(i <- a) println(i)

// Delete an element
scala> a - 1
res5: scala.collection.immutable.Set[Int] = Set(5.2.3.4)

// Splice two sets
scala> a ++ Set(6.7.8)
res2: scala.collection.immutable.Set[Int] = Set(5.1.6.2.7.3.8.4)

// Concatenate sets and lists
scala> a ++ List(6.7.8.9)
res6: scala.collection.immutable.Set[Int] = Set(5.1.6.9.2.7.3.8.4)
Copy the code

14.2 variable set

define

Mutable sets Immutable sets are created the same way, except that a mutable set class is imported in advance.

Manual import, an import scala. Collection. The mutable. Set

The sample

  1. Define a mutable set containing the following elements: 1,2,3, 4
  2. Add element 5 to the mutable set
  3. Removes element 1 from the mutable set

Reference code

scala> val a = Set(1.2.3.4)
a: scala.collection.mutable.Set[Int] = Set(1.2.3.4)                          

// Add elements
scala> a += 5
res25: a.type = Set(1.5.2.3.4)

// Delete elements
scala> a -= 1
res26: a.type = Set(5.2.3.4)
Copy the code

15. The mapping

A Map can be called a mapping. It’s a collection of key-value pairs. In Scala, maps are also divided into immutable maps and mutable maps.

15.1 Immutable Map

define

grammar

val/var map = Map(key -> value, key -> value, key -> value...)// It is recommended for better readability
val/var map = Map(key, value), (key, value), (key, value)...Copy the code

The sample

  1. Define a map that contains the following student name and age data

    "zhangsan".30
"lisi".40
Copy the code

  2. Get zhangsan’s age

Reference code

scala> val map = Map("zhangsan"->30."lisi"->40)
map: scala.collection.immutable.Map[String.Int] = Map(zhangsan -> 30, lisi -> 40)

scala> val map = Map(("zhangsan".30), ("lisi".30))
map: scala.collection.immutable.Map[String.Int] = Map(zhangsan -> 30, lisi -> 30)

// Get value based on key
scala> map("zhangsan")
res10: Int = 30
Copy the code

15.2 variable Map

define

Define syntax consistent with immutable Map. But define variable Map need to manually import import scala. Collection. The mutable. The Map

The sample

  1. Define a map that contains the following student name and age data

    "zhangsan".30
"lisi".40
Copy the code

  2. Example Change the age of Zhangsan to 20

scala> val map = Map("zhangsan"->30."lisi"->40)
map: scala.collection.mutable.Map[String.Int] = Map(lisi -> 40, zhangsan -> 30)

/ / modify the value
scala> map("zhangsan") = 20
Copy the code

15.3 Basic Map Operations

Basic operation

  • Get the value (map(key))
  • Get all keys (map.keys)
  • Get all values (map.values)
  • Iterate over the Map collection
  • getOrElse
  • Increase the key, the value of
  • Remove the key

The sample

  1. Define a map that contains the following student name and age data

    "zhangsan".30
"lisi".40
Copy the code

  2. Get zhangsan’s age

  3. Get all student names

  4. Get all student ages

  5. Print all student names and ages

  6. Gets the age of wangwu, or -1 if wangwu does not exist

  7. Add a new student: Wangwu, 35

  8. Remove LISi from mutable mapping

Reference code

scala> val map = Map("zhangsan"->30."lisi"->40)
map: scala.collection.mutable.Map[String.Int] = Map(lisi -> 40, zhangsan -> 30)

// Get zhagnsan's age
scala> map("zhangsan")
res10: Int = 30

// Get all student names
scala> map.keys
res13: 可迭代[String] = Set(lisi, zhangsan)

// Get all student ages
scala> map.values
res14: 可迭代[Int] = HashMap(40.30)

// Print all student names and ages
scala> for((x,y) <- map) println(s"$x $y")
lisi 40
zhangsan 30

// Get the age of wangwu, or -1 if wangwu does not exist
scala> map.getOrElse("wangwu".- 1)
res17: Int = - 1

// Add a new student: Wangwu, 35
scala> map + "wangwu"->35
res22: scala.collection.mutable.Map[String.Int] = Map(lisi -> 40, zhangsan -> 30, wangwu -> 35)

// Lisi is removed from the mutable map
scala> map - "lisi"
res23: scala.collection.mutable.Map[String.Int] = Map(zhangsan -> 30)
Copy the code

16. An iterator

Scala provides an iterator for each type of collection to access iteratively

Iterators are used to traverse the collection

  • useiteratorThe getIterator () method gets an iterator from the collection
  • Two basic operations of iterators
    • HasNext — Queries whether the next element is present in the container
    • Next — Returns the next element of the iterator, and if not, throws NoSuchElementException
  • Each iterator is stateful
    • Leave it in place of the last element after iteration
    • A second use throws a NoSuchElementException
  • You can use while or for to return elements one by one

The sample

  1. Define a list containing the following elements: 1,2,3,4,5
  2. Use a while loop and iterator to iterate over and print the list

Reference code

scala> val ite = a.iterator
ite: Iterator[Int] = non-empty iterator

scala> while(ite.hasNext) {
     | println(ite.next)
     | }
Copy the code

The sample

  1. Define a list containing the following elements: 1,2,3,4,5
  2. Iterate over and print the list using a for expression and iterator

Reference code

scala> val a = List(1.2.3.4.5)
a: List[Int] = List(1.2.3.4.5)

scala> for(i <- a) println(i)
Copy the code

17. Functional programming

Much of our future business code using Spark/Flink will use functional programming. The following operations are the focus of the study.

  • Traversal (foreach)
  • Mapping (map)
  • Mapping flattening (flatmap)
  • Filtering (filter)
  • Does it exist (exists)
  • Sorting (sorted,sortBy,sortWith)
  • Groups (groupBy)
  • Aggregate computation (reduce)
  • Fold (fold)

17.1 traversing | foreach

Earlier, you learned how to use a for expression to traverse a collection. We’ll take a look at Scala’s functional programming, using the Foreach method for traversal and iteration. It makes code much cleaner.

The method signature

foreach(f: (A) ⇒ Unit) :Unit
Copy the code

instructions

foreach API instructions
parameter F: (A) ⇒ Unit Receives a function object

The function takes an element of the collection as an input argument and returns a null value
The return value Unit empty

Foreach executes the procedure

The sample

There is a list of the following elements: 1,2,3,4. Print each element iterated through using the foreach method

Reference code

// Define a list
scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

// Print iteratively
scala> a.foreach((x:Int)=>println(x))
Copy the code

17.2 Use type inference to simplify function definitions

The above case function definition is a bit verbose, so we can write it more succinctly. Because foreach is used to iterate over a list, each element type in the list is determined

  • Scala can automatically infer the type of each element parameter in a collection
  • When you create a function, you can omit the type of its argument list

The sample

  1. There is a list of the following elements: 1,2,3,4. Print each element iterated through using the foreach method
  2. Use type inference to simplify function definitions

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

// Omit the parameter type
scala> a.foreach(x=>println(x))
Copy the code

17.3 Use underscores to simplify function definitions

Underlining can be used to simplify function definitions when function arguments occur only once in the function body and the function body has no nested calls

The sample

  1. There is a list of the following elements: 1,2,3,4. Print each element iterated through using the foreach method
  2. Use underscores to simplify function definitions

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

a.foreach(println(_))
Copy the code
  • If the method argument is a function, the Scala compiler automatically wraps the code into a function if an underscore is present
  • Argument lists are also handled automatically by the Scala compiler

17.4 mapping | map

The collection mapping operation is the operation we will use most in writing Spark/Flink in the future, and it is something we must master. When computing data, it is a process of converting one data type to another.

The map method takes a function, applies it to each element, and returns a new list

usage

The method signature


def map[B](f: (A) ⇒ B) :TraversableOnce[B]
Copy the code

Method resolution

The map method API instructions
The generic [B] Specifies the collection generic that the map method eventually returns
parameter F: (A) ⇒ B Pass in a function object

This function takes A type A (the list element to be converted) and returns A value of type B
The return value TraversableOnce[B] Set of type B

Map method parsing

Case a

  1. Create a list of elements 1,2,3,4

  2. Add 1 to each element in the List

Reference code

scala> a.map(x=>x+1)
res4: List[Int] = List(2.3.4.5)
Copy the code

Case 2

  1. Create a list of elements 1,2,3,4

  2. Use underscores to define functions, incrementing each element in the List by 1

Reference code

scala> val a = List(1.2.3.4)
a: List[Int] = List(1.2.3.4)

scala> a.map(_ + 1)
Copy the code

17.5 flat map | flatMap

Flat mapping is also an operation that will be used a lot in the future and must be mastered.

define

FlatMap can be understood as map first and then flatten

  • A map converts the elements of a List into a List
  • Flatten Flatten the entire list

The method signature

def flatMap[B](f: (A) ⇒ GenTraversableOnce[B) :TraversableOnce[B]
Copy the code

Method resolution

Flatmap method API instructions
The generic [B] The type of collection element to be converted
parameter F: (A) ⇒ GenTraversableOnce [B] Pass in a function object

The arguments to the function are elements of the collection

The return value of the function is a collection
The return value TraversableOnce[B] Set of type B

case

Case description

  1. There is a list of several lines of text: “Hadoop Hive Spark flink flume”,” Kudu hbase SQoop Storm”
  2. Get each word in the line of text and place each word in the list

Thought analysis

steps

  1. Use map to split lines of text into arrays
  2. And then we flatten the array

Reference code

// Define a text row list
scala> val a = List("hadoop hive spark flink flume"."kudu hbase sqoop storm")
a: List[String] = List(hadoop hive spark flink flume, kudu hbase sqoop storm)

// Use map to convert lines of text into arrays of words
scala> a.map(x=>x.split(""))
res5: List[Array[String]] = List(Array(hadoop, hive, spark, flink, flume), Array(kudu, hbase, sqoop, storm))

// Flatten the array
scala> a.map(x=>x.split("")).flatten
res6: List[String] = List(hadoop, hive, spark, flink, flume, kudu, hbase, sqoop, storm)
Copy the code

Use flatMap to simplify operations

Reference code

scala>  val a = List("hadoop hive spark flink flume"."kudu hbase sqoop storm")
a: List[String] = List(hadoop hive spark flink flume, kudu hbase sqoop storm)

scala> a.flatMap(_.split(""))
res7: List[String] = List(hadoop, hive, spark, flink, flume, kudu, hbase, sqoop, storm)
Copy the code

17.6 filter | filter

Filter elements that meet certain conditions

define

The method signature

def filter(p: (A) ⇒ Boolean) :TraversableOnce[A]
Copy the code

Method resolution

The filter method API instructions
parameter P: (A) ⇒ Boolean Pass in a function object

Accepts a parameter of the collection type

Returns a Boolean type, true if the condition is met, false if not
The return value TraversableOnce[A] The list of

case

  1. There is a list of numbers with the elements: 1,2,3,4,5,6,7,8,9

  2. Filter out all even numbers

Reference code


scala> List(1.2.3.4.5.6.7.8.9).filter(_ % 2= =0)
res8: List[Int] = List(2.4.6.8)
Copy the code

17.7 the sorting

In Scala collections, there are several ways to sort

  • Sorted by default
  • SortBy specifies field sorting
  • SortWith Custom sort

17.7.1 default sort | sorted

The sample

  1. Define a list containing the following elements: 3, 1, 2, 9, 7
  2. Sort lists in ascending order

Reference code

scala> List(3.1.2.9.7).sorted
res16: List[Int] = List(1.2.3.7.9)
Copy the code

17.7.2 specified field sorting | sortBy

After converting according to the function passed in, sort


def sortBy[B](f: (A) ⇒ B) :List[A]
Copy the code

Method resolution

SortBy method API instructions
The generic [B] By what type
parameter F: (A) ⇒ B Passing function object

Accepts an element parameter of a collection type

Returns elements of type B for sorting
The return value List[A] Returns the sorted list


The sample

  1. There is a list of the following lines: “01 Hadoop “, “02 Flume “, “03 Hive “, “04 Spark”
  2. Please sort the words alphabetically

Reference code

scala> val a = List("01 hadoop"."02 flume"."03 hive"."04 spark")
a: List[String] = List(01 hadoop, 02 flume, 03 hive, 04 spark)

// Get the word field
scala> a.sortBy(_.split("") (1))
res8: List[String] = List(02 flume, 01 hadoop, 03 hive, 04 spark)
Copy the code

17.7.3 custom sorting | sortWith

Custom sort, custom sort according to a function


The method signature

def sortWith(lt: (A.A) ⇒ Boolean) :List[A]
Copy the code

Method resolution

SortWith method API instructions
parameter Lt: (A, A) Tail Boolean Pass in a function object to compare the size

Accepts element parameters of two collection types

Returns two element sizes, less than true and greater than false
The return value List[A] Returns the sorted list


The sample

  1. There is a list containing the following elements: 2,3,1,6,4,5
  2. Use sortWith to sort the list in descending order

Reference code

scala> val a = List(2.3.1.6.4.5)
a: List[Int] = List(2.3.1.6.4.5)

scala> a.sortWith((x,y) => if(x<y)true else false)
res15: List[Int] = List(1.2.3.4.5.6)

scala> res15.reverse
res18: List[Int] = List(6.5.4.3.2.1)
Copy the code

Use underscores to abbreviate the above examples

Reference code

scala> val a = List(2.3.1.6.4.5)
a: List[Int] = List(2.3.1.6.4.5)

Function arguments appear only once in a function and can be replaced with underscores
scala> a.sortWith(_ < _).reverse
res19: List[Int] = List(6.5.4.3.2.1)
Copy the code

17.8 grouping | groupBy

. If we want to analyze the data in groups, we need to use the grouping method

define

GroupBy groups the list into groups by function

The method signature

def groupBy[K](f: (A) ⇒ K) :Map[K.List[A]]
Copy the code

Method resolution

GroupBy method API instructions
The generic [K] The type of the grouping field
parameter F: (A) ⇒ K Pass in a function object

Accepts a parameter of the collection element type

Return a key of type K that will be used to group the same key into a group
The return value Map[K, List[A]] Return a map where K is the group field and List is the group of data corresponding to the group field

GroupBy Performs process analysis

The sample

  1. There is a list with the names and genders of the students:

    "Zhang"."Male"
"Bill"."Female"
"Fifty"."Male"
Copy the code

  2. Please group them by gender and count the number of students of different genders

steps

  1. Define a tuple list to hold the student name and gender
  2. Group by gender
  3. List((” male “-> 2), (” female” -> 1))

Reference code

scala> val a = List("Zhang"->"Male"."Bill"->"Female"."Fifty"->"Male")
a: List[(String.String)] = List(Zhang SAN, male) li Si, female) Wang Wu, male)// Group by gender
scala> a.groupBy(_._2)
res0: scala.collection.immutable.Map[String.List[(String.String=)]]Map(male - >List(Zhang SAN, male), (Wang Wu, male), female ->List(Li Si, female))// Convert the grouped mapping to a list of gender/number tuples
scala> res0.map(x => x._1 -> x._2.size)
res3: scala.collection.immutable.Map[String.Int] = Map(male - >2Female - >1)
Copy the code

17.9 Aggregation Operations

An aggregation operation that merges data from a list into one. This operation is often used in statistical analysis

17.9.1 polymerization | reduce

Reduce means to pass the list into a function for aggregate calculation

define

The method signature

def reduce[A1> :A](op: (A1.A1) ⇒ A1) :A1
Copy the code

Method resolution

The reduce method API instructions
The generic [A1 >: A] A1 must be the parent of the collection element type
parameter Op: (A1, A1 Pass in a function object for continuous aggregation operations

The first A1 type argument is: the currently aggregated variable

The second A1 type argument is: the element currently being aggregated
The return value A1 The list is finally aggregated into one element

Reduce Execution process analysis

[!NOTE]

  • Reduce and reduceLeft have the same effect, indicating calculation from left to right

  • ReduceRight means calculate from right to left

case

  1. Define a list containing the following elements: 1,2,3,4,5,6,7,8,9,10
  2. Use reduce to calculate the sum of all elements

Reference code

scala> val a = List(1.2.3.4.5.6.7.8.9.10)
a: List[Int] = List(1.2.3.4.5.6.7.8.9.10)

scala> a.reduce((x,y) => x + y)
res5: Int = 55

// The first underscore represents the first parameter, which is the historical aggregate data result
// The second underscore represents the second parameter, which is the current data element to aggregate
scala> a.reduce(_ + _)
res53: Int = 55

// As with reduce, it is calculated from left to right
scala> a.reduceLeft(_ + _)
res0: Int = 55

// Aggregate from right to left
scala> a.reduceRight(_ + _)
res1: Int = 55
Copy the code

| 17.9.2 folding a fold

Fold is similar to Reduce, but with an additional parameter that specifies an initial value

define

The method signature

def fold[A1> :A](z: A1)(op: (A1.A1) ⇒ A1) :A1
Copy the code

Method resolution

The reduce method API instructions
The generic [A1 >: A] A1 must be the parent of the collection element type
Parameter 1 z: A1 The initial value
Parameter 2 Op: (A1, A1 Pass in a function object for continuous folding

The first A1 type argument is: the currently collapsed variable

The second A1 type argument is: the element currently being collapsed
The return value A1 The list eventually collapses into a single element

[!NOTE]

  • Fold and foldLet are calculated from left to right

  • FoldRight is calculated from right to left

case

  1. Define a list containing the following elements: 1,2,3,4,5,6,7,8,9,10
  2. The fold method is used to calculate the sum of all elements

Reference code

scala> val a = List(1.2.3.4.5.6.7.8.9.10)
a: List[Int] = List(1.2.3.4.5.6.7.8.9.10)

scala> a.fold(0)(_ + _)
res4: Int = 155
Copy the code

| | | return value Map [K, List [A]] | returns A Map, K for the grouping field, the List for the grouping field | corresponding to A set of data

GroupBy Performs process analysis

imG-9EEPG3HL-1625201993570

The sample

  1. There is a list with the names and genders of the students:

    "Zhang"."Male"
"Bill"."Female"
"Fifty"."Male"
Copy the code

  2. Please group them by gender and count the number of students of different genders

steps

  1. Define a tuple list to hold the student name and gender
  2. Group by gender
  3. List((” male “-> 2), (” female” -> 1))

Reference code

scala> val a = List("Zhang"->"Male"."Bill"->"Female"."Fifty"->"Male")
a: List[(String.String)] = List(Zhang SAN, male) li Si, female) Wang Wu, male)// Group by gender
scala> a.groupBy(_._2)
res0: scala.collection.immutable.Map[String.List[(String.String=)]]Map(male - >List(Zhang SAN, male), (Wang Wu, male), female ->List(Li Si, female))// Convert the grouped mapping to a list of gender/number tuples
scala> res0.map(x => x._1 -> x._2.size)
res3: scala.collection.immutable.Map[String.Int] = Map(male - >2Female - >1)
Copy the code

17.9 Aggregation Operations

An aggregation operation that merges data from a list into one. This operation is often used in statistical analysis

17.9.1 polymerization | reduce

Reduce means to pass the list into a function for aggregate calculation

define

The method signature

def reduce[A1> :A](op: (A1.A1) ⇒ A1) :A1
Copy the code

Method resolution

The reduce method API instructions
The generic [A1 >: A] A1 must be the parent of the collection element type
parameter Op: (A1, A1 Pass in a function object for continuous aggregation operations

The first A1 type argument is: the currently aggregated variable

The second A1 type argument is: the element currently being aggregated
The return value A1 The list is finally aggregated into one element

Reduce Execution process analysis

img-W3XVLJ6C-1625201993571

[!NOTE]

  • Reduce and reduceLeft have the same effect, indicating calculation from left to right

  • ReduceRight means calculate from right to left

case

  1. Define a list containing the following elements: 1,2,3,4,5,6,7,8,9,10
  2. Use reduce to calculate the sum of all elements

Reference code

scala> val a = List(1.2.3.4.5.6.7.8.9.10)
a: List[Int] = List(1.2.3.4.5.6.7.8.9.10)

scala> a.reduce((x,y) => x + y)
res5: Int = 55

// The first underscore represents the first parameter, which is the historical aggregate data result
// The second underscore represents the second parameter, which is the current data element to aggregate
scala> a.reduce(_ + _)
res53: Int = 55

// As with reduce, it is calculated from left to right
scala> a.reduceLeft(_ + _)
res0: Int = 55

// Aggregate from right to left
scala> a.reduceRight(_ + _)
res1: Int = 55
Copy the code

| 17.9.2 folding a fold

Fold is similar to Reduce, but with an additional parameter that specifies an initial value

define

The method signature

def fold[A1> :A](z: A1)(op: (A1.A1) ⇒ A1) :A1
Copy the code

Method resolution

The reduce method API instructions
The generic [A1 >: A] A1 must be the parent of the collection element type
Parameter 1 z: A1 The initial value
Parameter 2 Op: (A1, A1 Pass in a function object for continuous folding

The first A1 type argument is: the currently collapsed variable

The second A1 type argument is: the element currently being collapsed
The return value A1 The list eventually collapses into a single element

[!NOTE]

  • Fold and foldLet are calculated from left to right

  • FoldRight is calculated from right to left

case

  1. Define a list containing the following elements: 1,2,3,4,5,6,7,8,9,10
  2. The fold method is used to calculate the sum of all elements

Reference code

scala> val a = List(1.2.3.4.5.6.7.8.9.10)
a: List[Int] = List(1.2.3.4.5.6.7.8.9.10)

scala> a.fold(0)(_ + _)
res4: Int = 155
Copy the code