Golang version: 1.13.1


Lexical analysis

The lexical analysis stage is the first stage of the compilation process. The task at this stage is to read the source program character by character from left to right, that is, to scan the stream of characters that make up the source program and identify words (also known as word symbols or symbols) according to word-formation rules. The lexical analyzer performs this task. Lexical analyzer programs can be generated automatically using tools such as LEX.

scanner

Begin to implement the bootstrap go in (1.5), while the first phase of the lexical analysis by the CMD/compile/internal/syntax/scanner.

The realization of the scanner is simple, the main logic in the next, is to parse is a string, and then through the one big switch into tokens (tokens in the CMD/compile/internal/syntax/tokens). You can observe this process using TestScanner, the scanner test.



struct

First, let’s look at some definitions of the scanner structure

Definition of scanner struct



Source struct definition



TestScanner

As the previous test cases parsed the parser file, which was a bit large and not intuitive, a new AAA. go file was created to parse it




Test results:

=== RUN   TestScanner
1 package
1 name => syntax
1 ;
3 import
3 literal => "fmt"3; 5 func 5 name => aaaa 5 ( 5 ) 5 { 6 name => a 6 := 6 literal => 123 6 ; 7 name => fmt 7 . 7 name => Println 7 ( 7 name => a 7 ) 7 ; 8}; -- PASS: TestScanner (0.00s) PASS Debugger finished withexit code 0
Copy the code

Operation process

As you can see from the above test file, the first step of the scanner is the init data, init is a simple setting of the default value, I won’t go into the details here…

The main process is the logic of next. The execution of scanner is driven by Next. A Next is a token.



Ident string processing flow




conclusion

The main flow of scanner is as follows:



Lexical parsing is a relatively simple process, but very tedious, scanner+source 1000 lines of code there are some special cases of parsing, such as file names

Next step: Go Parser