Have you ever wondered how Web editors like Visual Studio (Online), CodeSandbox or Snack work? Or do you want to make a custom Web or desktop editor and don’t know how to get started? In this article, I’ll show you how the Web editor works, and we’ll create a custom language. The language editor we will build is simple. It declares a list of ToDos and then applies some predefined instructions to them. I call this language TodoLang. Here are some examples of the language:
ADD TODO "Make the world a better place"
ADD TODO "read daily"
ADD TODO "Exercise"
COMPLETE TODO "Learn & share"
Copy the codeWe simply add some TODOs using the following command:
ADD TODO "TODO_TEXT";
Copy the codeWe can use the COMPLETE TODO “todo_text” to indicate the completed TODO, so that the output that explains the code tells us about the remaining toDos and the toDos that have been completed so far. This is a simple language I invented for the purpose of this article. It may seem useless, but it contains everything I need to cover in this article.
We will enable the editor to support the following features:
- Automatic formatting
- Done automatically
- Syntax highlighting
- Syntax and semantic validation
Note: The editor supports only one code or file edit at a time. It does not support multiple files or code editing.
TodoLang Semantic rules Here are some semantics I will use for semantic validation of TodoLang code:
- If TODO is defined using the ADD TODO specification, we can ADD it again.
- In TODO applications, the COMPLETE directive should not be declared before adding TODO.
I’ll return to these semantic rules later in this article. Before diving into the code, let’s start with the general architecture of a Web editor, or any regular editor.
As you can see from the pattern above, there are usually two threads in any editor. One is responsible for UI content, such as waiting for the user to enter some code or perform some action. The other thread accepts the changes made by the user and performs heavy computations, including code parsing and other compilation work.
For each change in the editor, which could be each character entered by the user or until the user stops typing for 2 seconds, a message is sent to the Language Service Worker to perform some action. The Worker itself will respond with a message containing the result. For example, when a user enters some code and wants to format it (click Shift + Alt + F), the Worker receives a message containing the action “format” and the code to format. This should be done asynchronously for a good user experience.
The language service, on the other hand, parses the code, generates an abstract syntax tree (AST), looks for any possible syntax or lexical errors, uses the AST to find any semantic errors, formats the code, and so on.
We can use a new high-level way to handle the language service through LSP protocol, but in this example, the language service and the editor will be in the same process (that is, the browser) without any back-end processing. If you want to support your language in other editors (such as VS Code, Sublime, or Eclipse) without breaking the sweat, it’s best to separate language services from worker. Using LSP will enable you to make plug-ins for other editors to support your language. View the LSP page for more information.
The editor provides an interface that allows the user to enter code and perform some actions. As the user enters content, the editor should consult the configuration list to highlight code markers (keywords, types, and so on). This can be done through the language service, but for our example, we’ll do it in the editor. We’ll see how to do that later.
Monaco provides an API Monaco. Editor. CreateWebWorker to use the built-in ES6 Proxies to create a proxy Web worker. Get the proxy object (language service) using the getProxy method. To access any service in the language service worker, we will use this proxy object to invoke any method. All methods will return a Promise object.
Check out Comlink, a small library developed by Google to make interacting with Web workers enjoyable using ES6 Proxies.
Without further ado, let’s start writing some code.
What will we use?
React
View correlation.
ANTLR
According to the definition on its website, “ANTLR (another language recognition tool) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It is widely used to build languages, tools, and frameworks. ANTLR syntactically generates a parser that builds and traverses the parse tree.” ANTLR supports many languages as targets, which means it can generate parsers for Java, C#, and other languages. For this project, I’ll use ANTLR4TS, which is the Node.js version of ANTLR that generates a lexical parser and parser in TypeScript.
ANTLR uses special syntax to declare language syntax, which is usually placed in *.g4 files. It allows you to define lexical analyzer and parser rules in a single combined syntax file. In this repository, you will find syntax files for many well-known languages.
This language uses a language called the symbol Backus normal form (BNF) to describe grammar.
TodoLang grammar
This is our simplified syntax for TodoLang. It declares a root rule todoExpressions for TodoLang, which contains a list of expressions. Expressions in TodoLang can be addExpression or completeExpression. As with regular expressions, the asterisk (*) indicates that the expression may occur zero or more times.
Each expression begins with a terminal keyword (add, _todo, or _complete) and a string identifying TODO (“… “). ).
grammar TodoLangGrammar;
todoExpressions : (addExpression)* (completeExpression)*;
addExpression : ADD TODO STRING EOL;
completeExpression : COMPLETE TODO STRING EOL;
ADD : 'ADD';
TODO : 'TODO';
COMPLETE: 'COMPLETE';
STRING: '"' ~ ["]* '"';
EOL: [\r\n] +;
WS: [ \t] -> skip;
Copy the codeMonaco-Editor
The Monaco Editor is a Code Editor that supports VS Code. This is a JavaScript library that provides apis for syntax highlighting, auto-completion, and more.
The development tools
TypeScript, webpack, [webpack-dev-server](https://webpack.js.org/configuration/dev-server/), webpack-cli, [HtmlWebpackPlugin](https://github.com/jantimon/html-webpack-plugin), and [ts-loader](https://www.npmjs.com/package/ts-loader).
So let’s start by getting the project started.
Start a new TypeScript project
To that end, let’s launch our project:
npm init
Copy the codeCreate a tsconfig.json file with the following minimum content:
{
"compilerOptions": {
"target": "es6"."module": "commonjs"."allowJs": true."jsx": "react"}}Copy the codeAdd the webpack.config.js configuration file to webpack:
const path = require('path');
const htmlWebpackPlugin = require('html-webpack-plugin');
module.exports = {
mode: 'development'.entry: {
app: './src/index.tsx'
},
output: {
filename: 'bundle.[hash].js'.path: path.resolve(__dirname, 'dist')},resolve: {
extensions: ['.ts'.'.tsx'.'.js'.'.jsx']},module: {
rules: [{test: /.tsx? /,
loader: 'ts-loader'}},plugins: [
new htmlWebpackPlugin({
template: './src/index.html'}})]Copy the codeAdd dependencies for React and TypeScrip t:
npm add react react-dom
npm add -D typescript @types/react @types/react-dom ts-loader html-webpack-plugin webpack webpack-cli webpack-dev-server
Copy the codeCreate a SRC directory in the root path and create a div with an ID of Container in index.ts and index. HTML.
Add the Monaco Editor component
If you target existing languages like TypeScript, HTML, or Java, you don’t have to reinvent the wheel. The Monaco Editor and Monaco Languages support most of these Languages.
For our example, we will use the core version of the Monaco Editor called monaco-editor-core.
Add a package:
npm add monaco-editor-core
Copy the codeWe also need some CSS loaders because Monaco uses them internally:
npm add -D style-loader css-loader
Copy the codeAdd these rules to the Module property in the WebPack configuration:
{
test: /\.css$/,
use: ['style-loader', 'css-loader']
}
Copy the codeFinally, add CSS to the parsed extension:
extensions: ['.ts'.'.tsx'.'.js'.'.jsx'.'.css']
Copy the codeNow we are ready to create the editor component. Create a React component (we’ll call it Editor) and return an element with the ref attribute so that we can use its reference to have the Monaco API inject the Editor into it.
To create the Monaco editor, we need to call monace.editor.create. You pass in some parameters like Editor, languageId, theme, and so on. See the documentation for more details.
Add a file that will contain all of the following language configurations SRC /todo-lang:
export const languageID = 'todoLang' ;
Copy the codeAdd the Editor component to SRC/Components:
import * as React from 'react';
import * as monaco from 'monaco-editor-core';
interface IEditorPorps {
language: string;
}
const Editor: React.FC<IEditorPorps> = (props: IEditorPorps) = > {
let divNode;
const assignRef = React.useCallback((node) = > {
// On mount get the ref of the div and assign it the divNodedivNode = node; } []); React.useEffect(() = > {
if (divNode) {
const editor = monaco.editor.create(divNode, {
language: props.language,
minimap: { enabled: false },
autoIndent: true
});
}
}, [assignRef])
return <div ref={assignRef} style={{ height: '90vh' }}></div>;
}
export { Editor };
Copy the codeBasically, we use the callback hook at mount time to get a reference to the div, so we can pass it to the create function. Now you can add editor components to the application and add styles as needed.
Register our language using the Monaco API
In order to make the Monaco Editor support our definition language (for example, when we create the Editor, we specify the language ID), we need to use the API Monaco. Languages. Register to register. Let’s create a file setup named SRC /todo-lang in. We also need to realize the Monaco. Languages. OnLanguage a callback, in the language configuration calls the callback when ready. (We’ll use this callback later to register the language provider for syntax highlighting, auto-completion, formatting, and so on) :
import * as monaco from "monaco-editor-core";
import { languageExtensionPoint, languageID } from "./config";
export function setupLanguage() {
monaco.languages.register(languageExtensionPoint);
monaco.languages.onLanguage(languageID, () = >{}); }Copy the codeNow, call setupLanguage in index.tsx.
Add Worker for Monaco
So far, if you run the project and open it in a browser, you receive an error message about the Web Worker:
Could not create web worker(s). Falling back to loading web worker code in main thread, which might cause UI freezes. Please see https://github.com/Microsoft/monaco-editor#faq
You must define a function MonacoEnvironment.getWorkerUrl or MonacoEnvironment.getWorker
Copy the codeLanguage Services creates Web workers to compute heavy work outside of UI threads. They require almost no overhead, no worry, just normal use. The Monaco Editor uses a Web Worker, which I believe is used for highlighting and performing other behaviors. We will create another worker to handle the language service. The First step is to package Monaco’s Editor Web worker via Webpack. Add the worker to the entry:
entry: {
app: './src/index.tsx'."editor.worker": 'monaco-editor-core/esm/vs/editor/editor.worker.js'
},
Copy the codeChange the global variable for webpack’s output to self, which is the content of the WebPack configuration file so far:
const path = require('path');
const htmlWebpackPlugin = require('html-webpack-plugin');
module.exports = {
mode: 'development'.entry: {
app: './src/index.tsx'."editor.worker": 'monaco-editor-core/esm/vs/editor/editor.worker.js'
},
output: {
globalObject: 'self'.filename: (chunkData) = > {
switch (chunkData.chunk.name) {
case 'editor.worker':
return 'editor.worker.js';
default:
return 'bundle.[hash].js'; }},path: path.resolve(__dirname, 'dist')},resolve: {
extensions: ['.ts'.'.tsx'.'.js'.'.jsx'.'.css']},module: {
rules: [{test: /\.tsx? /,
loader: 'ts-loader'
},
{
test: /\.css/,
use: ['style-loader'.'css-loader']]}},plugins: [
new htmlWebpackPlugin({
template: './src/index.html'}})]Copy the codeAs you can see from the above error, Monaco calls the method getWorkerUrl from the global variable MonacoEnvironment. Go to setupLanguage and add the following:
import * as monaco from "monaco-editor-core";
import { languageExtensionPoint, languageID } from "./config";
export function setupLanguage() {(window as any).MonacoEnvironment = {
getWorkerUrl: function (moduleId, label) {
return './editor.worker.js';
}
}
monaco.languages.register(languageExtensionPoint);
monaco.languages.onLanguage(languageID, () = >{}); }Copy the codeThis will tell Monaco how to find the worker, and we will add a custom Language service worker. Running the application, you should see an editor that does not yet support any functionality:
Add syntax highlighting and language configuration
In this section, we will add some keyword highlighting. The Monaco Editor uses the Monarch library, which enables us to create declarative syntax highlighting displays using JSON. If you want to learn more about this syntax, check out its documentation. This is an example of a Java configuration for syntax highlighting, code folding, etc. Create config.ts in SRC /todo-lang. We will use the Monaco API configuration TodoLang highlighting and token generator: Monaco. Languages. SetMonarchTokensProvider. It takes two parameters, That is, the language ID and type The configuration of the [IMonarchLanguage] (https://microsoft.github.io/monaco-editor/api/interfaces/monaco.languages.imonarchlanguage.html). Here’s the configuration for TodoLang:
import * as monaco from "monaco-editor-core";
import IRichLanguageConfiguration = monaco.languages.LanguageConfiguration;
import ILanguage = monaco.languages.IMonarchLanguage;
export const monarchLanguage = <ILanguage>{
// Set defaultToken to invalid to see what you do not tokenize yet
defaultToken: 'invalid'.keywords: [
'COMPLETE'.'ADD',].typeKeywords: ['TODO'].escapes: / \ \? : [abfnrtv \ \ "'] | x [0-9 a - Fa - f] {1, 4} | u [0-9 a - Fa - f] {4} | u [0-9 a - Fa - f] {8}) /.// The main tokenizer for our languages
tokenizer: {
root: [
// identifiers and keywords
[/[a-zA-Z_$][\w$]*/, {
cases: {
'@keywords': { token: 'keyword' },
'@typeKeywords': { token: 'type' },
'@default': 'identifier'}}].// whitespace
{ include: '@whitespace' },
// strings for todos
[/ "([^ | \ \. \ \]") * $/.'string.invalid'].// non-teminated string
[/ /".'string'.'@string']],whitespace: [[/[ \t\r\n]+/.' ']],string: [[+ / / [^ \ \ "].'string'],
[/@escapes/.'string.escape'],
[/ \ \. /.'string.escape.invalid'],
[/ /".'string'.'@pop']]}},Copy the codeWe basically specify a CSS class or token name for each key in TodoLang. For example, for the keywords COMPLETE and ADD, we also configure Monaco to color strings by providing them with a class of type CSS, which is predefined by Monaco. You can use [defineTheme] (https://microsoft.github.io/monaco-editor/api/modules/monaco.editor.html#definetheme) and create a new apis The CSS class calls setTheme to override the theme.
To tell Monaco to consider this configuration, Use the setup function call in the onLanguage callback function [monaco.languages.setMonarchTokensProvider](https://microsoft.github.io/monaco-editor/api/modules/monaco.languages.html# Setmonarchtokensprovider) and configure it as the second parameter:
import * as monaco from "monaco-editor-core";
import { languageExtensionPoint, languageID } from "./config";
import { monarchLanguage } from "./TodoLang";
export function setupLanguage() {(window as any).MonacoEnvironment = {
getWorkerUrl: function (moduleId, label) {
return './editor.worker.js';
}
}
monaco.languages.register(languageExtensionPoint);
monaco.languages.onLanguage(languageID, () = > {
monaco.languages.setMonarchTokensProvider(languageID, monarchLanguage);
});
}
Copy the codeRun the application. The editor should now support syntax highlighting.
In the next part of this article, I’ll introduce language services. I’ll use ANTLR to generate the TodoLang lexical analyzer and parser, and use the AST provided by the parser to implement most of the editor’s functions. Then, we’ll see how to create workers to provide auto-complete language services.