The previous two articles focused on the result of a template file processed by Web.py’s template system: the __template__() function. This article focuses on how the template file becomes the __template__() function.
Render and frender
In general, the Render class is more common, which handles the entire directory of templates and also supports caching and nested templates. However, these are not related to the template itself, the implementation and purpose of this class will be explained later. Here we use the frender() function:
def frender(path, **keywords):
"""Creates a template from the given file path.
"""
return Template(open(path).read(), filename=path, **keywords)
This function is fairly simple. It does just one click: it reads the contents of the Template file, hands it to the Template class, and returns an instance of the Template class. You can also see from this that the entire Template parsing is only related to the Template class, and that Frender is there to do the chores.
The Template class
The effect of the Template instance
When we create a Template class instance T based on the contents of a Template, we can call that instance, which is equivalent to calling the Template’s corresponding __template__() function. The result is an instance of TemplateResult.
In [7]: t = web.template.frender("templates/hello.html")
# coding: utf-8
def __template__ (name):
__lineoffset__ = -4
loop = ForLoop()
self = TemplateResult(); extend_ = self.extend
extend_([u'hello, ', escape_(name, True), u'\n'])
return self
In [8]: print t("xxxx")
hello, xxxx
In [9]: print type(t("xxxx"))
<class 'web.template.TemplateResult'>
The Template instantiation process
The Template instantiation process is the substantive step of converting a Template to HTML content, but it is a more complex process. In a nutshell, however, this procedure is not all that different from the steps in the __init__() function of the Template.
__init__() def __init__(self, text, filename='< Template >', filter=None, globals=None, builtins=None, extensions=None): self.extensions = extensions or [] text = Template.normalize_text(text) code = self.compile_template(text, filename) _, ext = os.path.splitext(filename) filter = filter or self.FILTERS.get(ext, None) self.content_type = self.CONTENT_TYPES.get(ext, None) if globals is None: globals = self.globals if builtins is None: builtins = TEMPLATE_BUILTINS BaseTemplate.__init__(self, code=code, filename=filename, filter=filter, globals=globals, builtins=builtins)First of all, we will ignore all parameters except text, and then we will look at the processing of text (the content of the template). The whole process is summarized as follows:
-
Normalize_text (text) : text = template.normalize_text (text); delete the BOM string \n; replace \$with $$;
-
Compiling the template results in compiled Python bytecode code: code = self.pile_template (text, filename), which is the __template__() function mentioned earlier.
-
Call the __init__() function of the parent class (BaseTemplate) : create the __template__() function execution environment, and implement the callable function.
Other unspecified code can be ignored for the time being and will not affect your understanding of the instantiation process of the Template. As you can see from the above steps, the Template instantiation process consists of two main steps: generating and compiling the __template__() function, and creating the __template__() function execution environment.
Generates the code for the __template__() function
This is the longest and most complex part of the template generation process and applies to Python’s token analysis and dynamic compilation capabilities. Remember in the first post, when we set up the lab environment, we changed the web.py source code to insert a print statement in one place? Yes, this is the compile_template() function, so let’s see how it generates the code for the __template__() function.
def compile_template(self, template_string, filename): code = Template.generate_code(template_string, filename, parser=self.create_parser()) def get_source_line(filename, lineno): try: lines = open(filename).read().splitlines() return lines[lineno] except: Return None print code # return None print code #Obviously, the first line of the call generates the code for the __template__() function. Let’s move on:
def generate_code(text, filename, parser=None):
# parse the text
parser = parser or Parser()
rootnode = parser.parse(text, filename)
# generate python code from the parse tree
code = rootnode.emit(indent="").strip()
return safestr(code)
generate_code = staticmethod(generate_code)
def create_parser(self):
p = Parser()
for ext in self.extensions:
p = ext(p)
return p
These two functions work together to create an unextended Parser instance, which parses the template contents to produce the rootNode, and calls the rootNode emit() function to produce the final code (__template__()). So, the crux of the matter shifts to the Parser class.
Parser class study
The Parser class’s parse function parses the contents of the template and returns some node structure whose emit methods produce the actual Python code.
class Parser:
"""Parser Base.
"""
def __init__(self):
self.statement_nodes = STATEMENT_NODES
self.keywords = KEYWORDS
def parse(self, text, name="<template>"):
self.text = text
self.name = name
defwith, text = self.read_defwith(text)
suite = self.read_suite(text)
return DefwithNode(defwith, suite)
...
The above code is the two methods of the Parser class used in the Template class. The initializer function has nothing to say, while the parse function explains the top-level logic of Template parsing:
-
First parse the def with line.
-
Then parse the rest.
-
It returns an instance of defwithNode.
Parser’s parse function returns an instance of defwithNode, and calls to its emit() method generate the final code.
Compiles the code for the __template__() function
Once the compile_template method has the code for the template, it compiles the code to generate Python bytecode:
def compile_template(self, template_string, filename):
code = Template.generate_code(template_string, filename, parser=self.create_parser())
...
print code
try:
# compile the code first to report the errors, if any, with the filename
compiled_code = compile(code, filename, 'exec')
except SyntaxError, e:
# display template line that caused the error along with the traceback.
...
return compiled_code
This completes the second step of the Template class instantiation process: you get the compiled Template function code.
BaseTemplate
The next step is to call the initialization method of the parent class, the BaseTemplate class:
BaseTemplate.__init__(self, code=code, filename=filename, filter=filter, globals=globals, builtins=builtins)
The code that this method takes is the bytecode compiled above. Globals and builtins are used to build the __template__() function. Remember where the documentation for web.py says you can change these two? If you forget, the portal. The filter function is the filter function used to process the generated HTML content.
Globals and builtins
If not specified in your code, globals is empty by default and builtins contains the following:
TEMPLATE_BUILTIN_NAMES = [
"dict", "enumerate", "float", "int", "bool", "list", "long", "reversed",
"set", "slice", "tuple", "xrange",
"abs", "all", "any", "callable", "chr", "cmp", "divmod", "filter", "hex",
"id", "isinstance", "iter", "len", "max", "min", "oct", "ord", "pow", "range",
"True", "False",
"None",
"__import__", # some c-libraries like datetime requires __import__ to present in the namespace
]
These are just a subset of Python’s global built-in functions that you can use directly in your templates.
The initialization of the BaseTemplate
For this initialization process, let’s look directly at the code (list only the relevant functions) :
class BaseTemplate:
def __init__(self, code, filename, filter, globals, builtins):
self.filename = filename
self.filter = filter
self._globals = globals
self._builtins = builtins
if code:
self.t = self._compile(code)
else:
self.t = lambda: ''
def _compile(self, code):
env = self.make_env(self._globals or {}, self._builtins)
exec(code, env)
return env['__template__']
def make_env(self, globals, builtins):
return dict(globals,
__builtins__=builtins,
ForLoop=ForLoop,
TemplateResult=TemplateResult,
escape_=self._escape,
join_=self._join
)
The main thing to do is call self.t = self._compile(code). The _compile() method first uses the make_env() method to construct a function runtime env, which contains the objects used by the __template__() function, including the default built-in functions, and the templateResult. Then, we call exec(code, env) to execute the definition of the __template__() function in the env environment, and return the same __template__() function as we did in the env environment.
Note :* In Python 2, exec is a statement:
The first expression should evaluate to either a Unicode string, a Latin-1 encoded string, an open file object, a code object, or a tuple.
Exec also has the following form:
exec code in globals, locals
If you take a tuple as an argument, it takes the following two forms:
exec (code, globals)
exec (code, globals, locals)
An __builtins__ object can be inserted into the Globals dictionary to set references to built-in objects, such as the make_env() function above.
Parser class
Now you can see how the Parser class parses the template and generates a specific node structure. Let’s start with the parse function:
def parse(self, text, name="<template>"):
self.text = text
self.name = name
defwith, text = self.read_defwith(text)
suite = self.read_suite(text)
return DefwithNode(defwith, suite)
The read_defwith() method is called first to separate out the $def with(name) line, and the rest is parsed by read_suite(). Finally, a defwithNode is instantiated as the result. The read_suite() method does most of the work of the entire parsing.
Implementation conventions for the Parser class
Analytic function
The Parser class implements the following methods:
def read_assignment : function
def read_block_section : function
def read_expr : function
def read_indented_block : function
def read_keyword : function
def read_node : function
def read_section : function
def read_statement : function
def read_suite : function
def read_text : function
def read_var : function
def readline : function
The arguments and return values of these methods all follow the same pattern, so knowing this will help you read the code.
-
Each method is responsible for resolving a specific content (as shown by the method name), and some methods call other methods internally.
-
Each parameter is a Text, representing the content of the template that has not been parsed.
-
The return value is a tuple of two elements. The first element is the result of the function’s parsing (a string or an instance of a class), and the second element is the template content left after the function’s processing (to be parsed by the next function).
As you can see from this convention, the whole idea of template parsing is to start at the header of the template content, read it bit by bit, parse it as soon as it can be parsed, and continue to do so for the rest of the content.
In addition, these parsing functions also have inclusion relationships, some dealing with block contents, some dealing with a line, and some dealing with a word, which leads to a calling relationship: Coarse-grained functions call fine-grained parsing functions. As shown in the figure below:
Parsing the node
As mentioned in the previous section, the first value returned by a parse function is usually a parsed instance of a node class. So what is a parse node? Parsing nodes are just classes that implement the emit() function and produce the corresponding code string when calling the emit() function. Let’s take a look at the parsing nodes:
--
TextNode
ExpressoinNode
LineNode
VarNode
StatementNode
AssignmentNode
BlockNode
IfNode, ElseNode, ElifNode
ForNode
CodeNode
DefNode
SuiteNode
DefwithNode
Some nodes are simply wrappers around other nodes (such as Linenode), while others need to handle more complex cases (such as ForNode). But what these nodes do can be summarized as: handle initialization parameters so that the correct code is produced when the emit method is called. Let’s look at two examples:
AssignmentNode
Content of the template
$def with (name)
$ b = name
The function content
def __template__ (name):
__lineoffset__ = -4
loop = ForLoop()
self = TemplateResult(); extend_ = self.extend
b = name
return self
We know that when we parse to the $b = name line, an assignmentNode is generated. The code that calls read_assignment() is in the read_section() method of the Parser class. We can mimic this:
In [16]: node, _ = p.read_assignment(" b = name\n")
In [18]: print node
<assignment: 'b = name'>
In [20]: print node.emit(" ")
b = name
This is how Web.py’s template system produces the final code by calling the emit() method on each node.
ForNode
Let’s look at the more complex ForNode.
Content of the template
$def with (name_list)
$for name in name_list:
$name
The function content
def __template__ (name_list):
__lineoffset__ = -4
loop = ForLoop()
self = TemplateResult(); extend_ = self.extend
for name in loop.setup(name_list):
extend_([escape_(name, True), u'\n'])
return self
The method called to create an instance of forNode in web.py is read_block_section(), which we can simulate like this:
In [27]: node = web.template.ForNode("for name in name_list\n", " $name\n")
In [28]: print node
<block: 'for name in name_list\n', [<line: [t' ', $name, t'\n']>]>
In [31]: print node.emit(" ")
for name in loop.setup(name_list):
extend_([u' ', escape_(name, True), u'\n'])
When we create an instance of a forNode node, we pass the loop control statement on the first line and the rest of the loop internal code as two arguments to the forNode initializer. ForNode’s main job is to parse the first argument and convert it into code like loop.setup() (in order to be able to support the loop keyword in the template). Then, we call the initializer of the parent class BlockNode, which calls the read_suite() method of Parser class to parse the code inside the loop. Because we also use template syntax inside the loop (as in the example above, $name\n is converted to extend_([u “, escape_(name, True), u’\n’])).
class BlockNode:
def __init__(self, stmt, block, begin_indent=''):
self.stmt = stmt
self.suite = Parser().read_suite(block)
self.begin_indent = begin_indent
def emit(self, indent, text_indent=''):
text_indent = self.begin_indent + text_indent
out = indent + self.stmt + self.suite.emit(indent + INDENT, text_indent)
return out
def __repr__(self):
return "<block: %s, %s>" % (repr(self.stmt), repr(self.suite))
class ForNode(BlockNode):
def __init__(self, stmt, block, begin_indent=''):
self.original_stmt = stmt
tok = PythonTokenizer(stmt)
tok.consume_till('in')
a = stmt[:tok.index] # for i in
b = stmt[tok.index:-1] # rest of for stmt excluding :
stmt = a + ' loop.setup(' + b.strip() + '):'
BlockNode.__init__(self, stmt, block, begin_indent)
def __repr__(self):
return "<block: %s, %s>" % (repr(self.original_stmt), repr(self.suite))
SuiteNode
Inside SuiteNode is a list that holds all of the children. When you call the SuiteNode emit() method, you call the children’s emit() method in turn and concatenate it into a string:
def emit(self, indent, text_indent=''):
return "\n" + "".join([s.emit(indent, text_indent) for s in self.sections])
DefwithNode
DefWithNode, as the root node, does two things:
-
Generates a framework for the __template__() function.
-
Calling Suite. Emit () generates the rest of the code and splits it into a complete function.
Parser class summary
This chapter covers most of the implementation of the Parser class, but it doesn’t explain how the Parser class parses the contents of the template to determine which node to generate. The content of this part is the specific implementation details, and there are two main analysis techniques used:
-
String analysis to see if a string starts with some particular pattern.
-
Token analysis takes advantage of Python’s tokenize module to analyze.
conclusion
This article mainly analyzes the generation process from the Template file to the Template class instance, which consists of the following steps:
-
The fRender () function is called to read the contents of the Template file, which is passed as an argument to the initialization function of the Template class.
-
The Template class calls the Parser class to parse the Template content into the definition code for the __template__() function.
-
The Template class calls Python’s compile function to compile the generated __template__() definition.
-
The Template class calls the parent BaseTemplate class’s initializer to construct the __template__() function that executes in the specified context.