Python/AST

http://www.python.org/download/releases/2.5/NEWS.txt

- A new AST parser implementation was completed. The abstract

• syntax tree is available for read-only (non-compile) access to Python code; an _ast module was added.

http://docs.python.org/dev/whatsnew/node16.html

The design of the bytecode compiler has changed a great deal, to no longer generate bytecode by traversing the parse tree. Instead the parse tree is converted to an abstract syntax tree (or AST), and it is the abstract syntax tree that's traversed to produce the bytecode.

It's possible for Python code to obtain AST objects by using the compile() built-in and specifying |_ast.PyCF_ONLY_AST| as the value of the flags parameter:

from _ast import PyCF_ONLY_AST ast = compile("""a=0 for i in range(10):

• a += i

""", "<string>", 'exec', PyCF_ONLY_AST)

assignment = ast.body[0]

for_loop = ast.body[1]

No documentation has been written for the AST code yet. To start learning about it, read the definition of the various AST nodes in Parser/Python.asdl. A Python script reads this file and generates a set of C structure definitions in Include/Python-ast.h. The PyParser_ASTFromString() and PyParser_ASTFromFile(), defined in Include/pythonrun.h, take Python source as input and return the root of an AST representing the contents. This AST can then be turned into a code object by PyAST_Compile(). For more information, read the source code, and then ask questions on python-dev.

The AST code was developed under Jeremy Hylton's management, and implemented by (in alphabetical order) Brett Cannon, Nick Coghlan, Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters, Armin Rigo, and Neil Schemenauer, plus the participants in a number of AST sprints at conferences such as PyCon.

http://www.amk.ca/diary/2005/10/the_ast_branch_lands.html

The AST branch lands Last week, a historic event happened in the Python source tree: the AST branch was merged into the trunk. I've only followed this work from python-dev e-mails, and with only half my brain, so take the following notes with a grain of salt; I've probably made some silly errors.

To start, what's the AST branch? In all previous versions of CPython, Python source code is parsed and turned into a parse tree. Code then loops over this parse tree to generate Python bytecode. The problem with this design is that it's very difficult to make modifications or analyses of the parse tree, because the parse tree records details that aren't relevant from the point of view of the code's semantics. For example, the parse trees for these two statements are different, even though they implement the same computation: [...]

AST stands for Abstract Syntax Tree. The AST more closely matches the semantics of Python and cleans up special cases. [...]

The primary benefit is that it should now be easier to write optimization passes and tools such as PyChecker or refactoring browsers, particularly once there's a Python interface to the AST and once everything is documented. There's also some hope that the AST interface can be shared across Python implementations such as Jython and IronPython. We'll see where things go from here.

Originally from: http://lua-users.org/lists/lua-l/2006-09/msg00731.html