High-level-design-overview.md

+# High-Level Design Overview
+
+## parser.js
+
+The parser is a slightly modified fork of the [esprima](https://github.com/ariya/esprima/tree/harmony) harmony branch. It has two important functions:
+
+### `read` 
+
+`read` converts a string to a token tree. A token tree is similar to tokens produced by the standard esprima lexer but with the critical difference being that token trees match delimiters. So the standard esprima lexer would transform the string `"{ 42 }"` into three tokens:
+
+```js
+[{
+    type: 7,   // the punctuator type
+    value: "{"
+},
+{
+    type: 6,  // the numeric literal type
+    value: 42
+},
+{
+    type: 7,
+    value: "}"
+}]
+```
+
+But `read` will transform the same string into a single `{}` token tree with `inner` tokens:
+
+```js
+[{
+    type: 11,   // the delimiter type
+    value: "{}",
+    inner: [{
+        type: 6,
+        value: 42
+    }]
+}]
+```
+
+### `parse` 
+
+`parse` converts a token tree to an AST that conforms to the [Parser API](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API)
+
+Note that `parse` only understands the JavaScript grammar. All macros must be expanded from the token tree before calling `parse`.
+
+## expander.js
+
+This is where expansion happens. The expander takes a token tree, finds and loads any macro definitions, and the expands any invocations of those macros. The important functions in the expander are `expandToTermTree`, `expandTermTreeToFinal`, and `enforest`.
+
+To understand how things work in here you might want to read ["Macros that Work Together"](http://www.cs.utah.edu/plt/publications/jfp12-draft-fcdf.pdf) (which explains how hygiene works among other things) and ["Honu: A Syntactically Extensible Language"](http://www.cs.utah.edu/~rafkind/papers/honu-2012.pdf) (which explains how to do expansion for non-lispy languages).
+
+### `enforest`
+
+Enforest was first described in the honu paper. The idea is to look at the first few tokens in the token tree and convert them into a pseudo-AST called a term tree. Whereas a token tree is just flat tokens and nested delimiters, a term tree are complex objects that look similar to an AST like `ForStatement` or `FunctionExpression`. It basically adds structure to the mostly flat token tree. This is also where macros get expanded.
+
+### `expandToTermTree`
+
+This is similar to `parse1` in the honu paper. It calls `enforest` repeatedly until the entire token tree has been converted into a term tree. Each time `enforest` is called the result is checked; if the result is a macro definition the definition is loaded into the environment so it can be used later.
+
+### `expandTermTreeToFinal`
+
+This is similar to `parse2` in the honu paper. It takes the term tree generated by `expandToTermTree` and does some final processing to complete the expansion. Mainly this means handling issues of hygiene for function definitions.
+
+## patterns.js
+
+This is where the pattern matching functions live. They get called from macro definitions when a macro is invoked to bind syntax to pattern variables and substitute syntax into a template.
+
+## syntax.js
+
+This is where the definitions for syntax objects live. Some hygiene related stuff is here too.
+
+## sweet.js
+
+This ties everything together. It provides some convenience functions like `compile` that take a string and calls `read`, `expand`, `parse`, and `generate` to produce a string with all the macros expanded.
\ No newline at end of file