initial readtable implementation

Created by: jlongster

This is my initial implementation of readtables. It's very early and not ready to be merged yet, but I wanted to open it up as early as possible for feedback. There's definitely sloppy code in some places, which will be cleaned up. I'm mostly interested in feedback about the APIs and how we handle custom reader extensions.

I went ahead and included an example reader in the source, which will be removed before this is merged. The reader implements the JSX extension to embed HTML in JS. This has been a fantastic example of a reader and has uncovered practical needs that our APIs need to meet.

Theoretically, with this code you can expand this file:

let DOM = macro {
    rule { { $el $attrs } } => {
        $el($attrs)
    }

    rule { { $elStart $attrs $elEnd } } => {
        $elStart($attrs)
    }

    rule { { $elStart $attrs $children:expr ... $elEnd } } => {
        $elStart($attrs, $children (,) ...)
    }

    rule { } => { DOM }
}

<a foo="bar"> {value} <b><c /></b></a>

with the command:

./build/bin/sjs --load-readtable ./example-readtable.js jsx-macro.js

The "example-readtable.js" file will be looked up relative to the sweet files, not your working directory, so that should work. Just a stop gap for now. That should output:

React.DOM.a({ foo: 'bar' }, ' ', value, ' ', React.DOM.b(null, c(null)));                                               

There are 3 integration points that I'd like feedback on:

1. Hooking the readtable into the parser. This is in parser.js. Esprima doesn't have a parser object, it all functions within the same file and assumes closures. This makes is hard to call an external module and give it the parser. I had to create a fake parser object with APIs that can access the closure. This fake object is the parser API provided to a reader, and we need to think through what it should be.
2. readtables.js implements a base reader object that can be extended with CustomReader.create. This provides higher-level methods for a reader. It will scan for tokens and automatically queue them in a buffer, and provides methods like reset which will bail out and reset the parser to the state before the reader was invoked.
3. How we actually load readtables is up for debate. I think this is where we'll need to do the most research on. Racket has a powerful mechanism for specifying a language at the top of the file. JSX actually requires you to put an annotation at the top of the file. The nice thing about this is that you can run the expander across a whole bunch of JS files and know that it will only touch yours. We could provide an option for this mode, and have some sort of language file that specifies reader extensions and macros to load. Or we can just provide CLI options.

EDIT: A minor point to #3 (closed) above, the React compiler not only looks for the JSX annotation at the top of the file, but also parses an argument off of it. The annotation is /** @jsx React.DOM */, so it only prefixes the DOM elements with that you pass. (a is turned into React.DOM.a)

Racket only allows reader extensions to be invoked on expressions. I think that makes a lot of sense. In fact, for JSX I have go through more work to make sure that the < character is in place of an expression. The problem is that at the readToken stage we know nothing about the shape of the code or if we are at an expression or not. Not sure how Racket does this, but it's something to think about.

I made it so that you cannot override delimiters. Reader extensions are only checked after delimiters are parsed in readToken. It seems like too much power to override delimiters and I don't think we should allow it; the chances for something to go wrong are high.

That is my braindump of all of my work. It's probably somewhat scattered, but it feels nice to get it out there so that feedback can help shape it early on.