The idea is that you might be able to parallelize internally in the worker process, with the benefit of shared memory to cache details about dependencies. Async worker processes can receive & run multiple commands concurrently. They respond with results asynchronously in any order.
PR Notes:
- The DSL arg is
solo_async=True
becauseasync
is a python keyword. - I'm sort of lukewarm on the idea of
ListenableFuture
s - open to suggestions. The only real benefit is that it makes testing concurrent execution/cancellation easier. - In theory the
Sync
worker pool can be cleaned up with the new interface (less of a need for aBurrowedWorkerProcess
& lifecycle). I did not do this to keep this PR manageable.