Added
- Support
maxRequest
,allowedDomains
anduserAgent
option for crawler.queue([options]) - Support pluggable cache
- Add crawler.setMaxRequest(maxRequest), crawler.pause() and crawler.resume() methods
- Add crawler.pendingQueueSize and crawler.requestedCount read-only properties