Added
- Add HCCrawler.defaultArgs() method.
- Emit
requestretried
event.
changed
- Use
cache
option not only for remembering already requested URLs but for request queue for distributed environments. - Moved
onSuccess
,onError
andmaxDepth
options from HCCrawler.connect() and HCCrawler.launch() to crawler.queue().