Added
- Support
obeyRobotsTxt
for crawler.queue()'s options. - Support
persist
for RedisCache's constructing options.
changed
- Make
cache
to be required for HCCrawler.connect() and HCCrawler.launch()'s options. - Provide
skipDuplicates
to remember and skip duplicate URLs, instead of passingnull
tocache
option. - Modify
BaseCache
interface.