Skip to content
GitLab

Fix a bug of not following robots.txt

Merged yujiosaka requested to merge fix_a_bug_of_robots_txt into master

The bug started when updating puppeteer's version at https://github.com/yujiosaka/headless-chrome-crawler/pull/65 .