broken-link-checker: force follow links.

After 3256a2c2, blc honored robots.txt thus it didn't crawl anything. Ignore robots.txt to work around the issue.

Fixes #27585 (closed)