crawler.response becomes null when connecting to specific chrome instance
Created by: carlkentor
What is the current behavior? Okay, I dont know if this should be classified as a bug or not. But since I cant seem to find any trace of this in the docs, here it is.
When connecting to a specific chrome instance through the:
await HCCrawler.connect({ browserWSEndpoint: wsChromeEndpointurl, ...more options goes here})
method, the call to await crawl()
results in a null response.
The issue lies within:
\node_modules\headless-chrome-crawler\lib\crawler.js
async crawl(){
await this._prepare();
const response = await this._request(); //1. Results in Null
console.log(response); //2. ======> NULL INDEED
await this._waitFor();
const [
result,
screenshot,
cookies,
links,
] = await Promise.all([
this._scrape(),
this._screenshot(),
this._getCookies(),
this._collectLinks(response.url), //3. ====> Exception gets thrown here
]);
return {
options: this._options,
depth: this._depth,
previousUrl: this._previousUrl,
response: this._reduceResponse(response),
redirectChain: this._getRedirectChain(response),
result,
screenshot,
cookies,
links,
}; }
`
Steps to reproduce
- Close all chrome instances.
- Open CMD and run the following to start Chrome with remote debugging enabled:
start chrome.exe --remote-debugging-port=9222
- Navigate to http://localhost:9222/json/version and copy the value of "webSocketDebuggerUrl"
- Set the browserWSEndpoint to the value you just copied.
What is the expected behavior? When running through a clean instance with the
await HCCrawler.launch({
...options
})
command, everything works fineand the result comes back as expected. What am I missing here?
Please tell us about your environment:
- Version: 1.8.0
- Platform / OS version: Windows 10 / 1803
- Node.js version: v10.16.3