This PR is a suggestion to help fix https://github.com/twbs/bootstrap/commit/134dc7c5a0ac5144800170e28404d259a7983a3d based on the understanding I have for the moment of the reporters and the statuses of the several pa11y* projects. @XhmikosR I hope it will help, otherwise this PR can be closed.
Links
- HTML report / artifact: https://github.com/twbs/bootstrap/actions/runs/1586113297
- CLI output: https://github.com/twbs/bootstrap/runs/4543729540?check_suite_focus=true
Summary
According to the pa11y-ci documentation it is possible to do:
npm install pa11y-reporter-html --save
pa11y-ci --reporter=pa11y-reporter-html https://pa11y.org/
Maybe I have missed or don't understand something but I tried it and nothing was generated nor displayed. So I suppose that if it doesn't work there, it is normal that it doesn't work neither with multiple reporters defined as:
{
"defaults": {
"reporters": [
"cli",
"pa11y-reporter-html",
]
}
}
So I tried to understand what's going on here.
By adding some logs in node_modules/pa11y-reporter-html/lib/reporter.js, we can see that the HTML code is well generated. But it doesn't create a file containing this HTML code nor displays this HTML code as an output.
We can find another information in https://github.com/pa11y/pa11y-reporter-html that specifies that "The reporter has been merged into pa11y/reporters.".
Tried it and indeed, I was able to do this as well that produced exactly the same non-result:
{
"defaults": {
"reporters": [
"cli",
"./node_modules/pa11y/lib/reporters/html.js"
]
}
}
Note: node_modules/pa11y/bin/pa11y.js -r html https://example.com
displays the HTML as an output.
So let's dive into pa11y/reporters. They all create the content but never really create a file or display themselves their output.
Contrary to the reporter defined in pa11y-ci/reporters. At the end of the json.js, we can see:
if (fileName) {
writeReport(fileName, jsonString);
} else {
console.log(jsonString);
}
IMO, for a reporter to work with pa11y-ci, it must be capable of creating the file.
With all that in mind, I would say we don't have a choice for the moment and should generate a JSON file with ['json', { 'fileName': './pa11y-ci-results.json' }]
and then use pa11y-ci-reporter-html
to generate the HTML file from this JSON file