Created by: myspivey
Hey there Everyone! My name is James Spivey and wanted to submit a PR I think would be really helpful! I have recently joined a company called This Dot and been experimenting with visual regression testing. I am following up on fantastic work that @jayphelps has started in this area including the PR today. Applitools was kind enough to sponsor Jay and myself to help out and make some PRs to add visual regression tests to some open source projects and we immediately thought of Bootstrap.
If you're not familiar with visual regression testing there's a pretty good summary in the Storybook docs (most of the page is unrelated to Storybook) but the tl;dr it involves rendering components individually in the DOM and taking a snapshot of the pixels and doing comparisons of it against baselines.
Most solutions do normal 1:1 pixel comparing, but this often results in false positives because the exact pixels can vary. e.g. different GPUs render things differently, so your local tests will fail against baselines created on another computer, or if the CI changes its GPU or there's a browser update, etc, etc, etc. Applitools on the other hand tries to visually compare like a human would, ignoring differences that a human wouldn't perceive or care about. The whole AI thing and such.
Maintaining a separate visual tests can sometimes be a pain on such a large project so what I did instead was take advantage of the wonderful existing docs setup you have. Since you already had the ability to render the component examples mostly all by themselves e.g. https://getbootstrap.com/docs/4.1/components/alerts/ This made it pretty dang easy.
So I chose to use Cypress as the browser test harness, which Applitools has an SDK for. I started off with just one test for a handful of the most common components so it's sort of a "smoke test" but you can add more if you'd like--it'll take longer to run though.
What happens is the docs site is served, then Cypress launches a headless Chrome where it runs the tests. Applitool's SDK then takes DOM snapshots w/ styles (not screenshots) from that headless Chrome and sends them to Applitool's servers where it renders those results in Chrome and then takes a screenshot of that result and does its intelligent comparisons against the baseline. The baseline is created the first time you run it. If there's an issue, it'll be reported in TravisCI with an included link directly to Applitool's web app where you can see the visual differences and either accept or reject them or do other cool things like set ignored regions or tweak the matching behavior.
You'll need to create an Applitools account. and export APPLITOOLS_API_KEY=valuehere your API key in your local environment if you run it locally.
I also included the necessary changes for it to run in your TravisCI setup, all you need to do is add your APPLITOOLS_API_KEY to your TravisCI project environment variables in their UI.
While the default Applitools account has decent limits they also offer free unlimited accounts to open source projects. In addition to higher limits, it will also check your screenshots concurrently so it runs faster. If you'd like to unlock that you can either contact them directly or if you shoot me your account email I can have them hook you up. If you don't want to post your email here, you can Twitter DM it to me or find my email on my profile.