speed up Appveyor CI workflow
Appveyor (#2145 (closed)) has enabled us to move to a purely cloud-based workflow without each dev needing to locally test every platform. However, Appveyor's free solution for open-source projects has no build parallelization either within a job or between jobs, meaning there's a global serialized queue of all pull request and master commit jobs. Since each one takes ~26 minutes this can be a bottleneck when multiple devs are submitting PR's simultaneously. Contrast this to Travis which has both inter- and intra-build parallelism and which I got down to under 10 minutes by splitting the suites up.
VS2010 32-bit-only build with no tests is taking nearly 7 mins. VS2013 32-bit + 64-bit (+ vps) build + tests is taking 19 mins.
Ideas for speeding it up:
Go back to requiring each dev to run Windows tests locally
I think this is a non-starter.
Drop VS2010 support
Abandon VS2010 support and only build VS2013 on Appveyor. VS2012 Update 1 and VS2013 can still target XP so moving to VS2013 could be done. Xref #1424 (closed). Moving to VS2013 would also shrink pdb files (xref #2167).
Run random subset of tests for PR's that are not clearly windows-affecting
For PR's that don't change win32/ or suite/tests/*win32, run a random subset of tests instead of every test. Do this via ctest -I with a randomly chosen interval? Count on the merge to master running everything and the dev reverting on any Windows failure there.
But, shrinking the number of tests may not make much of a difference: the build time dominates.
Plus there's plenty of shared code not in win32/ that affects Windows.
Augment or switch to another hosted CI on Windows?
Travis may eventually add Windows but it doesn't look very promising: https://github.com/travis-ci/travis-ci/issues/2104
There don't seem to be any other hosted CI that supported Windows.
Set up our own VM's or real hardware for CI
We'd want to pick one of the CI agent software products that already integrates nicely with Github.