VS2017 running out of memory compiling ir_x86.c after recent AVX-512 additions
This happened with VS2013 too in the past: #3992 (closed). We split up some #included files to solve it then, but they have grown again.
The recent AVX-512 broadcast additions for #4534 (closed) in PR #4577 seem to have pushed it over the edge: but it seems non-deterministic since several prior builds on GA CI Server2016 VS2017 worked.
https://github.com/DynamoRIO/dynamorio/pull/4586/checks?check_run_id=1553077661
2020-12-14T21:09:44.3614062Z FAILED: suite/tests/CMakeFiles/api.ir.dir/api/ir_x86.c.obj
2020-12-14T21:09:44.3616541Z C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\cl.exe /nologo -DDR_HOST_ -DDR_REG_ENUM_COMPATIBILITY -DDYNAMORIO_STANDALONE -DX86_64 -I. -I..\suite\tests -I..\core\arch -Icmake\..\include -I..\core\drlibc -I..\core\lib -Iinclude\annotations -Isuite\tests\annotations -Iext\include -DSTANDALONE -DUSE_DYNAMO /nologo /MP /GF /FS /GS- /MTd /Zi /Od /W3 /WX -DNIGHTLY_REGRESSION -DNOT_DYNAMORIO_CORE /showIncludes /Fosuite\tests\CMakeFiles\api.ir.dir\api\ir_x86.c.obj /Fdsuite\tests\CMakeFiles\api.ir.dir\ /FS -c ..\suite\tests\api\ir_x86.c
2020-12-14T21:09:44.3618960Z d:\a\dynamorio\dynamorio\suite\tests\api\ir_x86.c(469) : fatal error C1002: compiler is out of heap space in pass 2
2020-12-14T21:09:44.3620066Z d:\a\dynamorio\dynamorio\suite\tests\api\ir_x86.c(458) : fatal error C1002: compiler is out of heap space in pass 2
2020-12-14T21:09:44.8225895Z ninja: build stopped: subcommand failed.
Compare these to the sizes listed in #3992 (closed):
$ wc suite/tests/api/ir*.h | sort -n
65 453 3684 suite/tests/api/ir_x86_2args_avx512_vex.h
68 506 4354 suite/tests/api/ir_x86_3args_avx512_vex.h
69 427 4149 suite/tests/api/ir_x86_4args_avx512_evex.h
97 459 5196 suite/tests/api/ir_x86_2args_avx512_evex_mask.h
113 627 5553 suite/tests/api/ir_x86_all_opc.h
193 876 6917 suite/tests/api/ir_x86_0args.h
193 992 9481 suite/tests/api/ir_x86_1args.h
217 1031 13206 suite/tests/api/ir_x86_3args_avx512_evex.h
247 1279 15896 suite/tests/api/ir_x86_2args_avx512_evex.h
327 1859 21705 suite/tests/api/ir_x86_2args_mm.h
331 1405 17631 suite/tests/api/ir_x86_4args.h
425 2055 26349 suite/tests/api/ir_x86_3args_avx.h
460 2634 27811 suite/tests/api/ir_x86_2args.h
485 2570 30857 suite/tests/api/ir_x86_3args.h
1041 4823 72815 suite/tests/api/ir_x86_5args_avx512_evex_mask.h
3247 12076 187105 suite/tests/api/ir_x86_3args_avx512_evex_mask.h
3417 13754 217118 suite/tests/api/ir_x86_4args_avx512_evex_mask_A.h
4105 16676 255355 suite/tests/api/ir_x86_4args_avx512_evex_mask_B.h
15100 64502 925182 total
I think we need to split ir_x86_4args_avx512_evex_mask_B.h in half.