AVX-512 asserts incorrectly fire on decoding user input
The decoder needs to support users trying to decode non-code, which is a common approach in various tools: try decoding somewhere and see whether it looks like legitimate code. The decoder should simply return invalid instructions through normal channels: it should not assert and abort. However, it does just that in multiple places via new AVX-512 asserts.
This breaks Dr. Memory's system call wrapper identification scheme:
Searching for system calls in "C:\WINDOWS\system32\ntdll.dll"
Found 465 system calls (0 usercalls) in "C:\WINDOWS\system32\ntdll.dll"
Searching for system calls in "C:\WINDOWS\system32\kernelbase.dll"
---------------------------
Microsoft Visual C++ Runtime Library
---------------------------
Debug Error!
Program: D:\derek\drmemory\git\build_DRtoo_x64_dbg\bin64\drmemory.exe
R6010
- abort() has been called
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------It's just decoding non-code as it looks through dll symbols searching for syscall wrappers. The decoder shouldn't assert on any user input.
CLIENT_ASSERT(!di->evex_encoded, "TODO i#1312: decode error: unsupported yet.");
0:000> U @@(sd->dll_base+modoffs) L10
0000026b`d8e11fa0 53 push rbx
0000026b`d8e11fa1 58 pop rax
0000026b`d8e11fa2 53 push rbx
0000026b`d8e11fa3 3a20 cmp ah,byte ptr [rax]
0000026b`d8e11fa5 2573282920 and eax,20292873h
0000026b`d8e11faa 62 ???
0000026b`d8e11fab 61 ???
0000026b`d8e11fac 64207061 and byte ptr fs:[rax+61h],dh
0000026b`d8e11fb0 7261 jb 0000026b`d8e12013
0000026b`d8e11fb2 6d ins dword ptr [rdi],dx
0000026b`d8e11fb3 657465 je 0000026b`d8e1201b
0d 00000077`ecb891b0 00007ff7`15cbe822 drmemory!abort+0x24 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 72]
0e 00000077`ecb891f0 00007ff7`15cd2c17 drmemory!external_error+0x32 [d:\derek\drmemory\git\src\dynamorio\core\arch\decodelib.c @ 133]
0f 00000077`ecb89220 00007ff7`15cd3824 drmemory!read_prefix_ext+0x127 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 901]
10 00000077`ecb89270 00007ff7`15cd6a63 drmemory!read_instruction+0xad4 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1128]
11 00000077`ecb89380 00007ff7`15ccfbdc drmemory!decode_common+0xe3 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2488]
12 00000077`ecb895e0 00007ff7`15c8b588 drmemory!decode+0x2c [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2655]
13 00000077`ecb89610 00007ff7`15c8bb13 drmemory!get_syscall_num+0xa8 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 666]
14 00000077`ecb89760 00007ff7`15cf1f86 drmemory!search_syms_cb+0xf3 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 784]
15 00000077`ecb89850 00007ff9`a63bbd9e drmemory!enum_cb+0x136 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 779]
16 00000077`ecb898a0 00007ff9`a63ab3db dbghelp!DoEnumCallback+0x232
17 00000077`ecb8a920 00007ff9`a63ab98e dbghelp!diaEnumScope+0x157
18 00000077`ecb8a9a0 00007ff9`a63abf2e dbghelp!diaSearchAll+0xda
19 00000077`ecb8aa40 00007ff9`a63cd3ef dbghelp!diaSearch+0x12a
1a 00000077`ecb8ba80 00007ff9`a63bc1f4 dbghelp!modSearch+0xa3
1b 00000077`ecb8bae0 00007ff9`a63c1a56 dbghelp!ModLoop+0x100
1c 00000077`ecb8bb60 00007ff9`a63c1b30 dbghelp!SearchSymbols+0x14e
1d 00000077`ecb8db70 00007ff7`15cf2571 dbghelp!SymSearch+0xa4
1e 00000077`ecb8dbf0 00007ff7`15cf0833 drmemory!drsym_search_symbols_local+0x2d1 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 914]
1f 00000077`ecb8dcb0 00007ff7`15c8c014 drmemory!drsym_search_symbols+0x83 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 1496]
20 00000077`ecb8dd10 00007ff7`15c892be drmemory!identify_syscalls+0x334 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 837]
21 00000077`ecb8dec0 00007ff7`15c26696 drmemory!drsys_generate_sysnum_file+0x10e [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 956]
22 00000077`ecb8df90 00007ff7`15c2aa16 drmemory!generate_sysnum_file+0x1b6 [d:\derek\drmemory\git\src\drmemory\frontend.c @ 728]
23 00000077`ecb8e150 00007ff7`15c56b7c drmemory!wmain+0x42a6 [d:\derek\drmemory\git\src\drmemory\frontend.cAfter fixing that it hits another assert:
CLIENT_ASSERT(!operand_is_ymm || !operand_is_zmm, "Internal reg size error.");
0:000> U 0x0000019a`c93b6f59
0000019a`c93b6f59 65 ???
0000019a`c93b6f5a 62 ???
0000019a`c93b6f5b 417574 jne 0000019a`c93b6fd2
0000019a`c93b6f5e 684e437461 push 6174434Eh
0000019a`c93b6f63 7049 jo 0000019a`c93b6fae
0d 0000003e`87fc8e20 00007ff6`f6b0e822 drmemory!abort+0x24 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 72]
0e 0000003e`87fc8e60 00007ff6`f6b245d0 drmemory!external_error+0x32 [d:\derek\drmemory\git\src\dynamorio\core\arch\decodelib.c @ 133]
0f 0000003e`87fc8e90 00007ff6`f6b249a4 drmemory!decode_reg+0x480 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1497]
10 0000003e`87fc8f30 00007ff6`f6b2595f drmemory!decode_modrm+0x94 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1564]
11 0000003e`87fc9050 00007ff6`f6b26bde drmemory!decode_operand+0x4df [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1934]
12 0000003e`87fc9390 00007ff6`f6b1fbdc drmemory!decode_common+0x2ae [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2519]
13 0000003e`87fc95f0 00007ff6`f6adb588 drmemory!decode+0x2c [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2660]
14 0000003e`87fc9620 00007ff6`f6adbb13 drmemory!get_syscall_num+0xa8 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 666]
15 0000003e`87fc9770 00007ff6`f6b41f36 drmemory!search_syms_cb+0xf3 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 784]
16 0000003e`87fc9860 00007ff9`a71ebd9e drmemory!enum_cb+0x136 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 779]
17 0000003e`87fc98b0 00007ff9`a71db3db dbghelp!DoEnumCallback+0x232
18 0000003e`87fca930 00007ff9`a71db98e dbghelp!diaEnumScope+0x157
19 0000003e`87fca9b0 00007ff9`a71dbf2e dbghelp!diaSearchAll+0xda
1a 0000003e`87fcaa50 00007ff9`a71fd3ef dbghelp!diaSearch+0x12a
1b 0000003e`87fcba90 00007ff9`a71ec1f4 dbghelp!modSearch+0xa3
1c 0000003e`87fcbaf0 00007ff9`a71f1a56 dbghelp!ModLoop+0x100
1d 0000003e`87fcbb70 00007ff9`a71f1b30 dbghelp!SearchSymbols+0x14e
1e 0000003e`87fcdb80 00007ff6`f6b42521 dbghelp!SymSearch+0xa4
1f 0000003e`87fcdc00 00007ff6`f6b407e3 drmemory!drsym_search_symbols_local+0x2d1 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 914]
20 0000003e`87fcdcc0 00007ff6`f6adc014 drmemory!drsym_search_symbols+0x83 [d:\derek\drmemory\git\src\dynamorio\ext\drsyms\drsyms_windows.c @ 1496]
21 0000003e`87fcdd20 00007ff6`f6ad92be drmemory!identify_syscalls+0x334 [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 837]
22 0000003e`87fcded0 00007ff6`f6a76696 drmemory!drsys_generate_sysnum_file+0x10e [d:\derek\drmemory\git\src\drsyscall\pdb2sysfile.cpp @ 956]
23 0000003e`87fcdfa0 00007ff6`f6a7aa16 drmemory!generate_sysnum_file+0x1b6 [d:\derek\drmemory\git\src\drmemory\frontend.c @ 728]
24 0000003e`87fce160 00007ff6`f6aa6b7c drmemory!wmain+0x42a6 [d:\derek\drmemory\git\src\drmemory\frontend.c @ 1619] In fact that same decode() call hits 2 further asserts:
CLIENT_ASSERT(false, "invalid vector length.");
0d 000000f8`c8bc8970 00007ff6`5953e822 drmemory!abort+0x24 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 72]
0e 000000f8`c8bc89b0 00007ff6`595568dc drmemory!external_error+0x32 [d:\derek\drmemory\git\src\dynamorio\core\arch\decodelib.c @ 133]
0f 000000f8`c8bc89e0 00007ff6`59550efe drmemory!decode_get_vector_length+0x6c [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2187]
10 000000f8`c8bc8a10 00007ff6`595550d9 drmemory!decode_get_compressed_disp_scale+0xde [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2204]
11 000000f8`c8bc8ac0 00007ff6`595558a0 drmemory!decode_modrm+0x809 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1727]
12 000000f8`c8bc8be0 00007ff6`59556d56 drmemory!decode_operand+0x460 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1926]
13 000000f8`c8bc8f20 00007ff6`5954fbdc drmemory!decode_common+0x466 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2548]
14 000000f8`c8bc9180 00007ff6`5950b588 drmemory!decode+0x2c [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2665]
default: CLIENT_ASSERT(false, "invalid vector length.");
0d 000000a2`e4338800 00007ff6`7893e822 drmemory!abort+0x24 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 72]
0e 000000a2`e4338840 00007ff6`789511d6 drmemory!external_error+0x32 [d:\derek\drmemory\git\src\dynamorio\core\arch\decodelib.c @ 133]
0f 000000a2`e4338870 00007ff6`789550d9 drmemory!decode_get_compressed_disp_scale+0x3b6 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2255]
10 000000a2`e4338920 00007ff6`789558a0 drmemory!decode_modrm+0x809 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1727]
11 000000a2`e4338a40 00007ff6`78956d36 drmemory!decode_operand+0x460 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 1926]
12 000000a2`e4338d80 00007ff6`7894fbdc drmemory!decode_common+0x466 [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2553]
13 000000a2`e4338fe0 00007ff6`7890b588 drmemory!decode+0x2c [d:\derek\drmemory\git\src\dynamorio\core\arch\x86\decode.c @ 2670]