TEB fields hold app values during detach client exit
Discovered in https://github.com/DynamoRIO/dynamorio/pull/5335#issuecomment-1033980049
Repeating some of that here:
These tests use dr_app_stop_and_cleanup(): for which DR does dispatch_enter_native() which does an fcache_enter to transfer control to the target which is DR's dr_app_stop_and_cleanup function. So that will swap the PEB fields to the app. dispatch_enter_native does call dynamo_thread_not_under_dynamo but all that does is turn off asynch interception: no PEB swap. (You'd think it should do a PEB swap: but the cxt sw gencode does; maybe we can split that up.) Then we run the detach_on_permanent_stack() w/ the fields as the app for the main thread: so client exit code is run with app TEB fields. dynamo_shared_exit() calls os_loader_exit() which does a swap (again) to the app, but only for the PEB pointer since it has no dcontext.
So we should call swap_peb_pointer(my_dcontext, to-priv) in detach_on_permanent_stack(), right? That seems to fix burst_replaceall: but traceopts now doesn't crash but has an assert on the re-attach on TEB stack bounds.
dstack is 0x000001b1d2085000
get_stack_bounds: app stack now is 0x0000000ef22fb000-0x0000000ef2300000
os_loader_exit: app stack now is 0x0000000ef22fa000-0x000001b1d2085000
os_loader_exit: app stack now is 0x0000000ef22fa000-0x000001b1d2085000
dstack is 0x000001b5d20a5000
get_stack_bounds: app stack now is 0x0000000ef22fa000-0x000001b1d2085000
So dstack from 1st attach is in TOP_STACK_TIB_OFFSET. That is the field we swap.
Aha: os_loader_exit() has no dcontext so all it does is swap the peb: no teb field swaps.
So we need teb field swaps on thread exit.
Oh: and we have to make dynamo_shared_exit() call thread exit for the current thread, not other -- see this comment:
#ifdef WINDOWS
/* If we use dynamo_thread_exit() when toexit is the current thread,
* it results in asserts in the win32.tls test, so we stick with this.
*/
d_r_mutex_lock(&thread_initexit_lock);
dynamo_other_thread_exit(toexit, false);
d_r_mutex_unlock(&thread_initexit_lock);
#else
/* On Linux, restoring segment registers can only be done
* on the current thread, which must be toexit.
*/
ASSERT(toexit->id == d_r_get_thread_id());
dynamo_thread_exit();
#endif
Maybe those asserts are b/c of the lack of teb field swap on detach? I hit a kstats assert w/ my fixes; running -no_kstats: no asserts w/ my fixes; but also no asserts w/o them. Hmm. But: w/o the priv swap I added it would have been swapped to app already.