Let’s assume that we’re all using an NT based OS (2k, XP, 2003). It’s a simple matter to detect that you’re being run unde a debugger – simply isue an IsDebuggerPresent call. Heck, you could even put it into a thread that intermittently polls and reacts (in)appropriately.
Of course, I’m not one for polling. Never have been. I just consider it ‘lower class’ to do something like that. This is of course why I like the next trick for detecting the arrival of a debugger.
what happens is that an external thread attaches and executed the ‘DbgUiRemoteBreakin’ function; which at some point issues an ‘DbgBreakPoint’, which of all things contains an int3 and a return statement (this is a kick the debugger operation).
Detecting the debugger consists of rewriting the function implementation.
What we want to do is invoke our function (fun) when a debugger attaches. The following code does just that, and as a result of the attachment of the process simply spits out a message and terminates.

#include <windows.h>
#include <stdio.h>
HANDLE wh;
void
message(void)
{
SetEvent(wh);
ExitThread(0);
}
int
main(int argc, char **argv)
{
unsigned char *abki;
HMODULE dlh;
wh = CreateEvent(NULL, FALSE, FALSE, NULL);
dlh = GetModuleHandle("ntdll.dll");
abki = (void *)GetProcAddress(dlh, "DbgUiRemoteBreakin");
if (abki != NULL) {
DWORD olp;
if (VirtualProtect(abki, 20, PAGE_EXECUTE_READWRITE, &olp)) {
*abki++ = 0xb8; // Mov EAX,
*(DWORD *)abki = (DWORD)message;
abki+=4;
*(WORD *)abki = 0xd0ff; // call EAX
} else {
printf("Page Protection Failed\n");
return (1);
}
}
WaitForSingleObject(wh, INFINITE);
printf("Debugger\n");
return (0);
}

This is the beginning of something bigger.