How do debuggers work? - P.1

Until now, although having more than 5 years working as a programmer, I actually have not understood what the hell the concept "debug" is and how it works. And hope that it's not too late for a investigation about the concept. Greatz thanks Alexander Sandler for ur article here
 
First of all, actual debugging requires operating system kernel support and here’s why. Think about it. We’re living in a world where one process reading memory belonging to another process is a serious security vulnerability. Yet, when debugging a program, we would like to access a memory that is part of debugged process’s (debuggie) memory space, from debugger process. It is a bit of a problem, isn’t it? We could, of course, try somehow to use same memory space for both debugger and debuggie, but then what if debuggie itself creates processes. This really complicates things.

Debugger support has to be part of the operating system kernel. Kernel able to read and write memory that belongs to each and every process in the system. Furthermore, as long as process is not running, kernel can see value of its registers and debugger have to be able to know values of the debuggie registers. Otherwise it won’t be able to tell you where the debuggie has stopped (when we pressed CTRL-C in gdb for instance).

As we spoke about where debugger support starts we already mentioned several of the features that we need in order to have debugging support in operating system. We don’t want just any process to be able to debug other processes. Someone has to monitor debuggers and debuggies. Hence the debugger has to tell the kernel that it is going to debug certain process and kernel has to either permit or deny this request. Therefore, we need an ability to tell the kernel that certain process is a debugger and it is about to debug other process. Also we need an ability to query and set values from debuggie’s memory space. And we need an ability to query and set values of the debuggie’s registers, when it stops.

And operating system lets us to do all this. Each operating system does it in it’s manner of course. Windows provides single system call named CreateProcess() (with the argument dwCreationFlags begins with DEBUG_...) (And ptrace() in Linux) , which allows to do all these operations and much more.
  • Types of Windows Debuggers
    • User-Mode Debugger
User-mode debuggers are used to debug any application that runs in user mode, which includes any GUI programs as well as applications you wouldn't expect, such as Windows 2000 services. Generally, user-mode debuggers use GUIs. The main hallmark of user-mode debuggers is that they use the Win32 Debugging application programming interface (API). Because the operating system marks the debuggee as running in a special mode, you can use the IsDebuggerPresent API function to find out whether your process is running under a debugger.

The Win32 Debugging API comes with an implied contract: once a process is running under the Debugging API, thus making it a debuggee, the debugger can't detach from that process. This symbiotic relationship means that if the debugger ends, the debuggee will end as well. The debugger is also limited to debugging only the debuggee and any processes spawned by the debuggee (if the debugger supports descendant processes).
For interpreted languages and run times that use a virtual machine approach, the virtual machines themselves provide the complete debugging environment and don't use the Win32 Debugging API. Some examples of those types of environments are the Microsoft or Sun Java Virtual Machines (VMs), the Microsoft scripting environment for Web applications, and the Microsoft Visual Basic p-code interpreter.

A surprising number of programs use the Win32 Debugging API. These include the Visual C++ debugger, the Windows Debugger (WinDBG), which I discuss in the kernel-mode debugger section that follows; Compuware NuMega's BoundsChecker; the Platform SDK HeapWalker program; the Platform SDK Depends program; the Borland Delphi and C++ Builder debuggers; and the NT Symbolic Debugger (NTSD). I'm sure there are many more.
    • Kernel-Mode Debugger
Kernel-mode debuggers sit between the CPU and the operating system. That means that when you stop in a kernel-mode debugger, the operating system also stops completely. As you can imagine, bringing the operating system to an abrupt halt is helpful when you're working on timing and synchronization problems. Except for one kernel-mode debugger, however, you can't debug user-mode code with kernel-mode debuggers.

There aren't that many kernel-mode debuggers. A few are the Windows 80386 Debugger (WDEB386), the Kernel Debugger (i386KD), WinDBG, and SoftICE. 
  • Simple Win32 Debugger
As said already, we can use CreateProcess and pass a special flag in the dwCreationFlags parameter: DEBUG_ONLY_THIS_PROCESS (or DEBUG_PROCESS if can handle multiple processes spawned by the initial debuggee).

Another requirement is that after the debuggee starts, the debugger must enter into a loop calling the WaitForDebugEvent API function to receive debugging notifications. When it has finished processing a particular debugging event, it calls ContinueDebugEvent. Be aware that only the thread that called CreateProcess with the special debug creation flags can call the Debugging API functions. The following pseudocode shows just how little code is required to create a Win32 debugger:
void main ( void )
{
    CreateProcess ( ..., DEBUG_ONLY_THIS_PROCESS ,... ) ;
 
    while ( 1 == WaitForDebugEvent ( ... ) )
    {
        if ( EXIT_PROCESS )
        {
            break ;
        }
        ContinueDebugEvent ( ... ) ;
 }
}
During the time a debugger sits in the debug loopit receives various notifications that certain events took place in the debuggee. The following DEBUG_EVENT structure, which is filled in by the WaitForDebugEvent function, contains all the interesting information about a debug event.
typedef struct _DEBUG_EVENT {
    DWORD dwDebugEventCode;
    DWORD dwProcessId;
    DWORD dwThreadId;
    union {
        EXCEPTION_DEBUG_INFO Exception;
        CREATE_THREAD_DEBUG_INFO CreateThread;
        CREATE_PROCESS_DEBUG_INFO CreateProcessInfo;
        EXIT_THREAD_DEBUG_INFO ExitThread;
        EXIT_PROCESS_DEBUG_INFO ExitProcess;
        LOAD_DLL_DEBUG_INFO LoadDll;
        UNLOAD_DLL_DEBUG_INFO UnloadDll;
        OUTPUT_DEBUG_STRING_INFO DebugString;
        RIP_INFO RipInfo;
    } u;
} DEBUG_EVENT;
 
When the debugger is processing the debug events returned by WaitForDebugEvent, it has full control over the debuggee because the operating system stops all the threads in the debuggee and won't reschedule them until ContinueDebugEvent is called. If the debugger needs to read from or write to the debuggee's address space, it can use ReadProcessMemory and WriteProcessMemory. If the memory is marked as read-only, you can use the VirtualProtect function to reset the protection levels. If the debugger patches the debuggee's code via a call to WriteProcessMemory, it must call FlushInstructionCache to clear out the instruction cache for the memory. If you forget to call FlushInstructionCache, your changes might work, but if the memory you changed is currently in the CPU cache, it might not. Calling FlushInstructionCache is especially important on multiprocessor machines. If the debugger needs to get or set the debuggee's current context or CPU registers, it can call GetThreadContext or SetThreadContext.

The only Win32 debug event that needs special handling is the loader breakpoint. After the operating system sends initial CREATE_PROCESS_DEBUG_EVENT and LOAD_DLL_DEBUG_EVENT notifications for the implicitly loaded modules, the debugger receives an EXCEPTION_DEBUG_EVENT. This debug event is the loader breakpoint. (When this exception occurs in the first time, just check if the value in ExceptionCode field is EXCEPTION_BREAKPOINT, we can now safety ensure that this exception happens when the debuggee was going to execute its very first instruction). The debuggee executes this breakpoint because the CREATE_PROCESS_DEBUG_EVENT indicates only that the process was loaded, not that it was executed. The loader breakpoint, which the operating system forces each debuggee to execute, is the first time the debugger knows when the debuggee is truly running. In real-world debuggers, the main data structure initialization, such as for symbol tables, is handled during process creation, and the debugger starts showing code disassembly or doing necessary debuggee patching in the loader breakpoint.

When the loader breakpoint occurs, the debugger should record that it saw the breakpoint so that the debugger can handle subsequent breakpoints accordingly. The only other processing needed for the first breakpoint (and for all breakpoints in general) depends on the CPU. For the Intel Pentium family, the debugger has to continue processing by calling ContinueDebugEvent and passing it the DBG_CONTINUE flag so that the debuggee resumes execution.

Here is the code for a small sample debugger from Debugging Applications (John Robbins):
/*----------------------------------------------------------------------
The world’s simplest debugger for Win32 programs
----------------------------------------------------------------------*/

/*//////////////////////////////////////////////////////////////////////
                           The Usual Includes
//////////////////////////////////////////////////////////////////////*/
#include "stdafx.h"
/*//////////////////////////////////////////////////////////////////////
                               Prototypes
//////////////////////////////////////////////////////////////////////*/
// Shows the minimal help.
void ShowHelp ( void ) ;

// Display functions 
void DisplayCreateProcessEvent ( CREATE_PROCESS_DEBUG_INFO & stCPDI ) ;
void DisplayCreateThreadEvent ( CREATE_THREAD_DEBUG_INFO & stCTDI ) ; 
void DisplayExitThreadEvent ( EXIT_THREAD_DEBUG_INFO & stETDI ) ; 
void DisplayExitProcessEvent ( EXIT_PROCESS_DEBUG_INFO & stEPDI ) ; 
void DisplayDllLoadEvent ( LOAD_DLL_DEBUG_INFO & stLDDI ) ;
void DisplayDllUnLoadEvent ( UNLOAD_DLL_DEBUG_INFO & stULDDI ) ; 
void DisplayODSEvent ( HANDLE hProcess , OUTPUT_DEBUG_STRING_INFO & stODSI    ) ; 
void DisplayExceptionEvent ( EXCEPTION_DEBUG_INFO & stEDI ) ;

*//////////////////////////////////////////////////////////////////////
                              Entry Point!
//////////////////////////////////////////////////////////////////////*/ 
void main ( int argc , char * argv[ ] )
{
    // Check that there is a command-line argument.
    if ( 1 == argc )
    {
        ShowHelp ( ) ;
        return ;
    }

    // Concatenate the command-line parameters.
    TCHAR szCmdLine[ MAX_PATH ] ;
    szCmdLine[ 0 ] = '\0' ;

    for ( int i = 1 ; i < argc ; i++ )
    {       strcat ( szCmdLine , argv[ i ] ) ;
        if ( i < argc )
        {
            strcat ( szCmdLine , " " ) ;
        }
    }

    // Try to start the debuggee process. The function call looks
    // like a normal CreateProcess call except for the special start 
    // option flag DEBUG_ONLY_THIS_PROCESS.
    STARTUPINFO         stStartInfo     ;
    PROCESS_INFORMATION stProcessInfo   ;

    memset ( &stStartInfo   , NULL , sizeof ( STARTUPINFO         ) ) ;
    memset ( &stProcessInfo , NULL , sizeof ( PROCESS_INFORMATION ) ) ;

    stStartInfo.cb = sizeof ( STARTUPINFO ) ;

    BOOL bRet = CreateProcess ( NULL                      ,
                                szCmdLine                 ,
                                NULL                      ,
                                NULL                      ,
                                FALSE                     ,
                                CREATE_NEW_CONSOLE |
                                  DEBUG_ONLY_THIS_PROCESS ,
                                NULL                      ,
                                NULL                      ,
                                &stStartInfo              ,
                                &stProcessInfo             ) ;

    // See whether the debuggee process started.
    if ( FALSE == bRet )
    {
        printf ( "Unable to start %s\n" , szCmdLine ) ;
        return ;
    }

    // The debuggee started, so let’s enter the debug loop.
    DEBUG_EVENT stDE                      ;
    BOOL        bSeenInitialBP   = FALSE  ;
    BOOL        bContinue        = TRUE   ;
    HANDLE      hProcess         = INVALID_HANDLE_VALUE ;
    DWORD       dwContinueStatus          ;

    // Loop until told to stop.
    while ( TRUE == bContinue )
    {
        // Pause until a debug event notification happens.
        bContinue = WaitForDebugEvent ( &stDE , INFINITE ) ;

        // Handle the particular debug events. Because MinDBG is only a
        // minimal debugger, it handles only a few events.
        switch ( stDE.dwDebugEventCode )
        {
            case CREATE_PROCESS_DEBUG_EVENT   :
            {
                DisplayCreateProcessEvent ( stDE.u.CreateProcessInfo ) ;
                // Save the handle information needed for later.
                hProcess = stDE.u.CreateProcessInfo.hProcess ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;
            case EXIT_PROCESS_DEBUG_EVENT   :
            {
                DisplayExitProcessEvent ( stDE.u.ExitProcess ) ;
                bContinue = FALSE ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;

            case LOAD_DLL_DEBUG_EVENT    :
            {
                DisplayDllLoadEvent ( stDE.u.LoadDll ) ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;
            case UNLOAD_DLL_DEBUG_EVENT  :
            {
                DisplayDllUnLoadEvent ( stDE.u.UnloadDll ) ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;

            case CREATE_THREAD_DEBUG_EVENT  :
            {
                DisplayCreateThreadEvent ( stDE.u.CreateThread ) ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;
            case EXIT_THREAD_DEBUG_EVENT    :
            {
                DisplayExitThreadEvent ( stDE.u.ExitThread ) ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;

            case OUTPUT_DEBUG_STRING_EVENT  :
            {
                DisplayODSEvent ( hProcess , stDE.u.DebugString ) ;
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;
            case RIP_EVENT  :
            {
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;

            case EXCEPTION_DEBUG_EVENT      :
            {
                DisplayExceptionEvent ( stDE.u.Exception ) ;

                // The only exception that I have to treat specially is
                // the initial breakpoint the loader provides.
                switch ( stDE.u.Exception.ExceptionRecord.ExceptionCode )
                {
                    case EXCEPTION_BREAKPOINT :
                    {

                        // If a breakpoint exception occurs and it’s the 
                        // first seen, I continue on my merry way;
                        // otherwise, I pass the exception on to the
                        // debuggee.
                        if ( FALSE == bSeenInitialBP )
                        {
                            bSeenInitialBP = TRUE ;
                            dwContinueStatus = DBG_CONTINUE ;
                        }
                        else
                        {
                            // Houston, we have a problem!
                            dwContinueStatus =
                                             DBG_EXCEPTION_NOT_HANDLED ;
                        }
                    }
                    break ;

                    // Just pass on any other exceptions to the 
                    // debuggee.
                    default : 
                    {
                        dwContinueStatus = DBG_EXCEPTION_NOT_HANDLED ;
                    }
                    break ;
                }
            }
            break ;

            // For any other events, just continue on.
            default :
            {
                dwContinueStatus = DBG_CONTINUE ;
            }
            break ;
        }

        // Pass on to the operating system.
        ContinueDebugEvent ( stDE.dwProcessId ,
                             stDE.dwThreadId  ,
                             dwContinueStatus  ) ;

    }
}

/*//////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////*/
void ShowHelp ( void )
{
    printf ( "MinDBG  "
             "\n" ) ;
}

void DisplayCreateProcessEvent ( CREATE_PROCESS_DEBUG_INFO & stCPDI )
{
    printf ( "Create Process Event      :\n" ) ;
    printf ( "   hFile                  : 0x%08X\n" , 
             stCPDI.hFile                            ) ;
    printf ( "   hProcess               : 0x%08X\n" , 
             stCPDI.hProcess                         ) ;
    printf ( "   hThread                : 0x%08X\n" , 
             stCPDI.hThread                          ) ;
    printf ( "   lpBaseOfImage          : 0x%08X\n" , 
             stCPDI.lpBaseOfImage                    ) ;
    printf ( "   dwDebugInfoFileOffset  : 0x%08X\n" , 
             stCPDI.dwDebugInfoFileOffset            ) ;
    printf ( "   nDebugInfoSize         : 0x%08X\n" , 
             stCPDI.nDebugInfoSize                   ) ;
    printf ( "   lpThreadLocalBase      : 0x%08X\n" , 
             stCPDI.lpThreadLocalBase                ) ;
    printf ( "   lpStartAddress         : 0x%08X\n" , 
             stCPDI.lpStartAddress                   ) ;
    printf ( "   lpImageName            : 0x%08X\n" , 
             stCPDI.lpImageName                      ) ;
    printf ( "   fUnicode               : 0x%08X\n" , 
             stCPDI.fUnicode                         ) ;
}
void DisplayCreateThreadEvent ( CREATE_THREAD_DEBUG_INFO & stCTDI )
{
    printf ( "Create Thread Event       :\n" ) ;
    printf ( "   hThread                : 0x%08X\n" , 
             stCTDI.hThread                          ) ;
    printf ( "   lpThreadLocalBase      : 0x%08X\n" , 
             stCTDI.lpThreadLocalBase                ) ;
    printf ( "   lpStartAddress         : 0x%08X\n" , 
             stCTDI.lpStartAddress                   ) ;
}
void DisplayExitThreadEvent ( EXIT_THREAD_DEBUG_INFO & stETDI )
{
    printf ( "Exit Thread Event         :\n" ) ;
    printf ( "   dwExitCode             : 0x%08X\n" , 
             stETDI.dwExitCode                       ) ;
}

void DisplayExitProcessEvent ( EXIT_PROCESS_DEBUG_INFO & stEPDI )
{
    printf ( "Exit Process Event        :\n" ) ;
    printf ( "   dwExitCode             : 0x%08X\n" , 
             stEPDI.dwExitCode                       ) ;
}
void DisplayDllLoadEvent ( LOAD_DLL_DEBUG_INFO & stLDDI )
{
    printf ( "DLL Load Event            :\n" ) ;
    printf ( "   hFile                  : 0x%08X\n" , 
             stLDDI.hFile                            ) ;
    printf ( "   lpBaseOfDll            : 0x%08X\n" , 
             stLDDI.lpBaseOfDll                      ) ;
    printf ( "   dwDebugInfoFileOffset  : 0x%08X\n" , 
             stLDDI.dwDebugInfoFileOffset            ) ;
    printf ( "   nDebugInfoSize         : 0x%08X\n" , 
             stLDDI.nDebugInfoSize                   ) ;
    printf ( "   lpImageName            : 0x%08X\n" , 
             stLDDI.lpImageName                      ) ;
    printf ( "   fUnicode               : 0x%08X\n" , 
             stLDDI.fUnicode                         ) ;
}
void DisplayDllUnLoadEvent ( UNLOAD_DLL_DEBUG_INFO & stULDDI )
{
    printf ( "DLL Unload Event          :\n" ) ;
    printf ( "   lpBaseOfDll            : 0x%08X\n" , 
             stULDDI.lpBaseOfDll                     ) ;
}
void DisplayODSEvent ( HANDLE                     hProcess ,
                       OUTPUT_DEBUG_STRING_INFO & stODSI    )
{
    printf ( "OutputDebugString Event   :\n" ) ;
    printf ( "   lpDebugStringData      : 0x%08X\n" , 
             stODSI.lpDebugStringData                ) ;
    printf ( "   fUnicode               : 0x%08X\n" , 
             stODSI.fUnicode                         ) ;
    printf ( "   nDebugStringLength     : 0x%08X\n" , 
             stODSI.nDebugStringLength               ) ;
    printf ( "   String                 :\n" ) ;

    char szBuff[ 512 ] ;
    if ( stODSI.nDebugStringLength > 512 )
    {
        return ;
    }

    DWORD dwRead ;
    BOOL bRet ;
    bRet = ReadProcessMemory ( hProcess                   ,
                               stODSI.lpDebugStringData   ,
                               szBuff                     ,
                               stODSI.nDebugStringLength  ,
                               &dwRead                    ) ;
    printf ( "%s" , szBuff ) ;
}void DisplayExceptionEvent ( EXCEPTION_DEBUG_INFO & stEDI )
{
    printf ( "Exception Event           :\n" ) ;
    printf ( "   dwFirstChance          : 0x%08X\n" , 
             stEDI.dwFirstChance                     ) ;
    printf ( "   ExceptionCode          : 0x%08X\n" , 
             stEDI.ExceptionRecord.ExceptionCode     ) ;
    printf ( "   ExceptionFlags         : 0x%08X\n" , 
             stEDI.ExceptionRecord.ExceptionFlags    ) ;
    printf ( "   ExceptionRecord        : 0x%08X\n" , 
             stEDI.ExceptionRecord.ExceptionRecord   ) ;
    printf ( "   ExceptionAddress       : 0x%08X\n" , 
             stEDI.ExceptionRecord.ExceptionAddress  ) ;
    printf ( "   NumberParameters       : 0x%08X\n" , 
             stEDI.ExceptionRecord.NumberParameters  ) ;
}
(To be continued ...)


Greatz thanks to: