Introduction to Windows API

The purpose of this page is to go through some of the Windows API concepts, how these functions are called, how errors are handled, and some naming conventions that Microsoft uses that are not always immediately clear.

Windows API Documentation

The Windows documentation related to their API is generally clear, but it helps to understand how to read it. For example, consider the documentation for CreateThread 1.

It starts by explaining what the function does, followed by the syntax to use to call it:

1
HANDLE CreateThread(
2
  [in, optional]  LPSECURITY_ATTRIBUTES   lpThreadAttributes,
3
  [in]            SIZE_T                  dwStackSize,
4
  [in]            LPTHREAD_START_ROUTINE  lpStartAddress,
5
  [in, optional]  __drv_aliasesMem LPVOID lpParameter,
6
  [in]            DWORD                   dwCreationFlags,
7
  [out, optional] LPDWORD                 lpThreadId
8
);

There is the return type (HANDLE, more on this later), followed by the name of the function (for instance CreateThread or CreateProcess 2) and the list of arguments it accepts.
Each argument has an in/out annotation and possibly optional. The in defines that this parameter is passed as an argument to the function and the value is never changed by the function but only used. out, on the other hand, are always pointers and are used to allow the caller of the function to extract data that the function defines. For example, in CreateThread the return value is a HANDLE to the thread, but the caller might need to get the ID of the thread. C does not allow returning multiple values, so the usual convention is to have the return value of a function be an error code or a handle, and additional data are returned in the out pointers. The caller can then take the value pointed to by the pointer and use it. It is optional because if the caller doesn’t need to use it, they can avoid passing it to the function.

Functions can have multiple out arguments like CryptDecrypt 3. This function also introduces another possibility: arguments that are both input and output.

1
BOOL CryptDecrypt(
2
  [in]      HCRYPTKEY  hKey,
3
  [in]      HCRYPTHASH hHash,
4
  [in]      BOOL       Final,
5
  [in]      DWORD      dwFlags,
6
  [in, out] BYTE       *pbData,
7
  [in, out] DWORD      *pdwDataLen
8
);

In this case, pbData is the buffer containing the bytes that need to be decrypted, and pdwDataLen is the number of bytes. When the decryption function is executed, the decrypted data is saved in pbData and the new size in pdwDataLen.

The documentation then continues with arguments which are pretty straightforward. One point worth noting is that some arguments, usually with the flags word in the name, allow multiple values to be passed separated by the | (bitwise OR) operator. Arguments separated by the OR operator are meant to be read as “both x and y”. For example, two of the available flags for the CreateFile function are GENERIC_READ and GENERIC_WRITE. To request both read and write access to the file, the flags would be specified as GENERIC_READ | GENERIC_WRITE.

The return value section of the documentation defines what the returned value means; only the return value, not the out arguments. This section often includes details on the error codes that are returned and how to access them. This is discussed later.

The remark section might be useful at times, but mostly for developers, as it includes specific information about the function. For example, CreateFile seems to be used to create a file, however it does support opening a file as well. The remark section explains why and the capabilities available.

API Names

The main point of this paragraph is to explain API calls that have leading or trailing capital letters. Before that, it is worth noting that assuming what a function does based only on the name can be misleading. For example, CreateFile might not create the file at all, but only open it. There are other examples, like GetProcAddress that seems like it might return the address of a process; however, it returns the address of a procedure (function). The documentation is usually the most reliable reference until that specific function becomes familiar.

Also, additional attention to leading and trailing letters is useful, as they might change the function slightly. The most popular ones are:

• suffix Ex → which means Extended. The extended version introduces new functionalities and can change the way you call the function. For example ReadFile and ReadFileEx.

Note how the lpBuffer is not optional in ReadFile but it is in ReadFileEx, lpOverlapped is both input and output in the extended version but not on the standard one, and the last argument is completely different.

1
BOOL ReadFile(
2
  [in]                HANDLE       hFile,
3
  [out]               LPVOID       lpBuffer,
4
  [in]                DWORD        nNumberOfBytesToRead,
5
  [out, optional]     LPDWORD      lpNumberOfBytesRead,
6
  [in, out, optional] LPOVERLAPPED lpOverlapped
7
);
8

9
BOOL ReadFileEx(
10
  [in]            HANDLE                          hFile,
11
  [out, optional] LPVOID                          lpBuffer,
12
  [in]            DWORD                           nNumberOfBytesToRead,
13
  [in, out]       LPOVERLAPPED                    lpOverlapped,
14
  [in]            LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
15
);

• suffix W → This doesn’t usually bring new features, except that input/output strings are wide-chars to allow for UTF-8/UTF-16 glyphs. An example is MessageBoxW.
• suffix A → This indicates that the strings arguments are ASCII. An example is MessageBoxA.
• prefix Zw and Nt → those are functions that are respectively the kernel-mode and user-mode implementations. They are, in theory, not meant to be used by developers as their implementation might change without notice; however, malware authors often use them to avoid detection. It might happen that the documentation of those functions is not available or redacted; for example, the SYSTEM_PROCESS_INFORMATION returned by the NtQuerySystemInformation has certain arguments redacted, see 4. Seeing these functions in pestudio often raises suspicion.

Windows Data Types

Looking at any of the functions used above, it is noticeable that the types of the arguments are not like those known by a C programmer. For example:

1
BOOL ReadFileEx(
2
  [in]            HANDLE                          hFile,
3
  [out, optional] LPVOID                          lpBuffer,
4
  [in]            DWORD                           nNumberOfBytesToRead,
5
  [in, out]       LPOVERLAPPED                    lpOverlapped,
6
  [in]            LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
7
);

None of HANDLE, LPVOID, DWORD or LPOVERLAPPED are common outside Windows. These are in fact either struct types defined in the Windows library or just simple aliases. To explain a few aliases:

• DWORD is the alias of unsigned long
• BYTE is the alias of unsigned char
• SIZE_T is the alias of unsigned long long
• BOOL is the alias of int
• NTSTATUS is the alias of long

There are many more than those listed above, but the point is that it is useful to become familiar with these types, and a full reference is given in 5.

Certain structs, like HANDLE, have a deeper meaning in the operating system than other structs like BYTE. In the next chapter, handles are discussed in more detail. For more on mutexes see the learn section Mutex.

Handles

A HANDLE type is nothing more than a pointer. It is defined as a PVOID meaning a pointer to void (void * in C). How can something be a pointer to void? This is a way to say that the pointer returned can be of any type, depending on the function that returned it. Internally it is mapped to a specific kernel object that the user does not need to care about. Of course, that might make it a bit harder to understand which type of HANDLE needs to be passed to a function; however, the documentation does a decent job in explaining it. A couple of examples of functions returning a HANDLE are:

1
HANDLE OpenProcess(
2
  [in] DWORD dwDesiredAccess,
3
  [in] BOOL  bInheritHandle,
4
  [in] DWORD dwProcessId
5
);
6

7

8
HANDLE CreateMutexA(
9
  [in, optional] LPSECURITY_ATTRIBUTES lpMutexAttributes,
10
  [in]           BOOL                  bInitialOwner,
11
  [in, optional] LPCSTR                lpName
12
);
13

14

15
HANDLE CreateFileA(
16
  [in]           LPCSTR                lpFileName,
17
  [in]           DWORD                 dwDesiredAccess,
18
  [in]           DWORD                 dwShareMode,
19
  [in, optional] LPSECURITY_ATTRIBUTES lpSecurityAttributes,
20
  [in]           DWORD                 dwCreationDisposition,
21
  [in]           DWORD                 dwFlagsAndAttributes,
22
  [in, optional] HANDLE                hTemplateFile
23
);

Three completely different meanings for the same object. Handles can be used to track down chains of execution; for example, if a sample opens multiple files and performs operations on them, the execution path might become hard to follow. However, looking at the HANDLE ID in the debugger usually helps.

Using a single unified object (HANDLE) for all the API calls that return a pointer type is a newer approach from Microsoft; previously, each function was returning a different HANDLE type, and some still do for backwards compatibility. For example RegCreateKeyExA 6, which returns a HKEY handle.

1
LSTATUS RegCreateKeyExA(
2
  [in]            HKEY                        hKey,
3
  [in]            LPCSTR                      lpSubKey,
4
                  DWORD                       Reserved,
5
  [in, optional]  LPSTR                       lpClass,
6
  [in]            DWORD                       dwOptions,
7
  [in]            REGSAM                      samDesired,
8
  [in, optional]  const LPSECURITY_ATTRIBUTES lpSecurityAttributes,
9
  [out]           PHKEY                       phkResult,
10
  [out, optional] LPDWORD                     lpdwDisposition
11
);

Once a handle is not needed anymore it is usually closed either with CloseHandle 7 or with a specific close function like InternetCloseHandle 8. There are some cases, like the HANDLE populated by EnumProcessModules that must not be closed, as specified in the documentation 9:

So, when encountering phrases like “a handle to a file” or “a handle to a process”, this can be read as “something that uniquely defines and points to the file/process”. It is possible to act on it; for example WriteFile 10 expects as first argument a HANDLE representing the file it can write to:

1
BOOL WriteFile(
2
  [in]                HANDLE       hFile,
3
  [in]                LPCVOID      lpBuffer,
4
  [in]                DWORD        nNumberOfBytesToWrite,
5
  [out, optional]     LPDWORD      lpNumberOfBytesWritten,
6
  [in, out, optional] LPOVERLAPPED lpOverlapped
7
);

Windows API Constructs

In this section a few topics are discussed that, if not known, might seem confusing, especially for readers who are not C programmers or have never programmed using the Windows API.

Calling the same function twice or more

There are scenarios in which the same function is called twice, and often the second call is inside a loop. One example is in the Example section of the RegQueryValueExA documentation 11, reported below:

1
#include <windows.h>
2
#include <malloc.h>
3
#include <stdio.h>
4

5
#define TOTALBYTES    8192
6
#define BYTEINCREMENT 4096
7

8
void main()
9
{
10
    DWORD BufferSize = TOTALBYTES;
11
    DWORD cbData;
12
    DWORD dwRet;
13

14
    PPERF_DATA_BLOCK PerfData = (PPERF_DATA_BLOCK) malloc( BufferSize );
15
    cbData = BufferSize;
16

17
    printf("\nRetrieving the data...");
18

19
    dwRet = RegQueryValueEx( HKEY_PERFORMANCE_DATA,
20
                             TEXT("Global"),
21
                             NULL,
22
                             NULL,
23
                             (LPBYTE) PerfData,
24
                             &cbData );
25
    while( dwRet == ERROR_MORE_DATA )
26
    {
27
        // Get a buffer that is big enough.
28

29
        BufferSize += BYTEINCREMENT;
30
        PerfData = (PPERF_DATA_BLOCK) realloc( PerfData, BufferSize );
31
        cbData = BufferSize;
32

33
        printf(".");
34
        dwRet = RegQueryValueEx( HKEY_PERFORMANCE_DATA,
35
                         TEXT("Global"),
36
                         NULL,
37
                         NULL,
38
                         (LPBYTE) PerfData,
39
                         &cbData );
40
    }
41
    if( dwRet == ERROR_SUCCESS )
42
        printf("\n\nFinal buffer size is %d\n", BufferSize);
43
    else printf("\nRegQueryValueEx failed (%d)\n", dwRet);
44
}

RegQueryValueEx is called once, but since the PerfData buffer might be too small to hold all the data stored in the registry key, it continues to be called with a bigger size every time until it succeeds inside the while loop. This is due to the fact that the size of the object being retrieved is unknown and there is no API that can give that value directly. Some APIs like EnumProcessModules specify this in their documentation:

Calling a First and a Next function

Sometimes functions like Process32First are followed by Process32Next. This is due to the fact that Microsoft sometimes uses a stateful enumeration pattern approach where the first call takes a snapshot of the processes, and the second call inside a loop returns each process. There are a few functions that behave this way:

• Process32First / Process32Next
• Thread32First / Thread32Next
• Module32First / Module32Next
• FindFirstFile / FindNextFile

An example usage of Process32First is shown in the learn section Process Enumeration.

Assembly Calling Conventions

When reading the decompiled assembly call from Ghidra, it is common to see functions preceded by a string like __thiscall, for example:

1
undefined  __thiscall  FUN_10001910 (void *  this ,  wchar_t)

The undefined is the return type that the decompiler was not able to understand, while the second string __thiscall is the calling convention. There are 4 main conventions that are commonly encountered. These conventions determine:

1. How to pass arguments to the function and which registers or stack to use
2. Where the return value is stored
3. Who cleans the stack used by the function

The conventions are summarized below:

Below is a comparison of the same function (FUN) which sums two numbers, with one example for each convention, and a more in-depth explanation of the convention.

cdecl

The arguments in a cdecl function are pushed on the stack in a very specific order, which is often mentioned as “from right to left”. If we were to look at the function in C we would have the signature as:

1
int FUN(int a, int b)

The rightmost argument, b, is the first one pushed on the stack, the leftmost is the last one.

In this convention, the caller clears the stack space used for the arguments after the function returns. printf is a good example of a cdecl function as it is variadic, so the callee cannot know how many arguments were passed. Therefore it uses a convention like cdecl where the caller cleans the stack, because the caller knows exactly how many arguments it pushed.

Looking at the FUN implementation, the function definition is very straightforward: it saves the base pointer (EBP), which is popped at the end so the program knows where to continue from once the function finishes. It moves the stack pointer to the base pointer to have a stable reference to the arguments and return values. At this point the stack looks like this:

1
[ebp+0]   = old EBP
2
[ebp+4]   = return address
3
[ebp+8]   = first argument (a)
4
[ebp+12]  = second argument (b)

The first element is loaded in EAX and the second is summed to it. It is moved to EAX because this is where a cdecl expects the return value to be, and since the registers do not clear up after the function finishes, the caller will be able to use EAX as needed.

1
FUN PROC
2
    push    ebp
3
    mov     ebp, esp
4
    mov     eax, [ebp+8]     ; a
5
    add     eax, [ebp+12]    ; + b
6
    pop     ebp
7
    ret                     ; no stack adjustment (caller cleans)
8
FUN ENDP

The caller will call the function pushing the arguments, and at the end of the function it cleans the stack of 8 bytes, 4 per argument, given that they are integers and take 4 bytes each in most of the implementations.

1
push b
2
push a
3
call FUN
4
add  esp, 8               ; caller cleans
5
; EAX = result

stdcall

stdcall is very similar to cdecl with the only exception that the function clears up the stack internally and the caller doesn’t have to care about it. At the end of the function it executes ret 8 which does two operations at once:

• by default it pops the return address into EIP
• if it is called with an operand, like 8 it cleans up the stack with ADD ESP, 8

1
FUN PROC
2
    push    ebp
3
    mov     ebp, esp
4
    mov     eax, [ebp+8]     ; a
5
    add     eax, [ebp+12]    ; + b
6
    pop     ebp
7
    ret     8               ; callee cleans 2 args (8 bytes)
8
FUN ENDP

The caller calls the function the same way as cdecl but after the call, the stack is already cleaned:

1
push b
2
push a
3
call FUN
4
; EAX = result

fastcall

In fastcall convention the first two arguments are stored in ECX and EDX while all the others, if needed, are pushed on the stack. In this specific function the ret call is used without any operand only due to the fact that no extra arguments were pushed onto the stack, so there is no need to clean it; in other functions it might be called like in the stdcall example.

1
FUN PROC
2
    ; a is already in ECX, b is already in EDX
3
    mov     eax, ecx         ; eax = a
4
    add     eax, edx         ; eax += b
5
    ret                      ; no stack args here, so plain ret
6
FUN ENDP

The caller initializes the ECX and EDX registers to be used by the function.

1
mov ecx, a
2
mov edx, b
3
call FUN
4
; EAX = result

thiscall

The thiscall convention is Microsoft-specific for C++, so it appears often in C++ samples. The this argument is passed via ECX, while the rest are pushed in the stack. this is a pointer to an instance of a struct, and inside the function the struct arguments are generally used or initialized. In the example below, the first argument of the struct (x) is used by dereferencing it [ecx]. If a second argument was needed it would be taken via [ecx + 4] or any other number based on the size of the previous arguments.

1
FUN PROC
2
    push    ebp
3
    mov     ebp, esp
4
    ; ECX = this
5
    mov     eax, [ecx]       ; eax = this->x
6
    add     eax, [ebp+8]     ; + b  (first stack arg)
7
    pop     ebp
8
    ret     4               ; callee cleans 1 stack arg (b)
9
FUN ENDP

The caller simply populates ECX and pushes any extra arguments on the stack:

1
mov ecx, thisPtr
2
push b
3
call FUN_thiscall
4
 ; EAX = result

Appendix