Monday, 31 January 2011

Structural decisions - Kernel notification, usermode action

In the "Pulling the digital rug out from under the malware's feet" article I posted a few months back, I mentioned the PsSetCreateProcessNotifyRoutine and PsSetCreateThreadNotifyRoutine APIs that allow a kernel mode driver to create a notification callback routine that is invoked whenever a process or thread is created or destroyed. There is in fact a third variety of this, PsSetLoadImageNotifyRoutine, which allows a notification callback to be created for executable image mapping events. Combining all three of these APIs provides a wonderful tool for profiling what the system is doing (w.r.t. process/thread objects) at any point in time.

Whilst a fully fledged kernel-mode module that allows for excellent protection against usermode malware by providing a direct method to interfere with their exection (i.e. avoid user mode security mechanisms such as ACLs), it would be infeasible to create such a KMD in this project. However, a KMD will be used to notify the service process of new processes, threads and the loading of executable images.

This kernel module has almost been completed, and is in late beta testing. Here's a screenshot of the DbgView output:



The next step will be to add kernel message notification code to the usermode service.

Sunday, 9 January 2011

Heuristic definitions

The problem with malware definitions is that they are specifically for one type of malware. If you write a definition for a single malware sample, the writer will simply modify their malware or obfuscate it with an executable packer. In order to detect generic malware in a way that cannot be easily obfuscated, we can use heuristics.

Heuristics are rules that are designed to identify malware through analysis of the logical behaviour of an application instead of calculating definitive signatures like hashes. This may seem difficult to do in the case of malware that is not currently executing, since behaviour is derived from activity. However, it is possible through basic interpretation of code and analysis of API calls what a program will do.

Here's an example of code I took from a keylogger:

push 0
push eax
push 0x041102A0
push 0x0D
call kernel32.SetWindowsHookEx

The code here sets a keyboard hook on the system. The prototype of SetWindowsHookEx is as follows:

HHOOK WINAPI SetWindowsHookEx(
  __in  int idHook,
  __in  HOOKPROC lpfn,
  __in  HINSTANCE hMod,
  __in  DWORD dwThreadId
); 

In the stdcall calling convention (the standard convention for Windows) parameters are pushed from right to left. This means that in our sample code the idHook parameter is 0x0D (decimal 13), which specifies a low level keyboard hook. Since the other parameters are arbitrary (i.e. not specific to a keyboard hook) we can simply say "look for any call to SetWindowsHookEx and if the last push done before it has a value of 13, treat it as a possible keylogger". Of course there are other reasons to set up a low level key hook, such as processing global hotkeys, so we have to look at a few other things. We can follow the third push address (HOOKPROC lpfn) to the hook function and see what it does there (e.g. does it call file write operations?) to test decide if it really is a keylogger.