Heuristic definitions

The problem with malware definitions is that they are specifically for one type of malware. If you write a definition for a single malware sample, the writer will simply modify their malware or obfuscate it with an executable packer. In order to detect generic malware in a way that cannot be easily obfuscated, we can use heuristics.

Heuristics are rules that are designed to identify malware through analysis of the logical behaviour of an application instead of calculating definitive signatures like hashes. This may seem difficult to do in the case of malware that is not currently executing, since behaviour is derived from activity. However, it is possible through basic interpretation of code and analysis of API calls what a program will do.

Here's an example of code I took from a keylogger:

push 0
push eax
push 0x041102A0
push 0x0D

call kernel32.SetWindowsHookEx

The code here sets a keyboard hook on the system. The prototype of SetWindowsHookEx is as follows:

HHOOK WINAPI SetWindowsHookEx(
  __in  int idHook,
  __in  HOOKPROC lpfn,
  __in  HINSTANCE hMod,
  __in  DWORD dwThreadId
); 

In the stdcall calling convention (the standard convention for Windows) parameters are pushed from right to left. This means that in our sample code the idHook parameter is 0x0D (decimal 13), which specifies a low level keyboard hook. Since the other parameters are arbitrary (i.e. not specific to a keyboard hook) we can simply say "look for any call to SetWindowsHookEx and if the last push done before it has a value of 13, treat it as a possible keylogger". Of course there are other reasons to set up a low level key hook, such as processing global hotkeys, so we have to look at a few other things. We can follow the third push address (HOOKPROC lpfn) to the hook function and see what it does there (e.g. does it call file write operations?) to test decide if it really is a keylogger.