n this blog post I'm describing an approach to force the execution flow to enter into out of memory (OOM) error conditions when the amount of memory to allocate is not controlled by the attacker as in the example below.
#define MAX_SIZE 0x08000000 /* 128M */
ptr = malloc(MAX_SIZE)
if (ptr == NULL)
{
return OUT_OF_MEMORY;
}
When testing software, it's reasonable to think that error conditions like above are unlikely to be reached as often as other error conditions. Some of them may never get reached because it's not obvious to make the allocation with fixed-size to fail.
read more.............http://reversingonwindows.blogspot.com/2013/01/entering-into-out-of-memory-condition.html?spref=tw