Abstract
With the rapid rise in the number of exploits targeting
the Java runtime environment, new tools are required
to detect these malicious Java applications. This paper
proposes one such tool, the Java Malware Detector
(JMD). JMD takes a hybrid approach that combines
symbolic execution, instrumentation and dynamic
analysis to detect malware that subverts Java’s
access control mechanisms. Using this approach, we
aim to derive any trigger conditions that may exist
before instrumenting and executing the malware in
a controlled environment to observe whether it escapes
the Java security sandbox. A key element of
this approach is our use of existing open-source software
platforms—specifically, Java Pathfinder and AspectJ.
By using real-world Java malware samples we
are able to evaluate the effectiveness of JMD. The results
of this evaluation show that JMD’s instrumentation
and dynamic analysis capabilities provide an
effective tool for detecting a wide range of Java malware:
we successfully detected malware variants that
represent fourteen of the known access control-related
CVEs disclosed over the past four years. However, our
success in using symbolic execution to derive trigger
conditions was limited, mainly due to the incomplete
state of the String handling implementation in Java
Pathfinder’s symbolic execution plugin
more here..........http://crpit.com/confpapers/CRPITV161Herrera.pdf
With the rapid rise in the number of exploits targeting
the Java runtime environment, new tools are required
to detect these malicious Java applications. This paper
proposes one such tool, the Java Malware Detector
(JMD). JMD takes a hybrid approach that combines
symbolic execution, instrumentation and dynamic
analysis to detect malware that subverts Java’s
access control mechanisms. Using this approach, we
aim to derive any trigger conditions that may exist
before instrumenting and executing the malware in
a controlled environment to observe whether it escapes
the Java security sandbox. A key element of
this approach is our use of existing open-source software
platforms—specifically, Java Pathfinder and AspectJ.
By using real-world Java malware samples we
are able to evaluate the effectiveness of JMD. The results
of this evaluation show that JMD’s instrumentation
and dynamic analysis capabilities provide an
effective tool for detecting a wide range of Java malware:
we successfully detected malware variants that
represent fourteen of the known access control-related
CVEs disclosed over the past four years. However, our
success in using symbolic execution to derive trigger
conditions was limited, mainly due to the incomplete
state of the String handling implementation in Java
Pathfinder’s symbolic execution plugin
more here..........http://crpit.com/confpapers/CRPITV161Herrera.pdf