An original entry point detection method with candidate-sorting for more effective generic unpacking

R Isawa, D Inoue, K Nakao - IEICE TRANSACTIONS on Information …, 2015 - search.ieice.org
R Isawa, D Inoue, K Nakao
IEICE TRANSACTIONS on Information and Systems, 2015search.ieice.org
Many malware programs emerging from the Internet are compressed and/or encrypted by a
wide variety of packers to deter code analysis, thus making it necessary to perform
unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system
named Justin that provides original entry point (OEP) candidates. Justin executes a packed
program, and then it extracts written-and-executed points caused by the decryption of the
original binary until it determines the OEP has appeared, taking those points as candidates …
Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.
search.ieice.org
Showing the best result for this search. See all results