2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference

on Embedded and Ubiquitous Computing (EUC)

Anti-Reverse-Engineering Tool of Executable Files on the Windows Platform

Luo Chenke1, Yuan Feng1, Gao Qiyuan1, Yang Jiateng1, Xu Jian1,2

1. Software College, Northeastern University, Shenyang 110169, China
2. State Key Laboratory of Information Security (Institute of Information Engineering, The Chinese Academy of Sciences),
Beijing 100093, China
corresponding author: xu jian, [email protected]

Abstract— In response to the problems of the Windows against analysis of reverse engineering. It can be very easy to
executable being reverse-analyzed easily, we introduce the decipher the secret key generation algorithm.
technology of INT3 breakpoint detection and characteristics
detection in the tail of the heap in the software anti-dynamic- (2) Monotonous Protection methods. Even if the developers
debugging. We also applied the multistage security policy which use some shelling technique as the protection mechanism, the
contains self-stored secret key decryption and the UKEY based diversity is apparently not enough. The attacker can still
decryption to the anti-static-analysis. When an attacker is trying remove the protection mechanism to achieve their malicious
to attack the program with the debugger, the detection thread
will check the existence of the INT3 breakpoint in the head of the
purpose by using some corresponding analysis technics,
key API and the debugging-characteristics in the tail of the heap. causing a huge loss to the developer of these commercial
If the analysis behavior is confirmed, the program will be software.
terminated. At the same time, the key function will be hidden in
(3) The little ability of dynamic debugging resistance.
exception handlers and the attacker will be unable to follow in
the key function. According to the multilevel security policy, the Dynamic debugging refers to tracking software running, using
program can be executed directly under low level policy, and it the built-in debugger of our integrated environment. The
should be authenticated by the U-KEY to execute under the high attacker often used this method to analyze the behavior of a
level policy. After the experiment of effectiveness and feasibility running software. If the software does not have any good anti-
test, we can conclude that this system is able to detect the dynamic debugging mechanism, it is likely for it to be cracked
execution environment and protect the software from reverse- by the attacker, and the purpose of malicious dissemination can
attacking effectively. be achieved.
Keywords: Anti-Reverse-Engineering; software attack; Windows (4) Although the existing VM shell technic is good enough
for software protection, but will still bring reduction of
efficiency, the increase of the software volume, and some other
I. INTRODUCTION INTRODUCTION issues. Also, the mapping relationship between some popular
VM shell OPCode and the actual assembly code has been
In recent years, there has been rapid progress in computer analyzed adequately. With the static analysis, the reduction
development [1, 2]. On the one hand, computer software has from OPCode to the actual assembly code can be implemented,
rapidly developed, along with the physical devices‘’ which means this method is not safe enough.puter monitoring
performance getting stronger and stronger, and has integrated and control platform.
into every corner of the society; On the other hand, with the
growing of the computer software functions, software II. DESING IDEA
development cost is rising, in the meantime the cost of copies is In order to meet the need of safety protection in the daily use
almost zero for those software pirates [3, 4]. It will become a of software, the overall design idea of our system is to use
disaster for most develop companies, even a deadly punch to more complex encryption technology, combined with a variety
the whole IT industry. Because this kind of rampant piracy will of anti-debugging techniques, and finally develop an anti-
seriously reduce the enthusiasm of the software developers and reverse and anti-debugging protection system with reasonable
the vested interests of software vendors. It will be difficult to structure, convenient way to use, wide application scope, an
achieve the effect if we just rely on the constraints of the harsh overall of safety and computing performance.
punishment of laws and regulations, or on masses themselves,
to curb the illegally copying and using of computer software. (1) Meet the basic needs of software security protection,
We must make it harder for implementation of those pirate and which refers to the implementation of software shelling,
abuse [5, 6]. encryption and decryption of key section and anti-debugging
functions.
By the analysis of the case of reverse attacks under Windows
platform, we have summed up several common problems of the (2) The realization of transparent software protection,
current protection mechanisms of the most software in use [7, guarantee that the system does not affect the normal operation
8]. of the original software, in the premise of increasing the
security level of the software.
(1) Shell protection algorithm is too simple that it can be
cracked easily after the analysis by attackers. The most (3) Friendly interface design, simple operation, and easy to
common software copyright protection technology in use is use. We can also use modular design for easily modifying and
called Name/Serial protection, which is not stable enough upgrading later.

978-1-5386-3221-5/17 $31.00 © 2017 IEEE 798

800
797
DOI 10.1109/CSE-EUC.2017.158
 III.DESIGN PRINCIPLE AND IMPLEMENTATION SCHEME
The PE file structure compiled by the VC++ compiler with
C/C++ language in Win 32 platform are shown in Figure 1.

Figure 3. Compress the encrypted program

Attach the compressed part to the last section of the packer

program, as shown in Figure 4.

Figure 1. General C/C++ compiler PE file structure

In the following part, we will discuss about packer program

which provides protection for the original software and the
shell program which recovers the packed program and get it
running.
A. Paker prgram
The packer program will encrypt the whole target program
using the AES-256 encryption algorithm to avoid the case that
many sensitive information being left because of the old
packing method encrypting sections one by one. It is
noteworthy that the AES-256 encryption algorithm is a
relatively safe encryption algorithm, mostly used in the
commercial block encryption, and it is difficult to get the Figure 4. Attack the compressed part to the shell program
original program if the attackers have only got the encrypted There are two options of how the AES-256 key gets stored in
edition. The process of encrypting the original program is given our multilevel security policy.
in Figure 2.
(1) scatter the secret key in various parts of the program,
read out all of them and automatically decrypt and run the
program at run time;
(2) Encrypt the key of AES128-CTR using the public key of
RSA-2048, and store the private key in the U-KEY, then
scatter the encrypted AES128-CTR key and store them in the
program. At the runtime, read and combine the scatter secrete
key of AES128-CTR, decrypt it with private key of RSA-2048,
stored in a U-KEY, and finally decrypt the program with the
decrypted AES128-CTR secrete key.
The advantages of using the first security strategy are more
convenience. The protected program can be run directly when
in need, without the need of additional data. And the original
program is encrypted, which means it can not be disassembled
directly to the original program. The secrete key is scattered
and hidden in all parts of the program and not that easy to get.
Figure 2. Encrypting the original program However, the disadvantage is that the security level is not as
high as the second strategy, for after a mass of analysis of the
After the completion of encryption, the encrypted program shell program, there exists possibility for attackers to recover
will also be compressed by LZMA compression algorithm to the target program.
get smaller. The process of compressing the encrypted program
is given in Figure 3.

798
801
799
 The advantage of second kind of security policy is absolute Then we get the AES-128 secrete key, and be ready for
safety, because the secret key of AES is encrypted by RSA decryption.
algorithm, and the RSA private key which is used to decrypt
the AES key is stored in the U-KEY. Without the U-KEY If the higher level of security strategy is chosen, the program
providing the RSA secrete key, our protected program, in the will get the AES128-CTR key after reading out the RSA-2048
field of encryption and decryption, is very safe now, and there private key from a U-KEY and finishing decryption of the AES
is no way to bypass the private key of RSA and get the original key, as shown in Figure 7.
program directly. The RSA keys are randomly generated for
each time, and cannot be shared between several target
programs.
We will implement these two kinds of security policies, but
which one to choose is decided by our users, considering the
ease of use and security trade-offs in the face of different
programs.
B. Shell program (recovering the protected program)
The responsibility of shell program is to restore the protected Figure 7. Decrypt the AES key
program. From the point of view of shell program, how to
If the user choose the lower level security strategy, the
prevent the program from being debugged and how to restore
program can start working at decryption after it has read out the
the protected program are the two most important things. Here
scattered AES secrete key.
we discuss how to restore the original program execution, and
the issues related to anti-debugging will be discussed later. Next step is about the decryption of target program with
AES128-CTR key, as shown in Figure 8.
We introduced the progress substitution technic, creating
another shell progress in the shell program, and then release the
memory of the child progress, after that we decompress and
decrypt the protected program and write it to the progress
memory created earlier. We then run the new progress from the
entry point of it and start executing and finally achieve our goal
of program recovery and sensitive-information hiding.
First, create a progress and release its memory, as shown in
Figure 5.

Figure 8. Decrypt the target program.

After the decryption of the target program, the shell will

move it to the free memory released from the child progress,
and start it from then entry point. As shown in Figure 9.

Figure 5. Release memory after creating a new progress

Then decompress the .ENCDATA section, as in Figure 6.

Figure 9. Run the target program

At this time, the protected program has started running

Figure 6. Decompress target program using LZMA algorithm normally, and the shell program returns.

799
802
800
C. Measures of anti-debugging IV.CONCLUSIONS
The part of anti-debugging lies in the shell program, which In the context of high occupancy of Windows platform, with
will first start some anti-debugging work before the recovery of the development of computer performance and the increasing
target program. The multi-thread anti-debugging mechanism power of computer software, more and more attackers have
will start first and the main thread will hold through the whole gathered their interests to attacking the windows executable file.
process of recovery. To cope with this trend, we developed the anti-reverse
The anti-debugging technique consists of the following engineering tool for the protection of windows executable file,
aspects: combining dynamic encryption and decryption process,
progress replacement, API detection, multi-level security
(1) Anti-debugging in main thread policy, exception hiding and time interval detection and many
other technologies for the enhancement of software security,
Detect if the program is running under the debugger with providing anti-static analysis and anti-dynamic testing with
API IsDebuggerPresent; high efficiency from multi aspects. After several tests, we can
Detect if there exists any debugger with API confirm that the tool can resist most of the popular cracking
CheckRemoteDebuggerPresent. software, causing a lot of interference for most software with
analysis function, and can work effectively on confusing and
Detect if there exists any debugger with undocumented API misleading the attackers. It can greatly enhance the security
NtQueryInformationprogress. level without affecting the normal use of the target program,
Add GetTickCount to the begging and ending of a block of protecting the property rights and economic benefits of the
codes to calculate the time interval from system starting to developers and security of software users.
current timing. The operation speed under debugging case and
normal situation differs, especially when it comes to step ACKNOWLEDGMENT
through a program. If the time interval exceeds a critical value, The authors would like to thank the reviewers for their
we can judge that a debugging behavior is being performed, detailed reviews and constructive comments, which have
and the program will enter a dead loop or exit directly. helped improve the quality of this paper. This work was
supported in part by the Fundamental Research Funds for the
Add a SEH node to the SEH chain, the handler of which Central Universities under Grant No. N151704002.
refers to the function for decryption and decompression. The
program will ultimately trigger an exception and transfer the
controller to system. The operating system will return the REFERENCES
controller to the handler after doing a series of exception [1] Jian Xu, Fucai Zhou, Zhihao Jiang, Rui Xue. Dynamic proofs of
process, and perform decompression and decryption in handler. retrievability with square-root oblivious RAM[J], Journal of Ambient
In this case, the exception will be captured by debugger and the Intelligence and Humanized Computing, 2016,7(5):611-621.
attacker will follow to the system airspace, which can [2] XU Jian, LI Mingjie, LI Fuxiang, YANG Qingsong, ZHOU Fucai.
effectively prevent the attacker following to the decryption part Optimized Algorithms for Flexible Length-Based Authenticated Skip
and achieve our purpose of anti-debugging. List [J],China Communications,2016,13(1): 124-138
[3] Abdelhak Mesbah, Jean-Louis Lanet, Mohamed Mezghiche. Reverse
Check the tail of heap and exit the program if the debugging engineering a Java Card memory management algorithm[J], computers
characteristics are confirmed. & security,2017(66):97-114.
[4] Anas Shatnawi, Abdelhak-Djamel Seriai, Houari Sahraoui, Zakarea
(2) Anti-debugging in multithreading: Alshara. Reverse engineering reusable software components from
object-oriented APIs[J], The Journal of Systems and Software, 2016,1-
Create a process at the start of the shell program to check the 19.
first dozens of byte in the head of some key API, such as [5] Jing Qiu, Babak Yadegari, Brian Johannesmeyer. Framework for
CreateProcess, WriteProcessMemory, ResumeThread and so Understanding Dynamic Anti-Analysis Defenses[C], Proc of PPREW-
on, to see if there exists OXCC, or the INT3 break point. If 4,2014.
exists, the program exits to prevent reverse engineers tracing [6] Sina Shahbazomhamdi, Lei Wang, John Chandy, Mark Tehranipoor. A
back to key codes with API break point. Survey on Chip to System Reverse Engineering[J], ACM Journal on
Emerging Technologies in Computing Systems, 2016,13(1):
Anti-debugging progress is shown in Figure 10. [7] Paolo Ciancarini, Daniel Russo , Alberto Sillitti . Reverse Engineering:a
European IPR perspective[C],Proc of SAC 2016,2016, 1498-1503.
[8] Carlos Silva . Reverse Engineering of GWT Applications[C],Proc of
EICS 2012, 2012, 325-328.

Figure 10. Anti-debugging

800
803
801