Adversarial Attacks against Windows PE Malware Detection: A

Survey of the State-of-the-Art

Xiang Linga , Lingfei Wub , Jiangyu Zhangd , Zhenqing Qud , Wei Dengd , Xiang Chend ,
Yaguan Qianc , Chunming Wud , Shouling Jid , Tianyue Luoa , Jingzheng Wua,∗, Yanjun Wua,∗
a
Institute of Software, Chinese Academy of Sciences, Beijing, 100190, Beijing, China
b
Pinterest, New York, 10018, NY, USA
c
Zhejiang University of Science and Technology, Hangzhou, 310023, Zhejiang, China
d
Zhejiang University, Hangzhou, 310027, Zhejiang, China
arXiv:2112.12310v5 [cs.CR] 17 Feb 2023

Abstract
Malware has been one of the most damaging threats to computers that span across multiple operat-
ing systems and various file formats. To defend against ever-increasing and ever-evolving malware,
tremendous efforts have been made to propose a variety of malware detection that attempt to
effectively and efficiently detect malware so as to mitigate possible damages as early as possible.
Recent studies have shown that, on the one hand, existing machine learning (ML) and deep learn-
ing (DL) techniques enable superior solutions in detecting newly emerging and previously unseen
malware. However, on the other hand, ML and DL models are inherently vulnerable to adversarial
attacks in the form of adversarial examples, which are maliciously generated by slightly and care-
fully perturbing the legitimate inputs to misbehave. Adversarial attacks are initially studied in the
domain of computer vision like image classification, and then quickly extended to other domains,
including natural language processing, audio recognition, and even malware detection.
In this paper, we focus on malware with the file format of portable executable (PE) in the
family of Windows operating systems, namely Windows PE malware, as a representative
case to study the adversarial attack methods in such adversarial settings. To be specific, we
start by first outlining the general learning framework of Windows PE malware detection based
on ML/DL and subsequently highlighting three unique challenges of performing adversarial at-
tacks in the context of Windows PE malware. Then, we conduct a comprehensive and system-
atic review to categorize the state-of-the-art adversarial attacks against PE malware detection,
as well as corresponding defenses to increase the robustness of Windows PE malware detec-
tion. Finally, we conclude the paper by first presenting other related attacks against Windows
PE malware detection beyond the adversarial attacks and then shedding light on future research
directions and opportunities. In addition, a curated resource list of adversarial attacks and de-
fenses for Windows PE malware detection is also available at https://github.com/ryderling/
adversarial-attacks-and-defenses-for-windows-pe-malware-detection.
Keywords: Portable Executable, Malware Detection, Machine Learning, Adversarial Machine
Learning, Deep Learning, Adversarial Attack

∗
Corresponding Author.
Email addresses: [email protected] (Xiang Ling), [email protected] (Lingfei Wu),

Preprint submitted to Computer & Security February 20, 2023

Accepted by ELSEVIER Computers & Security.

1. Introduction

With the rapid development and advancement of information technology, computer systems
are playing an indispensable and ubiquitous role in our daily lives. Meanwhile, the cyberattack
that attempts to maliciously exploit the computer system with malicious intentions (e.g., dam-
aging computers or gaining economic profits) has been an important type of ever-increasing and
constantly evolving security threat in our society. Malware (i.e., short for Malicious software) is
one of the most common and powerful cyberattacks for attackers to perform malicious activities
in computer systems, such as stealing confidential information without permissions, compromis-
ing the whole system, and demanding for a large ransom. While malware spans across multiple
operating systems (e.g., Windows, Linux, macOS, Android, etc.) with various file formats, such
as portable executable (PE), executable and linkable format (ELF), Mach-O, Android application
package (APK), and portable document format (PDF), we focus on malware with the PE files in
the family of Windows operating systems (namely Windows PE malware) in this paper due to
the following two reasons. First, malware analysis techniques (e.g., detection methods) for Win-
dows PE files are mostly different from those for other files like APK and PDF files because their
underlying operating systems, the file format, and execution modes are significantly different from
each other. Research shows there is no universal malware detection that can satisfactorily detect
all kinds of malware, and thus existing literature papers on malware analysis commonly point out
what specific operating system they target and what file format they are [1, 2]. That is the very
first and most important reason why we focus on Windows PE malware in this paper. Second,
Windows is the most worldwide popular and long-standing operation system for end users while
the malware in the file format of PE constitutes the earliest and maximum studied threat in the
wild [3]. According to the statistics of Kaspersky Lab at the end of 2020, there are an average
of 360,000 malware detected by Kaspersky per day and over 90% of which are Windows PE mal-
ware [4]. Similar statistical trends have been reported by Kaspersky Lab at the end of 2021 [5],
indicating Windows PE files are still not sufficiently protected until now.
To mitigate and address the ever-increasing number of security threats caused by Windows PE
malware, there are tremendous research efforts have been made to detect Windows PE malware
effectively and efficiently [6, 7, 1, 2, 8]. In particular, traditional malware detection can be traced
back to signature-based malware detection, which determines whether a given suspicious software
is malicious or not (i.e., malware or goodware) by comparing its signature with all signatures
from the maintained database of malware that has been previously collected and confirmed. It is
obvious that the fatal flaw of signature-based malware detection is that it can only detect previously
collected and known malware due to the heavy reliance on the malware database. In the last few
decades, inspired by the great successes of ML and DL in various long-standing real-world tasks
(e.g., computer vision, natural language processing, speech recognition, etc.), a variety of ML/DL-
based malware detection methods [7, 1, 2, 8] that leverage the high capacity of ML/DL models
have been adapted and presented to detect Windows PE malware. In general, these ML/DL-based
malware detection methods claim that they can generalize well to predict the new and previously
unseen (i.e., zero-day) malware instances due to the inherent generalizability of ML/DL models.

[email protected] (Jiangyu Zhang), [email protected] (Zhenqing Qu), [email protected]

(Wei Deng), [email protected] (Xiang Chen), [email protected] (Yaguan Qian),
[email protected] (Chunming Wu), [email protected] (Shouling Ji), [email protected] (Tianyue Luo),
[email protected] (Jingzheng Wu), [email protected] (Yanjun Wu)

2
Accepted by ELSEVIER Computers & Security.

Unfortunately, recent studies have demonstrated that existing ML/DL models are inherently
vulnerable to adversarial attacks in the form of adversarial examples, which are maliciously gen-
erated by slightly and carefully perturbing the legitimate inputs to confuse the target ML/DL
models [9, 10]. Since the creation of adversarial attacks, most research papers focused on studying
adversarial attacks in the domain of computer vision [11], e.g., slightly and carefully perturbing
a “Persian cat” image x such that the resulting adversarial image x0 can be misclassified as a
“lionfish” by the target image classifier. Normally, in the context of images, most proposed adver-
sarial attacks resolve to the feature-space attack like various gradient-based methods, which can
be directly applied to generate adversarial images. Until now, there have been a large number
of adversarial attack methods and corresponding defense mechanisms being proposed by security
researchers and practitioners in both academia and industry [12, 13, 14, 15, 16].
Inspired by those studies of adversarial attacks in the context of images, a natural question
arises is that, is it feasible to perform adversarial attacks against existing malware detec-
tion methods, especially against ML/DL based PE malware detection? To answer the
aforementioned question, in recent five years, security researchers and practitioners have proposed
lots of adversarial attacks in the context of malware [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55],
requiring that the generated adversarial malware file should not only be misclassified as the “good-
ware” by the target malware detection model, but also behaves exactly the same as the original
malware file. Compared to the adversarial attack in the context of images, exploring the adver-
sarial attack in the context of malware is completely different and extremely challenging due to
the highly structured nature of software files like PE files. To put it simply, even though we can
employ existing feature-space attacks to generate the “adversarial features of malware”, it is sig-
nificantly challenging to find the corresponding “adversarial malware file” that can preserve the
format, executability, and maliciousness the same as the original malware file.
Related work to this survey. In fact, for the general adversarial attacks and defenses,
there are lots of surveys being done on the image, audio, video, text, and graph [11, 12, 13, 14,
15, 16, 56, 57, 58, 59], but very few surveys focusing on the adversarial attacks in the context of
malware [60, 61, 62]. Here, we introduce those closely related surveys and highlight their limitations
and differences compared with our survey paper as follows.

• In [60], Pierazzi et al. are the first to present a general mathematical formalization of adver-
sarial attacks in the problem-space and further propose a novel problem-space attack against
Android malware detection. Although they identify four key constraints and commonalities
among different domains (i.e., image classification, face recognition, code attribution, PDF
malware, android malware, etc.) in terms of the problem-space adversarial attack, this pa-
per [60] is not a survey paper as it neither extensively collects all existing research efforts
nor systematically categorizes and summarizes these efforts in this research direction.

• In [61], Park and Yener actually conduct an incomplete and improper survey in re-
viewing existing adversarial attacks against malware classifiers, since the authors mistakenly
categorize them into two categories: gradient-driven and problem-driven approaches, which
are clearly not sufficient to cover all existing adversarial attacks against malware detec-
tion. For instance, the semantics-preserving reinforcement learning (SRL) attack proposed
by Zhang et al. [38] is neither a gradient-driven nor a problem-driven method. In addition,
this paper [61] lacks surveying existing defense methods against such adversarial attacks.

3
Accepted by ELSEVIER Computers & Security.

• In [62], Li et al. first make a series of formulations and assumptions in the context of ad-
versarial malware detection and then attempt to survey this research field of adversarial
malware detection in a broad spectrum of malware formats, including Windows PE, Android
Package Kit (APK), and Portable Document Format (PDF), which is supposed to be too
coarse-grained to fully understand the unique challenges adversarial attacks and defenses
for different malware formats. It is well-known that malware detection relies heavily on
the specific file formats of malware, and thus existing literature papers on malware analysis
commonly point out what specific operating system they target and what file format they
are [1, 2]. Therefore, our paper instead focuses on the malware format of Windows PE,
which allows us to specifically identify the distinct characteristics of Windows PE malware
and further gain a deeper and thorough understanding of adversarial attacks and defenses in
terms of Windows PE malware.

Contributions of this survey. Motivated by the ever-increasing attention of adversarial

malware detection, the purpose of this survey is to provide a comprehensive review on the state-
of-the-art research efforts of adversarial attacks against Windows PE malware detection as well
as corresponding defenses to increase the robustness of existing PE malware detection solutions.
We expect that this survey can serve successive researchers and practitioners who are interested
in attacking and defending Windows PE malware detection. In addition, this survey also aims
to provide the basic principles to solve this challenging question in generating “real” adversar-
ial PE malware rather than unpractical adversarial PE features that violate the principles (i.e.,
format-preserving, executability-preserving, and maliciousness-preserving). We believe that these
principles can constitute a useful guideline when related researchers and practitioners deal with
the generic task of malware detection, not only restricted to PE malware detection. To summarize,
the key contributions of this survey are as follows.

• To the best of our knowledge, this is the first work that summarizes and highlights three
unique challenges of adversarial attacks in the context of Windows PE malware in the wild,
i.e., format-preserving, executability-preserving, and maliciousness-preserving.

• We conduct a comprehensive and systematic review for adversarial attacks against Windows
PE malware detection and propose a complete taxonomy to categorize the state-of-the-art
adversarial attack methods from different viewpoints.

• We summarize the existing adversarial defenses for PE malware detection against the pro-
posed adversarial attacks. In addition, we discuss other types of attacks against Windows
PE malware detection beyond adversarial attacks.

Organization. The rest of this paper is organized as follows. We introduce the general layout
of PE files and outline the ML/DL-based learning framework of PE malware detection in § 2. In § 3,
we manifest three unique challenges of adversarial attacks against PE malware detection compared
with the general adversarial attacks in the context of images. § 4 first presents our taxonomy of
adversarial attacks against PE malware detection and then gives a detailed introduction to the
state of the art. We summarize the existing adversarial defenses for PE malware detection in § 5.
In § 6, we first discuss other types of attacks against PE malware detection beyond the adversarial
attacks and then point out some research directions and possible opportunities for future work.
We conclude our survey paper in § 7.

4
Accepted by ELSEVIER Computers & Security.

2. Machine Learning and Deep Learning for PE Malware Detection

This section aims to provide the basics to understand how to take advantage of ML and DL
for malware detection in terms of PE files in the family of Windows operating systems (OSs). In
particular, we first introduce the general layout of PE files and PE malware in § 2.1, and then
outline the general learning framework of PE malware detection models based on ML/DL in § 2.2.

2.1. PE file Layout and Malware

2.1.1. General Layout of PE files
The Portable Executable (PE) format [63, 64] is created to
provide a common file format for the Microsoft family of Win-

Header Information
dows OSs to execute code and store essential data that is nec- DOS Header

essary to execute the code on all supported CPUs, which has

made PE the dominant file format of executables on the Win- DOS Stub

dows platform since its creation. As shown in Fig. 1, the general

PE Header
file layout of a PE contains three groups of information, i.e.,
header information (avocado green), section information
PE Optional
(sandy brown) and the other un-mapped data (light blue). Header
The first group is header information which is organized in
a fixed manner to tell the OS how to map the PE file into mem- Section Table

ory for execution. In particular, every PE file starts with the

DOS Header followed by the DOS Stub, both of which are only .text section
Section Information
used for compatibility with the Microsoft disk operating system
(MS-DOS) [65]. Following the DOS Header and DOS Stub, the .data section

PE Header outlines the overview information about the entire

.idata section
PE file, including the number of sections, the creation time, and
various other attributes of PE, etc. Besides that, the subsequent .edata section
PE Optional Header is used to supplementally describe PE files
.reloc section
with more than 30 attributes (e.g., size of code, address of entry
point, etc.) in more detail. The last component in the header .rsrc section
information group usually is the Section Table, which provides
information about all associated sections, including names, off- ……

sets, sizes, and other information. It is worth noting that one

PE needs at least one section to be loaded and run. Other Un-mapped
The second group is section information which contains a Data
list of consecutive sections with the executable code and neces-
sary data. In general, the number and the order of these sections Figure 1: The general layout of a
are not fixed. Although the name of a section can be customized PE file consists of three groups of
by the user, Microsoft officially defines several naming conven- information: header information,
section information, and the other
tions based on its semantics and functionalities. In most cases,
un-mapped data.
the “.text” section in PE contains the executable code, while the
“.data” section contains necessary data, mainly storing global
variables and static variables. The “.idata” and “.edata” sections are used to store the address
and size information for the import table and export table, respectively. To be specific, the import
table specifies the APIs that will be imported by this executable, while the export table specifies
its own functions so that other executables can access them. The “.reloc” section has relocation
5
Accepted by ELSEVIER Computers & Security.
2.2 Learning Framework for PE Malware Detection

information to ensure the executable is positioned and executed correctly. The “.rsrc” section
contains all resources (e.g., icons, menus, etc.). The last group is the other un-mapped data
which will not be mapped into memory. In particular, the un-mapped data refers to the chunk of
unused bytes, like debug information, at the end of PE files.
Within the family of Windows OSs, PE mainly has two typical and most commonly used file
types, i.e., EXEcutable (EXE) and Dynamic Link Library (DLL), which are generally ended with
“.exe” and “.dll” as the suffix name. Normally, an “.exe” file can be run independently while a “.dll”
file contains the library of functions that other executables can use in the Windows platform [64].

2.1.2. PE Malware
Malicious software, i.e., malware, is purposely designed and implemented to satisfy the mali-
cious goals and intentions of attackers, e.g., accessing the system without user permissions, stealing
private or confidential information, asking for a large ransom, etc. Since the PE file format was first
created in the family of Windows OSs, PE files have been widely leveraged by malicious attackers
to build PE malware. Until now, according to the security reports from AV-TEST Institute [66]
and Avira Protection Labs [67], PE malware still remains the predominant threat for both personal
users and business users in the wild for the following two major reasons. First, the worldwide pop-
ularity of the family of Windows OSs and the commonality of PE files inside make the family of
Windows OSs, especially the PE file that can be executed, become the main target of attackers for
benefit maximization. Second, unlike other file types, PE files can be self-contained, which means
that PE malware can include all needed data and does not require additional data to launch the
attack. In addition, based on different types of proliferation and different malicious intentions, PE
malware can be further briefly classified as viruses, trojans, PUA, worms, adware, ransomware,
etc. For more details, we refer interested readers to [68, 69, 7].

2.2. Learning Framework for PE Malware Detection

2.2.1. Overview
To defend against PE malware, i.e., effectively and efficiently detecting PE malware so that po-
tential infections and damages can be mitigated or stopped, there are tremendous research efforts
have been made for PE malware detection. Traditional malware detection can actually be traced
back to classic signature-based malware detection. To be specific, signature-based malware detec-
tion typically maintains a database of signatures of malware that have been previously collected
and confirmed. For a given suspicious software like a PE file, signature-based malware detection
can determine whether it is malicious or not by comparing its signature with all signatures from
the maintained database of malware. Obviously, signature-based malware detection has a fatal
drawback in that it heavily relies on the maintained database of malware signatures, so that it can
only detect previously collected and known malware.
In recent years, inspired by the great successes of ML and DL techniques in various research
fields, various ML/DL-based malware detection methods that leverage the high learning capacity of
ML/DL models have been adapted and proposed for PE malware detection. These ML/DL-based
malware detection methods normally claim that, as ML/DL models generalize well to predictions
of new and unseen instances, they can also generalize to new and previously unseen (i.e., zero-day)
malware. Fig. 2 illustrates the overview learning framework of PE malware detection [8], which
generally consists of three steps, including data acquisition, feature engineering as well as learning
from models and predictions. In the following, we are going to introduce each step at a glance.

6
Accepted by ELSEVIER Computers & Security.
2.2 Learning Framework for PE Malware Detection

Data Acquisition (Input) Feature Engineering Learning from Models and Predictions

Naïve Bayes

Machine Learning
Byte Sequence
EXE EXE EXE

Static
Readable String Decision Tree

Header Information Random Forest

EXE EXE EXE

…… SVM

EXE EXE EXE ……

System Resource Status

Dynamic
File Status MLP
EXE EXE EXE Problem Space Registry Status Feature Space Label Space

Deep Learning
CNN
…… RNN = {0,1}
EXE EXE EXE
Hybrid Opcode LSTM

System/API Calls Graph Neural Network

EXE EXE EXE
…… ……

Figure 2: The Overview Learning Framework of PE Malware Detection.

2.2.2. Data Acquisition

It is well known that the data quality determines the upper limit of ML/DL models [70]. In
order to build any malware detection models, it is fundamental to collect and label sufficient PE
samples, including both malware and goodware of PE files. However, unlike the benchmark datasets
in the field of computer vision or natural language processing (e.g., MNIST [71], CIFAR [72],
ImageNet [73], Yelp Review [74], etc.), cybersecurity companies or individuals normally treat PE
samples, especially raw files of PE malware, as private property, and thus rarely release them to
public. Although a few security institutions or individuals have shared their datasets of PE files [75,
76], there is no established benchmark that has been widely used for most PE malware detection
until now. In addition to collecting PE samples, another important process is to distinguish
PE malware from all collected PE files, i.e., data labeling. Currently, a common practice to
label PE files is to rely on malware analysis services like VirusTotal, which can provide multiple
detection results (i.e., whether it is malware) from nearly 70 state-of-the-art anti-virus engines [77].
Unfortunately, for the same suspicious PE sample, it is well-known that its detection results from
different anti-virus engines are somewhat inconsistent with each other. To address these challenges,
a variety of methods [78, 79, 80, 81] have been proposed to unify the detection label by either picking
up a “credible” anti-virus (i.e., the most effective and recognized anti-virus software like Kaspersky
or Norton) or adopting a voting-based approach from multiple anti-virus engines.
Formally, suppose there is a problem space Z (also known as input space) that contains
objects of one specific domain (e.g., PE files, images, texts, etc.) from the real-world applications,
each object z in the problem space Z (z ∈ Z) is associated with a ground-truth label y ∈ Y, in
which Y denotes the corresponding label space depending on the specific real-world application.
In the scenario of PE malware detection, the problem space Z is referred to the space of all possible
PE files, and the label space Y (e.g., Y = {−1, 1} or Y = {0, 1}) denotes the space of detection
labels (e.g., −1 or 0 denotes malware and 1 denotes goodware.)

2.2.3. Feature Engineering

After collecting and labeling sufficient PE samples, it is necessary and important to perform
somewhat feature engineering over all PE samples before inputting them into ML/DL models, as
ML/DL models can only accept numeric input. Feature engineering aims to extract the intrinsic
properties of PE files that are most likely to be used for distinguishing malware from goodware,
and then generates corresponding numeric features for representation. From different perspectives
7
Accepted by ELSEVIER Computers & Security.
2.2 Learning Framework for PE Malware Detection

Table 1: Common Features and Corresponding Learning Models for PE Malware Detection.
Category
Representative ML/DL Representative PE Malware
Features
Static

Dynamic Model Architectures Detection Methods∗

Byte Sequence X Naı̈ve Bayes [82], SVM [83], Schultz et al. [3], Kolter and
DT [84], MLP [85], CNN [86], Maloof [88], Saxe and Berlin
LightGBM [87]. [89], Gibert et al. [90],
MalConv [91], EMBER [92].
Readable String X Naı̈ve Bayes [82], SVM [83], Schultz et al. [3], SBMDS [94],
MLP [85], DT [84], RF [93], Islam et al. [95], Saxe and
LightGBM [87]. Berlin [89], EMBER [92].
Header
X MLP [85], LightGBM [87], Saxe and Berlin [89],
Information
Naı̈ve Bayes [82], SVM [83], EMBER [92], PE-Miner [96].
DT [84].
Grayscale Image X kNN [97], CNN [86], Nataraj et al. [98], Kim [99],
SVM [83]. Visual-AT [100].
CPU/Memory/IO
X CNN [86], kNN [97], Rieck et al. [101], AMAL [102],
etc. Status
SVM [83]. Abdelsalam et al. [103].
File Status X Hierarchical Clustering [104], Bailey et al. [106], Rieck et al.
CNN [86], kNN [97], [101], AMAL [102],
SVM [83], GNN [105]. Abdelsalam et al. [103],
MatchGNet [107].
Registry Status X Hierarchical Clustering [104], Bailey et al. [106], Rieck et al.
kNN [97], SVM [83]. [101], AMAL [102].
Network Status X Hierarchical Clustering [104], Bailey et al. [106], Rieck et al.
CNN [86], kNN [97], [101], AMAL [102],
SVM [83], GNN [105]. Abdelsalam et al. [103],
MatchGNet [107].
Opcode X X kNN [97], SVM [83], DT [84], AMCS [109], Santos et al.
RNN [108], CNN [86], [110], IRMD [111],
Hierarchical Clustering [104]. RMVC [112].
System or API
X X RIPPER [113], SVM [83], Schultz et al. [3], SBMDS [94],
Calls
Hierarchical Clustering [104], Rieck et al. [101], Qiao et al.
CNN [86], LSTM [114]. [115], Zhang et al. [116].
Control Flow
X X Naı̈ve Bayes [82], SVM [83], Kapoor and Dhavale [117],
Graph
RF [93], GNN [105]. MAGIC [118], MalGraph [119].
Function Call
X X RF [93], AutoEncoder [120], Hassen and Chan [121],
Graph
CNN [86], GNN [105]. DLGraph [122], DeepCG [123],
MalGraph [119].
∗ If the paper does not clearly name the PE malware detection, we use the author name(s) of the paper with its reference.

8
Accepted by ELSEVIER Computers & Security.
2.2 Learning Framework for PE Malware Detection

of properties of PE files, there is a large body of work on extracting various features, which can
be generally categorized into three broad category: static features, dynamic features and hybrid
features [7, 2, 8] and summarized in Table 1.
First of all, static features are directly extracted from the PE samples themselves without
actually running them. For instance, byte sequence, readable string, header information, and the
grayscale image are commonly used static features for PE malware detection.

• Byte Sequence: A PE sample is essentially a binary file, which is typically considered to be a

sequence of bytes. Therefore, the byte sequence is the most straightforward and informative
way to represent a PE file. In fact, the byte sequence can either be directly input into
DL models [91, 124, 125], or be further converted into an intermediate representation, e.g.,
n-grams or entropy of byte sequences [3, 88, 89, 90].

• Readable String: A PE file might contain readable strings that reflect it intentions or se-
mantics, like file names, IP addresses, domain names, author signatures, etc. After ex-
tracting readable strings in a PE file, their numeric feature representation can be a set
of binary attributes (i.e., whether the string exists), frequencies, or even 2D histogram fea-
tures [3, 94, 95, 89].

• Header Information: As illustrated in Fig. 1, the PE header information occupies an impor-

tant place to describe and normalize the PE file globally so that it can be executed properly.
In particular, simple statistics on PE header information, such as the file size, numbers of
sections, sizes of sections, the number of imported or exported functions, etc., are commonly
used feature representations for PE malware detection [89, 92, 96].

• Grayscale Image: Since the value range of bytes in a PE file is the same as the pixel value in
an image, a visualization-based feature engineering approach is to transform a PE file into a
grayscale image, for which each byte in a PE file corresponds to a pixel in an image [126].
Inspired by the recent great successes of image classification methods, a lot of visualization-
based methods have also been proposed for PE malware detection [98, 99, 100].

Second, dynamic features refer to those features that can be extracted by first running the
executable in an isolated environment (e.g., sandbox, virtual machine, etc.) and then monitoring
their runtime status in terms of system resources, files, registries, network, and others.

• System Resource Status: The execution of malware inevitably occupies system resources
(e.g., CPU, memory, IO, etc.), whose runtime status can be considered as dynamic features for
malware detection, as a variety of malware within one specific family might follow a relatively
fixed pattern of system resources during execution. In particular, CPU usage, memory usage,
and I/O request packets are commonly monitored as dynamic features [101, 102, 103].

• File Status: Malware normally needs to operate on files of target users for reaching malicious
intentions by attackers. Thus, logging and counting for the files accessed, created, modified,
or deleted are commonly used dynamic features in malware detection [106, 101, 102, 103, 107].

• Registry Status: Registries that store the system/application-level configurations are im-
portant for the family of Windows OSs. The malware could operate on registries with
malicious intentions, like self-starting malware. Similar to file status, registry status like
9
Accepted by ELSEVIER Computers & Security.
2.2 Learning Framework for PE Malware Detection

counting the registries created, modified, and deleted can also be regarded as dynamic fea-
tures [106, 101, 102].
• Network Status: The spread of malware like trojans and ransomware mainly depends on the
network. Taking trojans as an example, they are likely to connect remote servers with certain
network ports. Therefore, when diving into the specific aspects of network status, there is
a variety of network-level information that can be used for creating a rich set of dynamic
features [106, 101, 102, 103, 107], such as the number of distinct IP addresses or certain ports,
the number of different HTTP requests (e.g., POST, GET, HEAD, PUT, etc.), the number
of common DNS record types (e.g., PTR, CNAMN, SOA, etc.), to name just a few.
Finally, we exemplify four commonly used hybrid features, i.e., opcode, system/API calls,
control flow graph (CFG), and function call graph, which can be extracted from executables with
either static analysis methods or dynamic analysis methods. For instance, opcodes of executables
can be obtained by either extracting from their disassembled instructions or monitoring their
runtime instructions in memory.
• Opcode: Executables, including malware, can be generally considered as a collection of in-
structions that are executed in a specific order. In machine assembly language, an instruction
consists of an opcode and several operands, in which the opcode specifies the operation to be
executed and the operand refers to the corresponding data or its memory location. As prior
studies suggest, the opcode distributions of malware statistically differ from goodware, and
thus various features are constructed from the opcodes, such as their frequency, n-grams of
opcode sequences, or even opcode images [109, 110, 111, 112].
• System/API Calls: System/API calls refer to how executables interact with system-level
or application-level libraries in the family of Windows OSs. Similar to the opcode, various
feature representations are thus constructed from system/API calls, such as the frequency of
system/API calls, and n-grams of system/API call sequences [3, 94, 101, 115, 116].
• CFG: The CFG is a graph-based feature representation that is commonly used to characterize
the control flow of executables, including PE malware [117, 118]. Building from assembly in-
structions of executables, each node in the CFG represents a sequence of instructions without
branching and each edge represents the control flow path between two nodes [127].
• Function Call Graph: The function call graph [128] that attempts to build the caller-callee
relation between different functions (including system/API or user-implemented functions), is
regarded as a more coarse-grained graph representation compared with CFG [121, 122, 123].
It is worth noting that, on the one hand, we just briefly review and categorize the commonly
used features in PE malware detection, and do not attempt to cover all, which is not the goal of
our paper. On the other hand, all the features mentioned above are not separate or independent,
they are actually mixed for PE malware detection in the wild. In essence, the process of feature
engineering can be broadly expressed as a feature mapping function that maps the problem space
into the feature space (i.e., the numeric features), which is formulated in Def. 1 as follows.
Def. 1 (Feature Mapping Function). A feature mapping function φ is formulated as φ : Z →
X, in which Z denotes the problem space from a specific real-world application, and X denotes
the corresponding feature space, numerically describing the intrinsic properties of objects in the
problem space.
10
Accepted by ELSEVIER Computers & Security.

2.2.4. Learning from Models and Predictions

After extracting and generating the numeric features from the executable, it is necessary to
choose a proper ML or DL model for PE malware detection that is generally regarded as a binary
classification task, i.e., predicting whether the given executable is malware or goodware. In recent
years, with the rapid development and great success of artificial intelligence technology in many
fields like computer vision, natural language processing, and even code analysis [129], a huge variety
of ML/DL models have been continuously proposed, such as Naı̈ve Bayes, SVM, DT, RF, MLP,
CNN, RNN, LSTM, or GNN. Regardless of how diverse of feature representation for executables,
almost all kinds of ML/DL models mentioned have been used for PE malware detection, as long as
the features obtained from feature engineering conform to the input format of the corresponding
ML/DL model. For instance, an LSTM model can accept the sequence data as the input, but
cannot accept the graph data, while GNN can process the graph data.
In essence, an ML/DL model refers to a mathematical discrimination function f with pa-
rameters to map the numeric features of executables into their binary labels (i.e., malware and
goodware), which is broadly formulated in Def. 2 as follows.

Def. 2 (Discrimination Function). A discrimination function f can be precisely formulated as

f : X → Y, in which X denotes the feature space and Y denotes the corresponding label space.

The training process of a malware detection model is to learn the model parameters based on
a large number of training samples, so that the malware detection model can approximate the
real relationship function between the feature patterns of executables and their binary detection
labels. After that, to predict whether a given executable is malware or not, the malware detection
model with learned parameters can effectively and efficiently compute the probabilities assigned to
both classes of malware and goodware. In order to find the most applicable model, it is actually
quite common to test different ML/DL models for PE malware detection depending on the specific
task. In Table 1, the last two columns present the representative ML/DL model architectures and
corresponding PE malware detection methods with references.

3. Challenges of Adversarial Attacks for PE Malware

In this section, we first introduce the general concept and taxonomy of adversarial attacks that
have been originally and extensively studied in the domain of image classification tasks, and then
manifest the most unique challenges of adversarial attacks for PE malware when compared with
other fields like images, audios, texts, etc.

3.1. Adversarial Attacks: The General Concept and Taxonomy

Although recent advances in ML and DL have led to breakthroughs in a variety of long-standing
real-world tasks (e.g., computer vision, natural language processing, speech recognition, etc.), un-
fortunately, it has been convincingly demonstrated that existing ML and DL models are inherently
vulnerable to adversarial attacks with carefully crafted adversarial examples. In particular, adver-
sarial examples are intentionally and maliciously designed inputs that aim to mislead the given
target model in the testing phrase rather than the training phrase. Generally, adversarial attacks
can be categorized along multiple different dimensions. In the following part, we broadly classify
adversarial attacks along two dimensions, i.e., adversary’s space and adversary’s knowledge.

11
Accepted by ELSEVIER Computers
3.1 & Adversarial
Security. Attacks: The General Concept and Taxonomy

Problem Space Feature Space

z x
Images Executables
Feature Mapping

𝝓𝝓
EXE

Malware: 94.3%
Persian Cat: 95.7% Goodware: 5.7% Problem-space Feature-space
Attack Attack

EXE

𝝓𝝓−𝟏𝟏
Malware: 10.0%
Lion Fish: 99.8%
Goodware: 90.0%
Inverse
Feature Mapping
z' x'

Figure 3: Illustration of the connection between the feature-space attack and the problem-space attack, in
which the feature mapping function φ and the inverse feature mapping function φ−1 act as bridges for transitions
between the feature-space and the problem-space.

3.1.1. Adversary’s Space: Feature-space Attack versus Problem-space Attack

Initially, the adversarial example is explored in the context of image classification tasks and
is normally addressed in the feature-space domain. In particular, for a given image classification
model f and a given input image z with feature representation x ∈ X (i.e., f (x) = y), the attacker
attempts to minimize the distance between x0 and x in the feature-space, such that the resulting
adversarial example x0 ∈ X in the feature-space can misclassify the classification model f . This
kind of adversarial attack is normally termed as the feature-space attack, which is formulated
in Eq. (1) as follows.
min
0
distance(x0 , x)
x
s.t. x0 ∈ X (1)
f (x0 ) = y 0 6= y
in which distance(x1 , x2 ) denotes any measurement function of distance between x1 and x2 in
the feature-space depending on the actual applications. For the feature-space attacks, as the fea-
ture representation in the feature-space is normally continuous, most of them generate adversarial
examples based on gradients-base methods like FGSM, PGD, C&W, etc.
In contrast with the aforementioned feature-space attack, the problem-space attack refers to
the adversarial attack that is performed in the problem-space, i.e., how to “modify” the real-world
input z ∈ Z with a minimal cost (e.g., executables, source code, PDF,etc.) such that the generated
adversarial example z 0 ∈ Z in the problem-space can also misclassify the target model f as follows.

min
0
cost(z 0 , z)
z
s.t. z 0 ∈ Z (2)
0 0
f (φ(z )) = y 6= y

12
 3.2 Three Unique Challenges of Adversarial Attacks for PE Malware: From Feature-space to
Accepted by ELSEVIER Computers & Security. Problem-space

in which cost(z2 , z1 ) denotes any cost function that transforms z1 into z2 in the problem-space of
the specific application.
When comparing the problem-space attack in Eq. (2) with the feature-space attack in Eq. (1),
it is easy to find the most fundamental and noticeable difference between them is, the problem-
space attack involves a feature mapping function φ that maps the problem-space into the feature-
space, which is usually neither invertible nor differentiable. Therefore, the problem-space attack
can hardly use gradient-based methods directly to generate adversarial examples. In Fig. 3, we
illustrate the connection between the feature-space attack and the problem-space attack.

3.1.2. Adversary’s Knowledge: White-box Attack versus Black-box Attack

Adversary’s knowledge specifies what we assume the adversary knows about the target model
to be attacked. In terms of adversary’s knowledge, adversarial attacks can be further categorized
into the white-box attack and the black-box attack. To be specific, the white-box attack refers
to the scenario that the attackers know all information about the target model (e.g., architectures,
weights/parameters, outputs, features, etc.) as well as the dataset to train the target model. By
contrast, the black-box attack refers to the scenario that the attackers know nothing about the
target model except the model output, e.g., the classification label with or without probability.1
Apparently, the black-box attack is much harder to satisfy than the white-box attack since the
white-box attack is equipped with more knowledge about the target model. Besides, there is a
wide spectrum between the white-box attack and the black-box attack, which is usually broadly
referred as the gray-box attack. Therefore, for any gray-box attack, it is significantly necessary
to show to what extent the adversary knows and does not know about the target model as well as
the training dataset.

3.2. Three Unique Challenges of Adversarial Attacks for PE Malware: From Feature-space to
Problem-space
Originally, adversarial attacks are explored in the domain of image classification tasks and a
variety of feature-space attacks are subsequently proposed to generate adversarial examples for the
malicious purpose of misclassification, e.g., misclassifying a Persian cat into a lionfish with a high
probability of 99.8% as depicted in Fig. 3. Actually, the main reason for the success of directly
performing the feature-space attack to generate adversarial examples of images is that, it is easy
to find the corresponding image z 0 from the generate adversarial feature x0 via the inverse feature
mapping function φ−1 (i.e., z 0 = φ−1 (x0 )), as indicated in Fig. 3. However, when considering the
adversarial attacks for the PE files, the circumstance becomes completely different and extremely
challenging due to the problem-feature space dilemma [130], which is mainly manifested in
the following two aspects.
1. The feature mapping function φimage for images is relatively fixed (i.e., an image can be
formatted as a two-dimensional array of pixels where each pixel value is a three-dimensional
RGB vector with a continuous value between 0 to 255), while the feature mapping function
φpe for PE files is not fixed and can take various and diverse approaches of feature engineering
as detailed in § 2.2.3. Especially in the setting of black-box attacks, the attacker cannot know
the specific feature mapping function φpe for PE files, which greatly increases the difficulty
of adversarial attacks for PE files.
1
There is no unified view on whether to treat the scenario of knowing the classification label with probability as
the black-box attack.

13
 3.2 Three Unique Challenges of Adversarial Attacks for PE Malware: From Feature-space to
Accepted by ELSEVIER Computers & Security. Problem-space

2. For images, although the inverse feature mapping function φ−1 image is not exactly bi-injective
(e.g., the pixel value might not be in the range of 0 to 255), it is continuously differentiable,
and thus the feature-space attack based on gradients can directly apply on images to generate
adversarial examples. However, for various different feature mapping functions of PE files,
to map a feature vector in the feature-space into an executable in the problem-space, it is
almost impossible to find an exact or approximate function of inverse feature mapping φ−1 pe
that is either bi-injective or differentiable.
As depicted in Fig. 4, in order to generate adversarial examples for PE files, although there
is a variety of adversarial attacks that exploit the feature-space attacks based on gradients have
been proposed, we argue that these adversarial attacks are impractical and unrealistic against
PE malware detection in the wild world. This is because what these adversarial attacks generate
is the “adversarial PE feature” rather than the “adversarial PE malware” in the end, and an
“adversarial PE feature” does not guarantee to correspond to an “adversarial PE malware” due
to the following two reasons. On the one hand, it is almost impossible to find a corresponding
adversarial PE malware z 0 based on the generated adversarial PE feature x0 , as the inverse feature
mapping function φ−1 pe is normally neither bi-injective nor differentiable. On the other hand, even
though we could find the exact PE malware z 0 in the problem-space that corresponds to the
generate adversarial feature x0 in the feature-space, there is no guarantee that the found z 0 is also
“adversarial”. Taking the x03 in Fig. 4 as an example, although its feature representation x03 in
feature-space is misclassified as benign (i.e., f (x03 ) = 0), but its corresponding PE malware object
z30 in problem-space is still detected as malicious (i.e., f (φ(z30 )) = 1 6= 0).
Therefore, to further generate practical and realistic adversarial PE malware against malware
detection in the wild, one of the possible or even the only way so far is to seek for the problem-
space attack to generate adversarial PE malware in the problem-space as defined in Eq. (2). To be
specific, as depicted in Fig. 4, current problem-space attacks normally attempt to find and apply
a series of self-defined problem-space transformations (i.e., T1 , T2 , T3 and T4 ) that sequentially
T T
transform the original PE malware z into the desired adversarial PE malware z 0 (i.e., z −−→
1
z10 −−→
2

T T
z20 −−→
3
z30 −−→
4
z 0 ), such that ¶ z 0 is no longer detected as malicious by the target malware detection,
0
and · z maintains the same semantics as the original z. In the following parts, we detail the three
unique challenges of maintaining the semantics of adversarial PE malware for practical and realistic
adversarial attacks against PE malware detection and present the relationship between the three
challenges in Fig. 5.

3.2.1. Challenge 1: Follow the format of PE files (format-preserving)

First of all, unlike images, audio, or even texts, PE malware must follow the standard and
strict format rules of PE files. As characterized in Fig. 1, the PE file normally has a relatively
fixed layout and structure that is necessary to first load the PE file in the system memory and then
begin to execute it in the Microsoft family of Windows OSs. Therefore, for PE files, their problem-
space transformations should be defined within the scope of the format specification requirements,
i.e., format-preserving. For example, one of the transformations that add a new section within
the group of Header Information will definitely violate the format of PE files, while adding a new
section inside the group of Section Information is acceptable in terms of the format of PE files. In
order to overcome the challenge of format-preserving, one of the straightforward and non-trivial
approaches is to carefully check and inspect each candidate transformation according to the formal
specification of PE format under the family of Windows OSs.
14
 benign

Benign
x'
x3
x1

x
x2
malicious

Malicious

3.2 Three Unique Challenges of Adversarial Attacks for PE Malware: From Feature-space to
Accepted by ELSEVIER Computers & Security. Problem-space

Problem-space benign

malicious Format-preserving

Feature-space benign Executability-

preserving

Maliciousness-
preserving

Malicious Figure 5: The schematic illustration of relation-

ship between the three unique challenges of ad-
versarial attacks for PE malware, i.e., format-
Figure 4: The schematic illustration of the feature-space at- preserving, executability-preserving and
tack versus the problem-space attack for PE malware, in maliciousness-preserving.
which the original PE malware z is manipulated in the problem-
space to continuously generate the adversarial PE malware (i.e.,
z10 , z20 , z30 and z 0 ), while the corresponding PE malware feature x
in the feature-space is mapped to continuously generate adver-
sarial PE malware features (i.e., x01 , x02 , x03 and x0 ).

3.2.2. Challenge 2: Keep the executability for PE files (executability-preserving)

Although one generated adversarial PE malware is format-preserving, it does not necessarily
mean it is executability-preserving as well, which is one of the most challenging properties to be
addressed in generating adversarial PE malware. This is mainly because, the format of PE files
only determines its layout and structure under standard specifications, but cannot particularly
determine the concrete content of each element inside the layout and structure, and thus cannot
guarantee that they can be properly executed. For example, applying a simple transformation
of flipping on byte or even one bit in the “.data” section of a given PE file usually does not
violate the format specifications of PE files. However, it is very likely to cause a runtime crash
for the transformed PE file as it cannot load the necessary data from the “.data” section, thereby
Format-preserving
preventing its normal execution.

3.2.3. Challenge 3: Keep the same maliciousness for PE malware (maliciousness-preserving)

Executability-
Recall that the ultimate goal of adversarial attacks against PE malware detection is to gen-
preserving practical and realistic adversarial PE malware, which could not only misclassify the target
erate
PE malware detection model, but also can keep the same maliciousness as the original PE mal-
ware. However, again one generated adversarial malware that preserves the executability does
not necessarily mean it still preserves the sample maliciousness as the original PE malware, i.e.,
Maliciousness-
maliciousness-preserving. Suppose that the generated adversarial malware cannot perform the
same maliciousness behaviors (e.g., deleting or encrypting files, modifying registry items, etc.) as
preserving
the original PE malware, it is totally meaningless and unprofitable for the adversary in the wild.
15
Accepted by ELSEVIER Computers & Security.

Therefore, the property of maliciousness-preserving is another significant challenge to be addressed

in generating adversarial PE malware.
In short, to address the two aforementioned challenges of executability-preserving and
maliciousness-preserving, most of the proposed problem-space attacks claim all transforma-
tions adopted are format/executability/maliciousness-preserving, but only with limited inspection
and analysis on every specific transformation in either concept or empiric. Furthermore, due to the
complexity of both PE malware and the corresponding executable environments, we argue that it
is almost impossible to theoretically prove whether the proposed adversarial attacks can generate
successful adversarial PE malware that satisfies the two properties of executability-preserving and
maliciousness-preserving. Alternatively, empirical verification has been reasonably employed in
evaluating both executability-preserving and maliciousness-preserving. In particular, if the gener-
ated adversarial PE malware can be properly executed and its runtime status is basically consistent
with the runtime status of the original PE malware in the same simulated environments (e.g., sand-
box), both executability-preserving and maliciousness-preserving can therefore be empirically and
reasonably concluded.

4. Adversarial Attacks against PE Malware Detection: The State of the Art

In order to explore the most promising advances of adversarial attacks against PE malware
detection, in this section, we comprehensively and systematically categorize state-of-the-art ad-
versarial attacks from different viewpoints, i.e., adversary’s knowledge, adversary’s space, target
malware detection, and attack strategy. Fig. 6 illustrates the general category of adversarial at-
tacks against PE malware detection of this paper. In the following subsections, in terms of the
adversary’s knowledge (i.e., white-box versus black-box), we will first introduce the white-box
adversarial attacks against PE malware detection in § 4.1, and then introduce the black-box ad-
versarial attacks in § 4.2. Finally, we highlight the summary of state-of-the-art adversarial attacks
against PE malware detection in § 4.3.

4.1. White-box Adversarial Attacks against PE Malware Detection

Recall that in § 3.1, the white-box attack refers to the scenario that the adversary knows full
information about the target malware detectors, including architectures, parameters, feature space,
the training dataset, etc. In the following parts, we first further divide the white-box attacks into
the feature-space white-box attacks (in § 4.1.1) and the problem-space white-box attacks (in § 4.1.2)
based on their corresponding adversary’s space (i.e., feature-space versus problem-space).

4.1.1. Feature-space White-box Attacks against PE Malware Detection

In this part, we focus on the feature-space white-box attacks against PE malware detection, in
which all adversarial manipulations (e.g., adding irrelevant API calls) are performed in the feature
space of PE malware (e.g., the API call list). In particular, to better understand all feature-space
white-box attacks in the face of different kinds of PE malware detection models, we group them
into the following categories according to the different types of PE malware detection models,
including raw byte based malware detectors, API call list based malware detectors, visualization
based malware detectors, and other miscellaneous malware detectors.
Raw Bytes based Malware Detectors. In order to evade the raw bytes based malware
detector like MalConv [91], Kreuk et al. [17] consider appending or injecting a sequence of bytes,
namely the adversarial payload, to the end of PE malware or the slack region, i.e., existing unused
16
 Maliciousness-
preserving

Accepted by ELSEVIER Computers

4.1 White-box
& Security.
Adversarial Attacks against PE Malware Detection
Target Malware Detection

Raw Byte based Malware Detectors Gradient-based Method

API Call List based Malware Detectors
Feature-Space FGSM
Attacks
Visualization based Malware Detectors
Section 4.1.1 C&W
Other Misc. Malware Detectors
White-Box DeepFool
Attacks

Section 4.1
……
Problem-Space
Raw Byte based Malware Detectors
Attacks
Section 4.1.2 Explaination-based
Method
Adversarial Attack
Attacks Strategy Reinforcement Learning
API Call List based Malware Detectors

Feature-Space Randomization Method

API Call Sequence based Malware Detectors
Attacks
Section 4.2.1 Other Misc. Malware Detectors Evolutionary Algorithm
Black-Box
Attacks Generative Adversarial
Network
Section 4.2
Raw Byte based Malware Detectors
Problem-Space
Attacks PE Header Info. based Malware Detectors
Heuristic Algorithm
Section 4.2.2
…… ……

Figure 6: The General Category of Adversarial Attacks against PE Malware Detection.

continuous bytes of sections in the middle of PE malware. First, they iteratively generate the
adversarial payload with the gradient-based method of FGSM [9] in the continuous feature space.
Then, to generate the practical adversarial PE malware, they attempt to reconstruct back to the
input problem space by directly searching the closest neighbor to the generated adversarial payload.
Similarly, Suciu et al. [18] extend the FGSM-based adversarial attacks with two previously
proposed strategies (i.e., append-FGSM and slack-FGSM [17]), and further perform a systematic
evaluation to compare the effectiveness of both append and slack strategies against MalConv. Their
experimental results show that slack-FGSM outperforms append-FGSM with a smaller number of
modified bytes. Possible reasons are that the appended bytes of append-FGSM might exceed the
maximum size of the model input (e.g., 2MB for MalConv), or that slack-FGSM can make use
of surrounding contextual bytes to amplify the power of FGSM since the CNN-based MalConv
detector requires the consideration of the contextual bytes within the convolution window.
Chen et al. [19] suggest that all those adversarial attacks [17, 18] append or inject adversarial
bytes that are first initialized by random noises and further iteratively optimized, which might
lead to inferior attack performance. To address this issue, Chen et al. propose two novel white-
box attacks (i.e., BFA and Enhanced-BFA) with the saliency vector generated by the Grad-CAM
approach [131]. For BFA, it selects the data blocks with significant importance from benign PE
files using computed saliency vectors and then appends those data blocks to the end of the original
PE malware. Besides that, Enhanced-BFA is presented to use FGSM to iteratively optimize these
perturbations generated by BFA. Experimental results show that Enhanced-BFA and BFA have
comparative attack performances when the number of appending bytes is large, but Enhanced-BFA
is ten times more effective than BFA when the number of appending bytes is small.
Qiao et al. [20] propose a white-box adversarial attack against raw bytes based malware detec-
tors like MalConv. In particular, it first generates a prototype sample to maximize the output of

17
Accepted by ELSEVIER Computers
4.1 White-box
& Security.
Adversarial Attacks against PE Malware Detection

malware detection model towards the target class (i.e., malware) by directly applying the gradient
descent algorithm. Next, to ensure that the generated adversarial PE malware preserves both exe-
cutability and maliciousness as the input PE malware, it modifies the modifiable part of input PE
malware (i.e., bytes between sections, bytes at the end of PE files, and bytes in the newly added
section) in a fine-grained manner under the guidance of the generated prototype sample.
API Call List based Malware Detectors. By flipping the bits of the binary feature vector
of malware (“1” denotes the presence of one Windows API call and “0” denotes the absence),
Al-Dujaili et al. introduce four kinds of white-box adversarial attacks with k multi-steps, namely
dFGSMk , rFGSMk , BGAk , and BCAk [21], to attack API call list based malware detectors. To
be specific, dFGSMk and rFGSMk are two white-box adversarial attacks that are adapted mainly
from the FGSM attack in the continuous feature-space [9, 132] but extended for the binary feature
space via deterministic or randomized rounding, respectively. BGAk and BCAk are two gradient
ascent based attacks that update multiple bits or one bit in each step, respectively.
As the winner of “Robust Malware Detection Challenge” [133] in both attack and defense tracks,
Verwer et al. [134] propose a novel white-box adversarial attack with greedy random accelerated
multi-bit search, namely GRAMS, which generates functional adversarial API call features and
also builds a more robust malware detector in a standard adversarial training setting. The main
idea of GRAMS is to perform a greedy search procedure that explores gradient information as the
heuristic to indicate which bits to flip among all the binary search space (i.e., 22761 API calls).
At each iteration, GRAMS flips k bits of API calls that have the largest absolute gradient and
exponentially increases or decreases the value of k depending on whether GRAMS finds a better
or worse solution. To ensure the functionality of the generated adversarial malware, both [21]
and [134] limit the attack to flipping ‘0’ to ‘1’, meaning both of them only add irrelevant API calls.
Visualization based Malware Detectors. Differently, to attack the visualization-based
malware detectors, Liu et al. [22] propose the first white-box adversarial attack approach, namely
Adversarial Texture Malware Perturbation Attack (ATMPA), based on adversarial attacks in the
domain of image classification tasks [9, 10]. In particular, ATMPA first converts the malware sample
to a binary texture grayscale image and then manipulates the corresponding adversarial example
with subtle perturbations generated from two existing adversarial attack approaches - FGSM [9] and
C&W [10]. However, the major limitation of ATMPA is that the generated adversarial grayscale
image of the malware sample destroys the structure of the original malware and thus cannot be
executed properly, which makes ATMPA unpractical for real-world PE malware detection.
Similar to ATMPA, Khormali et al. present an adversarial attack COPYCAT [23] against vi-
sualization based malware detectors with CNNs. COPYCAT also makes use of existing generic
adversarial attacks (e.g., FGSM, PGD, C&W, MIM, DeepFool, etc.) to generate an adversarial
image. After that, COPYCAT appends the generated adversarial image to the end of the original
image of malware rather than directly adding it to the original malware image.
Differently, to evade visualization based malware detectors, Park et al. [24] propose another
adversarial attack based on the adversarial malware alignment obfuscation (AMAO) algorithm.
Specifically, a non-executable adversarial image is first generated by the off-the-shelf adversarial
attacks in the field of image classification [9, 10]. Then, in order to attempt to preserve the exe-
cutability, the adversarial PE malware is finally generated by the AMAO algorithm that minimally
inserts semantic N OP s at the available insertion points of the original malware such that the
modified PE malware is as similar as possible to the generated no-executable adversarial image.
Other Miscellaneous Malware Detectors. In [25], Li et al. first train ML-based malware

18
Accepted by ELSEVIER Computers
4.1 White-box
& Security.
Adversarial Attacks against PE Malware Detection

detection models based on OpCode n-gram features, i.e., the n-gram sequence of operation codes
extracted from the disassembled PE file. Then, the authors employ an interpretation model of
SHAP [135] to assign each n-gram feature with an importance value and observe that the 4-gram
“move + and + or + move” feature is a typical malicious feature as it almost does not appear
in the benign PE samples. Thus, based on this observation, the authors consider a generation
method of adversarial PE malware by instruction substitution. For instance, the “move + and +
or + move” in 10 sampled malware samples can be replaced with “push + pop + and + or + push
+ pop”, which can be used to bypass the malware detectors in their evaluation.

4.1.2. Problem-space White-box Attacks against PE Malware Detection

Different from these aforementioned feature-space adversarial attacks that operate in the feature
space, there is a growing body of work being proposed to perform problem-space adversarial attacks
against PE malware detection. To be specific, since it is technically feasible to directly modify the
raw bytes of PE files with possible constraints, almost all existing problem-space white-box attacks
target at raw byte based malware detectors (e.g., MalConv), which are detailed as follows.
In [26], Kolosnjaji et al. introduce a gradient-based white-box attack to generate adversarial
malware binaries (AMB) against MalConv. To ensure the generated adversarial malware bina-
ries behave identically to the original malware as much as possible, they consider one semantic-
preserving manipulation of appending the generated bytes at the end of the original malware file.
The appended bytes are generated by a gradient-descent method of optimizing the appended bytes
to maximally increase the probability of the appended PE malware that is predicted as goodware.
To unveil the main characteristics learned by MalConv to discriminate PE malware from benign
PE files, Demetrio et al. [27] employ the integrated gradient technique [136] for meaningful expla-
nations and find that MalConv is primarily based on the characteristics learned from PE header
rather than malicious content in sections. Motivated by the observation, they further present a
variant gradient-based white-box attack that is almost the same as [26]. The only difference is
that, AMB [26] injects adversarial bytes at the end of the PE file while this work is limited to
changing the bytes inside the specific DOS header in the PE header.
In [28], Demetrio et al. propose a general adversarial attack framework (RAMEn) against PE
malware detectors based on two novel functionality-preserving manipulations, namely Extend and
Shift, which inject adversarial payloads by extending the DOS header and shifting the content
of the first section in PE files, respectively. In fact, the adversarial payload generation can be
optimized in both white-box and black-box settings. For white-box settings, they use the same
gradient-based approach as AMB [26] to generate adversarial payloads and then inject payloads
via Extend and Shift manipulations. It is simply noted that, for black-box settings, they use the
same genetic algorithm as [52] to generate adversarial payloads and then inject them via Extend
and Shift manipulations.
To make it more stealthy than previous adversarial attacks [26, 27, 28], Sharif et al. propose
a new kind of adversarial attack based on binary diversification techniques which manipulate the
instructions of binaries in a fine-grained function level via two kinds of functionality-preserving
transformations, i.e., in-place randomization and code displacement [29]. In order to guide the
transformations that are applied to the PE malware under the white-box setting, they use a
gradient ascent optimization to select the transformation only if it shifts its embeddings in a
direction similar to the gradient of the attack loss function [10] with respect to its embeddings.

19
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

Table 2: Summary of State-of-the-Art Adversarial Attacks against PE Malware Detection. WB/BB is short for the white-box
attack and the black-box attack, FS/PS is short for the problem-space attack and the feature-space attack, and / denotes
emptily/fully preserving the related property. In particular, as it is almost impossible to theoretically prove both properties
of executability-preserving and maliciousness-preserving, we thus use to denote that related property is preserved neither
conceptually nor empirically, to denote it is preserved both conceptually and empirically, and to denote it is only preserved
conceptually but not empirically without experimental verification.

PE Malware Detection Attack Methods Preservation

Adversary’s Knowledge

Adversary’s Space

Format

empirical verification)

empirical verification)
Executability (with

Maliciousness (with
Attack Names Year Category Detection Name Transformation Strategy

Append or inject
Kreuk et al.
2018 WB FS Static MalConv the adversarial FGSM
[17]
payload
Append or inject
Suciu et al.
2019 WB FS Static MalConv adversarial FGSM
[18]
payload
Append the
BFA,
selected or
Enhanced- Grad-CAM
2019 WB FS Static MalConv optimized bytes
BFA or FGSM
from benign PE
[19]
files
Append or inject
Qiao et al. Gradient-
2022 WB FS Static MalConv adversarial
[20] based
payload
dFGSMk ,
rFGSMk , API call list based Add irrelevant Gradient-
2018 WB FS Static
BGAk , BCAk malware detectors API calls based
[21]
API call list based Add irrelevant Gradient-
GRAMS [134] 2020 WB FS Static
malware detectors API calls based
Visualization- Add adversarial
ATMPA [22] 2019 WB FS Static based malware noise to the FGSM, C&W
detectors malware image
Visualization- Append FGSM, PGD,
COPYCAT
2019 WB FS Static based malware adversarial noise C&W, MIM,
[23]
detectors generated DeepFool
Visualization-
Insert the
AMAO [24] 2019 WB FS Static based malware FGSM, C&W
semantic N OP s
detectors
Opcode
Opcode-based Interpretation
Li et al. [25] 2020 WB FS Static instruction
malware detectors model SHAP
substitution
Append Gradient-
AMB [26] 2018 WB PS Static MalConv
adversarial bytes based
Modify specific
Demetrio Gradient-
2019 WB PS Static MalConv regions in the PE
et al. [27] based
header
MalConv, DOS Header
Gradient-
RAMEn [28] 2020 WB PS Static Byte-based DNN Extension,
based
Model [125] Content Shifting
Binary
Lucas et al. MalConv, Gradient-
2021 WB PS Static diversification
[29] AvastNet [124] based
techniques
Table to be continued.

20
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

Continued Table.

BB
API call list based Add irrelevant
MalGAN [30] 2017 w/o FS Static GAN
malware detectors API calls
prob.
BB
Improved API call list based Add irrelevant
2019 w/o FS Static GAN
MalGAN [31] malware detectors API calls
prob.
BB
API call list based Add or Remove Greedy
EvnAttack [32] 2017 w FS Static
malware detectors API calls Algorithm
prob.
BB API call sequence
Hu and Tan Insert irrelevant Generative
2017 w/o FS Dynamic based malware
[33] API calls Model
prob. detectors
API call sequence Insert irrelevant
Transferability,
GADGET [34] 2018 BB FS Dynamic based malware API calls with
Heuristics
detectors IAT Hooking
BB API call sequence Insert API calls
Greedy
ELE [35] 2019 w FS Dynamic based malware with IAT
Algorithm
prob. detectors Hooking
API call sequence Insert API calls
Evolutionary
BADGER [36] 2020 BB FS Dynamic based malware with IAT
Algorithm
detectors Hooking
BB Predefined Transferability,
Rosenberg
2020 w/ FS Static EMBER modifiable Explainable
et al. [37]
prob. features ML
BB Inject semantic
CFG-based Reinforcement
SRL [38] 2020 w/o FS Static N OP s into CFG
malware detectors Learning
prob. blocks
BB
gym-malware format-preserving Reinforcement
2017 w/o PS Static –
[39, 137] modifications Learning
prob.
BB
format-preserving Reinforcement
gym-plus [40] 2018 w/o PS Static –
modifications Learning
prob.
gym-malware- BB
format-preserving Reinforcement
mini 2020 w/o PS Static –
modifications Learning
[41] prob.
BB
format-preserving Reinforcement
DQEAF [42] 2019 w/o PS Static –
modifications Learning
prob.
BB
RLAttackNet format-preserving Reinforcement
2020 w/o PS Static –
[43] modifications Learning
prob.
BB
AMG-VAC format-preserving Reinforcement
2021 w/o PS Static –
[44] modifications Learning
prob.
BB
AIMED-RL format-preserving Reinforcement
2021 w/o PS Static –
[45] modifications Learning
prob.
BB
format-preserving Reinforcement
AMG-IRL [46] 2021 w/o PS Static –
modifications Learning
prob.
BB
format-preserving
ARMED [47] 2019 w/ PS Static – Randomization
modifications
prob.
BB Append strings
Dropper [48] 2019 w/o PS Static – from goodware & Randomization
prob. Packing
Experience-
BB Append bytes
Chen et al. based
2019 w/o PS Static MalConv from benign PE
[19] Randomiza-
prob. files
tion
Table to be continued.

21
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

Continued Table.
BB format-preserving Weighted
Song et al.
2020 w/o PS Static – (macro & micro) Randomiza-
[49]
prob. modifications tion
BB
format-preserving Genetic
AIMED [50] 2019 w/ PS Static –
modifications Programming
prob.
BB
format-preserving Genetic
MDEA [51] 2020 w/ PS Static MalConv
modifications Algorithm
prob.
BB Inject and pad
Genetic
GAMMA [52] 2020 w/ PS Static – sections from
Algorithm
prob. benign PE files
BB
Append bytes to
GAPGAN [53] 2020 w/o PS Static MalConv GAN
the end
prob.
BB
Obfuscation-like Convolutional-
MalFox [54] 2020 w/o PS Static –
techniques GAN
prob.
Targeted BB
Occlusion of Binary
occlusion 2018 w/ PS Static –
important bytes Search
attack [55] prob.
BB Binary
Lucas et al. MalConv, Hill-climbing
2021 w/ PS Static diversification
[29] AvastNet [124] algorithm
prob. techniques

4.2. Black-box Adversarial Attacks against PE Malware Detection

Recall that in § 3.1, the black-box attack refers to the scenario that the adversary knows nothing
about the target PE malware detection models except the outputs, i.e., the malicious/benign label
with/without probability. In the following parts, we first further divide the black-box attacks into
the feature-space black-box attacks (in § 4.2.1) and the problem-space black-box attacks (in § 4.2.2)
based on their corresponding adversary’s space (i.e., feature-space versus problem-space).

4.2.1. Feature-space Black-box Attacks against PE Malware Detection

In this part, we focus on the feature-space black-box attacks against PE malware detectors,
in which all adversarial operations (e.g., insert irrelevant API calls) are performed in the feature
space of PE malware (e.g., the API call sequence) instead of being performed in the PE malware
themselves. As the feature-space attacks rely on the different feature representations from different
types of PE malware detectors, we thus group all existing feature-space black-box attacks into the
following categories according to different types of PE malware detectors, including API call list
based malware detectors, API call sequence based malware detectors, and other miscellaneous
malware detectors.
It is worth noting that we distinguish the API call list from the API call sequence for PE
malware detectors as follows. The API call list is a binary feature (i.e., ‘1’ or ‘0’), indicating
whether or not the PE file calls the specific API. The API call sequence represents the sequence of
APIs sequentially that is called by the PE file. The API call list can be extracted by either static
or dynamic analysis techniques while the API call sequence can only be extracted by dynamic
analysis techniques.
API Call List based Malware Detectors. Hu and Tan [30] first present a black-box
adversarial attack MalGAN based on GAN to attack PE malware detectors based on the API call
list. MalGAN assumes the adversary knows the complete feature space (i.e., binary features of the
22
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

160 system-level API calls) of the target malware detector and considers only adding some irrelevant
API calls into the original malware sample for generating the adversarial malware samples in the
feature space. MalGAN first builds a differentiable substitute detector to fit the target black-box
malware detector and then trains a generator to minimize the malicious probability of generated
adversarial malware predicted by the substitute detector. Subsequently, Kawai et al. [31] further
present an Improved-MalGAN after addressing several issues of MalGAN from a realistic viewpoint.
For instance, Improved-MalGAN trains the MalGAN and the target black-box malware detector
with different API call lists while the original MalGAN trains with the same API call list.
In [32], Chen et al. introduce another black-box adversarial attack, namely EvnAttack. EvnAt-
tack first employs Max-Relevance [138] to calculate the importance of each API call in classifying
PE malware or goodware based on the training set and then ranks those API calls into two sets: M
and B. In particular, M contains API calls that are highly relevant to malware, while B contains
API calls that are highly relevant to goodware. Intuitively, EvnAttack is a simple and straight-
forward attack method that manipulates the API call list by either adding the API calls in B or
removing the ones in M . Specifically, EvnAttack employs a bidirectional selection algorithm that
greedily selects API calls for the manipulation of addition or removal based on the fact that how
the manipulation influences the loss of the target PE malware detector.
API Call Sequence based Malware Detectors. Aiming at attacking RNN-based malware
detection models that take the API call sequence as the input, Hu and Tan [33] propose a generative
model based black-box adversarial attack to evade such RNN-based PE malware detectors. In
particular, a generative RNN is trained based on PE malware to generate an irreverent API call
sequence that will be inserted into the original API call sequence of the input PE malware, while
a substitute RNN model is trained to fit the target RNN-based malware detector based on both
benign samples and the gradient information of malware samples from the generative RNN model.
In [34], Rosenberg et al. propose a generic end-to-end attack framework, namely GADGET,
against state-of-the-art API call sequence-based malware detectors under black-box settings by
the transferability property. GADGET is carried out in three steps: i) GADGET first trains a
surrogate model to approximate the decision boundaries of the target malware detector by using the
Jacobian-based dataset augmentation method [139]. ii) it then performs a white-box attack on the
surrogate model to generate the adversarial API call sequence by restricting the insertion of API
calls into the original API call sequence. In more detail, GADGET first randomly selects an insert
position and then uses a heuristic searching approach to iteratively find and insert the API calls
such that the generated adversarial sequence follows the direction indicated by the Jacobian. iii) to
generate practical adversarial malware samples from the adversarial API call sequence, GADGET
uses a proxy wrapper script to wrap the original malware by calling the additional APIs with valid
parameters in the corresponding position based on the generated adversarial API call sequence.
Fadadu et al. [35] propose an executable level evasion (ELE) attack under black-box settings
to evade PE malware detectors based on the API call sequence. The manipulation of ELE is
restricted only to the addition of new API calls, which are chosen by maximizing the fraction of
sub-sequences that have the added API call in the domain of benign samples and minimizing the
fraction of sub-sequences that have the added API call in the domain of malware samples. To
further make the modified PE malware can be executed properly, ELE uses a novel IAT (i.e.,
Import Address Table) hooking method to redirect the control in the adversarial code that is
attached to the PE malware. In particular, the adversarial code contains a wrapper function that
not only has identical arguments and returns values with the original API function, but also invokes

23
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

the added API function that is periodically called by the PE malware.

Following GADGET, Rosenberg et al. [36] subsequently propose and implement an end-to-end
adversarial attack framework, namely BADGER, which consists of a series of query-efficient black-
box attacks to misclassify such API call sequence-based malware detector as well as minimize the
number of queries. Basically, to preserve the original functionality, the proposed attacks are limited
to only inserting API calls with no effect or an irrelevant effect, e.g., opening a non-existent file.
To solve the problem of which and where the API calls should be inserted in, the authors propose
different attacks with or without knowledge of output probability scores, i.e., the score-based attack
and the decision-based attack. For the score-based attack, it uses the self-adaptive uniform mixing
evolutionary algorithm [140] to optimize the insertion position with the API calls generated by a
pre-trained SeqGAN that has been trained to mimic API call sequences of benign samples. For
the decision-based attack, it selects a random insertion position and then inserts the API call with
the same position from the pre-trained SeqGAN. Finally, to make the attacks query-efficient, they
first insert a maximum budget of API calls and then employ a logarithmic backtracking method
to remove some of the inserted API calls as long as evasion is maintained.
Other Miscellaneous Malware Detectors. Rosenberg et al. [37] present a transferability-
based black-box attack against traditional ML-based malware detection models (e.g., EMBER) by
modifying the features instead of just adding new features (e.g., adding API calls) like previous
attacks. The authors first train a substitute model to fit the target black-box malware detector
and then use the explainable algorithm to obtain a list of feature importance of the detection result
of the original malware on the substitute model. Subsequently, the authors modify those easily
modifiable features with a list of predefined feature values and select a particular value that results
in the highest benign probability.
Aiming at attacking graph-based (i.e., CFG) malware detectors [118], Zhang et al. [38] intro-
duce the first semantic-preserving RL-based black-box adversarial attack named SRL. To preserve
the original functionality of the malware and retain the structure of the corresponding control
flow graph, SRL trains a deep RL agent which could iteratively choose basic blocks in CFG and
semantic N OP s for insertion to modify the PE malware until the generated adversarial malware
can successfully bypass the target malware detector. Their experimental results show that SRL
achieves a nearly 100% attack success rate against two variants of graph neural network based
malware detectors.

4.2.2. Problem-space Black-box Attacks against PE Malware Detection

In this part, we focus on the problem-space black-box adversarial attacks against PE malware
detectors, in which all adversarial operations are performed in the problem space of PE malware
under black-box settings, i.e., directly operating the PE malware itself without any consideration
of its feature representation (indicating the problem-space) as well as the PE malware detectors
to be attacked (indicating the black-box setting). It means that, in theory, the problem-space
black-box adversarial attacks in the scenario of PE malware detection are completely agnostic to
specific PE malware detectors. Therefore, regardless of any kind of PE malware detectors, we
group the problem-space black-box adversarial attacks according to the attack strategies, includ-
ing reinforcement learning, randomization, the evolutionary algorithm, GAN, and the heuristic
algorithm, which are detailed as follows.
Reinforcement learning based attacks. To expose the weaknesses of current static anti-
virus engines, Anderson et al. [39, 137] are the first to study how to automatically manipulate the
original PE malware such that the modified PE malware are no longer detected as malicious by
24
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

the anti-virus engines while do not break the format and functionality. Particularly, with only
knowledge of the binary detection output, they propose a completely black-box adversarial attack
based on reinforcement learning (RL), namely gym-malware. The gym-malware first defines 10
kinds of format-preserving and functionality-preserving modifications for Windows PE files as the
action space available to the agent within the environment. Then, for any given PE malware,
gym-malware tries to learn which sequences of modifications in the action space can be used to
modify the PE malware, such that the resulting PE malware is most likely to bypass the static
anti-virus engines. Although, gym-malware has demonstrated its effectiveness against PE malware
detectors, its experimental results also show that RL with an agent of deep Q-network (DQN) or
actor-critic with experience replay (ACER) [141] offers limited improvement compared with the
random policy.
On the basis of gym-malware, there are multiple follow-up work [40, 41, 42, 43, 44, 45, 46]
proposing problem-space black-box adversarial attacks against static PE malware detection models.
In particular, Wu et al. [40] propose gym-plus based on gym-malware with the improvement
of adding more format-preserving modifications in the action space and their experimental results
show that gym-plus with DQN obtains a higher evasion rate than gym-plus with the random
policy. Differently, Chen et al. [41] propose gym-malware-mini based on gym-malware with a
limited and smaller action space. Based on the observation of most of the format-preserving
modifications of gym-malware and gym-plus are stochastic in nature (e.g., the appending bytes
to the new section are chosen at random for simplicity, etc.) and those modifications are not
exactly repeatable, gym-malware-mini makes 6 kinds of random format-preserving modifications
to deterministic modifications, making the RL algorithms easier to learn better policies among
limited action space.
Besides that, Fang et al. [42] present a general framework using DQN to evade PE malware
detectors, namely DQEAF, which is almost identical to gym-malware in methodology except for
three implementation improvements as follows. 1) DQEAF uses a subset of modifications employed
in gym-malware and guarantees that all of them would not lead to corruption in the modified mal-
ware; 2) DQEAF uses a vector with 513 dimensions as the observed state, which is much lower than
that in gym-malware; 3) DQEAF makes priority into consideration during the replay of past transi-
tions. Fang et al. [43] also observe that the modifications in the action space of gym-malware have
some randomness and further found that most effective adversarial malware from gym-malware are
generated by UPX pack/unpacked modifications, which could lead to some training problems with
RL due to the non-repeatability of those modifications. Thus, they first reduce the action space
to 6 categories having certain deterministic parameters and then propose an improved black-box
adversarial attack, namely RLAttackNet, based on the gym-malware implementation.
Ebrahimi et al. [44] suggest that the RL-based adversarial attacks against PE malware de-
tectors normally employ actor-critic or DQN, which are limited in handling environments with
combinatorially large state space. Naturally, they propose an improved RL-based adversarial at-
tack framework of AMG-VAC on the basis of gym-malware [39, 137] by adopting the variational
actor-critic, which has been demonstrated to be the state-of-the-art performance in handling en-
vironments with combinatorially large state space. As previous RL-based adversarial attacks tend
to generate homogeneous and long sequences of transformations, Labaca-Castro et al. [45] thus
present an RL-based adversarial attack framework of AIMED-RL as well. The main difference
between AIMED-RL and other RL-based adversarial attacks is that AIMED-RL introduces a
novel penalization to the reward function for increasing the diversity of the generated sequences of

25
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

transformations while minimizing the corresponding lengths. Li and Li [46] suggest that existing
RL-based adversarial attacks [39, 137, 42] employ the artificially defined instant reward function
and environment, which are highly subjective and empirical, potentially leading to non-converge of
the RL algorithm. Therefore, in order to address the issue of the subjective and empirical reward
function, they present an Inverse RL-based adversarial malware generation method, namely AMG-
IRL, which could autonomously generate the flexible reward function according to the current
optimal strategy.
In short, compared to [40, 41, 44, 45], the adversarial malware samples generated by [42, 43, 46]
are verified not only for executability within the Cuckoo sandbox [142], but also verified for the
original maliciousness via comparing the function call graph between the before and after malware
samples with IDA Pro [143].
Randomization based attacks. To fully automatize the process of generating adversarial
malware without corrupting the malware functionality under the black-box setting, Castro et al. [47]
propose ARMED – automatic random malware modifications to evade detection. ARMED first
generates the adversarial PE malware by randomly applying manipulations among 9 kinds of
format-preserving modifications from [39, 137], and then employs the Cuckoo sandbox to test the
functionality of the generated adversarial malware samples. In case the functionality test fails, the
above steps would be re-start with a new round until the functionality test successes.
Ceschin et al. [48] find that packing the original PE malware with a distinct packer [144, 145]
or embedding the PE malware in a dropper [146, 147] is an effective approach in bypassing ML-
based malware detectors when combined with appending goodware strings to malware. However,
some of the generated adversarial PE malware suffer from either not being executed properly or
being too large in size. To solve the challenges, the authors implemented a lightweight dropper,
namely Dropper, which first creates an entire new PE file to host the original PE malware and
then randomly chooses goodware strings to be appended at the end of the newly created PE file.
Similar to the white-box attack of Enhanced-BFA, Chen et al. [19] also introduce another black-
box version of adversarial attack against MalConv. First, it continuously selects data blocks at
random from goodware and appends them to PE malware to generate adversarial PE malware.
After performing multiple random attacks as above, it then calculates the contribution degree of
each data block based on the experience of successful trajectories of data blocks. Finally, it appends
the data blocks to the end of PE malware according to the order of their contribution degrees.
In [49], Song et al. propose an automatic black-box attack framework that applies a sequence
of actions to rewrite PE malware for evading PE malware detectors. In particular, to generate
adversarial malware samples with a minimal set of required actions from macro/micro actions,
the authors employ an action sequence minimizer that consists of three steps. First, it randomly
selects macro-actions according to the previously updated weights of actions as the action sequence
to rewrite the original malware. Second, it tries to remove some unnecessary macro-actions from the
action sequence to generate a minimized adversarial malware, and increases the weights of effective
actions for updating. Finally, for every macro-action in the minimized adversarial malware, it
attempts to replace the macro-action with a corresponding micro-action. Besides that, the proposed
framework can also help explain which features are responsible for evasion as every required action
in adversarial malware samples corresponds to a type of affected feature.
Evolutionary algorithm based attacks. Following the black-box adversarial attack frame-
work of ARMED [47], Castro et al. [50] propose AIMED, which employs a genetic programming
approach rather than randomization methods to automatically find optimized modifications for

26
Accepted by ELSEVIER Computers
4.2 Black-box
& Security.
Adversarial Attacks against PE Malware Detection

generating adversarial PE malware. Firstly, 9 kinds of format-preserving modifications are intro-

duced and applied to the original PE malware to create the population, and then each modified
sample in the population is evaluated by a fitness function in terms of functionality, detection ra-
tio, similarity, and the current number of generations. Secondly, if the modified PE malware fails
to bypass the PE malware detector, AIMED implements the classic genetic operations, including
selection, crossover, and mutation, which are repeated until all generated adversarial PE malware
can evade the PE malware detector. Experiments demonstrate the time of generating the same
number of successful adversarial malware is reduced up to 50% compared to the previous random
based approach [47].
Similarly, Wang and Miikkulainen [51] propose to retrain MalConv with the adversarial PE
malware, namely MDEA. To generate the adversarial malware samples, MDEA adjusts 10 kinds of
format-preserving manipulations from [39, 137] as the action space and employs a genetic algorithm
to optimize different action sequences by selecting manipulations from the action space until the
generated adversarial malware bypasses the target malware detectors. In particular, as some
manipulations are stochastic in nature, MDEA limits each manipulation with a parameter set to
make the adversarially trained models converge within an acceptable time. However, the generated
adversarial malware samples by MDEA are not tested for functionality like AIMED and ARMED.
To omit the functionality testing (e.g., sandbox required) and further speed up the computation
of prior work [50, 47], Demetrio et al. introduced an efficient black-box attack framework, namely
GAMMA [52]. For GAMMA, the generation approaches of adversarial malware are limited to two
types of functionality-preserving manipulations: section injection and padding. Specifically, benign
contents are extracted from the goodware as adversarial payloads to inject either into some newly-
created sections (section injection) or at the end of the file (padding). The main idea of GAMMA
is to formalize the attack as a constrained minimization problem which not only optimizes the
probability of evading detection, but also penalizes the size of the injected adversarial payload as
a regularization term. To solve the constrained minimization problem, GAMMA also employs a
genetic algorithm, including selection, crossover, and mutation, to generate adversarial PE malware
to bypass the malware detector with few queries as well as small adversarial payloads.
GAN based attacks. In [53], Yuan et al. present GAPGAN, a novel GAN-based black-
box adversarial attack framework that is performed at the byte-level against DL-based malware
detection, i.e., MalConv. Specifically, GAPGAN first trains a generator and a discriminator con-
currently, where the generator intends to generate adversarial payloads that would be appended at
the end of original malware samples, and the discriminator attempts to imitate the black-box PE
malware detector to recognize both the original PE goodware and the generated adversarial PE
malware. After that, GAPGAN uses the trained generator to generate the adversarial payload for
every input PE malware and then appends the adversarial payload to the corresponding input PE
malware, which generates the adversarial PE malware while preserving its original functionalities.
Experiments show that GAPGAN can achieve a 100% attack success rate against MalConv with
only appending payloads of 2.5% of the total length of the input malware samples.
Zhong et al. [54] propose a convolutional generative adversarial network-based (C-GAN) frame-
work, namely MalFox, which can generate functionally indistinguishable adversarial malware sam-
ples against realistic black-box antivirus products by perturbing malware files based on packing-
related methods. In general, MalFox consists of 5 components: PE Parser, Generator, PE Edi-
tor, Detector, and Discriminator, where Detector is the target black-box malware detector (e.g.,
antivirus product) and Discriminator is an API call-based malware detector, representing the dis-

27
Accepted by ELSEVIER Computers
4.3 Summary
& Security.
of Adversarial Attacks against PE Malware Detection

crimination model in GAN. First, the API call list (i.e., DLL and system functions) of the original
malware sample is extracted as a binary feature vector by PE Parser; Second, the Generator takes
both the malware feature vector and a sampled 3-dimensional Gaussian noise as input to produce a
3-dimensional perturbation path, indicating whether each of the three perturbation methods (i.e.,
Obfusmal, Stealmal, and Hollowmal) is adopted. Third, following the produced perturbation path,
the PE editor generates the adversarial malware sample with corresponding perturbation methods.
Finally, the Generator will stop training until the generated adversarial PE malware fails to be
recognized by Discriminator.
Heuristic based attacks. Aiming to quantify the robustness of PE malware detectors ranging
from two ML-based models to four commercial anti-virus engines, Fleshman et al. [55] propose one
novel targeted occlusion black-box attack for comparing with three pre-existing evasion techniques,
i.e., random-based gym-malware [39, 137], obfuscation through packing [145, 148], and malicious
ROP injection [149]. For the proposed targeted occlusion attack, it first uses the occlusion binary
search method to identify the most important byte region according to the changes of the malicious
probability for a given PE malware detector, and then replaces the identified region with completely
random bytes or a contiguous byte region selected randomly from benign samples. However, we
believe that adversarial malware samples generated by the targeted occlusion attack are destructive
because the replacement could prevent the generated malware from being executed, not to mention
maintain the original maliciousness.
Based on the two kinds of functionality-preserving transformations (i.e., in-place randomization
and code displacement) that manipulate the instructions of binaries in a fine-grained function level,
Lucas et al. [29] also propose another black-box version of adversarial attack based on a general
hill-climbing algorithm. This black-box attack is basically similar to the white-box version and the
only difference is how the attempted transformation is selected. Specifically, the black-box attack
first queries the model after attempting to apply one of the transformations to the PE malware,
and then accepts the transformation only if the corresponding benign probability increases.

4.3. Summary of Adversarial Attacks against PE Malware Detection

In the research field of PE malware detection, adversarial attack methods have been rapidly
proposed and developed in recent years since 2017. For all four adversarial attack categories (i.e.,
feature-space white-box, problem-space white-box, feature-space black-box, and problem-space
black-box) detailed above, their generated adversarial PE malware is becoming more and more
practical and effective in attacking the target PE malware detection. We summarize the state-
of-the-art adversarial attacks against PE malware detection in Table 2, which demonstrates the
adversarial attack methods and their corresponding categories and characteristics in detail.
For all white-box attacks against PE malware detection, regardless of the adversary’s space
(feature-space or problem-space), it is clearly observed from Table 2 that almost all of the white-
box attacks adopt the optimization of gradient-based methods as their attack strategies. Actually,
gradient-based optimization and its variants have been widely adopted in the domain of adver-
sarial attacks against image classification models [9, 10]. However, it is infeasible and impractical
to directly apply the gradient-based optimization methods to generate “realistic” adversarial PE
malware due to the problem-feature space dilemma [130]. Therefore, it is primarily important
to adapt the existing gradient-based methods (e.g., FGSM, C&W) within constraints of feasible
transformations (e.g., append adversarial bytes, add irrelevant API calls) according to the differ-
ent types of PE malware detectors, indicating the white-box attacks normally depend on specific
malware detectors.
28
Accepted by ELSEVIER Computers & Security.

Compared with the white-box attacks, the black-box attacks are more realistic and practical in
the wild due to their minimal reliance on knowledge about the target malware detector. For the
feature-space black-box attacks, as they are actually performed in the feature space of PE files,
existing adversarial attack methods generally devise corresponding feasible transformations (e.g.,
add irrelevant API calls) for PE malware detectors with different feature spaces (e.g., API call list
based malware detectors), indicating the black-box attacks are normally malware detector specific.
However, for the problem-space black-box attacks with the most strict requirements due to their
manipulations in the problem-space, most of them are malware detector agnostic, meaning that
these adversarial attack methods can be used to attack any kind of PE malware detectors in theory.
In terms of property preservation (i.e., format, executability, and maliciousness), for all kinds of
adversarial attacks against PE malware detection, it is also observed from Table 2 that most of them
can only guarantee the format-preserving rather than executability-preserving and maliciousness-
preserving. In particular, several adversarial attack methods (e.g., ATMPA [22], COPYCAT [23]
and the target occlusion attack [55]) might destroy the fixed layout and grammar of the PE format
that is necessary to load and execute the PE file. On the other hand, for those adversarial attacks
like [29, 42, 43, 46, 47, 48, 49, 50], they are verified not only for the executability, but also verified
experimentally whether the generated adversarial PE malware keeps the same maliciousness as the
original PE malware, which is strongly recommended and advocated in our opinion.

5. Adversarial Defenses for PE Malware Detection

As various adversarial attacks continue to be proposed and evaluated, adversarial defense meth-
ods are meanwhile proposed accordingly. In fact, the rapid development of adversarial attacks and
counterpart defenses constitutes a constant arms race, meaning a new adversarial attack can easily
inspire the defender to devise a novel corresponding defense, and a newly proposed defense method
will inevitably lead the attacker to design a new adversarial attack for profit. Therefore, it is im-
portant to explore the most promising advances of adversarial defenses for Windows PE malware
detection against adversarial attacks. Although there are currently few researchers that specifi-
cally and exclusively propose adversarial defenses for Windows PE malware detection, most of the
aforementioned research efforts on adversarial attacks might more or less present corresponding
defense methods. In this section, we summarize the state-of-the-art adversarial defenses for PE
malware detection in recent years, mainly including adversarial training and several other defenses.

5.1. Adversarial Training

Adversarial training is one of the mainstream adversarial defenses to resist adversarial attacks
regardless of specific applications, e.g., image classification, natural languages processing, speech
recognition, etc. Intuitively, adversarial training refers to the defense mechanism that attempts
to improve the robustness of the ML/DL model by re-training it with the generated adversarial
examples with/without original training examples. In the scenario of defending against adversarial
attacks for PE malware detection, adversarial training and its variants are also widely adopted and
we detail them as follows.
Most studies on adversarial defenses [30, 137, 32, 40, 19, 51, 38] based on adversarial train-
ing follow a similar procedure, in which adversarial PE malware or adversarial PE features are
first generated and then re-train a corresponding ML/DL model based on the adversarial PE mal-
ware/features with/without the original PE malware/features. For instance, Hu and Tan re-train
a random-forest-based malware detector based on the original API calls as well as the adversarial
29
Accepted by ELSEVIER Computers & Security. 5.2 Other Defense Methods

API calls generated by the adversarial attack of MalGAN [30]. Anderson et al. [137] first exploit
the gym-malware to generate adversarial PE malware, and then re-train the malware detection
model of EMBER based on the original PE files and the generated adversarial PE malware. Dif-
ferently, in addition to adversarial training with adversarial API calls generated by EvnAttack,
Chen et al. [32] also present a new secure-learning framework for PE malware detection, namely
SecDefender, which adds a security regularization term by considering the evasion cost of feature
manipulations by attackers.
To sum up, for those adversarial defenses based on adversarial training, it is generally observed
that, 1) adversarial training is experimentally demonstrated to mitigate one or several adversarial
attacks to some extent; 2) adversarial training inevitably introduces significant additional costs in
generating adversarial examples during the training process.

5.2. Other Defense Methods

As the first place in the defender challenge of the Microsoft’s 2020 Machine Learning Security
Evasion Competition [150], Quiring et al. present a combinatorial framework of adversarial defenses
– PEberus [151] based on the following three defense methods as follows.
1. A PE file is passed into the semantic gap detectors, which are used to check whether the PE
file is maliciously challenged based on three simple heuristics, i.e., slack space, overlay, and
duplicate. For instance, a considerably high ratio of the overlay to the overall size usually
indicates the PE file might have appended bytes to the overlay.
2. If the PE file is not detected by the semantic gap detectors, PEberus employs the ensemble
of existing malware detectors (e.g., EMBER [92], the monotonic skip-gram model [152], and
the signature-based model [153]) and use the max voting for predictions.
3. PEberus also employs a stateful nearest-neighbor detector [154] which continuously checks if
the PE file is similar to any of the previously detected malware in the history buffer.
In short, with PEberus, a PE file is predicted as malicious if any of the above three defense methods
predicts it as malicious.
Lucas et al. [29] attempt to mitigate their proposed adversarial attacks by exploiting two de-
fensive mechanisms, i.e., binary normalization and instruction masking. For the defense of binary
normalization, it applies the transformation of in-place randomization iteratively into the PE file
to reduce its lexicographic representation, so that its potential adversarial manipulations can be
undone before inputting it into the downstream PE malware detector. Similarly, the instruction
masking defense first selects a random subset of the bytes that pertain to instructions and then
masks it with zeros before inputting the PE file into the PE malware detector.

6. Discussions

The previous sections of § 4 and § 5 enable interested readers to have a better and faster under-
standing with regard to the adversarial attacks and defenses for Windows PE malware detection.
In the following subsections, we first present the other related attacks against PE malware detec-
tion beyond the aforementioned adversarial attacks in § 6.1 and then shed some light on research
directions as well as opportunities for future work in § 6.2.

30
Accepted by ELSEVIER Computers & Security. 6.1 Beyond Adversarial Attacks

6.1. Beyond Adversarial Attacks

6.1.1. Universal Adversarial Perturbation
Universal Adversarial Perturbation (UAP) is one special type of adversarial attack in which an
identical single perturbation can be applied over a large set of inputs for misclassifying the target
model in the testing phase. In order to generate the problem-space UAP against PE malware
classifiers in the wild, Labaca-Castro et al. [155] first prepare a set of available transformations
(e.g., adding sections to the PE file, renaming sections, pack/unpacking, etc.) for Windows PE files,
and then perform a greedy search approach to identify a short sequence of transformations as the
UAP for the PE malware classifier. In particular, if the identified short sequence of transformations
representing the UAP is applied to any PE malware, then the resulting adversarial PE malware
can evade the target PE malware classifier with high probability while preserving the format,
executability, and maliciousness.

6.1.2. Training-time Poisoning Attacks

Different from the adversarial attacks that are performed in the testing phase, the poisoning
attacks aim to manipulate the training phase, such that the resulting poisoned model fb has the
same prediction on a clean set of inputs as the cleanly trained model fc but has an adversarially-
chosen prediction on the poisoned input (i.e., the input associated a specific backdoor trigger).
Aiming at misclassifying a specific family of malware as goodware, Sasaki et al.[156] propose
the first poisoning attack against ML-based malware detectors. They assume the strictest attack
scenario that the adversary not only has full knowledge of the training dataset and the learning
algorithm, but also can manipulate the training samples (i.e., malware) in the feature space as
well as their labels. In particular, the poisoning attack framework first selects one specific malware
family, and then utilizes the same optimization algorithm like [157] to generate the poisoning
samples in the feature-space, which are finally adopted to train the poisoned model.
However, poisoning one entire family of malware is much easier to be detected if the defender
checks all malware families individually. In contrast, poisoning one specific instance of malware is
a more challenging problem, by which any malware associated with a backdoor trigger would be
misclassified as goodware, regardless of their malware families. In addition, it is almost impossible
to control and manipulate the labeling process for the poisoning data samples for the adversary,
since most of the training samples normally are labeled with multiple independent anti-virus en-
gines by security companies and further used to train the malware detection models. Therefore,
considering the practicality of the attack scenario in the wild, both following [158] and [159] belong
to the clean-label poisoning attack [160], in which the adversary can control and manipulate the
poison instance itself, but cannot control the labeling of the poison instance.
In [158], targeting the feature-based ML malware classifier, Severi et al. consider the scenario
where the adversary can know and manipulate the feature-space of software to build the poisoned
goodware (i.e., goodware with the backdoor trigger), some of which can be obtained by security
companies to train the feature-based ML malware classifier. To be specific, the backdoor trigger
is created by presenting a greedy algorithm to select coherent combinations of relevant features
and values based on the explainable machine learning technique (i.e., SHAP [135]). Experimental
results indicate that the case of 1% poison rate and 17 manipulated features results in an attack
success rate of about 20%.
Shapira et al. [159] argue that the attack assumption of feature-space manipulations in [158]
is unrealistic and unreasonable for real-world malware classifiers. In pursuit of a poisoning attack

31
Accepted by ELSEVIER Computers & Security. 6.2 Future Directions and Opportunities

in a problem space, Shapira et al. propose a novel instance poisoning attack by first selecting
the goodware that is most similar to the target malware instance and then adding sections to
the goodware for adversarially training the poisoned model. Actually, the manipulation of adding
sections acts as the backdoor trigger, which can remain the functionality of the associated goodware
as well as the malware instance. During the testing phase, the target malware instance associated
with the backdoor trigger will be misclassified as benign by the poisoned model.

6.1.3. Model Steal Attacks

In order to approximate (i.e., steal) a remote deployed detection model fb of PE malware under
a strictly black-box setting, Ali and Eshete [161] propose a best-effort adversarial approximation
method, which mainly relies on limited training samples and publicly accessible pre-trained mod-
els. The proposed best-effort adversarial approximation method leverages feature representation
mapping (i.e., transforming the raw bytes of each PE to an image representation) and cross-domain
transferability (i.e., taking advantage of pre-trained models from image classification) to approx-
imate the PE malware detection model fb by locally training a substitute PE malware detection
model fs . Experimental results show that the approximated model fs is nearly 90% similar to the
black-box model fb in predicting the same input samples.

6.2. Future Directions and Opportunities

6.2.1. Strong Demands for Robust PE Malware Detection
As described in § 2.2 and § 5, although there are a large number of detection methods for
PE malware, there are only very limited adversarial defense methods for building more robust PE
malware detection models against adversarial attacks. In general, almost all current adversarial
defense methods are empirical defenses based on various heuristics (e.g., adversarial training, input
transformation, etc.), which are usually only effective for one or a few adversarial attack methods,
indicating these empirical defenses are normally attack-specific. On the one hand, with the mas-
sive existence and continuous evolution of PE malware and corresponding adversarial malware, we
argue that the demand for empirical defense methods against them is also likely to rise accordingly
to build more robust PE malware detection models. On the other hand, the substantial work
of robustness certification applied in image classification tasks suggests a strong demand for PE
malware detection models with theoretically guaranteed robustness, as there is no work studying
the certified defenses against various adversarial attacks until now. We argue that certifying the
robustness of malware detection models not only helps users comprehensively evaluate their effec-
tiveness under any attack, but also increases the transparency of their pricing in cloud services
(e.g., malware detection as a service).

6.2.2. Practical and Efficient Adversarial Attacks against Commercial Anti-viruses in the Wild
As introduced and summarized in § 4 and Table 2, the current adversarial attack methods
against PE malware detection have been devoted to developing problem-space black-box adversarial
attacks, which usually take a similar and typical attack procedure. In general, the attack procedure
first defines a set of available transformations (e.g., inject adversarial payload, insert the semantic
N OP s, etc.) in the problem-space, and then employs a variety of search strategies (e.g., gradient-
based, reinforcement learning, genetic algorithm, etc.) to choose a sequence of transformations
which can be applied to the original PE malware for generating the adversarial PE malware. Based
on the above observation, we argue there is still much room for improving both the effectiveness
and efficiency of adversarial attacks against PE malware detection in two aspects: i) devising
32
Accepted by ELSEVIER Computers & Security.

and defining more practical and stealthier transformations for PE malware. For example, instead
of simply inserting the N OP s in the blocks of CFGs [38] that is easily noticed and removed by
defenders, the transformation of splitting one block of CFGs into multiple iteratively called blocks
is much stealthier to be noticed and removed. ii) designing and implementing more efficient search
strategies to accelerate the generation of adversarial PE malware. We argue that it is quite time-
consuming for both RL and genetic algorithm based search strategies.
In addition, it is clearly observed that most existing adversarial attacks target PE malware
detection based on static analysis rather than dynamic analysis, which is particularly unknown for
both attackers and defenders. However, the mainstream commercial anti-virus software/service
used by end users of laptops and servers normally employs a hybrid defense solution with both
static analysis and dynamic analysis. Therefore, it is extremely important and urgently demanded
to devise and implement practical and efficient adversarial attacks against PE malware detection
based on dynamic analysis and commercial anti-viruses.

6.2.3. Lack of Benchmark Platforms for Research

As the continuous emergence of adversarial attacks against Windows PE malware detection has
led to the constant arms race between adversarial attacks and defense methods, it is challenging to
quantitatively understand the strengths and limitations of these methods due to incomplete and
biased evaluation. In fact, in terms of other research fields like adversarial attacks in images, texts,
and graphs, there are a large number of corresponding toolboxes and platforms [162, 163, 58, 164]
that have been implemented and open-sourced. Based on these toolboxes and platforms, subse-
quent practitioners and researchers can not only exploit them to evaluate the actual effectiveness
of previously proposed methods, but also take them as a cornerstone to implement their own pro-
posed attacks and defenses, thereby reducing the time consumption for repetitive implementations.
Therefore, in order to further advance the research on adversarial attacks against Windows PE
malware detection, we argue that it is significantly important to design and implement a benchmark
platform with a set of benchmark datasets, representative PE malware detection to be targeted,
state-of-the-art adversarial attack & defense methods, performance metrics, and environments for
a comprehensive and fair evaluation.

7. Conclusion

In this paper, we conduct a comprehensive review on the state-of-the-art adversarial attacks

against Windows PE malware detection as well as the counterpart adversarial defenses. To the best
of our knowledge, this is the first work that not only manifests the unique challenges of adversarial
attacks in the context of Windows PE malware in the wild, but also systematically categorizes
the extensive work from different viewpoints in this research field. Specifically, by comparing
the inherent differences between PE malware and images that have been explored originally and
traditionally, we present three unique challenges (i.e., format-preserving, executability-preserving,
and maliciousness-preserving) in maintaining the semantics of adversarial PE malware for practical
and realistic adversarial attacks. Besides, we review recent research efforts in both adversarial
attacks and defenses, and further develop reasonable taxonomy schemes to organize and summarize
the existing literature, aiming at making them more concise and understandable for interested
readers. Moreover, beyond the aforementioned adversarial attacks, we discuss other types of attacks
against Windows PE malware detection and shed some light on future directions. Hopefully, this
paper can serve as a useful guideline and give related researchers/practitioners a comprehensive and
33
Accepted by ELSEVIER Computers & Security. REFERENCES

systematical understanding of the fundamental issues of adversarial attacks against PE malware

detection, thereby becoming a starting point to advance this research field.

Acknowledgement
This project is supported by the Strategic Priority Research Program of the Chinese Academy
of Sciences under Grant No.XDA0320000, the National Natural Science Foundation of China under
No.62202457 and No.U1936215, and the project funded by China Postdoctoral Science Foundation
under No.2022M713253. Yaguan Qian is also supported by the Key Program of Zhejiang Provincial
Natural Science Foundation of China under No.LZ22F020007.

References
[1] D. Ucci, L. Aniello, R. Baldoni, Survey of machine learning techniques for malware analysis, Computers &
Security 81 (2019) 123–147.
[2] E. Raff, C. Nicholas, A survey of machine learning methods and challenges for windows malware classification,
2020. ArXiv preprint arXiv:2006.09271.
[3] M. G. Schultz, E. Eskin, F. Zadok, S. J. Stolfo, Data mining methods for detection of new malicious executables,
in: IEEE Symposium on Security and Privacy, IEEE, Oakland, California, USA, 2001, pp. 38–49.
[4] Kaspersky Lab, The number of new malicious files detected every day increases
by 5.2% to 360,000 in 2020, https://www.kaspersky.com/about/press-releases/2020_
the-number-of-new-malicious-files-detected-every-day-increases-by-52-to-360000-in-2020, 2020.
Online (last accessed October 1, 2021).
[5] Kaspersky Lab, New malicious files discovered daily grow by 5.7% to
380,000 in 2021, https://www.kaspersky.com/about/press-releases/2021_
new-malicious-files-discovered-daily-grow-by-57-to-380000-in-2021, 2020. Online (last accessed
September 14, 2022).
[6] N. Idika, A. P. Mathur, A survey of malware detection techniques, Purdue University 48 (2007) 32–46.
[7] Y. Ye, T. Li, D. Adjeroh, S. S. Iyengar, A survey on malware detection using data mining techniques, ACM
Computing Surveys 50 (2017) 1–40.
[8] F. Ceschin, H. M. Gomes, M. Botacin, A. Bifet, B. Pfahringer, L. S. Oliveira, A. Grégio, Machine learning (in)
security: A stream of problems, 2020. ArXiv preprint arXiv:2010.16045.
[9] I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, in: International
Conference on Learning Representations, OpenReview.net, San Diego, CA, USA, 2015, pp. 1–11.
[10] N. Carlini, D. Wagner, Towards evaluating the robustness of neural networks, in: IEEE Symposium on Security
and Privacy, IEEE, San Jose, CA, USA, 2017, pp. 39–57.
[11] N. Akhtar, A. Mian, Threat of adversarial attacks on deep learning in computer vision: A survey, IEEE Access
6 (2018) 14410–14430.
[12] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay, Adversarial attacks and defenses:
A survey, arXiv preprint arXiv:1810.00069 (2018).
[13] J. Zhang, C. Li, Adversarial examples: Opportunities and challenges, IEEE Transactions on Neural Networks
and Learning Systems 31 (2019) 2578–2593.
[14] A. Serban, E. Poll, J. Visser, Adversarial examples on object recognition: A comprehensive survey, ACM
Computing Surveys (CSUR) 53 (2020) 1–38.
[15] G. R. Machado, E. Silva, R. R. Goldschmidt, Adversarial machine learning in image classification: A survey
toward the defender’s perspective, ACM Computing Surveys (CSUR) 55 (2021) 1–38.
[16] T. Long, Q. Gao, L. Xu, Z. Zhou, A survey on adversarial attacks in computer vision: Taxonomy, visualization
and future directions, Computers & Security 121 (2022) 102847.
[17] F. Kreuk, A. Barak, S. Aviv-Reuven, M. Baruch, B. Pinkas, J. Keshet, Deceiving end-to-end deep learning
malware detectors using adversarial examples, 2018. ArXiv preprint arXiv:1802.04528.
[18] O. Suciu, S. E. Coull, J. Johns, Exploring adversarial examples in malware detection, in: IEEE Security and
Privacy Workshops, IEEE, San Francisco, CA, USA, 2019, pp. 8–14.
[19] B. Chen, Z. Ren, C. Yu, I. Hussain, J. Liu, Adversarial examples for CNN-based malware detectors, IEEE
Access 7 (2019) 54360–54371.

34
Accepted by ELSEVIER Computers & Security. REFERENCES

[20] Y. Qiao, W. Zhang, Z. Tian, L. T. Yang, Y. Liu, M. Alazab, Adversarial malware sample generation method
based on the prototype of deep learning detector, Computers & Security (2022) 102762.
[21] A. Al-Dujaili, A. Huang, E. Hemberg, U.-M. O’Reilly, Adversarial deep learning for robust detection of binary
encoded malware, in: IEEE Security and Privacy Workshops, IEEE, San Francisco, CA, USA, 2018, pp. 76–82.
[22] X. Liu, J. Zhang, Y. Lin, H. Li, ATMPA: Attacking machine learning-based malware visualization detection
methods via adversarial examples, in: IEEE/ACM International Symposium on Quality of Service, IEEE,
Phoenix, AZ, USA, 2019, pp. 1–10.
[23] A. Khormali, A. Abusnaina, S. Chen, D. Nyang, A. Mohaisen, COPYCAT: practical adversarial attacks on
visualization-based malware detection, 2019. ArXiv preprint arXiv:1909.09735.
[24] D. Park, H. Khan, B. Yener, Generation & evaluation of adversarial examples for malware obfuscation, in:
International Conference on Machine Learning and Applications, IEEE, Boca Raton, FL, USA, 2019, pp.
1283–1290.
[25] X. Li, K. Qiu, C. Qian, G. Zhao, An adversarial machine learning method based on opcode n-grams feature
in malware detection, in: International Conference on Data Science in Cyberspace, IEEE, Hong Kong, China,
2020, pp. 380–387.
[26] B. Kolosnjaji, A. Demontis, B. Biggio, D. Maiorca, G. Giacinto, C. Eckert, F. Roli, Adversarial malware bina-
ries: Evading deep learning for malware detection in executables, in: European Signal Processing Conference,
IEEE, Roma, Italy, 2018, pp. 533–537.
[27] L. Demetrio, B. Biggio, G. Lagorio, F. Roli, A. Armando, Explaining vulnerabilities of deep learning to
adversarial malware binaries, 2019. ArXiv preprint arXiv:1901.03583.
[28] L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, F. Roli, Adversarial EXEmples: A survey and
experimental evaluation of practical attacks on machine learning for windows malware detection, 2020. ArXiv
preprint arXiv:2008.07125.
[29] K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, S. Shintre, Malware makeover: Breaking ml-based static analysis
by modifying executable bytes, in: ASIA Conference on Computer and Communications Security, ACM,
Hongkong, China, 2021, pp. 744–758.
[30] W. Hu, Y. Tan, Generating adversarial malware examples for black-box attacks based on GAN, 2017. ArXiv
preprint arXiv:1702.05983.
[31] M. Kawai, K. Ota, M. Dong, Improved MalGAN: Avoiding malware detector by leaning cleanware features,
in: International Conference on Artificial Intelligence in Information and Communication, IEEE, Okinawa,
Japan, 2019, pp. 40–45.
[32] L. Chen, Y. Ye, T. Bourlai, Adversarial machine learning in malware detection: Arms race between evasion
attack and defense, in: European Intelligence and Security Informatics Conference, IEEE, Athens, Greece,
2017, pp. 99–106.
[33] W. Hu, Y. Tan, Black-box attacks against RNN based malware detection algorithms, 2017. ArXiv preprint
arXiv:1705.08131.
[34] I. Rosenberg, A. Shabtai, L. Rokach, Y. Elovici, Generic black-box end-to-end attack against state of the
art API call based malware classifiers, in: International Symposium on Research in Attacks, Intrusions, and
Defenses, Springer, Heraklion, Crete, Greece, 2018, pp. 490–510.
[35] F. Fadadu, A. Handa, N. Kumar, S. K. Shukla, Evading API call sequence based malware classifiers, in:
International Conference on Information and Communications Security, Springer, Beijing, China, 2019, pp.
18–33.
[36] I. Rosenberg, A. Shabtai, Y. Elovici, L. Rokach, Query-efficient black-box attack against sequence-based
malware classifiers, in: Annual Computer Security Applications Conference, ACM, Virtual Event, 2020, p.
611–626.
[37] I. Rosenberg, S. Meir, J. Berrebi, I. Gordon, G. Sicard, E. O. David, Generating end-to-end adversarial
examples for malware classifiers using explainability, in: International Joint Conference on Neural Networks,
IEEE, Glasgow, United Kingdom, 2020, pp. 1–10.
[38] L. Zhang, P. Liu, Y.-H. Choi, Semantic-preserving reinforcement learning attack against graph neural networks
for malware detection, 2020. ArXiv preprint arXiv:2009.05602.
[39] H. S. Anderson, A. Kharkar, B. Filar, P. Roth, Evading machine learning malware detection, in: Black Hat
USA, blackhat.com, Las Vegas, NV, USA, 2017, pp. 1–6.
[40] C. Wu, J. Shi, Y. Yang, W. Li, Enhancing machine learning based malware detection model by reinforcement
learning, in: International Conference on Communication and Network Security, ACM, Qingdao, China, 2018,
pp. 74–78.
[41] J. Chen, J. Jiang, R. Li, Y. Dou, Generating adversarial examples for static PE malware detector based on

35
Accepted by ELSEVIER Computers & Security. REFERENCES

deep reinforcement learning, Journal of Physics: Conference Series 1575 (2020) 012011.
[42] Z. Fang, J. Wang, B. Li, S. Wu, Y. Zhou, H. Huang, Evading anti-malware engines with deep reinforcement
learning, IEEE Access 7 (2019) 48867–48879.
[43] Y. Fang, Y. Zeng, B. Li, L. Liu, L. Zhang, DeepDetectNet vs RLAttackNet: An adversarial method to improve
deep learning-based static malware detection model, PLoS One 15 (2020) e0231626.
[44] M. Ebrahimi, J. Pacheco, W. Li, J. L. Hu, H. Chen, Binary black-box attacks against static malware detectors
with reinforcement learning in discrete action spaces, in: IEEE Security and Privacy Workshops, IEEE, Virtual
Event, 2021, pp. 85–91.
[45] R. Labaca-Castro, S. Franz, G. D. Rodosek, AIMED-RL: Exploring adversarial malware examples with re-
inforcement learning, in: Joint European Conference on Machine Learning and Knowledge Discovery in
Databases, Springer, Bilbao, Spain, 2021, pp. 37–52.
[46] X. Li, Q. Li, An irl-based malware adversarial generation method to evade anti-malware engines, Computers
& Security 104 (2021) 102118.
[47] R. L. Castro, C. Schmitt, G. D. Rodosek, ARMED: How automatic malware modifications can evade static
detection?, in: International Conference on Information Management, IEEE, Cambridge, United Kingdom,
2019, pp. 20–27.
[48] F. Ceschin, M. Botacin, H. M. Gomes, L. S. Oliveira, A. Grégio, Shallow security: On the creation of
adversarial variants to evade machine learning-based malware detectors, in: Reversing and Offensive-Oriented
Trends Symposium, ACM, Vienna, Austria, 2019, pp. 1–9.
[49] W. Song, X. Li, S. Afroz, D. Garg, D. Kuznetsov, H. Yin, Automatic generation of adversarial examples for
interpreting malware classifiers, 2020. ArXiv preprint arXiv:2003.03100.
[50] R. L. Castro, C. Schmitt, G. Dreo, AIMED: Evolving malware with genetic programming to evade detection,
in: International Conference on Trust, Security and Privacy in Computing and Communications / International
Conference on Big Data Science and Engineering, IEEE, Rotorua, New Zealand, 2019, pp. 240–247.
[51] X. Wang, R. Miikkulainen, MDEA: Malware detection with evolutionary adversarial learning, 2020. ArXiv
preprint arXiv:2002.03331.
[52] L. Demetrio, B. Biggio, G. Lagorio, F. Roli, A. Armando, Functionality-preserving black-box optimization of
adversarial windows malware, 2020. ArXiv preprint arXiv:2003.13526.
[53] J. Yuan, S. Zhou, L. Lin, F. Wang, J. Cui, Black-box adversarial attacks against deep learning based malware
binaries detection with GAN, in: European Conference on Artificial Intelligence, volume 325, IOS Press,
Santiago de Compostela, Spain, 2020, pp. 2536–2542.
[54] F. Zhong, X. Cheng, D. Yu, B. Gong, S. Song, J. Yu, MalFox: Camouflaged adversarial malware example
generation based on C-GANs against black-box detectors, 2020. ArXiv preprint arXiv:2011.01509.
[55] W. Fleshman, E. Raff, R. Zak, M. McLean, C. Nicholas, Static malware detection & subterfuge: Quantifying
the robustness of machine learning and current anti-virus, in: International Conference on Malicious and
Unwanted Software, IEEE, Nantucket, MA, USA, 2018, pp. 1–10.
[56] B. Alshemali, J. Kalita, Improving the reliability of deep neural networks in nlp: A review, Knowledge-Based
Systems 191 (2020) 105210.
[57] J. Lan, R. Zhang, Z. Yan, J. Wang, Y. Chen, R. Hou, Adversarial attacks and defenses in speaker recognition
systems: A survey, Journal of Systems Architecture 127 (2022) 102526.
[58] G. Zeng, F. Qi, Q. Zhou, T. Zhang, Z. Ma, B. Hou, Y. Zang, Z. Liu, M. Sun, OpenAttack: An open-source
textual adversarial attack toolkit, 2020. ArXiv preprint arXiv:2009.09191.
[59] L. Sun, Y. Dou, C. Yang, J. Wang, P. S. Yu, L. He, B. Li, Adversarial attack and defense on graph data: A
survey, 2018. ArXiv preprint arXiv:1812.10528.
[60] F. Pierazzi, F. Pendlebury, J. Cortellazzi, L. Cavallaro, Intriguing properties of adversarial ml attacks in the
problem space, in: IEEE Symposium on Security and Privacy, IEEE, Virtual Event, 2020, pp. 1332–1349.
[61] D. Park, B. Yener, A survey on practical adversarial examples for malware classifiers, in: Reversing and
Offensive-oriented Trends Symposium, ACM, Vienna, Austria, 2020, pp. 23–35.
[62] D. Li, Q. Li, Y. Ye, S. Xu, Arms race in adversarial malware detection: A survey, ACM Computing Surveys
(CSUR) 55 (2021) 1–35.
[63] Microsoft, Inc., PE format, https://docs.microsoft.com/en-us/windows/win32/debug/pe-format, 2020.
Online (last accessed October 22, 2020).
[64] M. Pietrek, Inside Windows: An in-depth look into the Win32 portable executable file for-
mat, MSDN Magazine: https://docs.microsoft.com/en-us/archive/msdn-magazine/2002/february/
inside-windows-win32-portable-executable-file-format-in-detail, 2020. Online (last accessed Octo-
ber 22, 2020).

36
Accepted by ELSEVIER Computers & Security. REFERENCES

[65] T. Paterson, An inside look at MS-DOS, Byte 8 (1983) 230.

[66] AV-TEST Institute, Security report 2019/2020, https://www.av-test.org/fileadmin/pdf/security_
report/AV-TEST_Security_Report_2019-2020.pdf, 2020. Online (last accessed January 15, 2020).
[67] Avira, Inc., Malware Threat Report: Q4 and 2020 Malware Threat Report, https://www.avira.com/en/blog/
q4-and-2020-malware-threat-report, 2020. Online (last accessed January 17, 2021).
[68] M. Souppaya, K. Scarfone, et al., Guide to malware incident prevention and handling for desktops and laptops,
NIST Special Publication 800 (2013) 83.
[69] H. R. Zeidanloo, F. Tabatabaei, P. V. Amoli, A. Tajpour, All about malwares (malicious codes)., in: Security
and Management, 2010, pp. 342–348.
[70] C. Cortes, L. D. Jackel, W.-P. Chiang, et al., Limits on learning machine accuracy imposed by data quality,
in: International Conference on Knowledge Discovery and Data Mining, AAAI Press, Montreal, Canada, 1995,
pp. 57–62.
[71] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Pro-
ceedings of the IEEE 86 (1998) 2278–2324.
[72] A. Krizhevsky, Learning multiple layers of features from tiny images, 2009. http://www.cs.toronto.edu/
~kriz/learning-features-2009-TR.pdf.
[73] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: A large-scale hierarchical image database,
in: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Miami, Florida, USA, 2009, pp.
248–255.
[74] Y. Inc., Yelp open dataset: An all-purpose dataset for learning, https://www.yelp.com/dataset, 2020. Online
(last accessed October 22, 2020).
[75] Corvus Forensics, Virusshare.com – because sharing is caring, https://virusshare.com/, 2021. Online (last
accessed August 25, 2021).
[76] Ytisf, thezoo a live malware repo., https://github.com/ytisf/thezoo, 2021. Online (last accessed August
25, 2021).
[77] K. Chen, P. Wang, Y. Lee, X. Wang, N. Zhang, H. Huang, W. Zou, P. Liu, Finding unknown malice in 10
seconds: Mass vetting for new threats at the google-play scale, in: USENIX Security Symposium, USENIX
Association, Washington, D.C., USA, 2015, pp. 659–674.
[78] M. Sebastián, R. Rivera, P. Kotzias, J. Caballero, AVCLASS: A tool for massive malware labeling, in:
International Symposium on Research in Attacks, Intrusions, and Defenses, Springer, Paris, France, 2016, pp.
230–253.
[79] S. Sebastián, J. Caballero, AVClASS2: Massive malware tag extraction from AV labels, in: Annual Computer
Security Applications Conference, ACM, Virtual Event, 2020, pp. 42–53.
[80] A. Kantchelian, M. C. Tschantz, S. Afroz, B. Miller, V. Shankar, R. Bachwani, A. D. Joseph, J. D. Tygar,
Better malware ground truth: Techniques for weighting anti-virus vendor labels, in: ACM Workshop on
Artificial Intelligence and Security, ACM, Denver, Colorado, USA, 2015, pp. 45–56.
[81] S. Zhu, Z. Zhang, L. Yang, L. Song, G. Wang, Benchmarking label dynamics of virustotal engines, in: ACM
SIGSAC Conference on Computer and Communications Security, ACM, Virtual Event, 2020, p. 2081–2083.
[82] K. P. Murphy, et al., Naive bayes classifiers, Technical Report, University of British Columbia, 2006.
[83] C. Cortes, V. Vapnik, Support-vector networks, Machine Learning 20 (1995) 273–297.
[84] I. H. Witten, E. Frank, Data mining: practical machine learning tools and techniques with java implementa-
tions, ACM Sigmod Record 31 (2002) 76–77.
[85] R. Collobert, S. Bengio, Links between perceptrons, MLPs and SVMs, in: International Conference on Machine
Learning, ACM, Banff, Alberta, Canada, 2004, pp. 1–8.
[86] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks,
in: Advances in Neural Information Processing Systems, Curran Associates, Inc., Lake Tahoe, Nevada, United
States, 2012, pp. 1106–1114.
[87] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, LightGBM: A highly efficient
gradient boosting decision tree, in: Advances in Neural Information Processing Systems, Curran Associates,
Inc., Long Beach, CA, USA, 2017, pp. 3146–3154.
[88] J. Z. Kolter, M. A. Maloof, Learning to detect and classify malicious executables in the wild., Journal of
Machine Learning Research 7 (2006) 2721–2744.
[89] J. Saxe, K. Berlin, Deep neural network based malware detection using two dimensional binary program
features, in: International Conference on Malicious and Unwanted Software, IEEE, Fajardo, PR, USA, 2015,
pp. 11–20.
[90] D. Gibert, C. Mateu, J. Planes, R. Vicens, Classification of malware by using structural entropy on convolu-

37
Accepted by ELSEVIER Computers & Security. REFERENCES

tional neural networks, in: AAAI Conference on Artificial Intelligence, AAAI Press, New Orleans, Louisiana,
USA, 2018, pp. 7759–7764.
[91] E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, C. Nicholas, Malware detection by eating a whole
EXE, 2017. ArXiv preprint arXiv:1710.09435.
[92] H. S. Anderson, P. Roth, EMBER: an open dataset for training static PE malware machine learning models,
2018. ArXiv preprint arXiv:1804.04637.
[93] T. K. Ho, Random decision forests, in: International Conference on Document Analysis and Recognition,
volume 1, IEEE, Montreal, Canada, 1995, pp. 278–282.
[94] Y. Ye, L. Chen, D. Wang, T. Li, Q. Jiang, M. Zhao, Sbmds: An interpretable string based malware detection
system using svm ensemble with bagging, Journal in Computer Virology 5 (2009) 283–293.
[95] R. Islam, R. Tian, L. Batten, S. Versteeg, Classification of malware based on string and function feature
selection, in: Cybercrime and Trustworthy Computing Workshop, IEEE, Ballarat, Australia, 2010, pp. 9–17.
[96] M. Z. Shafiq, S. M. Tabish, F. Mirza, M. Farooq, PE-Miner: Mining structural information to detect malicious
executables in realtime, in: International Workshop on Recent Advances in Intrusion Detection, Springer,
Saint-Malo, France, 2009, pp. 121–141.
[97] N. S. Altman, An introduction to kernel and nearest-neighbor nonparametric regression, The American
Statistician 46 (1992) 175–185.
[98] L. Nataraj, V. Yegneswaran, P. Porras, J. Zhang, A comparative assessment of malware classification using
binary texture analysis and dynamic analysis, in: ACM Workshop on Security and Artificial Intelligence,
ACM, Chicago, IL, USA, 2011, pp. 21–30.
[99] H.-J. Kim, Image-based malware classification using convolutional neural network, in: Advances in Computer
Science and Ubiquitous Computing, Springer, Taichung, Taiwan, China, 2017, pp. 1352–1357.
[100] X. Liu, Y. Lin, H. Li, J. Zhang, A novel method for malware detection on ml-based visualization technique,
Computers & Security 89 (2020) 101682.
[101] K. Rieck, T. Holz, C. Willems, P. Düssel, P. Laskov, Learning and classification of malware behavior, in:
International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer,
Paris, France, 2008, pp. 108–125.
[102] A. Mohaisen, O. Alrawi, M. Mohaisen, AMAL: high-fidelity, behavior-based automated malware analysis and
classification, Computers & Security 52 (2015) 251–266.
[103] M. Abdelsalam, R. Krishnan, Y. Huang, R. Sandhu, Malware detection in cloud infrastructures using convo-
lutional neural networks, in: International Conference on Cloud Computing, IEEE, San Francisco, CA, USA,
2018, pp. 162–169.
[104] J. Franklin, The elements of statistical learning: data mining, inference and prediction, The Mathematical
Intelligencer 27 (2005) 83–85.
[105] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: International
Conference on Learning Representations, OpenReview.net, Toulon, France, 2017, pp. 1–14.
[106] M. Bailey, J. Oberheide, J. Andersen, Z. M. Mao, F. Jahanian, J. Nazario, Automated classification and
analysis of internet malware, in: International Workshop on Recent Advances in Intrusion Detection, Springer,
Gold Goast, Australia, 2007, pp. 178–197.
[107] S. Wang, Z. Chen, X. Yu, D. Li, J. Ni, L. Tang, J. Gui, Z. Li, H. Chen, P. S. Yu, Heterogeneous graph matching
networks for unknown malware detection, in: International Joint Conference on Artificial Intelligence, ijcai.org,
Macao, China, 2019, pp. 3762–3770.
[108] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning
phrase representations using rnn encoder–decoder for statistical machine translation, in: Empirical Methods
in Natural Language Processing, ACL, Doha, Qatar, 2014, pp. 1724–1734.
[109] Y. Ye, T. Li, Y. Chen, Q. Jiang, Automatic malware categorization using cluster ensemble, in: ACM SIGKDD
Conference on Knowledge Discovery and Data Mining, ACM, Washington, DC, USA, 2010, pp. 95–104.
[110] I. Santos, F. Brezo, X. Ugarte-Pedrero, P. G. Bringas, Opcode sequences as representation of executables for
data-mining-based unknown malware detection, Information Sciences 231 (2013) 64–82.
[111] J. Zhang, Z. Qin, H. Yin, L. Ou, Y. Hu, Irmd: malware variant detection using opcode image recognition, in:
International Conference on Parallel and Distributed Systems, IEEE, Wuhan, China, 2016, pp. 1175–1180.
[112] G. Sun, Q. Qian, Deep learning and visualization for identifying malware families, IEEE Transactions on
Dependable and Secure Computing 18 (2018) 283–295.
[113] W. W. Cohen, Learning trees and rules with set-valued features, in: AAAI/IAAI, AAAI Press, Portland,
Oregon, USA, 1996, pp. 709–716.
[114] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation 9 (1997) 1735–1780.

38
Accepted by ELSEVIER Computers & Security. REFERENCES

[115] Y. Qiao, Y. Yang, L. Ji, J. He, Analyzing malware by abstracting the frequent itemsets in api call sequences,
in: IEEE International Conference on Trust, Security and Privacy in Computing and Communications, IEEE,
Melbourne, Australia, 2013, pp. 265–270.
[116] Z. Zhang, P. Qi, W. Wang, Dynamic malware analysis with feature engineering and feature learning, in: AAAI
Conference on Artificial Intelligence, AAAI Press, New York, NY, USA, 2020, pp. 1210–1217.
[117] A. Kapoor, S. Dhavale, Control flow graph based multiclass malware detection using bi-normal separation,
Defence Science Journal 66 (2016) 138–145.
[118] J. Yan, G. Yan, D. Jin, Classifying malware represented as control flow graphs using deep graph convolu-
tional neural network, in: IEEE/IFIP International Conference on Dependable Systems and Networks, IEEE,
Portland, OR, USA, 2019, pp. 52–63.
[119] X. Ling, L. Wu, W. Deng, Z. Qu, J. Zhang, S. Zhang, T. Ma, B. Wang, C. Wu, S. Ji, MalGraph: Hier-
archical graph neural networks for robust Windows malware detection, in: IEEE Conference on Computer
Communications, IEEE, Virtual Event, 2022, pp. 1998–2007.
[120] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, L. Bottou, Stacked denoising autoencoders:
Learning useful representations in a deep network with a local denoising criterion, Journal of Machine Learning
Research 11 (2010) 3371–3408.
[121] M. Hassen, P. K. Chan, Scalable function call graph-based malware classification, in: ACM on Conference on
Data and Application Security and Privacy, ACM, Scottsdale, AZ, USA, 2017, pp. 239–248.
[122] H. Jiang, T. Turki, J. T. Wang, Dlgraph: Malware detection using deep learning and graph embedding,
in: International Conference on Machine Learning and Applications, IEEE, Orlando, FL, USA, 2018, pp.
1029–1033.
[123] S. Zhao, X. Ma, W. Zou, B. Bai, Deepcg: classifying metamorphic malware through deep learning of call
graphs, in: International Conference on Security and Privacy in Communication Systems, Springer, Orlando,
FL, USA, 2019, pp. 171–190.
[124] M. Krčál, O. Švec, M. Bálek, O. Jašek, Deep convolutional malware classifiers can learn from raw executables
and labels only, in: International Conference on Learning Representations – Workshop Track, OpenReview.net,
Vancouver, BC, Canada, 2018, pp. 1–4.
[125] S. E. Coull, C. Gardner, Activation analysis of a byte-based deep neural network for malware classification,
in: IEEE Security and Privacy Workshops, IEEE, San Francisco, CA, UAS, 2019, pp. 21–27.
[126] L. Nataraj, S. Karthikeyan, G. Jacob, B. S. Manjunath, Malware images: visualization and automatic classi-
fication, in: International Symposium on Visualization for Cyber Security, ACM, Pittsburgh, PA, USA, 2011,
pp. 1–7.
[127] X. Ling, L. Wu, S. Wang, T. Ma, F. Xu, A. X. Liu, C. Wu, S. Ji, Multilevel graph matching networks for deep
graph similarity learning, IEEE Transactions on Neural Networks and Learning Systems (TNNLS) (2021).
[128] B. G. Ryder, Constructing the call graph of a program, IEEE Transactions on Software Engineering 5 (1979)
216–226.
[129] X. Ling, L. Wu, S. Wang, G. Pan, T. Ma, F. Xu, A. X. Liu, C. Wu, S. Ji, Deep graph matching and searching
for semantic code retrieval, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (2021).
[130] E. Quiring, A. Maier, K. Rieck, Misleading authorship attribution of source code using adversarial learning,
in: USENIX Security Symposium, USENIX Association, Santa Clara, CA, USA, 2019, pp. 479–496.
[131] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual explanations
from deep networks via gradient-based localization, in: International Conference on Computer Vision, IEEE,
Venice, Italy, 2017, pp. 618–626.
[132] A. Kurakin, I. Goodfellow, S. Bengio, Adversarial examples in the physical world, in: International Conference
on Learning Representations, OpenReview.net, Toulon, France, 2017, pp. 1–14.
[133] MIT-IBM Watson AI Lab, Robust malware detection challenge, 1st Workshop on Adversarial Learning Meth-
ods for Machine Learning and Data Mining in KDD 2019 https://sites.google.com/view/advml/Home/
advml-2019/advml19-challenge, 2019. Online (last accessed October 15, 2020).
[134] S. Verwer, A. Nadeem, C. Hammerschmidt, L. Bliek, A. Al-Dujaili, U.-M. O’Reilly, The robust malware detec-
tion challenge and greedy random accelerated multi-bit search, in: ACM Workshop on Artificial Intelligence
and Security (AISec), ACM, Virtual Event, 2020, pp. 61–70.
[135] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Advances in Neural
Information Processing Systems, Curran Associates, Inc., Long Beach, CA, USA, 2017, pp. 4765–4774.
[136] M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in: International Conference on
Machine Learning, PMLR, Sydney, NSW, Australia, 2017, pp. 3319–3328.
[137] H. S. Anderson, A. Kharkar, B. Filar, D. Evans, P. Roth, Learning to evade static PE machine learning malware

39
Accepted by ELSEVIER Computers & Security. REFERENCES

models via reinforcement learning, 2018. ArXiv preprint arXiv:1801.08917.

[138] H. Peng, F. Long, C. Ding, Feature selection based on mutual information criteria of max-dependency, max-
relevance, and min-redundancy, IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (2005)
1226–1238.
[139] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, A. Swami, Practical black-box attacks against
machine learning, in: Asia Conference on Computer and Communications Security, ACM, Abu Dhabi, United
Arab Emirates, 2017, pp. 506–519.
[140] D.-C. Dang, P. K. Lehre, Self-adaptation of mutation rates in non-elitist populations, in: International
Conference on Parallel Problem Solving from Nature, Springer, Edinburgh, UK, 2016, pp. 803–813.
[141] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT Press, Cambridge, Massachusetts,
USA / London, England, 2018. Second edition (in progress).
[142] Cuckoo Team, Cuckoo Sandbox, https://cuckoosandbox.org, 2020. Online (last accessed September 13,
2020).
[143] SHex-Rays, IDA Pro, https://www.hex-rays.com/products/ida/, 2020. Online (last accessed September 13,
2020).
[144] Telock, Telock Version 0.98 for Windows, https://www.softpedia.com/get/Programming/
Packers-Crypters-Protectors/Telock.shtml, 2020. Online (last accessed October 25, 2020).
[145] B. Cheng, J. Ming, J. Fu, G. Peng, T. Chen, X. Zhang, J.-Y. Marion, Towards paving the way for large-
scale windows malware analysis: Generic binary unpacking with orders-of-magnitude performance boost, in:
ACM SIGSAC Conference on Computer and Communications Security, ACM, Toronto, ON, Canada, 2018,
pp. 395–411.
[146] K. Shoair, Dr0p1t-framework, https://github.com/D4Vinci/Dr0p1t-Framework, 2020. Online (last accessed
October 25, 2020).
[147] B. J. Kwon, J. Mondal, J. Jang, L. Bilge, T. Dumitraş, The dropper effect: Insights into malware distribution
with downloader graph analytics, in: ACM SIGSAC Conference on Computer and Communications Security,
ACM, Denver, Colorado, USA, 2015, pp. 1118–1129.
[148] H. Aghakhani, F. Gritti, F. Mecca, M. Lindorfer, S. Ortolani, D. Balzarotti, G. Vigna, C. Kruegel, When
malware is packin’heat; limits of machine learning classifiers based on static analysis features, in: Network and
Distributed System Security Symposium, The Internet Society, San Diego, California, USA, 2020, pp. 1–20.
[149] G. Poulios, C. Ntantogian, C. Xenakis, ROPInjector: Using return oriented programming for polymorphism
and antivirus evasion, in: Black Hat USA, blackhat.com, Las Vegas, NV, USA, 2015, pp. 1–11.
[150] Microsoft Azure, 2020 machine learning security evasion competition, https://github.com/Azure/
2020-machine-learning-security-evasion-competition, 2021. Online (last accessed January 20, 2021).
[151] E. Quiring, L. Pirch, M. Reimsbach, D. Arp, K. Rieck, Against all odds: Winning the defense challenge in an
evasion competition with diversification, 2020. ArXiv preprint arXiv:2010.09569.
[152] Í. Íncer Romeo, M. Theodorides, S. Afroz, D. Wagner, Adversarially robust malware detection using monotonic
classification, in: International Workshop on Security and Privacy Analytics, ACM, Tempe, AZ, USA, 2018,
pp. 54–63.
[153] VirusTotal, YARA in a nutshell, https://github.com/virustotal/yara, 2020. Online (last accessed December
15, 2020).
[154] S. Chen, N. Carlini, D. Wagner, Stateful detection of black-box adversarial attacks, in: ACM Workshop on
Security and Privacy on Artificial Intelligence, ACM, Taipei, Taiwan, China, 2020, pp. 30–39.
[155] R. Labaca-Castro, L. Muñoz-González, F. Pendlebury, G. D. Rodosek, F. Pierazzi, L. Cavallaro, Universal
adversarial perturbations for malware, 2021. ArXiv preprint arXiv:2102.06747.
[156] S. Sasaki, S. Hidano, T. Uchibayashi, T. Suganuma, M. Hiji, S. Kiyomoto, On embedding backdoor in
malware detectors using machine learning, in: International Conference on Privacy, Security and Trust, IEEE,
Fredericton, NB, Canada, 2019, pp. 1–5.
[157] L. Muñoz-González, B. Biggio, A. Demontis, A. Paudice, V. Wongrassamee, E. C. Lupu, F. Roli, Towards
poisoning of deep learning algorithms with back-gradient optimization, in: ACM Workshop on Artificial
Intelligence and Security, ACM, Dallas, TX, USA, 2017, pp. 27–38.
[158] G. Severi, J. Meyer, S. Coull, A. Oprea, Explanation-guided backdoor poisoning attacks against malware
classifiers, in: USENIX Security Symposium, USENIX Association, Virtual Event, 2021, pp. 1487–1504. (First
submitted to arXiv.org on March 2020 at https://arxiv.org/abs/2003.01031v1.).
[159] T. Shapira, D. Berend, I. Rosenberg, Y. Liu, A. Shabtai, Y. Elovici, Being single has benefits. instance poisoning
to deceive malware classifiers, 2020. ArXiv preprint arXiv:2010.16323.
[160] A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, T. Goldstein, Poison frogs! targeted

40
Accepted by ELSEVIER Computers & Security. REFERENCES

clean-label poisoning attacks on neural networks, in: Advances in Neural Information Processing Systems,
Curran Associates, Inc., Montréal, Canada, 2018, pp. 6106–6116.
[161] A. Ali, B. Eshete, Best-effort adversarial approximation of black-box malware classifiers, in: EAI International
Conference on Security and Privacy in Communication Networks, Springer, Washington DC, USA, 2020, pp.
318–338.
[162] N. Papernot, F. Faghri, N. Carlini, I. Goodfellow, R. Feinman, A. Kurakin, C. Xie, Y. Sharma, T. Brown,
A. Roy, et al., Technical report on the cleverhans v2.1.0 adversarial examples library, 2016. ArXiv preprint
arXiv:1610.00768.
[163] X. Ling, S. Ji, J. Zou, J. Wang, C. Wu, B. Li, T. Wang, DEEPSEC: A uniform platform for security analysis
of deep learning model, in: IEEE Symposium on Security and Privacy, IEEE, San Francisco, USA, 2019, pp.
673–690.
[164] Y. Li, W. Jin, H. Xu, J. Tang, DeepRobust: A pytorch library for adversarial attacks and defenses, 2020.
ArXiv preprint arXiv:2005.06149.

41