PANOPLY: Low-TCB Linux Applications

with SGX Enclaves

Shweta Shinde Dat Le Tien† Shruti Tople Prateek Saxena

National University of Singapore University of Oslo National University of Singapore National University of Singapore
[email protected] [email protected] [email protected] [email protected]

Abstract—Intel SGX, a new security capability in emerging has been a threat to privileged software layer, often targeting
CPUs, allows user-level application code to execute in hardware- vulnerabilities in privileged code such as the OS. In this paper,
isolated enclaves. Enclave memory is isolated from all other we envision providing the benefits of privilege separation and
software on the system, even from the privileged OS or hypervi- isolation based on a strong line of defense against OS-resident
sor. While being a promising hardware-rooted building block, malware. Such a defense is based on a new trusted computing
enclaves have severely limited capabilities, such as no native
primitive, which can isolate a sensitive user-level application
access to system calls and standard OS abstractions. These OS
abstractions are used ubiquitously in real-world applications. from a compromised OS. Hardware support for this primitive
has become available in commodity CPUs in the form of
In this paper, we present a new system called PANOPLY which Intel SGX, which can run such hardware-isolated application
bridges the gap between the SGX-native abstractions and the instances in enclaves [40]. Intel SGX provides a hardware-
standard OS abstractions which feature-rich, commodity Linux isolated memory region which can be remotely attested. SGX
applications require. PANOPLY provides a new abstraction called a hardware supports execution of many enclaves simultaneously.
micro-container (or a “micron”), which is a unit of code and data
isolated in SGX enclaves. Microns expose the standard POSIX Recent research has demonstrated how to enforce useful
abstractions to application logic, including access to filesystems, low-level guarantees using SGX — for instance, protection
network, multi-threading, multi-processing and thread synchro- of certain cryptographic keys in memory [30], [32], [42],
nization primitives. Further, PANOPLY enforces a strong integrity
[44], [47], verifiable execution of code snippets [51], and
property for the inter-enclave interactions, ensuring that the
execution of the application follows the legitimate control and authenticated data delivery [56]. While these properties are
data-flow even if the OS misbehaves. Thus, commodity Linux useful, their applicability has been limited to small, selected
applications can enhance security by splitting their application pieces of application logic rather than end-to-end applications.
logic in one or more microns, or by importing micron-libraries, For larger, richer applications, the best known approache has
with little effort. In contrast to previous systems that enable been to use library OSes [21], [52]. In such architectures, the
comparable richness, PANOPLY offers two orders of magnitude application is bundled together with a large TCB of millions
lower TCB (about 20 KLOC in total), more than half of which of lines of code, emulating the OS logic inside the enclave.
is boiler-plate and can be automatically verified in the future.
We demonstrate how PANOPLY enables much stronger security In this paper, we propose a new system called PANOPLY,
in 4 real-world applications — including Tor, OpenSSL, and web designed with an eye towards minimizing TCB and yet offering
services — which can base security on hardware-root of trust. rich OS abstractions to enclaved code. PANOPLY introduces
a new abstraction we call a micro-container (or “micron”
I. I NTRODUCTION for short). A micron is a unit of application logic which
runs on the Intel SGX hardware enclaves — thus it offers
Privilege separation and isolation are cornerstones in de- a strong isolation against an adversarial OS. Microns expose
sign of secure computer systems. Machine isolation is used the rich gamut of standard Linux abstractions to application
for designing fault-tolerant network services, virtualization logic, much more expressiveness in enclave-bound code than
for isolating OSes, library OSes and containers for isolat- several previous systems (e.g. Haven). For instance, micron-
ing applications. However, these primitives trust a privileged enabled logic can readily use multi-processing (fork-exec),
software component (e.g. a hypervisor or OS) for ensuring multi-threading, event registration / callbacks (e.g. signals), in
their claimed security guarantees. For several decades, malware addition to supporting the standard Linux system calls.
† This work was done while the author was a visiting graduate intern at PANOPLY prioritizes a minimal TCB over performance
National University of Singapore. as a goal. It uses a simple design philosophy of delegate-
rather-than-emulate that contrasts previous systems (e.g. li-
brary OSes). PANOPLY delegates the implementation of OS
Permission to freely reproduce all or part of this paper for noncommercial
purposes is granted provided that copies bear this notice and the full citation
abstractions to the underlying OS itself, rather than emulating
on the first page. Reproduction for commercial purposes is strictly prohibited it inside the enclave. PANOPLY microns implement a small
without the prior written consent of the Internet Society, the first-named author set of checks which enables them to detect malicious re-
(for reproduction of an entire paper only), and the author’s employer if the sponse from the OS, and abort if so. In line with this design
paper was prepared within the scope of employment. choice, unlike library OSes, PANOPLY does not “virtualize”
NDSS ’17, 26 February - 1 March 2017, San Diego, CA, USA
Copyright 2017 Internet Society, ISBN 1-1891562-46-0 microns by giving them each their own namespace. This choice
http://dx.doi.org/10.14722/ndss.2017.23500 eliminates a massive amount of namespace management logic
that emulates the underlying OS from the enclave TCB. With level protocol guarantees for their host application beyond
these simple design principles, we show that the total TCB key-protection. In all of our case studies, the porting effort
of PANOPLY can be about 20 KLOC (apart from the original is modest, incurring average 905 lines of code changes.
application logic), which is 2 orders of magnitude smaller than
previous LibraryOS systems. We believe such TCB is within We have compared PANOPLY application to a state-of-
realm of automated verification in the near future. the-art Linux library-OS called Graphene-SGX [3], [52] that
supports SGX hardware. First, PANOPLY applications have two
A second feature of PANOPLY is that it enables a plug- order of magnitude smaller TCB. Second, we find that the
and-play architecture, wherein security architects can create performance of the two on our case studies is comparable. We
as many microns as needed and host them across multiple provide a detailed breakdown of the performance overheads
OS processes. Such a design keeps compatibility with multi- on real hardware in both systems. We find that most source
process and multi-threaded application designs — several of overhead is from the Intel’s SDK, incurred for creating and
applications such as servers inherently use multiple processes initializing empty enclaves. PANOPLY-specific code introduces
for security (e.g. for privilege separation), performance, as well an additional average CPU overhead of 24%.
as for isolating crash failures (better availability). Security
architects can split monolithic applications across multiple Contributions. In summary, we make the following contribu-
microns easily, or import security-sensitive libraries that are tions in this paper:
implemented as microns. The PANOPLY architecture allows
analysts to partition application by adding modest amount • PANOPLY System. PANOPLY is the first system which
of annotation to source code and compiling with PANOPLY’s supports applications with multi-threading, multi-
infrastructure. PANOPLY instruments each micron to ensure processing, event management in enclaves. Our inter-
that all inter-micron control and data-flow interactions are micron flow integrity ensures that the applications
secured against the OS. In effect, PANOPLY ensures that an preserve the high-level guarantees.
application partitioned into multiple microns will either exe- • Usage in real-world Applications. We retrofit 4 appli-
cute with the same control and data-flow as the unpartitioned cations into PANOPLY architecture that require on an
original application, even under adversarial influence, or abort. average 905 lines of code changes.
To achieve such a guarantee, we propose defenses that extend • Evaluation. After porting to PANOPLY, we report a
beyond simple data tampering attacks (e.g. Iago attacks [26]) performance overhead of 24% and an average TCB in-
— our defenses enforce control and data-flow integrity for crease of 19.62 KLOC per application. In comparison
inter-enclave transitions, with freshness and authentication to previous systems, PANOPLY reduces the TCB by 2
guarantees built-in. orders of magnitude while lowering the performance
overhead by 5 − 10%.
PANOPLY microns expose expressive OS abstractions to ap-
plication logic. To support these, we make several conceptual
advances in its design that are not offered by off-the-shelf Intel
SDKs for SGX [8]. First, microns expose the POSIX abstrac- II. P ROBLEM
tion of creating threads on demand, dynamically requesting
as many threads as the application needs. Previous systems A. Background: Intel SGX Enclaves
have limited this design to a pre-determined number of static
threads executable in an enclave. PANOPLY runtime library Existing hardware-based / hypervisor-based defenses
multiplexes a dynamic number of threads over multiple under- against compromised OSes provide an isolated execution space
lying enclaves. Second, PANOPLY proposes several designs for for executing user-level applications. In the case of SGX,
supporting the semantics of fork-exec. PANOPLY allows these are referred to as enclaves, and a single application
microns to be hosted across multiple OS processes. Third, process can comprise of one or more enclaves. SGX assures
PANOPLY supports the pthreads synchronization interfaces, the confidentiality and integrity of all the sensitive code and
which includes mutexes, barriers, and so on using SGX- data contained within an enclave. The Intel SGX SDK provides
specific abstractions. Finally, as explained earlier, PANOPLY a function call mechanism for SGX applications via ECALL
automatically embeds checks at the micron interfaces, ensuring and OCALL. Specifically, an ECALL is a trusted function call
that inter-enclave control and data-flow conforms to that of the that enters an enclave and OCALL is an untrusted function call
original code. that leaves an enclave [17]. Thus, a user-level application can
invoke code inside an enclave via an Enclave Call (ECALL)
Results. We show 4 case studies of real applications that and get the return values. The enclave can invoke an Outer
use expressive features. The first case study is Tor, a popular Call (OCALL) to execute a function in the untrusted portion of
distributed anonymous communication service [11]. We show the application and receive a return value. The enclave code
how Tor can leverage PANOPLY micron to provide a strong can access all the application memory outside whereas, the
security for its directory service protocol, basing security on non-enclave code cannot access the enclave’s memory. SGX
the SGX hardware-root-of-trust. The second case study is on CPU supports local and remote attestation, so as to check if
a web-server called H2O, which can self-attest the correctness the enclave has loaded the correct code. To this end, the CPU
/ integrity of the served content (e.g., such as in CDNs, computes a measurement by securely hashing the enclaves
serving static content over HTTPS). We also support two case initial state. Thus an entity can attest the measurement to verify
studies of popular libraries, OpenSSL and FreeTDS, that can the initial state of the enclave confirming that the initial state
be imported as microns in other host application. The libraries is “clean” [18], [23]. The enclave is created and loaded as
can be secured to protect secret keys and provide higher- expected without OS tampering.

2
 1 session_t session; Unprivileged Privileged
2 certificate_credentials_t xcred; OS
3 /* Specify the callback function to be used*/ Enclave Enclave
4 #begin privilege_enclave
5 certificate_set_verify_function (xcred, _callback); certificate_set_verify_function()
(a)
6 #end privilege_enclave
7 /* Initialize TLS session */
8 init (&session, TLS_CLIENT); x509_crt_check_hostname()
9 /* Set non-default priorities */ x509_crt_check_hostname()
10 if(non-default)
11 #begin privilege_enclave (b)
false
12 priority_set_direct (session, "%UNSAFE_RENEGO"); true
13 #end privilege_enclave
14 err = handshake(session); priority_set_direct
15 ... } (“UNSAFE_RENEGOTIATION”)
(c)
16
17 static int _callback (session_t session) {
18 x509_crt_t cert;
19 const char *hostname; Fig. 2. (a) Call dropping, b) call spoofing and c) call replay attacks perpetrated
20 ... by the OS during inter-enclave interactions.
21 #begin privilege_enclave
22 ret = x509_crt_check_hostname (cert, hostname);
23 #end privilege_enclave
24 if (!ret) SGX enclaves. The trusted certificate manager service is shared
25 return CERTIFICATE_ERROR;
26 ...
across many applications, and by principle of least privilege,
27 /* Validation successful, continue handshake */ is the only service with access to the “root certificate”. As
28 return 0; can be seen in Figure 1, the FreeTDS application validates a
29 }
30 presented certificate by invoking the trusted certificate manager
31 static SSL_CTX *tds_init_ssl(void) via standard OpenSSL interfaces. Specifically, the FreeTDS
32 {... application does this in 3 steps: it registers a callback with
33 tds_mutex_lock(&tls_mutex);
34 if (!tls_initialized) { the trusted certificate manager (Line 5), passes configuration
35 SSL_library_init(); options (Line 12), and invokes the certificate check (Line 22).
36 tls_initialized = 1;
37 }
38 tds_mutex_unlock(&tls_mutex);
Attacks on Inter-Enclave Interactions. Note that the
39 ...} FreeTDS application enclave invokes the certificate manager
enclave several times, passing rich data. Such communication
necessarily goes through an adversarial channel under the OS
Fig. 1. Code snippet from FreeTDS application for certificate validating of
remote database server’s certificate using OpenSSL. The #pragmas are added control (e.g. an IPC call or control transfer in unprotected
while porting the application to PANOPLY. code). The OS knows the standard OpenSSL interfaces and
the semantics of its interfaces. If enclaves use the standard
interfaces of the Intel SDK, the OS can subvert the ap-
B. Attacks & Challenges plication’s guarantee by causing FreeTDS to accept invalid
certificates. We show concrete attacks on this interface in
To enable end-to-end security guarantees in real applica- Figure 2. First, the OS can drop the call on Line 5, thereby
tions, we aim to address three main challenges: (1) support for disallowing the application to register a callback. The attack
rich OS abstractions, (2) secure interactions between multiple is powerful because abort fails silently, with the effect that
enclaves, and (3) a minimal additional TCB. We demonstrate the certificate validity checks on Line 22 never execute. A
the gap in existing abstractions to meet these challenges with second attack opportunity is to effect a session downgrade
an example, for ensuring higher-level security properties. attack, by forcing the certificate manager to re-negotiate weak
SSL parameters [12], [13]. The SSL protocol has a known flaw
Example. Consider FreeTDS [2], an application that imple- called a session re-negotiation vulnerability, which is patched
ments a streaming protocol (TDS [9]) for remote databases. by the call on Line 12. However, the OS can abort this message
Figure 1 shows a FreeTDS code snippet, which shows that the and this causes the certificate validation to proceed with unsafe
application uses the OpenSSL SSL/TLS library to establish a defaults silently. As a third example, the OS can perform
secure channel with a remote database. Each time it connects a session downgrade by replaying a call from a different
to a remote database, FreeTDS checks if the remote database (previous) session. Specifically, the OS can record the inter-
is an authorized service and has a valid SSL certificate, by enclave message transcript from a different session consisting
connecting to trusted certificate manager service. Our goal is of a invalid certificate with weaker parameters (requesting
to fortify the FreeTDS application against a compromised OS, UNSAFE_RENEGOTIATION). The previous session would
ensuring a key end-to-end security property — the FreeTDS have failed; however, the OS can replay one recorded message
application accepts a certificate as valid if and only if the from that transcript in a session with a strong certificate to
original application would have accepted it as is valid. The cause it to downgrade. A final and fourth attack is on Line 22.
assumption is that the original application is bug-free, but the This is a data replay attack wherein the certificate manager
system administrators wish to secure it against the hosting returns false (signaling an invalid certificate), but the OS
service provider’s infrastructure, which may be infected. We drops the message and replays a true return value from a
can achieve such a defense by using Intel SGX enclaves. previous execution run.
In this example, we wish to run the FreeTDS application These attacks highlight that applications that aren’t de-
and the trusted certificate manager, each in their own separate signed with the objective of running on enclaved-abstraction

3
TABLE I. C OMPARISON OF PROGRAMMING CONSTRUCT SUPPORT AND TABLE II. POSIX API S UPPORT IN PANOPLY . C OLUMN 3 DENOTES
SECURITY PROPERTIES OF PANOPLY AND OTHER EXISTING SYSTEMS NUMBER OF API S FOR SUB - CLASSES OF SERVICES . C OLUMN 4 LISTS IF
THE PANOPLY CAN GUARANTEE TO PRESERVE THE SEMANTICS OF THE
Multi Thread
SERVICE (S A F E ) OR NOT (W I L D ).
Abstr Sys Threading Synch Fork Low
STI
actions Calls Stat Dyn Mutex All Exec TCB
POSIX # of API
Intel SDK 7 X 7 X 7 7 7 X Service Description
Standard APIs Type
Haven X X X X X 7 7 7
SGX Graphene X X X X X X 7 7
Process Creation and Control 5 Safe
PANOPLY 3 X X X 3 3 3 3
Signals 6 Wild
Timers 5 Wild
Core
File and Directory Operations 37 Safe
Services
Pipes 4 Safe
will be susceptible to subtle vulnerabilities. Further, there is C Library (Standard C) 66 Wild
gap between the SGX-native low-level guarantees (of remote I/O Port Interface and Control 40 Wild
Real-Time Signals 4 Wild
attestation and memory isolation) and those needed to ensure Clocks and Timers 1 Wild
safe end-to-end execution of applications. Several previous Semaphores 2 Safe
Real-time
works have enforced confidentiality of private keys and au- Extensions
Message Passing 7 Safe
Shared Memory 6 Safe
thenticated data delivery, while our emphasis is on securing Asynchronous and Synchronous I/O 29 Wild
the end-to-end application semantics. Memory Locking Interface 6 Wild
Thread Creation, Control & Cleanup 17 Safe
Supporting Rich OS Abstractions With Low TCB. The Thread Scheduling 4 Wild
Threads
example highlights an even more basic challenge in supporting Extensions
Thread Synchronization 10 Safe
real-world applications. The FreeTDS code snippet makes Signal Delivery 2 Wild
Signal Handling 3 Safe
extensive use of OS abstractions. The code is multi-threaded, Total Total 254
where a new thread of execution is dynamically created for
each request to initiate connection with a new server. Line
33−38 in Figure 1 shows that all threads use mutex objects for invoke system calls (e.g. filesystem, network, and so on)
synchronizing operations such as initializing the TLS library. just as non-enclave code in the same Linux process would.
The original application is not designed to be executed in Microns do have their private address space, which is isolated
multiple enclaves, which do not share any address space in enclaves, and can share an arbitrary amount of public
(unlike threads). The application assumes it can create an address space with the host process. By default, the code and
arbitrary number of threads at runtime on-demand. FreeTDS data of the micron-based logic is allocated in private memory.
uses standard system calls, such as gethostname() and There exists an explicit PANOPLY API by which micron can
poll () (implicitly for callback registrations). However, communicate with code outside.
SGX and Intel SDK do not provide native support for any
of these abstractions. Table I shows the gap between the Micron code has access to a rich subset of the POSIX v1.3
expressiveness offered by the Intel SDK, library OSes and the API, listed in Table II which PANOPLY supports. The exposed
requirements of real-world applications. The research question API includes system calls for filesystems, network, multi-
is how to enable support for such abstractions with minimal processing, multi-threading, synchronization primitives (via
effort, enabling security architects to quickly experiment with pthreads), signals and event management (via libenv).
ways to privilege-separate their applications. The service API’s are classified as SAFE or WILD based on
whether PANOPLY guarantees to preserve their semantics or
The state-of-the-art solutions for offering rich expressive- not. Specifically, for SAFE API’s PANOPLY guarantees POSIX
ness rely on library OSes (such as Haven [21] and Graphene- semantics and application can assume that semantic correct-
SGX [52]). Library OSes provide an abstraction of a vir- ness (barring aborts). The API design choice is intentionally
tualized process namespace to application code. Namespace chosen so as to eliminate much of shared libraries (e.g. libc)
virtualization requires emulating much of the OS logic within from the TCB of the enclave-enabled micron code. Figure 3
enclaves. This approach offers good compatibility with existing shows parts of a standard Linux application that would be in
code; however, it comes at the peril of bloating the enclave the TCB (shaded grey) in our architecture. PANOPLY embeds
TCB. Library OSes have reported TCB sizes in millions of a thin shim library which interfaces with the PANOPLY API.
lines code. Systems which have been formally verified have The shim library is added to each enclave at compile-time, and
been smaller by orders of magnitude [37]; thus, library OSes is largely transparent to the developer.
are not within the realm of practical verification in the near
future. Compiled micron code includes PANOPLY’s shim library,
which plays a key role in protecting against a malicious
III. S OLUTION OS. First, the shim library acts as controller or manager and
provides micron management functionalities such as support
A. PANOPLY Overview to create / destroy microns and providing identity management
for microns, using SGX-specific features. More importantly,
PANOPLY provides a new design point in systems that
the shim code enforces a stronger integrity property across all
enable rich Linux applications on SGX enclaves. PANOPLY
microns in an application — it ensures that control and data-
provides the abstraction of a micro-container or micron, which
flows between microns conforms to the control and data-flow
is a unit of application logic that is enclave-bound.
of the original application. We call this the inter-micron flow
A Linux process can import or host one or more microns. integrity property. The shim ensures inter-micron flow integrity
Microns do not get their own virtualized namespace, but by establishing an authenticated secure-channel protocol be-
instead share it with their host Linux process; that is, they tween microns automatically. To assert the importance of this

4
 Source 1 Compiler Creating
Micron 2
Code Instrumentation Microns
Enclave-bound Logic Enclave E1
Enclave-bound M1
Add calls to Code
Panoply Shim Lib Trusted SDK Lib
Panoply API E2 M2
Intel PL
Add Flow Panoply
Non-enclave Untrusted SGX
libc.so Checks Shim
Logic SDK Lib SDK Panoply
Programmer
Linux User-level Process Application
Annotations Panoply
Linux Kernel SGX Driver
Fig. 4. System Overview. PANOPLY takes in the original program and the
Fig. 3. Overview of codename architecture. All the regions within an enclave partitioning scheme as input. It first divides the application into enclaves and
are trusted, the regions shaded as black are untrusted, grey shaded regions are then enforces inter-micron flow integrity, to produce PANOPLY application.
newly added / modified as a part of codename system.

system. The compilation phase adds code for inter-micron flow

global property, we demonstrate real attacks on several case integrity and PANOPLY APIs.
studies similar to those in Section II-B, which succeed even if
Out-of-scope Goals. The choice of partitioning scheme is or-
memory isolation properties are enforced locally on all of the
thogonal to our work and is left to the security analyst. Existing
enclaves. Second, the shim library performs checks for Iago
tools for program partitioning could be leveraged here [22],
attacks [26], safeguarding against low-level data-tampering for
[24]. Instead, PANOPLY focuses on porting the partitioned code
OS services. It acts as an interface to invoke other services such
to enclaves. PANOPLY does not reason about the functional
as threading, synchronization and event management.
correctness of original application implementation. Any bugs
To achieve low TCB, PANOPLY chooses a delegate-rather- or vulnerabilities in the original application would persist in the
than-emulate strategy. First, PANOPLY delegates all the system micron-based application. PANOPLY cannot prevent denial-of-
calls to the untrusted OS. PANOPLY intercepts the calls to service attacks from the OS, since SGX itself does not provide
the glibc API, which allows us to leave the glibc library this guarantee. Our system currently does not take special
outside enclave TCB (Figure 3). Second, PANOPLY re-thinks measures to thwart enclave side-channels. However, one can
the design of threading, forking and other interfaces so as employ orthogonal defenses for enclave side-channels [30],
to not emulate the entire OS logic in the TCB, but instead [44]. Lastly, we blindly trust all analyst-inserted annotations
delegating it to the OS. PANOPLY delegates the scheduling and instrumentation to be secure and correct. We trust the
logic with the underlying OS. Thus, the application scheduling SGX hardware, which includes a secure implementation of
is not guaranteed to be same as the original code (hence API (a) isolated memory, (b) cryptographic attestation for enclaves,
is WILD). However, this trade-off is justified because OS can and (c) random-number generator.
anyways launch a DOS attack and is in-charge of enclave
scheduling. As an advantage, it allows us to place minimal IV. PANOPLY D ESIGN & S ECURITY
number of checks within the enclave, thus significantly re-
ducing the TCB. Lastly, PANOPLY modularly includes API In designing PANOPLY, we aim to support essential UNIX
calls in the enclave i.e., only the APIs which are used by a abstractions as well as provide necessary security guarantees
given enclave-bound code are included inside the enclave. This for single or multi-micron execution of an application. We
choice is inspired from micro-kernels to reduce the TCB. implement several checks within each micron, which allows
us to adhere to delegate-rather-than-emulate design decision.
We describe the important design choices in PANOPLY and its
B. Usage Model & Scope security guarantees.
PANOPLY consists of a set of runtime libraries (including
the shim library), and a build toolchain that assists developers A. Runtime Micron Management
in preparing microns. PANOPLY takes as input the application PANOPLY processes the source code along with user an-
program source code and per-function programmer annotations notations to identify how the analyst wishes to separate the
to specify which micron should execute that function. Thus, application logic among microns. As done in several other
if the analyst wishes to partition the application into multiple works [24], [35], PANOPLY partitions the micron application
microns, she can annotate different functions with correspond- code accordingly. PANOPLY instruments it to output a micron
ing micron identifiers. Functions that are not marked with binary file (a shared library) at the end. The PANOPLY shim
any micron identifiers can be bundled and delegated to one library ensures that the final micron code supports secure
separate micron by default. In cases that PANOPLY is not micron initialization and inter-micron flow integrity.
able to identify the micron for a function, it prompts the
analyst for providing additional annotations or sanitization Micron Initialization and Identity Establishment. PANOPLY
code. PANOPLY instruments the application, creating multiple creates an instance of a micron within an enclave via Intel
micron binaries, each embedded with its own shim code. Each SGX SDK API which takes the micron binary file as an input.
micron is compiled as a library package (e.g. micron-A.so). PANOPLY generates the micron binary file based on the devel-
It consists of 3 libraries internally: PANOPLY shim library, the oper annotations provided in the source code. If the micron is
Intel SDK library and any other libraries that micron code created successfully, the hardware returns an identifier which
uses. Figure 4 shows the schematic view of the PANOPLY is a unique value for that instance of micron. The OS assigns a

5
 Micron A Micron B Micron C Message Freshness: The OS can replay a previously cap-
B i A i B j
C j
tured message to arbitrarily invoke functions of the receiver
Call foo ()
microns (for e.g., line 25 in Figure 1). To prevent against
i
Execute
replay attacks, PANOPLY ensures that every call message
foo() Call bar ()
j from one micron to another is distinctly identifiable. This
Wait for Execute
allows us to maintain the freshness guarantees for every valid
Wait for
Ack_foo
i
Ack_bar
j
bar() message exchanged between microns. To achieve this, the
j = j+1 Send Ack_bar j = j+1 sender micron generates a 128-bit fresh nonce using Intel
Execute
foo()
j
SGX’s random number generator (RDRAND)1 at the start of
i = i+1 Send Ack_foo
i
i = i+1 session with each micron, as a session-id between a pair of
microns. The nonce is incremented as a counter / sequence
(a) (b) (c) number in all the subsequent messages and authenticated-
encrypted to prevent the OS from tampering it. Each micron
Fig. 5. (a) Micron A is a sender micron that makes a call to foo function stores a mapping of micron-id 7→ session-id for ev-
in micron B. (b) Micron B is both a sender as well as receiver micron that
executes function foo and invokes function bar in micron C. (c) Micron C is
ery micron it communicates with. Figure 5 shows the protocol
a receiver micron that executes function bar. for communicating between three microns. The sender attaches
this nonce in the first call message. The receiver verifies if the
incoming message is from an authorized sender. On validating
handle for this identifier, however PANOPLY does not trust the the authenticity of the sender, it checks if it is the first message
OS handles. Instead, PANOPLY shim library assigns a micron- from the particular sender during the application execution.
identity for each micron instance and internally maintains a If it is the first message of a session, the receiver sets the
mapping of this name and the hardware identifier returned nonce value as the session-id for communicating with the
by SGX. PANOPLY uses this micron-identity for all further sender micron. For other subsequent messages, the receiver
inter-micron interactions. Before starting any interactions, the micron checks if the session-id is in the expected sequence.
PANOPLY shim code attests other microns by the processor For all valid incoming call messages, the receiver micron
for an attestation quote and verifies the measurement from the executes the requested function and increments the session-id
public attestation service [33]. on successful execution of the function (Figure 5 (b), (c)). The
sender micron waits for the receiver micron to complete the
Inter-micron Flow Integrity. Inter-micron flow integrity en- execution and return (Figure 5 (a), (b)). Thus, the application
sures that the application exhibits the same control and data executes sequentially and proceeds in lock steps.
flow across multi-micron execution, as intended in the orig-
inal application. In our design, we consider the interaction Reliable Delivery: When delivering a message from one
between multiple microns as a communication protocol where micron to another, the OS can arbitrarily drop the call messages
the adversarial OS acts as a mediator between the microns. (for e.g., line 4 in Figure 1). This results in silent abort / failure
PANOPLY guarantees that micron execution is protected against of the communication between two microns. To avoid this,
attacks such as silent aborts or message replay from the PANOPLY introduces an acknowledgment message similar to
underlying OS. For this purpose, PANOPLY enforces a secure ACK in the TCP protocol. After the receiver micron executes
and authenticated inter-micron protocol with the following the requested function in the call message, it appends the
design. session-id with an ACK message to the sender micron and
increments the session-id. The sender micron checks whether
Confidentiality & Integrity: During inter-micron commu- the session-id attached with the ACK is valid. If the check
nication, the (mediator) OS can observe all the information is successful, the sender micron increments its own session-id
that is exchanged between the microns. Moreover, it can and is ready to make the next call message to the receiver
change the values of the incoming and outgoing messages from micron (Figure 5 (a), (b)). If the sender fails to receive
the microns. To block this capability of the OS, PANOPLY an acknowledgment, it successfully detects that the OS has
performs authenticated encryption of every incoming and silently dropped its messages. The sender micron aborts its
outgoing message of the micron. execution if the ACK is either not received or contains an
Sender / Receiver Authentication: When a micron is ex- invalid session-id.
ecuting, the OS can impersonate to be a sender micron and
spoof spurious calls to a receiver micron (for e.g., Line 13 in B. Expressiveness with Low TCB
Figure 1). It can also impersonate to be a receiver and hijack all
Enclaves are limited in terms of the programming ex-
the micron-bound calls. To prevent such spurious messaging
pressiveness they support. Specifically, standard C primitives
from the OS, PANOPLY ensures that only a pre-defined set of
such as system calls, networking APIs, file system operations,
authorized microns included in the application interact with
multi-threading, multi-processing and event handling do not
each other. For this, it makes use of the secure mapping of
work out-of-the-box within enclaves. As a recourse, PANOPLY
micron identities to its instance which is established during
addresses each of these limitations by creating microns.
the micron initialization phase. PANOPLY shim code checks the
authenticity of each micron-identity for all call-rets points System Calls. Enclave code cannot directly make a system
— only legitimate call sites can invoke the respective functions call to the OS. Therefore, PANOPLY redirects all the system
and return back. Further, it also checks if a particular micron is
authorized to perform a given interaction. PANOPLY discards 1 This number is passed through an extractor to generate a cryptographically
messages from unauthorized microns and aborts execution. secure random number.

6
calls from the enclave code to our custom wrappers in the Virtual Threads
PANOPLY shim library. The shim library is responsible to make
the correct OCALL and ECALL to invoke these calls in the
OS. It is in-charge of exiting out of the enclave, executing the
system call in the untrusted component and relaying the return
TCS TCS TCS TCS TCS TCS TCS TCS
values back to the enclave. 1 2 3 4 1 2 3 4

Inside the shim, PANOPLY performs custom checks on Micron A Micron A’

Panoply Shim Lib Panoply Shim Lib
the system call return values inside the enclave to defend
against well-known class of Iago attacks [26]. Nearly two third
(205/309) of the system calls return 0 or error, and another one Shared Variables
third (104/309) return an integer. In addition to return values, Thread Control
system calls can also return data via parameters passed by Manager
reference. Most of the parameters are data structures which
contain control fields. After checking the specifications of Fig. 6. Design for on demand multi-threading in PANOPLY.
all the system calls, we identify 50 system calls (16%) that
write information into 20 distinct structures. Most of them
(18 structures) only have integer field types, similar to return not performed by PANOPLY since the buffer size is not known
values. Other structures contain structure pointers or function statically. The developer can provide an upper bound based
pointers. Specifically, we add sanitization code to ensure that on the maximum file size. However, since there is no concrete
return values are consistent with the POSIX semantics. check on the size of buf, the original code itself is susceptible
For return values that have static data types such as integer to buffer overflow attacks.
fields with 0 or error calls, PANOPLY compares the return value Multi-threading. Enclaves have a limited support for thread-
with 0 and a set of valid error numbers per system call. If it ing — there is a gap between program abstractions in UNIX
detects return values such as invalid error numbers, then the and that provided by SGX. In SGX SDK, the application
check reports failure. For return values that lie within a range, has to statically determine how many and where the threads
PANOPLY checks that the return value conforms to a valid are initialized. Intel SGX ensures that all the thread-local
range depending on the return type. For e.g., the read system memory (such as thread stack) is isolated from each other.
call in Linux has an integer return type as shown below. It also provides a Thread Control Structure (TCS), a data
size_t read(int fd, void *buf, size_t len); structure to hold thread-specific information such as program
counter, stack pointer, register values, and execution context
The shim code knows the input length (len) requested state of the thread. Since the enclave has to know the total
by the application to read. Thus, it checks that the system call size to be allocated for saving TCS structures, the enclave
return value and pass-by-reference parameter (buf) is less than has a statically pre-specified number (say, k) of TCS data
or equal to the requested length. This check limits the return structures. Thus, it can only support maximum k concurrent
value within a much smaller range than the original range of threads during the entire execution of the enclave. Thus, SGX
variable type. Similar to return values, PANOPLY sanitization does not allow to dynamically execute arbitrary threads on-
logic checks all the structures with integer fields returned by demand. Although the application can create threads outside
the system call. The policy is similar to the check on return the enclave, the number of concurrent threads executing in the
values — based on the field type, the code ensures that the enclave at a given time is limited to the pre-determined value.
value is within a valid range. Many applications do not fit in this regime of thread usage.
For instance, number of concurrent threads in a web server is
Few of the system calls return structure pointers or function a function of number of requests that the server has to serve.
pointers which are handled specially in our system. For e.g.,
connect system call returns a pointer to a structure of On-demand Threading: PANOPLY gives the abstraction of
sockaddr type as shown below. For such dynamic data types, arbitrary number of threads by supporting POSIX threading
PANOPLY library needs developer assistanace for performing (pthread) API. Applications can dynamically create threads
deep checks for such structures. For example, PANOPLY can by calling the standard pthread API. To support this abstrac-
check the fields of sa_family_t structure which itself is a tion, PANOPLY realizes the notion of virtual threads for the
field of sockaddr that is returned by the connect system applications. Figure 6 shows an overview of PANOPLY design
call with developer annotations for the correct bounds. to support on-demand multi-threading. Under the hood, it
int connect(int sockfd,
multiplexes these virtual threads on the underlying enclave
const struct sockaddr *serv_addr, threads and uses the TCS structure and SGX threading APIs.
socklen_t addrlen); Specifically, PANOPLY threading API creates TCS structures
struct sockaddr { for a pre-determined pool of threads per micron (say k).
sa_family_t sa_family; When a specific thread wants to enter an enclave, PANOPLY
char sa_data[14]; } first checks if the host-enclave can accommodate the thread
concurrently. If so, PANOPLY sets the correct micron id, thread
Note that PANOPLY’s checks do not protect against any arguments, and redirects the thread execution to enter the
vulnerabilities such as buffer overflow that are present within micron. The challenge is when a micron has already reached
the original application. For example, in the read call if the its maximum concurrent thread limit. At this point, there are
len is not provided, then a check for the buf variable is two design choices: (a) evict an executing thread and schedule

7
the execution of the new thread (b) spawn a new micron to does not use all the parent data.
cater to the k + 1 thread. Although eviction is a clean solution,
it incurs additional operations of save-restore and scheduling Strategy 2. A second alternative is a copy-on-demand
inside the enclave. PANOPLY does not adopt this design and strategy, similar to copy-on-write optimization for fork calls
instead spawns a host enclave on-demand. Specifically, if the in model Linux implementations. To achieve this for microns,
micron reaches its maximum capacity of concurrent threads, we can implement a page-fault based on-demand data passing
PANOPLY launches a new micron thus increasing the size of from parent to child micron. Specifically, the parent replicates
threads from k to 2k. This design choice comes with a caveat and communicates its data to the child if and only if the
that all the shared memory between threads has to be accessible child accesses it. To achieve this, PANOPLY can mark all
across the two microns. To this end, PANOPLY introduces a the parent micron’s pages as read-only (copy-on-write) and
thread control manager for such global thread memory that register custom page-fault handlers 2 inside both the microns.
acts as a reference monitor. For supporting (k + 1)th thread, Whenever the child faults on a data page, the custom handler
PANOPLY launches a new micron and all the shared memory can request the parent to communicate the data. We point
operations are performed via the thread control manager, as out that the support for registering page fault handling within
done in a write-through cache. The PANOPLY shim library enclaves is not currently enabled in the SGX hardware. To
is responsible for redirecting the micron code to this thread overcome this limitation, we can modify the OS to notify the
control manager if the total number of concurrent threads micron on faults. The OS is not trusted to reliably uplift the
running is greater than k. When the threads are less than k, fault to the user level. However, even if the OS suppresses
they access the local copy via shim within the host enclave. faults, it does not weaken the security as the parent’s pages
will still be sealed.
Synchronization: SGX SDK only supports basic thread syn-
chronization primitives — spin locks and mutexes. However, Strategy 3. A final strategy is to statically identify what data
higher level synchronization primitives such as semaphores, is accessed by the forked child micron. When the application
conditional variables, barriers, and read / write locks are not yet performs a fork, the parent micron can replicate / communicate
supported in SGX. Thus, the programmer has to realize these these statically identified data values to the child micron. To
constructs by using mutexes available with the limited SGX reason about the incompleteness of the static analysis, the
threading support. Instead, PANOPLY exposes the full suite system can raise a run-time error to flag any data values which
of pthread synchronization primitives inside the enclave. were not replicated. This way, the developer can add custom
It implements these operations using the SGX’s mutex syn- code to replicate these values. In our experiment applications,
chronization support. PANOPLY supports the POSIX threading we observe that most of the data that is required by the
API pthread for semaphores, conditional variables, barriers, child micron is small and does not need full data space
and read / write locks operations based on SDK-supported replication. Thus, PANOPLY can statically identify the variables
mutexes. For all synchronization operations which are local to and communicate their values at fork and replicate them in
a micron’s code, PANOPLY keeps the synchronization objects the child micron. Developer can additionally annotate data
inside the micron. For global synchronizations across microns, variables that are shared between parent and children microns.
PANOPLY creates a notion of inter-micron locks. The thread In many cases, strategy 3 is cleaner.
control manager micron holds and manages all such shared PANOPLY provides a generic implementation using strategy
objects and all global changes are synchronized via this 1, such that any application which uses fork can continue to
micron. Specifically, to either release or get hold of a lock, execute without excessive developer efforts. In Section IV-D,
a micron has to invoke the thread control manager. The value we discuss the details for implementing strategy 1 in existing
of the global objects can only be changed by a well-defined SGX SDK. In the future when the SGX2 hardware is available,
set of microns, thus enforcing the right semantics of mutexes strategy 2 can also be integrated in PANOPLY, subject to
for the objects. specific design details of the SGX hardware. Currently for
Multi-processing. PANOPLY supports applications which use cases where performance is critical, developers can chose to
the UNIX fork and exec APIs. For a fork system call, resort to strategy 3.
PANOPLY library instructs the OS to launch a new untrusted In the case of exec, PANOPLY requests the OS to create
child process. This child process then creates a new micron a new untrusted process which launches a new micron for
with the same code as the parent micron. The PANOPLY the exec-ed code. This micron code is attested by using the
shim library performs micron initialization step and assigns SGX hardware primitive before the execution begins. The shim
a micron-identity to this micron instance. Next, PANOPLY library assigns a micron identity to this micron for further
establishes a communication channel between the parent and interactions with other microns. For exec, static identification
all the children microns for maintaining the inter-micron flow and copy-on-demand strategy saves a lot of redundant copy
integrity. As per the POSIX fork semantics, the child must operations, since the child micron doesn’t need access to parent
replicate the data memory state of the parent micron. There data. For cases where strategy 3 is not sufficient, PANOPLY
are three possible strategies to ensure that the child micron resorts to using strategy 2.
has access to the parent’s data.
Shared Memory. To support shared memory between two or
Strategy 1. A straightforward way is to do a full replica of more microns, PANOPLY establishes a shared secret between
parent micron’s data and communicate it to the child micron
over a secure communication channel at the fork call. This 2 With SGX2 extensions, the hardware can directly notify the enclave when
approach involves excessive micron-micron data copy and the enclave code incurs a page fault along with the faulting virtual address,
slows down the application needlessly, even when the child page permissions, fault type and the register context [16], [39].

8
these microns using secure inter-micron protocol. The microns correct target function to be executed in another micron, along
then use a public page as their shared memory resource. with the its current state. Hence, the adversary cannot tamper
Every microns seals the content and writes to the public page. the caller (checked at the entry) or forge the callee (created
Rest of the microns unseal the page content within their own at the exit). PANOPLY replaces all calls to non-micron code
address space. The sealing is standard authenticated encryption (such as system library APIs) with OCALLs. Further, each
with version control for preventing replay attacks [46]. This OCALL interface is instrumented with a set of checks on input
mechanism creates a notion of shared memory, such that each and output call parameters as discussed in Section IV. These
micron internally maintains a private copy of the memory and checks are best-effort, and the developer can add call-specific
all changes are synchronized and broadcasted by writing sealed sanitization logic at any of these interfaces.
data to the public page.
For supporting multi-threading, PANOPLY needs to know
Event Management. PANOPLY applies the delegate strategy the set of memory that is concurrently accessed by multiple
to support event based-programming inside the enclaves. SGX threads. The developer can annotate all the such variables
hardware natively supports synchronous and asynchronous which are subject to operations from multiple threads. In
exits from the enclave. For synchronous and asynchronous our current implementation, we manually identified concurrent
exits, the hardware saves the execution state before exiting memory accesses for our case studies. In the future, PANOPLY
the enclave, and restores it back when the enclave resumes. can perform precise pointer analysis to aid the developer
For example, if the enclave receives an interrupt, the hardware in identifying the complete set of corresponding operations
performs an Asynchronous Enclave Exit (AEX), saves the cur- on shared memory addresses. Once the shared variables and
rent processor state into enclave memory, enters the Interrupt operations are identified, PANOPLY redirects the access to all
Service Routine (ISR) and then finally restores the processor the shared variables, via the shim. The shim code is responsible
state for resuming the enclave execution. In this case, the OS for deciding if the operations on shared variables are to be
is responsible for executing the ISR and scheduling back the performed locally to the micron or are to be carried out as
enclave execution. The hardware ensures that the OS cannot global operations in tandem with threads in other microns. We
tamper any context registers during the exits. Thus, it is safe follow similar strategy to mark and mediate access to shared
to delegate the event listening and notification tasks to the OS. memory and fork-exec process memory semantics.
PANOPLY uses the OS primitives of signals and interrupts
to register event listeners, callbacks and add dispatch handlers. D. Implementation
Internally, PANOPLY hooks the underlying OS APIs and inter- We implement PANOPLY on top of the Intel SDK [8] 1.5
faces via OCALLs. Thus, PANOPLY enables event wait/notify Open Source Beta shipped for Linux Kernel v3.13. PANOPLY
mechanism, polling, event buffering, signal handling for en- comprises of a set of API libraries and build extensions. For
claves which are necessary for event management. Our API our case-studies, we annotate the application code to mark
is expressive to support event libraries such as libevent, which functions should execute in which micron. Then we
libev, in addition to hand-coded event loops. modify the application Makefile to use PANOPLY extensions
and library. We add the interface file (.edl) for specifying all
C. The PANOPLY Infrastructure the enclave entry-exit points. The Intel SDK edger8r tool
uses these files to generate boiler-plate code stubs for enclave.
To use PANOPLY, akin to several partitioning tools [22],
PANOPLY then adds specific checks to each of these stubs.
[24], [35], the developer marks all functions to denote which
micron executes these functions. In case of our example in We implement the support for multi-processing (specifi-
Section II, the functions called on Line 5, 13, and 25 are cally, fork and exec) by patching the SGX SDK [6] and
marked to be executed in a priviledged micron by annotations. the corresponding SGX linux driver [7]. By default, when a
Rest of the functions in the program are explicitly marked to new process is created by fork, the SGX kernel driver data
be executed in a non-priviledged mircon. structures which map the enclave virtual addresses to the EPC
physical addresses for the parent process are copied over to
After establishing the set of all such micron-bound func-
the child process. Hence, when the child process spawns its
tions, PANOPLY analysis identifies the inter-micron interaction
enclave, the driver reads the data structures and assumes that
boundaries. Specifically, it constructs the program dependence
the VA space and the EPC PA addresses are already taken
graph consisting of control flow and data flow. The control
up. However, in reality these are stale mappings from the
flow graph comprises of micron function nodes connected by
parent enclave. If left unmodified, the SDK driver spawns the
call edges. Data flow graph comprises of all the parameters
child enclave in a new VA space, which does not overlap
passed between microns and any shared memory such as global
with the existing mapping. This leads to a mismatch in the
variables. For cases where PANOPLY cannot precisely identify
child and parent’s VA layout. To work around this, our driver
the control and data-flow graph, it prompts the user to specify
code ensures that the start virtual addresses of all the children
the intended flows by adding annotations.
enclaves is same as that of the parent enclave. Specifically,
PANOPLY instruments all the boundaries of the micron i.e., when spawning a child enclave, PANOPLY ignores the VA-PA
the entry and exit points of the micron wherein the control mapping inherited from the parents enclave. Further, we ensure
starts or ends the micron-bound execution. Each ECALL to be that the child’s enclave starts at the same virtual address as
executed inside the micron is mapped to an entry point and a the parent’s enclave. Once the address layouts are identical,
return point. For all such entry points in the micron, PANOPLY PANOPLY shim code in the child enclave reads the sealed
adds code to check the caller’s identity as well as the caller’s contents (BSS, data, heap and stack sections) saved by parent
state. At all the exit point from the micron, PANOPLY adds the enclave and updates its own corresponding sections.

9
 V. E VALUATION ensure that even if 8 out of 9 DA servers are compromised, a
blacklisted node is not accepted in the network’s “good” view.
In this section, we show the effectiveness of PANOPLY
The attacker can only shut down the network. This high level
by porting popular real-world applications to microns and
property is hard to achieve without the use of byzantine fault
testing them against a suite of application-specific benchmarks.
tolerance protocol [25], [27], which are bandwidth hogging and
Specifically, we aim to evaluate PANOPLY for the following:
thus are not used in this application. Previous work by Jain et
• Expressiveness. Is PANOPLY successful in supporting al. has only looked at a low-level property of protecting the
expressive programming constructs inside microns? DA server’s private keys using SGX [32]. They port only the
• Stronger Security. Does PANOPLY enable stronger key related operations to enclaves, and execute the rest of the
security guarantees for applications? code outside the enclave. While this ensures that the attacker
• TCB. How much TCB reduction does PANOPLY does not get direct access to the key, it does not guarantee that
achieve over Library OSes? Tor will always maintain the true state of the network.
• Performance. How does PANOPLY perform compared
to Library OSes? The Tor DA servers act as nodes and the messages they
exchange act as control and data flow edges of the blacklisting
All our experiments are measured on Inspiron-13-7359 consensus protocol. Even though each DA server’s secrets
machine with 6th Generation Intel(R) Core(TM) i7-6500U are secured by enclaves, it does not ensure the integrity
processor and 8GB RAM. We configure the BIOS to use of interactions between the DA servers. Specifically, in our
128 MB memory for SGX EPC. We use Linux 1.5 Open extended technical report [45], we demonstrate 4 concrete
Source Beta version of Intel Software Guard Extensions SDK, attacks, similar to our running example in Section II. The
Intel SGX Platform Software (PSW), and driver on Ubuntu attacks use the following strategies to disrupt the consensus
Desktop-14.04-LTS 64-bits with Linux kernel version 3.13. protocol between DA servers:
All our applications are compiled with GCC v4.8 and built
for SGX hardware pre-release mode (HW_PRERELEASE) with • Call tampering to change the network status votes
default optimization flags and debug symbols. To measure • Force silent failures, leading to vote withholding
various statistics at run-time, we use Intel VTune Amplifier • Replace messages to allow compromised certificates
with SGX Hotspots analysis which is configured with the • Replay compromised public keys
standard parameters for SGX. All performance measurements
On the other hand, if we view the protocol amongst
are reported over an average of 5 runs.
these distributed nodes as a single process executing in a
A. Case Studies single contiguous isolated memory, then the correctness of the
consensus follows directly. In this case, all the messages (such
We select 4 application case-studies which highlight the as votes, and status objects) are generated and passed without
advantages of multi-micron architecture in applications. We any tampering from the byzantine adversary. There are only
successfully demonstrate that PANOPLY can enable end-to-end two things that the adversary can do — abort the process or
guarantees rooted on SGX primitives as building blocks and proceed. This property holds true because the adversary cannot
support expressive programming constructs. The case studies tamper the execution flow inside the isolated memory.
include:
If we move the entire logic of each DA server to a separate
• Tor v0.2.5.11, where multiple directory servers form
micron (i.e., 9 microns for 9 DA servers), then all the messages
a distributed network
generated inside the micron are ensured to be untampered,
• H2O v2.0.0 webserver, with privilege separated sup-
they are executed in an isolated memory. Further, by the
port for Neverbleed [5]
virtue of inter-micron flow integrity, PANOPLY ensures that
• OpenSSL v1.0.1m library-as-a-service which can be
the control and data flow of inter-micron interactions (inter-DA
imported by any application
interaction in this case) is preserved. Thus, PANOPLY allows us
• FreeTDS v0.95.81 database client application
to achieve the same security guarantee as executing the entire
1) Stronger Security Property — Tor: Tor is an anony- Tor blacklisting consensus protocol in a single contiguous
mous communication system that routes the client request isolated memory. As discussed above, this ensures that the
through a circuit of (three or more) nodes. Tor is an open byzantine adversary can only abort the consensus, it cannot
distributed network which uses a directory protocol to maintain bias it. We implement this architecture for Tor with the help
a global view of “good” (or non-blacklisted) nodes in its of PANOPLY, as discussed in Section V-B.
network. All the routers in the network periodically send their
2) Supporting Privilege-separation — H2O: In 2014, the
status to a directory authority server (DA). Each DA collects
Heartbleed bug in OpenSSL’s implementation of TLS protocol
these status reports to generate its local view of the network.
lead to leaking server’s private key due to a buffer over-read.
DA servers then run a form of consensus protocol to agree
Webserver implementations such as H2O HTTP2 server have
upon the set of global network nodes, wherein each DA sends
taken precautionary measures by separating the RSA private
a signed network status vote object to peer DA servers. Various
key operations in a privilege separated process since version
real-world attacks have targeted DAs by compromising either
1.5.0-beta4 [5]. This minimizes the risk of private key leakage
their private keys or forging status votes to create a dishonest
in case of vulnerability such as Heartbleed. In PANOPLY
view of the network, thereby causing malicious or blacklisted
design, the RSA key operations are moved to a separate micron
Tor nodes to be accepted in the network [10].
(RSA micron) whereas the rest of the webserver code executes
Goal: Our goal is to secure the Tor DA protocol against in another micron (H2O micron). The RSA micron comprises
a malicious OS on the DA servers. Specifically, we want to of functions to load the private key and to use it for encrypting

10
/ decrypting a given buffer and to sign the contents of a local attestation. We chose the same configuration as evaluated
buffer. The H2O micron invokes the RSA micron functions by previous works [32], [36]. We observe a maximum of 2
to perform functions such as rsa_sign, rsa_priv_enc threads executing in parallel for our configuration.
and rsa_priv_dec. PANOPLY ensures that the RSA keys
are safe in one micron different from the rest of the application. H2O. We configure H2O web server with Neverbleed plug-
The application micron invokes the key-storage micron via in, YAML parser and the in-built event-loop implemented by
PANOPLY’s interface. H2O. We further statically link H2O with OpenSSL library
which executes inside the webserver micron and includes
3) Supporting Enclaved-libraries — FreeTDS & the networking and multi-threading module of PANOPLY. We
OpenSSL: FreeTDS is an open source implementation of the make 154 LOC changes to the code to achieve this. The
TDS (Tabular Data Stream) protocol which allows programs to application exposes 3 functions for inter-micron invocation.
natively talk to Microsoft SQL Server and Sybase databases. Further, we evaluate H2O performance with h2load which
It links to OpenSSL library to support SSL/TLS for its traffic. is a benchmarking tool for HTTP/2 [4]. We observe a total
The running example in Section II demonstrates how PANOPLY of 2 inter-micron messages and 128 PANOPLY API calls at
enables to split the FreeTDS logic inside two microns. run-time for H2O.
OpenSSL is a widely used open-source library for SS- FreeTDS. Our FreeTDS application uses OpenSSL library,
L/TLS protocols. It provides client and server-side imple- along with threading and network support from PANOPLY.
mentations for SSLv3, TLS along with the X.509 support We make 473 LOC changes to the application to compile
needed by SSL/TLS (libssl). Further, the core library also it with PANOPLY. We configure FreeTDS client application
implements basic cryptographic functions (libcrypto). Sev- with a remote Microsoft SQL database server. Our benchmark
eral large-scale real life applications require SSL/TLS support makes 48 queries (1 create, 46 insert and 1 select) to the
for networking with clients, servers or peers. Hence, we use remote database server and collects the response time. For this
PANOPLY to execute OpenSSL inside a single micron. This workload, FreeTDS makes 3 inter-micron calls.
allows large scale applications to use OpenSSL library by
executing it inside the micron. It is up to the application’s OpenSSL. We port the OpenSSL library including the crypto-
security architect to decide if the application code executes graphic utilities (libcrypto) and the TLS/SSL implementa-
in the same micron as OpenSSL or as in a separate micron. tion (libssl). We re-use the partially ported OpenSSL code
For our case study we take a sample certificate verification available with the Intel SDK which only ports libcrypto.
application shipped with Intel SGX SDK [8]. The application Specifically, we enable the SSL protocol and the support for
comprises of two parts: (a) IO operations to read a list of executing OpenSSL engines inside the micron. We use the
certificates from an input file and display the results (b) X.509 PANOPLY networking and file system APIs to achieve this
certificate verification by using the root-certificate. We separate and change 307 LOC. We test the OpenSSL library with
the application into two microns — one for IO and one for its regression suite, and use the speed and tspeed utility
OpenSSL library. The OpenSSL micron uses programming to benchmark the performance of our OpenSSL library. We
constructs such as threading and network APIs. The IO micron observe a total of 8 PANOPLY API calls at run-time along
uses file system and standard IO APIs. Further, the attacker can with 1 inter-micron call.
attack the micron-micron interactions by replaying or dropping
the messages as outlined in our running example in Section II. Comparison to Graphene-SGX. We attempted to port our
These attacks trick the application to accept invalid certificates, 4 case-studies to Graphene-SGX, by following the public
thus violating the higher level guarantee of accepting only valid instructions [3] available at the time we performed our ex-
certificates. PANOPLY enforces inter-micron flow integrity and periments. We report that we were not able to port 3 out
ensures that all such attacks are thwarted. of 4 applications to Graphene-SGX— the applications either
crash or suspend during execution. Since there is no public
companion report, but only a large-scale system, it is hard to
B. Porting Effort evaluate the design reasons of why the applications crash. After
further investigation of the Graphene-SGX source-code, we
We make an average change of 905 lines of code per
were able to narrow down 2 cases which are problematic: (1)
application to port it to PANOPLY [54]. We test our applications
the fork semantics break in H2O which launches neverbleed
with their regression test suite to ensure that our porting does
in a separate process (2) the epoll call does not receive the
not break the application.
socket events, thus the applications such as Tor and FreeTDS
Tor. Tor code uses 4 libraries — OpenSSL’s libcrypto never progress. Thus, our evaluation comparisons are best-
and libssl for SSL/TLS protocol, libevent for event effort. We have informed the authors to implement possible
handling, and libz for compression. We port all 4 libraries mitigations in their system for these issues.
along with Tor code to microns. In doing so, we make use of
PANOPLY’s threading, multi-process and networking APIs. We C. Reduction in TCB
create two microns as per the design discussed in Section V-A. PANOPLY achieves 2 orders of magnitude lower TCB
We make a total of 2685 LOC changes to the complete code- as compared to state-of-art approaches such as Graphene-
base with the help of PANOPLY. To test for correctness, we SGX [52] and Haven [21]. The size of compiled microns in
ensure that micron-code passes all the 32 tests in the regression MB is an orders of magnitude smaller than Graphene-SGX.
suite shipped with Tor code. We evaluate Tor with a private
network comprising of 3 DA servers and 3 routers using PANOPLY TCB. PANOPLY adds an average of 19.62 KLOC
Chutney [1]. All the nodes execute on a single machine and use per application which is 5.8% of the original application code.

11
 TABLE III. TCB E VALUATION . T HE TABLE SUMMARISES THE TCB COMPONENTS OF PANOPLY AND G RAPHENE -SGX IN TERMS OF LOC AND SIZE OF COMPILED
ENCLAVE BINARIES . C OLUMNS 2-5 REPORT THE BREAK - DOWN FOR THE LOC OF EACH COMPONENT INCLUDED IN THE MICRON , AND C OLUMN 6 HIGHLIGHTS THE PERCENTAGE
INCREASE IN THE TCB DUE TO PANOPLY . C OLUMNS 7-13 REPORT THE LIBRARIES INCLUDED (3) IN THE COMPILED ENCLAVE BINARY, AND A 7 DENOTES THAT THE MICRONS
DO NOT REQUIRE THE LIBRARY. C OLUMN 14 DENOTES THE TOTAL SIZE OF COMPILED MICRONS . C OLUMNS 15-18 DENOTE THE COMPILED SIZE OF VARIOUS COMPONENTS
INCLUDED BY G RAPHENE -SGX TCB. C OLUMN 19 REPORTS THE TOTAL SIZE OF COMPILED ENCLAVES FOR G RAPHENE -SGX.

PANOPLY TCB in LOC Split-up for PANOPLY TCB (in MB) Graphene (in MB)
Case Study
Enclave Boilerplate PANOPLY libssl libcrypto libz libyrmcds libyaml libevent SDK Enclave Trusted Enclave
Total % Inc PAL Enclave SDK
Logic Code Logic (0.765) (3.7) (0.137) (0.06) (0.185) (0.579) (0.05) Size Libs Size
OpenSSL 256987 9004 10425 276416 7.56 3 3 7 7 7 7 3 5.88 1.84 0.067 64.58 0.05 64.69
H2O 414918 9189 10425 434532 4.72 3 3 7 3 3 7 3 7.98 1.84 16 64.70 0.05 80.75
FreeTDS 297108 9788 10425 317321 6.80 3 3 7 7 7 7 3 6.08 1.84 1.2 64.58 0.05 65.83
Tor 436385 8817 10425 455627 4.41 3 3 3 7 7 3 3 13.18 1.84 7.0 63.94 0.05 70.99

TABLE IV. P ERFORMANCE E VALUATION . C OLUMNS 2-5 REPORT THE STATISTICS ABOUT PANOPLY MICRONS . C OLUMNS 6-9 DENOTE THE CPU CYCLES CONSUMED BY
PANOPLY . C OLUMNS 10-12 DENOTE THE EXECUTION TIME FOR EACH CASE STUDY AND THE CONTRIBUTION OF PANOPLY TO THE TOTAL TIME .

Case No. of Inter- No. of No. of Split-up of CPU Cycles (in billions) CPU Time (in seconds)
Study Microns Micron OCALLs ECALLs Non-Micron Create Enclave Empty %
Total PANOPLY
APIs / SDK Delete Logic Enclave Increase
OpenSSL 2 1 23695 1 0.25 6.06 0.11 6.43 2.79 3.16 13
H2O 2 1 124287 5 0.35 17.08 6.11 23.54 6.56 8.79 34
FreeTDS 2 1 86198 1 0.47 22.31 0.07 22.64 8.60 8.74 1
Tor 6 30 286459 5 6.65 11.77 2.58 17.41 4.54 6.72 48
Average 24

TABLE V. B REAKDOWN OF LOC AND TCB S IZES OF G RAPHENE -SGX

COMPONENTS . GLIBC LOC AND SIZE COMPUTATION ACCOUNTS FOR THE WHOLE D. PANOPLY Performance
CODEBASE COMPRISING OF ALL LIBRARIES AND PLATFORM - SPECIFIC CODE .
PANOPLY adds a 24% CPU overhead to the application’s
Component LOC Size (in MB)
glibc 1156740 56.9 execution time (Table IV) on an average, over the baseline
libPAL-LinuxSGX 16901 0.9 cost of creating empty enclaves. It achieves comparable perfor-
libPAL-enclave 33103 0.2 mance as Graphene-SGX, with PANOPLY’s overheads higher
SGX SDK 119234 0.5
Total 1325978 58.5
by 5-10%. PANOPLY strictly prioritizes TCB over perfor-
mance, and does not include any optimizations such as buffer-
ing. However, future systems can improve over PANOPLY by
Columns 2-5 in Table III shows the LOC split-up for each incrementally adding optimization with careful considerations
case-study. Out of the total LOC included by PANOPLY in for TCB bloat.
each micron, 9.2 KLOC (46.3%) of code is automatically
generated boilerplate code to facilitate OCALLs and ECALLs Performance Breakdown. We measure the micron execution
mechanism. This code is generated by the Intel edger8r bottlenecks as well as the breakdown for the total number of
tool, which takes in a .edl interface file and creates stubs CPU cycles it consumes. (Table IV) Column 6-9 present the
for passing parameters across micron boundaries. We believe average number CPU cycles that each application consumes for
that this code is easy to automatically verify in the future. In various operations during its entire execution. Our preliminary
terms of compiled code size, each micron binary includes the performance measurements identified that bulk of the CPU
original application logic, corresponding application libraries, cycles are spent in the Intel SDK. On further investigation, we
the trusted libraries added by Intel SDK and PANOPLY shim identified three main factors which contribute to this.
library. Columns 7-14 in Table III show the split-up for the First, the operation of creating and destroying enclaves
micron binary size. Note that PANOPLY selectively adds a takes up 6-7 billion CPU cycles for various sizes of enclaves.
library to a micron, if-and-only-if the micron code needs it. We launch each of the application enclaves and destroy them
More importantly, PANOPLY does not include system libraries without executing any logic inside the enclave. Column 7 in
such as libc, libdl, libpthread in the micron. Table IV denotes the number of cycles to do this for each
Comparison to Graphene-SGX TCB. We refer to the public specific case-study. These operations are performed by the Intel
release of Graphene-SGX [3] to compute the total LOC SDK, and are not an artifact of PANOPLY’s design. Thus, 96%
of system. Table V shows the LOC of each component in of the CPU cycles are due to enclave launch and tear-down
Graphene-SGX, and the corresponding binary size. Graphene- (Column 7 vs Column 9 in Table IV). Taking this cost as a
SGX library comprises of a Platform Adaptation Layer, Linux baseline, PANOPLY has a 24% overhead.
library, and system libraries required to support a pico-process.
In terms of LOC, this amounts to an increase of 1.326 MLOC. Second, copying enclave data to-and-from non-enclave
When compared with PANOPLY, this amounts to 2 orders of memory contributes to significant fraction of CPU cycles
magnitude (127.19×) larger TCB. Note that the TCB and consumed by the application. Specifically, these operations
binary sizes are dependent on the application. We port our involve encryption-decryption of data entering/leaving the en-
four case studies to Graphene-SGX and measure the total clave memory, which consumes CPU cycles. For example, in
size of the binaries, since counting per-binary LoC is more the H2O application, the number and volume of such a copy
complicated. Columns 15-19 in Table III lists the breakdown of operation is directly proportional to the size of the served web-
size of each component. On an average, Graphene-SGX pico- page. To test this aspect, we measure the throughput of H2O
process binaries are an order of magnitude larger as compared webserver by serving two sizes of web-pages: 200 Bytes, 1
to PANOPLY microns. Haven reports a TCB of millions of lines KB, and 6 KB for a total of 100, 000 requests from 100 clients.
of code in the paper [21], however we do not have access to As shown in Figure 7, the webserver throughput decreases
the system to directly compare with PANOPLY. as we increase the size of the web-pages. The shaded-region

12
 100000 94933.2 12
Baseline H2O

Throughput in Requests/second
Graphene-SGX 10.13 10.18
90000 82480.69 9.64
Panoply H2O 10 Panoply
80000 8.84
66042.22
70000

Time (second)
8
60000
48490.84
50000 6
42293.46
40000
4
30000
20000 12818.84 2 0.9 1 0.9 1.02
10000 0.1 0.109 0.1 0.109
0
0 (512,10) (512,100) (512,1000) (1024,10) (1024,100) (1024,1000)
200 Bytes 1KB 6KB
(Bytes, Connec2ons)
Size of served sta8c web-page

Fig. 7. PANOPLY Performance for 100000 requests of various page sizes. Fig. 9. OpenSSL Throughput Performance of PANOPLY & Graphene-SGX.

120

35
Baseline Graphene-SGX Panoply 100 Graphene-SGX Panoply
Latency in miliseconds(ms)

Bandwidth
30 80

25
60

40
20

20
15
0

ni tcp pe
k s et
am

en ad
re m lose

co en d

l l e l l e b c al i g e
pa bc py ned

bc un ned

na ed
or mo ed

ar r>a rite

or rite

ro
10

M ad / e
n s
c b op rea

ar rea

e rit
em o mo a
oc ck

ze
un un lib py u clo
op re
pi

y u gn
em Me lign

M em Me al re
tre

y p pa y w

em w
g

>a l w
M 2c
x s so

yb
r> op ali
o p al i
2
lib ad ap

y p ry

r
>

lr
o
5

y
ad

or y
ro ro c

r
re
_u

d
al
ap
af

M
m

M
0

M
0 10 20 30 40 50 60 70

d
Load size in MB

Fig. 8. LMBench Latency Performance of PANOPLY & Graphene-SGX. Fig. 10. LMBench Bandwidth Performance of PANOPLY & Graphene-SGX.

of the bar-graph denotes the fraction of execution time spent a SSL client-server setup and measure the CPU execution
in the SDK routine for OCALLs. Thus, larger the number of time for 6 configurations of number of requests and size of
OCALLs, the more is the overhead. As we can also see from payloads: 512 / 2014 bytes of data for each of the 10 / 100
Table IV, applications which have less number of OCALLs, / 1000 client connections. Figure 9 depicts the performance
exhibit lower overheads. of Graphene-SGX and PANOPLY. For these configurations,
Third, all of our case studies use OpenSSL routines for PANOPLY has 5-10% higher overheads than Graphene-SGX.
various operations ranging from SSL connections to cryp- Since comparison of a single user-application may not be a
tographic operations. The OpenSSL library first checks if conclusive evidence for the overall performance of PANOPLY,
the underlying hardware supports AES-NI via cpuid. If it we present a fair comparison of these two systems. Specifi-
detects that the hardware supports, it uses the hardware AES cally, we port the LMBench benchmark to PANOPLY, which
instructions for its cryptographic operations, otherwise it falls is supported 3 by Graphene-SGX as well. Figures 8 and 10
back to a software implementation. In our experiments, we present the latency and bandwidth performance of PANOPLY
observe that the OpenSSL library executing inside the enclave and Graphene-SGX. Our first observation is that the perfor-
fails to detect that our hardware has AES-NI support. Thus, it mance overheads of both the systems are significant over the
uses a slower AES routine which adds latency by consuming absolute baseline of a native application executing without any
more CPU cycles. We suspect that the SDK has not rolled out enclave infrastructure. This re-iterates our earlier findings that
support for executing cpuid instruction inside the enclave. the slow-down due to enclave life-cycle operations (create and
This change was also proposed by previous work [21]. destroy) are common to any system that uses SGX. Secondly,
the memory latencies exhibited by PANOPLY and Graphene-
We point out that the Linux SDK itself does not use hard-
SGX are comparable. As shown in Figure 8, the PANOPLY
ware AES-NI instructions for encryption-decryption. Instead it
latency is almost-always lower than Graphene-SGX, whereas
uses software implementation of AES routine, as was pointed
both the systems exhibit similar overheads over an absolute
out by recent public disclosures [34]. In our case studies,
baseline. Finally, we measure the bandwidth for various types
this further amplifies the slow-down of each encrypt / decrypt
of operations including network, memory, file IO for the two
operation inside microns by 20% of the total execution.
systems. PANOPLY performs comparable to Graphene-SGX for
Comparison to Graphene-SGX. Our design does not sig- memory and network operations (Figure 10). For file-backed
nificantly degrade the performance as compared to Graphene- mmap operations, the overhead for PANOPLY is observably
SGX. We compare the performance of two systems for execut- 3 The Graphene-SGX system is not stable when executing the full LMBench
ing OpenSSL. Specifically, we test varied sizes and frequencies benchmark suite, and crashes non-deterministically with segmentation faults.
of data written out of the micron-enclave, since it is the main The results presented here are assimilated over 24 attempted runs to gather a
factor for PANOPLY overheads. To this end, we configure full set of evaluation.

13
larger than Graphene-SGX. Since PANOPLY performs these al. [20] propose four partitioning schemes ranging from coarse-
operations via libc interface, the number of enclave entry and grained partitioning (single enclave for whole application) to
exits per operation is larger. On the other hand, Graphene-SGX ultra-fine partitioning (one enclave per application secret) for
uses a narrower interface and hence for file IO, it incurs a lower executing OpenSSL library in enclaves.
number of enclave transitions. This is one of the factors which
causes the bandwidth variation for this subset of IO operations. SGX Containers & Sandboxes using Enclaves. Scone [19]
is a concurrent system which uses Intel SGX enclaves to
isolate docker containers running in a public cloud setting.
VI. R ELATED W ORK We summarise the key design differences between Scone and
PANOPLY is a new design point in SGX enclave design PANOPLY. Firstly, the interface exposed by PANOPLY is at
space that achieves low TCB while maintaining expressiveness POSIX level, whereas Scone exposes a system call interface
for enclave-bound code. PANOPLY’s inter-micron flow integrity to the enclaves. As an artifact of this design choice, PANOPLY
guarantees a higher level security property, unlike previous does not execute any libc library inside the enclave. On the
systems which target low-level confidentiality primitives [47], other hand, Scone executes the libc library (specifically musl
[48]. We discuss how PANOPLY differs from existing systems libc) inside the enclave. Secondly, PANOPLY does a syn-
in terms of TCB, design goals, scope and end-to-end guaran- chronous exit for executing code outside the enclave, whereas
tees. Scone does an asynchronous exit. These two variations lead to
a different design in terms of TCB, performance and system
TCB. PANOPLY design is driven by the delegate-rather-than- challenges. Thirdly, the on-demand threading model proposed
emulate philosophy, which is the key for lowering the TCB. by PANOPLY spawns new microns in separate enclaves to scale
Specifically, we do not perform namespace management inside the number of threads. This way, each thread in the application
the enclave, which is common approach for library OS designs. is associated with a unique thread in the enclave. Scone uses
The goal of library OSes is to achieve a narrow ABI [21], [41], a M:N threading model. Hence, when the application scales
[52], so as to maintain compatibility and portability. Hence, it threads, it is forced to multiplex on a limited number
these systems implement bulk of the system logic inside the of existing threads in a single enclave. Lastly, PANOPLY
library OS to map the system call APIs to their narrow ABI is designed for multi-process applications, which comprise
interface. In PANOPLY, we are not limited by these design of multiple micron containers and user processes. Hence its
choices. We expose a larger POSIX API to microns, and design comprises of in-built support for fork, exec, clone and
delegate all the system logic to underlying operating system. a secure communication interface between multiple microns
This is a reasonable choice because the OS can perpetrate the and processes. Scone only supports applications with a single-
same set of attacks even with a narrow interface. Thus, our container process running inside a single-enclave, which is a
design choice allows us to keep system libraries such as libc subset of PANOPLY.
outside of TCB, while achieving the same level of security.
Ryoan [31] is a concurrent work for executing distributed
Security of Single Enclave. PANOPLY is the first system to SGX native client (NaCl) sandboxes. PANOPLY’s execution
demonstrate control and data-flow attacks on enclave-enclave model of multi-micron applications varies from Ryoan, since in
interactions. It prevents such attacks by ensuring inter-micron PANOPLY, all microns that belong to the same application trust
flow integrity. Recent works have pointed out that enclaves each other. Ryoan introduces a request-oriented data model
are susceptible to side channel attacks via page faults [55] and where each enclave is in-charge of processing the input only
cache [28]. Currently, PANOPLY does not guarantee defenses once. Ryoan ensures that each service sandbox confines the
against such side-channels. However, applications can employ user-data to itself, while allowing mutually distrustful parties
off-the-shelf defenses proposed recently [29], [38], [43], [44]. to compute over sensitive data. In Ryoan, the NaCl executes
Weichbrodt et al. [53] recently showed that if the enclave the system calls and all the buffer and file IO operations are
logic has use-after-free or TOCTOU bugs, then the OS can backed by in-enclave memory.
exacerbate the effect of these bugs to perpetrate control-flow
attacks inside the enclave code. PANOPLY assumes that the VII. C ONCLUSION
enclave is free of any logic or memory bugs. Strackx et al. PANOPLY bridges the gap between expressiveness of
highlight that the adversary can shut down enclaves and abuse the SGX-native abstractions and requirements of feature-rich
the execution by doing a hardware state-replay attack [49], Linux applications. PANOPLY offers a new design point, prior-
[50]. PANOPLY can use their proposed solution to ensure itizing TCB over performance, without sacrificing compatibil-
hardware state contiguity in the future. ity. It achieves 2 orders of magnitude lower TCB than previous
Partitioning Applications for SGX. PANOPLY enables ex- systems.
pressive enclave-bound code with a low TCB. Thus, it can
ACKNOWLEDGMENTS
execute maximum application logic inside one or more mi-
crons while ensuring that the PANOPLY application maintains We thank Mona Vij and Simon Johnson from Intel for
the security guarantee. However, PANOPLY leaves the choice their feedback. Thanks to Chia-Che Tsai and Donald Porter
of partition design to the security architect [22], [24], [35]. for releasing code and discussions on Graphene-SGX. This
Jain et al. [32] propose the use of enclaves to protect Tor research was partially supported by a grant from the National
DA server keys inside enclaves to protect against well-known Research Foundation, Prime Ministers Office, Singapore under
attacks [14], [15]. Kim et al. [36] propose designs to use SGX its National Cybersecurity R&D Program (TSUNAMi project,
for networking applications such as SDN-based inter-domain No. NRF2014NCR-NCR001-21) and administered by the Na-
routing, Tor directory servers and ORs. Atamli-Reineh et tional Cybersecurity R&D Directorate.

14
 R EFERENCES [32] P. Jain, S. Desai, S. Kim, M.-W. Shih, J. Lee, C. Choi, Y. Shin,
T. Kim, B. Kang, and D. Han, “OpenSGX: An Open Platform for SGX
[1] “Chutney - The Chutney tool for testing and automating Tor network Research,” in NDSS 2016.
setup,” https://gitweb.torproject.org/chutney.git.
[33] S. Johnson, V. Scarlata, C. Rozas, E. Brickell, and F. Mckeen, “Intel
[2] “FreeTDS: Making the leap to SQL Server,” http://www.freetds.org/. Software Guard Extensions: EPID Provisioning and Attestation Ser-
[3] “Graphene-SGX Library OS - A Library OS for Linux Multi-process vices,” ser. Intel Corporation.
Applications, with Intel SGX support,” https://github.com/oscarlab/ [34] L. M. JP Aumasson, “SGX Secure Enclaves in Practice: Security and
graphene, (Accessed on 12/06/2016, Commit 9958214). Crypto Review — Kudelski Security,” Black Hat USA, 2016.
[4] “h2load - HTTP/2 Benchmarking Tool,” https://nghttp2.org/ [35] D. Kilpatrick, “Privman: A Library for Partitioning Applications,” in
documentation/h2load-howto.html. USENIX ATC 2003.
[5] “H2O Neverbleed: Privilege Separation Engine for OpenSSL / Li- [36] S. Kim, Y. Shin, J. Ha, T. Kim, and D. Han, “A First Step Towards
breSSL,” https://github.com/h2o/neverbleed. Leveraging Commodity Trusted Execution Environments for Network
[6] “Intel SGX for Linux,” https://github.com/01org/linux-sgx. Applications,” in HotNets 2015.
[7] “Intel SGX Linux Driver,” https://github.com/01org/linux-sgx-driver. [37] G. Klein, K. Elphinstone, G. Heiser, J. Andronick, D. Cock, P. Derrin,
[8] “Intel Software Guard Extensions SDK - Documentation — Intel D. Elkaduwe, K. Engelhardt, R. Kolanski, M. Norrish, T. Sewell,
Software,” https://software.intel.com/en-us/sgx-sdk/documentation. H. Tuch, and S. Winwood, “seL4: Formal Verification of an OS Kernel,”
in SOSP 2009.
[9] “[MS-TDS]: Tabular Data Stream Protocol,” https://
msdn.microsoft.com/en-us/library/dd304523.aspx. [38] S. Lee, M. Shih, P. Gera, T. Kim, H. Kim, and M. Peinado, “Inferring
fine-grained control flow inside SGX enclaves with branch shadowing,”
[10] “Tor Network Is Under Attack through Directory Authority Servers 2016. [Online]. Available: http://arxiv.org/abs/1611.06952
Seizures,” http://thehackernews.com/2014/12/tor-network-hacked.html.
[39] F. McKeen, I. Alexandrovich, I. Anati, D. Caspi, S. Johnson, R. Leslie-
[11] “Tor Project: Anonymity Online,” https://www.torproject.org/. Hurd, and C. Rozas, “Intel Software Guard Extensions Support for
[12] “CVE-2009-3555 TLS Session Renegotiation Vulnerability,” https:// Dynamic Memory Management Inside an Enclave,” in HASP 2016.
cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3555, 2009. [40] F. McKeen, I. Alexandrovich, A. Berenzon, C. V. Rozas, H. Shafi,
[13] “RFC 5746 - Transport Layer Security (TLS) Renegotiation Indication V. Shanbhogue, and U. R. Savagaonkar, “Innovative Instructions and
Extension,” https://tools.ietf.org/html/rfc5746, February 2010. Software Model for Isolated Execution,” in HASP 2013.
[14] “Tor Project Infrastructure Updates in Response to Security Breach,” [41] D. E. Porter, S. Boyd-Wickizer, J. Howell, R. Olinsky, and G. C. Hunt,
http://archives.seul.org/or/talk/Jan-2010/msg00161.html, 01 2010. “Rethinking the Library OS from the Top Down,” in ASPLOS 2011.
[15] “Possible Upcoming Attempts to Disable the Tor Network — The [42] F. Schuster, M. Costa, C. Fournet, C. Gkantsidis, M. Peinado,
Tor Blog,” https://blog.torproject.org/blog/possible-upcoming-attempts- G. Mainar-Ruiz, and M. Russinovich, “VC3: Trustworthy Data Analyt-
disable-tor-network, 12 2014. ics in the Cloud,” in IEEE Symposium on Security and Privacy 2015.
[16] “Software Guard Extensions Programming Reference Rev. 2.” [43] M.-W. Shih, S. Lee, T. Kim, and M. Peinado., “T-SGX: Eradicating
software.intel.com/sites/default/files/329298-002.pdf, Oct 2014. Controlled-Channel Attacks Against Enclave Programs,” in NDSS 2017.
[17] “Intel Software Guard Extensions Enclave Writer’s Guide,” [44] S. Shinde, Z. L. Chua, V. Narayanan, and P. Saxena, “Preventing page
https://software.intel.com/sites/default/files/managed/ae/48/Software- faults from telling your secrets,” in AsiaCCS 2016.
Guard-Extensions-Enclave-Writers-Guide.pdf, 2015. [45] Shweta Shinde and Dat Le Tien and Shruti Tople and Prateek Saxena,
[18] I. Anati, S. Gueron, S. P. Johnson, and V. R. Scarlata, “Innovative “Panoply: Low-TCB Linux Applications With SGX Enclaves,” National
Technology for CPU Based Attestation and Sealing,” in HASP 2013. University of Singapore, Tech. Rep., Dec 2016.
[19] S. Arnautov, B. Trach, F. Gregor, T. Knauth, A. Martin, C. Priebe, [46] Shweta Shinde and Shruti Tople and Deepak Kathayat and Prateek Sax-
J. Lind, D. Muthukumaran, D. O’Keeffe, M. L. Stillwell, D. Goltzsche, ena, “PodArch: Protecting Legacy Applications with a Purely Hardware
D. Eyers, R. Kapitza, P. Pietzuch, and C. Fetzer, “SCONE: Secure Linux TCB,” National University of Singapore, Tech. Rep., Feb 2015.
Containers with Intel SGX,” in OSDI 2016. [47] R. Sinha, M. Costa, A. Lal, N. P. Lopes, S. Rajamani, S. A. Seshia,
[20] A. Atamli-Reineh and A. Martin, ch. Securing Application with Soft- and K. Vaswani, “A Design and Verification Methodology for Secure
ware Partitioning: A Case Study Using SGX. Isolated Regions,” in PLDI 2016.
[21] A. Baumann, M. Peinado, and G. Hunt, “Shielding Applications from [48] R. Sinha, S. Rajamani, S. Seshia, and K. Vaswani, “Moat: Verifying
an Untrusted Cloud with Haven,” in OSDI 2014. Confidentiality of Enclave Programs,” in CCS 2015.
[22] A. Bittau, P. Marchenko, M. Handley, and B. Karp, “Wedge: Splitting [49] R. Strackx, B. Jacobs, and F. Piessens, “ICE: A Passive, High-speed,
Applications into Reduced-privilege Compartments,” in NSDI 2008. State-continuity Scheme,” in ACSAC 2014.
[23] E. Brickell, J. Camenisch, and L. Chen, “Direct Anonymous Attesta- [50] R. Strackx and F. Piessens, “Ariadne: A Minimal Approach to State
tion,” in CCS 2004. Continuity,” in USENIX Security 2016.
[24] D. Brumley and D. X. Song, “Privtrans: Automatically Partitioning [51] F. Tramer, F. Zhang, H. Lin, J.-P. Hubaux, A. Juels, and E. Shi,
Programs for Privilege Separation,” in USENIX Security 2004. “Sealed-Glass Proofs: Using Transparent Enclaves to Prove and Sell
Knowledge,” Cryptology ePrint Archive, Report 2016/635, 2016.
[25] M. Castro and B. Liskov, “Practical Byzantine Fault Tolerance,” in
OSDI 1999. [52] C.-C. Tsai, K. S. Arora, N. Bandi, B. Jain, W. Jannen, J. John, H. A.
Kalodner, V. Kulkarni, D. Oliveira, and D. E. Porter, “Cooperation and
[26] S. Checkoway and H. Shacham, “Iago Attacks: Why the System Call
Security Isolation of Library OSes for Multi-Process Applications,” in
API is a Bad Untrusted RPC Interface,” in ASPLOS 2013.
EuroSys 2014.
[27] A. Clement, E. Wong, L. Alvisi, M. Dahlin, and M. Marchetti, “Making
[53] N. Weichbrodt, A. Kurmus, P. Pietzuch, and R. Kapitza, “AsyncShock:
Byzantine Fault Tolerant Systems Tolerate Byzantine Faults,” in NSDI
Exploiting Synchronisation Bugs in Intel SGX Enclaves,” in ESORICS
2009.
2016.
[28] V. Costan and S. Devadas, “Intel SGX Explained,” Cryptology ePrint [54] D. Wheeler, “SLOCcount,” http://www.dwheeler.com/sloccount/.
Archive, Report 2016/086, 2016, http://eprint.iacr.org/2016/086.
[55] Y. Xu, W. Cui, and M. Peinado, “Controlled-Channel Attacks: De-
[29] V. Costan, I. Lebedev, and S. Devadas, “Sanctum: Minimal Hardware terministic Side Channels for Untrusted Operating Systems,” in IEEE
Extensions for Strong Software Isolation,” in USENIX Security 2016. Symposium on Security and Privacy 2015.
[30] T. T. A. Dinh, P. Saxena, E.-C. Chang, B. C. Ooi, and C. Zhang, “M2R: [56] F. Zhang, E. Cecchetti, K. Croman, A. Juels, and E. Shi, “Town Crier:
Enabling Stronger Privacy in MapReduce Computation,” in USENIX An Authenticated Data Feed for Smart Contracts,” Cryptology ePrint
Security ’15. Archive, Report 2016/168, 2016, http://eprint.iacr.org/2016/168.
[31] T. Hunt, Z. Zhu, Y. Xu, S. Peter, and E. Witchel, “Ryoan: A Distributed
Sandbox for Untrusted Computation on Secret Data,” in OSDI 2016.

15