Trusted Execution Environments

Chester Rebeiro
IIT Madras

Some of the slides borrowed from Intel; CDACH; ARM 1

 Previously in SSE…
• We looked at techniques to run an untrusted code safely

System

Run Program Here

If misbehaves
Kill it
Confinement

2
 Today in SSE…
• We now look at how to run sensitive code in an untrusted environment
– Besides other applications, the OS can also be untrusted.
– Attackers can probe hardware
• What to worry about:
– Code / Data of the sensitive app gets read / modified by the system
Untrusted System

Run Sensitive
Program Here

Trusted Execution
Environment

3
 Basic Problem
(Ring Architecture)

4
 Basic Problem
(Ring Architecture)

5
Invasive Attacks

6
 Trusted Execution Environments
Achieve confidentiality and integrity even when the OS is compromised!

• ARM : Trustzone (trusted execution environments)

• Intel : SGX (enclaves)

7
 ARM Trustzone

Trustzone Security Whitepaper, ARM

http://infocenter.arm.com
/help/topic/com.arm.doc.prd29-genc-009492c/PRD29GENC-009492C_trustzone_security_whitepaper.pdf
8
ARM System on Chips

9
 ARM Trustzone
(Main Idea)

Hardware and Software partitioned into two:

Normal and Secure worlds

A single hardware processor timesliced

between secure and normal worlds

Secure world provides an environment that

supports confidentiality and integrity.
- Can prevent software attacks
- Cannot prevent invasive attacks

10
A Typical Trustzone Application

11
 Switching Worlds
• Execution in time sliced manner (Secure <-> Normal)
• New mode (monitor mode) that is invoked during switching modes
• Mode switching
– triggered by secure monitoring call (SMC) instruction
– certain hardware exceptions (interrupts, aborts)

• Monitor Mode: saves state of the current world and restores the state of the
world being switched to. Restoration by return-from-exception.

• NS Bit: in configuration register indicates secure / normal operating mode.

NS = 1 -> indicates non-secure (normal) mode

12
NS Bit extends beyond the chip

13
NS Bit extends beyond the chip

14
 Memory Management
• Non Secure Table Identifier
current state of the processor CPU Core Memory (RAM)
(0 if secure world / 1 if normal Virtual address (VA) have an
world extra bit (33-rd bit) to

• If NSTID = 1 then force NS bit to 1

VA NSTID

page tables

Physical address
MMU
page tables

Physical address

15
 Memory Management
TLB stores NSTID and NS bit CPU Core Memory (RAM)
per entry

VA NSTID

page tables

VA NSTID PA NS
VA NSTID PA NS Physical address
MMU
VA NSTID PA NS
VA NSTID PA NS page tables

TLB
Page walk only Physical address
on TLB miss
16
 Memory Management
CPU Core Memory (RAM)
Secure world page tables
can map to normal world
memory

VA NSTID

page tables

VA NSTID PA NS
VA NSTID PA NS Physical address
MMU
VA NSTID PA NS
VA NSTID PA NS page tables

TLB
Page walk only Physical address
on TLB miss
17
 Memory Management
Cache Memory CPU Core Memory (RAM)
Tag NS cache line
Tag NS cache line
Tag NS cache line
Tag NS cache line Access RAM only
PA VA NSTID on cache miss

page tables

VA NSTID PA NS
VA NSTID PA NS MMU
VA NSTID PA NS page tables
VA NSTID PA NS

TLB
Page walk only
on TLB miss
18
 Memory Management Units
• Two virtual MMUs (one for each mode)
– Two page-tables active simultaneously
• A single TLB present
– A tag in each TLB entry determines the mode
(Normal and Secure TLB entries may co-exist; this allows for quicker switching of
modes)
– alternatively the monitor may flush the TLB whenever switching mode
• A single cache is present
– Tags (again) in each line used to store state
– Any non-locked down cache line can be evicted to make space for new data
– A secure line load can evict a non-secure line load (and vice-versa)

19
Secure and Normal Devices

20
 Interrupts
Normal world Secure world

User Code User Code

Privileged Code Privileged Code

IRQ Monitor IRQ

IRQ
All interrupts routed to monitor first.
Interrupts can be configured to go either to the normal world or secure world.
21
 Interrupts
Normal world Secure world Secure World
Interrupt Vector Table

User Code User Code

Normal World
Interrupt Vector Table Privileged Code Privileged Code

IRQ Monitor

Monitor Interrupt
IRQ Vector Table
All interrupts routed to monitor first.
Interrupts can be configured to go either to the normal world or secure world.
22
 Software Architecture
• The minimal secure world can just have implementations of synchronous code
libraries
• Typically has an entire operating system
– Qualcomm’s QSEE; Trustonics Kinibi; Samsung Knox; Genode
– The secure OS could be tightly couples to the rich OS so that a priority of a task in the
rich OS gets mapped accordingly in the secure OS
– Advantage of having a full OS is that we will have complete MMU support
• Intermediate Options

23
 Secure Boot
Why?
Attackers may replace the flash software with a malicious version, compromising
the entire system.

How?

Secure chain of trust.

Starting from a root device (root of trust) that cannot be easily tampered

24
Secure Boot Sequence

25
 Chain of Trust
check signature
check signature
Trustlet

Trustlet Trustlet
Rich OS
check signature

Secure OS check
signature

Boot loader check

signature
Inherently secure
Component check
(PUF/ TPM/ onchipROM) signature

Root of trust
26
 Intel’s SGX

Innovative Instructions and Software Model for Isolated Execution, HASP 2013 (F. McKeen et. al.)
27
Reduced Attack Surface with SGX
Normally With SGX enabled

App App App App App App

OS OS
VMM VMM
Hardware Hardware

Attack Surface

Malware that can subvert any one of Small attack surface (App + Hardware)
app, OS, VMM, or hardware Malware cannot steel secrets inspite
can steal secrets of subverting OS, BIOS, VMM, most
parts of the App, etc.
28
 Enclaves
(reverse sandbox)

• Enclave has its own code and data areas

Entry Table Provides confidentiality and integrity
Enclave
With controlled entry points
Heap
Enclave • However, enclave code and data cannot
Stack be accessed from outside the enclave not
Enclave even by the operating system.
Code
TCS • TCS: Thread control Structure
(SGX supports multi-threading;
one TCS for each thread supported)

29
 Enclave Properties
• Achieves confidentiality and integrity
– Tampering of code / data is detected and access to tampered code / data is
prevented.
• Code outside enclave cannot access code/data inside the enclave
• Even though OS is untrusted, it should still be able to manage page
translation and page tables of the enclave
• Enclave code and data
– Enclave code and data is in the clear when in the CPU package (eg. Registers /
caches), but unauthorized access is prevented
– Enclave code and data is automatically encrypted it leaves the CPU package

30
 Physical Memory
• PRM – processor related memory allocated by
the BIOS. Access to PRM is blocked by external
agents such as DMA, graphics engine, etc.)
– To the other devices, this range is treated as non-
existent memory
PRM EPC – All SGX enclaves mapped into the PRM
• EPC Pages: Enclave page cache holds enclaves
EPCM from any application.
– Divided into 4KB pages
– If an EPC page is valid, it either contains an SGX enclave
page or EPCM (EPC micro-architecture structure)

RAM
31
 Process 1
SGX Enclaves and PRM Process 2
Virtual address to
physical address
mapping. Done by
OS and MMU

RAM
Virtual Memory Virtual Memory 32
 Physical Memory
• EPCM: Enclave page cache map
– one for each EPC
– Used by hardware for access control
– It stores management related aspects for the
corresponding EPC
PRM EPC • Aspects such as valid / invalid; r/w/x permissions
• Type of page
• Virtual address range through which, the EPC can be
EPCM accessed
• It is an additional layer of security compared to legacy
paging and segmentation since we do not trust the OS

RAM
33
 Physical Memory
• SECS: SGX Enclave Control Store
SECS – One for each enclave
– 4KB (present in an EPC)
– Contains global metadata about the enclave
• EPC pages that are used
PRM EPC – Mapping information
– Crypto log of each used EPC page
• Range of protected addresses used by the enclave
• 32 / 64 bit operating mode
• Debug access

RAM
34
 EPC Encryption
• Hardware unit that encrypts and protects integrity
of each EPC

35
Memory
Access

36
 Application Execution Flow
App built with trusted and untrusted part

1. Untrusted part creates and executes the

enclave
1. Enclave is placed in the EPC. It is encrypted and
trusted
2. Trusted function is called and execution is
transferred into the enclave
3. Trusted function executes
4. Trusted function returns
5. Application continues execution

37
 Enclave Life Cycle
(creation)
ECREATE Instruction
Process
• Creates a SECS (SGX enclave control
structure)
– Contains global information about the enclave
• System software can choose where (in the
process virtual space) the enclave should
be present
• Also specifies
– Operating mode (32/64 bit)
– Processor features that is supported
– Debug allowed

38
 Enclave Life Cycle
(adding pages)
EADD Instruction
Process
• System software should select free ECS page
• EADD will initialize EPCM with
– Page type (TCS / REG)
– Linear address that will access the page
– RWX permissions
– Associate the page in SECS structure
• EADD will then record EPCM information in a
crypto log stored in the SECS
– This is the measurement of the enclave
– Used for gaining assurance
• Copy 4K bytes of data from unprotected
memory into the enclave

39
 Enclave Life Cycle
(measuring pages)
EEXTEND Process
• Measure a 256 byte region in an EPC page
– This region is specified by the developer
– The measurement comprising of a 64 bit address
and a 256 byte information in the SECS
– 16 invocations EEXTEND needed to measure the
whole page
• Correct construction of the enclave would
result in a matching with the enclave
owner
– The enclave owner’s signature is stored in a
SIGSTRUCT structure
– This can also be remotely verified

40
 Enclave Life Cycle
(initializing pages)
EINIT
Process
• Should be invoked after all pages have
been added
• Verify that the signature matches that of
the owner’s signature
• If EINIT is successful, it allows the enclave
to be entered

41
 Enclave Life Cycle
(enter/exit)
Process invokes the enclave through
pre-defined entry points using EENTER
instruction

EENTER
- Changes made to enclave mode
- Need to know the location to transfer
control and location where to save
state in case of an interrupt
- Defines an Asynch. Exit pointer,
which where IRET returns to after
servicing an interrupt
- It is outside the enclave
- And typically will have an instruction ERESUME

42
 Entry into the Enclave
• Set TCS to busy
• Change mode to enclave mode
• Save state of SP, BP, etc. for return in case of async. Exit
• Save AEP
• Transfer control from outside the enclave to inside

43
 Exit from Enclave
• EEXIT
- Clear enclave mode and flush TLB entries
- Mark TCS as free.
- Transfer control outside the enclave

44
 Asynchronous Exit (AEX)
• Occurs when an interrupt / exit occurs
• Processor state is securely saved inside the enclave and replaced with synthetic states
• AEP pushed onto the stack
(AEP is a location outside the enclave where execution goes to after IRET)
• After AEX completes, the logical processor is no longer in enclave mode

• Resuming after an interrupt

– EERESUME instruction is invoked, which restores all registers
– Typically EERESUME is present at the AEP location
• Resuming after a fault that occurred in the enclave?
– Eg. A divide by zero

45
 Instruction set Extensions for SGX
• Privileged Instructions
– Creation related: to create, add pages, extend, initialize, remove
enclave
– Paging related: evict page, load an evicted page
• User level instructions
– Enter enclave, leave enclave
– Interrupt related: asynchronous exit, resume

46
 Attestation
• system proves to somebody else that it has a particular SGX enclave
• Two attestation techniques
– Intra machine (prove to another enclave in the same machine)
– Inter machine (prove to a third party)
• Makes use of a register called MRENCLAVE
– Contains the SHA-256 hash of an internal log that measures the activity done
by the enclave
• The log contains the pages (code, data, stack, heap) in the enclave
• Relative position of the pages in the enclave
• Security flags associated with the pages

Innovative Technology for CPU Based Attestation and Sealing, HASP 2015, Ittai Anati et al 47
 Intra-Platform Enclave Attestation

1
1

Enclave Enclave
A B

• (1) Enclave A obtains enclave B’s MRENCLAVE

• Enclave A invokes EREPORT together with B’s MRENCLAVE to create a signed report destined for enclave B
– Enclave contains: attributes associated with the enclave
– Attributes of the Trusted Control Block
– MAC (produced by a key called report key, which is known only to the hardware and Enclave B)

48
 Intra-Platform Enclave Attestation

Enclave 2
1 Enclave
A B

• (1) Enclave A obtains enclave B’s MRENCLAVE

• Enclave A invokes EREPORT together with B’s MRENCLAVE to create a signed report destined for enclave B
• (2) Enclave A sends the report to B, via an untrusted channel
• Enclave B calls EGETKEY to retrieve the report key, re-computes the MAC accompanying the REPORT. If there is a match
with the MAC, then the enclave is indeed running on the same machine.
• Once the MACs have been verified, Enclave B can verify Enclave A’s report using the MRENCLAVE it just received

49
 Inter-Platform Enclave Attestation
External
Challenger

Enclave 2
1 Quoting
A Enclave

• Quoting enclave and external system uses asymmetric crypto. to transfer a

quote to the external system

50