Best Practices for

Embedded Software
Security Testing

1/13 page
What to Expect

Best Practices for Embedded Software Security Testing ............................................................. 3

Why Is it so Hard to Secure Embedded Applications? ............................................................... 3
Hardware Dependencies and Requirements............................................................................ 3
Challenges of Embedded Testing .............................................................................................. 4
1. Limited Hardware Access ................................................................................................... 4
2. Wide Range of Attack Vectors............................................................................................. 4
3. Time and Resource Constraints ........................................................................................... 5
4. Lack of Standardization in Embedded Systems ..................................................................... 5
5. Testing Under Realistic Conditions ...................................................................................... 5
6. Lack of Automated Testing Tools ........................................................................................ 5
7. Security vs. Safety .............................................................................................................. 6
Embedded Software Testing Tools and Technologies ................................................................. 6
Static Analysis Technologies and Tools .................................................................................... 6
Linting .................................................................................................................................. 7
Using Compiler Tools and Flags .............................................................................................. 7
Dynamic Analysis Technologies and Tools ................................................................................. 7
Unit Testing........................................................................................................................... 8
Fuzz Testing.......................................................................................................................... 8
Automating Test Case Creation With Fuzz Data ......................................................................... 9
Mocking on Steroids: Testing Positive and Negative Criteria With Fuzz Data ........................... 10
The Secret Sauce to Fuzzing Embedded Software: Instrumentation ........................................ 11
Leveraging the API Documentation to Generate Fuzz Tests .................................................... 11
3 Reasons Why You Should Fuzz Embedded Systems ............................................................... 12
1. Increased Code Coverage ................................................................................................. 12
2. Detecting Memory Corruption Accurately ......................................................................... 12
3. Early Bug Detection .......................................................................................................... 12
Enterprise Best Practice to Deal With Complexity in Embedded Software ................................ 13

2/13 page
Practices for Embedded Software
Security Testing
In embedded software, we need to make sure that growing complexity and
software dependency come at no cost to security.

Why Is it so Hard to Secure Embedded Applications?

Due to increasing connectivity and dependencies, modern embedded applications are
constantly growing more complex. This complexity comes with implications for
software security testing and requires plenty of manual effort, depending on the
toolchain. From an operational perspective, many embedded industries (automotive,
aviation, healthcare etc) are tightly staffed and work in long cycles with strict
deadlines. When things get tight, this can cause time-critical matters to be prioritized
over software testing.

From a technical perspective, embedded software security testing can be

distinguished from other ecosystems as well. In the early stages, you usually test
hardware, independently of the software, for compliance with physical requirements
and standards (e.g., electromagnetic compatibility). Later in the process, your final
hardware has to be tested for software development. For this purpose, most teams
I've worked with either rely on developer boards or simulate the hardware in software.
After completion, you integrate the software into the target platform and perform
integration tests.

Hardware Dependencies and Requirements

Additionally, embedded software tests examine if the final product, including

hardware and software, fulfills all functional and non-functional requirements. Usually,
so-called robustness tests are also performed manually by the tester. Depending on
the place of use, you’ll have to provide special evidence, such as high code coverage
or modified condition/decision coverage (MCDC).

Many languages used in embedded, such as C/C++ are dependent on the hardware
and operating system and can’t be cross-compiled. This leads to two significant
problems:

3/13 page
 1. Although the Public API documentation is available, you need to write
plenty of test harnesses, which is incredibly time-consuming.

2. The connection between the hardware and hardware-dependent API

requires testing to ensure that it can process unexpected inputs from the
PublicAPI securely and under all circumstances.

Moreover, security tests have to prevent the application from crashing under all
circumstances. They should cover all relevant states and behaviors, including edge
cases caused by unexpected or erroneous inputs.

Challenges of Embedded Testing

Due to the interdependence of hardware and software in embedded systems, it is
particularly important to test them for security and functional issues. However, it is
precisely this interdependence that can also pose a few challenges. Here are seven
common challenges in embedded software security testing:

1. Limited Hardware Access

Embedded software testing requires hardware, which can be difficult to access. As a

workaround, testing is often conducted using remote servers since testing teams
often do not have direct access to the hardware. An approach to simulate hardware in
software is by using emulators.

2. Wide Range of Attack Vectors

There are many potential attack vectors that can be exploited in an embedded
system, such as buffer overflows, memory corruption, and code injection. It can be
difficult to identify all the ways that an attacker could potentially exploit a system due
to the vast number of vulnerabilities. Coverage-guided testing can help uncover many
of these vulnerabilities.

4/13 page
3. Time and Resource Constraints
There are proven-effective approaches for embedded software testing at different
stages of the development process, such as unit tests, regression tests, HW/SW
integration and integration tests. However, time and resource constraints may force
teams to omit some of these testing efforts or run them on a reduced scale.
Simulating real-world usage requires testing a system under various conditions and
with different input types. Automated testing tools can help you ensure sufficient
testing despite time constraints.

4. Lack of Standardization in Embedded Systems

The unique nature and diverse range of embedded software systems make it difficult
to standardize testing approaches over different use cases. Each domain (e.g.,
automotive, avionics, medical, etc.) has unique requirements and constraints, making
it challenging to determine a blueprint testing approach for all embedded software.
Norms such as ISO/SAE 21434 (automotive) and DO-178C (avionics) are promoting
more standardization within their respective industries. Usually, compliance with such
standards requires a thorough audit of the development process.

5. Testing Under Realistic Conditions

Real-time testing is essential for embedded systems that have to respond promptly to
external events. Testing under realistic conditions ensures that a system will perform
as expected in the field, but simulating these conditions is a significant challenge.
Advanced mocking is a proven method for addressing this challenge and enabling
effective testing of your embedded systems' real-time capabilities.

6. Lack of Automated Testing Tools

Manual testing of embedded systems can be time-consuming and inaccurate.
Automated testing solutions can improve the speed and reliability of the testing
process. However, they can be challenging to implement for embedded systems.
Integrating security testing directly into your CI/CD environment is one way to reduce
manual efforts. It can also help you find security and functional issues that the human
eye tends to overlook.

5/13 page
7. Security vs. Safety
Embedded systems require a balance of security and safety. However, these two
objectives can be conflicting. Safety focuses on fail-safe behaviour and reliability,
while security focuses on protecting sensitive data from unauthorized access.
However, the complexity introduced by security features can impede the testing
process and impact safety. Proper tooling is necessary for embedded software teams
to identify and address both topics.

Embedded Software Testing Tools and Technologies

Broadly speaking, there are two kinds of embedded testing tools: static analysis and
dynamic analysis. Which combination of tools and technologies fits your
development process best will be largely specific to the projects you deal with.
Finding the right setup will help you alleviate a large chunk of the challenges listed
above. Here are some options:

Static Analysis Technologies and Tools

Static analysis is a method of evaluating source code without execution. This is
particularly useful for embedded C/C++ code, as the system under test does not need
to be compiled or deployed in production. These systems often have limited
resources and are not as user-friendly for developers. Advantages include:

• Being fast, as no compilation is required

• Compliance with code styles and guidelines
• Finding code smells and issues by parsing the code
• Providing code coverage data
• Testing and analyzing code in the development environment

Two popular methods for performing static code analysis in C++ are linting tools and
utilizing compiler tools.

6/13 page
Linting
Linting is a process of checking source code for bugs, defects, and formatting issues,
such as unused libraries in C/C++ or other languages. It's useful for finding errors
when updating compilers and ensuring style guidelines are followed to prevent
miscommunication on software projects. There are specialized linters for C++, such
as Clang-tidy, which finds and automatically fixes issues, and the IntelliSense Code
Linter developed by Microsoft for Visual C++.

Using Compiler Tools and Flags

Compilers play a crucial role during development. Not only can you use them to
produce executable programs but also for static analysis of the code base. Compilers
flag lines of code with warnings (potential issues) and errors (issues that prevent
compilation). For testing, you can configure your compilers to mark all warnings or
just specific warnings as errors. However, tooling options can be confusing and
specific to different compilers such as Clang, GCC, G++ and Visual C++. Using
compiler tools is effective but requires some expertise. It is best done alongside other
testing methods.

Dynamic Analysis Technologies and Tools

Dynamic Analysis tools test software for vulnerabilities by running it with randomized
or predefined inputs using tools such as libfuzzer. The goal is to trigger unexpected or
unusual behavior. Advantages include:

• Black-box approaches for dynamic analysis do not require the source code of
an application to test it
• Code can be executed on both production and development environments for
comparison
• Detecting runtime issues
• Testing with real integrations and dependencies

Two popular methods for dynamic code analysis are unit testing and fuzz testing.

7/13 page
Unit Testing
Unit testing is a technique in which automated tests are written by developers to
evaluate the functionality of a specific unit of code (e.g., a function or class). There
are various approaches to unit testing, such as test-driven development, during
which tests are written first to guide the development process. Another approach is
using unit tests as regression tests to identify bugs introduced into previously
working code. Unit testing is particularly useful for C/C++ developers working on
large projects, as it allows you to test specific portions of the codebase without
having to compile the entire project. In C++ for example, CppUnit, Catch2, and Boost
are popular solutions for unit testing. Visual Studio also provides a built-in C++
testing solution.

Fuzz Testing
Fuzz testing is a dynamic analysis method that involves feeding invalid or random
data, known as "fuzzy" data, into the software under test and observing how it
behaves. Similar to unit testing, the software is tested under various scenarios. The
key difference is that unit testing is deterministic, meaning that the expected
outcome is known in advance, while fuzz testing is non-deterministic or exploratory,
meaning that the expected outcome is not necessarily known ahead of time. This
makes fuzz testing useful for uncovering bugs and security vulnerabilities that may
not have been identified through other testing methods.

There are different techniques to simulate embedded dependencies using fuzz

testing, such as fuzzing hardware-dependent functions with static values or using
feedback-based fuzz data to mock the behavior of external sources in a realistic
environment. You can then combine these techniques with other dynamic and/or
static test methods to achieve even more comprehensive results. An easy-to-use
open-source tool for fuzz testing in C/C++ is our own CI Fuzz CLI, which lets you
write fuzz tests using a few simple commands.

8/13 page
Automating Test Case Creation With Fuzz Data
In my experience, writing test cases that adequately cover relevant program states
and behaviours manually takes time and bears the risk of missing things. A mocking
approach based on fuzz data is a proven effective way to speed things up and
pinpoint issues more accurately. Instead of manually trying to come up with test
cases, this approach uses fuzz data to automatically generate invalid, unexpected or
random data as input for your embedded application. This way, you can simulate the
behaviour of your embedded software units under realistic circumstances.

To make sure that your fuzzer is not just throwing random test inputs at your
application, you should opt for tools that are capable of leveraging information about
the software under test to refine fuzz data. Such a so-called “white-box” (or grey-box)
fuzzing approach enables your fuzzer to maximize test coverage based on runtime
information from previous inputs and thereby trigger (almost) all relevant program
states.

This approach to embedded software security testing is much faster and more
accurate than any form of manual testing or dumb fuzzing. Below I explained this in
more detail.

Recorded Coding Session: Mocking Embedded Systems With Fuzz Data

https://www.youtube.com/watch?v=2VrFPVr_p34

9/13 page
Mocking on Steroids: Testing Positive and Negative Criteria
With Fuzz Data
To test an application with dynamic inputs, you would usually have to compile and
run it first. But with embedded software, you often have the problem that it only runs
on specific hardware or that the application requires input from external sources. For
this reason, mocking/simulating these dependencies is needed for DAST, IAST, or
fuzzing.

A bare-bones approach to testing embedded systems can also be mocking for the
hardware-dependent functions to return static values. However, this setup is
basically blind, as it lacks runtime context, causing it to miss many possible
behaviours, resulting in comparably low code coverage.

As described above, a more accurate method for embedded security testing is by

mocking them with fuzz data. This allows you to dynamically generate return values
for your mocked functions, so you can make use of the magic of feedback-based
fuzzing to simulate the behavior of external sources under realistic conditions. Also,
this approach lets you cover both positive and negative test cases during your tests.
If you have the resources, I would always recommend combining different DAST,
IAST, and fuzzing approaches.

Two Excel sheets can be enough to generate a fuzz test that will achieve high code coverage
while using fuzz data to simulate the input from outer sources

10/13 page
The Secret Sauce to Fuzzing Embedded Software:
Instrumentation
One of the reasons why modern fuzzing technology is so effective is that it can
leverage feedback from previous test inputs to increase code coverage. To achieve
this, the source code has to be instrumented using sanitizers. A sanitizer is a
software library used to compile code with the goal of making programs crash more
often. By receiving information about the program under test from these sanitizers,
the fuzzing engine can constantly craft more effective test inputs.

Leveraging the API Documentation to Generate Fuzz Tests

You will most likely need some form of structured input to automatically generate
your test harnesses. To determine what kinds of input may be suitable for testing,
use the documentation of the Public API. In my experience, most software teams
that work on embedded hardware have such documentation stored as a CSV or
Excel table. This documentation can be used to automatically create sophisticated
fuzz tests without any further adaptations.

Example of an API documentation in a CSV table

Your fuzzer then starts by calling functions from your Public API documentation file
in random order, with random parameters. Through instrumentation, the fuzzer can
then gather feedback about the covered code. Based on this feedback, it can actively
adapt and mutate the called functions and parameters to increase code coverage
and trigger more interesting program states. This setup will allow you to generate
test cases you might not have thought of that can traverse large parts of the tested
code. With modern open-source tooling, fuzz testing can be simplified to a point
where it only takes a few commands to have a fuzzer up and running within your CLI.

11/13 page
3 Reasons Why You Should Fuzz Embedded Systems
In my experience, fuzzing is one of the most effective ways to test embedded
systems. Especially because the margin of error is very small in many embedded
industries, as software defects do not only affect the functionality of systems, they
can have a physical impact on our lives (e.g., in automotive brake systems).

1. Increased Code Coverage

By leveraging feedback about the software under test, modern fuzzers can actively
craft test inputs that maximize code coverage. Apart from reaching critical parts of
the tested code more reliably, this enables highly useful reports that let dev teams
know how much of their code actually was executed during a test. This reporting
may be helpful in identifying which parts of their code need additional testing or
more inspection

2. Detecting Memory Corruption Accurately

Uncovering memory corruption issues is one of the strong suits of feedback-based
fuzzing which is highly relevant for memory-unsafe languages such as C/C++
(nonetheless, fuzz testing is also very effective at securing memory-safe languages).
By feeding automotive software with unexpected or invalid inputs, modern fuzzing
tools can uncover dangerous edge cases and behaviours related to communication
with the memory.

3. Early Bug Detection

Feedback-based fuzzing tools can be integrated into the development process to
test your code automatically as soon as you have an executable program. By
leveraging the bug-finding capacities of fuzzing early, you can find bugs and
vulnerabilities before they become a real problem.

Example: If you find a bug in a JSON parser during unit testing, you can most likely
fix it in a couple of minutes. This is relatively easy compared to fixing a crash that
was possibly caused by a specific user input that only affects one certain
component. So do yourself a favour and fix the bugs before they pop up in
production.

12/13 page
Enterprise Best Practice to Deal With Complexity in
Embedded Software: CI/CD-Integrated Fuzz Testing
Many of the things that make fuzzing so useful are available within open-source
tooling. In fact, you could have your own fuzz test running locally in less than five
minutes by using our open-source CLI tool for C/C++.

For enterprise projects, I recommend CI/CD-integrated fuzz testing, as it allows you

and your team to fuzz your code automatically at each code change. Ideally, you
would implement such a testing set-up early in the development process to avoid
late-stage fixes and to speed up the development process. In my opinion, Thomas
Dohmke, CEO of GitHub, put it best when he said that modern fuzzing tools are "like
having an automated security expert who is always by your side".

Anyways, if you want to find out more about how you can set up fuzz testing for
large automotive projects, you can book a personal demo with one of my colleagues
or check out our product page. If you want to get started right away, I recommend
checking out CI Fuzz CLI.

CI Fuzz CLI is an open-source solution that lets you run feedback-based fuzz tests from your
command line with three simple commands: https://www.code-intelligence.com/cli-tool

Code Intelligence www.code-intelligence.com

Rheinwerkallee 6 [email protected]
53227 Bonn +49 228 28695830

13/13 page