Quickly Testing Legacy Code

Clare Macrae
5 February 2019

1
@ClareMacraeUK
 @ClareMacraeUK

2
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

3
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

4
@ClareMacraeUK
 5
@ClareMacraeUK
 6
@ClareMacraeUK
 A year of remote pairing later…

7
@ClareMacraeUK
 Goal:
Share techniques for easier testing
in challenging scenarios

8
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

9
@ClareMacraeUK
 Quickly Testing Legacy Code

10
@ClareMacraeUK
 11
@ClareMacraeUK
 What does Legacy Code really mean?
• Michael Feathers
– “code without unit tests”
• J.B. Rainsberger
– “profitable code that we feel afraid to change.”
• Kate Gregory
– “any code that you want to change, but are afraid to”

12
@ClareMacraeUK
 Typical Scenario Need to
change
• I've inherited some the code
legacy code
• It's valuable Can’t
refactor
• I need to add feature No tests
without
tests
• Or fix bug
• How can I ever break
out of this loop?

Needs
Not
refactoring
designed
to add
for testing
tests

13
@ClareMacraeUK
 Typical Scenario Need to
change
• the code

• Can’t
refactor
• No tests
without
tests
•
•
Topics of
Needs
Not
this talk
refactoring
designed
to add
for testing
tests

14
@ClareMacraeUK
 Assumptions
• Value of testing
• No worries about types of tests
– (unit, integration, regression)

15
@ClareMacraeUK
 Any existing tests?

16
@ClareMacraeUK
 What, no tests?
• If absolutely no tests…
• Stop now!
• Set one up!
• Existing framework

17
@ClareMacraeUK
 Popular Test Frameworks
Google Test Catch
• Google's C++ test framework • Phil Nash’s test framework
• https://github.com/google/googletest • https://github.com/catchorg/Catch2

18
@ClareMacraeUK
 19
@ClareMacraeUK
 How good are your existing tests?

20
@ClareMacraeUK
 First evaluate your tests
• If you do have tests….
• Test the tests!
• In area you are changing
• Reliable?

21
@ClareMacraeUK
 Code Coverage
• Caution!
• Unexecuted code

• Techniques
• Debugger
• Code coverage tool
– What to measure?

22
@ClareMacraeUK
 100% Test Coverage

Or is it?

OpenCppCoverage does not show

branch/condition coverage.

23
@ClareMacraeUK
 BullseyeCoverage:

Unrolls If conditions
(line 10)

73% of conditions
covered

24
@ClareMacraeUK
 Mutation testing: Sabotage the code!
• Test the tests
• Small changes
• Re-run tests
• Fail: ☺
• Pass: 

25
@ClareMacraeUK
 Mutation testing approaches
• By hand,
– e.g. + to -
• By tool
– e.g. Mutate++ - https://github.com/nlohmann/mutate_cpp

26
@ClareMacraeUK
 If tests need improving…
And unit tests not viable…

27
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

28
@ClareMacraeUK
 Quickly Testing Legacy Code

29
@ClareMacraeUK
 Key Phrase :
“Locking Down Current Behaviour”

30
@ClareMacraeUK
 Writing Unit Tests for Legacy Code
• Time-consuming
• What’s intended behaviour?
• Is there another way?

31
@ClareMacraeUK
 Alternative: Golden Master Testing

32
@ClareMacraeUK
 Golden Master Test Setup

Save
Input Existing
“Golden
Data Code
Master”

33
@ClareMacraeUK
 Golden Master Tests In Use

Yes
Pass
Input Updated Output
Data Code same?
Fail
No

34
@ClareMacraeUK
 Thoughts on Golden Master Tests
• Good to start testing legacy systems
• Poor Person’s Integration Tests)
• Depends on ease of
– Capturing output
– Getting stable output
– Reviewing any differences
– Avoiding overwriting Master by mistake!
• Doesn’t matter that it’s not a Unit Test

35
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

36
@ClareMacraeUK
 Quickly Testing Legacy Code

37
@ClareMacraeUK
 One approach: “ApprovalTests”
• Llewellyn Falco’s convenient, powerful, flexible Golden Master implementation
• Supports many language

GO .Net.ASP Perl
Java NodeJS Python
Lua Objective-C Swift
.NET PHP TCL

And now C++!

38
@ClareMacraeUK
 ApprovalTests.cpp
• New C++ library for applying Llewellyn Falco’s “Approval Tests” approach
• For testing cross-platform C++ code (Windows, Linux, Mac)
• For legacy and green-field systems
• It’s on github

39
@ClareMacraeUK
 ApprovalTests.cpp
• Header-only
• Open source - Apache 2.0 licence

40
@ClareMacraeUK
 ApprovalTests.cpp
• Works with a range of testing frameworks
• Currently supports Catch1, Catch2, Google Test and Okra

41
@ClareMacraeUK
 Getting Started with Approvals

42
@ClareMacraeUK
 StarterProject with Catch2
• https://github.com/approvals/ApprovalTests.cpp.StarterProject
• Download ZIP

43
@ClareMacraeUK
 How to use it: Catch2 Boilerplate
• Your main.cpp

#define APPROVALS_CATCH
#include "ApprovalTests.hpp"

44
@ClareMacraeUK
 Pure Catch2 Test
• A test file

#include "Catch.hpp"

// Catch-only test
TEST_CASE( "Sums are calculated" )
{
REQUIRE( 1 + 1 == 2 );
REQUIRE( 1 + 2 == 3 );
}

45
@ClareMacraeUK
 Approvals Catch2 Test
• A test file (Test02.cpp)

#include "ApprovalTests.hpp"
#include "Catch.hpp"

// Approvals test - test static value, for demo purposes

TEST_CASE("TestFixedInput")
{
Approvals::verify("Some\nMulti-line\noutput");
}

46
@ClareMacraeUK
 First run
• 1.

• 2. Differencing tool pops up (Araxis Merge, in this case)

47
@ClareMacraeUK
 Actual/Received Expected/Approved

48
@ClareMacraeUK
 Actual/Received Expected/Approved

49
@ClareMacraeUK
 Actual/Received Expected/Approved

50
@ClareMacraeUK
 Second run
• I approved the output

• Then we commit the test, and the approved file(s) to version control
• The diffing tool only shows up if:
– there is not (yet) an Approved file or
– the Received differs from the Approved

51
@ClareMacraeUK
 What’s going on here?
• It’s a very convenient form of Golden Master testing
• Objects being tested are stringified – that’s a requirement
• Captures snapshot of current behaviour
• Useful for locking down behaviour of existing code
– Not intended to replace Unit Tests

52
@ClareMacraeUK
 Example test directory

53
@ClareMacraeUK
 How to use it: GoogleTest Boilerplate
• Your main.cpp

#define APPROVALS_GOOGLETEST
#include "ApprovalTests.hpp"

54
@ClareMacraeUK
 Pure Google Test
• A test file

#include <gtest/gtest.h>

// Google-only test
TEST( Test01, SumsAreCalculated )
{
EXPECT_EQ( 1 + 1, 2 );
EXPECT_EQ( 1 + 2, 3 );
}

55
@ClareMacraeUK
 Approvals Google Test
• A test file

#include "ApprovalTests.hpp"
#include <gtest/gtest.h>

// googletest - test static value, for demo purposes

TEST( Test02, TestFixedInput )
{
Approvals::verify("Some\nMulti-line\noutput");
}

56
@ClareMacraeUK
 Reference: Add to existing GoogleTest
• Your main.cpp
#define APPROVALS_GOOGLETEST_EXISTING_MAIN
#include "ApprovalTests.hpp"

int main(int argc, char** argv)

{
::testing::InitGoogleTest(&argc, argv);
initializeApprovalTestsForGoogleTests();
return RUN_ALL_TESTS();
}

57
@ClareMacraeUK
 Reference: Supporting a new test framework
• Any test framework that supplies the following:
– Current test’s name
– Current test’s source file (with correct case of filename)
– Report unexpected exceptions as a test failure
• That’s all that’s needed

58
@ClareMacraeUK
 A note on Tools
• I’ll be using a variety of tools in the talk
• I’ll mention them as I go
• Slides of references at the end

59
@ClareMacraeUK
 How does this help with legacy code?

60
@ClareMacraeUK
 Applying this to legacy code
TEST_CASE("New test of legacy feature")
{
// Standard pattern:
// Wrap your legacy feature to test in a function call
// that returns some state that can be written to a text file
// for verification:
const LegacyThing result = doLegacyOperation();
Approvals::verify(result);
}
61
@ClareMacraeUK
 Implementation
class LegacyThing;

std::ostream &operator<<(std::ostream &os, const LegacyThing &result) {

// write interesting info from result here:
os << result…;
return os;
}
LegacyThing doLegacyOperation() {
// your implementation here..
return LegacyThing();
}

62
@ClareMacraeUK
 Feature: Consistency over machines
• Naming of output files
• Approved files version controlled

63
@ClareMacraeUK
 Feature: Consistency over OSs
• Line-endings

64
@ClareMacraeUK
 Feature: Quick to write tests
TEST_CASE("verifyAllWithHeaderBeginEndAndLambda")
{
std::list<int> numbers{ 0, 1, 2, 3};
// Multiple convenience overloads of Approvals::verifyAll()
Approvals::verifyAll(
"Test Squares", numbers.begin(), numbers.end(),
[](int v, std::ostream& s) { s << v << " => " << v*v << '\n' ; });
}

65
@ClareMacraeUK
 Feature: Quick to get good coverage
TEST_CASE("verifyAllCombinationsWithLambda")
{
std::vector<std::string> strings{"hello", "world"};
std::vector<int> numbers{1, 2, 3};
CombinationApprovals::verifyAllCombinations<
std::vector<std::string>, // The type of element in our first input container
std::vector<int>, // The type of element in our second input container
std::string>( // The return type from testing one combination of inputs
// Lambda that acts on one combination of inputs, and returns the result to be approved:
[](std::string s, int i) { return s + " " + std::to_string(i); },
strings, // The first input container
numbers); // The second input container
}

66
@ClareMacraeUK
 All values in single output file:
(hello, 1) => hello 1
(hello, 2) => hello 2
(hello, 3) => hello 3
(world, 1) => world 1
(world, 2) => world 2
(world, 3) => world 3

67
@ClareMacraeUK
 Challenge: Golden Master is a log file
• Dates and times?
• Object addresses?

2019-01-29 15:27

68
@ClareMacraeUK
 Options for unstable output
• Introduce date-time abstraction?
• Customised comparison function?
• Or: strip dates from the log file

69
@ClareMacraeUK
 Tip: Rewrite output file

2019-01-29 15:27 [date-time-stamp]

70
@ClareMacraeUK
 Customisability: ApprovalWriter interface
class ApprovalWriter
{
public:
virtual std::string getFileExtensionWithDot() = 0;
virtual void write(std::string path) = 0;
virtual void cleanUpReceived(std::string receivedPath) = 0;
};

71
@ClareMacraeUK
 Feature: Run on build servers
TEST_CASE("UseQuietReporter")
{
// QuietReporter does nothing if a failure occurs.
// Failing tests will still fail, but nothing is launched.
Approvals::verify(
"Some\nMulti-line\noutput",
QuietReporter());
}

72
@ClareMacraeUK
 Customisability: Reporters
• Very simple but powerful abstraction
• Gives complete user control over how to inspect and act on a failure
• If you adopt Approvals, do review the supplied reporters for ideas

73
@ClareMacraeUK
 Tip: currentReporter()
• Easy to switch reporter behaviour

74
@ClareMacraeUK
 Feature: Convention over Configuration
• Fewer decisions for developers
• Users only specify unusual behaviours

75
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

76
@ClareMacraeUK
 What I can do:

77
@ClareMacraeUK
 What I need to do:

78
@ClareMacraeUK
 Sugar - sucrose

79
@ClareMacraeUK
 Sugar - sucrose

80
@ClareMacraeUK
 Sugar - sucrose

81
@ClareMacraeUK
 Sugar - sucrose

82
@ClareMacraeUK
 Sugar - sucrose

83
@ClareMacraeUK
 84
@ClareMacraeUK
 Approving Images

85
@ClareMacraeUK
 What I’m aiming for
TEST(PolyhedralGraphicsStyleApprovals, CUVXAT)
{
// CUVXAT identifies a crystal structure in the database
const QImage crystal_picture = loadEntry("CUVXAT");
verifyQImage(crystal_picture, *currentReporter());
}

86
@ClareMacraeUK
 Foundation of Approval tests
void FileApprover::verify(
ApprovalNamer& namer,
ApprovalWriter& writer,
const Reporter& reporter);

87
@ClareMacraeUK
 Idiomatic verification of Qt image objects
void verifyQImage(
QImage image, const Reporter& reporter = DiffReporter())
{
ApprovalTestNamer namer;
QImageWriter writer(image);
FileApprover::verify(namer, writer, reporter);
}

88
@ClareMacraeUK
 Customisability: ApprovalWriter interface
class ApprovalWriter
{
public:
virtual std::string getFileExtensionWithDot() = 0;
virtual void write(std::string path) = 0;
virtual void cleanUpReceived(std::string receivedPath) = 0;
};

89
@ClareMacraeUK
 QImageWriter declaration
class QImageWriter : public ApprovalWriter
{
public:
explicit QImageWriter(QImage image, std::string fileExtensionWithDot = ".png");
std::string getFileExtensionWithDot() override;
void write(std::string path) override;
void cleanUpReceived(std::string receivedPath) override;

private:
QImage image_;
std::string ext;
};

90
@ClareMacraeUK
 QImageWriter::write()
void QImageWriter::write(std::string path)
{
// Have to convert std::string to QString
image_.save( QString::fromStdString(path) );
}

91
@ClareMacraeUK
 Useful feedback BeyondCompare 4

92
@ClareMacraeUK
 Back In work…
• Previously, working from home via Remote Desktop
• The tests started failing when I ran them in work

93
@ClareMacraeUK
 What? Re-running now gives random failures
Araxis Merge

94
@ClareMacraeUK
 BeyondCompare 4

95
@ClareMacraeUK
 96
@ClareMacraeUK
 97
@ClareMacraeUK
 98
@ClareMacraeUK
 99
@ClareMacraeUK
 100
@ClareMacraeUK
 Customisability: ApprovalComparator
class ApprovalComparator
{
public:
virtual ~ApprovalComparator() = default;

virtual bool contentsAreEquivalent(std::string receivedPath,

std::string approvedPath) const = 0;
};

101
@ClareMacraeUK
 Create custom QImage comparison class
// Allow differences in up to 1/255 of RGB values at each pixel, as saved in 32-bit images
// sometimes have a few slightly different pixels, which are not visible to the human eye.
class QImageApprovalComparator : public ApprovalComparator
{
public:
bool contentsAreEquivalent(std::string receivedPath, std::string approvedPath) const override
{
const QImage receivedImage(QString::fromStdString(receivedPath));
const QImage approvedImage(QString::fromStdString(approvedPath));

return compareQImageIgnoringTinyDiffs(receivedImage, approvedImage);

}
};

102
@ClareMacraeUK
 Register the custom comparison class
// Somewhere in main…
FileApprover::registerComparator(
".png",
std::make_shared<QImageApprovalComparator>());

103
@ClareMacraeUK
 Does the difference matter?
• Legacy code is often brittle
• Testing makes changes visible
• Then decide if change matters
• Fast feedback cycle for efficient development

104
@ClareMacraeUK
 Looking back
• User perspective
• Learned about own code
• Enabled unit tests for graphics problems
• Approvals useful even for temporary tests

105
@ClareMacraeUK
 ApprovalTests Customisation Points
• Reporter
• ApprovalWriter
• ApprovalComparator
• ApprovalNamer

106
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

107
@ClareMacraeUK
 References: Tools Used
• Diffing
– Araxis Merge: https://www.araxis.com/merge/
– Beyond Compare: https://www.scootersoftware.com/
• Code Coverage
– OpenCppCoverage and Visual Studio Plugin: https://github.com/OpenCppCoverage
– BullseyeCoverage: https://www.bullseye.com/

108
@ClareMacraeUK
 Beautiful C++: Updating Legacy Code
• https://app.pluralsight.com/library/courses/cpp-updating-legacy-code/table-of-contents

109
@ClareMacraeUK
 The Legacy Code Programmer’s Toolbox
• https://leanpub.com/legacycode
• The Legacy Code Programmer's Toolbox: Practical skills for software professionals working
with legacy code, by Jonathan Boccara

110
@ClareMacraeUK
 Videos

111
@ClareMacraeUK https://youtu.be/aWiwDdx_rdo
 Adam Tornhill’s Books
• Mining actionable information from historical code bases (by Adam Tornhill)
– Your Code as a Crime Scene
– Software Design X-Rays: Fix Technical Debt with Behavioural Code Analysis

112
@ClareMacraeUK
 Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

113
@ClareMacraeUK
 Adopting legacy code
• You can do it!
• Evaluate current tests
• Quickly improve coverage with Golden Master
• ApprovalTests.cpp makes that really easy
• Even for non-text types

114
@ClareMacraeUK
 ApprovalTests
• Powerful Golden Master tool
• verify(), verifyAll(), verifyAllCombinations()
• Adjust output files, after writing, to simplify comparison

115
@ClareMacraeUK
 ApprovalTests.cpp
• Not (always) a replacement for Unit Tests!

Archive
Golden Refactor Add unit
Approval
Master code tests
Tests?

116
@ClareMacraeUK
 Thank You: Any Questions?
• Example Code: https://github.com/claremacrae/cpponsea2019

• Slides: https://www.slideshare.net/ClareMacrae
– (later this week)

• ApprovalTests.cpp
– https://github.com/approvals/ApprovalTests.cpp
– https://github.com/approvals/ApprovalTests.cpp.StarterProject

117
@ClareMacraeUK