100% found this document useful (4 votes)
54 views

Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall 2024 Scribd Download

Comprehensive

Uploaded by

edelyconcon
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (4 votes)
54 views

Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall 2024 Scribd Download

Comprehensive

Uploaded by

edelyconcon
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 62

Download the full version of the textbook now at textbookfull.

com

Programming Persistent Memory: A


Comprehensive Guide For Developers 1st
Edition Steve Scargall

https://textbookfull.com/product/programming-
persistent-memory-a-comprehensive-guide-for-
developers-1st-edition-steve-scargall/

Explore and download more textbook at https://textbookfull.com


Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Camping Washington A Comprehensive Guide to Public Tent


and RV Campgrounds 3rd Edition Steve Giordano

https://textbookfull.com/product/camping-washington-a-comprehensive-
guide-to-public-tent-and-rv-campgrounds-3rd-edition-steve-giordano/

textbookfull.com

Scala Programming A comprehensive beginner s guide to


Scala 2nd Edition Claudia Alves

https://textbookfull.com/product/scala-programming-a-comprehensive-
beginner-s-guide-to-scala-2nd-edition-claudia-alves/

textbookfull.com

The Economics of Education: A Comprehensive Overview 2nd


Edition Steve Bradley (Editor)

https://textbookfull.com/product/the-economics-of-education-a-
comprehensive-overview-2nd-edition-steve-bradley-editor/

textbookfull.com

The examinations of Anne Askew 1st Edition Anne Askew


(Author) & Elaine V. Beilin (Editor)

https://textbookfull.com/product/the-examinations-of-anne-askew-1st-
edition-anne-askew-author-elaine-v-beilin-editor/

textbookfull.com
100 Weight Loss Bowls Build Your Own Calorie Controlled
Diet Plan Heather Whinney

https://textbookfull.com/product/100-weight-loss-bowls-build-your-own-
calorie-controlled-diet-plan-heather-whinney/

textbookfull.com

In His Sights (The Law Trilogy: Breaking the Law Book 1)


1st Edition Haven Rose [Rose

https://textbookfull.com/product/in-his-sights-the-law-trilogy-
breaking-the-law-book-1-1st-edition-haven-rose-rose/

textbookfull.com

Taking sacred back the complete guide to designing sharing


group rituals First Edition Linde

https://textbookfull.com/product/taking-sacred-back-the-complete-
guide-to-designing-sharing-group-rituals-first-edition-linde/

textbookfull.com

Molecular Basis of Nutrition and Aging: A Volume in the


Molecular Nutrition Series 1st Edition Marco Malavolta Phd

https://textbookfull.com/product/molecular-basis-of-nutrition-and-
aging-a-volume-in-the-molecular-nutrition-series-1st-edition-marco-
malavolta-phd/
textbookfull.com

Testing Vue js Applications 1st Edition Edd Yerburgh

https://textbookfull.com/product/testing-vue-js-applications-1st-
edition-edd-yerburgh/

textbookfull.com
Keeping Their Marbles: How the Treasures of the Past Ended
Up in Museums - And Why They Should Stay There 1st Edition
Tiffany Jenkins
https://textbookfull.com/product/keeping-their-marbles-how-the-
treasures-of-the-past-ended-up-in-museums-and-why-they-should-stay-
there-1st-edition-tiffany-jenkins/
textbookfull.com
Programming
Persistent
Memory
A Comprehensive Guide for Developers

Steve Scargall
Programming Persistent
Memory
A Comprehensive Guide for
Developers

Steve Scargall
Programming Persistent Memory: A Comprehensive Guide for Developers

Steve Scargall
Santa Clara, CA, USA

ISBN-13 (pbk): 978-1-4842-4931-4 ISBN-13 (electronic): 978-1-4842-4932-1


https://doi.org/10.1007/978-1-4842-4932-1
Copyright © 2020 by Intel
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction
on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use,
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this book are included in the book’s Creative Commons license, unless
indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons
license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every
occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion
and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified
as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither
the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may
be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Susan McDermott
Development Editor: Laura Berendson
Coordinating Editor: Jessica Vakili
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected],
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer
Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail [email protected], or visit http://www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses
are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at
http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to readers on
GitHub via the book's product page, located at www.apress.com/978-1-4842-4931-4. For more detailed
information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii
About the Technical Reviewer���������������������������������������������������������������������������������xv
About the Contributors������������������������������������������������������������������������������������������xvii
Acknowledgments��������������������������������������������������������������������������������������������������xxi
Preface�����������������������������������������������������������������������������������������������������������������xxiii

Chapter 1: Introduction to Persistent Memory Programming���������������������������������� 1


A High-Level Example Program���������������������������������������������������������������������������������������������������� 2
What’s Different?��������������������������������������������������������������������������������������������������������������������� 5
The Performance Difference��������������������������������������������������������������������������������������������������� 6
Program Complexity���������������������������������������������������������������������������������������������������������������� 7
How Does libpmemkv Work?��������������������������������������������������������������������������������������������������� 8
What’s Next?��������������������������������������������������������������������������������������������������������������������������������� 9
Summary�������������������������������������������������������������������������������������������������������������������������������������� 9

Chapter 2: Persistent Memory Architecture����������������������������������������������������������� 11


Persistent Memory Characteristics��������������������������������������������������������������������������������������������� 12
Platform Support for Persistent Memory������������������������������������������������������������������������������������ 13
Cache Hierarchy�������������������������������������������������������������������������������������������������������������������������� 14
Power-Fail Protected Domains���������������������������������������������������������������������������������������������������� 16
The Need for Flushing, Ordering, and Fencing���������������������������������������������������������������������������� 19
Data Visibility������������������������������������������������������������������������������������������������������������������������������ 23
Intel Machine Instructions for Persistent Memory���������������������������������������������������������������������� 24
Detecting Platform Capabilities�������������������������������������������������������������������������������������������������� 25

iii
Table of Contents

Application Startup and Recovery����������������������������������������������������������������������������������������������� 27


What’s Next?������������������������������������������������������������������������������������������������������������������������������� 29
Summary������������������������������������������������������������������������������������������������������������������������������������ 29

Chapter 3: Operating System Support for Persistent Memory������������������������������� 31


Operating System Support for Memory and Storage������������������������������������������������������������������ 31
Persistent Memory As Block Storage������������������������������������������������������������������������������������������ 33
Persistent Memory-Aware File Systems������������������������������������������������������������������������������������� 34
Memory-Mapped Files���������������������������������������������������������������������������������������������������������������� 35
Persistent Memory Direct Access (DAX)������������������������������������������������������������������������������������� 43
Summary������������������������������������������������������������������������������������������������������������������������������������ 53

Chapter 4: Fundamental Concepts of Persistent Memory Programming��������������� 55


What’s Different?������������������������������������������������������������������������������������������������������������������������ 55
Atomic Updates��������������������������������������������������������������������������������������������������������������������������� 56
Transactions�������������������������������������������������������������������������������������������������������������������������������� 57
Atomicity������������������������������������������������������������������������������������������������������������������������������� 57
Consistency��������������������������������������������������������������������������������������������������������������������������� 58
Isolation��������������������������������������������������������������������������������������������������������������������������������� 58
Durability������������������������������������������������������������������������������������������������������������������������������� 58
Flushing Is Not Transactional������������������������������������������������������������������������������������������������������ 59
Start-Time Responsibilities��������������������������������������������������������������������������������������������������������� 59
Tuning for Hardware Configurations������������������������������������������������������������������������������������������� 60
Summary������������������������������������������������������������������������������������������������������������������������������������ 60

Chapter 5: Introducing the Persistent Memory Development Kit��������������������������� 63


Background��������������������������������������������������������������������������������������������������������������������������������� 63
Choosing the Right Semantics���������������������������������������������������������������������������������������������������� 64
Volatile Libraries������������������������������������������������������������������������������������������������������������������������� 65
libmemkind���������������������������������������������������������������������������������������������������������������������������� 65
libvmemcache����������������������������������������������������������������������������������������������������������������������� 66
libvmem��������������������������������������������������������������������������������������������������������������������������������� 67

iv
Table of Contents

Persistent Libraries��������������������������������������������������������������������������������������������������������������������� 67
libpmem�������������������������������������������������������������������������������������������������������������������������������� 67
libpmemobj���������������������������������������������������������������������������������������������������������������������������� 68
libpmemobj-cpp�������������������������������������������������������������������������������������������������������������������� 68
libpmemkv����������������������������������������������������������������������������������������������������������������������������� 69
libpmemlog���������������������������������������������������������������������������������������������������������������������������� 69
libpmemblk���������������������������������������������������������������������������������������������������������������������������� 69
Tools and Command Utilities������������������������������������������������������������������������������������������������������� 70
pmempool������������������������������������������������������������������������������������������������������������������������������ 70
pmemcheck��������������������������������������������������������������������������������������������������������������������������� 70
pmreorder������������������������������������������������������������������������������������������������������������������������������ 71
Summary������������������������������������������������������������������������������������������������������������������������������������ 71

Chapter 6: libpmem: Low-Level Persistent Memory Support��������������������������������� 73


Using the Library������������������������������������������������������������������������������������������������������������������������� 74
Mapping a File���������������������������������������������������������������������������������������������������������������������������� 75
Copying to Persistent Memory���������������������������������������������������������������������������������������������������� 76
Separating the Flush Steps��������������������������������������������������������������������������������������������������������� 77
Summary������������������������������������������������������������������������������������������������������������������������������������ 79

Chapter 7: libpmemobj: A Native Transactional Object Store��������������������������������� 81


What is libpmemobj?������������������������������������������������������������������������������������������������������������������ 81
Why not malloc( )?����������������������������������������������������������������������������������������������������������������������� 82
Grouping Operations������������������������������������������������������������������������������������������������������������������� 83
Memory Pools����������������������������������������������������������������������������������������������������������������������������� 83
Creating Memory Pools��������������������������������������������������������������������������������������������������������� 83
Pool Object Pointer (POP) and the Root Object���������������������������������������������������������������������� 87
Opening and Reading from Memory Pools���������������������������������������������������������������������������� 88
Memory Poolsets������������������������������������������������������������������������������������������������������������������������ 90
Concatenated Poolsets���������������������������������������������������������������������������������������������������������� 90
Replica Poolsets�������������������������������������������������������������������������������������������������������������������� 91
Managing Memory Pools and Poolsets��������������������������������������������������������������������������������������� 92

v
Table of Contents

Typed Object Identifiers (TOIDs)�������������������������������������������������������������������������������������������������� 92


Allocating Memory���������������������������������������������������������������������������������������������������������������������� 93
Persisting Data���������������������������������������������������������������������������������������������������������������������������� 94
Atomic Operations����������������������������������������������������������������������������������������������������������������� 94
Reserve/Publish API�������������������������������������������������������������������������������������������������������������� 97
Transactional API����������������������������������������������������������������������������������������������������������������� 100
Optional Flags���������������������������������������������������������������������������������������������������������������������� 104
Persisting Data Summary���������������������������������������������������������������������������������������������������� 104
Guarantees of libpmemobj’s APIs��������������������������������������������������������������������������������������������� 105
Managing Library Behavior������������������������������������������������������������������������������������������������������� 106
Debugging and Error Handling�������������������������������������������������������������������������������������������������� 106
Summary���������������������������������������������������������������������������������������������������������������������������������� 108

Chapter 8: libpmemobj-cpp: The Adaptable Language - C++ and


Persistent Memory����������������������������������������������������������������������������������������������� 111
Introduction������������������������������������������������������������������������������������������������������������������������������� 111
Metaprogramming to the Rescue��������������������������������������������������������������������������������������������� 112
Persistent Pointers�������������������������������������������������������������������������������������������������������������� 112
Transactions������������������������������������������������������������������������������������������������������������������������ 113
Snapshotting����������������������������������������������������������������������������������������������������������������������� 115
Allocating����������������������������������������������������������������������������������������������������������������������������� 116
C++ Standard limitations���������������������������������������������������������������������������������������������������������� 118
An Object’s Lifetime������������������������������������������������������������������������������������������������������������ 119
Trivial Types������������������������������������������������������������������������������������������������������������������������� 120
Object Layout����������������������������������������������������������������������������������������������������������������������� 122
Pointers������������������������������������������������������������������������������������������������������������������������������� 123
Limitations Summary���������������������������������������������������������������������������������������������������������� 125
Persistence Simplified�������������������������������������������������������������������������������������������������������������� 126
The Ecosystem�������������������������������������������������������������������������������������������������������������������������� 133
Persistent Containers���������������������������������������������������������������������������������������������������������� 134
Examples of Persistent Containers�������������������������������������������������������������������������������������� 135
Summary���������������������������������������������������������������������������������������������������������������������������������� 138

vi
Table of Contents

Chapter 9: pmemkv: A Persistent In-­Memory Key-Value Store���������������������������� 141


pmemkv Architecture���������������������������������������������������������������������������������������������������������������� 143
A Phonebook Example�������������������������������������������������������������������������������������������������������������� 147
Bringing Persistent Memory Closer to the Cloud���������������������������������������������������������������������� 151
Summary���������������������������������������������������������������������������������������������������������������������������������� 152

Chapter 10: Volatile Use of Persistent Memory���������������������������������������������������� 155


Introduction������������������������������������������������������������������������������������������������������������������������������� 155
Background������������������������������������������������������������������������������������������������������������������������������� 156
Memory Allocation��������������������������������������������������������������������������������������������������������������� 156
How it Works����������������������������������������������������������������������������������������������������������������������� 156
Supported “Kinds” of Memory��������������������������������������������������������������������������������������������� 157
The memkind API���������������������������������������������������������������������������������������������������������������������� 159
Kind Management API��������������������������������������������������������������������������������������������������������� 159
Heap Management API�������������������������������������������������������������������������������������������������������� 164
Kind Configuration Management����������������������������������������������������������������������������������������� 167
Additional memkind Code Examples����������������������������������������������������������������������������������� 168
C++ Allocator for PMEM Kind��������������������������������������������������������������������������������������������������� 168
pmem::allocator methods���������������������������������������������������������������������������������������������������� 169
Nested Containers��������������������������������������������������������������������������������������������������������������� 169
C++ Examples�������������������������������������������������������������������������������������������������������������������������� 170
Using the pmem::allocator�������������������������������������������������������������������������������������������������� 170
Creating a Vector of Strings������������������������������������������������������������������������������������������������� 171
Expanding Volatile Memory Using Persistent Memory�������������������������������������������������������������� 173
libvmemcache: An Efficient Volatile Key-Value Cache for Large-Capacity
Persistent Memory�������������������������������������������������������������������������������������������������������������������� 177
libvmemcache Overview����������������������������������������������������������������������������������������������������� 178
libvmemcache Design��������������������������������������������������������������������������������������������������������� 180
Using libvmemcache����������������������������������������������������������������������������������������������������������� 183
Summary���������������������������������������������������������������������������������������������������������������������������������� 186

vii
Table of Contents

Chapter 11: Designing Data Structures for Persistent Memory��������������������������� 187


Contiguous Data Structures and Fragmentation����������������������������������������������������������������������� 187
Internal and External Fragmentation����������������������������������������������������������������������������������� 188
Atomicity and Consistency�������������������������������������������������������������������������������������������������� 189
Selective Persistence���������������������������������������������������������������������������������������������������������� 193
Example Data Structures����������������������������������������������������������������������������������������������������� 193
Summary���������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 12: Debugging Persistent Memory Applications������������������������������������� 207


pmemcheck for Valgrind����������������������������������������������������������������������������������������������������������� 208
Stack Overflow Example������������������������������������������������������������������������������������������������������ 208
Memory Leak Example�������������������������������������������������������������������������������������������������������� 209
Intel Inspector – Persistence Inspector������������������������������������������������������������������������������������ 210
Stack Overflow Example������������������������������������������������������������������������������������������������������ 211
Memory Leak Example�������������������������������������������������������������������������������������������������������� 212
Common Persistent Memory Programming Problems�������������������������������������������������������������� 214
Nonpersistent Stores����������������������������������������������������������������������������������������������������������� 214
Stores Not Added into a Transaction������������������������������������������������������������������������������������ 228
Memory Added to Two Different Transactions��������������������������������������������������������������������� 233
Memory Overwrites������������������������������������������������������������������������������������������������������������� 240
Unnecessary Flushes���������������������������������������������������������������������������������������������������������� 242
Out-of-Order Writes������������������������������������������������������������������������������������������������������������� 247
Summary���������������������������������������������������������������������������������������������������������������������������������� 259

Chapter 13: Enabling Persistence Using a Real-World Application���������������������� 261


The Database Example������������������������������������������������������������������������������������������������������������� 262
Different Persistent Memory Enablement Approaches������������������������������������������������������������� 262
Developing a Persistent Memory-Aware MariaDB* Storage Engine����������������������������������������� 263
Understanding the Storage Layer���������������������������������������������������������������������������������������� 264
Creating a Storage Engine Class����������������������������������������������������������������������������������������� 265
Summary���������������������������������������������������������������������������������������������������������������������������������� 276

viii
Table of Contents

Chapter 14: Concurrency and Persistent Memory������������������������������������������������ 277


Transactions and Multithreading����������������������������������������������������������������������������������������������� 278
Mutexes on Persistent Memory������������������������������������������������������������������������������������������������ 282
Atomic Operations and Persistent Memory������������������������������������������������������������������������������ 285
Lock-Free Algorithms and Persistent Memory�������������������������������������������������������������������� 285
Concurrent Data Structures for Persistent Memory������������������������������������������������������������������ 286
Concurrent Ordered Map����������������������������������������������������������������������������������������������������� 287
Concurrent Hash Map���������������������������������������������������������������������������������������������������������� 291
Summary���������������������������������������������������������������������������������������������������������������������������������� 293

Chapter 15: Profiling and Performance���������������������������������������������������������������� 295


Introduction������������������������������������������������������������������������������������������������������������������������������� 295
Performance Analysis Concepts����������������������������������������������������������������������������������������������� 295
Compute-Bound vs. Memory-Bound����������������������������������������������������������������������������������� 295
Memory Latency vs. Memory Capacity������������������������������������������������������������������������������� 296
Read vs. Write Performance������������������������������������������������������������������������������������������������ 296
Memory Access Patterns����������������������������������������������������������������������������������������������������� 296
I/O Storage Bound Workloads���������������������������������������������������������������������������������������������� 297
Determining the Suitability of Workloads for Persistent Memory��������������������������������������������� 297
Volatile Use Cases��������������������������������������������������������������������������������������������������������������� 298
Use Cases Requiring Persistence���������������������������������������������������������������������������������������� 301
Performance Analysis of Workloads Using Persistent Memory������������������������������������������������ 302
Characterizing the Workload����������������������������������������������������������������������������������������������� 303
Memory Bandwidth and Latency����������������������������������������������������������������������������������������� 303
Persistent Memory Read-Write Ratio���������������������������������������������������������������������������������� 305
Working Set Size and Memory Footprint����������������������������������������������������������������������������� 305
Non-Uniform Memory Architecture (NUMA) Behavior��������������������������������������������������������� 305
Optimizing the Software for Persistent Memory����������������������������������������������������������������� 307
Summary���������������������������������������������������������������������������������������������������������������������������������� 311

ix
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Table of Contents

Chapter 16: PMDK Internals: Important Algorithms and Data Structures������������ 313
A Pool of Persistent Memory: High-Level Architecture Overview��������������������������������������������� 313
The Uncertainty of Memory Mapping: Persistent Memory Object Identifier����������������������������� 315
Persistent Thread Local Storage: Using Lanes�������������������������������������������������������������������������� 318
Ensuring Power-Fail Atomicity: Redo and Undo Logging���������������������������������������������������������� 320
Transaction Redo Logging��������������������������������������������������������������������������������������������������� 320
Transaction Undo Logging��������������������������������������������������������������������������������������������������� 321
libpmemobj Unified Logging������������������������������������������������������������������������������������������������ 322
Persistent Allocations: The Interface of a Transactional Persistent Allocator���������������������������� 323
Persistent Memory Heap Management: Allocator Design for Persistent Memory�������������������� 324
ACID Transactions: Efficient Low-Level Persistent Transactions����������������������������������������������� 328
Lazy Reinitialization of Variables: Storing the Volatile State on Persistent Memory����������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 331

Chapter 17: Reliability, Availability, and Serviceability (RAS)������������������������������ 333


Dealing with Uncorrectable Errors�������������������������������������������������������������������������������������������� 333
Consumed Uncorrectable Error Handling���������������������������������������������������������������������������� 334
Unconsumed Uncorrectable Error Handling������������������������������������������������������������������������ 336
Clearing Uncorrectable Errors��������������������������������������������������������������������������������������������� 339
Device Health���������������������������������������������������������������������������������������������������������������������������� 339
ACPI-Defined Health Functions (_NCH, _NBS)��������������������������������������������������������������������� 342
Vendor-Specific Device Health (_DSMs)������������������������������������������������������������������������������ 342
ACPI NFIT Health Event Notification������������������������������������������������������������������������������������ 343
Unsafe/Dirty Shutdown������������������������������������������������������������������������������������������������������������� 343
Application Utilization of Data Loss Count (DLC)����������������������������������������������������������������� 344
Summary���������������������������������������������������������������������������������������������������������������������������������� 346

Chapter 18: Remote Persistent Memory��������������������������������������������������������������� 347


RDMA Networking Protocols����������������������������������������������������������������������������������������������������� 348
Goals of the Initial Remote Persistent Memory Architecture���������������������������������������������������� 351
Guaranteeing Remote Persistence�������������������������������������������������������������������������������������������� 351
General-Purpose Remote Replication Method��������������������������������������������������������������������� 353

x
Table of Contents

Appliance Remote Replication Method�������������������������������������������������������������������������������� 355


General Software Architecture�������������������������������������������������������������������������������������������������� 357
librpmem Architecture and Its Use in Replication��������������������������������������������������������������������� 358
Configuring Remote Replication Using Poolsets����������������������������������������������������������������� 362
Performance Considerations����������������������������������������������������������������������������������������������� 362
Remote Replication Error Handling������������������������������������������������������������������������������������� 364
Say Hello to the Replicated World��������������������������������������������������������������������������������������� 364
Summary���������������������������������������������������������������������������������������������������������������������������������� 370

Chapter 19: Advanced Topics������������������������������������������������������������������������������� 373


Nonuniform Memory Access (NUMA)���������������������������������������������������������������������������������������� 373
NUMACTL Linux Utility��������������������������������������������������������������������������������������������������������� 374
NDCTL Linux Utility�������������������������������������������������������������������������������������������������������������� 376
Intel Memory Latency Checker Utility���������������������������������������������������������������������������������� 378
NUMASTAT Utility����������������������������������������������������������������������������������������������������������������� 380
Intel VTune Profiler – Platform Profiler�������������������������������������������������������������������������������� 381
IPMCTL Utility���������������������������������������������������������������������������������������������������������������������� 381
BIOS Tuning Options������������������������������������������������������������������������������������������������������������ 382
Automatic NUMA Balancing������������������������������������������������������������������������������������������������� 382
Using Volume Managers with Persistent Memory�������������������������������������������������������������������� 383
The mmap( ) MAP_SYNC Flag��������������������������������������������������������������������������������������������������� 385
Summary���������������������������������������������������������������������������������������������������������������������������������� 386

Appendix A: How to Install NDCTL and DAXCTL on Linux������������������������������������� 389


Prerequisites����������������������������������������������������������������������������������������������������������������������������� 389
Installing NDCTL and DAXCTL Using the Linux Distribution Package Repository���������������������� 390
Searching for Packages Within a Package Repository�������������������������������������������������������� 391
Installing NDCTL and DAXCTL from the Package Repository���������������������������������������������� 392

xi
Table of Contents

Appendix B: How to Install the Persistent Memory Development Kit (PMDK)������ 395
PMDK Prerequisites������������������������������������������������������������������������������������������������������������������ 395
Installing PMDK Using the Linux Distribution Package Repository������������������������������������������� 395
Package Naming Convention����������������������������������������������������������������������������������������������� 396
Searching for Packages Within a Package Repository�������������������������������������������������������� 396
Installing PMDK Libraries from the Package Repository����������������������������������������������������� 398
Installing PMDK on Microsoft Windows������������������������������������������������������������������������������������ 402

Appendix C: How to Install IPMCTL on Linux and Windows���������������������������������� 403


IPMCTL Linux Prerequisites������������������������������������������������������������������������������������������������������ 404
libsafec�������������������������������������������������������������������������������������������������������������������������������� 404
IPMCTL Linux Packages������������������������������������������������������������������������������������������������������������ 404
IPMCTL for Microsoft Windows������������������������������������������������������������������������������������������������� 404
Using ipmctl������������������������������������������������������������������������������������������������������������������������������ 405

Appendix D: Java for Persistent Memory������������������������������������������������������������� 411


Volatile Use of Persistent Memory�������������������������������������������������������������������������������������������� 411
Heap Allocation on Alternative Memory Devices����������������������������������������������������������������� 412
Persistent Collections for Java (PCJ)���������������������������������������������������������������������������������������� 416
Using PCJ in Java Applications������������������������������������������������������������������������������������������������� 417
Low-Level Persistent Library (LLPL)����������������������������������������������������������������������������������������� 418
Using LLPL in Java Applications������������������������������������������������������������������������������������������������ 419
Summary���������������������������������������������������������������������������������������������������������������������������������� 419

Appendix E: The Future of Remote Persistent Memory Replication��������������������� 421

Glossary���������������������������������������������������������������������������������������������������������������� 425

Index��������������������������������������������������������������������������������������������������������������������� 429

xii
About the Author
Steve Scargall is a persistent memory software and cloud architect at Intel
Corporation. As a technology evangelist, he supports the enabling and development
effort to integrate persistent memory technology into software stacks, applications,
and hardware architectures. This includes working with independent software
vendors (ISVs) on both proprietary and open source development, original equipment
manufacturers (OEMs), and cloud service providers (CSPs).

Steve holds a Bachelor of Science in computer science and cybernetics from the
University of Reading, UK, where he studied neural networks, AI, and robotics. He
has over 19 years’ experience providing performance analysis on x86 architecture and
SPARC for Solaris Kernel, ZFS, and UFS file system. He performed DTrace debugging in
enterprise and cloud environments during his tenures at Sun Microsystems and Oracle.

xiii
About the Technical Reviewer
Andy Rudoff is a principal engineer at Intel Corporation, focusing on non-volatile
memory programming. He is a contributor to the SNIA NVM Programming Technical
Work Group. His more than 30 years’ industry experience includes design and
development work in operating systems, file systems, networking, and fault management
at companies large and small, including Sun Microsystems and VMware. Andy has
taught various operating systems classes over the years and is a coauthor of the popular
UNIX Network Programming textbook.

xv
About the Contributors
Piotr Balcer is a software engineer at Intel Corporation with many years’ experience
working on storage-related technologies. He holds a Bachelor of Science in engineering
from the Gdańsk University of Technology, Poland, where he studied system software
engineering. Piotr has been working on the software ecosystem for next-generation
persistent memory since 2014.

Eduardo Berrocal joined Intel Corporation as a cloud software engineer in 2017 after
receiving his PhD in computer science from the Illinois Institute of Technology. His
doctoral research focused on data analytics and fault tolerance for high-performance
computing. Past experience includes working as an intern at Bell Labs (Nokia), a research
aid at Argonne National Laboratory, a scientific programmer and web developer at the
University of Chicago, and an intern in the CESVIMA laboratory in Spain.

Adam Borowski is a software engineer at Intel Corporation, hailing from the


University of Warsaw, Poland. He is a Debian developer and has made many open
source contributions over the past two decades. Adam is currently working on
persistent memory stacks, both on upstream code and integrating it with downstream
distributions.

Igor Chorazewicz is a software engineer at Intel Corporation. His main focus is on


persistent memory data structures and enabling C++ applications for persistent
memory. Igor holds a Bachelor of Science in engineering from the Gdańsk University of
Technology, Poland.

Adam Czapski is a technical writer at Intel Corporation. He writes technical


documentation in the Data Center Group and is currently working in the persistent
memory department. Adam holds a Bachelor of Arts in English philology and a master’s
degree in natural language processing from the Gdańsk University of Technology, Poland.

Steve Dohrmann is a software engineer at Intel Corporation. He has worked on a variety


of projects over the past 20 years, including media frameworks, mobile agent software,
secure collaboration software, and parallel programming language implementation. He
is currently working on enabling the use of persistent memory in Java*.

xvii
About the Contributors

Chet Douglas is a principal software engineer at Intel Corporation and focuses on cloud
software architecture along with operating system and OEM enabling of non-volatile
memory technologies. He has over 14 years’ experience working on various enterprise
and client programs and 28 years of total storage experience. Chet has worked in all
aspects of storage, including storage controller hardware design, SCSI disk/tape/CD
writer firmware architecture, storage management software architecture, Microsoft
Windows* and Linux kernel-mode drivers, enterprise hardware RAID, and client/
workstation software RAID. He holds seven storage-related hardware and software
patents and has a dual Bachelor of Science in electrical engineering and computer
engineering from Clarkson University, New York.

Ken Gibson is the director of persistent memory software architecture within Intel
Corporation’s Data Center Group. Since 2012, Ken and his team have been working with
Intel’s server and software partners to create the open persistent memory programming
model.

Tomasz Gromadzki is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other (networking) technologies as well as
optimal persistent memory replication procedures and algorithms.

Kishor Kharbas is a software engineer on the Java runtime engineering team at Intel
Corporation. For the past eight years, he has been working to optimize Oracle’s OpenJDK
on Intel platforms. This involves Java garbage collection and compiler back-end
optimization.

Jackson Marusarz is a senior technical consulting engineer (TCE) in Intel Corporation's


Compute Performance and Developer Products Division. As the lead TCE for Intel
VTune Profiler, his main focus is on software performance analysis and tuning for both
serial and multithreaded applications. Jackson’s time is split between determining how
to analyze and tune software and creating tools that help others do the same.

Jan Michalski is a software engineer in Intel Corporation’s Non-­Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other technologies, as well as looking for
optimal persistent memory replication procedures and algorithms. He holds a master's
degree in computer engineering from the Gdańsk University of Technology, Poland,
where he studied system software engineering.

xviii
About the Contributors

Nicholas Moulin is a cloud software architect at Intel Corporation. Since joining Intel
in 2012, he has focused on enabling and developing persistent memory software for
operating systems and platform firmware and managing persistent memory hardware.
Nicholas is currently working with industry partners to define and improve RAS features
relevant to the persistent memory programming model.

Szymon Romik is a software engineer at Intel Corporation and is currently focused on


persistent memory programming. He previously worked as a lead software engineer
on 5G technologies at Ericsson. Szymon holds a master’s degree in mathematics from
Jagiellonian University, Poland.

Jakub Schmiegel is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group where he has been focused on enabling existing applications to
persistent memory and analyzing their performance for more than four years. Jakub
holds a master’s degree in computer science from the Gdańsk University of Technology,
Poland.

Kevin Shalkowsky is a Telly Award–winning creative director, graphic designer, and


animator with more than a decade of experience. While his contributions are in
technology today, Kevin has spent time in broadcast journalism and selling numerous
products through 30-minute late-night infomercials. He resides in Oregon with his wife
and son. From time to time, you can find Kevin lost in the woods, lost in a parking lot,
or lost in his design process – but this somehow got him to where he is today, and he
wouldn’t have it any other way.

Vineet Singh is a memory and storage tools software engineer at Intel Corporation.
He develops techniques to help developers adapt to the latest memory technologies.
Vineet holds a PhD in philosophy from the University of California and has a Bachelor
of Technology degree from the Indian Institute of Information Technology, Design, and
Manufacturing in Jabalpur.

Pawel Skowron is a software engineering manager at Intel Corporation with 20 years'


experience in the software industry. Pawel has worked in various roles related to the
whole-software development life cycle. His software engineering background lies in the
areas of embedded systems, database systems, and applications. For the past few years,
Pawel has led the development and validation of the Persistent Memory Development
Kit (https://github.com/pmem/pmdk).

xix
About the Contributors

Usha Upadhyayula has been with Intel Corporation for 20 years serving in many
different roles. Usha holds a master’s degree in computer science from the University
of South Carolina, and she spent the first few years at Intel developing user-level
applications in C and C++. She later moved to customer-­enabling roles for Intel media
processors and support for Intel RAID software. Usha is currently part of the Data Center
Group where she is focused on enabling cloud service providers to fully utilize and
accelerate the adoption of Intel persistent memory products.

Sergey Vinogradov is a senior software development engineer at Intel Corporation


where he spent more than seven years working on performance profiling tools and
threading runtime libraries. During the past four years, Sergey has been working on C++
programming models and performance profiling methodologies for persistent memory.

xx
Acknowledgments
First and foremost, I would like to thank Ken Gibson for masterminding this book idea
and for gifting me the pleasure of writing and managing it. Your support, guidance, and
contributions have been instrumental in delivering a high-quality product.
If the Vulcan mind-meld or The Matrix Headjack were possible, I could have cloned
Andy Rudoff’s mind and allowed him to work on his daily activities. Instead, Andy’s
infinite knowledge of persistent memory had to be tapped through good old verbal
communication and e-mail. I sincerely thank you for devoting so much time to me and
this project. The results read for themselves.
Debbie Graham was instrumental in helping me manage this colossal project. Her
dedication and support helped drive the project to an on-time completion.
To my friends and colleagues at Intel who contributed content, supported
discussions, helped with decision-making, and reviewed drafts during the book-writing
process. These are the real heroes. Without your heavily invested time and support, this
book would have taken considerably longer to complete. It is a much better product as a
result of the collaborative effort. A huge thanks to all of you.
I'd like to express my sincerest gratitude and appreciation to the people at Apress,
without whom this book could not have been published. From the initial contact and
outline discussions through the entire publishing process to this final polished product,
the Apress team delivered continuous support and assistance. Many thanks to Susan,
Jessica, and Rita. It was a real pleasure working with you.

xxi
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Preface
About This Book
Persistent memory is often referred to as non-volatile memory (NVM) or storage
class memory (SCM). In this book, we purposefully use persistent memory as an all-­
encompassing term to represent all the current and future memory technologies that
fall under this umbrella. This book introduces the persistent memory technology and
provides answers to key questions. For software developers, those questions include:
What is persistent memory? How do I use it? What APIs and libraries are available?
What benefits can it provide for my application? What new programming methods do I
need to learn? How do I design applications to use persistent memory? Where can I find
information, documentation, and help?
System and cloud architects will be provided with answers to questions such as:
What is persistent memory? How does it work? How is it different than DRAM or SSD/
NVMe storage devices? What are the hardware and operating system requirements?
What applications need or could benefit from persistent memory? Can my existing
applications use persistent memory without being modified?
Persistent memory is not a plug-and-play technology for software applications.
Although it may look and feel like traditional DRAM memory, applications need to be
modified to fully utilize the persistence feature of persistent memory. That is not to say
that applications cannot run unmodified on systems with persistent memory installed,
they can, but they will not see the full potential of what persistent memory offers without
code modification.
Thankfully, server and operating system vendors collaborated very early in the
design phase and already have products available on the market. Linux and Microsoft
Windows already provide native support for persistent memory technologies. Many
popular virtualization technologies also support persistent memory.
For ISVs and the developer community at large, the journey is just beginning. Some
software has already been modified and is available on the market. However, it will
take time for the enterprise and cloud computing industries to adopt and make the
hardware available to the general marketplace. ISVs and software developers need time
to understand what changes to existing applications are required and implement them.
xxiii
Preface

To make the required development work easier, Intel developed and open sourced
the Persistent Memory Development Kit (PMDK) available from https://pmem.io/
pmdk/. We introduce the PMDK in more detail in Chapter 5 and walk through most of
the available libraries in subsequent chapters. Each chapter provides an in-depth guide
so developers can understand what library or libraries to use. PMDK is a set of open
source libraries and tools based on the Storage Networking Industry Association (SNIA)
NVM programming model designed and implemented by over 50 industry partners. The
latest NVM programming model document can be found at https://www.snia.org/
tech_activities/standards/curr_standards/npm. The model describes how software
can utilize persistent memory features and enables designers to develop APIs that take
advantage of NVM features and performance.
Available for both Linux and Windows, PMDK facilitates persistent memory
programming adoption with higher-level language support. C and C++ support is fully
validated. Support for other languages such as Java and Python is work in progress
at the time this book was written. Other languages are expected to also adopt the
programming model and provide native persistent memory APIs for developers. The
PMDK development team welcomes and encourages new contributions to core code,
new language bindings, or new storage engines for the persistent memory key-value
store called pmemkv.
This book assumes no prior knowledge of persistent memory hardware devices
or software development. The book layout allows you to freely navigate the content in
the order you want. It is not required to read all chapters in order, though we do build
upon concepts and knowledge described in previous chapters. In such cases, we make
backward and forward references to relevant chapters and sections so you can learn or
refresh your memory.

B
 ook Structure
This book has 19 chapters, each one focusing on a different topic. The book has three
main sections. Chapters 1-4 provide an introduction to persistent memory architecture,
hardware, and operating system support. Chapters 5-16 allow developers to understand
the PMDK libraries and how to use them in applications. Finally, Chapters 17-19 provide
information on advanced topics such as RAS and replication of data using RDMA.

xxiv
Preface

• Chapter 1. Introduction to Persistent Memory – Introduces persistent


memory and dips our toes in the water with a simple persistent key-­
value store example using libpmemkv.

• Chapter 2. Persistent Memory Architecture – Describes the persistent


memory architecture and focuses on the hardware requirements
developers should know.

• Chapter 3. Operating System Support for Persistent Memory –


Provides information relating to operating system changes, new
features, and how persistent memory is seen by the OS.

• Chapter 4. Fundamental Concepts of Persistent Memory


Programming – Builds on the first three chapters and describes the
fundamental concepts of persistent memory programming.

• Chapter 5. Introducing the Persistent Memory Development Kit


(PMDK) – Introduces the Persistent Memory Development Kit
(PMDK), a suite of libraries to assist software developers.

• Chapter 6. libpmem: Low-Level Persistent Memory Support –


Describes and shows how to use libpmem from the PMDK, a low-­level
library providing persistent memory support.

• Chapter 7. libpmemobj: A Native Transactional Object Store –


Provides information and examples using libpmemobj, a C native
object store library from the PMDK.
• Chapter 8. libpmemobj-cpp: The Adaptable Language - C++ and
Persistent Memory – Demonstrates the C++ libpmemobj-cpp object
store from the PMDK, built using C++ headers on top of libpmemobj.

• Chapter 9. pmemkv: A Persistent In-Memory Key-Value Store –


Expands upon the introduction to libpmemkv from Chapter 1 with a
more in-depth discussion using examples.

• Chapter 10. Volatile Use of Persistent Memory – This chapter is


for those who want to take advantage of persistent memory but
do not require data to be stored persistently. libmemkind is a user-
extensible heap manager built on top of jemalloc which enables
control of memory characteristics and a partitioning of the heap

xxv
Preface

between different kinds of memory, including persistent memory.


libvmemcache is an embeddable and lightweight in-memory caching
solution. It is designed to fully take advantage of large-capacity
memory, such as persistent memory with DAX, through memory
mapping in an efficient and scalable way.

• Chapter 11. Designing Data Structures for Persistent Memory –


Provides a wealth of information for designing data structures for
persistent memory.
• Chapter 12. Debugging Persistent Memory Applications – Introduces
tools and walks through several examples for how software developers
can debug persistent memory–enabled applications.

• Chapter 13. Enabling Persistence using a Real-World Application –


Discusses how a real-world application was modified to enable
persistent memory features.

• Chapter 14. Concurrency and Persistent Memory – Describes how


concurrency in applications should be implemented for use with
persistent memory.

• Chapter 15. Profiling and Performance – Teaches performance


concepts and demonstrates how to use the Intel VTune suite of tools
to profile systems and applications before and after code changes are
made.

• Chapter 16. PMDK Internals: Important Algorithms and Data


Structures – Takes us on a deep dive of the PMDK design, architecture,
algorithms, and memory allocator implementation.

• Chapter 17. Reliability, Availability, and Serviceability (RAS) –


Describes the implementation of reliability, availability, and
serviceability (RAS) with the hardware and operating system layers.

• Chapter 18. Remote Persistent Memory – Discusses how applications


can scale out across multiple systems using local and remote persistent
memory.

• Chapter 19. Advanced Topics – Describes things such as NUMA, using


software volume managers, and the mmap() MAP_SYNC flag.

xxvi
Preface

The Appendixes have separate procedures for installing the PMDK and utilities
required for managing persistent memory. We also included an update for Java and the
future of the RDMA protocols. All of this content is considered temporal, so we did not
want to include it in the main body of the book.

Intended Audience
This book has been written for experienced application developers in mind. We
intend the content to be useful to a wider readership such as system administrators
and architects, students, lecturers, and academic research fellows to name but a few.
System designers, kernel developers, and anyone with a vested or passing interest in this
emerging technology will find something useful within this book.
Every reader will learn what persistent memory is, how it works, and how operating
systems and applications can utilize it. Provisioning and managing persistent memory
are vendor specific, so we include some resources in the Appendix sections to avoid
overcomplicating the main chapter content.
Application developers will learn, by example, how to integrate persistent memory
in to existing or new applications. We use examples extensively throughout this book
using a variety of libraries available within the Persistent Memory Development Kit
(PMDK). Example code is provided in a variety of programming languages such as C,
C++, JavaScript, and others. We want developers to feel comfortable using these libraries
in their own projects. The book provides extensive links to resources where you can find
help and information.
System administrators and architects of Cloud, high-performance computing,
and enterprise environments can use most of the content of this book to
understand persistent memory features and benefits to support applications and
developers. Imagine being able to deploy more virtual machines per physical server or
provide applications with this new memory/storage tier such that they can keep more
data closer to the CPU or restart in a fraction of the time they could before while keeping
a warm cache of data.
Students, lecturers, and academic research fellows will also benefit from many
chapters within this book. Computer science classes can learn about the hardware,
operating system features, and programming techniques. Lecturers are free use the
content in student classes or to form the basis of research projects such as new persistent
memory file systems, algorithms, or caching implementations.

xxvii
Random documents with unrelated
content Scribd suggests to you:
“Well, good-night to you”—and the other man, speaking
over his shoulder, said in an easy, cultivated voice
without any accent at all:

“Pleasant dreams, Miss Renata.”

Jane’s “Good-night” was just audible and no more, but


obviously it satisfied the two men, for the door was
shut, the key turned and withdrawn, and presently the
hall light went out, and the darkness was absolute and
unrelieved, except where the midnight sky showed just
less black than the interior of the room.

After what seemed a long, long time, Jane undressed 49


and got to bed. It was strange to grope for and find
Renata’s neatly folded nightdress.

Presently she lay down, and presently she slept. Time


ceased; the day was over.

She woke suddenly a few hours later. It was still dark.


She came broad awake at once, and sat up in bed as if
some one had called to her. Her mind was full of one
horrifying thought.

The plank—what had Arnold done with the plank?

Impossible that he should have helped Renata down the


fire-escape and carried the plank as well, and somehow
Jane did not see Arnold troubling to come back for it.

One thing was certain; if Arnold had left the plank in its
compromising position, it must be removed before
daylight.

Jane got out of bed, shivering. She went to the window,


opened it, and leaned out. The yard, mews, wall, and
parapet—all were veiled in the same thick dusk. She
strained her eyes, but it was impossible to distinguish
anything. There was nothing for it but to cross that
horrid little hall again, open the window, and make sure.

With the key in her hand, and mingled rage and terror
in her heart, she felt her way to the door, opened it
noiselessly, and crossed barefoot to the window. The
hasp was stiff, it creaked, and the window stuck.

Recklessness took possession of Jane. With a jerk she


pushed it up; as it chanced, recklessness made less
noise than caution would have done. She leaned right
out, and there, sure enough, was the plank.

Even Jane’s anger could provide her with nothing more 50


cutting than, “How exactly like Arnold Todhunter.”

She stood quite still and considered.

A bold course was the only one. Remembering the


plank’s previous fall and the perfect calm with which the
neighbourhood had received it, she decided to take the
same chance again—only, she must be quick and have it
all planned in her head: first a shove to the plank, then
down with the window and latch it, five steps—no, six—
across the hall, and then her own door, and on no
account must she forget the key.

She drew a long breath, leaned out, and pushed. The


board was heavier than she had supposed—harder to
move. She had to pull it in, until the sudden weight and
strain told her that it was clear of the coping upon
which the farther end had rested. Then she pushed with
all her might, and as it fell, her hands were on the
window quick and steady. Next moment she was
crouching in Renata’s bed, the clothes clutched about
her, the door key cold in her palm. She pushed it far
down beneath the clothes, and sat breathless—listening.

The crash with which the plank had landed seemed to


have deafened her, but as the vibrations died away, she
heard, sharp and unmistakable, the click of a latch and
hurrying footsteps.

The next moment her door was opened and her light
switched on. Quick as thought her hand was over her
eyes and the sheet up to her chin.

Molloy stood in the doorway, and beyond him the other.

“What’s doing? Did you hear it?” he stammered, and


then the other man pushed him aside.

“I’d like a look from your window if you’ll excuse me, 51


Miss Renata,” he said, and crossed the room.

As he leaned out, Jane watched him from beneath her


hand, and recalled Renata’s words, “He generally wears
a fur coat; they call him Number Two.” This man wore a
fur coat over pale blue silk pyjamas. When he turned,
saying, “I can’t see a thing,” she was ready with her
stammered, “What was it?”

“You heard it, then?” said Molloy.

“Such a fearful crash! It—it frightened me most


dreadfully,”—and here Jane spoke the literal truth.

“I don’t know.” It was Molloy who answered again, but


the other man’s eyes travelled round the room, and a
feeling of terror came over Jane.
If she had forgotten anything, if there were one shred
of incriminating evidence, those eyes would miss
nothing! She felt as if they must pierce the bedclothes
and see her bag and the hidden key, but he merely
nodded to Molloy, and they left the room, switching out
the light and locking the door.

Jane drew a long breath of relief, turned upon her side,


and in five minutes was asleep again.

The day came in with a thick mist. Jane opened her


eyes upon it sleepily.

She began to think what a strange dream she had had,


and then, as sleep ebbed from her, she remembered
that it was not a dream at all. She was Renata Molloy
under lock and key, and in front of her stretched a day
that might be even more crowded with adventure than
yesterday.

She jumped out of bed, and as she dressed her eyes 52


brightened and her courage rose. With Renata’s scissors
she unpicked the initials which marked her
underclothes. This was a game at which one must not
make a single slip. Her bag worried her a little, but it
was just such a plain leather bag as any one might
possess. She ransacked it carefully, and frowned over an
envelope addressed to Miss Jane Smith. What in the
world was she to do with it?

There were no matches, so it could not be burned. After


some thought she soaked it in water, scratched the
name to shreds with a hairpin, and crumpling the wet
paper into a ball, tossed it out of the window.

By the time her door was unlocked, she was very


hungry. This time, it appeared, she was being
summoned to bid the departing Mr. Molloy a fond
farewell.

His luggage was already being carried out to the lift,


and two or three men were coming and going. The man
in the fur coat stood with his back to the window,
smoking a cigarette. Obviously Molloy’s farewell was not
to be said in private.

Jane looked at him with some curiosity—a tall man,


strongly built, with a bold air and a florid complexion.

It was he who had opened the door, and he stood still


holding the handle and looking, not at Jane, but over
her shoulder. For this she felt grateful.

“Well, well then, I’m off,” said Molloy. “You’ll be a good


girl and do as you’re bid, and I’ll be having you out to
keep house for me in less than no time.”

From what she had seen of Renata, Jane fancied that a


sob would meet the occasion. She therefore sobbed,
and pressed her handkerchief to her eyes.

“There, there,” said Molloy hastily.

He bent and deposited an awkward kiss upon the top of


her head. Then he took his hand from the door and was
gone.

The lift gate clanged, and Jane realised that the real 53
adventure had begun.

The man by the window threw the end of his cigarette


into the fireplace and came towards her.
“Parental devotion is a beautiful thing, isn’t it, Miss
Renata? Suppose we have some breakfast.”

A meal, a proper meal, enough to eat! As she passed


into the dining-room and beheld a ham, coffee, and
boiled eggs, Jane felt as if she could confront any one
or anything. Besides, the first trick was hers.

In the full light of day, and under those cold, pale eyes,
she had passed as Renata.

She allowed herself to sigh and dab her eyes, and then
—oh, how good was the rather stale bread, the London
egg, and the indifferent ham.

The man watched her quizzically.

As she finished her second cup of coffee, he remarked


that she had a good appetite, and there was something
in his tone that cast a chill upon the proceedings.

Jane pushed back her chair.

“I’ve finished,” she said.

“Well, then,” said the man, “I think we must talk. Yes,


sit down again, please. I won’t keep you very long.”

Jane did as she was told.

“Well, Molloy’s gone,” he said. “You know what that 54


means? He’s washed his hands of you. Just in case—
just in case, you’ve been relying on Molloy, I would like
to point out to you that his own position is none too
secure. The firm he works for has not been entirely
satisfied with him for some time. It is, therefore, quite
out of the question that he should influence any
decision that may be come to with regard to yourself.
His going off like this shows that he realises the position
and accepts it. Self-preservation is Molloy’s trump suit,
first, last, and all the time. I shouldn’t advise you to
count upon trifles like parental devotion, or anything of
that sort. In a word—he can’t help you, but I can.”

The man leaned forward as he spoke, and a sudden


smile changed his features.

“Just be frank,” he went on. “Tell me what you really


heard, and I’ll see you through.”

Jane let her eyes meet his. That smile had puzzled her;
it was so spontaneous and charming, but it did not
reach his eyes.

She looked and found them cold and opaque, and as


she looked, she saw the pupils narrow, expand, and
then narrow again.

He got up from his chair, walked to the mantelpiece,


stopped for a light to his cigarette, and came back again
with a thin blue haze of smoke about him.

“Perhaps I haven’t been altogether frank with you,” he


said. “That little romance of mine about a firm of
chemists who employ your father—you didn’t really
believe it? No, I thought not. The fact is, that first night
I took you for just a schoolgirl, and one can’t tell
schoolgirls everything. But now, now I’m talking to you
as a woman. I can’t tell you everything, even so, but I
can tell you this. It’s a Government matter, a most
important one, and it is vital that I should know just
what you overheard.”

Jane looked down.


“I don’t understand,” she said in a low voice. “I was 55
dreaming and I waked up suddenly. There was a screen
in front of me, and some one on the other side of the
screen called out very loud, ‘The door, the door!’ That’s
what I heard.”

She felt the pale eyes upon her face. Then with an
abrupt movement the man came over to her.

“Stand up,” he said.

Jane stood up.

“Look at me.”

Jane looked at him.

After what seemed like a very long time, he threw out


his hand with an impatient gesture. It struck the table
edge with a sharp rap, the spring that held his wrist
watch gave, and the watch on its gold curb flew off and
fell on the floor behind Jane.

She turned, glad of an excuse to turn, and bent to pick


it up. The back of the watch was open; her fingers
caught and closed it instantly, but not for nothing had
she told Henry that she had gimlet eyes. The back of
the watch contained a photograph, and Jane had seen
the photograph before. Henry’s voice sounded in her
ears. “It was done from Amory’s portrait of her, in 1915
—the year of her marriage.”

Number Two, the man in the fur coat, Renata’s “worst


of them all,” had in the back of his watch a photograph
of Lady Heritage!
Jane laid the watch on the table without giving it a
second glance.

56
CHAPTER VI

As the watch slid back into its place beneath his shirt
cuff, the man spoke with an entire change of manner.

“Well, Miss Renata, that was all very stiff and


businesslike. You mustn’t hold it up against me, because
I hope we’re going to be friends. Don’t you want to
know your plans?”

Jane looked at him with a little frown.

“My plans?”

“What is going to happen to you. Oh, please, don’t look


so grave! It’s nothing very dreadful. You have heard of
Sir William Carr-Magnus?”

“Yes, of course,” said Jane. She hoped that she looked


innocent and surprised.

“Well,” said the man in the fur coat, “I happen to be his


secretary, and that reminds me, I don’t believe you
know my name. Your father and his friends use a
ridiculous nickname which sticks to me like a burr ... but
let me introduce myself—Jeffrey Ember, and your friend,
if you will have me.”
The charming smile just touched his face, and then he
said in a quiet, serious way:

“Sir William’s daughter, Lady Heritage, has 57


commissioned me to find her an amanuensis—
companion—no, that’s not quite right either. She doesn’t
want a trained stenographer, or a young person with a
business training, but she wants a girl in the house—
some one who’ll do what she’s told, write notes, arrange
the flowers.... I dare say you can guess the sort of
thing. She is willing to give you a trial, and your father
has agreed. As a matter of fact, I’m taking you down
there to-day.”

“Oh!” said Jane, because she seemed expected to say


something, and for the life of her she could not think of
anything else to say.

“I’m afraid you’ll have to submit to certain restrictions at


Luttrell Marches. You see, Sir William is engaged upon
some very important experiments for the Government,
and all the members of his household have to conform
to certain regulations. Their letters must be censored,
and they must not leave the grounds, which are,
however, extremely delightful and extensive. It isn’t
much of a hardship, really.”

“Oh no,” said Jane in her best schoolgirl manner.

And there the interview ended.

They made the journey to Luttrell Marches by car, but,


after the manner of Mrs. Gilpin’s post-chaise, it did not
pick them up at the door. An ordinary taxi conveyed
them to Victoria Station, and it was in the station yard
that they and their luggage were picked up by the Rolls-
Royce with the Carr-Magnus crest upon the door.
The mist was thinner, and as they came clear of
London, the sun came out. The day warmed into
beauty, and the green growth of the countryside
seemed to be expanding before their eyes. So many
long hedges running into a blur, so many miles of road
all slipping past. Jane fell fast asleep, and did not know
how long she slept.

It was in the late afternoon that they came into the 58


Marsh country—great flat stretches of it, set with boggy
tussocks and intersected by straight lanes of water.
Purple-brown and green it stretched for miles. To the
right a humped line of upland, but to the left, and as far
as the eye could see in front, nothing but marsh. Then
the road rose a little; the ground was firmer and carried
a black pine or two.

They came to a three-cross way and turned sharply to


the right. The ground rose more and more. They
climbed a steep hill, zigzagging between banked-up
hedges to make the rise, and came out upon a bare
upland. Ahead of them one saw a high stone wall
pierced by iron gates. The car stopped. Mr. Ember
leaned out, and after a pause the gates swung inwards.

For a mile the drive lay through a flat waste of springing


bracken, with here and there a group of wind-driven
trees, then a second gate through a high fencing topped
with wire. An avenue of trees led up to the house, a
huge grey pile set against a sky full of little racing
clouds.

Jane felt stiff and bewildered with the long drive. She
followed Mr. Ember up a flight of granite steps and
came into the great hall of Luttrell Marches with its
panelled walls and dark old portraits of half-forgotten
Luttrells.

Exactly opposite the entrance rose the stairway which


was the pride of the house. Its beautiful proportions,
the grapes and vine leaves of its famous carvings, were
lighted from beneath by the red glow of a huge open
fire, and from above by the last word in electric lighting.

Ember walked straight across the hall and up the stair,


and Jane followed him.

She thought she knew exactly how a puppy must feel 59


when, blinking from the warmth and straw of his
basket, he comes for the first time into the ordered
solemnity of his new master’s house.

And then she looked up and saw The Portrait.

It hung on the panelling at the top of the stair where


the long corridors ran off to right and left, and it took
Jane’s breath away—the portrait of Lady Heritage.

Amory had painted more than a beautiful woman


standing on a marble terrace: he had painted a woman
Mercury. The hands held an ivory rod—diamond wings
rose from the cloudy hair. Under the bright wings the
eyes looked out, looked far—dark, splendid, hungry
eyes.

“The earth belongs to her, and she despises it,” was


Jane’s thought.

She stood staring at the portrait. Nineteen-fifteen,


Henry had said—the year when other women posed
with folded linen hiding their hair and the red cross
worn like a blazon. She could think of several famous
beauties who had been painted thus. But this woman
wore her diamond wings, though, even as she wore
them, Fate had done its worst to her, for Anthony
Luttrell was a name with other names in a list of
missing, and no man knew his grave.

A sharp clang of metal upon metal startled Jane. She


looked quickly to her right, and saw that a steel gate
completely barred the entrance to the corridor on that
side. It had just closed behind a curious white-draped
figure.

“Ah, Jeffrey,” said a voice—a deep, rather husky voice—


and the figure came forward.

Jane saw that it was a woman wearing a long white 60


linen overall, and a curious linen head-dress, which she
was undoing and pushing back as she walked. She
pulled it off as she came up to them, saying, “It’s so hot
in there I can hardly breathe, but too fascinating to
leave. You’re early. Is this Miss Molloy?”

She put out her hand to Jane, and Jane, with her mind
full of the portrait, looked open-eyed at its original.

Afterwards she tried to formulate her sensations, but, at


the time, she received just that emotional shock which
most people experienced when they first met Raymond
Heritage.

Beautiful—but there are so many beautiful women.


Charming? No, there was rather something that
repelled, antagonised. In her presence Jane felt untidy,
shabby, gauche.

Lady Heritage unbuttoned her overall and slipped it off.


She wore a plain white knitted skirt and jersey. Her
fingers were bare even of the wedding ring which Jane
looked for and missed. Her black hair was a little ruffled,
and above the temples, where Amory had painted
diamond wings, there were streaks of grey.

Bewilderment came down on Jane like a thick mist,


which clung about her during the brief interchange of
sentences which followed, and went with her to her
room.

It was a queer room with a rounded wall set with three


windows and to right and left irregular of line, with a
jutting corner here and a blunted angle there. It faced
west, for the sun shone level in her eyes.

Crossing to the window, as most people do when they 61


come into a strange room, she looked out and caught
her breath with amazement.

The sea—why, it seemed to lie just beneath the


windows!

They had driven up from the landward side, and this


was her first hint that the sea was so near.

There was a wide gravel terrace, a stone wall set with


formal urns full of blue hyacinths, the sharp fall of the
cliff, and then the sea.

The tide was in, the sun low, and a wide golden path
seemed to stretch almost from Jane’s feet to the far
horizon. Overhead the little racing clouds that told of a
wind high up were golden too.

The humped ridge of upland, which Jane had seen as


they drove, ran out to sea on the right hand. It ended in
low, broken cliff, and a line of jagged rocks of which
only the points stood clear.

Jane turned from all the beauty outside to the ordered


comfort within. Hot water in a brass can that she could
see her face in, a towel of such fine linen that it was a
joy to touch it, this pretty white-panelled room, the
chintzes where bright butterflies hovered over roses and
sweet-peas—she stood and looked at it all, and she
heard Renata’s words, “At Luttrell Marches they will
decide whether I am to be eliminated.”

This curious dual sense remained with her during the


days that followed. Life at Luttrell Marches was simple
and regular. She wrote letters, gathered flowers,
unpacked the library books, and kept out of Sir William’s
way.

Sir William, she decided, was exactly like his 62


photograph, only a good deal more so; his eyebrows
more tufted, his chin more jutting, and his eyes harder.
For a philanthropist he had a singularly bad temper, and
for so eminent a scientist a very frivolous taste in
literature. One of Jane’s duties was to provide him with
novels. She ransacked library lists and trembled over the
results of her labours.

Sir William did not always join the ladies after dinner,
but when he did so he would read a novel at a sitting
and ask for more.

Mr. Ember was never absent, and when Lady Heritage


talked, it was to him that her words were addressed.
Sometimes she would disappear inside the steel gate for
hours.
Jane soon learnt that the whole of the north wing was
given up to Sir William’s experiments. On each floor a
steel gate shut it off from the rest of the house. All the
windows were barred from top to bottom.

She also discovered that the high paling where the


avenue began had, on its inner side, an apron of barbed
wire, and it was the upper strand of this apron which
she had seen as they approached from outside.

Sir William’s experiments employed a considerable


number of men. These, she learned, were lodged in the
stables, and neither they nor any of the domestic staff
were permitted to pass beyond the inner paling.

On the coast side there was a high wire entanglement—


electrified.

There were moments when Jane was cold with fear, and
moments when she told herself that Renata was a little
fool who had had nightmare.

63
CHAPTER VII

When Jane stood at her window and looked across the


sea, she saw what might have been a picture of life at
Luttrell Marches during those first few days. Such a
smooth stretch of water, pleasant to the eye, where
blue and green, amethyst, grey and silver came and
went, and under the play of colour and the shifting light
and shade of day and evening, the unchanging black of
rocks which showed for an instant and then left one
guessing whether anything had really broken the beauty
and the peace.

Over the surface all was pleasant enough, but incidents,


some of them almost negligible in themselves, kept
recurring to remind Jane that there were rocks beneath
the sea.

The first incident came up suddenly whilst she was


writing Lady Heritage’s letters on the second day.

She had beside her a little pile of correspondence,


mostly about trifles. Upon each letter there was
scrawled, “Yes”—“No”—“Tell them I’ll think it over,” or
some such direction.

Presently Jane arrived at a letter in French, upon which


Lady Heritage had written, “Make an English translation
and enclose to Mrs. Blunt.” Mrs. Blunt’s own letter lay
immediately underneath. It contained inquiries about
some conditions of factory labour amongst women in
France.

The French letter was an excellent exposition of the said 64


conditions.

Jane sat looking at it, and wondering whether Renata


could have translated a single line of it, and how much
ignorance it would behove her to display.

After a moment’s thought she turned round and said


timidly, “May I have a dictionary, please?”

Lady Heritage looked up from the papers before her.


She frowned and said:

“A dictionary?”

“Yes, for the French letter.”

“You don’t know French, then?”

Jane met the half-sarcastic look with protest.

“Oh yes, I do. But, if I might have a dictionary——”

Lady Heritage pointed to the bookcase and went back


to her papers.

An imp of mischief entered into Jane.

She took the dictionary and spent the next half-hour in


producing a translation with just the right amount of
faults in it. She put it down in front of her employer with
a feeling of triumph.
“Please, will this do?”

Lady Heritage looked, frowned, and tore the paper


across.

“I thought you said you knew French?”

Jane fidgeted with her pen:

“Of course I know I’m not really good at it, but I looked
out all the words I didn’t know.”

“There must have been a good many,” was Lady


Heritage’s comment, and the imp made Jane raise
innocent eyes and say:

“Oh, there were!”

She went back to her table, and Lady Heritage spoke 65


over her shoulder to Mr. Ember, who appeared to be
searching for a book at the far end of the room. She
spoke in French—the low, rapid French of the woman to
whom one language is the same as another.

“What do they teach at English schools, can you tell me,


Jeffrey? This girl says she knows French, and if she can
follow one word I am saying now——” She broke off
and shrugged. “Yet I dare say she went to an expensive
school. Now, I had a Bavarian maid, educated in the
ordinary village school, and she spoke English with
ease, and French better than any English schoolgirl I’ve
come across. Wait whilst I try her in something else.”

She turned back to Jane.

“Just send the original to Mrs. Blunt—I haven’t time to


bother with it—and make a note for me. I want it
inserted after para three on the second page of that
typewritten article that came back this morning.”

Jane supposed she might be allowed to know what a


“para” was. She turned over the leaves of the typescript
and waited for the dictation. The last sentence read,
“Woman through all the ages is at the disposal and
under the autocratic rule of man, but it is not of her
own volition.”

She wondered what was to come next, and waited,


keenly on the alert.

Lady Heritage began to speak:

“Write it in as neatly as possible, please; it’s only one


sentence: ‘It is Man who has forced “das ewig
Weibliche” upon us.’”

Jane wrote, “It is man——” and then stopped. She


repeated the words aloud and looked expectant.

“‘Das ewig Weibliche’”—there was a slight grimness in 66


Lady Heritage’s tone.

“I’m afraid—” faltered Jane.

“Never heard the quotation?”

“I’m so sorry.”

“You don’t know any German, then?”

“I’m so sorry,” said Jane.

“My dear girl, what did they teach you at that school of
yours? By the way, where was it?”
“At Ilfracombe.”

“English education is a disgrace,” said Lady Heritage,


and went back to her papers.

It was next day that she turned suddenly to Jane:

“By the way, you were at school at Ilfracombe—can you


give me the name of a china shop there? I want some
of that blue Devonshire pottery for a girls’ club I’m
interested in.”

Jane had a moment of panic. Renata’s shoes had fitted


her too easily. She had felt secure, and then to have her
security shattered by a trifle like this!

“A china shop?” she said meditatively; then, after a


pause, “It’s awfully stupid of me—I’m afraid I’ve
forgotten the name.”

Lady Heritage stared.

“A shop that you must have passed hundreds of times?”

“It’s very stupid of me.”

Lady Heritage smiled with a sudden brilliance. “Well, it


is rather,” she said.

It was on the fourth day that Jane really caught her first
glimpse of the black rocks.

She was writing in the library, dealing with an 67


apparently endless stream of begging letters, requests
for interviews, invitations to speak at meetings or to join
committees.
In four days Jane had discovered that Lady Heritage
was up to her eyes in a dozen movements relating to
feminist activities, women’s labour, and social reform.

Newspapers, pamphlets, and reports littered a table


which ran the whole length of the room. Jane was
required to open all these as they came, and separate
those which dealt with social reform and the
innumerable scientific treatises and reviews. These
latter arrived in every European language.

Jane sat writing. The day was clear and lovely, the air
sun-warmed and yet fresh as if it had passed over
snow. April has days like this, and they fill every healthy
person with a longing to be out, to stop working, and
take holiday.

The windows of the library looked out upon the gravel


terrace above the sea. The sun was on the blue water.

Jane put down her pen and looked at the hyacinths in


the grey stone urns. They were blue too. A yellow
butterfly played round them. She sat up and went to the
window.

Lady Heritage and Mr. Ember were walking up and


down the terrace, Lady Heritage bareheaded, all in
white with not even a scarf, and Jeffrey Ember with a
muffler round his neck, and the inevitable fur coat. They
were coming towards her, and Jane stood back so that
the curtains made a screen. She watched Raymond
Heritage as she had watched the sea and the flowers,
for sheer joy in her beauty.

Raymond’s face was towards her, and she was speaking.


Not a word reached Jane’s ears, but as she looked at 68
those beautiful lips, their movements spelt words to her
—words and sentences. She would have drawn back or
looked away, but the first sentence that she read riveted
her attention too closely.

“Are you satisfied about her Jeffrey?”

Ember must have spoken, but his head was turned


away. Then Raymond spoke again.

“Nor am I—not entirely. She seems intelligent and


unintelligent by turns, unbelievably stupid in one
direction and quick in another.” They passed level with
the window, and so on to the end of the terrace. Jane
went round the table to the other side of the window
and waited for them to come back.

Ember’s face was towards her when they turned, too far
away for her to see anything. But, as they came nearer,
she saw that he was speaking. Not easy to read from,
however, with those straight, thin lips that moved so
little. There was only one word she was sure of
—“overheard.”

It was too tantalising. If she had to wait until they


reached the far end of the terrace and turned again,
what might she not miss?

As the thought passed through her mind Lady Heritage


stopped, walked slowly to the grey stone wall, and sat
down on it, motioning to Ember to do the same.

Jane could see both faces now, and Raymond was


saying, “If she overheard anything, would she have the
intelligence to be dangerous?—that is what I ask
myself.”
Ember’s lips just moved, but the movements made no
sense.

“Perhaps you’re right,” said Lady Heritage; “despise not 69


thine enemy.”

She changed her position, leaned forward, displaying a


statuesque profile, and appeared to be speaking fast
and earnestly. Then Jane saw her lips again, and they
were saying, “Anything but Formula ‘A.’”

Jane gripped the curtain which she held until the gold
galon which bordered it marked her hand with its acorn
pattern.

“Formula ‘A’!” everything swam round her while she


heard Renata’s gasping voice:

“He said ‘With Formula “A” you have the key. When
Formula “B” is also complete, you will have the lock for
that key to fit; then the treasures of the world are
yours.’”

The mist cleared from her eyes; she looked again.

Raymond Heritage had risen to her feet. Ember and she


looked out to sea for a moment, then crossed the gravel
towards the house. They were talking of the sunshine
and the spring air.

“My bulbs have done well,” Lady Heritage said.

They passed out of sight.

Two days later Jane, coming down the corridor to the


library, was aware of voices in conversation. She opened
the door and saw Jeffrey Ember with his back to her. He
had pulled a deep leather chair close to the fire, and
was bending forward to warm his hands. Lady Heritage
stood a yard or two away. She had a large bunch of
violets in one hand; with the other she leaned against
the black marble mantel.

She and Ember were talking in German. Both glanced


round, and Raymond asked:

“What is it?”

“The letters for the post,” said Jane. 70

They went on talking whilst she sorted and stamped the


letters.

“Which of us is the better judge of character, it comes to


that.” Speaking German, Lady Heritage’s deep voice
sounded deeper than ever.

“Do we take different sides then?”

“I don’t know. I thought your verdict was inclined to be


‘Guilty, but recommended to mercy,’ whereas mine——”
She hesitated—stopped rather—for there was no
hesitation in her manner.

Ember made a gesture with the hand that held his


cigarette.

“Expound.”

“I doubt the guilt. But if I did not doubt, I should have


no mercy at all.”

Jane went out with the letters, and when she was in the
corridor again she put out her hand and leaned against
the wall. It would be horrible enough, she thought, to
be tried in an open court upon some capital count, but
how far less horrible than a secret judgment where
whispered words made unknown charges, where the
trial went on beneath the surface of one’s pleasant daily
life, and every word, every look, a turn of the head, an
unguarded sigh, a word too little, or a glance too much
might tip the scale and send the balance swinging down
to—what?

Next day Lady Heritage was deep in her


correspondence, when she suddenly flashed into anger.
Pushing back her chair, she got up and began to pace
the room. There was a letter in her hand, and as she
walked she tore it across and across, flung the
fragments into the fire, and pushed a blazing log down
upon them with her foot.

Jane and Ember watched her—the former with some 71


surprise and a good deal of admiration, the latter with
that odd something which her presence always called
out. She swung round, met his eyes, and burst into
speech.

“It’s Alington—to think that I ever called that man my


friend! I wonder if there’s a single man on this earth
who would translate professions of devotion to one
woman, into bare decent justice to all women.”

“What has Lord Alington done?” asked Mr. Ember, with a


slight drawl.

Jane, with a thrill, identified the President of the Board


of Trade.

“Nothing that I might not have expected. It is only


women that are different, Jeffrey. Men are all the same.”
“And still I don’t know what he has done,” said Jeffrey
Ember.

“Oh, it’s a long story! I’ve been pressing for women


inspectors in various directions. It seems inconceivable
that any one should cavil at a woman inspector
wherever women are employed. You have no idea of
what some of the conditions are. Stewardesses, for
instance; I’ve a letter there from a woman who has
been working on one of the largest liners—not a tramp
steamer, mind you, but one of the biggest liners afloat.
All the passengers’ trays, all the cabin meals had to be
carried up a perpendicular iron stair like a fire-escape—
not a permanent stair, you understand, but a ladder that
is let up and down. Those wretched women had to go
up and down it all day with heavy trays. They said they
couldn’t do it, and were told they had to. And that’s a
little thing compared to some of the other conditions. I
want an inspector for them.”

“And Alington?” 72

Lady Heritage came to a halt by the long, piled-up


table. She struck it with her open hand. “Lord Alington
is just a man,” she said. “He stands for what men have
always stood for, the sacred right of the vested interest.
What man ever wants to alter anything? And why
should he when the existing order gives him all he
wants? It doesn’t matter where you turn, what you do,
how hard you try, the vested interest blocks the way;
you are up against the Established Order of what has
always been. My God, how I’d like to smash it all, the
whole thing, the whole smug sham which we call
civilisation!”
Jane stared at her open-eyed. She had never dreamed
that the statue could wake into such vivid life as this.
The colour burned in Raymond’s cheeks, the sombre
eyes were sombre still, but they held sparks as if from
inward fire.

Ember touched the hand that was clenched at the


table’s edge. A sort of tremor passed over her from
head to foot. The colour died, the fire was gone. With a
complete change of manner she said:

“Alington was hardly worth all that, was he?” Then


without a change of key, but in German:

“Thank you, Jeffrey, the child’s eyes were nearly falling


out of her head. It was stupid of me; I forgot. These
things carry me away.”

The door opened on her last words, and Sir William


came in. He was frowning, and appeared to be in a
great hurry.

“Ridiculous business, ridiculous waste of time. These 73


damned departments appear to think I’ve nothing to do
with my time except to answer their infernal inquiries,
and entertain any interfering jackanapes that they
choose to let loose on me.”

“What is it Father?” said Lady Heritage—“Government


inspection?”

“Nonsense,” said Sir William slowly. “Henry March wants


to come down for the night.”

Jane bent forward over her papers. No one was looking


at her, no one was thinking of her, but she had felt her
cheeks grow hot, and was glad of an excuse to hide
them.

She did not know whether she was very much afraid or
very glad. A feeling unfamiliar but overwhelming
seemed to shake her to the depths. She was quite
unconscious of what was passing behind her.

At Henry’s name, Raymond Heritage uttered a sharp,


“Oh no!” She came quickly forward as she spoke and
caught the letter from Sir William’s hand.

“He can’t come—I can’t have him here—put him off,


Father; you can make some excuse!”

“Nonsense!” said Sir William again. “It’s a nuisance, of


course—it’s an infernal nuisance—but he’ll have to
come, confound him!”

Then, as she made a half-articulate protest, he went on


with increasing loss of temper:

“Good heavens! I can’t very well tell the man I won’t


have him in what is practically his own house.”

It was Ember, not her father, who saw how frightfully


pale Raymond became. In a very low voice she said:

“No, I suppose not.”

Sir William was fidgeting. He looked at Jane’s back. 74

“Of course, he’s coming down on business.”

Then he broke off and stared at Jane again.

Lady Heritage nodded.

You might also like