100% found this document useful (2 votes)
42 views

Instant Download Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall PDF All Chapters

Comprehensive

Uploaded by

kutrpancer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (2 votes)
42 views

Instant Download Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall PDF All Chapters

Comprehensive

Uploaded by

kutrpancer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 65

Experience Seamless Full Ebook Downloads for Every Genre at textbookfull.

com

Programming Persistent Memory: A Comprehensive


Guide For Developers 1st Edition Steve Scargall

https://textbookfull.com/product/programming-persistent-
memory-a-comprehensive-guide-for-developers-1st-edition-
steve-scargall/

OR CLICK BUTTON

DOWNLOAD NOW

Explore and download more ebook at https://textbookfull.com


Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Camping Washington A Comprehensive Guide to Public Tent


and RV Campgrounds 3rd Edition Steve Giordano

https://textbookfull.com/product/camping-washington-a-comprehensive-
guide-to-public-tent-and-rv-campgrounds-3rd-edition-steve-giordano/

textboxfull.com

Scala Programming A comprehensive beginner s guide to


Scala 2nd Edition Claudia Alves

https://textbookfull.com/product/scala-programming-a-comprehensive-
beginner-s-guide-to-scala-2nd-edition-claudia-alves/

textboxfull.com

The Economics of Education: A Comprehensive Overview 2nd


Edition Steve Bradley (Editor)

https://textbookfull.com/product/the-economics-of-education-a-
comprehensive-overview-2nd-edition-steve-bradley-editor/

textboxfull.com

Quick Boot A Guide for Embedded Firmware Developers 2nd


Edition Pete Dice

https://textbookfull.com/product/quick-boot-a-guide-for-embedded-
firmware-developers-2nd-edition-pete-dice/

textboxfull.com
The Rust Programming Language 1st Edition Steve Klabnik

https://textbookfull.com/product/the-rust-programming-language-1st-
edition-steve-klabnik/

textboxfull.com

Building Generative AI Powered Apps A Hands on Guide for


Developers 1st Edition Kansal

https://textbookfull.com/product/building-generative-ai-powered-apps-
a-hands-on-guide-for-developers-1st-edition-kansal/

textboxfull.com

Reservoir Modelling A Practical Guide 1st Edition Steve


Cannon

https://textbookfull.com/product/reservoir-modelling-a-practical-
guide-1st-edition-steve-cannon/

textboxfull.com

Linux for Developers Jumpstart Your Linux Programming


Skills William "Bo" Rothwell

https://textbookfull.com/product/linux-for-developers-jumpstart-your-
linux-programming-skills-william-bo-rothwell/

textboxfull.com

Executive Functioning: A Comprehensive Guide for Clinical


Practice 1st Edition Yana Suchy

https://textbookfull.com/product/executive-functioning-a-
comprehensive-guide-for-clinical-practice-1st-edition-yana-suchy/

textboxfull.com
Programming
Persistent
Memory
A Comprehensive Guide for Developers

Steve Scargall
Programming Persistent
Memory
A Comprehensive Guide for
Developers

Steve Scargall
Programming Persistent Memory: A Comprehensive Guide for Developers

Steve Scargall
Santa Clara, CA, USA

ISBN-13 (pbk): 978-1-4842-4931-4 ISBN-13 (electronic): 978-1-4842-4932-1


https://doi.org/10.1007/978-1-4842-4932-1
Copyright © 2020 by Intel
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction
on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use,
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this book are included in the book’s Creative Commons license, unless
indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons
license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every
occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion
and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified
as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither
the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may
be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Susan McDermott
Development Editor: Laura Berendson
Coordinating Editor: Jessica Vakili
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected],
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer
Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail [email protected], or visit http://www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses
are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at
http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to readers on
GitHub via the book's product page, located at www.apress.com/978-1-4842-4931-4. For more detailed
information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii
About the Technical Reviewer���������������������������������������������������������������������������������xv
About the Contributors������������������������������������������������������������������������������������������xvii
Acknowledgments��������������������������������������������������������������������������������������������������xxi
Preface�����������������������������������������������������������������������������������������������������������������xxiii

Chapter 1: Introduction to Persistent Memory Programming���������������������������������� 1


A High-Level Example Program���������������������������������������������������������������������������������������������������� 2
What’s Different?��������������������������������������������������������������������������������������������������������������������� 5
The Performance Difference��������������������������������������������������������������������������������������������������� 6
Program Complexity���������������������������������������������������������������������������������������������������������������� 7
How Does libpmemkv Work?��������������������������������������������������������������������������������������������������� 8
What’s Next?��������������������������������������������������������������������������������������������������������������������������������� 9
Summary�������������������������������������������������������������������������������������������������������������������������������������� 9

Chapter 2: Persistent Memory Architecture����������������������������������������������������������� 11


Persistent Memory Characteristics��������������������������������������������������������������������������������������������� 12
Platform Support for Persistent Memory������������������������������������������������������������������������������������ 13
Cache Hierarchy�������������������������������������������������������������������������������������������������������������������������� 14
Power-Fail Protected Domains���������������������������������������������������������������������������������������������������� 16
The Need for Flushing, Ordering, and Fencing���������������������������������������������������������������������������� 19
Data Visibility������������������������������������������������������������������������������������������������������������������������������ 23
Intel Machine Instructions for Persistent Memory���������������������������������������������������������������������� 24
Detecting Platform Capabilities�������������������������������������������������������������������������������������������������� 25

iii
Table of Contents

Application Startup and Recovery����������������������������������������������������������������������������������������������� 27


What’s Next?������������������������������������������������������������������������������������������������������������������������������� 29
Summary������������������������������������������������������������������������������������������������������������������������������������ 29

Chapter 3: Operating System Support for Persistent Memory������������������������������� 31


Operating System Support for Memory and Storage������������������������������������������������������������������ 31
Persistent Memory As Block Storage������������������������������������������������������������������������������������������ 33
Persistent Memory-Aware File Systems������������������������������������������������������������������������������������� 34
Memory-Mapped Files���������������������������������������������������������������������������������������������������������������� 35
Persistent Memory Direct Access (DAX)������������������������������������������������������������������������������������� 43
Summary������������������������������������������������������������������������������������������������������������������������������������ 53

Chapter 4: Fundamental Concepts of Persistent Memory Programming��������������� 55


What’s Different?������������������������������������������������������������������������������������������������������������������������ 55
Atomic Updates��������������������������������������������������������������������������������������������������������������������������� 56
Transactions�������������������������������������������������������������������������������������������������������������������������������� 57
Atomicity������������������������������������������������������������������������������������������������������������������������������� 57
Consistency��������������������������������������������������������������������������������������������������������������������������� 58
Isolation��������������������������������������������������������������������������������������������������������������������������������� 58
Durability������������������������������������������������������������������������������������������������������������������������������� 58
Flushing Is Not Transactional������������������������������������������������������������������������������������������������������ 59
Start-Time Responsibilities��������������������������������������������������������������������������������������������������������� 59
Tuning for Hardware Configurations������������������������������������������������������������������������������������������� 60
Summary������������������������������������������������������������������������������������������������������������������������������������ 60

Chapter 5: Introducing the Persistent Memory Development Kit��������������������������� 63


Background��������������������������������������������������������������������������������������������������������������������������������� 63
Choosing the Right Semantics���������������������������������������������������������������������������������������������������� 64
Volatile Libraries������������������������������������������������������������������������������������������������������������������������� 65
libmemkind���������������������������������������������������������������������������������������������������������������������������� 65
libvmemcache����������������������������������������������������������������������������������������������������������������������� 66
libvmem��������������������������������������������������������������������������������������������������������������������������������� 67

iv
Table of Contents

Persistent Libraries��������������������������������������������������������������������������������������������������������������������� 67
libpmem�������������������������������������������������������������������������������������������������������������������������������� 67
libpmemobj���������������������������������������������������������������������������������������������������������������������������� 68
libpmemobj-cpp�������������������������������������������������������������������������������������������������������������������� 68
libpmemkv����������������������������������������������������������������������������������������������������������������������������� 69
libpmemlog���������������������������������������������������������������������������������������������������������������������������� 69
libpmemblk���������������������������������������������������������������������������������������������������������������������������� 69
Tools and Command Utilities������������������������������������������������������������������������������������������������������� 70
pmempool������������������������������������������������������������������������������������������������������������������������������ 70
pmemcheck��������������������������������������������������������������������������������������������������������������������������� 70
pmreorder������������������������������������������������������������������������������������������������������������������������������ 71
Summary������������������������������������������������������������������������������������������������������������������������������������ 71

Chapter 6: libpmem: Low-Level Persistent Memory Support��������������������������������� 73


Using the Library������������������������������������������������������������������������������������������������������������������������� 74
Mapping a File���������������������������������������������������������������������������������������������������������������������������� 75
Copying to Persistent Memory���������������������������������������������������������������������������������������������������� 76
Separating the Flush Steps��������������������������������������������������������������������������������������������������������� 77
Summary������������������������������������������������������������������������������������������������������������������������������������ 79

Chapter 7: libpmemobj: A Native Transactional Object Store��������������������������������� 81


What is libpmemobj?������������������������������������������������������������������������������������������������������������������ 81
Why not malloc( )?����������������������������������������������������������������������������������������������������������������������� 82
Grouping Operations������������������������������������������������������������������������������������������������������������������� 83
Memory Pools����������������������������������������������������������������������������������������������������������������������������� 83
Creating Memory Pools��������������������������������������������������������������������������������������������������������� 83
Pool Object Pointer (POP) and the Root Object���������������������������������������������������������������������� 87
Opening and Reading from Memory Pools���������������������������������������������������������������������������� 88
Memory Poolsets������������������������������������������������������������������������������������������������������������������������ 90
Concatenated Poolsets���������������������������������������������������������������������������������������������������������� 90
Replica Poolsets�������������������������������������������������������������������������������������������������������������������� 91
Managing Memory Pools and Poolsets��������������������������������������������������������������������������������������� 92

v
Table of Contents

Typed Object Identifiers (TOIDs)�������������������������������������������������������������������������������������������������� 92


Allocating Memory���������������������������������������������������������������������������������������������������������������������� 93
Persisting Data���������������������������������������������������������������������������������������������������������������������������� 94
Atomic Operations����������������������������������������������������������������������������������������������������������������� 94
Reserve/Publish API�������������������������������������������������������������������������������������������������������������� 97
Transactional API����������������������������������������������������������������������������������������������������������������� 100
Optional Flags���������������������������������������������������������������������������������������������������������������������� 104
Persisting Data Summary���������������������������������������������������������������������������������������������������� 104
Guarantees of libpmemobj’s APIs��������������������������������������������������������������������������������������������� 105
Managing Library Behavior������������������������������������������������������������������������������������������������������� 106
Debugging and Error Handling�������������������������������������������������������������������������������������������������� 106
Summary���������������������������������������������������������������������������������������������������������������������������������� 108

Chapter 8: libpmemobj-cpp: The Adaptable Language - C++ and


Persistent Memory����������������������������������������������������������������������������������������������� 111
Introduction������������������������������������������������������������������������������������������������������������������������������� 111
Metaprogramming to the Rescue��������������������������������������������������������������������������������������������� 112
Persistent Pointers�������������������������������������������������������������������������������������������������������������� 112
Transactions������������������������������������������������������������������������������������������������������������������������ 113
Snapshotting����������������������������������������������������������������������������������������������������������������������� 115
Allocating����������������������������������������������������������������������������������������������������������������������������� 116
C++ Standard limitations���������������������������������������������������������������������������������������������������������� 118
An Object’s Lifetime������������������������������������������������������������������������������������������������������������ 119
Trivial Types������������������������������������������������������������������������������������������������������������������������� 120
Object Layout����������������������������������������������������������������������������������������������������������������������� 122
Pointers������������������������������������������������������������������������������������������������������������������������������� 123
Limitations Summary���������������������������������������������������������������������������������������������������������� 125
Persistence Simplified�������������������������������������������������������������������������������������������������������������� 126
The Ecosystem�������������������������������������������������������������������������������������������������������������������������� 133
Persistent Containers���������������������������������������������������������������������������������������������������������� 134
Examples of Persistent Containers�������������������������������������������������������������������������������������� 135
Summary���������������������������������������������������������������������������������������������������������������������������������� 138

vi
Table of Contents

Chapter 9: pmemkv: A Persistent In-­Memory Key-Value Store���������������������������� 141


pmemkv Architecture���������������������������������������������������������������������������������������������������������������� 143
A Phonebook Example�������������������������������������������������������������������������������������������������������������� 147
Bringing Persistent Memory Closer to the Cloud���������������������������������������������������������������������� 151
Summary���������������������������������������������������������������������������������������������������������������������������������� 152

Chapter 10: Volatile Use of Persistent Memory���������������������������������������������������� 155


Introduction������������������������������������������������������������������������������������������������������������������������������� 155
Background������������������������������������������������������������������������������������������������������������������������������� 156
Memory Allocation��������������������������������������������������������������������������������������������������������������� 156
How it Works����������������������������������������������������������������������������������������������������������������������� 156
Supported “Kinds” of Memory��������������������������������������������������������������������������������������������� 157
The memkind API���������������������������������������������������������������������������������������������������������������������� 159
Kind Management API��������������������������������������������������������������������������������������������������������� 159
Heap Management API�������������������������������������������������������������������������������������������������������� 164
Kind Configuration Management����������������������������������������������������������������������������������������� 167
Additional memkind Code Examples����������������������������������������������������������������������������������� 168
C++ Allocator for PMEM Kind��������������������������������������������������������������������������������������������������� 168
pmem::allocator methods���������������������������������������������������������������������������������������������������� 169
Nested Containers��������������������������������������������������������������������������������������������������������������� 169
C++ Examples�������������������������������������������������������������������������������������������������������������������������� 170
Using the pmem::allocator�������������������������������������������������������������������������������������������������� 170
Creating a Vector of Strings������������������������������������������������������������������������������������������������� 171
Expanding Volatile Memory Using Persistent Memory�������������������������������������������������������������� 173
libvmemcache: An Efficient Volatile Key-Value Cache for Large-Capacity
Persistent Memory�������������������������������������������������������������������������������������������������������������������� 177
libvmemcache Overview����������������������������������������������������������������������������������������������������� 178
libvmemcache Design��������������������������������������������������������������������������������������������������������� 180
Using libvmemcache����������������������������������������������������������������������������������������������������������� 183
Summary���������������������������������������������������������������������������������������������������������������������������������� 186

vii
Table of Contents

Chapter 11: Designing Data Structures for Persistent Memory��������������������������� 187


Contiguous Data Structures and Fragmentation����������������������������������������������������������������������� 187
Internal and External Fragmentation����������������������������������������������������������������������������������� 188
Atomicity and Consistency�������������������������������������������������������������������������������������������������� 189
Selective Persistence���������������������������������������������������������������������������������������������������������� 193
Example Data Structures����������������������������������������������������������������������������������������������������� 193
Summary���������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 12: Debugging Persistent Memory Applications������������������������������������� 207


pmemcheck for Valgrind����������������������������������������������������������������������������������������������������������� 208
Stack Overflow Example������������������������������������������������������������������������������������������������������ 208
Memory Leak Example�������������������������������������������������������������������������������������������������������� 209
Intel Inspector – Persistence Inspector������������������������������������������������������������������������������������ 210
Stack Overflow Example������������������������������������������������������������������������������������������������������ 211
Memory Leak Example�������������������������������������������������������������������������������������������������������� 212
Common Persistent Memory Programming Problems�������������������������������������������������������������� 214
Nonpersistent Stores����������������������������������������������������������������������������������������������������������� 214
Stores Not Added into a Transaction������������������������������������������������������������������������������������ 228
Memory Added to Two Different Transactions��������������������������������������������������������������������� 233
Memory Overwrites������������������������������������������������������������������������������������������������������������� 240
Unnecessary Flushes���������������������������������������������������������������������������������������������������������� 242
Out-of-Order Writes������������������������������������������������������������������������������������������������������������� 247
Summary���������������������������������������������������������������������������������������������������������������������������������� 259

Chapter 13: Enabling Persistence Using a Real-World Application���������������������� 261


The Database Example������������������������������������������������������������������������������������������������������������� 262
Different Persistent Memory Enablement Approaches������������������������������������������������������������� 262
Developing a Persistent Memory-Aware MariaDB* Storage Engine����������������������������������������� 263
Understanding the Storage Layer���������������������������������������������������������������������������������������� 264
Creating a Storage Engine Class����������������������������������������������������������������������������������������� 265
Summary���������������������������������������������������������������������������������������������������������������������������������� 276

viii
Table of Contents

Chapter 14: Concurrency and Persistent Memory������������������������������������������������ 277


Transactions and Multithreading����������������������������������������������������������������������������������������������� 278
Mutexes on Persistent Memory������������������������������������������������������������������������������������������������ 282
Atomic Operations and Persistent Memory������������������������������������������������������������������������������ 285
Lock-Free Algorithms and Persistent Memory�������������������������������������������������������������������� 285
Concurrent Data Structures for Persistent Memory������������������������������������������������������������������ 286
Concurrent Ordered Map����������������������������������������������������������������������������������������������������� 287
Concurrent Hash Map���������������������������������������������������������������������������������������������������������� 291
Summary���������������������������������������������������������������������������������������������������������������������������������� 293

Chapter 15: Profiling and Performance���������������������������������������������������������������� 295


Introduction������������������������������������������������������������������������������������������������������������������������������� 295
Performance Analysis Concepts����������������������������������������������������������������������������������������������� 295
Compute-Bound vs. Memory-Bound����������������������������������������������������������������������������������� 295
Memory Latency vs. Memory Capacity������������������������������������������������������������������������������� 296
Read vs. Write Performance������������������������������������������������������������������������������������������������ 296
Memory Access Patterns����������������������������������������������������������������������������������������������������� 296
I/O Storage Bound Workloads���������������������������������������������������������������������������������������������� 297
Determining the Suitability of Workloads for Persistent Memory��������������������������������������������� 297
Volatile Use Cases��������������������������������������������������������������������������������������������������������������� 298
Use Cases Requiring Persistence���������������������������������������������������������������������������������������� 301
Performance Analysis of Workloads Using Persistent Memory������������������������������������������������ 302
Characterizing the Workload����������������������������������������������������������������������������������������������� 303
Memory Bandwidth and Latency����������������������������������������������������������������������������������������� 303
Persistent Memory Read-Write Ratio���������������������������������������������������������������������������������� 305
Working Set Size and Memory Footprint����������������������������������������������������������������������������� 305
Non-Uniform Memory Architecture (NUMA) Behavior��������������������������������������������������������� 305
Optimizing the Software for Persistent Memory����������������������������������������������������������������� 307
Summary���������������������������������������������������������������������������������������������������������������������������������� 311

ix
Table of Contents

Chapter 16: PMDK Internals: Important Algorithms and Data Structures������������ 313
A Pool of Persistent Memory: High-Level Architecture Overview��������������������������������������������� 313
The Uncertainty of Memory Mapping: Persistent Memory Object Identifier����������������������������� 315
Persistent Thread Local Storage: Using Lanes�������������������������������������������������������������������������� 318
Ensuring Power-Fail Atomicity: Redo and Undo Logging���������������������������������������������������������� 320
Transaction Redo Logging��������������������������������������������������������������������������������������������������� 320
Transaction Undo Logging��������������������������������������������������������������������������������������������������� 321
libpmemobj Unified Logging������������������������������������������������������������������������������������������������ 322
Persistent Allocations: The Interface of a Transactional Persistent Allocator���������������������������� 323
Persistent Memory Heap Management: Allocator Design for Persistent Memory�������������������� 324
ACID Transactions: Efficient Low-Level Persistent Transactions����������������������������������������������� 328
Lazy Reinitialization of Variables: Storing the Volatile State on Persistent Memory����������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 331

Chapter 17: Reliability, Availability, and Serviceability (RAS)������������������������������ 333


Dealing with Uncorrectable Errors�������������������������������������������������������������������������������������������� 333
Consumed Uncorrectable Error Handling���������������������������������������������������������������������������� 334
Unconsumed Uncorrectable Error Handling������������������������������������������������������������������������ 336
Clearing Uncorrectable Errors��������������������������������������������������������������������������������������������� 339
Device Health���������������������������������������������������������������������������������������������������������������������������� 339
ACPI-Defined Health Functions (_NCH, _NBS)��������������������������������������������������������������������� 342
Vendor-Specific Device Health (_DSMs)������������������������������������������������������������������������������ 342
ACPI NFIT Health Event Notification������������������������������������������������������������������������������������ 343
Unsafe/Dirty Shutdown������������������������������������������������������������������������������������������������������������� 343
Application Utilization of Data Loss Count (DLC)����������������������������������������������������������������� 344
Summary���������������������������������������������������������������������������������������������������������������������������������� 346

Chapter 18: Remote Persistent Memory��������������������������������������������������������������� 347


RDMA Networking Protocols����������������������������������������������������������������������������������������������������� 348
Goals of the Initial Remote Persistent Memory Architecture���������������������������������������������������� 351
Guaranteeing Remote Persistence�������������������������������������������������������������������������������������������� 351
General-Purpose Remote Replication Method��������������������������������������������������������������������� 353

x
Table of Contents

Appliance Remote Replication Method�������������������������������������������������������������������������������� 355


General Software Architecture�������������������������������������������������������������������������������������������������� 357
librpmem Architecture and Its Use in Replication��������������������������������������������������������������������� 358
Configuring Remote Replication Using Poolsets����������������������������������������������������������������� 362
Performance Considerations����������������������������������������������������������������������������������������������� 362
Remote Replication Error Handling������������������������������������������������������������������������������������� 364
Say Hello to the Replicated World��������������������������������������������������������������������������������������� 364
Summary���������������������������������������������������������������������������������������������������������������������������������� 370

Chapter 19: Advanced Topics������������������������������������������������������������������������������� 373


Nonuniform Memory Access (NUMA)���������������������������������������������������������������������������������������� 373
NUMACTL Linux Utility��������������������������������������������������������������������������������������������������������� 374
NDCTL Linux Utility�������������������������������������������������������������������������������������������������������������� 376
Intel Memory Latency Checker Utility���������������������������������������������������������������������������������� 378
NUMASTAT Utility����������������������������������������������������������������������������������������������������������������� 380
Intel VTune Profiler – Platform Profiler�������������������������������������������������������������������������������� 381
IPMCTL Utility���������������������������������������������������������������������������������������������������������������������� 381
BIOS Tuning Options������������������������������������������������������������������������������������������������������������ 382
Automatic NUMA Balancing������������������������������������������������������������������������������������������������� 382
Using Volume Managers with Persistent Memory�������������������������������������������������������������������� 383
The mmap( ) MAP_SYNC Flag��������������������������������������������������������������������������������������������������� 385
Summary���������������������������������������������������������������������������������������������������������������������������������� 386

Appendix A: How to Install NDCTL and DAXCTL on Linux������������������������������������� 389


Prerequisites����������������������������������������������������������������������������������������������������������������������������� 389
Installing NDCTL and DAXCTL Using the Linux Distribution Package Repository���������������������� 390
Searching for Packages Within a Package Repository�������������������������������������������������������� 391
Installing NDCTL and DAXCTL from the Package Repository���������������������������������������������� 392

xi
Table of Contents

Appendix B: How to Install the Persistent Memory Development Kit (PMDK)������ 395
PMDK Prerequisites������������������������������������������������������������������������������������������������������������������ 395
Installing PMDK Using the Linux Distribution Package Repository������������������������������������������� 395
Package Naming Convention����������������������������������������������������������������������������������������������� 396
Searching for Packages Within a Package Repository�������������������������������������������������������� 396
Installing PMDK Libraries from the Package Repository����������������������������������������������������� 398
Installing PMDK on Microsoft Windows������������������������������������������������������������������������������������ 402

Appendix C: How to Install IPMCTL on Linux and Windows���������������������������������� 403


IPMCTL Linux Prerequisites������������������������������������������������������������������������������������������������������ 404
libsafec�������������������������������������������������������������������������������������������������������������������������������� 404
IPMCTL Linux Packages������������������������������������������������������������������������������������������������������������ 404
IPMCTL for Microsoft Windows������������������������������������������������������������������������������������������������� 404
Using ipmctl������������������������������������������������������������������������������������������������������������������������������ 405

Appendix D: Java for Persistent Memory������������������������������������������������������������� 411


Volatile Use of Persistent Memory�������������������������������������������������������������������������������������������� 411
Heap Allocation on Alternative Memory Devices����������������������������������������������������������������� 412
Persistent Collections for Java (PCJ)���������������������������������������������������������������������������������������� 416
Using PCJ in Java Applications������������������������������������������������������������������������������������������������� 417
Low-Level Persistent Library (LLPL)����������������������������������������������������������������������������������������� 418
Using LLPL in Java Applications������������������������������������������������������������������������������������������������ 419
Summary���������������������������������������������������������������������������������������������������������������������������������� 419

Appendix E: The Future of Remote Persistent Memory Replication��������������������� 421

Glossary���������������������������������������������������������������������������������������������������������������� 425

Index��������������������������������������������������������������������������������������������������������������������� 429

xii
About the Author
Steve Scargall is a persistent memory software and cloud architect at Intel
Corporation. As a technology evangelist, he supports the enabling and development
effort to integrate persistent memory technology into software stacks, applications,
and hardware architectures. This includes working with independent software
vendors (ISVs) on both proprietary and open source development, original equipment
manufacturers (OEMs), and cloud service providers (CSPs).

Steve holds a Bachelor of Science in computer science and cybernetics from the
University of Reading, UK, where he studied neural networks, AI, and robotics. He
has over 19 years’ experience providing performance analysis on x86 architecture and
SPARC for Solaris Kernel, ZFS, and UFS file system. He performed DTrace debugging in
enterprise and cloud environments during his tenures at Sun Microsystems and Oracle.

xiii
About the Technical Reviewer
Andy Rudoff is a principal engineer at Intel Corporation, focusing on non-volatile
memory programming. He is a contributor to the SNIA NVM Programming Technical
Work Group. His more than 30 years’ industry experience includes design and
development work in operating systems, file systems, networking, and fault management
at companies large and small, including Sun Microsystems and VMware. Andy has
taught various operating systems classes over the years and is a coauthor of the popular
UNIX Network Programming textbook.

xv
About the Contributors
Piotr Balcer is a software engineer at Intel Corporation with many years’ experience
working on storage-related technologies. He holds a Bachelor of Science in engineering
from the Gdańsk University of Technology, Poland, where he studied system software
engineering. Piotr has been working on the software ecosystem for next-generation
persistent memory since 2014.

Eduardo Berrocal joined Intel Corporation as a cloud software engineer in 2017 after
receiving his PhD in computer science from the Illinois Institute of Technology. His
doctoral research focused on data analytics and fault tolerance for high-performance
computing. Past experience includes working as an intern at Bell Labs (Nokia), a research
aid at Argonne National Laboratory, a scientific programmer and web developer at the
University of Chicago, and an intern in the CESVIMA laboratory in Spain.

Adam Borowski is a software engineer at Intel Corporation, hailing from the


University of Warsaw, Poland. He is a Debian developer and has made many open
source contributions over the past two decades. Adam is currently working on
persistent memory stacks, both on upstream code and integrating it with downstream
distributions.

Igor Chorazewicz is a software engineer at Intel Corporation. His main focus is on


persistent memory data structures and enabling C++ applications for persistent
memory. Igor holds a Bachelor of Science in engineering from the Gdańsk University of
Technology, Poland.

Adam Czapski is a technical writer at Intel Corporation. He writes technical


documentation in the Data Center Group and is currently working in the persistent
memory department. Adam holds a Bachelor of Arts in English philology and a master’s
degree in natural language processing from the Gdańsk University of Technology, Poland.

Steve Dohrmann is a software engineer at Intel Corporation. He has worked on a variety


of projects over the past 20 years, including media frameworks, mobile agent software,
secure collaboration software, and parallel programming language implementation. He
is currently working on enabling the use of persistent memory in Java*.

xvii
About the Contributors

Chet Douglas is a principal software engineer at Intel Corporation and focuses on cloud
software architecture along with operating system and OEM enabling of non-volatile
memory technologies. He has over 14 years’ experience working on various enterprise
and client programs and 28 years of total storage experience. Chet has worked in all
aspects of storage, including storage controller hardware design, SCSI disk/tape/CD
writer firmware architecture, storage management software architecture, Microsoft
Windows* and Linux kernel-mode drivers, enterprise hardware RAID, and client/
workstation software RAID. He holds seven storage-related hardware and software
patents and has a dual Bachelor of Science in electrical engineering and computer
engineering from Clarkson University, New York.

Ken Gibson is the director of persistent memory software architecture within Intel
Corporation’s Data Center Group. Since 2012, Ken and his team have been working with
Intel’s server and software partners to create the open persistent memory programming
model.

Tomasz Gromadzki is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other (networking) technologies as well as
optimal persistent memory replication procedures and algorithms.

Kishor Kharbas is a software engineer on the Java runtime engineering team at Intel
Corporation. For the past eight years, he has been working to optimize Oracle’s OpenJDK
on Intel platforms. This involves Java garbage collection and compiler back-end
optimization.

Jackson Marusarz is a senior technical consulting engineer (TCE) in Intel Corporation's


Compute Performance and Developer Products Division. As the lead TCE for Intel
VTune Profiler, his main focus is on software performance analysis and tuning for both
serial and multithreaded applications. Jackson’s time is split between determining how
to analyze and tune software and creating tools that help others do the same.

Jan Michalski is a software engineer in Intel Corporation’s Non-­Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other technologies, as well as looking for
optimal persistent memory replication procedures and algorithms. He holds a master's
degree in computer engineering from the Gdańsk University of Technology, Poland,
where he studied system software engineering.

xviii
About the Contributors

Nicholas Moulin is a cloud software architect at Intel Corporation. Since joining Intel
in 2012, he has focused on enabling and developing persistent memory software for
operating systems and platform firmware and managing persistent memory hardware.
Nicholas is currently working with industry partners to define and improve RAS features
relevant to the persistent memory programming model.

Szymon Romik is a software engineer at Intel Corporation and is currently focused on


persistent memory programming. He previously worked as a lead software engineer
on 5G technologies at Ericsson. Szymon holds a master’s degree in mathematics from
Jagiellonian University, Poland.

Jakub Schmiegel is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group where he has been focused on enabling existing applications to
persistent memory and analyzing their performance for more than four years. Jakub
holds a master’s degree in computer science from the Gdańsk University of Technology,
Poland.

Kevin Shalkowsky is a Telly Award–winning creative director, graphic designer, and


animator with more than a decade of experience. While his contributions are in
technology today, Kevin has spent time in broadcast journalism and selling numerous
products through 30-minute late-night infomercials. He resides in Oregon with his wife
and son. From time to time, you can find Kevin lost in the woods, lost in a parking lot,
or lost in his design process – but this somehow got him to where he is today, and he
wouldn’t have it any other way.

Vineet Singh is a memory and storage tools software engineer at Intel Corporation.
He develops techniques to help developers adapt to the latest memory technologies.
Vineet holds a PhD in philosophy from the University of California and has a Bachelor
of Technology degree from the Indian Institute of Information Technology, Design, and
Manufacturing in Jabalpur.

Pawel Skowron is a software engineering manager at Intel Corporation with 20 years'


experience in the software industry. Pawel has worked in various roles related to the
whole-software development life cycle. His software engineering background lies in the
areas of embedded systems, database systems, and applications. For the past few years,
Pawel has led the development and validation of the Persistent Memory Development
Kit (https://github.com/pmem/pmdk).

xix
About the Contributors

Usha Upadhyayula has been with Intel Corporation for 20 years serving in many
different roles. Usha holds a master’s degree in computer science from the University
of South Carolina, and she spent the first few years at Intel developing user-level
applications in C and C++. She later moved to customer-­enabling roles for Intel media
processors and support for Intel RAID software. Usha is currently part of the Data Center
Group where she is focused on enabling cloud service providers to fully utilize and
accelerate the adoption of Intel persistent memory products.

Sergey Vinogradov is a senior software development engineer at Intel Corporation


where he spent more than seven years working on performance profiling tools and
threading runtime libraries. During the past four years, Sergey has been working on C++
programming models and performance profiling methodologies for persistent memory.

xx
Acknowledgments
First and foremost, I would like to thank Ken Gibson for masterminding this book idea
and for gifting me the pleasure of writing and managing it. Your support, guidance, and
contributions have been instrumental in delivering a high-quality product.
If the Vulcan mind-meld or The Matrix Headjack were possible, I could have cloned
Andy Rudoff’s mind and allowed him to work on his daily activities. Instead, Andy’s
infinite knowledge of persistent memory had to be tapped through good old verbal
communication and e-mail. I sincerely thank you for devoting so much time to me and
this project. The results read for themselves.
Debbie Graham was instrumental in helping me manage this colossal project. Her
dedication and support helped drive the project to an on-time completion.
To my friends and colleagues at Intel who contributed content, supported
discussions, helped with decision-making, and reviewed drafts during the book-writing
process. These are the real heroes. Without your heavily invested time and support, this
book would have taken considerably longer to complete. It is a much better product as a
result of the collaborative effort. A huge thanks to all of you.
I'd like to express my sincerest gratitude and appreciation to the people at Apress,
without whom this book could not have been published. From the initial contact and
outline discussions through the entire publishing process to this final polished product,
the Apress team delivered continuous support and assistance. Many thanks to Susan,
Jessica, and Rita. It was a real pleasure working with you.

xxi
Preface
About This Book
Persistent memory is often referred to as non-volatile memory (NVM) or storage
class memory (SCM). In this book, we purposefully use persistent memory as an all-­
encompassing term to represent all the current and future memory technologies that
fall under this umbrella. This book introduces the persistent memory technology and
provides answers to key questions. For software developers, those questions include:
What is persistent memory? How do I use it? What APIs and libraries are available?
What benefits can it provide for my application? What new programming methods do I
need to learn? How do I design applications to use persistent memory? Where can I find
information, documentation, and help?
System and cloud architects will be provided with answers to questions such as:
What is persistent memory? How does it work? How is it different than DRAM or SSD/
NVMe storage devices? What are the hardware and operating system requirements?
What applications need or could benefit from persistent memory? Can my existing
applications use persistent memory without being modified?
Persistent memory is not a plug-and-play technology for software applications.
Although it may look and feel like traditional DRAM memory, applications need to be
modified to fully utilize the persistence feature of persistent memory. That is not to say
that applications cannot run unmodified on systems with persistent memory installed,
they can, but they will not see the full potential of what persistent memory offers without
code modification.
Thankfully, server and operating system vendors collaborated very early in the
design phase and already have products available on the market. Linux and Microsoft
Windows already provide native support for persistent memory technologies. Many
popular virtualization technologies also support persistent memory.
For ISVs and the developer community at large, the journey is just beginning. Some
software has already been modified and is available on the market. However, it will
take time for the enterprise and cloud computing industries to adopt and make the
hardware available to the general marketplace. ISVs and software developers need time
to understand what changes to existing applications are required and implement them.
xxiii
Preface

To make the required development work easier, Intel developed and open sourced
the Persistent Memory Development Kit (PMDK) available from https://pmem.io/
pmdk/. We introduce the PMDK in more detail in Chapter 5 and walk through most of
the available libraries in subsequent chapters. Each chapter provides an in-depth guide
so developers can understand what library or libraries to use. PMDK is a set of open
source libraries and tools based on the Storage Networking Industry Association (SNIA)
NVM programming model designed and implemented by over 50 industry partners. The
latest NVM programming model document can be found at https://www.snia.org/
tech_activities/standards/curr_standards/npm. The model describes how software
can utilize persistent memory features and enables designers to develop APIs that take
advantage of NVM features and performance.
Available for both Linux and Windows, PMDK facilitates persistent memory
programming adoption with higher-level language support. C and C++ support is fully
validated. Support for other languages such as Java and Python is work in progress
at the time this book was written. Other languages are expected to also adopt the
programming model and provide native persistent memory APIs for developers. The
PMDK development team welcomes and encourages new contributions to core code,
new language bindings, or new storage engines for the persistent memory key-value
store called pmemkv.
This book assumes no prior knowledge of persistent memory hardware devices
or software development. The book layout allows you to freely navigate the content in
the order you want. It is not required to read all chapters in order, though we do build
upon concepts and knowledge described in previous chapters. In such cases, we make
backward and forward references to relevant chapters and sections so you can learn or
refresh your memory.

B
 ook Structure
This book has 19 chapters, each one focusing on a different topic. The book has three
main sections. Chapters 1-4 provide an introduction to persistent memory architecture,
hardware, and operating system support. Chapters 5-16 allow developers to understand
the PMDK libraries and how to use them in applications. Finally, Chapters 17-19 provide
information on advanced topics such as RAS and replication of data using RDMA.

xxiv
Preface

• Chapter 1. Introduction to Persistent Memory – Introduces persistent


memory and dips our toes in the water with a simple persistent key-­
value store example using libpmemkv.

• Chapter 2. Persistent Memory Architecture – Describes the persistent


memory architecture and focuses on the hardware requirements
developers should know.

• Chapter 3. Operating System Support for Persistent Memory –


Provides information relating to operating system changes, new
features, and how persistent memory is seen by the OS.

• Chapter 4. Fundamental Concepts of Persistent Memory


Programming – Builds on the first three chapters and describes the
fundamental concepts of persistent memory programming.

• Chapter 5. Introducing the Persistent Memory Development Kit


(PMDK) – Introduces the Persistent Memory Development Kit
(PMDK), a suite of libraries to assist software developers.

• Chapter 6. libpmem: Low-Level Persistent Memory Support –


Describes and shows how to use libpmem from the PMDK, a low-­level
library providing persistent memory support.

• Chapter 7. libpmemobj: A Native Transactional Object Store –


Provides information and examples using libpmemobj, a C native
object store library from the PMDK.
• Chapter 8. libpmemobj-cpp: The Adaptable Language - C++ and
Persistent Memory – Demonstrates the C++ libpmemobj-cpp object
store from the PMDK, built using C++ headers on top of libpmemobj.

• Chapter 9. pmemkv: A Persistent In-Memory Key-Value Store –


Expands upon the introduction to libpmemkv from Chapter 1 with a
more in-depth discussion using examples.

• Chapter 10. Volatile Use of Persistent Memory – This chapter is


for those who want to take advantage of persistent memory but
do not require data to be stored persistently. libmemkind is a user-
extensible heap manager built on top of jemalloc which enables
control of memory characteristics and a partitioning of the heap

xxv
Preface

between different kinds of memory, including persistent memory.


libvmemcache is an embeddable and lightweight in-memory caching
solution. It is designed to fully take advantage of large-capacity
memory, such as persistent memory with DAX, through memory
mapping in an efficient and scalable way.

• Chapter 11. Designing Data Structures for Persistent Memory –


Provides a wealth of information for designing data structures for
persistent memory.
• Chapter 12. Debugging Persistent Memory Applications – Introduces
tools and walks through several examples for how software developers
can debug persistent memory–enabled applications.

• Chapter 13. Enabling Persistence using a Real-World Application –


Discusses how a real-world application was modified to enable
persistent memory features.

• Chapter 14. Concurrency and Persistent Memory – Describes how


concurrency in applications should be implemented for use with
persistent memory.

• Chapter 15. Profiling and Performance – Teaches performance


concepts and demonstrates how to use the Intel VTune suite of tools
to profile systems and applications before and after code changes are
made.

• Chapter 16. PMDK Internals: Important Algorithms and Data


Structures – Takes us on a deep dive of the PMDK design, architecture,
algorithms, and memory allocator implementation.

• Chapter 17. Reliability, Availability, and Serviceability (RAS) –


Describes the implementation of reliability, availability, and
serviceability (RAS) with the hardware and operating system layers.

• Chapter 18. Remote Persistent Memory – Discusses how applications


can scale out across multiple systems using local and remote persistent
memory.

• Chapter 19. Advanced Topics – Describes things such as NUMA, using


software volume managers, and the mmap() MAP_SYNC flag.

xxvi
Preface

The Appendixes have separate procedures for installing the PMDK and utilities
required for managing persistent memory. We also included an update for Java and the
future of the RDMA protocols. All of this content is considered temporal, so we did not
want to include it in the main body of the book.

Intended Audience
This book has been written for experienced application developers in mind. We
intend the content to be useful to a wider readership such as system administrators
and architects, students, lecturers, and academic research fellows to name but a few.
System designers, kernel developers, and anyone with a vested or passing interest in this
emerging technology will find something useful within this book.
Every reader will learn what persistent memory is, how it works, and how operating
systems and applications can utilize it. Provisioning and managing persistent memory
are vendor specific, so we include some resources in the Appendix sections to avoid
overcomplicating the main chapter content.
Application developers will learn, by example, how to integrate persistent memory
in to existing or new applications. We use examples extensively throughout this book
using a variety of libraries available within the Persistent Memory Development Kit
(PMDK). Example code is provided in a variety of programming languages such as C,
C++, JavaScript, and others. We want developers to feel comfortable using these libraries
in their own projects. The book provides extensive links to resources where you can find
help and information.
System administrators and architects of Cloud, high-performance computing,
and enterprise environments can use most of the content of this book to
understand persistent memory features and benefits to support applications and
developers. Imagine being able to deploy more virtual machines per physical server or
provide applications with this new memory/storage tier such that they can keep more
data closer to the CPU or restart in a fraction of the time they could before while keeping
a warm cache of data.
Students, lecturers, and academic research fellows will also benefit from many
chapters within this book. Computer science classes can learn about the hardware,
operating system features, and programming techniques. Lecturers are free use the
content in student classes or to form the basis of research projects such as new persistent
memory file systems, algorithms, or caching implementations.

xxvii
Preface

We introduce tools that profile the server and applications to better understand CPU,
memory, and disk IO access patterns. Using this knowledge, we show how applications
can be modified to take full advantage of persistence using the Persistent Memory
Development Kit (PMDK).

A Future Reference
The book content has been written to provide value for many years. Industry
specification such as ACPI, UEFI, and the SNIA non-volatile programming model will,
unless otherwise stated by the specification, remain backward compatible as new
versions are released. If new form factors are introduced, the approach to programming
remains the same. We do not limit ourselves to one specific persistent memory vendor
or implementation. In places where it is necessary to describe vendor-specific features
or implementations, we specifically call this out as it may change between vendors or
between product generations. We encourage you to read the vendor documentation for
the persistent memory product to learn more.
Developers using the Persistent Memory Development Kit (PMDK) will retain a stable
API interface. PMDK will deliver new features and performance improvements with each
major release. It will evolve with new persistent memory products, CPU instructions,
platform designs, industry specifications, and operating system feature support.

Source Code Examples


Concepts and source code samples within this book adhere to the vendor neutral
SNIA non-volatile memory programming model. SNIA which is the Storage
Networking Industry Association is a non-profit global organization dedicated to
developing standards and education programs to advance storage and information
technology. The model was designed, developed, and is maintained by the SNIA NVM
Technical Working Group (TWG) which includes many leading operating system,
hardware, and server vendors. You can join this group or find information at https://
www.snia.org/forums/sssi/nvmp.

xxviii
Preface

The code examples provided with this book have been tested and validated using
Intel Optane DC persistent memory. Since the PMDK is vendor neutral, they will also
work on NVDIMM-N devices. PMDK will support any future persistent memory product
that enters the market.
The code examples used throughout this book are current at the time of
publication. All code examples have been validated and tested to ensure they compile
and execute without error. For brevity, some of the examples in this book use assert()
statements to indicate unexpected errors. Any production code would likely replace
these with the appropriate error handling actions which would include friendlier
error messages and appropriate error recovery actions. Additionally, some of the code
examples use different mount points to represent persistent memory aware file systems,
for example “/daxfs”, “/pmemfs”, and “/mnt/pmemfs”. This demonstrates persistent
memory file systems can be mounted and named appropriately for the application, just
like regular block-based file systems. Source code is from the repository that accompanies
this book – https://github.com/Apress/programming-persistent-memory.
Since this is a rapidly evolving technology, the software and APIs references
throughout this book may change over time. While every effort is made to be backward
compatible, sometimes software must evolve and invalidate previous versions. For this
reason, it is therefore expected that some of the code samples may not compile on newer
hardware or operating systems and may need to be changed accordingly.

B
 ook Conventions
This book uses several conventions to draw your attention to specific pieces of
information. The convention used depends on the type of information displayed.

C
 omputer Commands
Commands, programming library, and API function references may be presented in line
with the paragraph text using a monospaced font. For example:
To illustrate how persistent memory is used, let’s start with a sample program
demonstrating the key-value store provided by a library called libpmemkv.

xxix
Preface

Computer Terminal Output


Computer terminal output is usually taken directly from a computer terminal presented
in a monospaced font such as the following example demonstrating cloning the
Persistent Memory Development Kit (PMDK) from the GitHub project:

$ git clone https://github.com/pmem/pmdk


Cloning into 'pmdk'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 100169 (delta 2), reused 7 (delta 2), pack-reused 100157
Receiving objects: 100% (100169/100169), 34.71 MiB | 4.85 MiB/s, done.
Resolving deltas: 100% (83447/83447), done.

S
 ource Code
Source code examples taken from the accompanying GitHub repository are shown with
relevant line numbers in a monospaced font. Below each code listing is a reference to
the line number or line number range with a brief description. Code comments use
language native styling. Most languages use the same syntax. Single line comments
will use // and block/multiline comments should use /*..*/. An example is shown in
Listing 1.

Listing 1. A sample program using libpmemkv


    37  #include <iostream>
    38  #include "libpmemkv.h"
    39  
    40  using namespace pmemkv;
    41  
    42  /*
    43   * kvprint -- print a single key-value pair
    44   */
    45  void kvprint(const string& k, const string& v) {
    46      std::cout << "key: " << k << ", value: " << v << "\n";
    47  }

xxx
Preface

• Line 45: Here we define a small helper routine, kvprint(), which prints
a key-value pair when called.

Notes
We use a standard format for notes, cautions, and tips when we want to direct your
attention to an important point, for example.

Note Notes are tips, shortcuts, or alternative approaches to the current


discussion topic. Ignoring a note should have no negative consequences, but you
might miss out on a nugget of information that makes your life easier.

xxxi
Discovering Diverse Content Through
Random Scribd Documents
judges of that court.

"(2) For the purpose of hearing and determining appeals under


this Act, and for the purpose of any other proceedings under
this Act, the Court of Criminal Appeal shall be summoned in
accordance with directions given by the Lord Chief Justice of
England with the consent of the Lord Chancellor, and the court
shall be duly constituted if it consists of not less than
three judges and of an uneven number of judges.

"If the Lord Chief Justice so directs, the court may sit in
two or more divisions. The court shall sit in London except in
cases where the Lord Chief Justice gives special directions
that it shall sit at some other place. …

"3. A person convicted on indictment may appeal under this Act


to the Court of Criminal Appeal—

(a) against his conviction on any ground of appeal


which involves a question of law alone; and

(b) with the leave of the Court of Criminal Appeal


or upon the certificate of the Judge who tried him that it
is a fit case for appeal against his conviction on any
ground of appeal which involves a question of fact alone or
a question of mixed law and fact, or any other ground which
appears to the court to be a sufficient ground of appeal;
and

(c) with the leave of the Court of Criminal Appeal


against the sentence passed on his conviction, unless the
sentence is one fixed by law.

"4.
(1) The Court of Criminal Appeal on any such appeal against
conviction shall allow the appeal if they think that the
verdict of the jury should be set aside on the ground that it
is unreasonable or cannot be supported having regard to the
evidence, or that the judgment of the court before whom the
appellant was convicted should be set aside on the ground of a
wrong decision of any question of law or that on any ground
there was a miscarriage of justice, and in any other case
shall dismiss the appeal: Provided that the court may,
notwithstanding that they are of opinion that the point raised
in the appeal might be decided in favour of the appellant,
dismiss the appeal if they consider that no substantial
miscarriage of justice has actually occurred.

"(2) Subject to the special provisions of this Act, the Court


of Criminal Appeal shall, if they allow an appeal against
conviction, quash the conviction and direct a judgment and
verdict of acquittal to be entered.

"(3) On an appeal against sentence the Court of Criminal


Appeal shall, if they think that a different sentence should
have been passed, quash the sentence passed at the trial, and
pass such other sentence warranted in law by the verdict
(whether more or less severe) in substitution therefor as they
think ought to have been passed, and in any other case shall
dismiss the appeal.

"5.
(1) If it appears to the Court of Criminal Appeal that an
appellant, though not properly convicted on some count or part
of the indictment, has been properly convicted on some other
count or part of the indictment, the court may either affirm
the sentence passed on the appellant at the trial, or pass
such sentence in substitution therefor as they think proper,
and as may be warranted in law by the verdict on the count or
part of the indictment on which the court consider that the
appellant has been properly convicted.

"(2) Where an appellant has been convicted of an offence and


the jury could on the indictment have found him guilty of some
other offence, and on the finding of the jury it appears to
the Court of Criminal Appeal that the jury must have been
satisfied of facts which proved him guilty of that other
offence, the court may, instead of allowing or dismissing the
appeal, substitute for the verdict found by the jury a verdict
of guilty of that other offence, and pass such sentence in
substitution for the sentence passed at the trial as may be
warranted in law for that other offence, not being a sentence
of greater severity.

"(3) Where on the conviction of the appellant the jury have


found a special verdict, and the Court of Criminal Appeal
consider that a wrong conclusion has been arrived at by the
court before which the appellant has been convicted on the
effect of that verdict, the Court of Criminal Appeal may,
instead of allowing the appeal, order such conclusion to be
recorded as appears to the court to be in law required by the
verdict, and pass such sentence in substitution for the
sentence passed at the trial as may be warranted in law.

"(4) If on any appeal it appears to the Court of Criminal


Appeal that, although the appellant was guilty of the act or
omission charged against him, he was insane at the time the
act was done or omission made so as not to be responsible
according to law for his actions, the court may quash the
sentence passed at the trial and order the appellant to be
kept in custody as a criminal lunatic."

{412}

LAW AND ITS COURTS: France:


Reform of Judicial Procedure in Criminal Trials.

Criticism of French judicial procedure in criminal trials,


under the system which puts the duties of a prosecuting
attorney on the judge, was much sharpened in the autumn of
1909 by the attention drawn to a sensational murder trial at
Paris—the Steinheil case. The result was to impel the
Government to undertake measures of reform, beginning with the
appointment, November 20, of an extra-Parliamentary commission
to study the whole question of reform. Within a month after
the appointment of the commission one of its leading members,
in an article in the Matin, indicated the main points
of the recommendations which the commission was already
prepared to make. It would recommend that the authority of the
President of the Assize Court should remain intact, and that
the Judge should as heretofore continue to direct the jury and
preside over the whole process of the instruction or
preliminary inquiry. In the view of the Commission the Judge's
moral authority cannot but be augmented by the proposal to
relieve him of the duty of cross-examining a prisoner at the
bar. It would be recommended that in future a summary
statement of the case by the Public Prosecutor, or in a civil
suit by the plaintiff, should be followed by a presentation of
the defendant’s case on the part of counsel for the defence.
The jury would thus be made acquainted with the issue, and the
witnesses would then be called. Each witness would be liable
to cross-examination on behalf both of the defence and of the
prosecution, and the Judge, remaining aloof from the
discussion in his new rule as arbitrator, could not but gain
moral authority in a degree which would materially promote the
ends of even-handed justice.

A Bill on these lines was introduced by the Minister of


Justice in the following month.

LAW AND ITS COURTS: International:


Naval Prize Court, and proposed Judicial Arbitration Court.

See (in this Volume)


WAR, THE REVOLT AGAINST: A. D. 1909 (OCTOBER).

LAW AND ITS COURTS: United States:


The Question of Injunctions in Labor Disputes.
The question of the issuance of writs of injunction by the
courts in connection with labor disputes came much into
discussion during the canvass preliminary to the American
presidential election of 1908, and was a prominent subject of
declaration in the platforms of the political parties.

See (in this Volume)


UNITED STATES: A. D. 1908, APRIL-NOVEMBER).

Subsequently, President Taft, in his first annual message to


Congress, cited the pronouncement of the Republican party on
this question, and said: "I recommend that in compliance with
the promise thus made, appropriate legislation be adopted. The
ends of justice will best be met and the chief cause of
complaint against ill-considered injunctions without notice
will be removed by the enactment of a statute forbidding
hereafter the issuing of any injunction or restraining order,
whether temporary or permanent, by any Federal court, without
previous notice and a reasonable opportunity to be heard on
behalf of the parties to be enjoined; unless it shall appear
to the satisfaction of the court that the delay necessary to
give such notice and hearing would result in irreparable
injury to the complainant and unless also the court shall from
the evidence make a written finding, which shall be spread
upon the court minutes, that immediate and irreparable injury
is likely to ensue to the complainant, and shall define the
injury, state why it is irreparable, and shall also endorse on
the order issued the date and the hour of the issuance of the
order. Moreover, every such injunction or restraining order
issued without previous notice and opportunity by the
defendant to be heard should by force of the statute expire
and be of no effect after seven days from the issuance
thereof, or within any time less than that period which the
court may fix, unless within such seven days or such less
period, the injunction or order is extended or renewed after
previous notice and opportunity to be heard."
LAW AND ITS COURTS: National and State Legislation.
Need of Uniformity.
Movements to secure it.

Speaking in 1906 at a dinner of the Pennsylvania Society, the


Honorable Elihu Root, then United States Secretary of State,
addressed, in a few words, a very pregnant suggestion and
admonition to the lawmakers of the States in the American
Union. He spoke first of the strongly nationalized sentiment
of patriotism that has had its rapid growth of late in the
country, saying: "Our country as a whole, the noble and
beloved land of every citizen of every State, has become the
object of pride and devotion among all our people. North and
South, within the limits of the proud old colonial
commonwealths, through out that vast region where Burr once
dreamed of a separate empire dominating the valley of the
Mississippi, and upon the far distant shores of the Pacific:
and by the side of this strong and glowing loyalty to the
nation, sentiment for the separate States has become dim and
faint in comparison." Then he added, warningly: "There is but
one way in which the States of the Union can maintain their
power and authority under the conditions which are now before
us, and that way is by an awakening on the part of the States
to a realization of their own duties to the country at large.
Under the conditions which now exist, no State can live unto
itself alone and regulate its affairs with sole reference to
its own treasury, its own convenience, its own special
interests. Every State is bound to frame its legislation and
its administration with reference not only to its own social
affairs but with reference to the effect upon all its sister
States."

Quoting and affirming these remarks of the thoughtful


statesman, the National Civic Federation Review, of
July, 1909, says: "The plain truth is that the movement of
people and of merchandise goes on in our day without any
regard to State lines; and it is becoming increasingly clear
that unless the States will legislate with substantial
uniformity on a number of subjects the tendency toward
centralization and a corresponding increase of Federal power
cannot permanently be resisted."

In its preceding issue, of March, the Review had made


the following announcement: "The National Civic Federation,
through its experience in holding national conferences on such
subjects as the trusts, taxation, immigration and election
reform—conferences to which the Governors of States sent
official representatives—has become impressed with the
necessity for a systematic national effort toward securing,
within reasonable limits, more uniform legislation in the
States of the Union.

{413}

"There are useful national organizations of farmers,


manufacturers, wage-earners, bankers, merchants, lawyers,
economists and other organizations which hold national
meetings for the discussion of affairs peculiar to their own
pursuits and callings. The Civic Federation, however, provides
a forum in its annual conference for representatives of all
these elements to discuss national problems in which they have
a common interest. Heretofore there has been no effort to
crystallize into State organizations this representative
membership for the accomplishment of concrete aims.

"A committee has been appointed to organize a Council of one


hundred representative men in each State. Mr. John Hays
Hammond has accepted the chairmanship of this committee, of
which the following are also members: Messrs. Alton B. Parker,
New York; Myron T. Herrick, Ohio; David R. Francis, Missouri;
Curtis Guild, Jr., Massachusetts; Nahum J. Bachelder, New
Hampshire; Edwin Warfield, Maryland; Herman Ridder, New York;
C. F. Brooker, Connecticut; Bruce Haldeman, Kentucky; Victor
Rosewater, Nebraska; Clark Howell, Georgia; P. I. Bonebrake,
Kansas; James Lynch, Indiana; Harry Pratt Judson, Illinois; A.
H. Revell, Illinois; John B. Lennon, Illinois; John H.
Holliday, Indiana, and Benjamin Ide Wheeler, California.

"The continued existence for eighteen years of the Annual


Conference of Commissioners on Uniform State Laws, created by
the different States at the instance of the American Bar
Association, shows that the State Executives and Legislatures
are fully alive to the importance of this subject. The
last-named organization has been instrumental in securing the
passage in thirty-five States of a uniform negotiable
instruments law, and is promoting other commercial measures,
including a uniform food law to conform to the national law.

"This necessity for uniform legislation is further illustrated


by the proceedings at the annual meetings of the National
Association of the State Attorneys General and of the State
Labor Commissioners, Insurance Commissioners, etc., etc."

Discussing the subject in the July issue of the Review,


President Amasa M. Eaton of the Commissioners on Uniform State
Laws, said: "The subject of uniform legislation is in the air
all over the United States. At the instance of the President,
a National Conference to secure the conservation of our
national resources has been held in Washington, and to carry
into effect the conclusions of this Conference there must
follow uniform State legislation. At the instance of Governor
Guild a conference of the Governors of the New England States,
with other delegates, met in Boston last fall on the subject
of forestry, shell fisheries and automobiles, all subjects
calling for uniform legislation. A similar conference of the
Governors of New York and the adjoining States has met in New
York, at the instance of Governor Hughes of New York, to
consider a uniform automobile law. A National Divorce
Congress, called by Governor Pennypacker by virtue of an act
of the Legislature of Pennsylvania, has framed a uniform
divorce law which has been indorsed by the Conference of
Commissioners on Uniform State Laws. In March a Conference on
Uniform Child Labor Laws in the Southern States was held in
New Orleans at the call of the Governor of Louisiana, at which
the Governors and Delegates of those States were present. The
result was the formation of a permanent organization, with the
Governor of Louisiana as Chairman, and the executive committee
of that organization is to draft a Uniform Child Labor Law and
to submit it to the legislatures of the several Southern
States.

"All these are but expressions of the deep-seated necessity


for uniform legislation that has existed ever since we
acquired our independence of Great Britain, intensified by the
requirements of a progressive civilization knitting us ever
more and more closely into union as a nation."

The whole movement was planned to receive effective


organization at a National Conference in Washington which the
National Civic Federation, after consultation with other
bodies, announced, in the summer of 1909, its intention to
call, for January 5-7, 1910. The Conference was held
accordingly, in conjunction with a meeting of the Governors of
States, which gave attention to the same subject.

LAW AND ITS COURTS:


President Taft’s Recommendations for Expediting Procedure.

The following is from President Taft’s first annual Message to


Congress, December, 1909:

"The deplorable delays in the administration of civil and


criminal law have received the attention of committees of the
American Bar Association and of many State Bar Associations,
as well as the considered thought of judges and jurists. In my
judgment, a change in judicial procedure, with a view to
reducing its expense to private litigants in civil cases and
facilitating the dispatch of business and final decision in
both civil and criminal cases, constitutes the greatest need
in our American institutions. I do not doubt for one moment
that much of the lawless violence and cruelty exhibited in
lynchings is directly due to the uncertainties and injustice
growing out of the delays in trials, judgments, and the
executions thereof by our courts. Of course, these remarks
apply quite as well to the administration of justice in State
courts as to that in Federal courts, and without making
invidious distinction, it is, perhaps, not too much to say
that, speaking generally, the defects are less in the Federal
courts than in the State courts. But they are very great in
the Federal courts. The expedition with which business is
disposed of both on the civil and the criminal side of English
courts, under modern rules of procedure, makes the delays in
our courts seem archaic and barbarous.

"The procedure in the Federal courts should furnish an example


for the State courts. I presume it is impossible, without an
amendment to the Constitution, to unite under one form of
action the proceedings at common law and proceedings in equity
in the Federal courts, but it is certainly not impossible by a
statute to simplify and make short and direct the procedure
both at law and in equity in those courts. It is not
impossible to cut down still more than it is cut down the
jurisdiction of the Supreme Court so as to confine it almost
wholly to statutory and constitutional questions. Under the
present statutes, the equity and admiralty procedure in the
Federal courts is under the control of the Supreme Court, but
in the pressure of business to which that court is subjected,
it is impossible to hope that a radical and proper reform of
the Federal equity procedure can be brought about.
{414}
I therefore recommend legislation providing for the
appointment by the President of a commission with authority to
examine the law and equity procedure of the Federal courts of
first instance, the law of appeals from those courts to the
courts of appeals and to the Supreme Court, and the costs
imposed in such procedure upon the private litigants and upon
the public treasury, and make recommendation with a view to
simplifying and expediting the procedure as far as possible,
and making it as inexpensive as may be to the litigant of
little means."

See (in this Volume),


CRIME AND CRIMINOLOGY.

LEAGUE, ALL-INDIA MOSLEM.

See (in this Volume)


INDIA: A. D. 1907 (DECEMBER).

LEAGUE OF LIBERATION.

See (in this Volume)


RUSSIA: A. D. 1905-1907.

LEAGUE OF UNION AND PROGRESS.

See (in this Volume)


TURKEY: A. D. 1908 (JULY-DECEMBER), and after.

LECOT, Cardinal.

See (in this Volume)


FRANCE: A. D. 1905-1906.

LEGARDA, BENITO.

See (in this Volume)


PHILIPPINE ISLANDS: A. D. 1901.

LEGISLATION.
See (in this Volume)
LAW AND ITS COURTS.

LEGUIA, Augusto B.:


President of Peru.

See (in this Volume)


PERU: A. D. 1908-1909.

LENARD, PHILIPPE.

See (in this Volume)


NOBEL PRIZES.

LEO XIII.: Death.

See (in this Volume)


PAPACY: A. D. 1903 (JULY-AUGUST).

LEOPOLD II., KING OF BELGIUM:


His Administration of the Congo State.

See (in this Volume)


CONGO STATE.

LEOPOLD II., KING OF BELGIUM:


His death.

See (in this Volume)


BELGIUM: A. D. 1909 (DECEMBER).

LERROUX, Señor:
Socialist-Republican Leader in Spain.

See (in this Volume)


SPAIN: A. D. 1907-1909.
LÈSE MAJESTÉ:
Prosecutions in Germany.

See (in this Volume)


GERMANY: A. D. 1903 .

LETCHWORTH PARK.

See (in this Volume)


NEW YORK STATE: A. D. 1907.

LEWIS, THOMAS L.:


President of the United Mine Workers of America.

See (in this Volume)


LABOR ORGANIZATION: UNITED STATES: A. D. 1909.

LEWIS AND CLARK EXPOSITION.

See (in this Volume)


PORTLAND, OREGON.

LEWIS ESTATE, Evicted Tenants of the.

See (in this Volume)


IRELAND: A. D. 1907.

LHASA: A. D. 1904.
Reached by British Expedition under Colonel Younghusband.

See (in this Volume)


TIBET: A. D. 1902-1904.

LIAO-TUNG PENINSULA.

See (in this Volume)


JAPAN: A. D. 1904 (FEBRUARY-JULY).
LIAO-YANG, Battle of.

See (in this Volume)


JAPAN: A. D. 1904 (JULY-SEPTEMBER).

LIAUTEY, General:
Operations in Morocco.

See (in this Volume)


MOROCCO: A. D. 1907-1909, and 1909.

LIBERAL-CONSERVATIVE SEPARATIST PARTY.

See (in this Volume)


AUSTRIA-HUNGARY: A. D. 1904.

LIBERIA: A. D. 1904-1905.
Good Relations between Colonists and Natives.
Improved Prospects.

"When it was decided in the United States to found a home for


repatriated Africans, the prior experiment of Sierra Leone
turned attention toward the same coast, and in 1821 and at
subsequent dates settlements were effected, firstly at
Monrovia, and later on at Roberts Port, Grand Basa, Sino, and
Harper (Cape Palmas). Usually those who conducted the
enterprise went through the form of buying small plats of land
from local headmen or chiefs; but, as a rule, the promoters of
this movement did not trouble overmuch about the rights of the
‘bush niggers,’ as the indigenous natives were termed.
Consequently the first fifty years of the history of Liberia
were marked by constant struggles between the
American-Liberian invaders and the native blacks. During the
last ten years, however, there has been a marked advance in
good relations between the American settlers and their native
subjects, as many of them may fairly be called. The wise
policy of President Barclay has greatly promoted this good
feeling since 1904. He has been able to assemble at different
times at the capital chiefs or their representatives from
almost all parts of Liberia, even from the Mandingo districts
just beyond the limits of the coast belt. Therefore they have
no subject of disagreement. Curiously enough one example of
this mild rule of black by black is that the white man in
Liberia is everywhere received with great friendliness,
because he is not associated in the minds of the natives with
anything like conquest or oppression.

"How far the original experiment will succeed the next twenty
years will, perhaps, indicate. The negroes of American origin
who have settled in Liberia have not, as a general rule, been
able to stand the climate very much better than Europeans,
and, as a rule, they have not been able to rear large families
of children. Yet it seems to me as though Liberians of the new
generation born in the country are beginning to take hold, but
this is partly due to the increasing and I think very sensible
practice of intermarriage with women of the fine, vigorous,
indigenous races. Probably the future of Liberia will be a
negro state very like Sierra Leone in its development, with
English as its government language, and such English or
American institutions as may prove to be suited to an African
country, a coast belt inhabited by negroes professing
Christianity and wearing clothes of European cut, and a
hinterland of Mohammedans dressed in the picturesque and
wholly suitable costume worn at the present day by the
Mandingos and by most Mohammedan negroes between Senegal and
the White Nile."

Sir Harry Johnston,


Liberia
(Annual Report, Smithsonian Institution, 1904-1905,
pages 254-255).

LIBERIA: A. D. 1907-1909.
English, French, and American attention to Conditions
in the Republic.

"The policy of the Liberian Republic has caused anxiety for


some time past both to England and to France, the Powers whose
territory adjoins the Liberian boundary. Some two years ago
President Barclay came to Europe to discuss the situation with
the British and French Governments. As a result of this
exchange of views, Liberia appointed Europeans to her Customs
Department, secured a gunboat to patrol her coast-line, and
arranged for a frontier force. These measures were approved by
the British and French Governments and also by the American
Government, and their execution was facilitated by a loan
negotiated on behalf of Liberia by the Liberian Development
Company.
{415}
The growth of British interests in the Republic led the
Foreign Office to appoint a Consul-General at Monrovia, the
capital, in the person of Captain Braithwaite Wallis, formerly
acting district commissioner in Sierra Leone. So far as the
first part of the reform programme was concerned the
consequences have been eminently satisfactory. Liberia has
been able to pay off some of her debts, and her revenue has
increased."

Correspondent London Times,


April 22, 1909.

While these movements were in progress, in June, 1908, three


commissioners from Liberia came to Washington asking for aid
in maintaining and administering its government. Probably in
course of this application, the American Ambassador in London,
Mr. Reid, addressed a note to the British Secretary of State
for Foreign Affairs, on the 29th of June, in which he wrote:

"We should be glad to have your views as to how the two


Governments could best co-operate at the present time towards
promoting the welfare of Liberia."

In his reply to this Sir Edward Grey said:

"As I had the honour to explain in March last to the United


States Charge d’Affaires, his Majesty’s Government have in any
measure they may be called upon to take in Liberia no designs
whatever upon the independence or integrity of the country,
and they do not intend to undertake any responsibility with
regard to it. The services of British officials have been lent
to the Liberians solely with a view to the better preservation
of order, more particularly in that part of Liberia which
marches with Sierra Leone, and improved administration.

"The French Government also, as your Excellency is doubtless


aware, takes a special interest in the affairs of the
Republic, and his Majesty’s Government have already assured
them that they would have no objection to the services of some
French officials being lent for the same objects as the
British officials. It is doubtful, therefore, whether there is
at the present time any scope for the co-operation of the
United States Government in the Customs or police, and if they
desire to render active assistance to the Liberian Government
they will perhaps prefer to direct their attention to other
branches of the administration which are as urgently in need
of reform.

"That reforms are required in one other branch at least his


Majesty’s Government have reason to know, for among the chief
difficulties which his Majesty’s Government experience in
regard to Liberia are the frequent complaints received from
British subjects as to the treatment they receive in the
Liberian Courts. If therefore the United States could see
their way to introducing reforms into the judiciary, either by
lending the services of an official to act as judicial advisor
or in some other manner, much good would in the opinion of his
Majesty’s Government be derived not only by the various
subjects of foreign nationalities resident in the country but
also by the Liberians themselves.

"While calling attention more specially to this one branch of


the administration, which has been a frequent source of
trouble, I need hardly add that his Majesty’s Government would
welcome the co-operation of the Government of the United
States with them in Liberia in any other manner which may
appear more suitable or more desirable on a consideration of
all the circumstances."

This and other information obtained by the State Department


led President Roosevelt, on the 18th of January, 1909, to ask
Congress for an appropriation of $20,000 to pay the expenses
of a commission to go to Liberia "to examine into the
situation, confer with the officers of the Liberian
government, and with the representatives of other governments
actually present in Monrovia, and report recommendations as to
the specific action on the part of the United States most apt
to render effective relief to the Republic of Liberia under
the present critical circumstances." The conclusion reached by
the State Department was that it "is quite clear that Liberia
is very much in need of assistance, that the United States can
help her substantially, and that it is our duty to help her."

The seriousness of the situation was set forth by Secretary


Root in a memorandum to the President. Between forty and fifty
thousand civilized negroes, for the most part descendants of
the original colonists from the United States, occupy a
territory comprising 43,000 square miles, in which there are
also over a million and a half members of uncivilized native
tribes. The civilized part of the population have been to a
great degree cut off from any intimate relation with the rest
of the civilized world for two-thirds of a century. They began
with little education, with no acquired skill in the art of
government, and they have had little opportunity to improve
through intercourse with other and more advanced communities.
They find it especially difficult to control the native
tribes, or to conduct their own government in accordance with
modern requirements.

The British colony of Sierra Leone to the north and the French
possessions closing in their hinterland to the east are almost
continuously complaining of the failure of Liberia to maintain
order upon the border. "Notwithstanding the very kindly
disposition on the part of Great Britain, and the similar
disposition on the part of France, there is imminent danger
that the republic, unless it receives outside assistance, will
not be able to maintain itself very long," said Secretary
Root.

The Commission to visit Liberia was appointed in the following


April, and was conveyed soon afterward to Monrovia by a
squadron of three cruiser scouts. It was composed of three
members, Mr. W. Morgan Shuster, who had been for a number of
years in the Philippine service of the United States, Dr.
George Sale, and Mr. Emmett J. Scott, private Secretary of Dr.
Booker Washington. These Commissioners were accompanied by
Captain Sydney A. Cloman, as Military Attache, and by Major
Percy W. Ashburn, of the United States Of America Medical
Department, who would study the sanitary conditions in
Liberia.

Early in October the Commission returned, but its report to


the State Department was not transmitted to Congress until the
25th of March, 1910. It recommended an extension of prompt and
effective aid to the Liberian Government, in the refunding of
its debt, the reform of its finances, the settlement of its
boundary disputes, and the organizing of a competent
constabulary force. Also that the United States establish in
Liberia a naval coaling station and a research station.

{416}
LICENSE LAWS.

See (in this Volume)


ALCOHOL PROBLEM.

LIFE INSURANCE.

See (in this Volume)


INSURANCE.

LILIENTHAL, Otto.

See (in this Volume)


Science and Invention: Aeronautics.

LIMA, WENCESLAO DE.

See (in this Volume)


PORTUGAL: A. D. 1906-1909.

LINCOLN, ABRAHAM.

February 12, 1909, the 100th anniversary of his birth, was


made a legal holiday by act of Congress. The same bill
appropriated $50,000 for making a highway from Washington to
Gettysburg, to be known as the Lincoln Way.

LINDSEY, JUDGE BEN D.:


His Juvenile Court at Denver.

See (in this Volume)


CHILDREN, UNDER THE LAW; AS OFFENDERS.

LINEVITCH, General.

See (in this Volume)


JAPAN: A. D. 1904-1905 (SEPTEMBER-MARCH).
LIPPMAN, GABRIEL.

See (in this Volume)


NOBEL PRIZES.

LIQUOR QUESTION.

See (in this Volume)


ALCOHOL PROBLEM.

LLOYD-GEORGE, David:
President of the Board of Trade.

See (in this Volume)


ENGLAND: A. D. 1905-1906.

LLOYD-GEORGE, David:
Address at the Imperial Conference of 1907
on Preferential Trade.

See (in this Volume)


BRITISH EMPIRE: A. D. 1907.

LLOYD-GEORGE, David:
Success in arranging for the Pacific Settlement
of Labor Disputes in the English Railway Service.

See (in this Volume)


LABOR ORGANIZATION: ENGLAND: A. D. 1907-1909.

LLOYD-GEORGE, David:
Chancellor of the Exchequer.

See (in this Volume)


ENGLAND: A. D. 1908 (APRIL).
LLOYD-GEORGE, David:
On the Working of the Old Age Pensions Act and
its Disclosures of Poverty.

See (in this Volume)


POVERTY, PROBLEMS OF: PENSIONS.

LLOYD-GEORGE, David:
On the Development of the Natural Resources of Great Britain.

See (in this Volume)


CONSERVATION OF NATURAL RESOURCES: GREAT BRITAIN.

LLOYD-GEORGE, David:
His Budget of 1909.
His speech on it.

See (in this Volume)


ENGLAND: A. D. 1909 (APRIL-DECEMBER).

LOCAL OPTION:
Progress in the United States.

See (in this Volume)


ALCOHOL PROBLEM: UNITED STATES.

LOCKOUTS.

See (in this Volume)


LABOR ORGANIZATION.

LODGE, SIR OLIVER.

See (in this Volume)


SCIENCE, RECENT: ELECTRICAL.

LODZ, Disturbances in.


See (in this Volume)
RUSSIA: A. D. 1904-1905.

LOEB, WILLIAM, JR.:


Collector of Customs at New York.
His unearthing of Corruptions.

See (in this Volume)


UNITED STATES: A. D. 1909 (OCTOBER-NOVEMBER).

LOISY, ABBÉ:
Appointment to be Professor of the History of Religions
in the College de France.

See (in this Volume)


FRANCE: A. D. 1909 (MARCH).

LONDON, ENGLAND: A. D. 1907-1909.


Control of the London County Council lost by the Progressives.
Defeat in Borough Councils Elections of 1909.

The local party of Progressives, so called, who had controlled


the London County Council since 1889, lost their majority in
the elections of the spring of 1907, and the Conservatives, or
Moderates, or Reformers, as they are variedly styled, were
brought into power, electing 120 members, against 85. The
Progressives, in their eighteen years of ascendancy, had
wrought immense changes in the great city, widening congested
streets, such as the Strand, opening great new thoroughfares
and new parks, electrifying the street railways, remodelling
antiquated public institutions, and the like. The cost of
their works had been heavy, and ratepayers had become
persuaded that there was extravagance in the progressiveness
of the party. It had antagonized many powerful interests in
the city, moreover, and the wonder seems to be that it had
been permitted to conduct the City Government so long.
Again, in elections to the borough councils, in 1909, the
Progressives lost heavily, and the Conservatives, who have
taken the name of Municipal Reformers, are strongly entrenched
in most of the boroughs. Several women were elected, 61 of
their sex having been candidates.

LONDON, ENGLAND: A. D. 1908.


Statistics of Elementary Schools.

See (in this Volume)


EDUCATION: ENGLAND: A. D. 1909.

LONDON, ENGLAND: A. D. 1908-1909.


International Naval Conference.

See (in this Volume)


War, The Revolt against: A. D. 1907
(appended to account of Second Peace Conference at The Hague).

LONG, John D.:


Secretary of the Navy.

See (in this Volume)


UNITED STATES: A. D. 1901-1905.

LOPUKHIN, M.:
His exposure of the Police Spy, Azeff,
to the Russian Revolutionists.

See (in this Volume)


RUSSIA: A. D. 1909 (JANUARY-JULY).

LORDS, British House of:


Decision in case of the Free Church of Scotland.

See (in this Volume)


SCOTLAND: A. D. 1904.

LORDS, British House of:


Defeat of Education Bill, 1906.

See (in this Volume)


EDUCATION: ENGLAND: A. D. 1906.

LORDS, British House of:


Menaced Limitation of its Legislative Powers
by the House of Commons.
Its own proposals of Constitutional Change.

See (in this Volume)


ENGLAND: A. D. 1906 (APRIL-DECEMBER); 1907-1908;
1909 (APRIL-DECEMBER), and 1910.

LORDS, British House of:


Rejection of Licensing Bill.

See (in this Volume)


ALCOHOL PROBLEM: ENGLAND: A. D. 1908.

LORDS, British House of:


Rejection of Budget of 1909.

See (in this Volume)


ENGLAND: A. D. 1909 (APRIL-DECEMBER).

LORENTZ, HENRIK ANTON.

See (in this Volume)


NOBEL PRIZES.

LOS ANGELES, CALIFORNIA:


Recent Rapid Growth of the City.
"The advance of this city to the important position of
metropolis of Southern California falls into two quite
distinct periods, each, however, beginning with the advent of
a transcontinental railroad. The first period opened with the
completion of the Southern Pacific Railroad as a through line
from San Francisco to the East, in 1881, and saw the
transformation of Los Angeles from a sleepy, half-Spanish town
of about 12,000 souls into a bustling progressive city of
70,000 population. The second period of advance began with the
entrance of the Atchison, Topeka and Santa Fe Railroad in
1885. This improved communication with the States east of the
Rocky Mountains gave an impetus to tourist travel, especially
in the winter season, and the fame of the city and of near-by
localities as places of winter resort spread far and wide. The
people of Los Angeles were quick to recognize the opportunity
for gain and the whole community joined in methods of
advertising of the most systematic character. By the aid of
its local press and through the agency of an energetic Chamber
of Commerce Los Angeles has become one of the best known
cities of North America.

{417}

"Since 1900, railroad communication has been further improved


by the opening of an additional road to San Francisco by way
of the ocean shore and the Salinas and Santa Clara Valleys.
This line, known as the Southern Pacific ‘Coast Line,’ avoids
the heavy grades of the Tehachapi Mountains and greatly
shortens the running time between Los Angeles and San
Francisco. The opening of the San Pedro, Los Angeles and Salt
Lake Railroad eastward of Los Angeles in 1903 gave the city
direct connection with the central Rocky Mountain region.

"Two other important influences within the past decade


contributed to the city’s remarkable advance in wealth and
population. These are the building of a vast system of
suburban electric railways making a large region of fertile
attractive land, now densely populated, directly tributary to
Los Angeles, and secondly, the introduction of cheap fuel
through the discovery of local supplies of oil. The net-work
of suburban electric railways of which Los Angeles is the
center is one of the most perfect in the world. These lines
reach out in every direction through distances of from 10 to
50 miles, and connect Los Angeles with the many rapidly
growing cities of Los Angeles County and its neighbor, Orange
County.

"Manufacturing in Los Angeles was for a long time handicapped


by the high cost of fuel. This difficulty has been removed by
the introduction of crude oil as fuel, and the city now has
over 1500 manufacturing establishments, employing over 12,000
people, with an annual output of over $40,000,000. These
include rolling mills, brass-works, paper-box factories,
manufactories of mining machinery, pumps, glass, etc. Los
Angeles is becoming a manufacturing center for the mining and
agricultural lands of Utah, Southern Nevada, Arizona, New
Mexico and the Northern parts of Mexico, as well as Southern
California itself.

"The steady expansion of Los Angeles has been maintained by a


policy of annexation of suburbs. The latest event in this line
of growth has elevated the city into the rank of a sea-port.
The city has long enjoyed abundant means of ocean traffic by
way of Santa Monica, Redondo and San Pedro, but by the
annexation of San Pedro and Wilmington, in 1906, with a
connecting strip of territory 19 miles long by ½ mile in
width, Los Angeles itself becomes a sea-port with the control
of traffic on San Pedro Bay. The city thus achieves an extreme
length from north to south of 33 miles."

Frederick H. Clark, Head of History Department,


Lowell High School, San Francisco.

LOS ANGELES, CALIFORNIA:


Experiments and Experiences in Municipal Government.

See (in this Volume)


MUNICIPAL GOVERNMENT.

LOS ANGELES, CALIFORNIA: A. D. 1905-1909.


Water Supply.
The Owens River Aqueduct.

"The present water supply of the city of Los Angeles is taken


from the flow of the Los Angeles River, supplemented by the
underground flow of the San Fernando Valley in which the river
lies. The demand for water within the city is supplemented by
the need for water for irrigation purposes in the surrounding
country. Some years ago it became evident that an increased
supply must be obtained, or the further development of the
city and its environs be brought to a standstill. Extensive
investigations resulted in the decision that Owens River
offered the best source of supply. This river, the principal
drainage of the Owens Valley region, at the base of the Sierra
Nevada Mountains, has a large number of tributaries, and
empties into Owens Lake, from which the waters escape by
evaporation only. The Los Angeles authorities adopted the plan
of an aqueduct to conduct the waters of this river along the
mountain slopes, over the Mojave Desert, and, by tunnel,
through the San Fernando Mountains, to their city,--a total
distance of 217½ miles. On the 7th of September, 1905, an
election was held at which the voters of Los Angeles, by a
majority of about fourteen to one, declared in favor of a bond
issue of $23,000,000 for the undertaking. Besides the
construction of the conduit, the project includes the building
of a large reservoir in Long Valley, above the Owens Valley
proper, for the storage of flood waters; also the construction
of a system of additional reservoirs along the line of the
aqueduct for the regulation of flow as well as for storage;
and a terminal reservoir from which the distributing system
proceeds. All of this work is well under way at this date
(1909), and according to the last published report of the
Aqueduct Bureau the chief engineer confidently expects that
this great project will be brought to completion within the
estimated period of five years--and within the estimated cost
of $23,000,000.

"Outside of the above estimates, the City also plans to build


a great electric power plant which will utilize the drop of
1500 feet where the aqueduct emerges from the San Fernando
Mountains. This plant is estimated to cost from $4,500,000 to
$5,000,000, and through the sale of electric power will become
the source of very considerable revenue to the City. Taken
altogether this Owens River Aqueduct is the greatest municipal
undertaking in California at the present time, and one of the
most important engineering achievements of recent years."

Frederick H. Clark, Head of History Department,


Lowell High School, San Francisco.

LOUBET, Emile:
President of France.

See (in Volume VI.)


FRANCE: A. D. 1899 (FEBRUARY-JUNE).

LOUBET, Emile:
Visit to the King of Italy.

See (in this Volume)


FRANCE: A. D. 1905-1906.

LOUBET, Emile:
Expiration of term as President of the French Republic.

See (in this Volume)


FRANCE: A. D. 1906.
LOUISIANA: A. D. 1908.
Enactment against Race-track Gambling.

See (in this Volume)


GAMBLING.

LOUISIANA PURCHASE EXPOSITION.

See (in this Volume)


ST. LOUIS: A. D. 1904.

LOW, Seth:
Mayor of New York.

See (in this Volume)


NEW YORK CITY: A. D. 1901-1903.

LOWELL, ABBOTT LAWRENCE:


President of Harvard University.

See (in this Volume)


EDUCATION: UNITED STATES: A. D. 1901-1909.

LOWTHER, JAMES WILLIAM:


Elected Speaker of the House of Commons.

See (in this Volume)


ENGLAND: A. D. 1905 (JUNE).

LUBIN, DAVID:
Originator of International Institute of Agriculture.

See (in this Volume)


AGRICULTURE.

{418}
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

textbookfull.com

You might also like