When setting up data-at-rest encryption (also known as transparent data encryption) in Percona Server for MongoDB, one has three options for storing a master encryption key:
- Encryption key file on a filesystem,
- KMIP server,
- HashiCorp’s Vault.
An encryption key file is only suitable for testing due to its lack of proper security. One can read more about KMIP in the documentation and the older blog post. The earlier post by Jaime Sicam and Michal Nosek provides complete instructions on setting up Vault as a master encryption key storage for Percona Server for MongoDB. In this blog post, I will focus on how new versions of Percona Server for MongoDB prevent potential key loss when using HashiCorp’s Vault.
Essentially, to make sure there is no risk of key loss in a deployment using HashiCorp’s Vault, you should:
- Upgrade Percona Server for MongoDB to the versions 6.0.18-15, 7.0.15-9, 8.0.4-1 or higher,
- Configure max_versions on the Vault server,
- Grant additional read permissions on the Vault server to Percona Server for MongoDB.
In the rest of the post, I will describe in detail how the key loss can occur and what Percona Server for MongoDB does to prevent it.
Percona Server for MongoDB’s integration with Vault
Percona Server for MongoDB supports the Vault’s versioned key-value secrets engine, kv-v2. The engine’s stored entity — secret — is essentially a set of key-value pairs. Percona Server for MongoDB constructs a single-entry secret, where a base64-encoded master encryption key is a value for an implementation-defined key.
On a Vault server, Percona Server for MongoDB locates a secret using the {<full_secret_path>, <version>} pair as a compound identifier, where the full secret path is a concatenation of engine’s mount path with the relative secret path on that engine over a fixed data string. For instance, in the following compound identifier
{mongokeys/prod/data/cluster-zulu/host-alpha, 5}
- mongokeys/prod is the versioned key-value secrets engine mount path,
- cluster-zulu/host-alpha is the relative secret path on that engine,
- 5 is the version of the secret.
When configuring Percona Server for MongoDB, you must pass the full secret path as the value of security.vault.secret configuration file parameter.
A user rarely needs to think about a particular secret version while configuring Percona Server for MongoDB because it is taken care of automatically. When Percona Server for MongoDB initializes a clean data directory or does key rotation for an existing data directory, it internally generates a new random master encryption key. Then, it wraps it into a secret and puts the latter on a Vault server at the configured path. Vault increments the value of current_version, associates the resulting value with a new secret, and returns the version. Percona Server for MongoDB then saves the full path and the version in the metadata and uses them later to get the key from the Vault server. Figure 1 illustrates the process.
Figure 1. Putting an encryption key to a Vault server
Secret overwriting and upgrading Percona Server for MongoDB
Adding a new version to a secret at a particular path is often illustrated similarly to what you can see in Figure 2 below, which suggests Vault stores secret versions in a sequential container.
Figure 2. An assumed model of a secret version storage
That, however, is not the case. By default, Vault keeps the ten most recent versions for each secret. Afterward, uploading each new version permanently removes the oldest one at the time. A more precise model for secret versions’ storage would be a circular buffer, as shown in Figure 3. The buffer’s capacity (i.e., the number of kept versions) can be configured, though, via the max_versions parameter either on a particular secret itself or the whole secret engine.
Figure 3. A realistic model of a secret version storage
A low default value for the number of kept secret versions poses a danger when several nodes intentionally or accidentally share a full secret path. For instance, eleven or more nodes having a shared secret path result in key loss during the cluster initialization. When the eleventh node puts its master encryption key on the Vault server, the latter irrecoverably removes the secret containing the master encryption key for the very first initialized node. Consequently, the first node won’t be able to decrypt the data directory after the restart.
Starting from versions 6.0.18-15, 7.0.15-9, and 8.0.4-1, Percona Server for MongoDB prevents secret overwriting by implementing the security.vault.checkMaxVersions configuration file option. The option is enabled by default instructing Percona Server for MongoDB to do an additional verification before putting a new secret on a Vault server. Percona Server for MongoDB checks whether the secret’s current_version on the Vault server reached the effective (more on this below) value of max_versions. If so, it aborts putting a new key to the Vault server, logs an error, and exits. The error text asks the user to raise the max_versions to place more secrets at that particular secret path.
Configuring max_versions
If you want to set max_vesions for the key at the path mongokeys/prod/data/cluster-zulu/host-alpha to some value N, you can do that using one of the two shell commands:
1 | vault kv metadata put -max-versions=N -mount=mongokeys/prod cluster-zulu/host-alpha |
or
1 | vault write mongokeys/prod/config max_versions=N |
The first command updates that secret’s metadata, so it can now have up to N versions. The second one changes the configuration of the whole secrets engine at the path mongokeys/prod. As a result, each secret in that engine can now have up to N versions. But what if one uses both commands with different values of max_versions? Vault uses the maximum of two values as the following table shows:
| Engine’s max_versions | Secret’s max_versions | Effective max_versions |
|---|---|---|
| Not set | Not set | 10 |
| N | Not set | N |
| Not set | M | M |
| N | M | max(N, M) |
As per Vault’s documentation, the maximum value of max_versions is at least 24,000. In practice, it is much higher. I successfully wrote 550,000 encryption-key-like versions in my experiment after raising max_versions on the secrets engine. That should be enough for most deployments to last for decades.
Granting permissions to Percona Server for MongoDB
To determine the effective value of max_versions, Percona Server for MongoDB reads the secret’s metadata and the secrets engine configuration; therefore, it needs appropriate permissions. Providing that the kv-v2 secrets engine is mounted at the path mongokeys/prod, a Vault policy granting those permissions may look as follows:
1 2 3 4 5 6 7 8 9 | path "mongokeys/prod/data/*" { capabilities = ["create", "read", "update"] } path "mongokeys/prod/metadata/*" { capabilities = ["read"] } path "mongokeys/prod/config" { capabilities = ["read"] } |
In the example above, the first block grants read-write permissions for the engine’s secrets themselves. The next block allows for reading metadata about any secret in the engine. Finally, the last block grants read access to the engine’s configuration.
Conclusion
The post described how Percona Server for MongoDB versions 6.0.18-15, 7.0.15-9, and 8.0.4-1 prevent potential key loss on a Vault server. The new versions require extended permissions on a Vault server, but configuring them is not difficult. I recommend upgrading your deployments to the above versions as soon as possible to avoid the situation when the master encryption key is lost and the data can’t be decrypted. If you need assistance, don’t hesitate to contact Percona experts.
Run MongoDB on your terms! We deliver secure, rigorously tested software that includes enterprise-grade advanced features such as encryption and backup, without traditional enterprise constraints.
Learn Why Customers Choose Percona for MongoDB
About the Author
Konstantin TrushinKonstantin is a senior software engineer working on Percona Server for MongoDB. His interests include multithreaded programming and distributed systems. Before Percona, Konstantin had been developing a high-performance network server for the online advertising industry. Outside work, Konstantin enjoys reading and exploring new cities.
