What is TDE?

The Oracle Transparent Data Encryption feature encrypts the sensitive data (database tables and tablespaces) stored on the disk. This prevents misuse of the data, if the disks or storage mediums are lost or stolen. The data is transparently decrypted for the authorized users when they access the data.

Data is encrypted with the help of encryption keys which are stored in an external module or file, known as the wallet or keystore.  (Note: terms which are in italics on their first usage are summarized in the Definitions section further in this document.) The keystore is managed by an authorized user, and can be either a hardware or software keystore. In order to decrypt the data successfully, the encryption keys must be made available to the database by the authorized user.

In the Oracle database, the data is organized into tables which are located within a tablespace, which is in turn made up of one or more files on disk.  With TDE either individual tables or an entire tablespace can be encrypted.  The keys used to encrypt the data are stored within the database itself, in the data dictionary.  The keys themselves are further encrypted using the wallet keys.  By using 2 layers of encryption in this manner, access to all of the encrypted data can be achieved with just access to the keys in the keystore.

In Figure 2, PDBs 1 and 2 are encrypted, but PDB 3 is not.  Note that the keystore contains all of the keys for the entire multitenant database, in this case keys for the CDB, PDB 1, and PDB 2.

In order to unplug a TDE-encrypted PDB, the key(s) for that PDB must be exported from the keystore into a new file, known as an exported keyfile.  Oracle will prevent the PDB from being unplugged unless the keys have first been exported.  The exported keyfile is encrypted with a password (known as the keyfile secret) that is specified during the export.  Figure 3 illustrates the scenario where PDB 2 has been unplugged.  Note that the key for PDB 2 still remains in the keystore.  This illustrates an important point about Oracle keystores - once added, a key cannot be easily removed from a keystore.

In order to be plugged back into a new container, the keys must first be imported into the new container's keystore.

The design of TDE allows for the keys for a given CDB or PDB to be changed, via the ADMINISTER KEY MANAGEMENT SET KEY command.  This process is known as key rotation, and will update the master encryption key in the keystore, while leaving the table keys in the data dictionary.  The table keys themselves will be encrypted with the new master encryption key, and any future updates to that data will use the new key.  Existing data already written to the datafile or archive log will still be encrypted with the original key, however.  For this reason, keys are never removed from an existing wallet, only new keys are added.  In order to decrypt all of the data in a given database, all current and prior keys will be needed. 

For more information on TDE and how it is configured and enabled, see the Oracle documentation.

Delphix Implementation of TDE in 6.0.6

Oracle recommends that the keystore be stored on a separate disk from the datafiles.  In accordance with this recommendation, neither keystores nor exported keyfiles are stored on Delphix storage.  Rather, they are placed on customer storage.  Exported keyfiles generated by Delphix are stored in the artifact directory under the toolkit directory, while keystores generated by Delphix are stored in the location specified by sqlnet.ora for the target container database.  It is the customer's responsibility to maintain these storage locations and ensure they are backed up as needed, just like database files.  If the keystore or exported keyfile is lost, the data in the associated vPDB may not be recoverable and the vPDB will cease to operate.

Delphix support for TDE as of 6.0.6 has the following requirements:

• TDE-enabled vPDBs must be provisioned to a linked CDB, not a vCDB.
• RAC dSources and target CDBs are not supported.
• The Oracle version must be 12cR2 or higher (12cR1 is not supported).
• System tables or tablespaces (either in the PDB or CDB) must not be encrypted.
• Keystores must not be on ASM storage.
• Migration of a TDE-enabled VPDB from one CDB to another requires manual steps which must be completed for the migrate operation to be successful and to support refresh and rewind operations on the migrated vPDB.
• The location of the keystore must be configured in sqlnet.ora, as opposed to the WALLET_ROOT initialization parameter.  This is true for both the dSource as well as the target container.
• Only software keystores on the same host as the database files are supported; in particular Oracle Key Vault is not supported.
• The dSource from which the initial provision is done must be encrypted when it is linked.  Existing dSources can not be encrypted without unlinking and creating a new dSource.
• There is currently no supported transition path from existing TDE-enabled vPDBs using the TDE workaround to the full product solution.  The TDE workaround continues to be supported in 6.0.6 for approved customers.

Some or all of these restrictions may be relaxed in future versions of Delphix.

Wallet Location Configuration

Oracle requires that the keystore location be specified to the database, so that it can be accessed when reading from or writing to the database files.  This location can be specified in 2 ways:

1. The ENCRYPTION_WALLET_LOCATION parameter in sqlnet.ora
2. The WALLET_ROOT initialization parameter

The WALLET_ROOT initialization parameter is available starting in Oracle 18, while ENCRYPTION_WALLET_LOCATION is available in Oracle 12.2.  Oracle has indicated that support for ENCRYPTION_WALLET_LOCATION will end starting with Oracle 20.  Delphix supports only the configuration in sqlnet.ora via ENCRYPTION_WALLET_LOCATION, for example:

ENCRYPTION_WALLET_LOCATION=
 (SOURCE=
  (METHOD=FILE)
   (METHOD_DATA=
    (DIRECTORY=/u03/app/ora12201/admin/$ORACLE_UNQNAME/wallet/)))

As there is only one sqlnet.ora file found under $ORACLE_HOME, it will be used for all databases that use that home.  This has implications for the configuration for Delphix, as the auxiliary container database used during provisioning will use the same home as the target container database.  As a result, it is necessary that the path to the wallet be unique for each database, even though they are both using the same configuration file.  This is accomplished by using an environment variable which is unique to each database, such as $ORACLE_UNQNAME or $ORACLE_SID.  Such a specification in sqlnet.ora is required for Delphix in 6.0.6.  Any environment variable referenced in sqlnet.ora must always be set in the environment for the Oracle user.  Delphix explicitly sets only $ORACLE_HOME, $ORACLE_SID, and $ORACLE_UNQNAME in the connections which are established by the Delphix engine, so it is recommended that only these variables be referenced in sqlnet.ora.

Required O/S Permissions for the Delphix User

The provision itself is executed within the context of the environment user specified during the provision.  This user does not have to be the Oracle user, and in fact often is not.  The Delphix user must be a member of the oracle group.  During a TDE-enabled vPDB provision, the parent keystore is copied from a user specified location to the location specified in sqlnet.ora.  This copy is done by the Delphix user, and thus the Delphix user must have the necessary O/S privileges to read the parent keystore, as well as the necessary O/S privileges to write into the wallet location.  After the parent keystore is copied, Delphix will invoke Oracle commands to create an autologin wallet.  Since this will be done by the Oracle user, it is necessary that the Oracle user be able to also create files in the wallet location.

The privilege requirements are satisfied by ensuring that the parent keystore has group read privileges, and the wallet location has group write privileges.

In Delphix version 6.0.6, these requirements are necessary for all Oracle versions.  In Delphix version 6.0.7 and Oracle versions 18c or higher, the Delphix engine will create a wallet location specifically for the use of the auxiliary database with the necessary permissions.  Note that for Oracle version 12c the wallet location must have group write privileges for both 6.0.6 and 6.0.7.

Definitions

The following terms are used throughout this document and are summarized here for clarity.  Note that the first occurrence of these terms may be further in the document.

Provisioning a TDE-enabled vPDB

Provisioning a vPDB to a linked container database first involves the user specifying the vPDB parameters, such as the vPDB name and target container, along with the snapshot from which to provision, through the GUI or CLI.  Once the provision job is started with these parameters, the Delphix engine does the following:

1. Mounts the snapshot files on the target host
2. Creates and opens (in mount mode) the auxiliary container database on the target host, using the snapshot files.  The auxiliary container database will have both the CDB and PDB data files from the dSource.
3. Completes recovery to bring the auxiliary container database into a consistent state.
4. Finalizes the state of the auxiliary database and unplugs the vPDB datafiles
5. Plugs the vPDB into the target database, and opens it in read write mode for general use.

1. Mounts the snapshot files on the target host.
2. Creates a keystore with the necessary keys to apply encrypted archived log files.
3. Creates and opens (in mount mode) the auxiliary container database on the target host, using the snapshot files.  The auxiliary container database will have both the CDB and PDB data files from the dSource.
4. Completes recovery to bring the auxiliary container database into a consistent state.
5. Exports the vPDB keys to an exported keyfile to enable unplug.
6. Finalizes the state of the auxiliary database and unplugs the vPDB datafiles.
7. Imports the vPDB keys from the exported keyfile into the target keystore.
8. Plugs the vPDB into the target database, and opens it in read write mode for general use

Figure 4 illustrates the provisioning steps.

To initiate the provision, Delphix needs 4 pieces of information, all of which can be specified in the GUI or CLI:

The target keystore password is specified in the GUI in the Databases tab under Environments:

Click on the pencil icon to set or update the password.

The remaining 3 pieces of information are specified during the provision itself by clicking on the Enabled checkbox:

Once a TDE-enabled vPDB is provisioned, it can be used the same as a non TDE-enabled vPDB within Delphix, with the exception of migrate.  There are however a few caveats:

• A refresh operation will use the parent keystore for the recovery.  If the dSource is rekeyed then the user will need to update the parent keystore with the new keys.  Similarly, if the location or password to the parent keystore has changed then they should be updated before the refresh.
• A rewind operation will use the target keystore for the recovery.  If the vPDB is rekeyed after it is provisioned, then the rekey will update the target keystore, so it does not need to be updated in Delphix.
• Each disable operation will result in the keys being exported to an exported keyfile in the artifact directory, to be used for a subsequent enable.  Refresh and rewind operations will first disable the existing vPDB, so those will also result in a new exported keyfile in the artifact directory.
• Provisioning a second generation vPDB (vvPDB) from a TDE-enabled vPDB is done in the same manner as a first generation vPDB, by specifying the TDE parameters during provision.  The current keystore for the vPDB can be specified as the parent keystore.
  

Example Illustrating Movement of Keys

Consider the example illustrated by the vPDB provision above.  The vPDB tde_vpdb is provisioned from the dSource CDOMSHSR52CAPDB2 on the VM tde-source18, which is an Oracle database running version 18.11.0.  Connecting to the dSource, we can query v$encryption_keys to determine the current keys in use by the each PDB:

SQL> show pdbs

CON_ID CON_NAME               OPEN MODE  RESTRICTED
------ ---------------------- ---------- ----------
     2 PDB$SEED               READ ONLY  NO
     3 CDOMSHSR52CAPDB1       READ WRITE NO
     4 CDOMSHSR52CAPDB2       READ WRITE NO
     5 CDOMSHSR52CAPDB3       READ WRITE NO

SQL> select con_id, key_id from v$encryption_keys order by con_id;

CON_ID KEY_ID
------ ----------------------------------------------------
     1 Ac9MY5kQwU8GvwlYMXImXmMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     3 AedrXL3aUk9zv+9t7J8ZsVYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     4 

     5 AWdc3ZRaP09Pvw4+2FmLwHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

SQL> show pdbs

CON_ID CON_NAME         OPEN MODE  RESTRICTED
------ ---------------- ---------- ----------
     2 PDB$SEED         READ ONLY  NO
     3 CDOMSHTG93CFPDB1 READ WRITE NO
     4 CDOMSHTG93CFPDB2 READ WRITE NO
     5 CDOMSHTG93CFPDB3 READ WRITE NO
     6 TDE_VPDB         READ WRITE NO

SQL> select con_id, key_id from v$encryption_keys order by con_id;

CON_ID KEY_ID
------ ----------------------------------------------------
      1 AZTc9eKqlk98v8GkQ8/AmaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
      6 

The key which was originally present in the wallet on the dSource - AdDdKibL - has been imported into the target keystore, and has a con_id of 6, which corresponds to the con_id of the vPDB.  There are several things to note about the behavior of Oracle and the v$encryption_keys table:

1. Keys are never deleted from existing keystores by Oracle, only new keys are added.  Therefore, if we were to disable the vPDB, which will unmount and unplug it from the CDB, v$encryption_keys will still show the key as present, with its original con_id, even though it has been unplugged:

SQL> show pdbs

CON_ID CON_NAME         OPEN MODE  RESTRICTED
------ ---------------- ---------- ----------
     2 PDB$SEED         READ ONLY  NO
     3 CDOMSHTG93CFPDB1 READ WRITE NO
     4 CDOMSHTG93CFPDB2 READ WRITE NO
     5 CDOMSHTG93CFPDB3 READ WRITE NO

SQL> select con_id, key_id from v$encryption_keys order by con_id;

CON_ID KEY_ID
------ ----------------------------------------------------
     1 AZTc9eKqlk98v8GkQ8/AmaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     6 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA

2. If the wallet is closed for a particular PDB,v$encryption_keys will not show any entries for that PDB. The wallet status can be determined by querying v$encryption_wallet.
3. Querying v$encryption_wallet while the session is attached to CDB$ROOT will return information about the entire CDB, otherwise only the keys for the current PDB are returned.

TDE Artifact Directory

The artifact directory stores the exported keyfiles used during the workflows for TDE-enabled vPDBs.  It is located under the toolkit directory in 6.0.6, in the directory oracle_tde_keystores  (Future releases will allow the artifact directory root to be user-specified for a particular host.).  Each TDE-enabled vPDB will have its own directory within the oracle_tde_keystores directory, identified by the vPDB name, group name, and a unique identifier, separated by an underscore.

For example, if the toolkit directory is /work, the artifact directory for the vPDB tde_vpdb in the group Encrypted could be

/work/oracle_tde_keystores/tde_vpdb_Encrypted_ce7a47e6-8860-4398-bab0-cf0233fc5e3c

Within the artifact directory, there is a subdirectory exported_keys which contains within it the exported keyfiles for each timeflow associated with that vPDB.  Each time an export is performed, a new exported keyfile is generated with a timestamp.  The contents of the artifact directory may change for future releases, but the path to the artifact directory and the naming convention is not anticipated to change.

If a vPDB is moved to a different host (either through the migrate workflow or an enable after a failover), the artifact directory will need to be copied to the new target host.  See the following section for details on the manual steps needed for the migrate workflow.

Maintenance of the Artifact Directory

As the artifact directory root is at the same level as the toolkit directory, it will not be overwritten if a host is refreshed through the Delphix engine and the toolkit updated.  It is the customer's responsibility to maintain the artifact directory and ensure that the contents are not lost, as a disk failure could prevent a TDE-enabled vPDB from being accessed.

The artifact directory is not removed when a TDE-enabled vPDB is deleted; the customer can remove it after confirming that the vPDB has been removed (including from any replicated Delphix engines).

Refreshing and Rewinding a TDE-enabled vPDB

Just like a non TDE-enabled vPDB, a TDE-enabled vPDB can be refreshed from the dSource or rewound to a previous snapshot or point in time.  In each case, no additional manual steps or input from the user is required.  The first step of a refresh or rewind operation is to disable the existing vPDB, which will result in a new keyfile exported to the artifiact directory.  The appropriate snapshot files are then mounted for the auxiliary database so that it can be recovered and brought to a consistent state.  Since the vPDB is TDE-enabled, a keystore is needed for the recover operation.  For a refresh, the Delphix engine will use the parent keystore, and for a rewind the Delphix engine will use the target keystore, as illustrated by Figure 5.

There are 2 potential places for keys to be rotated in a vPDB environment:

1. dSource: If the dSource keys are rotated and a new snapshot taken with the new key, the customer is responsible for updating the parent keystore before refreshing from the later snapshot encrypted with the new key.  The parent keystore would then contain both the new key and the original keys.
2. Target: If the target CDB keys are rotated, the target keystore will be updated.  This is why the Delphix engine uses the target keystore for rewind operations.

In either scenario, the keystore used for recovery will contain the current and all prior keys used to encrypt the datafiles and archive logs, for both the vPDB and CDB used in the auxiliary container.

Manual steps needed for Migrate workflow

Migrating a vPDB allows the customer to move a vPDB from one container database to another.  The new CDB can either be on the same host as the original CDB, or a different host.  For a non-TDE enabled vPDB, there are 3 steps:

1. Disable the vPDB

2. Migrate

3. Enable

1. Move the parent TDE keystore from the original target host to the new host.  If the vPDB is not being migrated to a new host, then this step is not needed.  Similarly, if the new host has the parent keystore in a different location, then the parent keystore path for the vPDB needs to be updated.
2. Move the artifact directory for the vPDB from the original target host to the new host.  If the vPDB is not being migrated to a new host, then this step is not needed.
3. Ensure that the new target container has the TDE keystore password set.
4. Merge the original target keystore into the new target keystore.  This is required to support a rewind operation to a snapshot taken before the migrate.

Figure 6 illustrates the scenario of a migrated TDE-enabled vPDB.  

Example

Consider the following TDE-enabled vPDB:

This vPDB is currently provisioned to the container database CDOMLOTG9620 on the host tde-target18.  Connecting to that database, we can see the keys for the target container and the vPDB:

SQL> show pdbs
CON_ID CON_NAME               OPEN MODE  RESTRICTED
------ ---------------------- ---------- ----------
     2 PDB$SEED               READ ONLY  NO
     3 CDOMLOTG9620PDB1       READ WRITE NO
     4 CDOMLOTG9620PDB2       READ WRITE NO
     5 CDOMLOTG9620PDB3       READ WRITE NO
     6 TDE_VPDB               READ WRITE YES

SQL> select con_id, key_id from v$encryption_keys;
CON_ID KEY_ID
------ ----------------------------------------------------
     1 ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     6 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA

The key for the target container starts with ASFwmcfa and the vPDB key starts with AdDdKibL.  We also have an artifact directory for this vPDB found in /work/oracle_tde_keystores:

$ ls /work/oracle_tde_keystores/
tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334

We are going to migrate this vPDB to the container database CDOMLOTGCAE8 located on the host tde-target18b.  Currently that container has the following configuration:

SQL> show pdbs
CON_ID CON_NAME               OPEN MODE  RESTRICTED
------ ---------------------- ---------- ----------
     2 PDB$SEED               READ ONLY  NO
     3 CDOMLOTGCAE8PDB1       READ WRITE NO
     4 CDOMLOTGCAE8PDB2       READ WRITE NO
     5 CDOMLOTGCAE8PDB3       READ WRITE NO

SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ----------------------------------------------------
     1 Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     3 AetaL+IKpE+ov5avXouApwUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     4 AXZTgaBbxk+6v6x2yHQ4ArgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     5 AayTgbjkJE//v82/hBBBAWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

The new target container has a CDB key starting with Ad344VSU.  The migration steps are as follows:

1. Disable the vPDB in the original container.
2. Migrate the vPDB to point to the new container on the new host.  The UI will report the following warnings during the migrate operation, which indicate the manual steps which need to be done:
   

   

3. Set the TDE keystore password for the new target container CDOMLOTGCAE8.
4. Move the parent keystore to the new target host tde-target18b.  In this case we will keep the existing location /u01/app/oracle/product/18.11.0.0/dbhome_1/admin/CDOMSHSR52CA/wallet on both hosts, so the configuration does not need to be updated.
5. Move the TDE artifact directory /work/oracle_tde_keystores/tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334 to the new target host.  If there are no TDE-enabled vPDBs on the new host, the oracle_tde_keystores directory may need to be created first.
6. Merge the original target keystore into the new target keystore.  Oracle provides the ADMINISTER KEY MANAGEMENT MERGE KEYSTORE command to facilitate this.  Note that you cannot successfully merge into a keystore that is currently in use by the database, so the recommended process is to first copy the original target keystore to the new host, merge the 2 keystores into a new keystore, replace the existing new target keystore and finally bounce the CDB to start using the merged keystore:
1. Copy the original keystore to /tmp/tde-target18a on tde-target18b
2. While connected to the new container, issue the merge keystore command:
   SQL> administer key management merge keystore '/tmp/tde-target18a' identified by *** and keystore '/u01/app/oracle/product/18.11.0.0/dbhome_1/admin/CDOMLOTGCAE8/wallet' identified by *** into new keystore '/tmp/merged' identified by ***;
3. Copy the new merged keystore into the existing keystore location (backing up the existing keystore first).
4. If there is an autologin wallet configured for the container, it must be recreated.
5. Shutdown and restart the new target container database.
7. Confirm that the keystore now contains both the original key and the new key:

SQL> select key_id from v$encryption_keys where con_id = 1;
KEY_ID
----------------------------------------------------
ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Note that we have the key starting with ASFwmcfa from the original container, and the key starting with Ad344VSU from the new container.

8. Enable the vPDB to complete the migration operation.  We can now see that the vPDB is successfully started in the new container, with its key starting with AdDdKibL:

SQL> show pdbs
CON_ID CON_NAME                       OPEN MODE  RESTRICTED
------ ------------------------------ ---------- ----------
     2 PDB$SEED                       READ ONLY  NO
     3 CDOMLOTGCAE8PDB1               READ WRITE NO
     4 CDOMLOTGCAE8PDB2               READ WRITE NO
     5 CDOMLOTGCAE8PDB3               READ WRITE NO
     6 TDE_VPDB                       READ WRITE NO

SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ---------------------------------------------
     1 Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     1 ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     3 AetaL+IKpE+ov5avXouApwUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     4 AXZTgaBbxk+6v6x2yHQ4ArgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     5 AayTgbjkJE//v82/hBBBAWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     6 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
6 rows selected.

After Replication Failover

A failover is similar to a migrate operation for TDE-enabled vPDBs.  If after the failover the original target hosts are still available, then the vPDB can be enabled on the failed over Delphix engine without requiring additional manual steps.  If the vPDB is to be restored to a different target host, then the same manual steps are required as for migrate, as the vPDB will now be located on a new target and thus needs the artifiact directory, parent keystore, and merged keystore available.

#oracle #TDE #6.0.6