Motivation

With the development of enterprises, data security has become increasingly important, and there are some industries that have mandatory legal requirements for it, such as bank, transportation, etc. As a data lake framework, it is very important for paimon in supporting data encryption to meet enterprise security standards. This document describes how to extend the current paimon architecture to provide users with out-of-the-box encryption capabilities.

...

Compatibility, Deprecation, and Migration Plan

Compatibility

By default, plain text mode will be used, allowing users to perform normal read and write operations on previous tables. If the user wants to enable encryption mode, they can specify the corresponding encryption parameters when creating the table or alter table.

Migration

If user wants to migrate previous unencrypted table to encrypted table, they first need to modify the table's properties and add corresponding encryption parameters. Then, during the subsequent compaction rewrite process, the system will change the newly written data to encrypted data file to achieve smooth migration.

Test Plan

Unit Tests

Encrypt and decrypt data using built-in mock kms so that we can conduct unit testing.

Benchmark Tests

Provide test case in the paimon-micro-benchmarks module to verify the read and write performance after encryption.

Rejected Alternatives

.