Status

Current state: Committed

Discussion thread: https://lists.apache.org/thread/qsmxsymozymy6dy9tp5xw9gn5fhz9nt4

JIRA: Unable to render Jira issues macro, execution error.

Released: 5.0-alpha1

Prototypes: CQL functions

Motivation

Many users have the need of masking sensitive data, such as contact info, age, gender, credit card numbers, etc. Dynamic data masking (DDM) allows to obscure sensitive information while still allowing access to the masked columns, and without changing the stored data. It is a feature available in multiple databases, but not in Apache Cassandra. This CEP aims to provide DDM for Apache Cassandra using both native and user-defined CQL functions.

DDM will be applied to specific columns once they are read from the storage engine, so unprivileged users won’t be able to see the real data that is stored in the database. Instead, they will see a redacted version of the data, such as “XXXX”. However, the stored data will remain unmasked.

There will be different types of masking, including partial masking. For example, a masking function can redact all the digits of a credit card number except the two last digits. Users will be able to provide their own masking functions if they need something more elaborate or personalized than the standard masking functions.

DDM aims to provide some degree of protection against accidental data exposure. However, it’s not a replacement for other security techniques such as column-level permissions or data encryption.

Audience

Cassandra developers and operators. 

Goals

• Add a set of native CQL functions providing generic masking for all, or almost all, CQL data types.
• Permit attaching the native masking functions to specific table columns, so any query over those columns will return the masked data. 
• Allow the use of UDFs as masking functions attached to specific table columns.
• Add a table-level UNMASK permission allowing certain users to both see and query the unmasked columns, assuming that they also have the SELECT permission on the table.
• Add a table-level SELECT_MASKED permission allowing certain users to query but not see the unmasked columns, assuming that they also have the SELECT permission on the table.
• Prevent users without UNMASK nor SELECT_MASKED permissions to indirectly guess the real value of the masked value by trying values on the WHERE clause of SELECT queries.

Non-Goals

• DDM won’t mask, encrypt nor transform in any way the data that is stored in the database. The data will be dynamically transformed after reading it from the storage engine, and before presenting it to the unprivileged user.
• DDM won’t provide any kind of column-level access control nor finer granularity than what the current authorization system allows. The new UNMASK and SELECT_MASKED permissions will be analogous to the current SELECT permission.
• DDM won’t provide any additional protection against direct access to the sstable files.
• DDM masking functions will always be scalar functions applied column value per column value. We won’t support, for now, more complex strategies such as shuffling column values. However, the proposed CQL syntax is not incompatible with that type of masking if we wanted to add it in the future.

Timeline

Dynamic data masking will target 4.2, with development happening mostly on trunk. It will be developed in the following incremental steps:

1. Add basic native CQL functions for masking data.
2. Allow to attach native masking functions to table columns with CREATE/ALTER TABLE.
3. Add a new UNMASK permission allowing users with that permission to see and query unmasked data. Steps 2 and 3 might be done together or in a short-lived feature branch.
4. Add a new SELECT_MASKED permission allowing users with that permission to query but not see unmasked data.
5. Allow to attach user-defined functions (UDF) as masking functions with CREATE/ALTER TABLE.
6. Maybe add more sophisticated and/or specific masking functions.

Proposed Changes

Masking functions

There will be a set of native CQL functions providing different types of masking for column values, such as replacing the real value of a column by a fixed value. Here is a more detailed list of possible functions:

This list is not exhaustive and of course we can add more functions if we think they are useful.

It would be possible to use the masking functions on SELECT queries, like any other native CQL function. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text,
      birth date);
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-01-02');

> SELECT name, mask_with_default(name), mask_with_partial(name, 2, 1) FROM patients;

 name  | mask_with_default(name) | mask_with_partial(name, 2, 1)
-------+-------------------------+-------------------------------
 alice | XXXX                    | alXXXXe

> SELECT birth, mask_with_default(birth), mask_with_null(birth) FROM patients;

 birth      | mask_with_default(birth) | mask_with_null(birth)
------------+--------------------------+-----------------------
 1982-01-02 | 1900-01-01               | null

These native masking functions have a value on their own, even if we don't attach them to the schema as we’ll see in the next section. So adding them would probably be the first step of the implementation. Once we add a few of them and establish the common superclasses, testing framework etc., adding more functions should be easy. Adding extra functions could be a good source of easy and satisfactory tickets for newcomers.

Masked table columns

It will be possible to attach the masking functions to schema columns. By doing so, all SELECT queries involving those columns would produce masked values, without the need to specify the masking function on the SELECT query. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH partial(2, 1),
      birth date MASKED WITH default()
      );
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21);

> SELECT name, birth FROM patients;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01

In addition to masking native CQL functions, it will also be allowed to use regular UDFs to mask a column. That way users will be able to provide their own masking functions.

It will be possible to change the masking of a column at any moment with the ALTER TABLE command:

> ALTER TABLE patients ALTER name MASKED WITH partial(2, 1);
> ALTER TABLE patients ALTER name WITHOUT MASK;

It won't be possible to use masked columns in the WHERE and IF clauses of SELECT and UPDATE statements without the proper permissions. This is done to prevent malicious users from inferring the underlying data:

> UPDATE patients SET name='alice' WHERE id=ed976860-1e3b-11ed-814b-e5a3bd61806e IF name='alice';  

Unauthorized: Error from server: code=2100 [Unauthorized] message="User has no UNMASK nor SELECT_UNMASK permission on <table k.patients>"

> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING; 

Unauthorized: Error from server: code=2100 [Unauthorized] message="User has no UNMASK nor SELECT_UNMASK permission on <table k.patients>"

Although we might not support it in an initial version, materialized views should inherit the same masking as its base table, so:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );
> CREATE MATERIALIZED VIEW patients_by_name AS 
      SELECT * FROM patients 
      WHERE id IS NOT NULL AND name IS NOT NULL 
      PRIMARY KEY (name, id);
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21);

> SELECT name, birth FROM patients_by_name;

 name | birth
------+------------
 XXXX | 1900-01-01

UNMASK and SELECT_MASKED permissions

There will be a special table-level permission for unmasking values, so users would need that permission to see the clear data. That permission will be named UNMASK. Users having the UNMASK (and SELECT) permission will see the real data, as if the masking function didn’t exist. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );

> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21');

> CREATE USER unprivileged_user WITH PASSWORD 'xyz';
> CREATE USER privileged_user WITH PASSWORD 'zyx';

> GRANT SELECT ON ALL TABLE patients TO unprivileged_user;
> GRANT SELECT ON ALL TABLE patients TO privileged_user;
> GRANT UNMASK ON ALL TABLE patients TO privileged_user;

> LOGIN unprivileged_user
> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING; 

Unauthorized: Error from server: code=2100 [Unauthorized] message="User has no UNMASK nor SELECT_UNMASK permission on <table k.patients>"

> SELECT name, birth FROM patients WHERE id=db2b372f-f91b-4537-b46b-c478f8330c29;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01  
            
> LOGIN privileged_user
> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING;

 name  | birth
-------+------------
 alice | 1982-12-21 

The UNMASK permission will also allow using the masked columns in the WHERE/IF clauses of statements.

There will also be a new SELECT_MASKED permission that will allow using the masked columns in the WHERE/IF clauses of statements. However, it will not allow to see the clear data in the query results. This permission is meant to be granted to not-malicious users that are not trying to exploit the data, but generating masked data for its consumption by other agents. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );

> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21');  

> CREATE USER privileged_user WITH PASSWORD 'zyx';
> GRANT SELECT ON TABLE patients TO privileged_user;
> GRANT SELECT_MASKED ON TABLE patients TO privileged_user;  

> LOGIN privileged_user
> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01

New or Changed Public Interfaces

CQL’s syntax for creating and altering tables will be extended to allow the specification of masked columns. This will be done by adding new clauses to the existing statements, and they will keep backward compatibility.

The IAuthorizer, IAuthenticator and IResource interfaces will be extended with two new types of permission. This should be done without breaking binary compatibility. Existing implementations will keep working as long as DDM is not used. However, implementations will need to be updated if they want to support DDM. Masking table columns will be forbidden if the authenticator or the authorizer don’t explicitly declare that they support it.

Compatibility, Deprecation, and Migration Plan

The new feature will be backward compatible, so anything that worked before should continue working if we don’t use DDM. 

We might forbid enabling DDM during rolling upgrades to avoid problems with the additions to the schema. However, depending on the implementation details, we might allow it so only the upgraded coordinator nodes will mask the data.

Clients reading tables with masked columns should be prepared to receive columns with data types different to those defined in the table metadata, depending on what masking function is used. The default masking function won’t change the column type.

Test Plan

We will include the usual unit and integration tests, testing a reasonable number of use cases. We will also include upgrade tests covering both partial and full upgrades.

Rejected Alternatives

Columns are presented with their original name after masking, so one can't know that they are masked by just looking at them. Users would only know that they are seeing masked data because they know the table schema:

> SELECT name, birth FROM patients;

name     | birth
---------+------------
alXXXXe  | 1900-01-01

An alternative approach would be replacing the name of the column with the call to the function, this way:

> SELECT name, birth FROM patients;

 mask_with_partial(name, 2, 1) | mask_with_default(birth)
-------------------------------+--------------------------
 alXXXXe                       | 1900-01-01

That would make it clear that the column is masked. However, that might be problematic for applications receiving unexpected column names, especially if the column masking has been added after the application was written. Also, keeping the original column name seems to be standard practice of other databases out there.

Other databases

The following table contains some information and links about how other databases out there deal with data masking. The (incomplete) list of databases comes from the top of DB-Engines ranking. Most of the databases on the top of the list support data masking some way or another, with Apache Cassandra being an exception.