Status

Current state: Under Discussion

Discussion thread: https://lists.apache.org/thread/qsmxsymozymy6dy9tp5xw9gn5fhz9nt4

JIRA: here

Released: <Cassandra Version>

Motivation

Many users have the need of masking sensitive data, such as contact info, age, gender, credit card numbers, etc. Dynamic data masking (DDM) allows to obscure sensitive information while still allowing access to the masked columns, and without changing the stored data. It is a feature available in multiple databases, but not in Apache Cassandra. This CEP aims to provide DDM for Apache Cassandra using both native and user-defined CQL functions.

DDM will be applied to specific columns once they are read from the storage engine, so unprivileged users won’t be able to see the real data that is stored in the database. Instead, they will see a redacted version of the data, such as “XXXX”. However, the stored data will remain unmasked.

There will be different types of masking, including partial masking. For example, a masking function can redact all the digits of a credit card number except the two last digits. Users will be able to provide their own masking functions if they need something more elaborate or personalized than the standard masking functions.

DDM aims to provide some degree of protection against accidental data exposure. However, it’s not a replacement for other security techniques such as column-level permissions or data encryption.

Audience

Cassandra developers and operators. 

Goals

• Add a set of native CQL functions providing generic masking for all, or almost all, CQL data types.
• Permit attaching the native masking functions to specific table columns, so any query over those columns will return the masked data. 
• Allow the use of UDFs as masking functions attached to specific table columns.
• Add a table-level UNMASK permission allowing certain users to see the unmasked column values, assuming that they also have the SELECT permission on the table.

Non-Goals

• DDM won’t mask, encrypt nor transform in any way the data that is stored in the database. The data will be dynamically transformed after reading it from the storage engine, and before presenting it to the unprivileged user.
• DDM won’t provide any kind of column-level access control nor finer granularity than what the current authorization system allows. The new UNMASK permission will be analogous to the current SELECT permission.
• DDM won’t prevent users with SELECT permissions to indirectly guess the real value of the masked value by trying values on the WHERE clause of SELECT queries. For example, if we have masked a “birthdate” column in a “patients” table, a user can still find patients within an age range by running a query such as “SELECT * FROM patients WHERE birthdate > ‘1960-01-01’”.
• DDM won’t provide any additional protection against direct access to the sstable files.
• DDM masking functions will always be scalar functions applied column value per column value. We won’t support, for now, more complex strategies such as shuffling column values. However, the proposed CQL syntax is not incompatible with that type of masking if we wanted to add it in the future.

Timeline

Dynamic data masking will target 4.2, with development happening mostly on trunk. It will be developed in the following incremental steps:

1. Add basic native CQL functions for masking data.
2. Allow to attach native masking functions to table columns with CREATE/ALTER TABLE.
3. Add a new UNMASK permission allowing users with that permission to see unmasked data. Steps 2 and 3 might be done together or in a short-lived feature branch.
4. Allow to attach user-defined functions (UDF) as masking functions with CREATE/ALTER TABLE.
5. Maybe add more sophisticated and/or specific masking functions.

Proposed Changes

Masking functions

There will be a set of native CQL functions providing different types of masking for column values, such as replacing the real value of a column by a fixed value. Here is a more detailed list of possible functions:

This list is not exhaustive and of course we can add more functions if we think they are useful.

It would be possible to use the masking functions on SELECT queries, like any other native CQL function. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text,
      birth date);
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-01-02');

> SELECT name, mask_with_default(name), mask_with_partial(name, 2, 1) FROM patients;

 name  | mask_with_default(name) | mask_with_partial(name, 2, 1)
-------+-------------------------+-------------------------------
 alice | XXXX                    | alXXXXe

> SELECT birth, mask_with_default(birth), mask_with_null(birth) FROM patients;

 birth      | mask_with_default(birth) | mask_with_null(birth)
------------+--------------------------+-----------------------
 1982-01-02 | 1900-01-01               | null

These native masking functions have a value on their own, even if we don't attach them to the schema as we’ll see in the next section. So adding them would probably be the first step of the implementation. Once we add a few of them and establish the common superclasses, testing framework etc., adding more functions should be easy. Adding extra functions could be a good source of easy and satisfactory tickets for newcomers.

Masked table columns

It will be possible to attach the masking functions to schema columns. By doing so, all SELECT queries involving those columns would produce masked values, without the need to specify the masking function on the SELECT query. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH partial(2, 1),
      birth date MASKED WITH default()
      );
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21);

> SELECT name, birth FROM patients;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01

In addition to masking native CQL functions, it will also be allowed to use regular UDFs to mask a column. That way users will be able to provide their own masking functions.

It will be possible to change the masking of a column at any moment with the ALTER TABLE command:

> ALTER TABLE patients ALTER name MASKED WITH partial(2, 1);
> ALTER TABLE patients ALTER name WITHOUT MASK;

Masking will only be applied to the selection of columns of SELECT queries, but not to the WHERE part of those queries. INSERT and UPDATE queries won’t be affected by masking at all:

> UPDATE patients SET name='alice' WHERE id=ed976860-1e3b-11ed-814b-e5a3bd61806e IF name='alice';

 [applied]
-----------
      True

> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01

Note that, as it is mentioned in the non-goals section, this might allow users to guess the real value of masked columns. That’s why dynamic data masking is not a replacement for column-level permissions.

Although me might not support it in an initial version, materialized views should inherit the same masking as its base table, so:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );
> CREATE MATERIALIZED VIEW patients_by_name AS 
      SELECT * FROM patients 
      WHERE id IS NOT NULL AND name IS NOT NULL 
      PRIMARY KEY (name, id);
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21);

> SELECT name, birth FROM patients_by_name;

 name | birth
------+------------
 XXXX | 1900-01-01

Unmask permission

There will be a special table-level permission for unmasking values, so users would need that permission to see the clear data. That permission will be named UNMASK. Users having the UNMASK (and SELECT) permission will see the real data, as if the masking function didn’t exist. For example:

> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );

> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21');

> CREATE USER unprivileged_user WITH PASSWORD 'xyz';
> CREATE USER privileged_user WITH PASSWORD 'zyx';

> GRANT SELECT ON ALL TABLE patients TO unprivileged_user;
> GRANT SELECT ON ALL TABLE patients TO privileged_user;
> GRANT UNMASK ON ALL TABLE patients TO privileged_user;

> LOGIN unprivileged_user
> SELECT name, birth FROM patients;

 name    | birth
---------+------------
 alXXXXe | 1900-01-01  
            
> LOGIN privileged_user
> SELECT name, birth FROM patients;

 name  | birth
-------+------------
 alice | 1982-12-21 

New or Changed Public Interfaces

CQL’s syntax for creating and altering tables will be extended to allow the specification of masked columns. This will be done by adding new clauses to the existing statements, and they will keep backward compatibility.

The IAuthorizer, IAuthenticator and IResource interfaces will be extended with a new type of permission. This should be done without breaking binary compatibility. Existing implementations will keep working as long as DDM is not used. However, implementations will need to be updated if they want to support DDM. Masking table columns will be forbidden if the authenticator or the authorizer don’t explicitly declare that they support it.

Compatibility, Deprecation, and Migration Plan

The new feature will be backward compatible, so anything that worked before should continue working if we don’t use DDM. 

We might forbid enabling DDM during rolling upgrades to avoid problems with the additions to the schema. However, depending on the implementation details, we might allow it so only the upgraded coordinator nodes will mask the data.

Clients reading tables with masked columns should be prepared to receive columns with data types different to those defined in the table metadata, depending on what masking function is used. The default masking function won’t change the column type.

Test Plan

We will include the usual unit and integration tests, testing a reasonable number of use cases. We will also include upgrade tests covering both partial and full upgrades.

Rejected Alternatives

Columns are presented with their original name after masking, so one can't know that they are masked by just looking at them. Users would only know that they are seeing masked data because they know the table schema:

> SELECT name, birth FROM patients;

name     | birth
---------+------------
alXXXXe  | 1900-01-01

An alternative approach would be replacing the name of the column with the call to the function, this way:

> SELECT name, birth FROM patients;

 mask_with_partial(name, 2, 1) | mask_with_default(birth)
-------------------------------+--------------------------
 alXXXXe                       | 1900-01-01

That would make it clear that the column is masked. However, that might be problematic for applications receiving unexpected column names, especially if the column masking has been added after the application was written. Also, keeping the original column name seems to be standard practice of other databases out there.

Other databases

The following table contains some information and links about how other databases out there deal with data masking. The (incomplete) list of databases comes from the top of DB-Engines ranking. Most of the databases on the top of the list support data masking some way or another, with Apache Cassandra being an exception.