Masking flow allows data administrators to produce masked copies of data for data scientists, business analysts, and application testers. Data is protected with data protection rules that apply automatically to all data imported to the catalog.
Masking flow also introduces advanced masking options for data protection rules, such as enhanced format preservation, one-way hash tokenization, ability to maintain relationships, and to increase utility of masked data. Data protection rules with advanced masking work only in projects.
- Required services
- IBM Knowledge Catalog
- Data Privacy (Masking flow)
- Data format
- Relational: Tables in relational data sources
- Data size
- Any size
Before creating masking flows, the data admin must complete these prerequisite tasks.
After the prerequisite tasks are completed, both data admins and data users can do one of the following tasks:
- Create a new project and add data assets to be masked in the project.
- Choose an existing project with data assets.
After completing one of the tasks, click New asset > Copy and mask data.
User roles in Masking flow
As a data administrator (or data engineer), you have a strong knowledge of data assets and data requirements of the data users. You are responsible for preparing data for masking and configuring user access to masked data. See the tasks that data admins must complete.
As a data user, such as data scientists, business analysts, testers, and developers, you rely on the data admin to curate and provide protected data that you need to do your work. See the tasks that data users can do.
Supported data sources
Masking flow supports the following relational and non-relational data sources:
- Apache Hive
- Db2 LUW
- Db2 Warehouse
- MySQL
- Netezza
- Oracle
- PostgreSQL
- SQL Server
- Teradata
Prerequisite tasks for data admins
- Required permissions
- You must be an IBM Cloud account administrator.
At the time that Masking flow is installed, there is at least one admin account set up in your organization. This admin can give other users admin access.
Prepare for privatizing data by completing the following tasks:
-
Add data assets to catalogs by automatically importing data assets with metadata. You create connections to your data in the metadata catalog. When importing the data assets, select the catalog that is created in the previous step as the import target. See Publishing assets from a project into a catalog.
-
Setting up data protection rules. Data protection rules apply to all governed catalogs and are enforced by Masking flow when you create masked copies of data by using masking flows. Advanced data masking options are only enabled for data classes.
-
Managing user access by adding users to an IBM Cloud Account and setting up Cloud Pak for Data as a Service for your organization.
-
Adding data users to catalogs by managing access to a catalog.
Avoiding unintentional data leakage
Moving assets from catalogs to projects
By default, data protection rules are not enforced for the asset owner, the user who added the asset to the catalog. This means for the asset owner, catalog previews are not protected for the data assets that they own.
- When you move an asset from a catalog to a project, the asset in the project is a copy of the catalog asset. Project assets are not linked to data protection rules.
- If the person moving the asset is the asset owner, the asset preview is unmasked for all users in the project.
- If the person moving the asset is not the asset owner, the asset preview is masked for all users in the project.
Because data protection rules aren't enforced for asset owners, when asset owners run a masking flow, the data copy loaded to a target database is not masked. Data is only masked when data users run the masking flow.
Best practice to avoid unintentional data leakage
Consider the following best practices to avoid data leakage:
-
The project used by the admin to import metadata to the catalog should not be used for masking flows. If you want to use the same project for metadata imports and masking flows, ensure that all users in the project have permissions to see unmasked data.
-
Data admins should not move data from catalogs to projects for creating masking flows. Data admins should add data users as viewers to the catalog, and then only data users should move data from the catalog to the project. They can optionally add other users to the project.
Avoiding out-of-memory errors
During a masking flow job, Spark might attempt to read all of a data source into memory. Errors might occur when there isn't you don't enough memory to support the job. The largest volume of data that can fit into the largest deployed Spark processing node is approximately 12GBs.
For the masking flow jobs that have high memory usage, to avoid out-of memory errors:
- Limit the number of executors and size of executors for the job.
- Set the columns in the source table to partition the data.
- When Masking flow jobs involves moving large amount of data, ensure that you select the columns by which data can be partitioned during the masking flow job.
Output truncated to accommodate column length restrictions
The column length is the maximum length that is defined for a column in a database for the string type data.
Previously, the generated masking output did not account for the column length, and the masking flow job would fail if any of the output values surpassed the column length.
Now, the generated output is truncated to ensure that it doesn't exceed column length restrictions.
Prerequisite tasks for data users
Data users must already be a member of the platform or have the level of permission for the data scientist role.
- Required permissions
- You must have an IBM Cloud account and be entitled to IBM Knowledge Catalog Lite plan.
Optional permissions
- Data admins can give you Editor or Viewer access to catalogs.
- Data admins or other data users can also give you access to individual projects that they create.
Prepare masked data copies by completing the following tasks: