Creating a metadata enrichment asset
Configure and run metadata enrichment to add several layers of metadata to your data assets.
You can create a data profile to classify a data asset and compile statistics about the values that it contains. Augment your assets with AI-generated alternative column names and descriptions for data assets and the columns that they contain. Use predefined data quality checks for an initial quality assessment of your data. Enrich assets with business vocabulary that describes the semantic meaning of the data for your organization. Identify relationships between data assets.
- Required services
-
IBM Knowledge Catalog
DataStage for advanced key or relationship analysis and advanced profiling - Required permissions
-
To create and run a metadata enrichment, you must have the Admin or the Editor role in the project, and you must have at least view access to the categories that you want to use in the enrichment. Also, you must be authorized to access the connections to the data sources of the data assets to be enriched.
All operations that are run as part of a metadata enrichment require credentials for secure authorization. Typically, your user API key is used to execute such long-running operations without disruption. If credentials are not available when you create a metadata enrichment or try to run any type of enrichment, you are prompted to create an API key. That API key is then saved as your task credentials. See Managing the user API key.
You can also create metadata enrichments with APIs instead of the user interface. The links to these APIs are listed in the Learn more section.
To create a metadata enrichment asset and a job for enriching data:
-
Open a project and click New asset > Enrich data assets with metadata. After you create the first metadata enrichment in this way, you can add new metadata enrichment assets from the project's Asset page.
-
Define details:
- Specify a name for the metadata enrichment.
- Optional: Provide a description.
- Optional: Select or create tags to be assigned to the metadata enrichment asset to simplify searching. You can create new tags by entering the tag name and pressing Enter.
-
Set the initial data scope.
Select the data assets that you want enrich from Data assets. See Initial data scope. Review the selected scope before you proceed. You can directly delete assets from the data scope or you can rework the entire scope by clicking Edit data scope.
You can skip this step to create an empty metadata enrichment asset, and set the scope later.
-
Define the objective of this metadata enrichment asset. You can add several layers of metadata to a data asset:
- Profile the data to classify it and compile statistics about the values.
- Add alternative names and AI-generated descriptions.
- Enrich assets with business vocabulary that describes the semantic meaning of the data for your organization.
- Run predefined data quality checks for an initial quality assessment.
- Identify primary keys and key relationships.
-
Select categories to determine the business vocabulary that can be applied during the enrichment. See Category selection.
-
Select a sampling type. See Sampling.
-
Define whether you want to run scheduled enrichment jobs. If you don't set a schedule, you run the enrichment when you save the metadata enrichment asset. You can rerun the enrichment manually at any time. See Scheduling options.
-
Select the data scope for the reruns of the enrichment, whether scheduled or run manually. See Scope of reruns of the enrichment
-
Review the metadata enrichment configuration. To make changes, click the Edit icon
on the tile and update the settings.
-
Click Create. The metadata enrichment asset is added to the project, and several jobs are created:
- A metadata enrichment job
- A job for deep primary key analysis named metadata-enrichment-name (PK Detection)
- A job for deep relationship analysis named metadata-enrichment-name (Relationship Detection)
If you didn't configure a schedule, the enrichment is run immediately. If you configured a schedule, the enrichment will run on the defined schedule.
After the enrichment is complete, you can access a high-level overview of the enrichment results by viewing the metadata enrichment asset. From there, you can drill down into and work with the results for each asset. See Working with the enrichment results.
Metadata enrichment is run on assets that are available in the project. Thus, the list of enriched assets might not correspond to the configured scope of included metadata import assets in these cases:
- Metadata import was not yet complete when the enrichment started.
- Metadata import failed for a set of assets or failed completely.
For information about how to update, rerun, or delete a metadata enrichment, see Managing an existing metadata enrichment.
Learn more
Next steps
Parent topic: Managing metadata enrichment