Publishing virtual data to the catalog in Data Virtualization

Last updated: Nov 26, 2024

You can publish your virtual data to catalogs in IBM Knowledge Catalog.

By publishing your virtual data to a catalog, you can complete the following tasks.

Organize, label, classify, and search for the published data assets by using global search.
Govern the virtual data asset and subject it to data protection rules. For more information, see Governing virtual data with data protection rules in Data Virtualization.

In Data Virtualization, you can use two methods to publish assets to a catalog. You can choose to enforce publishing of all assets to a primary catalog or you can allow users to choose to publish to any catalog that they have the Manager or Editor role for.

If you want to enforce publishing to a primary catalog, a Data Virtualization Manager must enable Enforce publishing to a governed catalog in Service settings > Governance and choose a primary catalog that all virtualized objects that are created with the user interface will be published to. If this setting is enabled, users will not be able to choose the catalog that they publish to when they virtualize data. All assets will be published to the primary catalog automatically.

If publishing to a primary catalog is not enforced, a user can choose to publish to any catalog that they have the Manager or Editor role for. The user can choose the catalog from the drop-down list on the Virtualize page.

Note: If you enforce publishing to a primary catalog, the service ID is added as a collaborator on the catalog in the background. The service ID performs the automatic publishing. Therefore, if you enforce publishing to a primary catalog, the service ID will take up one catalog collaborator spot from your plan quota.

Do not remove this service ID from the catalog. It is required for automatic publishing to the primary catalog. The service ID will appear as Unavailable user in your selected primary catalog collaborators list and will have the Manager role assigned.

Before you begin

The creator of the Data Virtualization instance must complete the following tasks.

Enable personal credentials. Navigate to Menu > Administration > Account and enable Personal credentials.
Create an instance of the Cloud Object Storage service and configure IBM Knowledge Catalog to use it. For more information, see Configuring Cloud Object Storage for project and catalog creation.
Authorize the Watson Knowledge Catalog service to access data in your Data Virtualization instance.
1. Log in to your IBM® Cloud account where IBM Knowledge Catalog and Data Virtualization instances are provisioned.
2. From the menu, select Manage > Access (IAM) > Authorizations.
3. Select Create + to create a new service-to-service authorization.
4. Select IBM Knowledge Catalog service as the Source and Data Virtualization service as the Target.
5. Select DataAccess role for service access.
6. Click Authorize.
Ensure that at least one data class exists in your account under Menu > Governance > Data classes.
Ensure that the user who runs profiling runs it from the same IBM Cloud account that the Data Virtualization instance belongs to.

About this task

If you have the Data Virtualization Manager or Engineer roles, when you virtualize data by using the Data Virtualization console, your virtual data is published to a governed catalog automatically if you enforced publishing to a catalog. Otherwise, you can choose a catalog to publish your virtual data to.

Note: If you publish duplicate catalog assets for the same object, you must ensure that all duplicates have the same assigned metadata, such as terms, data classes, tags, and classifications, to avoid inconsistent behavior for enforcement of data protection rules.

Data Virtualization Managers, Engineers, and Stewards can publish virtual data to the catalog manually. Data Virtualization users with the User role can publish virtual objects that they created. Data Virtualization Managers can also enable enforced automatic publishing to a pre-selected catalog in the Data Virtualization service settings.

When you publish a Data Virtualization object to a catalog that does not yet have a Data Virtualization connection, a Data Virtualization connection is added automatically to the target catalog.

Procedure

To manually publish virtual data to the catalog.

On the service menu, click Virtualization > Virtualized data.
Select the virtual objects that you want to publish and click Publish to catalog.
Alternately, click Publish to catalog from the overflow menu of a virtual object.
To publish assets as duplicates, click Publish as duplicates if assets already exist in the catalog.

Remember: If you publish duplicate catalog assets for the same object, you must ensure that all duplicates have the same assigned metadata, such as terms, data classes, tags, and classifications, to avoid inconsistent behavior for enforcement of data protection rules.

Results

When published, virtual views have the type:view tag assigned to them in IBM Knowledge Catalog to distinguish them from the virtual tables.

When you publish a virtual table or view to a catalog, if a Data Virtualization connection from this Data Virtualization instance to the target catalog has not been created, it will be added automatically.

What to do next

If you want catalog collaborators to use virtual objects that you publish to the catalog, you must grant these collaborators access to those virtual objects. For more information, see Managing access to virtual objects.

Restriction:

Data Virtualization access control is not applied when data masking or row-level filtering applies to the preview in Watson services (other than Data Virtualization). The Data Virtualization internal access controls, which are controlled by using Manage access in the Data Virtualization UI, do not apply to the preview from the other Watson services with masking or row-level filtering. You must define your rules to manage access to the catalogs, projects, data assets, or connections for access control in the other Watson services.

The preview is subject to the data protection rules and catalog or project access control only.

Even though a user does not have access to query an object from Data Virtualization, they might be able to preview it in a catalog or project if they have access to that catalog or project the data asset.