Manta Data Lineage is now also available in the Sydney region

21 January 2025

Manta Data Lineage is now also available in the Sydney data center. You can select Sydney as your preferred region when you sign up.

For more information about product features that are available in the Sydney region, see Regional availability for services and features.

Deploy models converted from CatBoost and LightGBM to ONNX format

15 January 2024

You can now deploy machine learning and generative AI models that are converted from CatBoost and LightGBM to ONNX format and use the endpoint for inferencing. These models can also be adapted to dynamic axes. For more information, see Deploying models coverted to ONNX format.

New Evaluation Studio tutorial and video

13 Jan 2025

Try the new Evaluation Studio tutorial and video to help you learn how to evaluate and compare the performance of your generative AI assets.

Deprecation of data location and sovereignty rules for Data Privacy

13 January 2025

The data location and sovereignty rules are experimental features that provide attributes-based access control of data assets based on their location or sovereignty. These experimental features are deprecating and might be removed in March 2025. For details, see Data location rules (experimental).

If you have questions or concerns that are related to the deprecation, you can open a support ticket.

Deploy models converted to ONNX format

20 December 2024

You can now deploy machine learning and generative AI models that are converted to ONNX format and use the endpoint for inferencing. These models can also be adapted to dynamic axes. For more information, see Deploying models coverted to ONNX format.

Deploy multi-source SPSS Modeler flows

20 December 2024

You can now create deployments for SPSS Modeler flows that use multiple input streams to provide data to the model. For more information, see Deploying multi-source SPSS Modeler flows.

New data sources for lineage metadata import

12 December 2024

You can now import lineage metadata from the following data sources. After the data is imported, you can visualize it on a lineage graph. For more information, see Supported data sources for curation and data quality.

• MicroStrategy
• OpenLineage

Data quality monitoring and remediation workflows (IBM Knowledge Catalog)

12 December 2024

To focus quality improvement efforts on the data that is most important for your organization, you identify critical data elements, define quality expectations, and ensure remediation of data quality issues.

You can now build data quality SLA rules to:

• Monitor the quality of critical data against specific quality criteria as part of metadata enrichment.
• Trigger remediation workflows if the quality doesn’t meet the expectations. You can work with the default remediation workflow or create custom workflows.

You can view information about SLA rule compliance or violations and the status of remediation tasks on a monitored data asset’s Data quality page.

For more information, see:

• Data quality SLA rules
• Designing metadata enrichment
• Managing workflows for data quality remediation tasks

Remove suggested names and descriptions from the metadata enrichment results (IBM Knowledge Catalog)

12 December 2024

In the metadata enrichment results, you can now remove suggested display names or descriptions in bulk that are suggested when the enrichment is run with the Expand metadata option. See Making bulk changes to metadata enrichment results.

Use data source definitions to manage and protect data that is accessed from connections

12 December 2024

Data source definitions are a new type of asset that you define based on a connection or connected data asset's endpoints. When you create a data source definition, you can monitor where your data is stored across multiple projects, catalogs, or multi-node data sources. You can also apply the correct protection solution (enforcement engine) based on the data source definition. For details, see Data protection with data source definitions.

These new data source definition features are now available in all regions.

Defining a data source definition with a protection solution (IBM Knowledge Catalog)

09 December 2024

A protection solution is a method of enforcing the data protection rules either in governed catalogs or by a deep enforcement solution.

To configure the platform with a deep enforcement solution, you can create a data source definition to set the data source type. The data source type determines which types of connections the data source definition can be associated with and your available protection solution options. For details, see Protection solutions for data source definition.

These new data source definition features are now available in all regions.

Deprecation of features for Masking flow

11 December 2024

The following features are deprecated and are now removed:

• Reversible option is now removed for obfuscating data, which you can later reverse the masking to recover the original values.
• Reversible encryption is no longer available for creating copies of data by creating masking flows and one-way hash tokenization for flexible compliance.
• Decrypt reversible masked data is no longer available.

Updated SPSS Modeler tutorials

11 December 2024

Get hands-on experience with SPSS Modeler by trying the 15 updated SPSS Modeler tutorials.

IBM Knowledge Catalog is available in the Sydney region

09 December 2024

IBM Knowledge Catalog is now also available in the Sydney data center. You can select Sydney as your preferred region when you sign up.

For more information about product features that are available in the Sydney region, see Regional availability for services and features.

IBM DataStage is available in the Sydney region

09 December 2024

DataStage is now generally available in the Sydney data center. You can select Sydney as your preferred region when signing-up.

For more information about the product features that are available in the Sydney region, see Regional availability for services and features.

IBM watsonx.governance is available in the Sydney region

9 December 2024

IBM watsonx.governance is now generally available in the Sydney data center. You can select Sydney as your preferred region when signing-up.

For more information about the product features that are available in the Sydney region, see Regional availability for services and features.

New data sources for lineage metadata import

06 December 2024

You can now import lineage metadata from the following data sources. After the data is imported, you can visualize it on a lineage graph. For more information, see Supported data sources for curation and data quality.

• Amazon RDS for Oracle
• Amazon RDS for PostgreSQL
• IBM Cloud Databases for PostgreSQL
• IBM Db2
• IBM Db2 for z/OS
• IBM Db2 on Cloud
• Oracle
• PostgreSQL
• Tableau

Deprecation of IBM Cloud Object Storage Lite plan (IBM Knowledge Catalog)

05 December 2024

The Cloud Object Storage Lite plans with unlimited duration that you provisioned prior to 1 July 2024 are deprecated and might be removed after 15 December 2024. To retain your data and other assets, you must upgrade your Cloud Object Storage service to a Standard plan before 15 December 2024. If you do not upgrade your Cloud Object Storage plan to Standard, your workspaces might become inaccessible after 15 December 2024, and your data might be permanently deleted. For more information, see Lite plan (deprecated).

Microsoft Excel files are deprecated for OPL models in Decision Optimization

05 December 2024

Microsoft Excel workbook (.xls and .xlsx) files are now deprecated for direct input and output in Decision Optimization OPL models. To connect to Excel files, use a data connector instead. The data connector transforms your Excel file into a .csv file. For more information, see Referenced data.

Enhanced scheduling of metadata enrichment jobs (IBM Knowledge Catalog)

05 December 2024

You can now configure execution windows for your metadata enrichment jobs to balance workloads. Jobs then run only within the configured time frames. For more information, see Managing scheduling of enrichment jobs.

Segment data assets by column values to focus on the information you need (IBM Knowledge Catalog)

05 December 2024

You can now chunk data assets into smaller data assets based on selected column values to help you access only the data that you’re interested in. For more information, see Creating data assets by segmenting column data.

New sample notebooks for deploying models converted to ONNX format

03 December 2024

You can now deploy machine learning and generative AI models that are converted to ONNX format and use the endpoint for inferencing. These models can also be adapted to dynamic axes. See the following sample notebooks:

• Convert ONNX neural network from fixed axes to dynamic axes
• Use ONNX model converted from PyTorch
• Use ONNX model converted from TensorFlow to recognize hand-written digits

For more information, see watsonx.ai Runtime Python client samples and examples.

Improved documentation on write options for Data Refinery

28 November 2024

The write options and table options for exporting data flows depends on your connection. These options are now explained so that you are better guided to select your target table options. For more information, see Target connection options for Data Refinery.

Name change for Watson Query service

25 November 2024

The Watson Query service was renamed to Data Virtualization Data Virtualizion.

Name change for Watson Studio and Watson Machine Learning services

21 November 2024

The following services were renamed:

• Watson Machine Learning is now named watsonx.ai Runtime.
• Watson Studio is now named watsonx.ai Studio.

Some videos, notebooks, and code samples might continue to refer to these services by their former names.

Cloud Pak for Data as a Service is available in the Sydney region

21 November 2024

Cloud Pak for Data as a Service is now generally available in the Sydney data center with watsonx.ai Runtime and watsonx.ai Studio services. When you sign-up, you can select Sydney as the preferred region.

Not all of the services are available in the Sydney region yet. For more information about product features that are available in the Sydney region, see Regional availability for services and features.

Enhanced monitoring of metadata enrichment jobs (IBM Knowledge Catalog)

21 November 2024

On the new run metrics dashboard, you can monitor the progress of the individual enrichment tasks for an active metadata enrichment job run. In addition, you can explore run information for completed job runs to identify if and where issues occurred. For more information, see Monitoring runs of enrichment jobs.

Promote SPSS Modeler flows to deployment spaces

19 November 2024

You can now directly promote SPSS Modeler flows from projects to deployment spaces without having to export the project and then import it into the deployment space. For more information, see Promoting SPSS Modeler flows and models.

Task credentials are now required to deploy assets and run jobs from a deployment space

11 November 2024

To improve the security for running deployment jobs, you must enter your task credentials to deploy the following assets from a deployment space:

• Prompt templates
• AI services
• Models
• Python functions
• Scripts

Additionally, you must enter your task credentials to create the following deployments from your deployment space:

• Online
• Batch

You must also use your task credentials to create and manage deployment jobs from your deployment space.

To learn how to set up your task credentials and generate an API key, see Adding task credentials.

Editor mode for custom properties (IBM Knowledge Catalog)

14 November 2024

When viewing governance artifacts, you can now switch on the editor mode for custom properties. When the Edit values toggle is switched off in the Details section, you can only see those custom properties for which values were definded for the artifact. Switch the editor mode on, and you can see all available custom properties and edit their values. For more information, see Custom properties, relationships, and asset types.

Connect to new data sources with SPSS Modeler

7 November 2024

You can now connect SPSS Modeler to Databricks and Microsoft Azure Synapse Analytics, and SPSS Modeler has read and write access to both data sources. For more information, see Microsoft Azure Databricks connection and Microsoft Azure Synapse Analytics connection.

Deprecation of IBM Runtime 23.1

28 October 2024

IBM Runtime 23.1 is deprecated. Beginning November 21, 2024, you cannot create new notebooks or custom environments by using 23.1 runtimes. Also, you cannot create new deployments with software specifications that are based on the 23.1 runtime. To ensure a seamless experience and to leverage the latest features and improvements, switch to IBM Runtime 24.1.

• For information about changing environments, see Changing notebook environments.
• For details on deployment frameworks, see Managing frameworks and software specifications.

Compare tables in Decision Optimization experiments to see differences between scenarios

23 October 2024

You can now compare tables in a Decision Optimization experiment in either the Prepare data or Explore solution view. This comparison can be useful to see data value differences between scenarios displayed next to each other.
For more information, see Compare scenario tables.

Account resource scoping is enabled by default

17 October 2024

The Resource scope setting for your account is now set to ON by default. However, if you previously set the value for the Resource scope setting to either ON or OFF, the current setting is not changed.

When resource scoping is enabled, you can’t access projects that are not in your currently selected IBM Cloud account. If you belong to more than one IBM Cloud account, you might not see all your projects listed together. For example, you might not see all your projects on the All projects page. You must switch accounts to see the projects in the other accounts.

Analyze Japanese text data in SPSS Modeler with Text Analytics

9 October 2024

You can now use the Text Analytics nodes in in SPSS Modeler, such as the Text Link Analysis node and Text Mining node, to analyze text data written in Japanese.

Introducing IBM Manta Data Lineage: a new service that provides data lineage for your data

04 October 2024

IBM Manta Data Lineage is a data lineage service that increases data pipeline transparency so you can determine data accuracy throughout business models and systems. For information about data lineage, see Data lineage.

This service requires IBM Knowledge Catalog service and enabling data lineage on your IBM Cloud account. See, Enable data lineage. It is available only in the Dallas region.

You can access your imported lineages in the new workspace Data lineage or view lineage for a specific asset through Catalogs or Projects page.

You can import lineage metadata from the following sources:

• Microsoft Azure SQL Database connection
• Microsoft SQL Server connection
• Microsoft Power BI (Azure) connection
• Snowflake connection
• InfoSphere DataStage
• IBM DataStage for Cloud Pak for Data

For more information about metadata import, see Importing metadata.

Improved Draft tab for governance artifacts (IBM Knowledge Catalog)

3 October 2024

For each artifact type, you can now view all available drafts in the Draft tab. To view it, select the artifact type from the main menu and click Draft. The tab is visible only if you have the required permissions and if any drafts are available. When viewing all your drafts in the tab, you can select multiple drafts and use the bulk actions menu to edit or process them at once. Note that the All drafts page is no longer available from the main menu. For more information, see Managing governance artifacts.

Bulk actions on catalog assets (IBM Knowledge Catalog)

3 October 2024

You can now edit and remove classifications and custom properties for multiple assets in a catalog at the same time.

Automatically updated common properties of data assets (IBM Knowledge Catalog)

3 October 2024

With global asset identification, you can ensure that the common properties of data assets that have the same resource key and reference the same physical resource stay the same even if they're in different projects or catalogs. This way, you can manage such data assets properly and consistently. For more information, see Globla asset identification.

Assign user groups as asset members (IBM Knowledge Catalog)

3 October 2024

You can now assign user groups as asset members. Previously, you could add only individual catalog users as asset members.

Upload and update assets in bulk (IBM Knowledge Catalog)

3 October 2024

To upload and update multiple assets in bulk, you can now import and export CSV files with either asset metadata details or asset relationship details, or both. For more information, see Adding and updating assets and asset metadata from CSV files to catalogs).

Availability of watsonx.governance plan in Frankfurt region and deprecation of OpenScale legacy plan

3 October 2024

The watsonx.governance legacy plan to provision Watson OpenScale in the Frankfurt region is deprecated. IBM Watson OpenScale will no longer be available for new subscription or to provision new instances. For OpenScale capabilities, subscribe to the watsonx.governance Essentials plan, which is now available in Frankfurt as well as Dallas.

• To view plan details, see watsonx.governance plans.
• To get started, see Provisioning and launching watsonx.governance.

Notes:

• Existing legacy plan instances will continue to operate and will be supported until the End of Support date which remains to be determined.
• Existing customers on IBM Watson OpenScale can continue to open support tickets using IBM Watson OpenScale.

Updated environments and software specifications

3 October 2024

The Tensorflow and Keras libraries that are included in IBM Runtime 23.1 are now updated to their newer versions. This might have an impact on how code is executed in your notebooks. For details, see Library packages included in watsonx.ai Studio (formerly Watson Studo) runtimes.

Runtime 23.1 will be discontinued in favor of IBM Runtime 24.1 later this year. To avoid repeated disruption we recommend that you switch to IBM Runtime 24.1 now and use related software specifications for deployments.

• For information about changing environments, see Changing notebook environments.
• For details on deployment frameworks, see Managing frameworks and software specifications.

New menu terms to open the Platform connections page

03 October 2024

Previously the path to the Platform connections page in the navigation menu was Data > Platform connections. The new path is Data > Connectivity. The Connectivity page has a tab for Platform connections.

Use data source definitions to manage and protect data that is accessed from connections

04 October 2024

Data source definitions are a new type of asset that you define based on a connection or connected data asset's endpoints. When you create a data source definition, you can monitor where your data is stored across multiple projects, catalogs, or multi-node data sources. You can also apply the correct protection solution (enforcement engine) based on the data source definition. For details, see Data protection with data source definitions.

These new data source definition features are available only in the Dallas region.

Defining a data source definition with a protection solution (IBM Knowledge Catalog)

04 October 2024

A protection solution is a method of enforcing the data protection rules either in governed catalogs or by a deep enforcement solution.

To configure the platform with a deep enforcement solution, you can create a data source definition to set the data source type. The data source type determines which types of connections the data source definition can be associated with and your available protection solution options. For details, see Protection solutions for data source definition.

These new data source definition features are available only in the Dallas region.

Review and manage data class and term assignments in a spreadsheet (IBM Knowledge Catalog)

04 October 2024

If you prefer to work in a familiar spreadsheet program when you review and update metadata enrichment results, you can now install the Review metadata add-in for Microsoft Excel. Use the spreadsheet template provided with the product in combination with the add-in:

• To download enriched data assets from a specific project and metadata enrichment.
• To review and update suggested and assigned data classes and terms for these data assets.
• To upload the updated data assets to the project.

For more information, see Reviewing and updating enrichment results in an external program.

Removal of Spark 3.3 runtime

23 September 2024

Support for Spark 3.3 runtime in IBM Analytics Engine will be removed by October 29, 2024 and the default version will be changed to Spark 3.4 runtime. To ensure a seamless experience and to leverage the latest features and improvements, switch to Spark 3.4.

Beginning October 29, 2024, you cannot create or run notebooks or custom environments by using Spark 3.3 runtimes. Also, you cannot create or run deployments with software specifications that are based on the Spark 3.3 runtime.

• To upgrade your instance to Spark 3.4, see Replace Instance Default Runtime.
• For details on available notebook environments, see Changing the environment of a notebook.
• For details on deployment frameworks, see Managing frameworks and software specifications.

Group data quality rules (IBM Knowledge Catalog)

20 September 2024

You can now group certain types of data quality rules into a single DataStage flow and run them together. For more information, see Grouping rules.

Create batch jobs for SPSS Modeler flows in deployment spaces

10 September 2024

You can now create batch jobs for SPSS Modeler flows in deployment spaces. Flows give you the flexibility to decide which terminal nodes to run each time that you create a batch job from a flow. When you schedule batch jobs for flows, the batch job uses the data sources and output targets that you specified in your flow. The mapping for these data sources and outputs is automatic if the data sources and targets are also in your deployment space. For more information about creating batch jobs from flows, see Creating deployment jobs for SPSS Modeler flows.

For more information about flows and models in deployment spaces, see Deploying SPSS Modeler flows and models.

Change pipeline node shape

30 August 2024

You can now change pipeline nodes' appearance to turn them from uniform card style into more compact sized shapes which reflect the type of node. For more information, see Pipelines settings.

Create global parameter sets

30 August 2024

You can now add PROJDEF parameters to your pipeline parameter sets. The parameters can be referenced from both DataStage and Orchestration Pipelines flows at the same project level. For more information, see Configuring global objects for Orchestration Pipelines.

Add user groups as collaborators in projects and spaces

22 August 2024

You can now add user groups as collaborators in projects and spaces if your IBM Cloud account contains IAM access groups. Your IBM Cloud account administrator can create access groups, which are then available as user groups in projects. While creating a project, you must leave the Restrict who can be a collaborator option enabled to add user groups as collaborators. For more information, see Working with IAM access groups.

Support ending for anomaly prediction feature for AutoAI time-series experiments

19 August 2024

The feature to predict anomalies (outliers) in AutoAI time-series model predictions, currently in beta, is deprecated and will be removed on Sep 23, 2024. Standard AutoAI time-series experiments are still fully supported. For details, see Building a time series experiment.

Assign classifications in metadata enrichment (IBM Knowledge Catalog)

22 August 2024

You can now assign classifications to data assets and columns in metadata enrichment, either automatically based on term or data-class assignment or manually in the enrichment results. See Designing metadata enrichment: Assign terms and classifications.

Archive and unarchive projects and spaces

16 August 2024

Projects and spaces are now archived after 90 days of inactivity to preserve resources. To work with such projects or spaces again, unarchive them by opening them directly on the project or space page. Depending on the size of the project or space, unarchiving might take a varied amount of time.

Configure asset removal

16 August 2024

Now, when you create a new catalog, you can also decide how you want to configure the removal of assets. You can either select to purge the assets automatically either immediately after the removal or 30 days after the removal. For previously created catalogs, you can change asset removal settings on the catalog Settings page.

For more information, see:

• Creating a catalog
• Changing catalog settings
• Removing an asset from a catalog

Task credentials are now required to run jobs in a deployment space

15 August 2024

To improve the security for running deployment jobs, you must enter your task credentials to run job in a deployment space. For more information, see Creating jobs in deployment spaces.

To learn how to set up your task credentials and generate an API key, see Adding task credentials.

Pausing metadata enrichment (IBM Knowledge Catalog)

25 July 2024

You can now pause and resume metadata enrichment job runs. For details, see Pausing and resuming enrichment job runs.

Announcing support for Python 3.11 and R4.3 frameworks and software specifications on runtime 24.1

25 July 2024

You can now use IBM Runtime 24.1, which includes the latest data science frameworks based on Python 3.11 and R 4.3, to run Jupyter notebooks and R scripts, and train models. Starting on July 29, you can also run deployments. Update your assets and deployments to use IBM Runtime 24.1 frameworks and software specifications.

• For information on the IBM Runtime 24.1 release and the included environments for Python 3.10 and R 4.2, see Notebook environments.
• For details on deployment frameworks, see Managing frameworks and software specifications.

Enhanced version of Jupyter Notebook editor is now available

25 July 2024

If you're running your notebook in environments that are based on Runtime 24.1, you can use these enhancements to work with your code:

• Automatically debug your code
• Automatically generate a table of contents for your notebook
• Toggle line numbers next to your code
• Collapse cell contents and use side-by-side view for code and output, for enhanced productivity

For more information, see Jupyter notebook editor.

Natural Language Processor transformer embedding models supported with Runtime 24.1

25 July 2024

In the new Runtime 24.1 environment, you can now use natural language processing (NLP) transformer embedding models to create text embeddings that capture the meaning of a sentence or passage to help with retrieval-augmented generation tasks. For more information, see Embeddings.

New specialized NLP models are available in Runtime 24.1

25 July 2024

The following new, specialized NLP models are now included in the Runtime 24.1 environment:

• A model that is able to detect and identify hateful, abusive, or profane content (HAP) in textual content. For more information, see HAP detection.
• Three pre-trained models that are able to address topics related to finance, cybersecurity, and biomedicine. For more information, see Classifying text with a custom classification model.

Extract detailed insights from large collections of texts by using Key Point Summarization

25 July 2024

You can now use Key Point Summarization in notebooks to extract detailed and actionable insights from large collections of texts that represent people’s opinions (such as product reviews, survey answers, or comments on social media). The result is delivered in an organized, hierarchical way that is easy to process. For more information, see Key Point Summarization

RStudio version update

25 July 2024

To provide a consistent user experience across private and public clouds, the RStudio IDE for the Cloud Pak for Data as a Service will be updated to RStudio Server 2024.04.1 and R 4.3.1 on July 29, 2024. The new version of RStudio provides a number of enhancements and security fixes. See the RStudio Server 2024.04.1 release notes for more information. While no major compatibility issues are anticipate, users should be aware of the version changes for some packages described in the following table below.

When launching the RStudio IDE from a project after the upgrade, reset the RStudio workspace to ensure that the library path for R 4.3.1 packages is picked up by the RStudio Server.

Tracking data protection rule enforcement decisions

9 July 2024

You can now track enforcement decisions as audit events when the Send policy evaluations to audit logs checkbox is selected from the Managing rule settings page.

Connectors grouped by data source type

05 July 2024

When you create a connection, the connectors are now grouped by data source type so that the connectors are easier to find and select. For example, the MongoDB data source type includes the IBM Cloud Databases for MongoDB and the MongoDB connectors.

In addition, a new Recents category shows the six latest connectors that you used to create a connection.

For instructions, see Adding connections to data sources in a project or Adding connections to data sources in a catalog.

Bulk edits for governance artifact properties

05 July 2024

You can now change the primary or secondary category for multiple governance artifacts at once. Bulk edits are also available when updating relationships. For more information, see Managing governance artifacts.

Setting an assignment threshold for results of relationship analyses (IBM Knowledge Catalog)

05 July 2024

You now also set a threshold for when results of a relationship analysis should be assigned automatically. You can set a project default but overwrite the setting for each analysis run. For details, see Identifying relationships.

Changes to Cloud Object Storage Lite plans

01 July 2024

Starting on 1 July 2024, the Cloud Object Storage Lite plan that is automatically provisioned when you sign up for a 30 day trial of Cloud Pak for Data as a Service expires after the trial ends. You can upgrade your Cloud Object Storage Lite instance to the Standard plan with the Free Tier option at any time during the 30 day trial.

Existing Cloud Object Storage service instances with Lite plans that you provisioned prior to 1 July 2024 will be retained until 15 December 2024. You must upgrade your Cloud Object Storage service to a Standard plan before 15 December 2024.

See Cloud Object Storage service plans.

Adding catalog assets to projects

20 June 2024

Added a Add catalog assets to projects user permission. Now, to add assets to projects, you must have the Add catalog assets to projects, the Admin, Editor or Viewer role in the catalog, and be the asset owner or editor. Users that don't have an existing role with the Manage catalogs or Access catalogs permission must be explicitly granted the Add catalog assets to projects permission.

Cognos Dashboard removal postponed

20 June 2024

Any existing dashboards that you created with the Cognos Dashboards Embedded service will now continue working until 30 September 2024. You can no longer provision an instance of the Cognos Dashboards Embedded service. You can use Cognos Analytics on Cloud On-Demand as a replacement for Cognos Dashboards Embedded. For more information, see IBM Cognos Analytics Pricing Plans.

Task credentials will be required for deployment job requests

19 Jun 2024

To improve security for running deployment jobs, the user requesting the job will be required to provide task credentials in the form of an API key. The requirement will be enforced starting August 15, 2024. See Adding task credentials for details on generating the API key.

Enhanced data enrichment in IBM Knowledge Catalog

20 Jun 2024

In addition to the existing capabilities, metadata enrichment now provides options for semantic and AI-augmented data enrichment:

• Recommend descriptive names for tables and columns based on the collected metadata and a predefined glossary.
• Suggest and assign semantic descriptions for the contents of tables and columns based on the surrounding columns and the context of the tables.
• Complete semantic term assignment for tables and columns.

For details, see Designing metadata enrichments.

These new gen AI based metadata enrichment features are available only in the Dallas region.

IBM Federated Learning Python client change

20 Jun 2024

Federated Learning's Python client library has been merged with the watsonx.ai library. Your code samples must be updated with the newest Python client. See Connecting to the aggregator.

Connect to a new data source in DataStage: IBM Planning Analytics

14 Jun 2024

You can now include data from an IBM Planning Analytics data source in your DataStage flows.

For the full list of DataStage connectors, see Supported data sources in DataStage.

Bulk edits for governance artifacts

7 Jun 2024

You can now make changes to multiple governance artifacts at once when you want to edit tags or stewards. For more information, see Managing governance artifacts.

Changing parent category for individual artifacts

7 Jun 2024

When viewing artifact details, you can now change the parent category by selecting Move to from the three-dot action menu.

Data protection rules no longer enforced in projects

7 June 2024

Data protection rules are now only enforced either in governed catalogs or by a deep enforcement solution. A deep enforcement solution is a protection solution to enforce rules on data that is outside of Cloud Pak for Data when the data source is integrated with one of these services:

• IBM Data Virtualization
• IBM watsonx.data

Assets that are added into projects from a governed catalog no longer have preview, download or profiling restricted by data protection rules unless you have configured a deep enforcement solution.

You will be reminded of the revised data protection rule enforcement protocols when you:

• Creating a data protection rule.
• Copying an asset from a governed catalog into a project

For details, see Accept revised protocol for enforcing data protection rules.

Adding related items to data quality rules (IBM Knowledge Catalog)

7 June 2024

You can now add data assets or columns with the new relationship type Validates data quality of to any type of data quality rule to have the quality score and any data quality issues reported for this item on the Data quality page. With this enhancement, data quality rules with externally managed bindings and SQL-based data quality rules can now also contribute to the quality scores of assets and columns.

For details, see Creating rules from data quality definitions and Creating SQL-based data quality rules.

Managing reports settings

6 June 2024

IBM Cloud account owners or administrators can now manage the reports settings on the Account page. For more information, see Managing your account settings.

IBM Watson Pipelines is now IBM Orchestration Pipelines

30 May 2024

The new service name reflects the capabilities for orchestrating parts of the AI lifecycle into repeatable flows.

Tag projects for easy retrieval

31 May 2024

You can now assign tags to projects to make them easier to group or retrieve. Assign tags when you create a new project or from the list of all projects. Filter the list of projects by tag to retrieve a related set of projects. For more information, see Creating a project.

Connect to a new data source: Milvus

31 May 2024

Use the Milvus connection to store and confirm the accuracy of your credentials and connection details to access a Milvus vector store. For information, see Milvus connection.

Asset user and role

24 May 2024

Updated the asset membership roles for catalogs. Now, users can hold the asset owner, asset editor, or asset viewer role. The asset editor role replaced the asset member role. Now, to complete any asset-related actions, you must be an asset owner or asset editor.

Also, assets might have more than one owner now.

You can change asset user roles on the Access control page of an asset by selecting a role from the Role dropdown menu.

Bulk actions on catalog assets

24 May 2024

You can now edit and remove the business terms, owners or tags on up to 20 catalog assets at a time.

New filters for enrichment results (IBM Knowledge Catalog)

10 May 2024

You can now apply additional filters to your enrichment results:

• Assigned, suggested, or no business terms
• Assigned, suggested, or no data class

Name changes for DataStage connections and connectors

10 May 2024

The following DataStage connections and connectors have new names:

• "Apache Cassandra (optimized)" is now "Apache Cassandra for DataStage".
• "IBM Db2 (optimized") is now "IBM Db2 for DataStage".
• "IBM Netezza Performance Server (optimized)" is now "IBM Netezza Performance Server for DataStage".
• "Oracle (optimized)" is now "Oracle Database for DataStage".
• "Salesforce.com (optimized)" is now "Salesforce API for DataStage".
• "Teradata (optimized)" is now "Teradata database for DataStage".

Your previous settings for the connections, connectors, and their associated jobs remain the same. Only the connection and connector names have changed.

Name change for the IBM Watson Query connection

26 Apr 2024

The "IBM Watson Query" connection has been renamed to "IBM Data Virtualization". Your previous settings for the connection remain the same. Only the connection name has changed.

Name change for the DataStage IBM Watson Query connector

26 Apr 2024

The DataStage "IBM Watson Query" connector name has changed to "IBM Data Virtualization". This change coincides with the connection name change. Your previous settings for the connection, connector, and the associated jobs remain the same. Only the connection and connector name have changed.

Masking watsonx.data in IBM Knowledge Catalog

26 Apr 2024

You can protect sensitive data in watsonx.data by using masking capabilities of IBM Knowledge Catalog. For more information, see Masking watsonx.data assets in IBM Knowledge Catalog.

Enhanced project list view in catalogs

18 Apr 2024

Now, when you are adding assets from a catalog to a project, you can view more than 100 projects in your project list page and add up to 50 assets at a time to your project. For more information, see Add assets from within the catalog.

Evaluate machine learning deployments in spaces

18 Apr 2024

Configure watsonx.governance evaluations in your deployment spaces to gain insights about your machine learning model performance. For example, evaluate a deployment for bias or monitor a deployment for drift. When you configure evaluations, you can analyze evaluation results and model transaction records directly in your spaces.

For more information, see Evaluating deployments in spaces.

19 Apr 2024

Factsheets available from AI use cases on main navigation menu

Factsheets that track lifecycle details for machine learning models are now stored in AI uses cases rather than model use cases. AI use cases and external models are displayed on the main navigation menu for easy access.

Revised data protection rule enforcement protocol across Cloud Pak for Data

12 Apr 2024

A revised version of the data protection rule enforcement protocol is now in place across Cloud Pak for Data. When you're inside of a governed catalog and click Add to project, information about the new data protection rule enforcement protocol appears. You must acknowledge it to continue.

Cognos Dashboards Embedded service is deprecated

11 Apr 2024

You can no longer provision an instance of the Cognos Dashboards Embedded service. However, any existing dashboards that you created with the Cognos Dashboards Embedded service will continue working until 20 June 2024. You can use Cognos Analytics on Cloud On-Demand as a replacement for Cognos Dashboards Embedded. For more information, see IBM Cognos Analytics Pricing Plans.

Use pivot tables to display data aggregated in Decision Optimization experiments

5 Apr 2024

You can now use pivot tables to display both input and output data aggregated in the Visualization view in Decision Optimization experiments. For more information, see Visualization widgets in Decision Optimization experiments.

Access the list of connection API properties from the user interface

05 Apr 2024

Previously the only way to view the connection properties was to open a new web page at https://dataplatform.cloud.ibm.com/connections/docs. Now you can access the same information from Data > Connectivity. Expand Connection resources, and select Connection properties.

You can use these properties to create connections with the connections in the Watson Data API. For example, if you create a connection in a notebook programmatically, you can use this information to identify the properties that you need.

Create dynamic views of connected data (IBM Knowledge Catalog)

21 March 2024

A new type of connected data asset provides filtered access to data from data sources that support SQL queries so you can access only relevant data. In a project, provide an SQL query to create a view of specific columns or rows from one or more tables. You can use these data assets in metadata enrichment and data quality analysis just like any other connected data asset.

For more information, see Adding a dynamic view of connected data to a project.

Use Delta Lake or Apache Iceberg table formats in the Amazon S3 and the Apache HDFS connectors

22 March 2024

The Amazon S3 and the Apache HDFS connectors now include properties for the Delta Lake and Apache Iceberg table formats. These table formats are integral to data lakes, which provide a centralized repository for managing large data volumes. Data lakes serve as a foundation for collecting and analyzing structured, semi-structured, and unstructured data in its original format for long-term storage and to drive insights and predictions.

The table format property is included in the interaction properties for the supported tools. For example, in the connector Stage properties in DataStage.

New Watson OpenScale tutorial and video

23 Feb 2024

Try the new Watson OpenScale tutorial to help you learn how to evaluate a machine learning model for fairness, accuracy, drift, and explainability.

Access data from DataStax Enterprise

23 Feb 2024

You can now work with data from DataStax Enterprise.

Case-sensitive codes in reference data sets in IBM Knowledge Catalog

16 Feb 2024

Reference data values consist of at least two columns: code and value. For all new reference data sets the code column is now case-sensitive. When you add values to a new reference data set, the code is saved exactly as you type it. Note that any reference data sets that were created before this change was introduced remain case-insensitive, and any new values added there will be saved in upper case. These reference data sets are marked with a Case-insensitive tag in the UI. For details, see Case-sensitive code.

Improved search, filter and sort options for reference data sets in IBM Knowledge Catalog

16 Feb 2024

When you view a list of reference data values, you can use the following methods to find the required values faster:

• Use a search bar to type a query for a code, value or a custom column value.
• Use one of the 6 advanced filter options.
• Use the sorting feature.

The search, filter, and sort options can be combined. For details, see Viewing reference data sets.

New Spark 3.4 environment for running Data Refinery flow jobs

09 Feb 2024

When you select an environment for a Data Refinery flow job, you can now select Default Spark 3.4 & R 4.2, which includes enhancements from Spark.

The Default Spark 3.3 & R 4.2 environment is deprecated and will be removed in a future update.

Update your Data Refinery flow jobs to use the new Default Spark 3.4 & R 4.2 environment. For details, see Compute resource options for Data Refinery in projects.

More task-oriented Decision Optimization documentation

09 Feb 2024

You can now more easily find the right information for creating and configuring Decision Optimization experiments. See Decision Optimization experiments and its subsections.

Pagination view feature to publish assets to a catalog

08 Feb 2024

When you are publishing project assets to a catalog, you can now view 20 catalogs and assets on each page with the pagination view. Previously, you can view your assets on a list. See Publishing assets to a catalog.

Advanced analysis types in metadata enrichment are available in the Frankfurt region (IBM Knowledge Catalog)

09 Feb 2024

Advanced primary key and relationship analysis and advanced profiling are now also available in the Frankfurt region, in addition to the Dallas region.

IBM Cloud Data Engine connection is deprecated

08 Feb 2024

The IBM Cloud Data Engine connection is deprecated and will be discontinued in a future release. See Deprecation of Data Engine for important dates and details.

Save your searches for catalog assets

02 Feb 2024

Each user can now save up to 25 searches within each of their catalogs. The user who saves a search in a catalog is the only user who can view, run, edit, and remove the search. For more information, see Saving searches for catalog assets.

Gallery renamed to Resource hub

02 Feb 2024

The Gallery is renamed to Resource hub. The Resource hub contains sample projects, data sets, and notebooks. See Resource hub.

IBM Cloud Databases for DataStax connection is discontinued

02 Feb 2024

The IBM Cloud Databases for DataStax connection has been removed from Cloud Pak for Data as a Service.

Dremio connection requires updates

02 Feb 2024

Previously the Dremio connection used a JDBC driver. Now the connection uses a driver based on Arrow Flight.

Action required Important: Update the connection properties. Different changes apply to a connection for a Dremio Software (on-prem) instance or a Dremio Cloud instance.

Additional analysis types in metadata enrichment (IBM Knowledge Catalog)

31 Jan 2024

Metadata enrichment now provides these additional analysis options:

• Primary key analysis to detect primary keys in your data that uniquely identify each record in a data asset.

  Shallow analysis is automatically included when you select the Profile data enrichment option. Advanced analysis can be run on selected assets from the enrichment results.

• Relationship analysis to identify relationships between data asset or to find overlapping and redundant data in columns.

  Shallow key relationship analysis is run when you select the new Set relationships enrichment option. Advanced analysis can be run on selected assets from the enrichment results.

• Advanced profiling to get more exact results for certain metrics, such as frequency distribution and uniqueness of values within a column.

  Advanced profiling can be run on selected assets from the enrichment results.

Advanced primary key and relationship analysis and advanced profiling require the DataStage service in addition to the IBM Knowledge Catalog service and are available only in the Dallas region.

For more information, see Creating a metadata enrichment asset, Identifying primary keys, Identifying relationships, and Advanced data profiles.

AutoAI supports ordered data for all experiments

25 Jan 2024

You can now specify ordered data for all AutoAI experiments rather than just time series experiments. Specify if your training data is ordered sequentially, according to a row index. When input data is sequential, model performance is evaluated on newest records instead of a random sampling, and holdout data uses the last n records of the set rather than n random records. Sequential data is required for time series experiments but optional for classification and regression experiments.

Set to dark theme

25 Jan 2024

You can now set your Cloud Pak for Data as a Service user interface to dark theme. Click your avatar and select Profile and settings to open your account profile. Then, set the Dark theme switch to on. Dark theme is not supported in RStudio and Jupyter notebooks. For information on managing your profile, see Managing your settings.

View native type information in the details panel for asset columns

19 Jan 2024

Now, you can view both standardized and native data types directly in the column details panel. To view the native type information, click an asset column name from the Overview page of an asset.

New option for rule action precedence (IBM Knowledge Catalog)

18 Jan 2024

Rule action precedence enables you to specify how rules are applied when there are multiple rules with different actions on a data set. You can use the new Hierarchical enforcement option to configure a two-layer evaluation of data protection rules.

• The first layer evaluates the rules for an Allow or Deny action without considering any masking actions. The decision from this first layer must be to allow access to move to the second layer.
• The second layer evaluates the rules for a Transform action.

You can set this option from the user interface or from the access_decision_precedence API.

For more information, see Managing rule settings.

Store the results of data quality analysis (IBM Knowledge Catalog)

18 Jan 2024

You now have the option to write the output of the predefined data quality checks that are run as part of metadata enrichment to a database. For example, you might want to store this data so that you can use the tables for tracking quality issues and as input to remediation processes. For more information, see Creating a metadata enrichment.

Connect to a new data source in DataStage: Tableau

18 Jan 2024

You can now include data from a Tableau data source in your DataStage flows.

For the full list of DataStage connectors, see Supported data sources in DataStage.

Support for IBM Runtime 22.2 deprecated in watsonx.ai Runtime (formerly Watson Machine Learning)

11 Jan 2024

IBM Runtime 22.2 is deprecated and will be removed on 11 April 2024. Beginning 7 March 2024, you cannot create notebooks or custom environments by using the 22.2 runtimes. Also, you cannot train new models with software specifications that are based on the 22.2 runtime. Action required Update your assets and deployments to use IBM Runtime 23.1 before 7 March 2024.

• To learn more about migrating an asset to a supported framework and software specification, see Managing outdated software specifications or frameworks.
• To learn more about the notebook environment, see Compute resource options for the notebook editor in projects.
• To learn more about changing your environment, see Changing the environment of a notebook.