What's new

Decision Optimization Java models

20 Sept 2023

Decision Optimization Java models can now be deployed in Watson Machine Learning. By using the Java worker API, you can create optimization models with OPL, CPLEX, and CP Optimizer Java APIs. You can now easily create your models locally, package them and deploy them on Watson Machine Learning by using the boilerplate that is provided in the public Java worker GitHub. For more information, see Deploying Java models for Decision Optimization.

Reminder: Watson Knowledge Catalog profiling of unstructured data will be discontinued

08 Sept 2023

Profiling of unstructured data assets will no longer be supported starting on October 10, 2023.

Deprecation of comments in notebooks

31 Aug 2023

As of today it is not possible to add comments to a notebook from the notebook action bar. Any existing comments were removed.

Use new environment variable in DataStage

28 Aug 2023

You can now add the environment variable APT_SHOW_METRICS to the flow parameters of your DataStage flows.

Quickly find catalogs with name and date sorting

24 Aug 2023

You can now find catalogs by sorting the list of catalogs on the View all Catalogs page by name or date created. Click on the Name header to sort the catalogs alphabetically by name. Click on the Date created header to sort the catalogs by ascending or descending dates.

Data quality at a glance in Watson Knowledge Catalog

22 Aug 2023

Data quality information has a new home. For each data asset in a catalog or a project, a Data quality page is populated with quality information that comes from predefined data quality checks and data quality rules. You can see the applicable data quality dimensions and the results of individual quality checks. You can drill down into the results for each check or even into the results for each column.

For more information, see Data quality.

Similar information is available from metadata enrichment results.

All data quality analysis is now run in the context of metadata enrichment or data quality rules. When you run profiling from the Profile page in a project or a catalog, data quality is not analyzed anymore and no data quality scores are generated.

Additional cache enhancements available for Watson Pipelines

21 August 2023

More options are available for customizing your pipeline flow settings. You can now exercise greater control over when the cache is used for pipeline runs. For details, see Managing default settings.

Plan name updates for Watson Machine Learning service

18 August 2023

Starting immediately, plan names are updated for the IBM Watson Machine Learning service, as follows:

• The v2 Standard plan is now the Essentials plan. The plan is designed to give your organization the resources required to get started working with foundation models and machine learning assets.

• The v2 Professional plan is now the Standard plan. This plan provides resources designed to support most organizations through asset creation to productive use.

Changes to the plan names do not change your terms of service. That is, if you are registered to use the v2 Standard plan, it will now be named Essentials, but all of the plan details will remain the same. Similarly, if you are registered to use the v2 Professional plan, there are no changes other than the plan name change to Standard.

For details on what is included with each plan, see Watson Machine Learning plans. For pricing information, find your plan on the Watson Machine Learning plan page in the IBM Cloud catalog.

Connect to more data sources in DataStage

18 Aug 2023

You can now include data from these data sources in your DataStage flows:

• Cloudera Impala
• Presto

For the full list of DataStage connectors, see Supported data sources in DataStage.

Connect to Google BigQuery data with ODBC (DataStage)

18 Aug 2023

The ODBC connection now includes the Google BigQuery data source.

For the full list of data sources that are available for the ODBC connection in DataStage, see ODBC connection.

Use new functions in the DataStage Transformer stage

8 August 2023

• You can now use data masking, encryption, and regex functions in the Transformer stage as part of your DataStage flows.
• You can now drag and drop columns on the Output tab of the Transformer stage.
• You can now bulk edit columns in the Transformer stage from the Input tab.

Deprecation of comments in notebooks

7 August 2023

On 31 August 2023, you will no longer be able to add comments to a notebook from the notebook action bar. Any existing comments that were added that way will be removed.

Custom text analytics template (SPSS Modeler)

4 August 2023

For SPSS Modeler, you can now upload a custom text analytics template to a project. This provides you with more flexibility to capture and extract key concepts in a way that is unique to your context.

Enhanced capabilities for evaluating models with Watson OpenScale

25 July 2023

Use these new features to monitor and evaluate model deployments and interpret results.

Manage asset column relationships in a catalog

14 July 2023

Admins can now create and manage asset column relationships in a catalog. Column relationships can be created between columns and assets, columns and artifacts, or between columns.

To add a column relationship, click a column row on the Overview page of an asset. In the side pane, click the Related items overflow menu. Select one of the relationship types from the dropdown to add a relationship.

To learn more about creating relationships, see Asset relationships in a catalog.

Deprecation of the profiling support for unstructured data in Watson Knowledge Catalog

12 July 2023

Profiling of data assets that contain unstructured data, such as Microsoft Word, PDF, HTML, and plain text documents, is deprecated. Support will be discontinued on 10 October 2023. Until then, unstructured data assets of the supported types will continue to be profiled automatically when added to a project or a catalog. Starting on 11 October 2023, newly added unstructured data assets will no longer be profiled. Existing profiles will be available while the respective data assets live in the project or catalog.

Microsoft Azure SQL Database connection supports Azure Active Directory authentication (Azure AD)

14 July 2023

You can now select Active Directory for the Microsoft Azure SQL Database connection. Active Directory authentication is an alternative to SQL Server authentication. With this enhancement, administrators can centrally manage user permissions to Azure. For more information, see Microsoft Azure SQL Database connection.

Updates to Watson Machine Learning plans

07 July 2023

All Watson Machine Learning plans now include foundation model inferencing. Foundation model inferencing is available only on watsonx.ai. You can switch to watsonx.ai and use the new Prompt Lab tool or access foundation models with a notebook. You use the same Watson Machine Learning service instance on watsonx.ai as you use on Cloud Pak for Data as a Service.

If you have the Watson Machine Learning Lite plan, you can use up to 25,000 tokens for foundation model inferencing per month.

If you have the Watson Machine Learning v2 Standard or v2 Professional plan, your account will incur charges when your account users perform foundation model inferencing in the Prompt Lab or in notebooks.

For details on how foundation model inferencing is tracked and billed, see Watson Machine Learning plan. For the pricing of foundation model inferencing, find your plan on the Watson Machine Learning plan page in the IBM Cloud catalog.

Enhanced Natural Language Processing capabilities in Runtime 23.1

07 July 2023

Runtime 23.1 contains the Watson Natural Language Processing library 4.1 and a new set of pre-trained models. The NLP library contains the following enhancements and updates:

• Many included models are now transformer-based. These models were trained on the Slate large language model (LLM), which was created by IBM. The models are available in two versions:
  • Optimized for CPU-only environments
  • For environments with GPUs or CPUs
• Many included models for different NLP tasks are now workflow-based instead of block-based, so you can apply the models directly on input text without worrying about preprocessing steps.

NLP includes a Slate foundation model that you can use for fine-tuning your NLP tasks. You can use the Slate model or any transformer-based model from Hugging Face as a base to build your own models with Watson NLP.

All models provided by IBM are now exclusively trained on unbiased data with state-of-the-art filtering for hate, bias, and profanity.

These capabilities are currently available in the following environments:

• NLP Runtime 23.1 on Python 3.10
• GPU V100 Runtime 23.1 on Python 3.10
• GPU 2xV100 Runtime 23.1 on Python 3.10

You can use these environments for NLP processing, but not for general model development. The data science libraries used in these environments are not yet supported by Watson Machine Learning.

For more information, see Watson Natural Language Processing.

Enhanced Data Privacy content in Knowledge Accelerators (Watson Knowledge Catalog)

28 June 2023

The Knowledge Accelerator for Cross Industry now has Data Privacy content that includes a set of classified business terms and data classes to accelerate the discovery and governance of personal information. In addition, sample data privacy policies and rules are available to describe the activities that are related to processing personal information.

The business terms and data classes have classifications to guide the identification of personal information (PI) and sensitive personal information (SPI). You can use metadata enrichment in Watson Knowledge Catalog to assign the business terms to imported data assets to identify assets that contain personal data.

See IBM Knowledge Accelerator for Cross Industry.

Reporting now available for custom assets (Watson Knowledge Catalog)

28 June 2023

You can now create queries, reports, and dashboards based on custom-defined properties for any asset in a project or in a catalog. You can define new custom properties for assets to extend any provided or custom asset types and then create reports based on these relationships. For example, you can create a report on your data quality rules and artifact relationships to extrapolate the accuracy of your data. For more information, see Setting up reporting.

Reporting improvements for data quality rules (Watson Knowledge Catalog)

28 June 2023

You can now monitor data quality rules in the following ways:

• Receive and manage reports on data quality issues for each data asset in a catalog or a project.
• Monitor ongoing data quality for data assets in projects and catalogs by using reporting for data quality scores and data quality dimensions scores. The data quality score is based on a weighted average from data quality dimension scores. The data quality dimensions scores are based on results from relevant data quality checks.
• For data quality rules that include multiple rule definitions, see the data quality check statistics (results) by rule definition in the BI reporting schema.

For more information, see Data model.

Govern models more effectively with enhancements for AI Factsheets

23 June 2023

AI Factsheets now offers more ways for you to track solutions for business problems, govern a wider range of assets, capture more information with factsheet attachments, and generate improved reports.

Announcing support for Python 3.10 Spark 3.3 runtime for notebooks (Watson Studio)

23 June 2023

Python 3.10 Spark 3.3 is now supported as a runtime for notebooks. Python 3.9 Spark 3.3 is deprecated and will be discontinued on July 20, 2023. Starting on July 6, 2023, you will be restricted from creating notebooks with a Python 3.9 Spark 3.3 environment, but existing notebooks will continue to run until July 30, 2023. Change your notebook environment to use Python 3.10 Spark 3.3 before the deprecated environment is removed. For details on notebook environments, see Compute resource options for the notebook editor in projects.

Coming soon: General availability of time series anomaly prediction in AutoAI experiments

15 June 2023

Create a time series anomaly prediction experiment to train a model that can detect anomalies, or unexpected results, when the model predicts results based on new data. This capability of AutoAI is currently offered in beta, and is not supported for production. Once the feature is generally available and fully supported, training for time series anomaly prediction experiments will consume capacity unit hours (CUH) as part of your Watson Machine Learning plan. For more details, see:

• Creating a time series anomaly prediction experiment
• Watson Machine Learning plans

Customize engine parameters for Decision Optimization experiments (Watson Studio)

15 June 2023

You can now add an engine settings file in your Decision Optimization experiment. With this file, you can view and customize the engine parameters that are used to solve your model in a new visual editor. You can also import an engine settings file and search for existing settings.

See Python model engine settings.

Manage AI lifecycle events with the cpdctl tool

2 June 2023

You can now manage and automate your assets hosted on Cloud Pak for Data as a Service using the Cloud Pak for Data Command Line Interface tool (cpdctl). Use automatic configuration from IBM Cloud to easily connect with the cpdctl API commands. For details and an example, see these resources:

• IBM Cloud Pak for Data Command Line Interface documentation.
• Exporting space assets for an example of using cpdctl for managing assets.
• IBM cpdctl CLI on IBM Cloud blog post for details about connecting to cpdctl from Cloud Pak for Data as a Service.

Find your catalogs easily with search

1 June 2023

With the updated Catalogs page, you can now search for a catalog by name and see more catalogs on the page for easier scanning.

Reminder: End of support approaching for Runtime 22.1 on Python 3.9 and R 3.6

15 May 2023

IBM Runtime 22.1 on Python 3.9 and R 3.6 environments will be removed on June 15, 2023. You can no longer create new notebooks or create custom environments using the 22.1 runtimes or R 3.6, or train new models with Python 3.9 software specifications. Update your assets and deployments to use IBM Runtime 22.2 on Python 3.10 or R 4.2 before June 15, 2023.

• For details on migrating an asset to a supported framework and software specification, see Managing frameworks and software specifications.
• For details on notebook environments, see Compute resource options for the notebook editor in projects.
• For information on changing your environments, see Changing the environment of a notebook.
• For details on the libraries and packages for R versions, see the CRAN release notes.

Introducing key-value search for advanced users

18 May 2023

Using key:value pairs in the search bar, you can now search within asset and artifact properties, such as the description, tags, custom properties, column names, and many more. See Searching for properties.

Name change for the IBM Cloud Compose for MySQL connection

18 May 2023

The IBM Cloud Compose for MySQL connection was renamed to IBM Cloud Databases for MySQL. Your previous settings for the connection remain the same. Only the connection name has changed.

Discontinued connections

18 May 2023

The following connections are discontinued and have been removed from Cloud Pak for Data as a Service:

• IBM Db2 Event Store
• IBM Db2 Hosted

Renaming data assets also renames file attachments in projects

19 May 2023

When you change the name of data assets with file attachments that you uploaded into the project, the file attachments are also renamed. However, changing the name of data assets imported from catalogs does not rename any attachments. You must update any references to the data asset in code-based assets, like notebooks, to the new data asset name, otherwise, the code-based asset won't run. See more information about Managing assets in projects.

New UI capabilities for creating custom assets and managing custom properties for columns

11 May 2023

Catalog collaborators with the Admin or Editor role can now complete the following tasks from the web client:

• Create custom assets from the catalog. To add a custom asset, select Custom asset from the Add to catalog drop-down menu.
• Manage custom properties for data asset columns. To manage custom properties, select a column in the Overview of an asset and edit the properties in the side pane.

To learn more about custom properties for data assets, see Custom asset types, properties, and relationships.

Add generated code from the Code snippets pane

04 May 2023

A new Code snippets icon was added to the notebook toolbar. Clicking the icon, opens the Code snippets pane from where you can read data from a file or connection that was added to the project. The existing "Insert to code" function logic for generating code that loads data to a notebook cell has been moved under Read data. The former Find and load data pane can now only be used to upload data to a project. See Loading and accessing data in a notebook.

Access more data with the new Presto connection

27 Apr 2023

You can now work with data from Presto data sources. For information, see Presto connection.

Drill down into the details of profiling results (Watson Knowledge Catalog)

20 Apr 2023

You can now access detailed profiling information from within a metadata enrichment or from an asset’s Profile tab in a project or a catalog. For each column, view statistical information about the column data, information about data classes, data types and formats, and the frequency distribution of values in the column. For the statistical information, you can also choose between several types of visualizations. To populate these views for an existing profile, update the profile.

For details, see Column-level profile details.

Default Python and CPLEX versions updated (Decision Optimization)

13 Apr 2023

The default Python for Decision Optimization users is now 3.10 and the default CPLEX version is 22.1. These versions are used by default when you create a new experiment. Python 3.9 is deprecated and will soon be removed. To update your environment, see Configuring Environments. To update existing deployed models, see Model deployment.

Enhancements to data quality rules (Watson Knowledge Catalog)

13 Apr 2023

You can now also run data quality rules on data assets from these data sources:

• Amazon S3 (CSV files only)
• Apache Cassandra
• SAP ASE

When you configure a data quality rule with externally managed bindings, you can now select additional content for output links in the associated DataStage flow. For more information, see Creating rules from data quality definitions.

New: Time Series anomaly detection experiment (Beta)

7 Apr 2023

Use AutoAI to train a time series anomaly prediction model that can detect anomalies, or unexpected results, when the model predicts results based on new data. Model candidate pipelines generated by the experiment are ranked according to how well they perform measured by the optimizing metric. Save a model as a notebook to review the code, or save and deploy a model to detect potential anomalies in new data. For details, see Creating a time series anomaly prediction model (Beta). This feature is offered as beta and is not yet supported for use in production environments.

Filter your asset activity in a project

6 Apr 2023

In the Assets pane on the Overview tab of a project, you can filter assets by selecting By you or By all using the dropdown. By you lists assets edited by you, ordered by most recent at the top. By all lists assets edited by others and also by you, ordered by most recent at the top.

Upgrade to Spark with R 4.2 in Watson Studio

3 Apr 2023

Action required Spark R 3.6 environments for Watson Studio are upgraded to R 4.2. All Spark R 3.6 environments are now deprecated and will be removed on 15 June 2023. Starting on 11 May 2023, you can no longer create new notebooks or new Data Refinery flows with Spark R 3.6. Additionally, you will not be able to create new Spark R 3.6 custom environments. At that time, you might need to update some package versions and scripts for your notebooks. You must update your assets and deployments to use Spark with R 4.2 before 15 June 2023.

See Changing the environment for a notebook. For details on the libraries and packages for R versions, see the CRAN release notes.

New Spark with R 4.2 environment for running Data Refinery flow jobs

3 Apr 2023

You can now select Default Spark 3.3 & R 4.2 when you select an environment for a Data Refinery flow job. The new environment uses the same capacity unit hours (CUHs) as the other Default environments.

For information about environments for Data Refinery, see Compute resource options for Data Refinery in projects.

Action required The environment change affects two GUI operations. If you have existing Data Refinery flows that include these GUI operations, you must update the Data Refinery flow.

• Split
• Tokenize

To update a flow, open it, save it. For details, see Managing Data Refinery flows.

Create custom assets from a catalog

31 Mar 2023

Admins and editors can now create custom assets inside the Catalog UI. To add a new custom asset, select Custom asset from the Add to catalog dropdown menu. To learn more about custom assets, see Custom asset types, properties, and relationships in Adding assets to a catalog (Watson Knowledge Catalog).

Improvements and enhancements in Watson Query

29 Mar 2023

Watson Query has been updated to provide the following capabilities:

• With asynchronous virtualization, you can view the status details of a virtualization job any time on the Virtualized data page. If the virtualized tables are large and the job takes longer, you can work on other tasks, such as virtualizing more tables, while the job finishes.
• With asynchronous publishing and assignments on the Virtualized data page, you can work on other tasks while the publishing and assignment jobs finish.
• You can use jobs in the web client to collect statistics on virtualized tables. For more information, see Collecting statistics in the web client in Watson Query.
• You can view the publishing or assignment history of an object on the Virutualized data page. Click an object row from the list to view its publishing and assignment history in the right side panel of the Virutualized data page.

Federated Learning runs on Mac computers with M-series chips

23 Mar 2023

Run your Federated Learning experiments on M1 Mac and M2 Mac computers in the latest runtime. For requirements, see Set up your system.

Define composite keys in reference data sets (Watson Knowledge Catalog)

17 Mar 2023

You can now specify multiple columns to create a composite key for your reference data sets. Without a composite key, reference data values in a set are identified by a unique string in the code column. A composite key is a combination of the code column and up to 5 custom columns in a reference data set. A composite key is used to uniquely identify each reference data value. With a composite key, the values in the code column no longer need to be unique. Uniqueness is guaranteed only when the values of all the specified columns are combined. For details, see Designing reference data sets.

Create queries, reports, or dashboards based on custom relationships (Watson Knowledge Catalog)

9 Mar 2023

When you create custom relationships between assets and governance artifacts, you can sync them to Watson Knowledge Catalog Reporting Data Mart, so that you can create reports. For example, you can use the custom relationships reporting to:

• Obtain quality analytics at various levels of granularity (by domain, by metadata, by user, by team)
• Certify the data quality of your data
• Count the number of assets that have a specific privacy property

To learn how to create custom relationships, see Custom properties and relationships for governance artifacts and catalog assets (Watson Knowledge Catalog).

To learn how to create reports, see Setting up reporting for Watson Knowledge Catalog.

Runtime 22.1 on Python 3.9 deprecation for Watson Studio and Watson Machine Learning

9 Mar 2023

Action required IBM Runtime 22.1 on Python 3.9 is now deprecated and will be removed on Jun 15, 2023. Starting on May 11, 2023, you can no longer create new notebooks or create custom environments using the 22.1 runtimes. You will also be unable to train new models with Python 3.9 software specifications. Update your assets and deployments to use IBM Runtime 22.2 on Python 3.10 before June 15, 2023:

• For details on migrating an asset to a supported framework and software specification, see Managing frameworks and software specifications.
• For details on notebook environments, see Compute resource options for the notebook editor in projects.
• For information on changing your environments, see Changing the environment of a notebook.

Run data quality rules on additional data sources (Watson Knowledge Catalog)

9 Mar 2023

You can now run data quality rules on data assets from these data sources:

• IBM Watson Query
• Microsoft Azure Data Lake Storage
• Snowflake

New option for binding variables in data quality rules (Watson Knowledge Catalog)

9 Mar 2023

You can now also use job parameters to bind rule variables to data columns and manage those parameters centrally in a project. Thus, you don’t need to update the rules when, for example, you want to change the binding to a different column. See Creating rules from data quality definitions.

Enhancements for AI Factsheets (Watson Machine Learning)

03 March 2023

You can now attach files and images to a factsheet. For details, see Customizing details for a factsheet. Factsheets also display additional Watson OpenScale metrics from explainability and custom monitors. For details, see Viewing factsheets.

Create, store, and share machine learning features (Beta) (Watson Studio)

02 March 2023

You can now speed the development of machine learning models by creating and sharing features. You add a feature group to a data asset in a project to identify the features of that data set. You can share the features with your organization by publishing the data asset to a catalog, which acts as a feature store. See Managing feature groups.

Manage custom relationships (Watson Knowledge Catalog)

24 February 2023

Now, you can manage custom relationships between catalog assets and governance artifacts in the Overview page of an asset.

To learn how to create custom relationships, see Custom properties and relationships for governance artifacts and catalog assets (Watson Knowledge Catalog).

Data Refinery Calculate operation works on Date columns

17 Feb 2023

You can now use the Calculate operation on Date data type columns to add or subtract day or month values.

For information about GUI operations, see GUI operations in Data Refinery.

New library to access project assets in Watson Studio

17 Feb 2023

The ibm-watson-studio-lib library contains a set of functions that help you to interact with Watson Studio projects and project assets. The library can be used in notebooks that are created in the notebook editor and is available for Python and R. It is the successor of the project_lib library. For details, see Using ibm-watson-studio-lib.

"Default Spark 3.2 & R 3.6 " environment discontinued (Data Refinery)

17 Feb 2023

The Default Spark 3.2 & R 3.6 environment will no longer be available effective February 17, 2023.

Action required If you have any Data Refinery flow jobs set up with the Default Spark 3.2 & R 3.6 environment or a custom environment that uses Spark 3.0, the jobs will fail. Change the environment to Default Spark 3.3 & R 3.6 or Default Data Refinery XS or a custom environment that does not use Spark 3.0.

For information about environments for Data Refinery, see Compute resource options for Data Refinery in projects.

New features for data quality rules (Watson Knowledge Catalog)

16 Feb 2023

These new capabilities are available:

• Use more than one data quality definition in a single data quality rule. In addition, you can include an individual definition more than once to apply the same definition to different columns. For details, see Creating rules from data quality definitions.
• Download rule output as CSV file. If an output table is defined for the rule, you can now also download the rule output as a CSV file from the rule's run history, for example, for use in a spreadsheet program.
• Run rules on data from Amazon Redshift and Greenplum data sources. See Supported data sources for metadata import, metadata enrichment, and data quality rules.
• Export and import data quality assets. When you export a project to desktop, you can now include data quality assets. See Exporting a project.

Import assets from a project or space into an existing space (Watson Machine Learning)

09 Feb 2023

You can now import a deployment space or a project (in .zip format) into an existing deployment space. Add assets or update existing assets to a space. For example, you can replace a model with a newer version. For details, see Importing spaces and projects into existing spaces.

Use more macros in DataStage

10 Feb 2023

You can add the DSJobController macro to stage properties or in the transformer functions.

The macro acts as DataStage function and outputs data without the need for arguments, simplifying the setup of DataStage jobs and flows.

For more information, see Macros.

Use more macros in DataStage

06 Feb 2023

You can add the following macros to stage properties or in the transformer functions:

• DSProjectId
• DSJobRunId
• DSJobId

The macros act as DataStage functions and output data without the need for arguments, simplifying the setup of DataStage jobs and flows.

For more information, see Macros.

Edit input columns in DataStage stages

20 Jan 2023

You can now edit columns through the input tab of a stage in DataStage. Your changes are propagated to the previous stage in the flow.

New options for metadata import (Watson Knowledge Catalog)

19 Jan 2023

To ensure that the target project or catalog of your metadata import doesn't contain stale data, you can now configure the import to clean up data assets that can't be reimported. Select to delete assets that are no longer available in the data source, that were removed from the import scope, or both from the import target when the metadata import is rerun. See Importing metadata.

Export data from Decision Optimization experiments to your project

18 Jan 2023

You can now export tables to your project from either the Prepare data or Explore solution view in your Decision Optimization experiment. This enables you to reuse your data in other models or services. You can also export data using the Decision Optimization Python client.

See Exporting data from Decision Optimization experiments.

Updated Data fabric use cases

12 Jan 2023

The Data fabric uses cases are updated to better reflect how you use our products:

• Data integration: This use case now includes Pipelines.
• Data governance: This use case now includes Match 360.
• AI governance: This use case now focuses on monitoring, maintaining, automating, and governing AI models in production.
• Data Science and MLOps: This new use case explains how to operationalize data analysis and model creation.

See Data fabric use cases.

Customize the web browser to support your brand

12 Jan 2023

As an administrator, you can add custom product names, logos, and other graphics to customize the branding of the web browser for Cloud Pak for Data as a Service.

See Customizing the branding of the web browser.

Connect to more data sources in DataStage

06 Jan 2023

You can now include data from these data sources in your DataStage flows:

• Dremio
• SingleStoreDB

For the full list of DataStage connectors, see DataStage connectors.

Native Python for scripting in SPSS Modeler

13 Dec 2022

You can now use Native Python for scripting in the Extension nodes. Invoke native Python APIs from your scripts to interact with SPSS Modeler. For details, see the new Native Python APIs documentation.

Data quality features are live in the Frankfurt region

12 Dec 2022

Data quality features are now live in our Frankfurt region, in addition to the Dallas region.

Use the DSFlowName macro in DataStage

09 Dec 2022

You can add the macro DSSFlowName to stage properties or in the transformer functions. The macro acts as a DataStage function and outputs data without the need for arguments, simplifying the setup of DataStage jobs and flows. Where you specify this macro, "DSFlowName" is replaced with the name of the flow at run time.

For more information, see Macros.

Account administrators can join any project and view all projects with new Manager role permissions

09 Dec 2022

As an account administrator, you can now join any project as Admin and view all projects in the account. You must assign yourself the Manager role in the IBM Cloud Pak for Data service in IBM Cloud IAM to gain these permissions. For details, see Managing all projects in the account.

Simplified configuration (Watson OpenScale)

08 Dec 2022

When you configure fairness evaluations and explainability in Watson OpenScale, you can run a custom notebook to generate configuration files. You can upload the configuration files in Watson OpenScale to specify settings.

For more information, see Configuring model monitors.

Upload payload data (Watson OpenScale)

08 Dec 2022

To provide model details to configure model evaluations for production deployments, you can now use a CSV file to upload payload data to Watson OpenScale. For more information, see Configuring endpoint evaluation.

Configure explainability methods (Watson OpenScale)

08 Dec 2022

When you configure your model evaluations in Watson OpenScale, you can now select different settings to generate local and global explanations:

• For global explanations, you can use the SHAP (SHapley Additive exPlanations) method.
• For local explanations, you can use the SHAP method or the LIME (local interpretable model-agnostic explanations) method.

For more information, see Configuring explainability.

New fairness metrics (Watson OpenScale)

08 Dec 2022

You can now configure the following fairness metrics in Watson OpenScale:

• Statistical parity difference
• Average odds difference
• Average absolute odds difference
• False negative rate difference
• False positive rate difference
• False discovery rate difference
• False omission rate difference
• Error rate difference

For more information see, Fairness metrics overview.

RStudio environment runtimes use R 4.2

08 Dec 2022

All of the default RStudio environment templates now use R 4.2. See Compute resource options for RStudio in projects for details.

Extensive new query capabilities Watson Knowledge Catalog

08 Dec 2022

You can now create custom reports on:

• Workflow data
• Metadata imports
• User profiling
• Metadata enrichment

For example, to ensure the quality of automatic term assignments for your discovered data sets and columns, you can generate a report to list the assigned and rejected terms for the data sets and columns.

To learn more about creating custom reports, see Setting up reporting for Watson Knowledge Catalog.

Data quality rules are coming to Cloud Pak for Data as a Service (Watson Knowledge Catalog)

09 Dec 2022

Data quality features are now available in the Dallas region. To be able to work with these features, you need the DataStage service as well as the Watson Knowledge Catalog service.

Identify data quality issues by evaluating your data against common data quality dimensions. Data quality definitions and rules are now available as assets in projects:

• Design and run data quality rules on data from a variety of sources.
• Automate your quality checks to monitor changes in data quality over time.
• Identify records in your data that do not meet the defined quality criteria and require remediation.

See Managing data quality.

Python 3.10 support and other enhancements to Decision Optimization (Watson Studio)

08 Dec 2022

Python 3.10 is now supported in Decision Optimization experiments in Watson Studio and for deployment in Watson Machine Learning. The default version remains Python 3.9. See Configuring Environments and Model deployment.

For DOcplex notebooks, the new Runtime 22.2 with Python 3.10 and CPLEX 22.1 is now available.

You can now search for OPL engine settings in Decision Optimization experiments in Watson Studio using the new filtering capabilities. See OPL settings.

New manage governance artifacts permission

09 Dec 2022

You can grant the manage governance artifacts permission to allow users to view all governance artifacts in all categories, regardless of whether the users are collaborators in those categories. With this permission, users can also run all API calls for governance artifacts.

When you grant this new permission, you should also grant the manage categories and access governance artifacts permissions at the same time if you want the users to fully control category and governance artifacts.

For more information, see User roles and permissions for Watson Knowledge Catalog and Watson Studio.

Homomorphic encryption for Federated Learning

07 Dec 2022

You can now apply Fully Homomorphic Encryption (FHE) in IBM Federated Learning for select model frameworks and computer architecture. With FHE, you can add an additional layer of security and privacy when using Federated Learning to train your model by encrypting model information that is sent to the aggregator. For more information, see Applying encryption.

Please check out our blog for additional information.

JDBC connector is automatically converted to platform connection when migrated to modern DataStage

2 Dec 2022

When you migrate a job from traditional DataStage to the modern version, the job might contain a source or target that has a JDBC connector. The stage is automatically converted to its respective platform connection on Cloud Pak for Data as a Service when you migrate such a job.

For more information, see Migrating DataStage jobs.

Support for IBM Cloud App ID for some services

1 Dec 2022

Some services on Cloud Pak for Data as a Service support IBM Cloud App ID to integrate customer's user registries for user authentication. You configure App ID on IBM Cloud and then provide an alias to the people in your organization to log in to Cloud Pak for Data as a Service. This beta release supports Watson Studio, Watson Knowledge Catalog, Watson Machine Learning, Watson OpenScale, and Watson Query. Other services have not been tested. See Setting up IBM Cloud App ID (beta).

Improvements and enhancements in Watson Query

30 Nov 2022

Watson Query has been updated to provide the following capabilities:

• In Data virtualization > User management, you can now add a Watson Query user by using their App ID and email address instead of an IBMid. For more information, see Setting up IBM Cloud App ID (beta).
• Sharing your virtualized objects is quicker and easier. When you virtualize objects, you can assign the objects to multiple projects, and you can publish the objects to a catalog, all in one step.
• When your join process is taking a long time, you can cancel the preview and improve the query performance before joining virtual tables. For more information, see Improving query performance in Watson Query.
• If your lite plan is expiring soon, Watson Query warns you how many days you have left. When your plan expires, you cannot use the Watson Query service.

New Runtime 2022 release for Python 3.10 and R 4.2

17 Nov 2022

You can now use Runtime 22.2 environments, which include the latest data science frameworks on Python 3.10 and R 4.2, to run Watson Studio Jupyter notebooks, train models, and run Watson Machine Learning deployments. Notebook environments with R 3.6 are now deprecated. Update your R assets and deployments to use Runtime 22.2 accordingly:

• For information on the Runtime 22.2 release and the included environments for Python 3.10 and R 4.2, see Compute resource options for the notebook editor in projects.
• For details on deployment frameworks, see Managing frameworks and software specifications.

Spark 3.3 replaces Spark 3.2 for Watson Studio and Watson Machine Learning

16 Nov 2022

Spark 3.3 is now supported for Watson Studio and Watson Machine Learning. Spark 3.2 is deprecated as a machine learning framework, notebook environment, and RStudio runtime. Update your assets to use Spark 3.3 instead. Support for training assets with Spark 3.2 will be discontinued on Jan 4, 2023. Support for deploying and scoring models with Spark 3.2 will be discontinued on Feb 16, 2023 and existing deployments using Spark 3.2 specifications will be removed. For details on migrating an asset to a supported framework and software specification, see Managing frameworks and software specifications. For details on notebook environments, see Compute resource options for the notebook editor in projects.

Refine data from a selected Excel worksheet in a connection or a connected data asset (Data Refinery)

18 Nov 2022

If you have an Excel file with multiple worksheets in a connection or in a connected data asset, you can select the individual worksheet of data in Data Refinery. Previously only the first worksheet was read.

Manage settings for data protection rules (Watson Knowledge Catalog)

11 Nov 2022

You can now have more control over how data protection rules are enforced. You can set the following behavior:

• Set the rule data access convention to control whether access to data is allowed or denied by default.
• Set the rule action and masking precedence to determine how multiple rules combine different actions and masking methods into a single decision.

See Managing rule settings.

Use new functions in the DataStage Transformer stage

11 Nov 2022

You can now use the UrlEncode and UrlDecode functions in the Transformer stage as part of your DataStage flows. For the full list of available functions, see Parallel transform functions.

Use triggers in the DataStage Transformer stage

11 Nov 2022

You can now use the Triggers tab to choose routines to be run at specific execution points as the Transformer stage runs in a DataStage job. The available built-in routines are SetCustomSummaryInfo and SetUserStatus. For more information, see Triggers in the Transformer stage.

Planning to implement data governance (Watson Knowledge Catalog)

10 Nov 2022

You can now understand how to plan your data governance implementation with Watson Knowledge Catalog, including the choices that you have, the implications of those choices, and how those choices affect the order of implementation tasks. See Planning to implement data governance.

Mark a project as sensitive

10 Nov 2022

As Admin, you can mark a project as sensitive when you create the project. Marking a project as sensitive prevents members of a project from moving data assets out of the project. You cannot mark the project as sensitive after the project was created. For details, see Marking a project as sensitive.

The tool for Advanced Data Privacy is renamed to Masking flow (Watson Knowledge Catalog)

07 Nov 2022

To run masking flows that create permanently masked data assets, select the Masking flow option on the New asset page. See Masking data with Masking flow.

New options available for the Apache HDFS connector (DataStage)

11 Nov 2022

Use new connector properties in the Apache HDFS connector that are specific for DataStage. These properties provide more features and granular control of the flow execution, similar to the "optimized" connectors. Select Use DataStage properties in the properties panel.

Performance enhancement for the Split column GUI operation in flows that use large data assets (Data Refinery)

04 Nov 2022

The Split column operation has been enhanced to work faster on large data assets.
Action required If you have existing Data Refinery flows that use the Split column operation, you must update the flows. To update a flow, open it, save it, and run a job for the flow. For details, see Managing Data Refinery flows.

Batch import connected data assets

03 Nov 2022

You can now import multiple connected data assets from the same connection at the same time. For details, see Adding data from a connection to a project.

GPU environment upgrades for Watson Studio

27 Oct 2022

We're excited to announce that Watson Studio now supports NVIDIA V100 GPUs to power its runtime environments in the Dallas region. The V100 GPU offers performance that is orders of magnitude faster than previous generations, allowing it to efficiently support advanced AI and parallel computing tasks. The new GPU environments come in the following two configurations with accelerated compute and memory:

• 40 vCPU + 186 GB + 1 NVIDIA V100 (1 GPU)
• 80 vCPU + 372 GB + 2 NVIDIA V100 (2 GPU)

Additionally, the NVIDIA K80 GPU environments are now deprecated. You will be unable to create new K80 environments from 18 November 2022 onwards, and the K80 environments will be fully removed on 8 December 2022. See Changing the environment of a notebook to select another environment for your asset.

Improvements to managing your notification settings

27 Oct 2022

You can now select Do not disturb to turn off push notifications that appear briefly on screen, and continue to see the number of notifications on the bell. To select Do not disturb, click the notification bell icon and then click the settings icon. See Managing your settings for more details on notification settings.

New capabilities added to the Watson Natural Language Processing Library

27 Oct 2022

Two new components (blocks) that encapsulate extracting DBPedia concepts and the relations between two entities from input data are now included in the Watson Natural Language Processing library. In addition, Entity extraction now contains the extraction of PII information. For details, see Watson Natural Language Processing.

New Spark 3.3 environment for running Data Refinery flow jobs

28 Oct 2022

You can now select Default Spark 3.3 & R 3.6 when you select an environment for a Data Refinery flow job. The Default Spark 3.3 & R 3.6 environment includes enhancements from Spark. The new environment uses the same capacity unit hours (CUHs) as the other Default environments.

For information about environments for Data Refinery, see Compute resource options for Data Refinery in projects.

Spaces user interface enhanced to improve productivity

Spaces are enhanced to align more closely with the asset organization in projects. Explore the enhanced asset organization, asset import flow, improved navigation, and built-in guidance - all designed to make it easier and more efficient to work and collaborate in a space. For details, Deployment spaces.

Customize report templates for AI Factsheets

20 Oct 2022

If the default report templates provided with AI Factsheets do not meet your needs, you can download a default report template, customize it for your needs, and upload the new template. Currently, you must use the AI Factsheets API to download the template, but you can upload it from the Model inventory UI. See Generating reports for factsheets and model entries.

Support for Oracle database sequences in DataStage

21 Oct 2022

You can now use Oracle database sequences in Surrogate Key Generator, Slowly Changing Dimension, and Transformer operators. The password for the Oracle connection must be an encrypted parameter.

For more information, see Updating the state file, Surrogate keys in a DataStage Slowly Changing Dimension stage, Surrogate Key tab, and Creating and using parameters and parameter sets.

Filter rows with data protection rules

21 Oct 2022

You can now specify that the action for a data protection rule filters rows from the affected data asset. You can include or exclude rows based on values in a specified column in the same asset or in a reference asset. For details, see Filtering rows.

Customize the learning scope for ML-based term assignment in metadata enrichment (Watson Knowledge Catalog)

21 Oct 2022

You can now determine at project-level whether your models for ML-based term assignment are trained from assets in the project or from a catalog of your choice.

See Default enrichment settings.

Capture data changes better through improved sampling in metadata enrichment (Watson Knowledge Catalog)

21 Oct 2022

When you set up customized sampling for metadata enrichment, you can now choose between sequential and random sampling. In addition, you can select to include a certain percentage of the table rows in the sample instead of a fixed number of rows. Random sampling is available only for data assets from data sources that support such sampling method.

For details, see Enriching your data assets.

Name change for the IBM Data Virtualization connection

21 Oct 2022

The IBM Data Virtualization connection has been renamed to IBM Watson Query. Your previous settings for the connection remain the same. Only the connection name has changed.

Name change for the IBM Data Virtualization connector (DataStage)

21 Oct 2022

The IBM Data Virtualization connector in the DataStage canvas has been renamed to IBM Watson Query. Your previous settings for the connector remain the same. Only the connector name has changed.

Predefined business terms for personal data (Watson Knowledge Catalog)

21 Oct 2022

For new accounts in Cloud Pak for Data as a Service Lite and Standard plans predefined business terms are available in Knowledge Accelerator Sample Personal Data category. For more information, see Predefined business terms.

Asset previews are more current

14 Oct 2022

Asset previews are now refreshed more often by default. Previously, asset previews were refreshed every 10 days. Now, asset previews are refreshed every day. You can manually refresh an asset preview at any time. See Asset previews.

Customize engine parameters for Decision Optimization experiments (Watson Studio)

13 Oct 2022

You can now add OPL engine settings file in your Decision Optimization experiment. This enables you to view and customize the engine parameters used to solve your model in a new visual editor. You can also import existing OPL settings.

See Engine settings.

View metadata of blocked assets in catalogs

13 Oct 2022

Users who are denied access to assets by a data protection rule can now see the metadata of the assets. For example, when users click on a blocked asset in a catalog they can now see the description, assigned terms, custom properties, relationships, and column names of the blocked asset.

Focus on the catalogs that matter with resource scoping

14 Oct 2022

With resource scoping, you limit the catalogs that you see to those that you own and those that are shared with you within the catalog’s account. Go to your account settings to enable resource scoping for your existing accounts. New accounts will use resource scoping by default. For catalog accounts that have resource scoping enabled, federated users can collaborate only if they are invited by an admin.

Connect to a new data source in DataStage: Elasticsearch

14 Oct 2022

You can now include data from an Elasticsearch data source in your DataStage flows.

For the full list of DataStage connectors, see DataStage connectors.

Use multiple input links on the ODBC connector for DataStage

07 Oct 2022

You can now use multiple input links on the ODBC connector and assign a different action to each link.

Use multiple input links on the Apache Cassandra connector for DataStage

07 Oct 2022

You can now use multiple input links on the Apache Cassandra connector.

Use multiple reject links on the Db2, Oracle, and ODBC connectors for DataStage

07 Oct 2022

You can now use multiple reject links on the Db2, Oracle, and ODBC connectors.

AutoAI experiments with joined data deprecated

06 Oct 2022

Action required The AutoAI experiment feature for joining multiple data sources to create a single training data set is deprecated. Support for joining data in an AutoAI experiment will be removed on Dec 7, 2022. After Dec 7, 2022, AutoAI experiments with joined data and deployments of resulting models will no longer run. To join multiple data sources, use a data preparation tool such as Data Refinery or DataStage to join and prepare data, then use the resulting data set for training an AutoAI experiment. Redeploy the resulting model. For details, see Joining data sources.

Assign all users Viewer access to the connections in the Platform assets catalog

30 Sept 2022

When you create the Platform assets catalog, you must add collaborators and assign them roles. Now, instead of assigning individual users the Viewer role, you can add the Public Access group as a collaborator and assign the Viewer role to the group. You can add the Public Access group as a collaborator with the Viewer role in an existing Platform assets catalog as well. The Viewer role allows users to find connections and use them in projects. By default, all users in your account are members of the Public Access group. See Creating the catalog for platform connections.

New Slowly Changing Dimension stage in DataStage

30 Sept 2022

You can now use the Slowly Changing Dimension stage in your DataStage flows. Use the Slowly Changing Dimension stage to store and manage current and historical data over time. For more information, see Slowly Changing Dimension stage.

Use the DStageName server macro in DataStage

30 Sept 2022

You can add the macro DSStagename to stage properties or in the transformer functions. The macro acts as a DataStage function and outputs data without the need for arguments, simplifying the setup of DataStage jobs and flows. Where you specify this macro, "DSStageName" will be replaced with the name of the stage as part of the job compilation.

For more information, see Macros.

Boolean and List parameter types available for DataStage jobs

30 Sept 2022

In DataStage jobs, you can use the parameter types 'Boolean' and 'List'. You use the Boolean parameter type to specify a true or false value and the List parameter type to specify a list of values that are available for selection in a job. For more information about parameters, see Creating and using parameters and parameter sets.

Add and edit column metadata for data definitions in DataStage

30 Sept 2022

In data definitions, you can add and edit metadata properties at the column level. For example, you can set properties such as field level, delimiter, quotation mark, and string type. For more information about data definitions, see Defining data definitions.

Copy and paste subflows in DataStage

30 Sept 2022

You can conveniently copy and paste shared subflows both within a DataStage flow or between different DataStage flows in the same project. A subflow can be copied and pasted as part of a larger flow or as just the subflow itself. For more information about subflows, see Subflows.

Oracle, Snowflake, and Teradata connectors can now have multiple input links, each with an individual action

30 Sept 2022

Previously the Oracle, Snowflake, and Teradata connectors had only one input link, and you specified the link's properties in the Stage properties. Now the connectors can have multiple input links, and each link can have a different property. This enhancement means that each link can have an individual action, such as read, write, and append. You can view the properties by switching the links in the Input tab.

IBM Watson Pipelines beta is live in the Frankfurt region

26 Sep 2022

IBM Watson Pipelines is now live in our Frankfurt region, in addition to the Dallas region. The tool provides a graphical interface for orchestrating an end-to-end pipeline of assets from creation through deployment. For details, see IBM Watson Pipelines.

Deprecation of notebook environments with Spark 3.2 in Watson Studio

23 Sep 2022

Spark 3.2 is deprecated as a notebook environment runtime. Update your notebooks to use Spark 3.3 environments instead.

For more information, see Compute resource options for the notebook editor in projects.

Improvements and enhancements in Watson Query

21 Sept 2022

Watson Query has been updated to provide the following capabilities:

• You can connect to TM1 databases that store data in multidimensional OLAP cubes by using the IBM Planning Analytics connection type. You cannot use CAM credentials as an authentication method when you create a connection to an IBM Planning Analytics data source in Watson Query. For more limitations, see Supported data sources in Watson Query.
• You can gather statistics on a virtualized table by using the new DVSYS.COLLECT_STATISTICS procedure. This procedure replaces the SYSPROC.NNSTAT procedure and incorporates the following improvements:
  • Collect statistics from remote data sources that support statistics collection.
  • Include table cardinality, the number of null values in a column of a table.
• You can virtualize text files that contain column headers in data sources in Cloud Object Storage. Column headers help categorize the data in columns for readability.
  • For more information, see Creating a virtualized table from files in Cloud Object Storage in Watson Query.

Documentation is translated into more languages

19 Sept 2022

You can now view the Cloud Pak for Data as a Service documentation in these languages:

• Brazilian Portuguese
• Simplified Chinese
• Traditional Chinese
• Czech
• French
• German
• Italian
• Japanese
• Korean
• Polish
• Spanish
• Turkish

The documentation is now automatically translated weekly. See Language support.

Provide feedback on the documentation

16 Sept 2022

You can now provide feedback on documentation content. Just scroll to the bottom of any page and select an option.

Stored procedures in DataStage flows are supported for more data sources

09 Sept 2022

You can now use stored procedures in the following connectors:

• Db2 for i
• Db2 for z/OS

For more details, see Using stored procedures.

Deprecated connections

09 Sept 2022

The following connections are deprecated:

• The IBM Cloud Databases for MySQL connection is deprecated by IBM Cloud. All instances on IBM Cloud will be removed after March 1st, 2023.
• The IBM Db2 Event Store connection is deprecated and will be removed in a future update of Cloud Pak for Data as a Service.

New Data governance tutorial for the Data fabric trial

02 Sept 2022

You can now experience how to govern data that you virtualized with Watson Query to implement a data fabric solution with the Data governance use case by taking this new tutorial: Govern virtualized data

This tutorial is a continuation of three other tutorials from the Data governance use case, which requires Watson Knowledge Catalog and the Virtualize external data tutorial from the Data integration use case, which requires the Watson Query service.

For more information on what the data fabric is, see The Cloud Pak for Data as a Service data fabric solution.

To take this tutorial:

• If you're a new user, sign up for the Data governance use case, and then take the Govern virtualized data tutorial.
• If you're an existing user of Cloud Pak for Data as a Service, you don't need to sign up again. You can try the Govern virtualized data tutorial by provisioning the required services and completing the prerequisite tutorials.

Support for migrating Db2 server-type data connection objects from traditional DataStage

02 Sept 2022

Traditional DataStage supports data connection objects of the type Db2 server. When you migrate these data connection objects to modern DataStage, they are automatically converted to Db2 connector objects so that you can still use them in your DataStage flows and jobs.

Use new functions in the DataStage Transformer stage

2 Sept 2022

• You can now use the ConvertDatum, NextValidDate, Fold, Fmt, and Rmunprint functions in the Transformer stage as part of your DataStage flows. For the full list of available functions, see Parallel transform functions.
• The Transformer stage now supports partitions.
• You can now use type-ahead search in the Transformer stage for functions,columns, and variables.

Connect to more data sources in DataStage

2 Sept 2022

You can now include data from these data sources in your DataStage flows:

• Cognos Analytics
• IBM Match 360
• SAP IQ

For the full list of DataStage connectors, see DataStage connectors.

Orchestrate DataStage flows with Watson™ Studio Pipelines

2 Sept 2022

You can now create a pipeline to run a sequence of DataStage flows. You can add conditions, loops, expressions, and scripts to a pipeline. For details, see Orchestrating flows.

Support for migrating sequence jobs in DataStage to Watson™ Studio Pipelines

2 Sept 2022

You can now migrate sequence jobs from traditional DataStage to modern DataStage as pipeline flows. For details, see Migrating DataStage jobs.

Use multiple input links on the Db2 (optimized) connector in DataStage

2 Sept 2022

You can now use multiple input links on the Db2 (optimized) connector and assign an individual action to each link.

Create jobs to run SPSS Modeler flows

1 Sept 2022

You can now create jobs to run SPSS Modeler flows. See Creating and managing jobs in a project and Creating jobs in SPSS Modeler.

Add catalog assets from within a project

18 Aug 2022

You can now add catalog assets to a project from within that project. Previously, you had to add catalog assets to the project from within a catalog. For details, see Adding catalog assets to a project.

Migrate older SPSS Modeler flows before 18 November 2022

18 Aug 2022

Action required If you have SPSS Modeler flows that you created before January 2019, migrate them by opening them before 18 November 2022. Otherwise, the flows might become unavailable.

Export reports for model factsheets and entries (Watson Knowledge Catalog)

19 Aug 2022

Generate a report from a factsheet or model entry in PDF, HTML, and DOCX format so you can share or print the details about a model being tracked in a model inventory. See Generating reports for factsheets and model entries.

Watson Natural Language Processing is GA! (Watson Studio)

11 Aug 2022

The Watson Natural Language Processing library is now generally available.

Use the Watson Natural Language Processing library to turn unstructured data into structured data, making data easier to understand and use in your Python notebooks. This premium library gives you instant access to pre-trained, high-quality text analysis models in over 20 languages. These models are created, maintained, and evaluated for quality by experts from IBM Research and IBM Software for each language. The Watson Natural Language Processing library is now included with the Decision Optimization library in a premium environment template. For details, see Watson Natural Language Processing library.

You can continue to use existing beta environment template for Natural Language Processing, Default Python 3.8 + Watson NLP XS (beta), until August 31. Switch to the new environment template, DO + NLP Runtime 22.1 on Python 3.9, to continue working. See Changing environments in notebooks.

Removal of the prefix "IBM" from notebook environment templates (Watson Studio)

11 Aug 2022

The prefix "IBM" has been removed from all IBM Runtime 22.1 environment templates. For example, the IBM Runtime 22.1 on Python 3.9 XXS template is now called Runtime 22.1 on Python 3.9 XXS. Similarly, when you create your own template, the prefix "IBM" has been dropped from the software version you can select. For details, see Compute resource options for the notebook editor in projects.

Access data from SingleStoreDB

11 Aug 2022

Use the new SingleStoreDB connection to access data from its storage and analytics service. For information, see SingleStoreDB connection.

Automatic term assignment now considers removed terms (Watson Knowledge Catalog)

11 Aug 2022

In metadata enrichment results, users can remove terms from a column that they think are inaccurate. A new machine learning model that is trained on such negative feedback now contributes to the overall confidence score for automatic term assignment to reduce inaccuracies. See Term assignment.

Updates to Watson Query

11 Aug 2022

Watson Query has a new navigation menu that makes it easy to manage more than one set of virtualized data at once. Launch Watson Query to use the new side menu, updated breadcrumbs, and an improved scaling interface for enterprise plans.

Watch videos using picture-in-picture

05 Aug 2022

Documentation topics with embedded videos just got better! When the video is playing, you can scroll through the rest of the page and still see the video in picture-in-picture mode. This allows you to watch the video while you are completing the steps in a tutorial. And you can click the timestamps to watch a preview of the next task in picture-in-picture mode.

Pause animated image

Play animated image

Try out the data fabric tutorials to see video picture-in-picture in action!

New API capabilities and behaviors

01 Aug 2022

The IBM Watson Data assets API for assigning roles includes the following improvements:

• You can assign user groups as asset members in bulk.
• You can specify asset editor and asset viewer roles when you assign asset members.
• You can assign multiple asset owners and an asset creator to an asset.
• When you add an asset to a project or publish or promote an asset, you become the asset creator and the list of asset owners in the source asset is preserved in the target asset.

Easier access to what's new

26 July 2022

You can now jump to the what's new from the tile in the welcome area of the Cloud Pak for Data as a Service home page.

Increased flexibility for data tables and Python extensions in Decision Optimization experiments (Watson Studio and Watson Machine Learning)

28 July 2022

You can now change the data types (number or string) of table columns in the Prepare data view of your Decision Optimization experiment. These types will be used when you save your scenario as a model for deployment.

See Prepare data view.

You can now add Python extensions to your Decision Optimization experiment environments so that you can include additional Python libraries.

See Configuring environments.

Name change for the IBM SQL Query connection

22 July 2022

The IBM SQL Query connection has been renamed to IBM Cloud Data Engine. Your previous settings for the connection remains the same. Only the connection name has changed.

Visualize your data with Dataview visualizations

22 July 2022

Now you can use Dataview visualizations to explore data from different perspectives so you can identify patterns, connections, and relationships to quickly understand large amounts of information.

To create and work with visualizations in your project, you select a data asset from the Assets tab and click the Visualization tab. Select a chart type and create and save the visualization. Your saved Dataview visualizations are listed as Visualization assets in your project. Graphical charts are generated based on a sample data set of up to 5000 records.

For details, see Visualizing your data in Data Refinery.

Add relationships between assets more easily

20 July 2022

When you add a relationship between assets in a catalog, you can now easily find the target asset:

• You can filter by the workspace (catalog, project, or deployment space) or by the asset type.
• You can search for assets by name.
• On the asset page in a catalog, the section for relationships is now called Related assets.

For details, see Adding relationships between assets.

Create relationships between assets across catalogs, projects, and spaces

20 July 2022

You can now create and edit relationships between assets across different catalogs, projects, and spaces that you have access to. With the new asset relationships tearsheet, you can search for assets outside of the current catalog with filters for asset type and asset location.

Add supporting features to improve your AutoAI Time Series model predictions

15 July 2022

When you create an AutoAI Time Series experiment, you can now specify supporting (or exogenous) features, to improve the forecast. For example, in a time series experiment that forecasts energy usage, you can train the model to consider supporting features such as daily temperatures to make the forecast more accurate. If you know the future value for a supporting feature, you can supply that as input when you deploy the model. For example, if you are predicting t-shirt sales, you can include future data on sales and promotions that could influence the forecast. For details on how to include supporting features in your time series experiment, see Building a time series experiment.

Improved test interface for online deployments

15 July 2022

When you create an online deployment for a model, you now have improved methods for providing input data from the Test tab of the deployment. These include:

• Enter data directly in the form
• Download a CSV template, enter values, and upload the input data
• Upload a file containing input data from your local file system or from the space
• Change to the JSON tab and upload or enter your input data as JSON code

For details, see Creating an online deployment.

Active Directory supported for the Microsoft SQL Server connection

11 July 2022

You can now select Active Directory for Microsoft SQL Server authentication. This enhancement means that you can take advantage of the credentials that are stored in an NTLM account database instead of on the Microsoft SQL Server. For information, see Microsoft SQL Server connection.

Use in-app assistance to find information in the documentation

08 July 2022

The new in-app assistance provides recommended articles in the documentation based on what page you're viewing in the product. No need to search the documentation in a separate tab or window. The assistance will do that for you. Open the assistance from the top banner . Close and open the assistance when you move to a new page to see updated recommended articles. You can also enter search terms to find information fast, launch tours where applicable, and access links for Additional support.

Easier upgrades for data fabric services

07 July 2022

You can now quickly upgrade Cloud Pak for Data services that are included in the data fabric use cases. Simply click the Buy button in the dashboard, and you can see a list of your provisioned data fabric services and their current plan. Checkmark the services you want to upgrade and select a plan. You can also view a pricing summary for each service, then upgrade all of them in one step. For upgrade instructions, see Buying Cloud Pak for Data services.

"Default Spark 3.0 & R 3.6" environment discontinued (Data Refinery)

04 July 2022

The Default Spark 3.0 & R 3.6 environment will no longer be available effective July 07, 2022.

Action required If you have any Data Refinery flow jobs set up with the Default Spark 3.0 & R 3.6 environment or a custom environment that uses Spark 3.0, the jobs will fail. Change the environment to Default Spark 3.2 & R 3.6 or Default Data Refinery XS or a custom environment that does not use Spark 3.0.

For information about environments for Data Refinery, see Compute resource options for Data Refinery in projects.

Learn more about the data fabric

30 June 2022

You can now learn more about how to implement the data fabric solution with Cloud Pak for Data as a Service. See Data fabric solution overview. To experience implementing the data fabric, take the data fabric tutorials.

Monitor workflow tasks (Watson Knowledge Catalog)

30 June 2022

Workflow administrator can now view metrics for the active tasks. The Task status page includes graphic overview of ownership status and due date for all active tasks. You can also filter the task list and set multiple tasks back to unclaimed at once.

See Managing workflow tasks.

New Spark 3.2 environment for running Data Refinery flow jobs

24 June 2022

You can now select Default Spark 3.2 & R 3.6 when you select an environment for a Data Refinery flow job. The Default Spark 3.2 & R 3.6 environment includes enhancements from Spark. The new environment uses the same capacity unit hours (CUHs) as the other Default environments.

For information about environments for Data Refinery, see Compute resource options for Data Refinery in projects.

New PMML software specification for (Watson Studio and Watson Machine Learning)

23 June 2022

Action required PMML models with spark-mllib_3.0 are deprecated but will not be removed. Model deployments with the deprecated specification will stop working on July 7, 2022. Create new PMML models with the pmml-3.0_4.3 software specification or update existing pmml models with the pmml-3.0_4.3 software specification if there are no existing deployments. For details on changing notebook environments for PMML models, see Changing notebook environments. For details on managing deployment frameworks, see Managing outdated software specifications.

New translations of the documentation!

22 June 2022

The Cloud Pak for Data as a Service documentation is newly translated into the following languages:

• Brazilian Portuguese
• French
• German
• Spanish
• Japanese
• Korean

You can now easily switch between languages when you view the documentation. Previously, to view the documentation in a different language, you reset your browser preferences. Now you can select the language you want from the language selector at the bottom of each page.

Publishing enrichment results just got easier (Watson Knowledge Catalog)

23 June 2022

You can now publish enrichment results without getting redirected to the project's publish flow. After publishing, you're back in the enrichment results UI and can continue to work there. The Publish status for each asset is shown on the Assets tab of the enrichment results.

See Publishing enrichment results.

Improve your IBM Match 360 matching algorithm by reviewing record pairs

10 June 2022

Review pairs of records to train the IBM Match 360 matching algorithm how to decide which records get matched into master data entities. During a pair review, a data steward compares records to determine whether they are a match.

When the pair review is complete, IBM Match 360 analyzes the responses and recommends adjustments to your matching algorithm's weights and matching thresholds. The more pairs you review, the better the tuning recommendations will be. A data engineer can then decide whether to apply the recommendations.

For information about pair reviews, see Customizing and strengthening your matching algorithm.

Define and work with relationships between your IBM Match 360 records

10 June 2022

Find new connections within your master data by adding relationship information to IBM Match 360. Now you can add relationship types to your data model, and then either bulk load relationship data assets or manually define relationships between records. Explore the relationships between your records to gain new insight about your data.

For information about working with relationships in your master data, see Exploring relationship data.

Save and load snapshots of your IBM Match 360 configuration

10 June 2022

Now you can use configuration snapshots to create point-in-time versions of your master data configuration settings, including your data model and matching settings. Load a snapshot to return your master data configuration to a previous version, or share snapshots across service instances to ensure consistency.

For information about working with snapshots, see Saving and loading master data configuration snapshots.

Support for Spark 3.2 and deprecation of Spark 3.0 for Watson Studio and Watson Machine Learning

01 June 2022

Spark 3.2 is now supported and Spark 3.0 is deprecated as a machine learning framework, notebook environment, and RStudio runtime. Update your assets to use Spark 3.2 instead. Support for training assets will be discontinued on Jun 22, 2022. Support for deploying and scoring models will be discontinued on Jul 7 2022 and existing deployments using Spark 3.0 specifications will be removed. For details on migrating an asset to a supported framework and software specification, see Managing frameworks and software specifications. For details on notebook environments, see Compute resource options for the notebook editor in projects.

Environment updates for Decision Optimization (Watson Studio and Watson Machine Learning)

25 May 2022

Action required You must change the enviroments for your Decision Optimization experiments and models that run on Python 3.8 and CPLEX 12.10 environments:

• Python 3.8 is now removed. You must use the default version Python 3.9. To change your default environment for Decision Optimization experiments, see Selecting a different run environment for a particular scenario. For deployed models that use older versions, you must update your Python version with the REST API, see Changing Python version for an existing deployed model with the REST API.
• CPLEX 12.10 is now removed and its equivalent do_12.10 runtime is no longer supported. CPLEX 20.1 remains the default and CPLEX 22.1 with its new runtime do_22.1 is now available. If you have already deployed your model with a CPLEX runtime that is no longer supported, you can update your existing deployed model using either the REST API or the UI.

Metadata enrichment: assign or remove business terms or data classes from selected assets in one go (Watson Knowledge Catalog)

26 May 2022

In the enrichment results, you can now assign business terms to or remove them from a selected set of assets or columns at once. For columns, you can also assign data classes to or unassign them from a several columns in one go. See Making bulk changes to term and data class assignments.

To add collaborators or change collaborator roles, project Admins must belong to the project creator's IBM Cloud account

26 May 2022

If you are a project Admin in a different IBM Cloud account than the project creator, you don't have permission to add collaborators or change collaborator roles. Ask another project Admin to add collaborators or make the change.

New stages in DataStage

26 May 2022

The following stages are now available for you to use in DataStage flows:

• Complex Flat File (CFF)
• Hierarchical stage: REST step
• Match Frequency stage
• One-source Match stage

For more information and the full list of stages, see DataStage stages and QualityStage stages.

Download a DataStage flow and its dependencies as a single file

26 May 2022

You can download an individual DataStage flow and its dependencies conveniently bundled together as a ZIP file. You can then import the file into another project. Dependencies include items such as connections, subflows, and parameter sets.

For details, see Downloading and importing a DataStage flow and its dependencies.

Generate new nodes from table output in SPSS Modeler

16 May 2022

When viewing table output, you can now select one or more fields, click Generate, then select a node to add to your flow.

New "Flow settings" give you more options for Data Refinery flows

20 May 2022

The Data Refinery flow settings give you more properties that you can use to control the data in your Data Refinery flows and offer a new capability to edit the sample size of the data while you refine your data.

Access the Data Refinery flow settings from the toolbar in Data Refinery.

Use the Data Refinery flow settings to do the following actions:

Source data sets:

• Edit the sample size: Use this new feature to adjust the sample size while you are refining the data. Adjusting the sample size can help you run the Data Refinery flows faster when you have a large data set.
• Edit the source properties: Previously you could only specify format options for CSV or delimited files. Now there are options for more file types and more options for data from connections.
• Change the source of a Data Refinery flow: Now you can replace more than one source data set in one place. (For Join and Union operations)

Target data set:

• Change the target location of a Data Refinery flow
• Edit the target properties: You have more options for the different types of data, including data from connections.
• Enter a description of the target data

Existing Data Refinery flows or Data Refinery flow jobs are not affected by these changes unless you open the Flow settings and make changes.

For information, see Managing Data Refinery flows.

New step options give you more control of your Data Refinery flow

20 May 2022

Data Refinery introduces new options for the steps: Duplicate, Insert step before, and Insert step after. These options give you greater flexibility and control of the Data Refinery flow.

Access these options from the Steps pane.

For information about all the actions you can do with steps, see Managing Data Refinery flows.

Control the placement of a new column in a Data Refinery flow

20 May 2022

When you use an operation that can create a new column in the Data Refinery flow and you select Create a new column for results, you can now select to place the new column to the right of the original column.

This new selection is available for these operations:

• Calculate
• Conditional replace
• Convert column type
• Convert column value to missing
• Extract date or time value
• Math
• Replace missing values
• Replace substring
• Text
• Tokenize

For information about GUI operations, see GUI operations in Data Refinery.

Metadata enrichment now also provides suggestions for data classes (Watson Knowledge Catalog)

20 May 2022

When you run metadata enrichment, profiling now also provides data class suggestions for columns. You can see them in a column's governance details. Assigned and suggested data classes are picked based on the new thresholds that you can set in the project settings for metadata enrichment. See Data class assignment settings.

Enhancements for DataStage connectors

20 May 2022

Certain connectors now provide a faster way to test and add metadata from their associated connections.

When you create the connection, the Test connection button on Add connection page now works for these connections. (Previously, you did not have a way to test the connection in the user interface.)

• Apache Kafka
• Db2 (optimized)
• Netezza Performance Server (optimized)
• ODBC
• Oracle (optimized)
• Salesforce.com (optimized)
• Teradata (optimized)

After you create the connection, in DataStage you can drag the Asset browser to the canvas, select a connection and drill down to add or preview the data for these connectors. (Previously, your only option was to drag a connector to the canvas, double-click it to open its Details card, and then go to Properties > Connection and select the connection.)

• Db2 (optimized)
• Netezza Performance Server (optimized)
• ODBC

For the full list of DataStage connectors, see DataStage connectors.

SPSS Modeler: Text Analytics improvements

12 May 2022

SPSS Modeler provides specialized nodes for handling text. From a Text Mining node, you can choose to launch the newly improved Text Analytics Workbench (formerly known as the Interactive Workbench). After extensive user research, the workbench has been redesigned. The documentation has also been updated to reflect the new design, including a new video and updated tutorial. See Text Analytics.

Connect to more data sources in DataStage

13 May 2022

You can now include data from these data sources in your DataStage flows:

• Generic S3
• Teradata (optimized)

For the full list of DataStage connectors, see DataStage connectors.

Running metadata enrichment made easy (Watson Knowledge Catalog)

13 May 2022

You can now run enrichment from the metadata enrichment results directly instead of rerunning the job from the Jobs page. Plus, you can select to run enrichment for the entire scope of assets or only for a selected subset. See Running enrichments manually.

Try more Watson Knowledge Catalog features with new plans

05 May 2022

You can now try out almost all Watson Knowledge Catalog features for free with the updated Lite plan, or pay only for what you use with the new Standard plan.

You can choose from the following new Watson Knowledge Catalog offering plans:

• The new Standard plan charges per catalog asset and for compute usage, based on capacity unit hour (CUH) rates when you run profiling, tools, and jobs. The plan does not include monthly instance fees or authorized user fees.
• The new Enterprise Bundle plan charges a monthly instance fee for 100,000 catalog assets and 2500 CUH per month. You pay for more catalog assets and compute usage. The plan does not include authorized user fees.

If you have the Lite plan, your plan is automatically updated. You now have access to most Watson Knowledge Catalog features. Many of the limits for assets and governance artifacts are increased. However, the monthly compute usage limit is decreased to 25 CUH.

If you previously provisioned the Standard, Professional, or Enterprise plan, you can keep your legacy plan for the next year. If you want to change to the new Standard or Enterprise Bundle plan, you can follow the steps for Managing services.

See Watson Knowledge Catalog service plans.

New home for asset activities (Watson Knowledge Catalog)

05 May 2022

In catalogs and projects, information about asset activities is now available in a side panel. Open an asset in a catalog or a project, and access its activities by clicking . See Activities.

Control data based on location (Experimental) (Watson Knowledge Catalog)

04 May 2022

You can now try the experimental feature of controlling access to data assets based on location. You can create data location rules to ensure that data privacy and location-aware regulations are enforced when you move data from one physical or sovereign location to another.

To try out this experimental feature, respond to this post for an example tutorial and additional information about the API.

See Data location rules.

New compute usage limit for Watson Studio Lite plans

29 April 2022

Watson Studio Lite plans now have a monthly compute usage limit of 10 CUH to run jobs and tools. This limit applies to all existing and new Lite plans. May 2022 is the first full month with the lower CUH limit.

If you use more than 10 CUH per month, you have these choices:

• Upgrade to the Professional plan. Because the Professional plan charges only for the CUH that you use, you can upgrade without incurring other charges.
• You can prolong your compute usage by updating your assets to use environments with lower CUH rates. For example, you can change your notebook environment.

Save an AutoAI Time Series model pipeline as a notebook (Watson Studio, Watson Machine Learning)

29 April 2022

You can now save a pipeline from an AutoAI Time Series experiment as a notebook so that you can review the code and algorithms used to generate the pipeline. For details, see Building a time series experiment.

Metadata enrichment: new service for automatic term assignment (Watson Knowledge Catalog)

29 April 2022

Linguistic name matching is now also available as a service for automatic term assignment. When this service is enabled, terms can be assigned based on the similarity between the term and the name of the asset or column. By default, this service is enabled for all existing and new projects. See Metadata enrichment default settings.

New pricing plans for Watson Query (Effective 1 May 2022)

29 April 2022

Enterprise pricing has changed to remove charges per Watson Query instance and to lower charges for Virtual Processor Core (VPC) hours for your Watson Query service. The service is metered and consumed when it is provisioned, even when you are not working in the service. The 250 free Virtual Processor Core-Hours per month has been discontinued. See Watson Query offering plans.

Change to deployment serving name requires action (Watson Machine Learning)

21 April 2022

Action required Starting on 4 May 2022, serving names that users assign to online deployments must be unique per region. You can check if an existing serving name is unique using the API call GET /ml/v4/deployments?serving_name={serving_name}&conflict=true API. If the GET call returns a status code of 204, the name is unique and available for use. If the call returns a status code of 409, the serving name already exists or might have a conflict. Review the response and take action to update the serving name using PATCH API if required. Starting on 4 May 2022, prediction requests associated with serving names where serving name exists more than once will fail with an error requiring the user to update the name. For details on serving names, see Creating an online deployment. For details on using the PATCH command, see Update the deployment metadata. If you need assistance with the update, contact IBM Support.

View your Data Refinery data in a CSV file without running a Data Refinery flow job

22 April 2022

You can now export the data at the current step in your Data Refinery flow to a CSV file without saving or running a Data Refinery flow job. This enhancement gives you the ability to quickly save and view data that is in progress. Click the text under the Export icon on the toolbar.

For more information, see Managing Data Refinery flows.

Metadata enrichment at a glance

22 April 2022

A new side panel provides a summary of relevant information about a metadata enrichment such as enrichment and sampling options, the associated job and its schedule.

Updates for DataStage

15 April 2022

Reject links are now supported for the MQ, Teradata, and ODBC connectors. Stored procedures within the SQL Server connector are now supported. You can now disable compilation when you import DataStage flows. You can import and download individual flows along with dependencies in the UI.

For the full list of DataStage connectors, see DataStage connectors.

Scripting enhancements in SPSS Modeler

13 April 2022

A new Scripting icon is available on the toolbar that opens a redesigned scripting panel. See Scripting overview.

Decision Optimization updates (Watson Studio and Watson Machine Learning)

13 April 2022

You can see the following updates to Decision Optimization:

• The default Python for Decision Optimization users is now 3.9. Python 3.8 is now deprecated and Python 3.7 will be removed soon. For more details see Decision Optimization notebooks.
• You can now run and delete multiple scenarios in a Decision Optimization experiment. For more details see Decision Optimization views and scenarios.

The new projects UI replaces the legacy UI

07 April 2022

The new projects UI has replaced the legacy UI and your work has not been affected. The projects experience has been updated to make it easier and more efficient to work and collaborate in a project. Experience enhanced asset organization, asset relations, improved navigation, and built-in guidance.

Find what you need quickly with the new search experience

07 April 2022

You can now quickly evaluate results when you search for assets or governance artifacts with the global search field. The new search results experience shows the context for your search term and provides many filters based on more properties.

You also get better results. More asset and artifact properties are searched. When you search for phrases in English, natural language analysis prioritizes common phrases and discards unimportant words.

You can now include a quoted phrase within a longer search string.

See Searching for assets and artifacts across the platform.

Deprecation and removal of IBM Analytics Engine classic plans and Amazon EMR

07 April 2022

Starting 07 April 2022, new users will not be able to create IBM Analytics Engine instances using the Lite, Standard-Hourly, or Standard-Monthly plans or any Amazon Elastic Map Reduce (EMR) instances in which to run notebooks.

Existing users can still create IBM Analytics Engine classic instances and any Amazon EMR instances until 30 June 2022. Thereafter all associated notebooks should be reassigned to supported Spark runtime environments available in Watson Studio.

The IBM Analytics Engine Classic plans and Amazon EMR will be removed on 9 November 2022.

The new governance artifacts experience replaces the legacy experience (Watson Knowledge Catalog)

08 April 2022

If you were using the legacy governance artifacts experience with Watson Knowledge Catalog, you were switched to the new governance artifacts experience on 07 April 2022. You had the legacy experience only if you provisioned Watson Knowledge Catalog before April 2021 and you hadn't already moved to the new experience.

Here's what happened during the move:

• All your existing business terms, policies, and data protection rules were permanently deleted. You can't revert to the legacy experience.
• All business term, data classes, and classification assignments on data assets became invalid.
• Any data masking that you configured with data protection rules was removed.
• Profiles of data assets are updated so that the classification results use the new data classes.

Action required Here's what you need to do now:

• Re-create your business terms, classifications, and data protection rules.
• Remove invalid business terms and classification assignment from assets in catalogs.
• Assign your new business terms and classification to assets in catalogs.
• Assign Watson Knowledge Catalog roles to your users. See Assign Watson Knowledge Catalog roles to users.

If you have any questions or concerns related to moving to new version of governance artifacts, you can open a support ticket.

Upcoming changes to Watson Knowledge Catalog plans

07 April 2022

Starting on 02 May 2022, you can choose from the following new Watson Knowledge Catalog offering plans:

• The new Standard plan will charge per catalog asset and for compute usage, based on capacity unit hour (CUH) rates when you run profiling, tools, and jobs. It will not include instance or authorized user fees.
• The new Enterprise bundle plan will charge a monthly instance fee for 100,000 catalog assets and 2500 CUH per month. You pay for more catalog assets and compute usage. It will not include authorized user fees.

If you have the Lite plan, your plan will automatically update. You will have access to all Watson Knowledge Catalog features, except Knowledge Accelerators. Many of the limits for assets and governance artifacts are increased, however, the monthly compute usage limit is decreased to 25 CUH.

If you have the current Standard, Professional, or Enterprise plan, you can keep your plan for the next year. If you want to change to the new Standard or Enterprise bundle plan, you can follow the steps for Managing services, starting on 02 May 2022.

View the data types from the automatic first step in the Data Refinery "Convert column type" operation

08 April 2022

When you open a file in Data Refinery, the Convert column type operation is automatically applied as the first step if it detects any nonstring data types in the data. Data types are automatically converted to inferred data types. Now you can confirm what data type each column's data was converted to. The information includes the format for date or timestamp data. Click Edit from the overflow menu to view the data types.

For information, see GUI operations.

Changes to Data Refinery "Convert column type" for timestamp and date data

08 April 2022

The following data type is no longer automatically converted:

• Date and Timestamp strings that use two digits for the year

The automatic conversions in existing Data Refinery flows are not affected.

Metadata enrichment updates (Watson Knowledge Catalog)

07 April 2022

The metadata enrichment results now include the enrichment status for each asset in the enrichment. Also, you can now change the review status for several assets or columns at once. See Metadata enrichment results.

In addition, you will now receive notifications for enrichment job run events such as start or completion.

Simplified Watson Studio plans

01 Apr 2022

The new Professional plan for Watson Studio is available now. Changes to the Lite plan are coming later this month.

Watson Studio now has a single paid plan, called the Professional plan, that replaces the Standard and Enterprise plans. The Professional plan charges only for compute usage, based on capacity unit hour (CUH) rates when you run tools and jobs. It does not include instance and authorized user fees. As of 1 April 2022, the Professional plan is the only paid plan option that you can select. For more information about the Watson Studio Professional plan, see Watson Studio service plans. You can also refer to IBM Cloud catalog: Watson Studio.

If you currently have the Standard or Enterprise plan, you can keep that plan indefinitely. If you want to change to the Professional plan, follow the steps for Managing services.

Starting 29 April 2022, all new and existing Watson Studio Lite plans will have a monthly limit of 10 CUH to run jobs and tools. Because the Professional plan charges only for the CUH that you use, you can upgrade to a paid plan without incurring other charges. May 2022 is the first full month with the lower CUH limit. If you want to prolong your runtime usage, you can update your assets to use environments with lower CUH rates. For example, you can change your notebook environment.

End of support for Core ML deployments for iOS

1 Apr 2022

Core ML, or virtual, deployments for use with iOS are deprecated. Support for this deployment type will end on May 4, 2022.

New PMML software specification for (Watson Studio and Watson Machine Learning)

1 Apr 2022

Action required PMML models with spark-mllib_2.4 are deprecated but will not be removed. Model deployments with the deprecated specification will stop working on May 4, 2022. Create new PMML models with the pmml-3.0_4.3 software specification or update existing pmml models with the pmml-3.0_4.3 software specification if there are no existing deployments. For details on changing notebook environments for PMML models, see Changing notebook environments. For details on managing deployment frameworks, see Managing outdated software specifications.

Reminder: Switch from the legacy governance artifacts experience (Watson Knowledge Catalog)

24 Mar 2022

If you are using the legacy governance artifacts experience with Watson Knowledge Catalog, you will be switched to the new governance artifacts experience on April 4, 2022. You have the legacy experience only if you provisioned Watson Knowledge Catalog before April, 2021 and you haven't already moved to the new experience. The new governance artifact experience became the default experience in April, 2021.

If you have any questions or concerns related to moving to new version of governance artifacts, you can open a support ticket.

Data Refinery flow jobs that use an environment template with Spark 2.4 must be updated

24 Mar 2022

Action required If you have a Data Refinery flow job that uses Spark 2.4, for example, the "Default Spark 2.4 & R 3.6" environment template, the job will fail. Change the environment template to "Default Spark 3.0 & R 3.6," "Default Data Refinery XS," or create your own "Spark 3.0 & R 3.6" environment template. For information, see Compute resource options for Data Refinery in projects.

Framework and software specification changes for (Watson Studio and Watson Machine Learning)

17 Mar 2022

Action required The following changes to framework and software specifications might require user action to update assets.

• CPLEX 12.10 model type is deprecated in Watson Studio and Watson Machine Learning. Support for CPLEX 12.10 will end on May 18, 2022. Migrate to the latest version, CPLEX 20.1. For details on Decision optimization model types, see Model deployment.
• Python 3.8 is deprecated and will be removed on May 18, 2022. Update your assets and deployments to use IBM Runtime 22.1, based on Python 3.9, with associated notebook environments and software specifications. For details on supported notebook environments for IBM Runtime 22.1, see Changing notebook environments. For details on deployment frameworks, see Managing frameworks and software specifications.

New connector for DataStage: Microsoft Azure Cosmos DB

11 Mar 2022

You can now include data from a Microsoft Azure Cosmos DB data source in your DataStage flows.

For the full list of DataStage connectors, see DataStage connectors.

Federated learning now supports Python 3.9

10 Mar 2022

Use Python 3.9 for your Federated Learning experiments with these frameworks:

• Tensorflow 2.7
• PyTorch 1.10
• Scikit-learn 1.0.2

Python 3.8 and all associated frameworks are being deprecated. Upgrade your Federated Learning experiments to Python 3.9 and implement fully supported frameworks. For more information, see Frameworks and Python version compatibility.

New connector for DataStage: Microsoft Azure SQL Database

04 Mar 2022

You can now include data from a Microsoft Azure SQL Database data source in your DataStage flows.

For the full list of DataStage connectors, see DataStage connectors.

New DataStage features

04 Mar 2022

You can now use the UniChar and UniSeq functions to convert decimal values into unicode in the Transformer stage. See Transformer stage.

End of support for Deep Learning as a Service (Watson Machine Learning)

2 March 2022

Support for Deep Learning as a Service and the Deep Learning Experiment Builder is deprecated and will be discontinued on April 2, 2022. No replacement is planned on Cloud Pak for Data as a Service, but support for Deep Learning experiments will continue to be supported on Cloud Pak for Data, with Watson Machine Learning Accelerator. Note that this discontinuation does not affect Watson Studio k80 GPU notebooks. You can continue to run the GPU notebooks, but Deep Learning notebooks, models, and deployments that rely on Watson Machine Learning REST APIs will not be supported.

DataStage features

25 Feb 2022

The following stages are now available for you to use in DataStage flows:

• Combine Records
• Make Subrecords
• Make Vector
• Promote Subrecords
• Split Subrecord
• Split Vector

For more information, see DataStage stages.

Support for Python 3.9 and deprecation of Python 3.7 (Watson Studio and Watson Machine Learning)

25 Feb 2022

Action required You can now use IBM Runtime 22.1, which includes the latest data science frameworks on Python 3.9, to run Watson Studio Jupyter notebooks, train models, and run Watson Machine Learning deployments. Python 3.7 is now deprecated and will be removed on Apr 14, 2022. Update your assets and deployments to use IBM Runtime 22.1 instead. Similarly, XL Python environments in Watson Studio and Watson Machine Learning are now deprecated and will also be removed on April 14th, 2022. Reassign any associated assets to supported configurations accordingly.

• For information on the IBM Runtime 22.1 release and the included environments for Python 3.9, see Changing notebook environments.
• For details on deployment frameworks, see Managing frameworks and software specifications.
• For details on selecting a Python version for your Decision Optimization experiments, see Run configuration pane and Solve parameters

Federated Learning (Watson Machine Learning) announcements

24 Feb 2022

There are several new feature enhancements for Federated Learning.

• The Federated Learning module is now part of the Python client for Watson Machine Learning. All API functions that contain ibmfl will be removed. Please update to the latest version of the Watson Machine Learning module and the party connector script. For more details, see Creating the Federated Learning experiment.
• Python 3.7 is being deprecated from older versions. Please update to at least Python 3.8 for continuation of use in Federated Learning.
• Scikit-learn 1.0 is now supported with Python 3.9.

Create custom constraints in the Decision Optimization Modeling Assistant

24 Feb 2022

The Modeling Assistant provides you with many constraint suggestions for your problem domain which can be customized. You might, however, want to express constraints beyond those that are predefined for the given domains. You can now achieve this by using more advanced custom constraints that use Python DOcplex. See Advanced custom constraints for an example illustrating how you can create these.

Easily configure the environment for your Decision Optimization experiment

25 Feb 2022

When building models in an experiment, the Run parameters pane in the Build model view now contains an Environment tab. Here you can see the default run environment that is used for the solve when you click Run in the Build model view. You can create environments using the Environment tab in the Information pane in the Overview. For more details see Configuring your hardware and software.

Import metadata from more data sources (Watson Knowledge Catalog)

24 Feb 2022

You can now run metadata import also for Apache Cassandra and Teradata data sources.

Access data from S3-compatible data sources

18 Feb 2022

Use the new Generic S3 connection to access data from a storage service that is compatible with the Amazon S3 API. For information, see Generic S3 connection.

Snowflake connection supports federated authentication provided by Okta for enhanced security

18 Feb 2022

If your company uses single-sign on (SSO) with native Okta authentication, the user interface has a new field where you can enter the Okta URL endpoint for your Okta account.

For information about the Snowflake connection, see Snowflake connection.

DataStage supports creating message handlers.

11 Feb 2022

Error messages and warnings are written to the log when you run a job. You view messages and alerts in the Logs panel. You can choose to handle specific errors differently by creating message handlers.

Message handlers are rules that define how messages are expressed. Youc an use them to suppress messages from the log or determine whether an error message should be a warning or informational.

Try this feature by expanding a log message, selecting the ellipsis next to the Message ID, and promoting or demoting to make the message a warning or informational. You can also choose to suppress from logs. This option is not available for errors.

PostgreSQL is a supported database to use with reporting on Watson Knowledge Catalog data

04 Feb 2022

When you send your Watson Knowledge Catalog data to an external database to generate reports, you can now choose a PostgreSQL database in addition to a Db2 database. For details, see Reporting on Watson Knowledge Catalog data.

Get ready for the switch from the legacy governance artifacts experience (Watson Knowledge Catalog)

04 Feb 2022

If you are using the legacy governance artifacts experience with Watson Knowledge Catalog, you will be switched to the new governance artifacts experience on April 4, 2022. You have the legacy experience only if you provisioned Watson Knowledge Catalog before April, 2021 and you haven't already moved to the new experience. The new governance artifact experience became the default experience in April, 2021.

The new governance artifacts experience includes these new features:

• More types of governance artifacts, such as reference data sets and governance rules
• More relationships between artifacts and assets
• Fine-grained control of user permissions to view and manage governance artifacts with categories

Action required Before the move, make a note of the details of your business terms, custom classifications, and data protection rules.

Here's what happens during the move:

• All your existing business terms, policies, and data protection rules are permanently deleted. You can't revert to the legacy experience.
• All business term, data classes, and classification assignments on data assets become invalid.
• Any data masking that you configured with data protection rules is removed.
• Profiles of data assets are updated so that the classification results use the new data classes.

Here's what you need to do after the move:

• Re-create your business terms, classifications, and data protection rules.
• Remove invalid business terms and classification assignment from assets in catalogs.
• Assign your new business terms and classification to assets in catalogs.
• Assign Watson Knowledge Catalog roles to your users. See Assign Watson Knowledge Catalog roles to users.

If you have any questions or concerns related to moving to new version of governance artifacts, you can open a support ticket.

New scripting documentation

04 Feb 2022

Although scripting isn't required to use SPSS Modeler, it can be a powerful tool for automating processes in the user interface. Scripts can perform the same types of actions that you perform with a mouse or a keyboard, and you can use them to automate tasks that would be highly repetitive or time consuming to perform manually.

A new scripting and automation guide describes this functionality in detail.

Data Refinery supports SAS files with the "sas7bdat" extension

04 Feb 2022

You can now refine SAS data assets that use the .sas7bdat extension. SAS files are supported as source files only. You cannot use SAS files as a target of a Data Refinery flow.

For the full list of file types that are supported by Data Refinery, see Refining data.

Data Refinery flows with large data sets need updating when using certain GUI operations

04 Feb 2022

Action required For running Data Refinery jobs with large data assets, the following GUI operations have performance enhancements that require you to update any Data Refinery flows that use them:

• Convert column type to Integer when you specify a thousands grouping symbol (comma, dot, or custom)
• Convert column type to Decimal with a comma decimal marker or when you specify a thousands grouping symbol (comma, dot, or custom)
• Text > Trim quotes

To improve the job performance of a Data Refinery flow that uses these operations, update the Data Refinery flow by opening it and saving it, and then running a job for it. New Data Refinery flows automatically have the performance enhancements. For instructions, see Managing Data Refinery flows.

New connector for DataStage

04 Feb 2022

DataStage now supports the Box connector. For the full list of DataStage connectors, see DataStage connectors.

Decision Optimization models move to Python 3.8

04 Feb 2022

For Decision Optimization models, the default version for Python models is now Python 3.8. If you have Decision Optimization models on Python 3.7, re-create or re-deploy the model with Python 3.8 to avoid possible issues.

See Decision Optimization notebooks and Solve parameters.

Keeping catalog names unique

28 Jan 2022

When you create a catalog in the Create a catalog page, you must now use a unique name. Unique catalog names will avoid ambiguity problems and sync errors. If you need to use a duplicate name for a catalog, use the API to rename or create a catalog.

Data Scientist role has Access governance artifacts permission (Watson Knowledge Catalog)

28 Jan 2022

With the Access governance artifacts permission, data scientists can see the details of governance artifacts that are assigned to assets to better understand the data.

Deprecation of Spark 2.4 for Watson Studio and Watson Machine Learning

27 Jan 2022

Spark 2.4 is deprecated as a machine learning framework, notebook environment, and RStudio runtime. Update your assets to use Spark 3.0 instead. Support for training assets will be discontinued on February 16, 2022. Support for deploying and scoring models will be discontinued on March 10, 2022 and existing deployments using Spark 2.4 specifications will be removed. For details on migrating an asset to a supported framework and software specification, see Managing frameworks and software specifications. For details on notebook environments, see Compute resource options for the notebook editor in projects.

Support for Large size hardware specification for Decision Optimization (Watson Machine Learning)

27 Jan 2022

You can now use a Large size hardware specification (8 vCPU and 32 GB) with Decision Optimization jobs. Additionally, the number of jobs that can run in parallel is increased to 100. For details, see Running jobs.

New connectors for DataStage

21 Jan 2022

DataStage includes these new connectors:

• Amazon RDS for Oracle
• Compose for MySQL

For the full list of DataStage connectors, see DataStage connectors.

Metadata enrichment: automatic term assignment and more (Watson Knowledge Catalog)

20 Jan 2022

Automatic term assignment can now be part of your metadata enrichment and you can pick from more sampling options. Enrichment results at column level and a variety of additional insights at the asset level are also new. Plus, you can publish assets and results directly to any catalog you have access to. For details, see Enriching your data.

Watson Natural Language Processing for notebooks

20 Jan 2022

The Watson Natural Language Processing library (beta release) for notebooks provides basic natural language processing functions for syntax analysis and out-of-the-box pre-trained models with which you can turn unstructured data into structured data, enabling you to work with a mix of unstructured and structured data. Examples of data are call center records, customer complaints, social media posts, or problem reports. For details, see Watson Natural Language Processing library (beta).

More industry accelerators for end-to-end solutions (Watson Studio)

14 Jan 2022

One new industry accelerators is available as predefined assets you can use to address common business challenges:

Full support for testing AutoAI experiments for fairness (Watson Machine Learning)

12 Jan 2022

Evaluate an experiment for fairness to ensure your results are not biased in favor of one group over another. You can now evaluate experiments with joined data as well as experiments with a single data source. You cannot evaluate a time series experiment for fairness. For details on fairness testing, see Applying fairness testing to AutoAI experiments.

Securely connect to data sources with IBM Cloud Satellite

07 Jan 2022

With IBM Cloud Satellite, you use your own compute infrastructure that is in your on-premises data center or in another cloud provider to create a Satellite location. Then, you use the capabilities of Satellite to run IBM Cloud services on your infrastructure, and consistently deploy, manage, and control your app workloads.

For Cloud Pak for Data as a Service, you set up a Satellite location for the data source and then select Satellite Link in the Private connectivity section in the Create connection page.

All data sources that support Secure Gateway now support Satellite Link. For instructions, see Securing connections.