0 / 0
Known issues and limitations for Watson OpenScale

Known issues and limitations for Watson OpenScale

The following list contains the limitations and known issues for IBM Watson OpenScale.

Limitations

  • When you configure settings for SHAP global explanations, Watson OpenScale has the following limitations:

    • The sample size that you use to configure explanations can affect the number of explanations that Watson OpenScale can generate during specific time periods. If you attempt to generate multiple explanations for large sample sizes, Watson OpenScale might fail to process your transactions.
    • If you configure explanations for multiple Watson OpenScale subscriptions, you must specify the default values for the sample size and number of perturbations settings when your deployment contains 20 features or less.
  • Watson OpenScale doesn't support IAM access groups.

  • Watson OpenScale does not support models where the data type of the model prediction is binary. You must change such models so that the data type of their prediction is a string or integer data type.

  • Drift is supported for structured data only.

  • Although classification models support both data and accuracy drift, regression models support only data drift.

  • Drift is not supported for Python functions.

  • Support for the XGBoost framework has the following limitations for classification problems: For binary classification, Watson OpenScale supports the binary:logistic logistic regression function with an output as a probability of True. For multiclass classification, Watson OpenScale supports the multi:softprob function where the result contains the predicted probability of each data point belonging to each class.

  • Fairness and drift metrics are not supported for unstructured (image or text) data types.

  • Having an equals sign (=) in the column name of a dataset causes an issue with explainability and generates the following error message: Error: An error occurred while computing feature importance. Do not use an equals sign (=) in a column name. It is not supported.

  • The database and IBM Watson Machine Learning instance must be deployed in the same account.

  • Watson OpenScale uses a PostgreSQL or Db2 database to store model-related data (feedback data, scoring payload) and calculated metrics. Lite Db2 plans are not currently supported.

  • The free Lite plan database is not GDPR-compliant. If your model processes personally identifiable information (PII), you must purchase a new database or use an existing database that does conform to GDPR rules.

  • If you upload test data for your preproduction model evaluations that exceeds the default maximum 10485760 bytes data size for the payload-logging-service-api pod, your upload might cause an error. To avoid this error, you must set the value for the -Dservice.defaults.import.max_csv_line_length option in the ADDITIONAL_JVM_OPTIONS environment variable to a larger size that fits your data set.

  • For proper processing of payload analytics, Watson OpenScale does not support column names with double quotation marks (") in the payload. This affects both scoring payload and feedback data in CSV and JSON formats.

  • Explainability is not supported for SPSS multiclass models that return only the winning class probability.

  • For IBM Watson Machine Learning, scoring input for image classification models that are sent for payload logging cannot exceed 1 MB. To avoid time out issues, images must not exceed 100 x 100 x 3 pixels and must be sent sequentially so that the explanation for the second image is requested when the first one is completed.

  • The Amazon SageMaker BlazingText algorithm input payload format is not supported in the current release of Watson OpenScale.

  • Scoring payloads for a model must fit within the maximum width allowed for the table created by payload logging in the datamart database (with some buffer for the internal-use columns that IBM Watson OpenScale itself adds). In addition, apart from the width there is also a hardcoded limit of 1012 features. Because many models have features of mixed types, the following sample configurations can be used for planning purposes:

    • For int64 or float64 or strings of length 64 or less, count as 64.
    • For strings from 65 to 2048, count as 2048.
    • For strings from 2048 to 32 K, count as 32 K.
    • The total length of all features should be no more than ~900 K.

Known issues

Watson OpenScale has the following known issues:

Drift configuration is started but never finishes

Drift configuration is started but never finishes and continues to show the spinner icon. If you see the spinner run for more than 10 minutes, it is possible that the system is left in an inconsistent state. There is a workaround to this behavior: Edit the drift configuration. Then, save it. The system might come out of this state and complete configuration. If drift reconfiguration does not rectify the situation, contact IBM Support.

Next steps

High availability and disaster recovery

Parent topic: Evaluating AI models with Watson OpenScale

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more