Watson Natural Language Processing library

Using Watson Natural Language Processing in a notebook

You can run your Python notebooks using the Watson Natural Language Processing library in any of the following provided environments. The GPU environment templates include the Watson Natural Language Processing library.

DO + NLP: Indicates that the environment templates includes both the CPLEX and the DOcplex libraries to model and solve decision optimization problems and the Watson Natural Language Processing library.

~ : Indicates that the environment template requires the Watson Studio Professional plan. See Offering plans.

* : Indicates that the environment is deprecated.

The DO + NLP Runtime 22.2 on Python 3.10 or DO + NLP Runtime 22.1 on Python 3.9 environment should be large enough to run notebooks that use the prebuild models. If you need a larger environment, for example to train your own models, you can use a GPU V100 environment.

If you don't use a provided environment template, you can create a custom template that includes the Watson Natural Language Processing library. See Creating your own environment template.

• Create a custom template without GPU by selecting the engine type Default, the hardware configuration size that you need, and choosing DO + NLP Runtime 22.2 on Python 3.10 or DO + NLP Runtime 22.1 on Python 3.9 as the software version.
• Create a custom template with GPU by selecting the engine type GPU, the hardware configuration size that you need, and choosing GPU Runtime 22.1 on Python 3.9 as the software version.

Working with the pre-trained models

Watson Natural Language Processing encapsulates natural language functionality via blocks where each block supports functions to:

• load(): load a block model.
• run(): run the block on input argument(s).
• train(): train the block on your own data. Not all blocks support training.
• save(): save the block model you trained on your own data.

There are two types of blocks:

• Blocks that operate directly on the input document
• Blocks that depend on other blocks

import watson_nlp

# Load the syntax model for English
syntax_model = watson_nlp.load(watson_nlp.download('syntax_izumo_en_stock'))

# Run the syntax model and print the result
syntax_prediction = syntax_model.run('Welcome to IBM!')
print(syntax_prediction)

import watson_nlp

# Load Syntax and a Entity Mention model for English
syntax_model = watson_nlp.load(watson_nlp.download('syntax_izumo_en_stock'))
entity_model = watson_nlp.load(watson_nlp.download('entity-mentions_bert_multi_stock'))

# Run the syntax model on the input text
syntax_prediction = syntax_model.run('IBM announced new advances in quantum computing')

# Now run the entity mention model on the result of syntax
entity_mentions = entity_model.run(syntax_prediction)
print(entity_mentions)

Loading and running a model

Watson Natural Language Processing contains the load() and download() functions to allow you to quickly load pre-trained models to your notebook. To download a model, you first need to know its name. Model names follow a standard convention encoding the type of model (like classification or entity extraction), type of algorithm (like BERT or SVM), language code and details of the type system.

To find the right block to use, use the block catalog. See Watson NLP block catalog.

You can find the expected input for a given block class (for example to the Entity Mentions model) by using help() on the block class run() method:

import watson_nlp

help(watson_nlp.blocks.entity_mentions.BERT.run)

Sample project and notebooks

To help you get started with the Watson Natural Language Processing library, you can download a sample project and notebooks from the Gallery. The notebooks demonstrate how to use the different Watson Natural Language Processing blocks and how to train your own models.

You can access the Gallery by sellecting Gallery from the Cloud Pak for Data navigation menu.

Sample notebooks

• Financial complaint analysis

  This notebook shows you how to analyze financial customer complaints using Watson Natural Language Processing. It uses data from the Consumer Complaint Database published by the Consumer Financial Protection Bureau (CFPB). The notebook teaches you to use the Tone classification and Emotion classification models.

• Car complaint analysis

  This notebook demonstrates how to analyze car complaints using Watson Natural Language Processing. It uses publicly available complaint records from car owners stored by the National Highway and Transit Association (NHTSA) of the US Department of Transportation. This notebook shows you how use syntax analysis to extract the most frequently used nouns, which typically depict the problems that review authors talk about and combine these results with structured data using association rule mining.

• Complaint classification with Watson Natural Language Processing

  This notebook demonstrates how to train different text classifiers using Watson Natural Language Processing. The classifiers predict the product group from the text of a customer complaint. This could be used, for example to route a complaint to the appropriate staff member. The data that is used in this notebook is taken from the Consumer Complaint Database that is published by the Consumer Financial Protection Bureau (CFPB), a U.S. government agency and is publicly available. You will learn how to train a custom CNN model and a VotingEnsemble model and evaluate their quality.

• Entity extraction on Financial Complaints with Watson Natural Language Processing

  This notebook demonstrates how to extract named entities from financial customer complaints using Watson Natural Language Processing. It uses data from the Consumer Complaint Database published by the Consumer Financial Protection Bureau (CFPB). In the notebook you will learn how to do dictionary-based term extraction to train a custom extraction model based on given dictionaries and extract entities using the BERT model.

Sample project

If you don't want to download the sample notebooks to your project individually, you can download the entire sample project Text Analysis with Watson Natural Language Processing from the Gallery.

The sample project contains the sample notebooks listed in the previous section, including:

• Analyzing hotel reviews using Watson Natural Language Processing

  This notebook shows you how to use syntax analysis to extract the most frequently used nouns from the hotel reviews, classify the sentiment of the reviews and use aspect-oriented sentiment analysis for the most frequently extracted aspects. The data file that is used by this notebook is included in the project as a data asset.

You can run all of the sample notebooks with the DO + NLP Runtime 22.1 on Python 3.9 environment except for the Complaint classification with Watson Natural Language Processingg notebook. To run this notebook, you need to create an environment template that is large enough to complete the training of the classification models on the training data.

Learn more

• Creating your own environment template

Parent topic: Notebooks and scripts