Machine-learning-based extraction for general entities

The machine-learning-based extraction models are trained on labeled data for the more complex entity types such as person, organization and location.

Capabilities

The entity models extract entities from the input text. The following types of entities are recognized:

• Date
• Duration
• Facility
• Geographic feature
• Job title
• Location
• Measure
• Money
• Ordinal
• Organization
• Person
• Time

Available workflows and blocks differ, depending on the runtime used.

import watson_nlp
# Load the workflow model
entities_workflow = watson_nlp.load('entity-mentions_transformer-workflow_multilingual_slate.153m.distilled')
# Run the entity extraction workflow on the input text
entities = entities_workflow.run('IBM\'s CEO Arvind Krishna is based in the US', language_code="en")
print(entities.get_mention_pairs())

[('IBM', 'Organization'), ('CEO', 'JobTitle'), ('Arvind Krishna', 'Person'), ('US', 'Location')]

import watson_nlp

# Load Syntax Model for English, and the multilingual BERT Entity model 
syntax_model = watson_nlp.load('syntax_izumo_en_stock')
bert_entity_model = watson_nlp.load('entity-mentions_bert_multi_stock')

# Run the syntax model on the input text
syntax_prediction = syntax_model.run('IBM\'s CEO Arvind Krishna is based in the US')

# Run the entity mention model on the result of syntax model
bert_entity_mentions = bert_entity_model.run(syntax_prediction)
print(bert_entity_mentions.get_mention_pairs())

[('IBM', 'Organization'), ('CEO', 'JobTitle'), ('Arvind Krishna', 'Person'), ('US', 'Location')]

Machine-learning-based extraction for PII entities

Block names entity-mentions_bilstm_en_pii

The entity-mentions_bilstm_en_pii machine-learning based extraction model is trained on labeled data for types person and location.

Capabilities

The entity-mentions_bilstm_en_pii block recognizes the following types of entities:

Dependencies on other blocks

The following block must run before you can run the entity-mentions_bilstm_en_pii block:

• syntax_izumo_en_stock

Code sample

import os

import watson_nlp

# Load Syntax and a Entity Mention BiLSTM model for English

syntax_model = watson_nlp.load('syntax_izumo_en_stock')

entity_model = watson_nlp.load('entity-mentions_bilstm_en_pii')

text = 'Denver is the capital of Colorado. The total estimated government spending in Colorado in fiscal year 2016 was $36.0 billion. IBM office is located in downtown Denver. Michael Hancock is the mayor of Denver.'

# Run the syntax model on the input text

syntax_prediction = syntax_model.run(text)

# Run the entity mention model on the result of the syntax analysis

entity_mentions = entity_model.run(syntax_prediction)

print(entity_mentions)

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 0,
        "end": 6,
        "text": "Denver"
      },
      "type": "Location",
      "producer_id": {
        "name": "BiLSTM Entity Mentions",
        "version": "1.0.0"
      },
      "confidence": 0.6885626912117004,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 25,
        "end": 33,
        "text": "Colorado"
      },
      "type": "Location",
      "producer_id": {
        "name": "BiLSTM Entity Mentions",
        "version": "1.0.0"
      },
      "confidence": 0.8509215116500854,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 78,
        "end": 86,
        "text": "Colorado"
      },
      "type": "Location",
      "producer_id": {
        "name": "BiLSTM Entity Mentions",
        "version": "1.0.0"
      },
      "confidence": 0.9928259253501892,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 151,
        "end": 166,
        "text": "downtown Denver"
      },
      "type": "Location",
      "producer_id": {
        "name": "BiLSTM Entity Mentions",
        "version": "1.0.0"
      },
      "confidence": 0.48378944396972656,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 168,
        "end": 183,
        "text": "Michael Hancock"
      },
      "type": "Person",
      "producer_id": {
        "name": "BiLSTM Entity Mentions",
        "version": "1.0.0"
      },
      "confidence": 0.9972871541976929,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "BiLSTM Entity Mentions",
    "version": "1.0.0"
  }
}

Rule-based extraction for general entities

The rule-based model entity-mentions_rbr_xx_stock identifies syntactically regular entities.

Block name entity-mentions_rbr_xx_stock

Capabilities

Rule-based extraction handles syntactically regular entity types. The entity block extract entities from the input text. The following types of entities are recognized:

• PhoneNumber
• EmailAddress
• Number
• Percent
• IPAddress
• HashTag
• TwitterHandle
• URLDate

Supported languages

Entity extraction is available for the following languages. For a list of the language codes and the corresponding language, see Language codes.

ar, cs, da, de, en, es, fi, fr, he, hi, it, ja, ko, nb, nl, nn, pt, ro, ru, sk, sv, tr, zh-cn, zh-tw

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load a rule-based Entity Mention model for English
rbr_entity_model = watson_nlp.load('entity-mentions_rbr_en_stock')

# Run the entity model on the input text
rbr_entity_mentions = rbr_entity_model.run('My email is [email protected]')
print(rbr_entity_mentions)

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 12,
        "end": 27,
        "text": "[email protected]"
      },
      "type": "EmailAddress",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}

Rule-based extraction for PII entities

The rule-based model entity-mentions_rbr_multi_pii handles the majority of the types by identifying common formats of PII entities and performing possible checksum or validations as appropriate for each entity type. For example, credit card number candidates are validated using the Luhn algorithm.

Block name entity-mentions_rbr_multi_pii

Capabilities

The entity block entity-mentions_rbr_multi_pii recognizes the following types of entities:

Some PII entity type names are country-specific. The _ in the following entity types is a placeholder for a country code.

• BankAccountNumber.BBAN._ : These are more variable national bank account numbers and the extraction is mostly language-specific without a general checksum algorithm.
• BankAccountNumber.IBAN._ : Highly standardized IBANs are supported in a language-independent way and with a checksum algorithm.
• NationalNumber.NationalID._: These national IDs don’t have a (published) checksum algorithm, and are being extracted on a language-specific basis.
• NationalNumber.Passport._ : Checksums are implemented only for the countries where a checksum algorithm exists. These are specifically extracted language with additional context restrictions.
• NationalNumber.TaxID._ : These IDs don't have a (published) checksum algorithm, and are being extracted on a language-specific basis.

Which entity types are available for which languages and which country code to use is listed in the following table.

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load the RBR PII model. Note that this is a multilingual model supporting multiple languages.
rbr_entity_model = watson_nlp.load('entity-mentions_rbr_multi_pii')

# Run the RBR model. Note that language code of the input text is passed as a parameter to the run method.
rbr_entity_mentions = rbr_entity_model.run('Please find my credit card number here: 378282246310005. Thanks for the payment.', language_code='en')
print(rbr_entity_mentions)

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 40,
        "end": 55,
        "text": "378282246310005"
      },
      "type": "BankAccountNumber.CreditCardNumber.Amex",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}

Parent topic: Watson Natural Language Processing task catalog