Entity extraction

Machine-learning-based extraction for general entities

The machine-learning-based extraction models are trained on labeled data for the more complex entity types such as person, organization and location.

Capabilities

The entity models extract entities from the input text. The following types of entities are recognized:

• Date
• Duration
• Facility
• Geographic feature
• Job title
• Location
• Measure
• Money
• Ordinal
• Organization
• Person
• Time

Available workflows and blocks differ, depending on the runtime used.

Workflow names

• entity-mentions_transformer-workflow_multilingual_slate.153m.distilled: this workflow can be used on both CPUs and GPUs.
• entity-mentions_transformer-workflow_multilingual_slate.153m.distilled-cpu: this workflow is optimized for CPU-based runtimes.

Supported languages

Entity extraction is available for the following languages.

ar, cs, da, de, en, es, fi, fr, he, hi, it, ja, ko, nb, nl, nn, pt, ro, ru, sk, sv, tr, zh-cn

For a list of language codes and corresponding languages, see Language codes

Code sample

import watson_nlp
# Load the workflow model
entities_workflow = watson_nlp.load('entity-mentions_transformer-workflow_multilingual_slate.153m.distilled')
# Run the entity extraction workflow on the input text
entities = entities_workflow.run('IBM\'s CEO Arvind Krishna is based in the US', language_code="en")
print(entities.get_mention_pairs())

Output of the code sample:

[('IBM', 'Organization'), ('CEO', 'JobTitle'), ('Arvind Krishna', 'Person'), ('US', 'Location')]

Rule-based extraction for general entities

The rule-based model entity-mentions_rbr_xx_stock identifies syntactically regular entities.

Block name entity-mentions_rbr_xx_stock

Capabilities

Rule-based extraction handles syntactically regular entity types. The entity block extract entities from the input text. The following types of entities are recognized:

• PhoneNumber
• EmailAddress
• Number
• Percent
• IPAddress
• HashTag
• TwitterHandle
• URLDate

Supported languages

Entity extraction is available for the following languages. For a list of the language codes and the corresponding language, see Language codes.

ar, cs, da, de, en, es, fi, fr, he, hi, it, ja, ko, nb, nl, nn, pt, ro, ru, sk, sv, tr, zh-cn, zh-tw

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load a rule-based Entity Mention model for English
rbr_entity_model = watson_nlp.load('entity-mentions_rbr_en_stock')

# Run the entity model on the input text
rbr_entity_mentions = rbr_entity_model.run('My email is [email protected]')
print(rbr_entity_mentions)

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 12,
        "end": 27,
        "text": "[email protected]"
      },
      "type": "EmailAddress",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}

Rule-based extraction for PII entities

The rule-based model entity-mentions_rbr_multi_pii handles the majority of the types by identifying common formats of PII entities and performing possible checksum or validations as appropriate for each entity type. For example, credit card number candidates are validated using the Luhn algorithm.

Block name entity-mentions_rbr_multi_pii

Capabilities

The entity block entity-mentions_rbr_multi_pii recognizes the following types of entities:

Some PII entity type names are country-specific. The _ in the following entity types is a placeholder for a country code.

• BankAccountNumber.BBAN._ : These are more variable national bank account numbers and the extraction is mostly language-specific without a general checksum algorithm.
• BankAccountNumber.IBAN._ : Highly standardized IBANs are supported in a language-independent way and with a checksum algorithm.
• NationalNumber.NationalID._: These national IDs don’t have a (published) checksum algorithm, and are being extracted on a language-specific basis.
• NationalNumber.Passport._ : Checksums are implemented only for the countries where a checksum algorithm exists. These are specifically extracted language with additional context restrictions.
• NationalNumber.TaxID._ : These IDs don't have a (published) checksum algorithm, and are being extracted on a language-specific basis.

Which entity types are available for which languages and which country code to use is listed in the following table.

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load the RBR PII model. Note that this is a multilingual model supporting multiple languages.
rbr_entity_model = watson_nlp.load('entity-mentions_rbr_multi_pii')

# Run the RBR model. Note that language code of the input text is passed as a parameter to the run method.
rbr_entity_mentions = rbr_entity_model.run('Please find my credit card number here: 378282246310005. Thanks for the payment.', language_code='en')
print(rbr_entity_mentions)

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 40,
        "end": 55,
        "text": "378282246310005"
      },
      "type": "BankAccountNumber.CreditCardNumber.Amex",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}

Parent topic: Watson Natural Language Processing task catalog