ibm-watson-machine-learning
¶This notebook facilitates Decision Optimization and Watson Machine Learning services. It contains steps and code to work with ibm-watson-machine-learning library available in PyPI repository. It also introduces commands for getting model and training data, persisting model, deploying model and scoring it.
Some familiarity with Python is helpful.
The learning goals of this notebook are:
This notebook contains the following parts:
Before you use the sample code in this notebook, you must:
Install and then import the Watson Machine Learning client library.
Note: ibm-watson-machine-learning
documentation can be found here.
# Install WML client API
!pip install ibm-watson-machine-learning
Requirement already satisfied: ibm-watson-machine-learning in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (1.0.338) Requirement already satisfied: requests in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (2.31.0) Requirement already satisfied: urllib3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (1.26.18) Requirement already satisfied: pandas<1.6.0,>=0.24.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (1.5.3) Requirement already satisfied: certifi in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (2023.11.17) Requirement already satisfied: lomond in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (0.3.3) Requirement already satisfied: tabulate in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (0.8.10) Requirement already satisfied: packaging in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (23.0) Requirement already satisfied: importlib-metadata in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (6.0.0) Requirement already satisfied: ibm-cos-sdk<2.14.0,>=2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning) (2.12.0) Requirement already satisfied: ibm-cos-sdk-core==2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning) (2.12.0) Requirement already satisfied: ibm-cos-sdk-s3transfer==2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning) (2.12.0) Requirement already satisfied: jmespath<1.0.0,>=0.10.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning) (0.10.0) Requirement already satisfied: python-dateutil<3.0.0,>=2.8.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk-core==2.12.0->ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas<1.6.0,>=0.24.2->ibm-watson-machine-learning) (2022.7) Requirement already satisfied: numpy>=1.21.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas<1.6.0,>=0.24.2->ibm-watson-machine-learning) (1.23.5) Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->ibm-watson-machine-learning) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->ibm-watson-machine-learning) (3.4) Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from importlib-metadata->ibm-watson-machine-learning) (3.11.0) Requirement already satisfied: six>=1.10.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from lomond->ibm-watson-machine-learning) (1.16.0)
!pip install -U wget
Collecting wget Downloading wget-3.2.zip (10 kB) Preparing metadata (setup.py) ... done Building wheels for collected packages: wget Building wheel for wget (setup.py) ... done Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9656 sha256=bf798e28395c3c4d6f31ffa38a5ef575cf2925f468cb6691c594e497f41a3d95 Stored in directory: /tmp/wsuser/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769 Successfully built wget Installing collected packages: wget Successfully installed wget-3.2
from ibm_watson_machine_learning import APIClient
# Instantiate a client using credentials
wml_credentials = {
"apikey": "<API_key>",
"url": "<instance_url>"
}
client = APIClient(wml_credentials)
First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use Deployment Spaces Dashboard to create one.
Tip: You can also use SDK to prepare the space for your work. More information can be found here.
# Find the space ID
space_name = "<space_name>"
space_id = [x['metadata']['id'] for x in client.spaces.get_details()['resources'] if x['entity']['name'] == space_name][0]
client.set.default_space(space_id)
'SUCCESS'
In this section, you will download externally created Decision Optimization model and data used for training it.
Action: Get your Decision Optimization model.
import os
import wget
model_path = 'do-model.tar.gz'
if not os.path.isfile(model_path):
wget.download("https://github.com/IBM/watson-machine-learning-samples/raw/master/cloud/models/decision_optimization/do-model.tar.gz")
In this section, you will learn how to store your model in Watson Machine Learning repository by using the Watson Machine Learning Client.
Define model name, autor name and email.
Get software specification for Decision Optimization model
sofware_spec_uid = client.software_specifications.get_uid_by_name("do_22.1")
Output data schema for storing model in WML repository
output_data_schema = [{'id': 'stest',
'type': 'list',
'fields': [{'name': 'age', 'type': 'float'},
{'name': 'sex', 'type': 'float'},
{'name': 'cp', 'type': 'float'},
{'name': 'restbp', 'type': 'float'},
{'name': 'chol', 'type': 'float'},
{'name': 'fbs', 'type': 'float'},
{'name': 'restecg', 'type': 'float'},
{'name': 'thalach', 'type': 'float'},
{'name': 'exang', 'type': 'float'},
{'name': 'oldpeak', 'type': 'float'},
{'name': 'slope', 'type': 'float'},
{'name': 'ca', 'type': 'float'},
{'name': 'thal', 'type': 'float'}]
},
{'id': 'teste2',
'type': 'test',
'fields': [{'name': 'age', 'type': 'float'},
{'name': 'sex', 'type': 'float'},
{'name': 'cp', 'type': 'float'},
{'name': 'restbp', 'type': 'float'},
{'name': 'chol', 'type': 'float'},
{'name': 'fbs', 'type': 'float'},
{'name': 'restecg', 'type': 'float'},
{'name': 'thalach', 'type': 'float'},
{'name': 'exang', 'type': 'float'},
{'name': 'oldpeak', 'type': 'float'},
{'name': 'slope', 'type': 'float'},
{'name': 'ca', 'type': 'float'},
{'name': 'thal', 'type': 'float'}]
}]
model_meta_props = {
client.repository.ModelMetaNames.NAME: "LOCALLY created DO model",
client.repository.ModelMetaNames.TYPE: "do-docplex_22.1",
client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: sofware_spec_uid,
client.repository.ModelMetaNames.OUTPUT_DATA_SCHEMA: output_data_schema
}
published_model = client.repository.store_model(model=model_path, meta_props=model_meta_props)
Note: You can see that model is successfully stored in Watson Machine Learning Service.
import json
published_model_uid = client.repository.get_model_id(published_model)
model_details = client.repository.get_details(published_model_uid)
print(json.dumps(model_details, indent=2))
{ "entity": { "hybrid_pipeline_software_specs": [], "schemas": { "input": [], "output": [ { "fields": [ { "name": "age", "type": "float" }, { "name": "sex", "type": "float" }, { "name": "cp", "type": "float" }, { "name": "restbp", "type": "float" }, { "name": "chol", "type": "float" }, { "name": "fbs", "type": "float" }, { "name": "restecg", "type": "float" }, { "name": "thalach", "type": "float" }, { "name": "exang", "type": "float" }, { "name": "oldpeak", "type": "float" }, { "name": "slope", "type": "float" }, { "name": "ca", "type": "float" }, { "name": "thal", "type": "float" } ], "id": "stest", "type": "list" }, { "fields": [ { "name": "age", "type": "float" }, { "name": "sex", "type": "float" }, { "name": "cp", "type": "float" }, { "name": "restbp", "type": "float" }, { "name": "chol", "type": "float" }, { "name": "fbs", "type": "float" }, { "name": "restecg", "type": "float" }, { "name": "thalach", "type": "float" }, { "name": "exang", "type": "float" }, { "name": "oldpeak", "type": "float" }, { "name": "slope", "type": "float" }, { "name": "ca", "type": "float" }, { "name": "thal", "type": "float" } ], "id": "teste2", "type": "test" } ] }, "software_spec": { "id": "e51999ba-6452-5f1f-8287-17228b88b652", "name": "do_22.1" }, "type": "do-docplex_22.1" }, "metadata": { "created_at": "2024-01-12T08:51:10.217Z", "id": "8c0932d0-4afa-44d7-9350-8e0e6fa1ec87", "modified_at": "2024-01-12T08:51:11.889Z", "name": "LOCALLY created DO model", "owner": "IBMid-270006YQEG", "resource_key": "98fdf7c6-a6b0-4a5a-b98e-c7765e36d3d5", "space_id": "0d715374-9c5f-4faf-bad7-b8356d478bda" }, "system": { "warnings": [] } }
client.repository.list_models()
------------------------------------ ------------------------ ------------------------ --------------- ---------- ---------------- ID NAME CREATED TYPE SPEC_STATE SPEC_REPLACEMENT 8c0932d0-4afa-44d7-9350-8e0e6fa1ec87 LOCALLY created DO model 2024-01-12T08:51:10.002Z do-docplex_22.1 supported ------------------------------------ ------------------------ ------------------------ --------------- ---------- ----------------
ID | NAME | CREATED | TYPE | SPEC_STATE | SPEC_REPLACEMENT | |
---|---|---|---|---|---|---|
0 | 8c0932d0-4afa-44d7-9350-8e0e6fa1ec87 | LOCALLY created DO model | 2024-01-12T08:51:10.002Z | do-docplex_22.1 | supported |
In this section you will learn how to create batch deployment to create job using the Watson Machine Learning Client.
You can use commands bellow to create batch deployment for stored model (web service).
meta_data = {
client.deployments.ConfigurationMetaNames.NAME: "deployment_DO",
client.deployments.ConfigurationMetaNames.BATCH: {},
client.deployments.ConfigurationMetaNames.HARDWARE_SPEC: {"name": "S", "num_nodes": 1}
}
deployment_details = client.deployments.create(published_model_uid, meta_props=meta_data)
####################################################################################### Synchronous deployment creation for uid: '8c0932d0-4afa-44d7-9350-8e0e6fa1ec87' started ####################################################################################### ready. ------------------------------------------------------------------------------------------------ Successfully finished deployment creation, deployment_uid='e9e839fd-20c5-4c73-a66b-1ab28365a580' ------------------------------------------------------------------------------------------------
Note: Here we use deployment url saved in published_model object. In next section, we show how to retrive deployment url from Watson Machine Learning instance.
deployment_uid = client.deployments.get_uid(deployment_details)
Now, you can list all deployments.
client.deployments.list()
------------------------------------ ------------- ----- ------------------------ ------------- ---------- ---------------- GUID NAME STATE CREATED ARTIFACT_TYPE SPEC_STATE SPEC_REPLACEMENT e9e839fd-20c5-4c73-a66b-1ab28365a580 deployment_DO ready 2024-01-12T08:51:13.609Z do supported ------------------------------------ ------------- ----- ------------------------ ------------- ---------- ----------------
GUID | NAME | STATE | CREATED | ARTIFACT_TYPE | SPEC_STATE | SPEC_REPLACEMENT | |
---|---|---|---|---|---|---|---|
0 | e9e839fd-20c5-4c73-a66b-1ab28365a580 | deployment_DO | ready | 2024-01-12T08:51:13.609Z | do | supported |
client.deployments.get_details(deployment_uid)
{'entity': {'asset': {'id': '8c0932d0-4afa-44d7-9350-8e0e6fa1ec87'}, 'batch': {}, 'custom': {}, 'deployed_asset_type': 'do', 'hardware_spec': {'id': 'e7ed1d6c-2e89-42d7-aed5-863b972c1d2b', 'name': 'S', 'num_nodes': 1}, 'name': 'deployment_DO', 'space_id': '0d715374-9c5f-4faf-bad7-b8356d478bda', 'status': {'state': 'ready'}}, 'metadata': {'created_at': '2024-01-12T08:51:13.609Z', 'id': 'e9e839fd-20c5-4c73-a66b-1ab28365a580', 'modified_at': '2024-01-12T08:51:13.609Z', 'name': 'deployment_DO', 'owner': 'IBMid-270006YQEG', 'space_id': '0d715374-9c5f-4faf-bad7-b8356d478bda'}}
Prepare test data
# Import pandas library
import pandas as pd
# Initialize list of lists
diet_food = pd.DataFrame([ ["Roasted Chicken", 0.84, 0, 10],
["Spaghetti W/ Sauce", 0.78, 0, 10],
["Tomato,Red,Ripe,Raw", 0.27, 0, 10],
["Apple,Raw,W/Skin", 0.24, 0, 10],
["Grapes", 0.32, 0, 10],
["Chocolate Chip Cookies", 0.03, 0, 10],
["Lowfat Milk", 0.23, 0, 10],
["Raisin Brn", 0.34, 0, 10],
["Hotdog", 0.31, 0, 10]] , columns = ["name","unit_cost","qmin","qmax"])
diet_food_nutrients = pd.DataFrame([
["Spaghetti W/ Sauce", 358.2, 80.2, 2.3, 3055.2, 11.6, 58.3, 8.2],
["Roasted Chicken", 277.4, 21.9, 1.8, 77.4, 0, 0, 42.2],
["Tomato,Red,Ripe,Raw", 25.8, 6.2, 0.6, 766.3, 1.4, 5.7, 1],
["Apple,Raw,W/Skin", 81.4, 9.7, 0.2, 73.1, 3.7, 21, 0.3],
["Grapes", 15.1, 3.4, 0.1, 24, 0.2, 4.1, 0.2],
["Chocolate Chip Cookies", 78.1, 6.2, 0.4, 101.8, 0, 9.3, 0.9],
["Lowfat Milk", 121.2, 296.7, 0.1, 500.2, 0, 11.7, 8.1],
["Raisin Brn", 115.1, 12.9, 16.8, 1250.2, 4, 27.9, 4],
["Hotdog", 242.1, 23.5, 2.3, 0, 0, 18, 10.4 ]
] , columns = ["Food","Calories","Calcium","Iron","Vit_A","Dietary_Fiber","Carbohydrates","Protein"])
diet_nutrients = pd.DataFrame([
["Calories", 2000, 2500],
["Calcium", 800, 1600],
["Iron", 10, 30],
["Vit_A", 5000, 50000],
["Dietary_Fiber", 25, 100],
["Carbohydrates", 0, 300],
["Protein", 50, 100]
], columns = ["name","qmin","qmax"])
job_payload_ref = {
client.deployments.DecisionOptimizationMetaNames.INPUT_DATA: [
{
"id": "diet_food.csv",
"values": diet_food
},
{
"id": "diet_food_nutrients.csv",
"values": diet_food_nutrients
},
{
"id": "diet_nutrients.csv",
"values": diet_nutrients
}
],
client.deployments.DecisionOptimizationMetaNames.OUTPUT_DATA: [
{
"id": ".*.csv"
}
]
}
Create job using Watson Machine Learning client
job = client.deployments.create_job(deployment_uid, meta_props=job_payload_ref)
Checking created job status and calculated KPI.
import time
job_id = client.deployments.get_job_uid(job)
elapsed_time = 0
while client.deployments.get_job_status(job_id).get('state') != 'completed' and elapsed_time < 300:
elapsed_time += 10
time.sleep(10)
if client.deployments.get_job_status(job_id).get('state') == 'completed':
job_details_do = client.deployments.get_job_details(job_id)
kpi = job_details_do['entity']['decision_optimization']['solve_state']['details']['KPI.Total Calories']
print(f"KPI: {kpi}")
else:
print("Job hasn't completed successfully in 5 minutes.")
KPI: 2000.0
If you want to clean up all created assets:
follow up this sample notebook.
You've successfully completed this notebook!
You've learned how to:
Check out our online documentation for more samples, tutorials and documentation:
Wojciech Jargielo, Software Engineer