Use watsonx, and mixtral_8x7b_instruct_v01
to generate code based on instruction¶
Disclaimers¶
- Use only Projects and Spaces that are available in watsonx context.
Notebook content¶
This notebook contains the steps and code to demonstrate support for code generating in watsonx. It introduces commands for defining prompt and model testing.
Some familiarity with Python is helpful. This notebook uses Python 3.11.
Learning goal¶
The goal of this notebook is to demonstrate how to generate code using IBM mixtral_8x7b_instruct_v01
watsonx model based on instruction provided by the user.
Contents¶
This notebook contains the following parts:
Set up the environment¶
Before you use the sample code in this notebook, you must perform the following setup tasks:
- Create a Watson Machine Learning (WML) Service instance (a free plan is offered and information about how to create the instance can be found here).
Install dependecies¶
!pip install -U ibm-watsonx-ai | tail -n 1
Defining the WML credentials¶
This cell defines the WML credentials required to work with watsonx Foundation Model inferencing.
Action: Provide the IBM Cloud user API key. For details, see documentation.
import getpass
from ibm_watsonx_ai import Credentials
credentials = Credentials(
url="https://us-south.ml.cloud.ibm.com",
api_key=getpass.getpass("Please enter your WML api key (hit enter): "),
)
Defining the project id¶
The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.
import os
try:
project_id = os.environ["PROJECT_ID"]
except KeyError:
project_id = input("Please enter your project_id (hit enter): ")
Foundation Models on watsonx.ai
¶
List available models¶
All avaliable models are presented under ModelTypes class. For more information refer to documentation.
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes
print([model.name for model in ModelTypes])
['FLAN_T5_XXL', 'FLAN_UL2', 'MT0_XXL', 'GPT_NEOX', 'MPT_7B_INSTRUCT2', 'STARCODER', 'LLAMA_2_70B_CHAT', 'LLAMA_2_13B_CHAT', 'GRANITE_13B_INSTRUCT', 'GRANITE_13B_CHAT', 'FLAN_T5_XL', 'GRANITE_13B_CHAT_V2', 'GRANITE_13B_INSTRUCT_V2', 'ELYZA_JAPANESE_LLAMA_2_7B_INSTRUCT', 'MIXTRAL_8X7B_INSTRUCT_V01_Q', 'CODELLAMA_34B_INSTRUCT_HF', 'GRANITE_20B_MULTILINGUAL', 'MERLINITE_7B', 'GRANITE_20B_CODE_INSTRUCT', 'GRANITE_34B_CODE_INSTRUCT', 'GRANITE_3B_CODE_INSTRUCT', 'GRANITE_7B_LAB', 'GRANITE_8B_CODE_INSTRUCT', 'LLAMA_3_70B_INSTRUCT', 'LLAMA_3_8B_INSTRUCT', 'MIXTRAL_8X7B_INSTRUCT_V01']
You need to specify model_id
that will be used for inferencing:
model_id = ModelTypes.MIXTRAL_8X7B_INSTRUCT_V01
Defining the model parameters¶
You might need to adjust model parameters
for different models or tasks, to do so please refer to documentation.
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
parameters = {
GenParams.DECODING_METHOD: "greedy",
GenParams.MAX_NEW_TOKENS: 200,
GenParams.STOP_SEQUENCES: ["<end·of·code>"]
}
Initialize the model¶
Initialize the ModelInference
class with previous set params.
from ibm_watsonx_ai.foundation_models import ModelInference
model = ModelInference(
model_id=model_id,
params=parameters,
credentials=credentials,
project_id=project_id)
Model's details¶
model.get_details()
{'model_id': 'mistralai/mixtral-8x7b-instruct-v01', 'label': 'mixtral-8x7b-instruct-v01', 'provider': 'Mistral AI', 'source': 'Hugging Face', 'functions': [{'id': 'text_generation'}], 'short_description': 'The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.', 'long_description': "This model is made with AutoGPTQ, which mainly leverages the quantization technique to 'compress' the model weights from FP16 to 4-bit INT and performs 'decompression' on-the-fly before computation (in FP16). As a result, the GPU memory, and the data transferring between GPU memory and GPU compute engine, compared to the original FP16 model, is greatly reduced. The major quantization parameters used in the process are listed below.", 'input_tier': 'class_1', 'output_tier': 'class_1', 'number_params': '46.7b', 'min_shot_size': 1, 'task_ids': ['summarization', 'retrieval_augmented_generation', 'classification', 'generation', 'code', 'extraction'], 'tasks': [{'id': 'summarization', 'ratings': {'quality': 4}}, {'id': 'retrieval_augmented_generation', 'ratings': {'quality': 3}}, {'id': 'classification', 'ratings': {'quality': 4}}, {'id': 'generation'}, {'id': 'code'}, {'id': 'extraction', 'ratings': {'quality': 4}}], 'model_limits': {'max_sequence_length': 32768}, 'limits': {'lite': {'call_time': '5m0s', 'max_output_tokens': 16384}, 'v2-professional': {'call_time': '10m0s', 'max_output_tokens': 16384}, 'v2-standard': {'call_time': '10m0s', 'max_output_tokens': 16384}}, 'lifecycle': [{'id': 'available', 'start_date': '2024-04-17'}]}
Generate code based on instruction¶
Define instructions for the model with at-least one example.
instruction = """Using the directions below, generate Python code for the given task.
Input:
# Write a Python function that prints 'Hello World!' string 'n' times.
Output:
def print_n_times(n):
for i in range(n):
print("Hello World!")
<end of code>
"""
Prepare question for the model.
question = """Input:
# Write a Python function, which generates sequence of prime numbers.
# The function 'primes' will take the argument 'n', an int. It will return a list which contains all primes less than 'n'."""
Generat the code using IBM mixtral_8x7b_instruct_v01
model.¶
Inter the model to generate the code, according to provided instruction.
result = model.generate_text(" ".join([instruction, question]))
Formatting the text to get the function itself
code_as_text = result.split('Output:')[1].split('<end of code>')[0]
Generated code testing¶
The resulting code looks as below.
print(code_as_text)
def primes(n): primes = [] for possiblePrime in range(2, n): isPrime = True for num in range(2, int(possiblePrime ** 0.5) + 1): if possiblePrime % num == 0: isPrime = False break if isPrime: primes.append(possiblePrime) return primes
Use generated code to make it as function.
exec(code_as_text)
Define the number 'n' for which the primes() function should process prime numbers.
n = 25
Test and run the generated function.
primes(n)
[2, 3, 5, 7, 11, 13, 17, 19, 23]
Summary and next steps¶
You successfully completed this notebook!.
You learned how to generate code based on instuction with IBM mixtral_8x7b_instruct_v01
on watsonx.
Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.
Authors:¶
Mateusz Szewczyk, Software Engineer at Watson Machine Learning.
Copyright © 2024 IBM. This notebook and its source code are released under the terms of the MIT License.