Running a streams flow and monitoring its metrics

After you run your streams flow, the Metrics page will provide information about the general health of your streams flow, real-time metrics, runtime views, and embedded logging and error notification.

 

Let’s take a deeper look at what’s in the Metric page. The following screen capture is an example.

Metrics UI

  • Streams Flow pane shows a dynamic, bird’s-eye view of the data as it flows among operators in your streams flow. You can see the throughput of the flow. In the previous example, the mouse pointer is hovering over the flow between the Sample Data and the Filter operators. While the pointer is hovering over a flow, you can see the current throughput and the total events of that flow. You can drill down even further to view the events that flow in the Flow of Events table.

  • Flow of Events table shows the events that flow between operators in both table and JSON formats. This table opens when you click the data flow between operators in the Streams Flow pane. This table is helpful if you need to understand the data better or to tweak an operator. In the previous screen capture, two tables are shown because both the flow between the Sample Data and the Filter operators and the flow between the Filter and the Cloud Object Storage operators were clicked.

  • Ingest Rate graph shows the number of events that are submitted to the streams flow per second. For example, a Sample Data operator might submit one million events per second to the streams flow. If several Sample Data operators submitted data, then the ingest rate is the sum of the events of those operators.

  • Throughput graph shows the throughput of a selected operator’s data flow (input, output, or both), the number of dropped events from the network, and the number of events that are not valid for any reason. In the previous example, the Filter operator is selected and the input and output flows are shown in the graph. You can zero in on different operators in the streams flow to see their input and output rates, and to see if any errors are occurring.

From the Metrics page you can start, stop, duplicate, and download a streams flow. If your streams flow needs to be tweaked to give you the exact information that you need, you can go directly to the edit canvas. You can view all stream flows in a table or tile view. The Notifications icon can indicate compilation or runtime errors. In addition, you can download logs and restart your Streaming Analytics service.

.

 

Prerequisites

You must have either the Editor or the Admin role.

 

To monitor or work with a streams flow, perform the following steps:

  1. From the Projects menu, click View All Projects, and then click the name of the project where your streams flow is located.

  2. In the Project page, click the Assets tab. In the Streams Flow section, click the name of the streams flow that you want.

  3. In the Metrics page, you can do any of the following actions:

    a. Run or stop the streams flow. If the Run icon Run is disabled, click the Notifications icon Error to open a panel that lists the errors. In the panel, click an error to open the streams flow in the canvas.

    b. Make a duplicate of the streams flow, and then save it with a different name in the same project.

    c. Download the streams flow to a file on your local disk. The file is in STP format. Then, you can create a new streams flow in a different project by importing the downloaded file. This functionality is useful if you want to run the streams flow in a different project or instance of Watson Studio.

    d. Change the structure of the streams flow by clicking the Edit icon Edit.

    Editing and saving a streams flow creates an updated version with the same name. If the streams flow is running and you edit and save the streams flow, a new draft of the streams flow is saved. You can redeploy with the updated streams flow.

    e. Show the throughput, number of events, and ingestion rate by doing the following actions.

    • To see all operators and the flow of data between them in the streams flow, see the Streams Flow pane. Hover your mouse pointer over a data flow to see its throughput rate and total number of events per second. When the Status is in “running” state, each data flow has a distinct color.

    • To see the number of events that are submitted to the streams flow per second for each streams flow source, see the Ingest Rate graph. Each streams flow source has a distinct color.

    • To see the throughput of a specific operator, click the operator in the Streams Flow pane. The throughput of the operator is displayed in the Throughput graph. You can see the number of events that flow into or out from the specific operator, the number of events that are dropped from the network, and the number of events that are not valid for any reason.

    • To see the events in a data flow, click anywhere in the flow. The Flow of Events table opens to show events in table and in JSON formats. Use this table to see what events are going into and out of an operator. This information is useful to debug your streams flow by comparing the incoming and outgoing events from a specific operator.

    f. If the Notifications icon Notifications icon is enabled, click the icon to see what errors occurred in the streams flow. The messages there indicate as clearly as possible what and where the problem is. The icon is enabled only when the streams flow has errors. See Troubleshooting streams flow for specific types of errors.

    In the Notifications taskbar, click any of the following icons:

    Metrics notification bar

    • Streaming Analytics instance icon Streaming Analytics icon to check the instance. In the Manage page of IBM Cloud, you can start or stop the instance.

    • Download user log icon Download user log icon to download the user log file.

    The user log contains logging messages that you put into the Code operator and the Python Machine Learning operator.

    • Download logs icon Download logs icon to download system log files. The log files are saved to your local disk in a compressed format. These logs are needed if you contact Support.

Learn more