Annotate with the Figure Eight platform

With Figure Eight, you can improve the quality of your data, for example, by interpreting mood, intention, or tone for text analysis, or by labeling images, in preparation for machine learning. This helps raise the model confidence in those areas in which confidence is low.

You can use the Figure Eight platform to create an annotation job within Watson Studio. For the plan and role requirements for creating annotation jobs, see Annotate data.

An annotation job describes the input data and the expected output. It includes instructions for the contributors, including rules and examples. Before you launch the job, you need to create test questions to test the performance of the contributors. For each test question that you create, you also specify a reason for the answer that you expect and an overall accuracy threshold for the job. Only contributors who have performed above this threshold are permitted to work on further jobs. See the Figure Eight documentation.

Figure Eight annotation job restrictions:

  • The data for sentiment analysis must have a column with text to analyze and a column with a URL that shows the text snippet in context. For example, a column with tweets and a column with links to the tweets in Twitter. If you don’t have a URL, you can specify another column, for example, and column with IDs.
  • Supported formats for image files are: JPEG, PNG, WebP, TIFF, GIF and SVG.
  • Image files must be packaged in a ZIP file for upload.

Set up integration between Watson Studio and Figure Eight

You must have the Admin role in the project to set up integration.

To set up integration with Figure Eight:

  1. From your project’s Settings page page, in the Integration section, click the link to create an account with Figure Eight.
  2. Add funds to your Figure Eight account. Add at least $10 into the account.
  3. Back in Watson Studio, add your access key to the Integration section.

Create an annotation job

You must have the Admin or Editor role in the project and be a team member of the Figure Eight account to create an annotation job.

To create an annotation job:

  1. On the project’s Assets page, upload, then preview the data asset that you want to base your annotation job on.
  2. Click Annotate.
  3. Choose a job template for your sentiment analysis project or for image annotation.
    • Click a template tile to see the input to and output from the job.
    • Click the eye (eye icon) icon to see a preview of the template.

    If none of the predefined templates fit your annotation needs, choose Start from scratch to create your own job template. Your whole data set for this job is sent to Figure Eight.

  4. If you choose a predefined job template for sentiment analysis, specify the columns from your data set to annotate. Only the data from those columns is sent to Figure Eight.
  5. Create the job. The CSV file or .zip file is validated to ensure consistency and uniformity and your job template is uploaded to Figure Eight where you can design your job. For image annotation, a unique URL is assigned for each image file.
  6. In Figure Eight, modify the template to fit your data set, for example, reword the questions or the instructions to answer the questions
  7. Create the test questions which contributors must pass to be able to work on your job.
  8. Launch a test run of your job, specifying the scope of the test run, such as the number of rows to order, the amount to pay per row and the number of judgments per row. You are shown the expected cost of your job. If necessary, add more funds to cover the expected cost of the job.
  9. Launch your annotation job.

In the information pane on the asset’s Preview page in Watson Studio, you can see the annotation job and its status. You can also click the link to open the Figure Eight monitoring dashboard. The dashboard displays information such as the job progress, incurred costs, number of trusted judgements, and contributor performance.

You are notified when the annotation job is complete. The results appear in a new data asset in your project. The annotated CSV file for sentiment analysys contains the original columns plus the annotations, in the what_is_the_authors_sentiment_feeling column, and other columns with information about the annotations. The file for image annotation contains the URL for each image, information about the bounding coordinates for the image, and the annotation information for each image.

Manage annotation jobs

You can add funds to annotation platform account by clicking the link in the Integrations section on the Settings page.

You can add project collaborators as team members to your annotation platform account. See Figure Eight account team management.