Adding extracted text to a RAG solution

Based on the file type you configure in your text extraction request, the structure and format of the extracted output differs. You may need to perform some post-processing on the result before you can use the contents as grounding data in your RAG solution.

You can convert the generated Markdown file into a text file by changing the file extension from .md to .txt. The resulting text file includes the Markdown tags. If you want to remove the tagging, you can use a parser library to find and convert the tags.

You can use a JSON processor library to extract text from the generated JSON file and store it as plain text. For example, the following command extracts the text from each token for all structures in the document and stores the text in a file named parsed_output_text.txt:

cat output_retail.json | jq '[.all_structures.tokens[].text] | join(" ")' > parsed_output_text.txt

After you convert the generated file to a TXT file, you can use the extracted text as contextual information for a foundation model prompt in the following ways:

• Reference the extracted text from a Python notebook.

  For example, you can use your TXT file instead of the state_of_the_union.txt file in the Use watsonx, Chroma, and LangChain to answer questions (RAG) sample notebook.

• You can use the TXT file as a grounding document in Prompt Lab. For more information, see Grounding foundation model prompts in contextual information.