watsonx.governance is IBM’s AI governance offering to manage and monitor (Generative) AI solutions. It is available as SaaS or as software to be run everywhere. This post demonstrates how to do this for a Question Answering scenario with an external model running on Azure.
Here is a definition of IBM watsonx.governance:
IBM watsonx.governance was built to help you direct, manage and monitor the artificial intelligence (AI) activities of your organization: 1. Govern generative AI (gen AI) and machine learning (ML) models from any vendor including IBM watsonx.ai, Amazon Sagemaker and Bedrock, Google Vertex and Microsoft Azure. 2. Evaluate and monitor for model health, accuracy, drift, bias and gen AI quality. 3. Access powerful governance, risk and compliance capabilities featuring workflows with approvals, customizable dashboards, risk scorecards and reports. 4. Use factsheet capabilities to collect and document model metadata automatically across the AI model lifecycle.
Documentation and blog posts:
- Documentation: Generative AI quality evaluations
- Evaluating deployments in spaces
- Generative AI Quality Metrics in watsonx.governance
- Custom LLM Metrics in watsonx.governance
Overview
For the Question Answering use case GPT is used running on Azure, but the same mechanism works for other models too. Question Answering is often utilized in RAG (Retrieval Augmented Generation) pipelines.
Evaluations
For this model different GenAI evaluations, for example answer quality, faithfulness and answer relevance, have been defined which are provided by watsonx. For the evaluations test sets need to provide ground truth information.
Results are displayed after the evaluations.
For RAG scenarios the documents are displayed which were passed in to the models to generate answers.
You can even select different parts of the answer to see from which documents they came from.
Faithfulness, answer relevance and unsuccessful requests are also displayed graphically.
Code
Since this example uses an external model, a detached prompt template needs to be created, for example in a notebook.
Next Steps
To learn more, check out the Watsonx.ai documentation and the Watsonx.ai landing page.