watsonx.governance is IBM’s AI governance offering to manage and monitor (Generative) AI solutions. It is available as SaaS or as software to be run everywhere. This post demonstrates how to do this for a simple summarization scenario.
Here is a definition of IBM watsonx.governance:
IBM watsonx.governance was built to help you direct, manage and monitor the artificial intelligence (AI) activities of your organization: 1. Govern generative AI (gen AI) and machine learning (ML) models from any vendor including IBM watsonx.ai, Amazon Sagemaker and Bedrock, Google Vertex and Microsoft Azure. 2. Evaluate and monitor for model health, accuracy, drift, bias and gen AI quality. 3. Access powerful governance, risk and compliance capabilities featuring workflows with approvals, customizable dashboards, risk scorecards and reports. 4. Use factsheet capabilities to collect and document model metadata automatically across the AI model lifecycle.
Documentation, videos and blog posts:
- Documentation: Generative AI quality evaluations
- Evaluating deployments in spaces
- Video: IBM watsonx.governance: Direct, manage and monitor your Generative AI and ML models, anywhere
- Evaluting Question Answering Scenarios with watsonx.governance
- Custom LLM Metrics in watsonx.governance
Scenario
In the example below summarization is done via a 13b Granite model running on watsonx.ai. A prompt template has been defined which allows passing in different data for evaluations via a variable.
Evaluations
For this model different GenAI evaluations have been defined which are provided by watsonx. For the evaluations test sets need to provide ground truth information.
Results are displayed after the evaluations.
The home page provides a dashboard with results of all deployed models.
The model health page displays information about number of requests, token counts and more.
For GenAI scenarios different out-of-the-box metrics are shown.
The factsheet displays the same information.
Evaluations can be run on all stages in the AI lifecycle.
Use Cases
Use cases have approval workflows which can be defined with watsonx.governance.
The same use cases can also be viewed in watsonx (.ai).
Next Steps
To learn more, check out the Watsonx.ai documentation and the Watsonx.ai landing page.