heidloff.net - Building is my Passion
Post
Cancel

Generative AI Quality Metrics in watsonx.governance

watsonx.governance is IBM’s AI governance offering to manage and monitor (Generative) AI solutions. It is available as SaaS or as software to be run everywhere. This post demonstrates how to do this for a simple summarization scenario.

Here is a definition of IBM watsonx.governance:

IBM watsonx.governance was built to help you direct, manage and monitor the artificial intelligence (AI) activities of your organization: 1. Govern generative AI (gen AI) and machine learning (ML) models from any vendor including IBM watsonx.ai, Amazon Sagemaker and Bedrock, Google Vertex and Microsoft Azure. 2. Evaluate and monitor for model health, accuracy, drift, bias and gen AI quality. 3. Access powerful governance, risk and compliance capabilities featuring workflows with approvals, customizable dashboards, risk scorecards and reports. 4. Use factsheet capabilities to collect and document model metadata automatically across the AI model lifecycle.

Documentation, videos and blog posts:

Scenario

In the example below summarization is done via a 13b Granite model running on watsonx.ai. A prompt template has been defined which allows passing in different data for evaluations via a variable.

image

Evaluations

For this model different GenAI evaluations have been defined which are provided by watsonx. For the evaluations test sets need to provide ground truth information.

image

Results are displayed after the evaluations.

image

image

The home page provides a dashboard with results of all deployed models.

image

The model health page displays information about number of requests, token counts and more.

image

For GenAI scenarios different out-of-the-box metrics are shown.

image

The factsheet displays the same information.

image

Evaluations can be run on all stages in the AI lifecycle.

image

Use Cases

Use cases have approval workflows which can be defined with watsonx.governance.

image

The same use cases can also be viewed in watsonx (.ai).

image

Next Steps

To learn more, check out the Watsonx.ai documentation and the Watsonx.ai landing page.

Featured Blog Posts
Disclaimer
The postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.
Contents
Trending Tags