Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add how to guide for on fail actions #1098

Merged
merged 1 commit into from
Sep 30, 2024
Merged

add how to guide for on fail actions #1098

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/concepts/error_remediation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ Guardrails provides a number of `OnFailActions` for when a validator fails. The
| `OnFailAction.FIX_REASK` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking. | No |


Custom OnFailActions can also be implemented, see the `custom` section in the [how to use on fail actions guide](../how_to_guides/use_on_fail_actions).

## Guidance on dealing with Validator errors

When a validator fails, Guardrails does two things
Expand Down
356 changes: 356 additions & 0 deletions docs/how_to_guides/use_on_fail_actions.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,356 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Use `on_fail` Actions\n",
"\n",
"`on_fail` actions are instructions to Guardrails that direct action when a validator fails. They are set at the validator level, not the guard level. \n",
"\n",
"The full set of on_fail actions are avialable in the [Error remediation concepts doc](https://www.guardrailsai.com/docs/concepts/error_remediation), and will not all be covered here.\n",
"\n",
"Instead, this interactive doc will serve to guide you on how and when to use different `on_fail` actions."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:654: UserWarning: Can't initialize NVML\n",
" warnings.warn(\"Can't initialize NVML\")\n"
]
}
],
"source": [
"# setup, run imports\n",
"from guardrails import Guard, install\n",
"try:\n",
" from guardrails.hub import DetectPII\n",
"except ImportError:\n",
" install(\"hub://guardrails/detect_pii\")\n",
" from guardrails.hub import DetectPII"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "plaintext"
}
},
"source": [
"## Raise Exceptions\n",
"\n",
"If you do not want to change the outputs of an LLM when validation fails, you can instead wrap your guard in a try/catch block.\n",
"\n",
"This works particularly well for input validation."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"output validation failed\n",
"Validation failed for field with errors: The following text in your response contains PII:\n",
"Hello, my name is John Doe and my email is john.doe@example.com\n",
"input validation failed\n",
"Validation failed for field with errors: The following text in your response contains PII:\n",
"Hello, my name is John Doe and my email is john.doe@example.com\n"
]
}
],
"source": [
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\")\n",
")\n",
"\n",
"try:\n",
" guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
"except Exception as e:\n",
" print(\"output validation failed\")\n",
" print(e)\n",
"\n",
"\n",
"# on input validation\n",
"\n",
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n",
" on=\"msg_history\"\n",
")\n",
"\n",
"\n",
"# on input validation\n",
"\n",
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n",
" on=\"msg_history\"\n",
")\n",
"\n",
"try:\n",
" guard(\n",
" model='gpt-4o-mini',\n",
" messages=[{\n",
" \"role\": \"user\",\n",
" \"content\": \"Hello, my name is John Doe and my email is john.doe@example.com\"\n",
" }]\n",
" )\n",
"except Exception as e:\n",
" print(\"input validation failed\")\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `noop` to log and continue\n",
"\n",
"If you want to log the error and continue, you can use the `noop` action. This is useful for when you want to log the error, but not stop the LLM from running."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"guarded just fine\n",
"Check if validation passed: False\n",
"Show that the validated text and raw text remain the same: True\n"
]
}
],
"source": [
"# the on_fail parameter does not have to be set, as the default is \"noop\"\n",
"\n",
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"noop\")\n",
")\n",
"\n",
"res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
"print(\"guarded just fine\")\n",
"print(\"Check if validation passed: \", res.validation_passed)\n",
"print(\"Show that the validated text and raw text remain the same: \",\n",
" res.validated_output == res.raw_llm_output)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `fix` to automatically fix the error\n",
"\n",
"Note, not all validators implement a 'fix' value. You can view the FailResult implementation in a validator to see if it has a fix value.\n",
"\n",
"Here's [an example](https://github.com/guardrails-ai/detect_pii/blob/48b15716460fe9e4e5b83ba9607ce3764d9b6d2e/validator/main.py#L188) that shows how Detect PII is written to return anonymized text as a fix value"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Check if validated_output is valid text: True\n",
"Scrubbed text: Hello, my name is <PERSON> and my email is <EMAIL_ADDRESS>\n"
]
}
],
"source": [
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"fix\")\n",
")\n",
"\n",
"res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
"\n",
"print(\"Check if validated_output is valid text: \", res.validation_passed)\n",
"print(\"Scrubbed text: \", res.validated_output)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `reask` to automatically ask for an output that passes validation\n",
"\n",
"This reask prompt is computed from the validators themselves. It's an interpolation \n",
"\n",
"In order for the reask prompt to work, the following additional params must be provided:\n",
"\n",
"- messages\n",
"- llm_api OR model"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Validated output: Sure! Here's a fictional person without any personal identifiable information:\n",
"\n",
"**Name:** <PERSON> \n",
"**Email:** <EMAIL_ADDRESS> \n",
"\n",
"Feel free to use this for any creative purposes!\n",
"Number of reasks: 1\n"
]
}
],
"source": [
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=\"reask\"),\n",
")\n",
"\n",
"res = guard(\n",
" messages=[{\n",
" \"role\": \"user\",\n",
" \"content\": \"Make up a fake person and email address\",\n",
" }],\n",
" model='gpt-4o-mini',\n",
" num_reasks=1\n",
")\n",
"\n",
"print(\"Validated output: \", res.validated_output)\n",
"print(\"Number of reasks: \", len(guard.history.last.iterations) - 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `custom` to anything else"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CUSTOM LOGIC COMPLETE!\n"
]
}
],
"source": [
"# A custom on_fail can be as simple as a function\n",
"\n",
"def custom_on_fail(value, fail_result):\n",
" # This will turn up in validated output\n",
" return \"CUSTOM LOGIC COMPLETE!\"\n",
"\n",
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
")\n",
"res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
"\n",
"print(res.validated_output)"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"String validated: Hello, my name is John Doe and my email is john.doe@example.com\n",
"\n",
"Reason it failed: [ErrorSpan(start=18, end=26, reason='PII detected in John Doe')]\n",
"\n"
]
}
],
"source": [
"# Of course, the function also has access to the fail_result and source text, \n",
"# so interesting logic/formatting over those is also possible\n",
"# Here, we show the specific char spans where the validator detected malfeasance\n",
"\n",
"def custom_on_fail(value, fail_result):\n",
" return f\"\"\"\n",
"String validated: {value}\n",
"\n",
"Reasons it failed: {fail_result.error_spans}\n",
"\"\"\"\n",
"\n",
"guard = Guard().use(\n",
" DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
")\n",
"res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
"\n",
"print(res.validated_output)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
1 change: 1 addition & 0 deletions docusaurus/sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ const sidebars = {
"how_to_guides/enable_streaming",
"how_to_guides/generate_structured_data",
"how_to_guides/custom_validators",
"how_to_guides/use_on_fail_actions",
"how_to_guides/hosting_validator_models",
"how_to_guides/hosting_with_docker",
"how_to_guides/continuous_integration_continuous_deployment",
Expand Down
Loading