Skip to content

Commit

Permalink
Updated docs
Browse files Browse the repository at this point in the history
  • Loading branch information
penguine-ip committed Jul 3, 2024
1 parent a01860b commit 442c5bd
Showing 1 changed file with 69 additions and 19 deletions.
88 changes: 69 additions & 19 deletions docs/docs/evaluation-datasets-synthetic-data.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,13 @@ There are 3 approaches a `deepeval`'s `Synthesizer` can generate synthetic `Gold

1. Generating synthetic `Golden`s using **context extracted from documents.**
2. Generating synthetic `Golden`s from a **list of provided context.**
3. Generating synthetic red-teaming `Golden`s.
3. Generating synthetic adversarial `Golden`s for red teaming.

### 1. Generating From Documents

To generate synthetic `Golden`s from documents, simply provide a list of document paths:

:::info
:::note
The `generate_goldens_from_docs` method employs a token-based text splitter to manage document chunking, meaning the `chunk_size` and `chunk_overlap` parameters do not guarantee exact context sizes. This approach is designed to ensure **meaningful and coherent context extraction**, but might lead to variations in the expected size of each `context`.
:::

Expand All @@ -64,19 +64,38 @@ There are one mandatory and seven optional parameters when using the `generate_g

- `document_paths`: a list strings, representing the path to the documents from which contexts will be extracted from. Supported documents types include: `.txt`, `.docx`, and `.pdf`.
- [Optional] `include_expected_output`: a boolean which when set to `True`, will additionally generate an `expected_output` for each synthetic `Golden`. Defaulted to `False`.
- [Optional] `max_goldens_per_document`: the maximum number of golden data points to be generated for each document. Defaulted to 5.
- [Optional] `max_goldens_per_document`: the maximum number of goldens to be generated per document. Defaulted to 5.
- [Optional] `chunk_size`: specifies the size of text chunks (in characters) to be considered for context extraction within each document. Defaulted to 1024.
- [Optional] `chunk_overlap`: an int that determines the overlap size between consecutive text chunks during context extraction. Defaulted to 0.
- [Optional] `num_evolutions`: the number of evolution steps to apply to each generated input. This parameter controls the **complexity and diversity** of the generated dataset by iteratively refining and evolving the initial inputs. Defaulted to 1.
- [Optional] `enable_breadth_evolve`: a boolean which when set to `True`, introduces a **wider variety of context modifications**, enhancing the dataset's diversity. Defaulted to `False`.
- [Optional] `evolution_types`: a list of `Evolution`, specifying methods used during data evolution. Defaulted to all `Evolution`s.
- [Optional] `evolutions`: a list of `Evolution`s, specifying the type of data evolution used. Defaulted to all `Evolution`s.

:::info
`Evolution` is an `ENUM` that specifies the different data evolution techniques you wish to employee to make synthetic `Golden`s more realistic.

```python
from deepeval.synthesizer import Evolution

available_evolutions = [
Evolution.REASONING,
Evolution.MULTICONTEXT,
Evolution.CONCRETIZING,
Evolution.CONSTRAINED,
Evolution.COMPARATIVE,
Evolution.HYPOTHETICAL
]
```

For those interested in what these evolutions mean, you can [read this article here.](https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms)
:::

### 2. Generating From Provided Contexts

`deepeval` also allows you to generate synthetic `Goldens` from a manually provided a list of context instead of directly generating from your documents.

:::tip
This is especially helpful if you already have an indexed and/or embedded knowledge base. For example, if you already have documents parsed and stored in an existing vector database, you may consider handling the logic to retrieve text chunks yourself.
This is especially helpful if you already have an embedded knowledge base. For example, if you already have documents parsed and stored in an existing vector database, you may consider handling the logic to retrieve text chunks yourself.
:::

```python
Expand All @@ -96,53 +115,70 @@ There are one mandatory and five optional parameters when using the `generate_go

- `contexts`: a list of context, where each context is itself a list of strings, ideally sharing a common theme or subject area.
- [Optional] `include_expected_output`: a boolean which when set to `True`, will additionally generate an `expected_output` for each synthetic `Golden`. Defaulted to `False`.
- [Optional] `max_goldens_per_context`: the maximum number of golden data points to be generated from each context. Adjusting this parameter can influence the size of the resulting dataset. Defaulted to 2.
- [Optional] `max_goldens_per_context`: the maximum number of goldens to be generated per context. Defaulted to 2.
- [Optional] `num_evolutions`: the number of evolution steps to apply to each generated input. This parameter controls the **complexity and diversity** of the generated dataset by iteratively refining and evolving the initial inputs. Defaulted to 1.
- [Optional] `enable_breadth_evolve`: a boolean indicating whether to enable breadth evolution strategies during data generation. When set to True, it introduces a **wider variety of context modifications**, enhancing the dataset's diversity. Defaulted to `False`.
- [Optional] `evolution_types`: a list of `Evolution`, specifying methods used during data evolution. Defaulted to all `Evolution`s.
- [Optional] `evolutions`: a list of `Evolution`s, specifying the type of data evolution used. Defaulted to all `Evolution`s.

:::caution
While the previous methods first use an LLM to generate a series of inputs based on the provided context before evolving them, `generate_goldens_from_inputs` simply evolves the provided list of inputs into more complex and diverse `Golden`s. It's also important to note that this method will only populate the input field of each generated `Golden`.
:::

### 3. Generating Red-Teaming `Goldens`
### 3. Generating Adversarial `Goldens` for Red Teaming

`deepeval` also allows you to generate synthetic red-teaming `Goldens`, with or without a list of contexts.
`deepeval` also allows you to generate adversarial `Goldens` to [red team LLM applications](https://www.confident-ai.com/blog/red-teaming-llms-a-step-by-step-guide). You can also optionally provide a list of contexts to keep each generated synthetic `Golden` grounded in your data.

:::tip
This is especially helpful if you are trying to **assess your LLM application's vulnerabilities**.
This is especially helpful for assessing your LLM application's **risks and vulnerabilities**.
:::

```python
from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
synthesizer.generate_red_team_goldens(
# Provide a list of context for synthetic data generation
synthesizer.generate_red_teaming_goldens(
# Optionally provide a list of context to keep adversarial goldens grounded
contexts=[
["The Earth revolves around the Sun.", "Planets are celestial bodies."],
["Water freezes at 0 degrees Celsius.", "The chemical formula for water is H2O."],
]
)
```

There are six optional parameters when using the `generate_red_team_goldens` method:
There are six optional parameters when using the `generate_red_teaming_goldens` method:

- [Optional] `contexts`: a list of context, where each context is itself a list of strings, ideally sharing a common theme or subject area.
- [Optional] `contexts`: a list of context, where each context is itself a list of strings, ideally sharing a common theme or subject area. When `contexts` is supplied, the `Synthesizer` generates adversarial goldens **based on information presented in contexts**. If not, they are generated from scratch.
- [Optional] `include_expected_output`: a boolean which when set to `True`, will additionally generate an `expected_output` for each synthetic `Golden`. Defaulted to `False`.
- [Optional] `max_goldens`: the maximum number of golden data points to be generated if `contexts` isn't supplied, or the max number of golden datapoints per context if `contexts` is supplied. Adjusting this parameter can influence the size of the resulting dataset. Defaulted to 2.
- [Optional] `num_evolutions`: the number of evolution steps to apply to each generated input. This parameter controls the **complexity and diversity** of the generated dataset by iteratively refining and evolving the initial inputs. Defaulted to 1.
- [Optional] `evolution_types`: a list of `RedTeamEvolution`, specifying methods used during data evolution. Defaulted to all `RedTeamEvolution`s.
- [Optional] `responses`: a list of `Response`, representing the red-teaming behaviour you are trying to elicit. Defaulted to all `Response`s.
- [Optional] `evolutions`: a list of `RedTeamEvolution`s, specifying methods used during data evolution. Defaulted to all `RedTeamEvolution`s.
- [Optional] `responses`: a list of `Response`s, representing the red teaming behaviour you are trying to elicit. Defaulted to all `Response`s.

:::info
`RedTeamEvolution` is an `ENUM` that specifies the different **types of red-teaming** prompts (prompt injection, prompt probing, etc.) you wish to include in your final dataset, while the `Response` `ENUM` is the **type of response you are trying to get your LLM to produce** (hallucination, bias, etc.).
`RedTeamEvolution` is an `ENUM` that specifies the different **types of red-teaming** prompts (prompt injection, prompt probing, etc.) you wish to include in your final dataset, while the `Response` `ENUM` is the **type of response you are trying to get your LLM to produce** (hallucination, bias, etc.):

The `contexts` is supplied, the Synthesizer simply generates red-teaming prompts **based on the contexts**. If not, they are generated from scratch.
```python
from deepeval.synthesizer import RedTeamEvolution, Response

available_red_teaming_evolutions = [
RedTeamEvolution.PROMPT_INJECTION,
RedTeamEvolution.PROMPT_PROBING,
RedTeamEvolution.GRAY_BOX_ATTACK,
RedTeamEvolution.JAIL_BREAKING
]

available_red_teaming_responses = [
Response.HALLUCINATION,
Response.OFFENSIVE,
Response.BIAS,
Response.DATA_LEAKAGE,
Response.UNFORMATTED
]
```

:::

### Saving Generated Goldens
## Saving Generated Goldens

To not accidentally lose any generated synthetic `Golden`, you can use the `save_as()` method:

Expand Down Expand Up @@ -198,6 +234,20 @@ dataset.generate_goldens(
)
```

Or, to `generate_red_teaming_goldens`:

```python
...

dataset.generate_red_teaming_goldens(
synthesizer=synthesizer,
contexts=[
["The Earth revolves around the Sun.", "Planets are celestial bodies."],
["Water freezes at 0 degrees Celsius.", "The chemical formula for water is H2O."],
]
)
```

Lastly, don't forget to call `save_as()` to perserve any generated synthetic `Golden`:

```python
Expand Down

0 comments on commit 442c5bd

Please sign in to comment.