Skip to content

Latest commit

 

History

History
44 lines (34 loc) · 1.43 KB

README.md

File metadata and controls

44 lines (34 loc) · 1.43 KB
Raw

Viya/Airtrain

This repo contains some code for pre-processing ViyaMD data for purposes of usage with Airtrain. It's primary purpose is to start from a directory containing json files with synthetic chats and convert them to jsonl files ready for training and tuning in Airtrain.

Setup

  • Make sure you have python3.10 installed
  • Change your working directory to the root of this repo
  • make prep # will prepare a python virtial env in .venv within the repo

Basic Usage

  • Every time you want to work in the repo: source .venv/bin/activate
  • To automatically format and address simple lint errors: make fix
  • To check formatting, linting, and type-checks: make lint
  • To add a new dependency, edit requirements.in then make refresh-dependencies which will update requirements.txt
  • To sync with the contents of requirements.txt: make sync

Converting Data

Make sure you have a directory filled with json files, each of which contains a full simulated session between the user and Viya.

To see script usage:

python3 viya_airtrain/scripts/prepare_jsonl.py --help

Example:

python3 viya_airtrain/scripts/prepare_jsonl.py \
    --source-dir ~/datasets/viya1/all/ \
    --destination ~/datasets/viya1/jsonls/train-history.jsonl \
    --output-mode turn \
    --test-destination ~/datasets/viya1/jsonls/test-history.jsonl \
    --agent MedicalHistoryAgent \
    --agent FamilyHistoryAgent \
    --agent SocialHistoryAgent