Update retraining to work with RunMe

[jlewi]

We need to update our retraining code to work with the data produced by RunMe

• Related to Support using Foyle's AI capabilities as a plugin stateful/runme#574

[jlewi]

Needed Changes

We need to update the Logs Analyzer to process the RunMe logs.

To Process the RunMe logs there's a couple things we need to do

• We need to get the log entries associated with Execute Requests

• We need to build a map from RunMe ULID to Foyle BlockIDs

  • We need to parse out the (ULID,ID) mapping from the LogEntry
  • This requires updating combineEntriesForTrace to store the ULID as part of the trace

• buildTraces needs to support RunMe Execute Requests

  • Option 1: Reuse ExecuteTrace to represent a RunMe execute request
  • Option 2: Define a new RunmeExecuteTrace
  • I think Option 2 is more consistent with where we are headed. For a RunMe trace different protos are used and we should log those rather than doing some sort of Lossy conversion to Foyle protos. If we just add those protos as fields to ExecuteTrace then it becomes harder to tell what type of trace we have and therefore what fields should be set
  • Option 2 is more consistent with introducing a proto which will use a OneOf to combine types that are unique for each type of trace Use Pebble to store traces #91
  • For RunMe execute traces we could use the _id field. This is a ULID but that shouldn't matter.

• We will need to change buildTraces because right now we iterate over all the traces (Generate and Execute) and add them to the associated block log

  • Now we need to do the following
  • Iterate over all the Generate Traces
    • Create a mapping from RunMe ULID to BlockID
  • Iterate over all the ExecuteTraces
    • Map ULID to BlockID
    • Lookup BlockID in BlockLog map.

Configuration

• I think we should define a LearnerConfig and add it to our Config this should allow people to configure additional log locations.

[jlewi]

Alternative: Use ULIDs for BlocKIDs

• An alternative is to change Foyle to use ULIDs for blockIDs.
• I think having a single consistent ID will simplify things and avoid the headaches caused by constantly needing to map back and forth.
• Its possible RunMe can work with UUIDs but we can just use ULIDs. IDs are string and we don't much care what they are as long as they are unique.

[jlewi]

Right now each type of trace is stored in a flat JSONL file

foyle/app/pkg/analyze/analyzer.go

Line 135 in 23a7b1e

genEnc := json.NewEncoder(genFile)

We rely on the filename to know the type of trace.
If we don't fix #91 first; we will have to add another set of JSONL files.

[jlewi]

With #121 we should be code complete but will need to do some E2E testing to see if its all working.

[jlewi]

So how do we verify everything is working

1. Use RunMe to generate a completion
2. Correct the completion
3. Train
4. Verify example is added to the training set
5. Generate a similar completion
6. Verify the example is used in the prompt

Since the RunMe changes haven't been released yet. We need to use our locally built version of RunMe.

[jlewi]

No examples are produced based on the foyle logs.
When I looked at the blocklog for block 01HZ0N1228NJ7PSRYB6WEMH08M there was no exec id.

[jlewi]

I didn't update the learning code to include the directory of the RunMe logs