Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async eval callback #702

Merged
merged 66 commits into from
Dec 19, 2023
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
e70a2f4
Async eval callback
aspfohl Oct 27, 2023
acd8b2e
add very basic tests
aspfohl Nov 8, 2023
aef814b
more tests
aspfohl Nov 8, 2023
007ae90
bump mcli
aspfohl Nov 8, 2023
d04aa40
Merge branch 'main' into anna/asynceval
aspfohl Nov 9, 2023
6cd020f
woop, missing import
aspfohl Nov 9, 2023
9fbe7a1
instance not specified error
aspfohl Nov 9, 2023
547ec21
fixes
aspfohl Nov 9, 2023
9819456
fix typing
aspfohl Nov 9, 2023
ba871d7
small testing fixes
aspfohl Nov 9, 2023
21e9880
launch new run only on main process
aspfohl Nov 9, 2023
47a8255
logger name
aspfohl Nov 9, 2023
08c24be
items
aspfohl Nov 9, 2023
bf415f0
format
aspfohl Nov 10, 2023
ecacdac
Merge branch 'main' into anna/asynceval
aspfohl Nov 10, 2023
5616ae4
Update llmfoundry/callbacks/async_eval_callback.py
aspfohl Nov 10, 2023
bc1647a
Update llmfoundry/callbacks/async_eval_callback.py
aspfohl Nov 10, 2023
3358837
feedback
aspfohl Nov 10, 2023
28e47df
Apply suggestions from code review
aspfohl Nov 13, 2023
78cc0b8
small updates
aspfohl Nov 13, 2023
7baa53f
Merge branch 'main' into anna/asynceval
aspfohl Nov 13, 2023
b58ccf9
use parameters from train.py to capture overrides and mounted paramet…
aspfohl Nov 13, 2023
d85ee5e
config_overrides
aspfohl Nov 14, 2023
0e96fea
Merge branch 'main' into anna/asynceval
aspfohl Nov 14, 2023
194774d
updates
aspfohl Nov 15, 2023
ba1280a
Merge branch 'main' into anna/asynceval
aspfohl Nov 15, 2023
08857d5
fix test
aspfohl Nov 15, 2023
6ce8b77
small fixes
aspfohl Nov 15, 2023
de155f7
add logging
aspfohl Nov 15, 2023
e5f9e9e
remove last launch check
aspfohl Nov 15, 2023
3f518f9
better logging
aspfohl Nov 15, 2023
ea742ef
fix parameters
aspfohl Nov 16, 2023
e3623f3
fix double unit in the name
aspfohl Nov 16, 2023
2deef3f
sadz
aspfohl Nov 16, 2023
91bdf43
Merge branch 'main' into anna/asynceval
aspfohl Nov 16, 2023
e8f4661
Merge branch 'main' into anna/asynceval
dakinggg Nov 30, 2023
f9e2dc7
Merge branch 'main' into anna/asynceval
aspfohl Dec 2, 2023
add7fbb
fies
aspfohl Dec 2, 2023
238086f
git integration path validation and update
aspfohl Dec 4, 2023
53a9943
detect forks, better error/comment
aspfohl Dec 4, 2023
1f35a7b
version import
aspfohl Dec 4, 2023
99f48cb
merge with main
aspfohl Dec 4, 2023
1184531
last checkpoint
aspfohl Dec 5, 2023
7af7383
post_close -> close
aspfohl Dec 6, 2023
9337af0
add todos, fix path bug
aspfohl Dec 6, 2023
87ffd86
add missing args
aspfohl Dec 6, 2023
e940f1c
remove eval_loader in callback too
aspfohl Dec 6, 2023
bb040d1
remove fit end event (already doing on close)
aspfohl Dec 6, 2023
14f386f
misc fixes
aspfohl Dec 6, 2023
ac37d09
fix test
aspfohl Dec 6, 2023
535d5fb
Merge branch 'main' into anna/asynceval
aspfohl Dec 7, 2023
9e11cf7
add back eval interval
aspfohl Dec 8, 2023
aa652f3
build_loggers and add tests
aspfohl Dec 8, 2023
818f4ac
Merge branch 'main' into anna/asynceval
aspfohl Dec 8, 2023
0e4f085
updates
aspfohl Dec 11, 2023
bf06c19
Merge branch 'main' into anna/asynceval
aspfohl Dec 11, 2023
dc25b2a
typing
aspfohl Dec 11, 2023
6865b96
changes
aspfohl Dec 11, 2023
9754b03
Merge branch 'main' into anna/asynceval
aspfohl Dec 11, 2023
1ac70cc
typing?
aspfohl Dec 11, 2023
1a485d4
Merge branch 'main' into anna/asynceval
aspfohl Dec 13, 2023
cd2a31d
metadata in eval.py
aspfohl Dec 18, 2023
f6393a9
Merge branch 'main' into anna/asynceval
aspfohl Dec 18, 2023
04865db
actually, just log metadata on every model eval
aspfohl Dec 19, 2023
557aca8
Merge branch 'main' into anna/asynceval
aspfohl Dec 19, 2023
2fd2317
Merge branch 'main' into anna/asynceval
aspfohl Dec 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/pytest-gpu.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ jobs:
run: |
set -ex
python -m pip install mosaicml-cli
mcli init --mcloud
aspfohl marked this conversation as resolved.
Show resolved Hide resolved
mcli version
- name: Submit Run
id: tests
Expand Down
2 changes: 2 additions & 0 deletions llmfoundry/callbacks/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# SPDX-License-Identifier: Apache-2.0

try:
from llmfoundry.callbacks.async_eval_callback import AsyncEval
from llmfoundry.callbacks.eval_gauntlet_callback import EvalGauntlet
from llmfoundry.callbacks.fdiff_callback import FDiffMetrics
from llmfoundry.callbacks.generate_callback import Generate
Expand All @@ -28,4 +29,5 @@
'EvalGauntlet',
'ModelGauntlet',
'HuggingFaceCheckpointer',
'AsyncEval',
]
Loading
Loading