dvclive: lightning log_model #4714

dberenbaum · 2023-07-21T20:45:27Z

Docs for iterative/dvclive#613

github-actions · 2023-07-21T21:01:26Z

Link Check Report

There were no links to check!

shcheklein

A few comments. Good stuff, thanks @dberenbaum .

shcheklein · 2023-07-21T21:40:56Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

@@ -69,6 +86,17 @@ checkpointing at all as described in the

 ## Examples

+- Using `log_model` to save the checkpoints and annotate the best one for use in


Can we show an output / end result of it?

like actual dvc.yaml and directory structure?

shcheklein · 2023-07-21T22:46:08Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

+  created by
+  [`ModelCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html)
+  and annotate the best checkpoint with `type=model` and `name=best` for use in
+  [Studio model registry]. DVCLive will <abbr>cache</abbr> the checkpoint


It took me a while to understand the part about cache and being able to recover it. I think most of the ppl will be confused by this (I expect they don't know much about DVC mechanics). Two suggestions:

remove it

try to be more explicit: DVCLive saves checkpoints with DVC - it means they are cached in DVC cache, also could be pushed to remote storage, etc. It also means that DVCLive drops the directory (it's safe) when it runs a new experiments. Model weights for any experiment are stored in DVC and can be restored ...

Btw, what happens, if save_dvc_exp is False? How do I recover those?

shcheklein · 2023-07-21T22:47:51Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

+- `log_model` - (`False` by default) - use
+  [`live.log_artifact()`](/doc/dvclive/live/log_artifact) to log checkpoints
+  created by
+  [`ModelCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html)


Do I need to use ModelCheckpoint along side? Ot it's being used internally?

shcheklein · 2023-07-21T22:48:39Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

+  directory and delete checkpoints from previous <abbr>experiments</abbr>, but
+  you can recover the checkpoints from any experiment using DVC.
+
+  - if `log_model == 'all'`, checkpoints are logged during training.


all checkpoints are logged during training?

also, what does it practically mean? does it create a record per each checkpoint in dvc.yaml?

shcheklein · 2023-07-21T22:49:21Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

@@ -61,6 +61,23 @@ checkpointing at all as described in the

 - `prefix` - (`None` by default) - string that adds to each metric name.

+- `log_model` - (`False` by default) - use
+  [`live.log_artifact()`](/doc/dvclive/live/log_artifact) to log checkpoints


log checkpoints - does it means that we might have multiple records? per checkpoint?

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

dberenbaum · 2023-07-22T14:19:22Z

Updated to address the comments, moving most of the details to the examples. Hopefully the docs answer the questions, so I will hold off on answering each question to see if you find the doc answers them sufficiently.

dberenbaum · 2023-07-22T14:25:47Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

+`Live.log_artifact()`. At the end of training, DVCLive will annotate the
+[`best_model_path`][`ModelCheckpoint`] with `type=model` and `name=best` for use
+in [Studio model registry].


Need a good example of how to use the best model, but I think it's blocked by iterative/dvc#9100 (comment)

content/docs/dvclive/live/log_artifact.md

dberenbaum · 2023-07-25T20:53:17Z

Adding reminders here to update vs code and studio snippets (cc @mattseddon @yathomasi )

shcheklein · 2023-07-26T18:38:46Z

content/docs/dvclive/ml-frameworks/pytorch-lightning.md

@@ -69,6 +82,52 @@ checkpointing at all as described in the

 ## Examples

+### Log model checkpoints
+
+Use `log_model` to save the checkpoints. DVCLive will first delete checkpoints


It's still so complicated :(

Sorry, Dave. I know you put tons of effort into this. And may be that's the best we can do ... but nonetheless we should recognize that's quire complex.

Let's start with this. Can it be just:

Use log_model to save the checkpoints (it will use Live.log_artifact() internally to save those). At the end of training, DVCLive will also mark the best the [best_model_path][ModelCheckpoint] (e.g. for then to be consumed in the Model Registry or automation scenarios).

Even this one is still too complicated. Do ppl care about MR at this point? I don't know . Can it go after the code example?

TBH I don't have that strong an opinion here at this point. I'm fine to go with that suggestion for now. We can always revisit.

shcheklein · 2023-07-26T18:41:35Z

content/docs/dvclive/live/log_artifact.md

-Uses `dvc add` to track `path` with DVC, generating a `{path}.dvc` file. When
-combined with [`save_dvc_exp=True`](/doc/dvclive#initialize-dvclive), it will
-ensure that `{path}.dvc` is included in the experiment.
+Uses `dvc add` to [track] `path` with DVC, saving it to the DVC


tbh, I still feel that we go right away into implementation details. High level (to me) this command first of all creates an entry somewhere (in this case dvc.yaml) with some information about the model - that we now even know that there is a model in the repo.

dvc add can be there or not, we should support other storages and other ways to get data then eventually.

tbh, I still feel that we go right away into implementation details.

I added an intro paragraph to introduce it at a high level.

High level (to me) this command first of all creates an entry somewhere (in this case dvc.yaml) with some information about the model - that we now even know that there is a model in the repo.

dvc add can be there or not, we should support other storages and other ways to get data then eventually.

I don't really agree here. Every other logger I know uses log_artifact to store the artifact data.

don't other loggers have any means to log a remote model?

in our case I think we should support all the ways to store the models. that's why it feels a bit low level and artificial (it can be even a file in Git after all)

Yes, we should support other ways to store models (we are discussing in iterative/dvclive#551 and we already have `Live.log_artifact(cache=False) that goes a little bit in this direction), but the expected default behavior for logging artifacts is to save them to the logger's storage so that it can be recovered later. In the intro, I think we should emphasize it as the primary scenario for the method.

dberenbaum · 2023-07-31T01:01:36Z

@shcheklein Let me know if you feel there are any blockers left.

content/docs/dvclive/live/log_artifact.md

dvclive: lightning log_model

9cca530

dberenbaum requested a review from daavoo July 21, 2023 20:45

shcheklein temporarily deployed to dvc-org-lightning-model-yorcas July 21, 2023 20:49 Inactive

dberenbaum mentioned this pull request Jul 21, 2023

auto save model in lightning iterative/dvclive#613

Closed

4 tasks

shcheklein reviewed Jul 21, 2023

View reviewed changes

daavoo reviewed Jul 22, 2023

View reviewed changes

content/docs/dvclive/ml-frameworks/pytorch-lightning.md Outdated Show resolved Hide resolved

expand lightning log_model examples

c04ee1b

shcheklein temporarily deployed to dvc-org-lightning-model-yorcas July 22, 2023 14:17 Inactive

dberenbaum commented Jul 22, 2023

View reviewed changes

dberenbaum requested review from shcheklein and daavoo July 25, 2023 20:31

shcheklein reviewed Jul 25, 2023

View reviewed changes

content/docs/dvclive/live/log_artifact.md Outdated Show resolved Hide resolved

drop mention of save_dvc_exp in log_artifact

f8ee570

shcheklein temporarily deployed to dvc-org-lightning-model-yorcas July 25, 2023 21:27 Inactive

daavoo approved these changes Jul 26, 2023

View reviewed changes

shcheklein reviewed Jul 26, 2023

View reviewed changes

tweak log_artifact language

94df019

shcheklein temporarily deployed to dvc-org-lightning-model-yorcas July 31, 2023 01:00 Inactive

dberenbaum requested a review from shcheklein July 31, 2023 01:01

shcheklein approved these changes Jul 31, 2023

View reviewed changes

daavoo reviewed Jul 31, 2023

View reviewed changes

content/docs/dvclive/live/log_artifact.md Outdated Show resolved Hide resolved

Update content/docs/dvclive/live/log_artifact.md

da0bdaf

shcheklein temporarily deployed to dvc-org-lightning-model-yorcas July 31, 2023 11:29 Inactive

dberenbaum merged commit dbc446e into main Jul 31, 2023
2 checks passed

dberenbaum deleted the lightning-model branch July 31, 2023 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dvclive: lightning log_model #4714

dvclive: lightning log_model #4714

dberenbaum commented Jul 21, 2023

github-actions bot commented Jul 21, 2023 •

edited

Loading

shcheklein left a comment

shcheklein Jul 21, 2023

shcheklein Jul 21, 2023

shcheklein Jul 21, 2023

shcheklein Jul 21, 2023

shcheklein Jul 21, 2023

shcheklein Jul 21, 2023

dberenbaum commented Jul 22, 2023

dberenbaum Jul 22, 2023 •

edited

Loading

dberenbaum commented Jul 25, 2023

shcheklein Jul 26, 2023

dberenbaum Jul 31, 2023

shcheklein Jul 26, 2023

dberenbaum Jul 31, 2023

shcheklein Jul 31, 2023

dberenbaum Jul 31, 2023

dberenbaum commented Jul 31, 2023

		@@ -69,6 +86,17 @@ checkpointing at all as described in the

		## Examples

		- Using `log_model` to save the checkpoints and annotate the best one for use in

dvclive: lightning log_model #4714

dvclive: lightning log_model #4714

Conversation

dberenbaum commented Jul 21, 2023

github-actions bot commented Jul 21, 2023 • edited Loading

Link Check Report

shcheklein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dberenbaum commented Jul 22, 2023

dberenbaum Jul 22, 2023 • edited Loading

Choose a reason for hiding this comment

dberenbaum commented Jul 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dberenbaum commented Jul 31, 2023

github-actions bot commented Jul 21, 2023 •

edited

Loading

dberenbaum Jul 22, 2023 •

edited

Loading