feat: [VRD-1120, VRD-1130, VRD-1132] Add `finetune()` #4085

liuverta · 2023-09-19T00:12:21Z

Impact and Context

Creates the ER for tracking the fine-tuning job, creates the RMV for storing the fine-tuned model, logs datasets and internal attributes, and POSTs /finetuning-job.

Risks and Area of Effect

Is this a breaking change?

There's a very touchy refactor of a handful of Client methods, but I covered them in Testing.

Testing

Unit test
Deployed to dev env
Other (explain)

Ran the new test_finetune integration test, along with coverage of entity CRUD methods on the client to check for regressions on the refactor

% pytest finetune test_entities.py registry/test_model.py
==================================================== test session starts ====================================================
platform darwin -- Python 3.7.10, pytest-7.4.2, pluggy-1.2.0                                                                 
rootdir: /Users/miliu/Documents/modeldb/client/verta/tests                                                                   
configfile: pytest.ini                                                                                                       
plugins: xdist-3.3.1, hypothesis-6.79.4                                                                                      
collected 78 items                                                                                                           
                                                                                                                             
finetune/test_finetune.py .                                                                                           [  1%] 
test_entities.py ...........................ss......s.s                                                               [ 50%]
registry/test_model.py .....F..........s.......s......s......s                                                        [100%]

==================================== 1 failed, 69 passed, 8 skipped in 433.59s (0:07:13) ====================================

The one failure on registry/test_model.py::TestModel::test_find is unrelated.

Reverting

Contains Migration - Do Not Revert

Revert this PR.

liuverta · 2023-09-19T00:12:47Z

client/verta/verta/client.py

-            request = conn.make_proto_request(
-                "GET", "/api/v2/uac-proxy/organization"
+            request = conn.make_proto_request("GET", "/api/v2/uac-proxy/organization")
+            response = conn.must_proto_response(
+                request, OrganizationV2_pb2.ListOrganizationsV2.Response
            )
-            response = conn.must_proto_response(request, OrganizationV2_pb2.ListOrganizationsV2.Response)


Ignore; black did this.

I mean this was bad

liuverta · 2023-09-19T00:12:50Z

client/verta/verta/client.py

-            raise ValueError("cannot provide both `organization_id` and `organization_name`")
+            raise ValueError(
+                "cannot provide both `organization_id` and `organization_name`"
+            )


Ignore; black did this.

liuverta · 2023-09-19T00:15:15Z

client/verta/verta/client.py

+    @staticmethod
+    def _get_or_create_project(


This whole file's changes are basically factoring out static versions of these get_or_create methods so that they can be called without needing to instantiate a new Client. See the absolute nightmare that is Client.__init__()—this company's oldest extant Python function—for why we want to avoid that.

Ideally these would actually be moved to new classes—which I have vague, vague plans for—but I wanted to keep this diff as readable as possible for now.

liuverta · 2023-09-19T00:17:41Z

client/verta/verta/registry/entities/_modelversion.py

-    def create_external_build(self, location: str, requires_root: Optional[bool] = None, scan_external: Optional[bool] = None, self_contained: Optional[bool] = None) -> Build:
+    def create_external_build(
+        self,
+        location: str,
+        requires_root: Optional[bool] = None,
+        scan_external: Optional[bool] = None,
+        self_contained: Optional[bool] = None,
+    ) -> Build:


Ignore; black did this.

liuverta · 2023-09-19T00:17:46Z

client/verta/verta/registry/entities/_modelversion.py

-        return Build._create_external(self._conn, self.workspace, self.id, location, requires_root, scan_external, self_contained)
+        return Build._create_external(
+            self._conn,
+            self.workspace,
+            self.id,
+            location,
+            requires_root,
+            scan_external,
+            self_contained,
+        )


Ignore; black did this.

github-actions · 2023-09-19T00:18:38Z

Common Code Coverage

Overall Project	15.01%	🍏

There is no coverage information present for the Files changed

github-actions · 2023-09-19T00:18:40Z

Backend Code Coverage

Overall Project	61.32%	🍏

There is no coverage information present for the Files changed

liuverta · 2023-09-19T18:58:46Z

client/verta/verta/client.py

+        if id is None and self._ctx.proj is None:
+            self.set_project()


This is moved here from the old lines 634–635 below, because the static method can't access self.set_project()

liuverta · 2023-09-20T23:03:38Z

client/verta/verta/registry/entities/_modelversion.py

+        :class:`~verta.registry.entities.RegisteredModelVersion`
+            New fine-tuned model version.
+
+        """


liuverta · 2023-09-21T16:52:34Z

client/verta/verta/registry/entities/_modelversion.py

+        ],
+        train_dataset: _dataset_version.DatasetVersion,
+        eval_dataset: Optional[_dataset_version.DatasetVersion] = None,
+        test_dataset: Optional[_dataset_version.DatasetVersion] = None,


Not actually sure whether we want/need test_dataset. I included it because it felt reasonable for calculating end-of-training performance, but eval_dataset might just serve the same purpose.

I say leave it 🤷🏼‍♀️

ghost · 2023-09-21T18:30:53Z

client/verta/verta/registry/entities/_modelversion.py

+        # TODO: [VRD-1131] check `enable_mdb_versioning`
+
+        ctx = _Context(self._conn, self._conf)
+        ctx.workspace_name = self.workspace


Do you know if we'll also need to handle orgs here?

I've verified that Grpc-Metadata-organization-id is faithfully sent on every request. It's encapsulated in conn.

liuverta commented Sep 19, 2023

View reviewed changes

Base automatically changed from liu/lora-config to main September 19, 2023 18:54

liuverta added 8 commits September 19, 2023 11:54

Consolidate constants

1c14266

Add fine-tuning constants

c431d3a

Get or create project

329d80b

Get or create RM

97dd8c8

Get or create experiment

cce2cff

Launch fine-tuning

25b2a78

Remove completed TODO

1640b23

Set ER name

c48f9ef

liuverta force-pushed the liu/finetune branch from 5a40e49 to c48f9ef Compare September 19, 2023 18:54

liuverta commented Sep 19, 2023

View reviewed changes

liuverta changed the title ~~feat: [VRD-1120] Add finetune()~~ feat: [VRD-1120, VRD-1130, VRD-1114] Add finetune() Sep 20, 2023

Clean up ER and RMV on error

817907c

liuverta changed the title ~~feat: [VRD-1120, VRD-1130, VRD-1114] Add finetune()~~ feat: [VRD-1120, VRD-1130, VRD-1132] Add finetune() Sep 20, 2023

liuverta added 7 commits September 20, 2023 12:25

Revamp entity names

3e5ed53

Use LoRA by default

fbb161e

Add test

0a1267d

Add docstrings

ebabcd0

Fix test and docstring

340a77d

Amend docstring

c752616

Fix typo

6698949

liuverta commented Sep 20, 2023

View reviewed changes

Make small fixes

62e229e

liuverta marked this pull request as ready for review September 20, 2023 23:07

liuverta requested review from conradoverta, a user and jared-verta September 20, 2023 23:08

liuverta commented Sep 21, 2023

View reviewed changes

Mention enable_mdb_versioning in docstring

8d726bf

ghost reviewed Sep 21, 2023

View reviewed changes

ghost approved these changes Sep 21, 2023

View reviewed changes

jared-verta approved these changes Sep 21, 2023

View reviewed changes

liuverta merged commit f462615 into main Sep 21, 2023
1 check failed

liuverta deleted the liu/finetune branch September 21, 2023 22:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: [VRD-1120, VRD-1130, VRD-1132] Add `finetune()` #4085

feat: [VRD-1120, VRD-1130, VRD-1132] Add `finetune()` #4085

liuverta commented Sep 19, 2023 •

edited

Loading

liuverta Sep 19, 2023

ghost Sep 21, 2023

liuverta Sep 19, 2023

ghost Sep 21, 2023

liuverta Sep 19, 2023 •

edited

Loading

liuverta Sep 19, 2023

liuverta Sep 19, 2023

github-actions bot commented Sep 19, 2023 •

edited

Loading

github-actions bot commented Sep 19, 2023 •

edited

Loading

liuverta Sep 19, 2023 •

edited

Loading

liuverta Sep 20, 2023 •

edited

Loading

liuverta Sep 21, 2023

ghost Sep 21, 2023

ghost Sep 21, 2023

liuverta Sep 21, 2023

liuverta Sep 21, 2023

feat: [VRD-1120, VRD-1130, VRD-1132] Add finetune() #4085

feat: [VRD-1120, VRD-1130, VRD-1132] Add finetune() #4085

Conversation

liuverta commented Sep 19, 2023 • edited Loading

Impact and Context

Risks and Area of Effect

Testing

Reverting

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuverta Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 19, 2023 • edited Loading

Common Code Coverage

github-actions bot commented Sep 19, 2023 • edited Loading

Backend Code Coverage

liuverta Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

liuverta Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: [VRD-1120, VRD-1130, VRD-1132] Add `finetune()` #4085

feat: [VRD-1120, VRD-1130, VRD-1132] Add `finetune()` #4085

liuverta commented Sep 19, 2023 •

edited

Loading

liuverta Sep 19, 2023 •

edited

Loading

github-actions bot commented Sep 19, 2023 •

edited

Loading

github-actions bot commented Sep 19, 2023 •

edited

Loading

liuverta Sep 19, 2023 •

edited

Loading

liuverta Sep 20, 2023 •

edited

Loading