Add support for using the ObjectStore pattern #78

mmmries · 2023-07-01T10:10:51Z

Coneptual documentation: https://docs.nats.io/nats-concepts/jetstream/obj_store/obj_walkthrough
ADR: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-20.md

Similar to the Key-Value functionality, this is not actually a specific thing built into the nats-server, but a documented pattern for how to store chunked files. This allows for efficient storage/streaming of larger files. I'll be modeling my changes from the ADR notes as well as the time-honored tradition of doing nats-server -js -DV and reverse engineering the logs that are generated when using the nats command-line tool for creating buckets, storing objects, retrieving objects, etc.

I'm going to start a checklist of functionality I want to cover before we merge/release this API

create a bucket
validate bucket names before creating
put a file in a bucket
overwrite a file in a bucket (cleanup prior chunks)
put a file in a bucket with TTL
get a file from a bucket (provide a callback function to accept each chunk)
delete a file from a bucket
list the files in a bucket

validate bucket names

Put a file in a bucket

add ability to list files

brandynbennett

Looking good so far!

Get object

support deleting an object from a bucket

mmmries · 2023-07-04T19:57:15Z

.github/workflows/CI.yml

@@ -18,7 +18,7 @@ jobs:
        pair:
          - otp: "24.3.4"
            elixir: "1.12"
-            nats: "2.2.0"
+            nats: "2.3.0"


NATS 2.2.0 doesn't support a PURGE with filter: "my subject" which is needed to support deleting objects from a bucket. So I had to bump this to 2.3.0

Put object ttl

mmmries · 2023-07-05T11:20:38Z

@brandynbennett this is ready for a review now. A couple of things I'm wondering about:

I tried to mostly follow the naming convention that I saw in the go client for things like list and info, but wavered for things like create_bucket and put_object. I wonder if we should move bucket operations to a separate module?
- Bucket.create, Bucket.delete, Bucket.list
- Object.put, Object.delete, Object.info
I should probably add some more type specs and docs for these functions, it would depend on the answer to my last question so didn't want to invest the time until we feel good about the API
Release schedule: Should I go ahead and make a release candidate with these changes? That way we could get the code running for an internal tool project as Spiff and have some real world validation?

mmmries · 2023-07-05T12:46:30Z

Since it's still a few hours before the US team is online, I decided to go ahead and release 0.0.8-alpha1 so I could test these changes

brandynbennett · 2023-07-05T16:14:49Z

test/jetstream/api/object_test.exs

+  # create a random 2MB binary file
+  # re-use it on subsequent test runs if it already exists
+  defp generate_big_file do
+    filepath = Path.join("tmp", "big_file.bin")


Could we us the ExUnit tmp_dir here?

brandynbennett · 2023-07-05T16:17:13Z

.gitignore

@@ -24,3 +24,6 @@ jetstream-*.tar

 # ASDF tool versions
 .tool-versions
+
+# tmp directory to test files
+tmp


Could we us the ExUnit tmp_dir here?

brandynbennett · 2023-07-05T16:23:19Z

lib/jetstream/api/stream.ex

+      :ok
+
+  """
+  @spec purge(conn :: Gnat.t(), stream_name :: binary()) :: :ok | {:error, any()}


spec is missing method

brandynbennett · 2023-07-05T16:27:42Z

lib/jetstream/api/object.ex

+  @stream_prefix "OBJ_"
+  @subject_prefix "$O."
+
+  def create_bucket(conn, bucket_name, params \\ []) do


I could see the case for making an Object.Bucket module to separate responsibilities. I think from an interface perspective I would still want to delegate so calling Object.create_bucket is the public interface.

👍 I can get behind that. I'll keep create_bucket and delete_bucket as-is. And Then change the rest of the functions to skip the _object since that focus of this API is to work with the files in a bucket.

mmmries · 2023-07-06T13:36:43Z

I temporarily installed dialyxir into this project locally to validate that my new typespecs are all valid. When I did that, I found some errors from existing parts of this library. So I think we should probably start up a separate PR to add dialyxir to this project and run it in CI to keep ourselves honest with all of our typespecs.

brandynbennett

Good idea about Dialyxir pr

start support for object store

6b279cb

mmmries force-pushed the object-store branch from a80f3ef to 6b279cb Compare July 1, 2023 10:12

mmmries requested a review from brandynbennett July 2, 2023 16:08

mmmries added 6 commits July 2, 2023 18:11

validate bucket names

2505232

Merge pull request #80 from mmmries/validate_bucket_name

88bd2f4

validate bucket names

happy path for adding a file to the object store

b247390

add error test

5e14636

Merge pull request #79 from mmmries/obj-put

348e876

Put a file in a bucket

add ability to list files

bb1cd11

mmmries mentioned this pull request Jul 3, 2023

add ability to list files #81

Merged

mmmries added 4 commits July 3, 2023 16:39

stash

c9eaac1

add info/3 to get details about an object

251124b

Merge pull request #81 from mmmries/list_files

fd6cc43

add ability to list files

stash

16a3a97

brandynbennett reviewed Jul 3, 2023

View reviewed changes

mmmries added 5 commits July 4, 2023 15:16

support retrieving files from an object store

eb0bdaa

Merge pull request #82 from mmmries/get_object

707c87a

Get object

support deleting an object from a bucket

8d72c61

Merge pull request #83 from mmmries/del_object

4d20b0e

support deleting an object from a bucket

update NATS supported version

7e4e33b

mmmries commented Jul 4, 2023

View reviewed changes

mmmries added 4 commits July 5, 2023 11:48

overwriting a file

b60a520

add test for buckets with a TTL

b3b39f7

Merge pull request #84 from mmmries/put_object_ttl

e6473d0

Put object ttl

mix format

b0b3531

mmmries marked this pull request as ready for review July 5, 2023 11:16

brandynbennett reviewed Jul 5, 2023

View reviewed changes

simplify interface function names

a88135e

mmmries added 3 commits July 6, 2023 13:51

add typespecs

0183273

use exunit tmp dir

4488a0a

mix format

820493f

brandynbennett approved these changes Jul 6, 2023

View reviewed changes

mmmries merged commit 30ddfb5 into master Jul 6, 2023
2 checks passed

mmmries deleted the object-store branch July 6, 2023 17:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for using the ObjectStore pattern #78

Add support for using the ObjectStore pattern #78

mmmries commented Jul 1, 2023 •

edited

Loading

brandynbennett left a comment

mmmries Jul 4, 2023

mmmries commented Jul 5, 2023

mmmries commented Jul 5, 2023

brandynbennett Jul 5, 2023

brandynbennett Jul 5, 2023

mmmries Jul 6, 2023

brandynbennett Jul 5, 2023

mmmries Jul 6, 2023

brandynbennett Jul 5, 2023

mmmries Jul 6, 2023

mmmries commented Jul 6, 2023

brandynbennett left a comment

Add support for using the ObjectStore pattern #78

Add support for using the ObjectStore pattern #78

Conversation

mmmries commented Jul 1, 2023 • edited Loading

brandynbennett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmmries commented Jul 5, 2023

mmmries commented Jul 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmmries commented Jul 6, 2023

brandynbennett left a comment

Choose a reason for hiding this comment

mmmries commented Jul 1, 2023 •

edited

Loading