Apply suggestions from code review

Co-authored-by: Erin Cochran <erin.k.cochran@gmail.com>
dagster-io · Aug 10, 2024 · a9842e4 · a9842e4
1 parent 65e8da0
commit a9842e4
Show file tree

Hide file tree

Showing 7 changed files with 15 additions and 9 deletions.
diff --git a/docs/content/api/modules.json.gz b/docs/content/api/modules.json.gz
diff --git a/docs/content/api/searchindex.json.gz b/docs/content/api/searchindex.json.gz
diff --git a/docs/content/api/sections.json.gz b/docs/content/api/sections.json.gz
diff --git a/docs/content/concepts/dagster-pipes/aws-glue.mdx b/docs/content/concepts/dagster-pipes/aws-glue.mdx
@@ -7,7 +7,9 @@ description: "Learn to integrate Dagster Pipes with AWS Glue to launch external
 
 This tutorial gives a short overview on how to use [Dagster Pipes](/concepts/dagster-pipes) with [AWS Glue](https://aws.amazon.com/glue/).
 
-## The [dagster-aws](/\_apidocs/libraries/dagster-aws) integration library provides the <PyObject object="PipesGlueClient" module="dagster_aws.pipes" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
+The [dagster-aws](/\_apidocs/libraries/dagster-aws) integration library provides the <PyObject object="PipesGlueClient" module="dagster_aws.pipes" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
+
+---
 
 ## Prerequisites
 
@@ -129,7 +131,9 @@ defs = Definitions(
 
 Dagster will now be able to launch the AWS Glue job from the `glue_pipes_asset` asset.
 
-By default the client will be using the CloudWatch log stream (`.../output/<job-run-id>`) created by the Glue job to receive Dagster events and will forward this stream to `stdout`. If this is undesired, instead, the client can be configured to use <PyObject object="PipesS3MessageReader" module="dagster_aws.pipes" />, and the Glue job can use <PyObject object="PipesS3MessageWriter" module="dagster_pipes" /> .
+By default, the client uses the CloudWatch log stream (`.../output/<job-run-id>`) created by the Glue job to receive Dagster events. The client will also forward the stream to `stdout`.
+
+To customize this behavior, the client can be configured to use <PyObject object="PipesS3MessageReader" module="dagster_aws.pipes" />, and the Glue job to use <PyObject object="PipesS3MessageWriter" module="dagster_pipes" /> .
 
 ---
 

diff --git a/docs/next/public/objects.inv b/docs/next/public/objects.inv
diff --git a/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst b/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst
@@ -119,6 +119,7 @@ Clients
 .. autoclass:: dagster_aws.pipes.PipesLambdaClient
 
 .. autoclass:: dagster_aws.pipes.PipesGlueClient
+   :noindex:
    :members: run
 
 Legacy

diff --git a/python_modules/libraries/dagster-aws/dagster_aws/pipes.py b/python_modules/libraries/dagster-aws/dagster_aws/pipes.py
@@ -23,7 +23,7 @@
 import dagster._check as check
 from botocore.exceptions import ClientError
 from dagster import PipesClient
-from dagster._annotations import experimental
+from dagster._annotations import experimental, public
 from dagster._core.definitions.resource_annotation import TreatAsResourceParam
 from dagster._core.errors import DagsterExecutionInterruptedError
 from dagster._core.execution.context.compute import OpExecutionContext
@@ -211,11 +211,10 @@ def consume_cloudwatch_logs(
             log_group (str): CloudWatch log group name
             log_stream (str): CLoudWatch log stream name
             start_time (Optional[int]): The start of the time range, expressed as the number of
-                milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to this
-                time or later than this time are included.
-                Events with a timestamp earlier than this time are not included.
+                milliseconds after ``Jan 1, 1970 00:00:00 UTC``. Only events with a timestamp equal to this
+                time or later are included.
             end_time (Optional[int]): The end of the time range, expressed as the number of
-                milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to or
+                milliseconds after ``Jan 1, 1970 00:00:00 UTC``. Events with a timestamp equal to or
                 later than this time are not included.
         """
         handler = check.not_none(
@@ -297,6 +296,7 @@ def __init__(
     def _is_dagster_maintained(cls) -> bool:
         return True
 
+    @public
     def run(
         self,
         *,
@@ -362,9 +362,9 @@ class PipesGlueClient(PipesClient, TreatAsResourceParam):
         message_reader (Optional[PipesMessageReader]): A message reader to use to read messages
             from the glue job run. Defaults to :py:class:`PipesCloudWatchsMessageReader`.
             When provided with :py:class:`PipesCloudWatchMessageReader`,
-            it will be used to recieve logs and events from the `.../output/<job-run-id>`
+            it will be used to recieve logs and events from the ``.../output/<job-run-id>``
             CloudWatch log stream created by AWS Glue. Note that AWS Glue routes both
-            `stderr` and `stdout` from the main job process into this LogStream.
+            ``stderr`` and ``stdout`` from the main job process into this LogStream.
         client (Optional[boto3.client]): The boto Glue client used to launch the Glue job
         forward_termination (bool): Whether to cancel the Glue job run when the Dagster process receives a termination signal.
     """
@@ -385,6 +385,7 @@ def __init__(
     def _is_dagster_maintained(cls) -> bool:
         return True
 
+    @public
     def run(
         self,
         *,