From 689299c68e1fde29fc2c9ce72579e399aa737c1e Mon Sep 17 00:00:00 2001 From: David Gardner Date: Tue, 14 Nov 2023 09:33:21 -0800 Subject: [PATCH 01/19] Replace stub stages document with links to the stage API --- docs/source/developer_guide/guides.md | 4 +- docs/source/index.rst | 2 +- docs/source/stages/morpheus_stages.md | 81 --------------------------- 3 files changed, 3 insertions(+), 84 deletions(-) delete mode 100644 docs/source/stages/morpheus_stages.md diff --git a/docs/source/developer_guide/guides.md b/docs/source/developer_guide/guides.md index 45d0c0c228..0a3ca52313 100644 --- a/docs/source/developer_guide/guides.md +++ b/docs/source/developer_guide/guides.md @@ -20,9 +20,9 @@ limitations under the License. ## Morpheus Stages Morpheus includes a number of pre-defined stage implementations to choose from when building a custom -pipeline, each of which can be included and configured to suit your application. +pipeline, each of which can be included and configured to suit your application. The stages included in Morpheus can be found in the stages API reference: -- [List of available Morpheus stages](../stages/morpheus_stages.md) +- [Morpheus stages API](../_modules/morpheus.stages) There are likely going to be situations that require writing a custom stage. Morpheus stages are written in Python and optionally may include a C++ implementation. The following guides outline how to create your own stages diff --git a/docs/source/index.rst b/docs/source/index.rst index e400523eb6..341e8b025e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -110,7 +110,7 @@ Deploying Morpheus :maxdepth: 20 :hidden: - stages/morpheus_stages + _modules/morpheus.stages .. toctree:: :caption: Morpheus Modules diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md deleted file mode 100644 index 44821c352a..0000000000 --- a/docs/source/stages/morpheus_stages.md +++ /dev/null @@ -1,81 +0,0 @@ - - -# Stages Documentation - -## Boundary - -- Linear Boundary Stage - -## General - -- Buffer Stage -- Delay Stage -- Linear Modules Stage -- Monitor Stage -- Multi Port Module Stage -- Trigger Stage - -## Inference - -- Auto Encoder Inference Stage -- Identity Inference Stage -- Inference Stage -- PyTorch Inference Stage -- Triton Inference Stage - -## Input - -- AppShield Source Stage -- Autoencoder Source Stage -- Azure Source Stage -- Cloud Trail Source Stage -- Control Message File Source Stage -- Control Message Kafka Source Stage -- Duo Source Stage -- File Source Stage -- Kafka Source Stage - -## Output - -- Write To File Stage -- Write To Kafka Stage - -## Post-process - -- Add Classifications Stage -- Add Scores Stage -- Filter Detections Stage -- Generate Viz Frames Stage -- ML Flow Drift Stage -- Serialize Stage -- Timeseries Stage -- Validation Stage - -## Pre-process - -- Deserialize Stage -- Drop Null Stage -- Preprocess AE Stage -- Preprocess Base Stage -- Preprocess FIL Stage -- Preprocess NLP Stage -- Train AE Stage - -## Training - -- Training Stage From 609301b07d44df724efc4dac5abd20220e125ee8 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Tue, 14 Nov 2023 10:56:12 -0800 Subject: [PATCH 02/19] The top-level README should direct users to the examples dir as this is the entry-point for github viewers --- README.md | 2 +- examples/README.md | 16 +++++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 6a5c374cb5..945d9eb112 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ NVIDIA Morpheus is an open AI application framework that provides cybersecurity * [Getting Started with Morpheus](./docs/source/getting_started.md) - Using pre-built Docker containers, building Docker containers from source, and fetching models and datasets * [Morpheus CLI Overview](./docs/source/basics/overview.rst) - Brief overview of the `morpheus` command line interface * [Building a Pipeline](./docs/source/basics/building_a_pipeline.md) - Introduction to building a pipeline using the command line interface -* [Morpheus Examples](./docs/source/examples.md) - Example pipelines using both the Python API and command line interface +* [Morpheus Examples](./examples/README.md) - Example pipelines using both the Python API and command line interface * [Pre-built Models and Datasets](./models/README.md) - Pretrained models with corresponding training, validation scripts, and datasets * [Developer Guides](./docs/source/developer_guide/guides.md) - Covers extending Morpheus with custom stages diff --git a/examples/README.md b/examples/README.md index c80d88ee9f..47dd3adf15 100644 --- a/examples/README.md +++ b/examples/README.md @@ -15,10 +15,12 @@ See the License for the specific language governing permissions and limitations under the License. --> -## Morpheus CLI Examples - -Examples run with the Morpheus CLI (`morpheus ...`) should be run from the repository root; otherwise, some filepath arguments may need to be changed. - -## Morpheus run.py Examples - -Examples run with python (`python run.py`) should be run from the example's directory; otherwise, relative Python imports may be broken. +# Examples +* [Anomalous Behavior Profiling with Forest Inference Library (FIL) Example](./abp_nvsmi_detection/README.md) +* [ABP Detection Example Using Morpheus](./abp_pcap_detection/README.md) +* [GNN Fraud Detection Pipeline](./gnn_fraud_detection_pipeline/README.md) +* [Example cyBERT Morpheus Pipeline for Apache Log Parsing](./log_parsing/README.md) +* [Sensitive Information Detection with Natural Language Processing (NLP) Example](./nlp_si_detection/README.md) +* [Example Ransomware Detection Morpheus Pipeline for AppShield Data](./ransomware_detection/README.md) +* [Root Cause Analysis Acceleration & Predictive Maintenance Example](./root_cause_analysis/README.md) +* [SID Visualization Example](./sid_visualization/README.md) From fff7c63e560215fade01b76200219c95c556dac0 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Tue, 14 Nov 2023 13:32:56 -0800 Subject: [PATCH 03/19] Revert "The top-level README should direct users to the examples dir as this is the entry-point for github viewers" This reverts commit 609301b07d44df724efc4dac5abd20220e125ee8. --- README.md | 2 +- examples/README.md | 16 +++++++--------- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 945d9eb112..6a5c374cb5 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ NVIDIA Morpheus is an open AI application framework that provides cybersecurity * [Getting Started with Morpheus](./docs/source/getting_started.md) - Using pre-built Docker containers, building Docker containers from source, and fetching models and datasets * [Morpheus CLI Overview](./docs/source/basics/overview.rst) - Brief overview of the `morpheus` command line interface * [Building a Pipeline](./docs/source/basics/building_a_pipeline.md) - Introduction to building a pipeline using the command line interface -* [Morpheus Examples](./examples/README.md) - Example pipelines using both the Python API and command line interface +* [Morpheus Examples](./docs/source/examples.md) - Example pipelines using both the Python API and command line interface * [Pre-built Models and Datasets](./models/README.md) - Pretrained models with corresponding training, validation scripts, and datasets * [Developer Guides](./docs/source/developer_guide/guides.md) - Covers extending Morpheus with custom stages diff --git a/examples/README.md b/examples/README.md index 47dd3adf15..c80d88ee9f 100644 --- a/examples/README.md +++ b/examples/README.md @@ -15,12 +15,10 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Examples -* [Anomalous Behavior Profiling with Forest Inference Library (FIL) Example](./abp_nvsmi_detection/README.md) -* [ABP Detection Example Using Morpheus](./abp_pcap_detection/README.md) -* [GNN Fraud Detection Pipeline](./gnn_fraud_detection_pipeline/README.md) -* [Example cyBERT Morpheus Pipeline for Apache Log Parsing](./log_parsing/README.md) -* [Sensitive Information Detection with Natural Language Processing (NLP) Example](./nlp_si_detection/README.md) -* [Example Ransomware Detection Morpheus Pipeline for AppShield Data](./ransomware_detection/README.md) -* [Root Cause Analysis Acceleration & Predictive Maintenance Example](./root_cause_analysis/README.md) -* [SID Visualization Example](./sid_visualization/README.md) +## Morpheus CLI Examples + +Examples run with the Morpheus CLI (`morpheus ...`) should be run from the repository root; otherwise, some filepath arguments may need to be changed. + +## Morpheus run.py Examples + +Examples run with python (`python run.py`) should be run from the example's directory; otherwise, relative Python imports may be broken. From 1b4f7dc322048b3db14b5073c8105fcbefc76734 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 08:22:38 -0800 Subject: [PATCH 04/19] Revert "Replace stub stages document with links to the stage API" This reverts commit 689299c68e1fde29fc2c9ce72579e399aa737c1e. --- docs/source/developer_guide/guides.md | 4 +- docs/source/index.rst | 2 +- docs/source/stages/morpheus_stages.md | 81 +++++++++++++++++++++++++++ 3 files changed, 84 insertions(+), 3 deletions(-) create mode 100644 docs/source/stages/morpheus_stages.md diff --git a/docs/source/developer_guide/guides.md b/docs/source/developer_guide/guides.md index 0a3ca52313..45d0c0c228 100644 --- a/docs/source/developer_guide/guides.md +++ b/docs/source/developer_guide/guides.md @@ -20,9 +20,9 @@ limitations under the License. ## Morpheus Stages Morpheus includes a number of pre-defined stage implementations to choose from when building a custom -pipeline, each of which can be included and configured to suit your application. The stages included in Morpheus can be found in the stages API reference: +pipeline, each of which can be included and configured to suit your application. -- [Morpheus stages API](../_modules/morpheus.stages) +- [List of available Morpheus stages](../stages/morpheus_stages.md) There are likely going to be situations that require writing a custom stage. Morpheus stages are written in Python and optionally may include a C++ implementation. The following guides outline how to create your own stages diff --git a/docs/source/index.rst b/docs/source/index.rst index 341e8b025e..e400523eb6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -110,7 +110,7 @@ Deploying Morpheus :maxdepth: 20 :hidden: - _modules/morpheus.stages + stages/morpheus_stages .. toctree:: :caption: Morpheus Modules diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md new file mode 100644 index 0000000000..44821c352a --- /dev/null +++ b/docs/source/stages/morpheus_stages.md @@ -0,0 +1,81 @@ + + +# Stages Documentation + +## Boundary + +- Linear Boundary Stage + +## General + +- Buffer Stage +- Delay Stage +- Linear Modules Stage +- Monitor Stage +- Multi Port Module Stage +- Trigger Stage + +## Inference + +- Auto Encoder Inference Stage +- Identity Inference Stage +- Inference Stage +- PyTorch Inference Stage +- Triton Inference Stage + +## Input + +- AppShield Source Stage +- Autoencoder Source Stage +- Azure Source Stage +- Cloud Trail Source Stage +- Control Message File Source Stage +- Control Message Kafka Source Stage +- Duo Source Stage +- File Source Stage +- Kafka Source Stage + +## Output + +- Write To File Stage +- Write To Kafka Stage + +## Post-process + +- Add Classifications Stage +- Add Scores Stage +- Filter Detections Stage +- Generate Viz Frames Stage +- ML Flow Drift Stage +- Serialize Stage +- Timeseries Stage +- Validation Stage + +## Pre-process + +- Deserialize Stage +- Drop Null Stage +- Preprocess AE Stage +- Preprocess Base Stage +- Preprocess FIL Stage +- Preprocess NLP Stage +- Train AE Stage + +## Training + +- Training Stage From 30fef3f052af37065262dcd5a8dfe81e59b4f060 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 10:10:28 -0800 Subject: [PATCH 05/19] WIP: Add sphinx API links [no ci] --- docs/source/stages/morpheus_stages.md | 90 ++++++++++++++------------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 44821c352a..77f454d171 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -19,63 +19,67 @@ limitations under the License. ## Boundary -- Linear Boundary Stage +- Linear Boundary Egress Stage {py:class}`~morpheus.stages.boundary.linear_boundary_stage.LinearBoundaryEgressStage` +- Linear Boundary Ingress Stage {py:class}`~morpheus.stages.boundary.linear_boundary_stage.LinearBoundaryIngressStage` + +## Doca + +- Doca Stage {py:class}`~morpheus.stages.doca.doca_source_stage.DocaSourceStage` ## General -- Buffer Stage -- Delay Stage -- Linear Modules Stage -- Monitor Stage -- Multi Port Module Stage -- Trigger Stage +- Linear Modules Stage {py:class}`~morpheus.stages.general.linear_modules_stage.LinearModulesStage` +- Monitor Stage {py:class}`~morpheus.stages.general.monitor_stage.MonitorStage` +- Multi Port Module Stage {py:class}`~morpheus.stages.general.multi_port_modules_stage.MultiPortModulesStage` +- Trigger Stage {py:class}`~morpheus.stages.general.trigger_stage.TriggerStage` ## Inference -- Auto Encoder Inference Stage -- Identity Inference Stage -- Inference Stage -- PyTorch Inference Stage -- Triton Inference Stage +- Auto Encoder Inference Stage {py:class}`~morpheus.stages.inference.auto_encoder_inference_stage.AutoEncoderInferenceStage` +- PyTorch Inference Stage {py:class}`~morpheus.stages.inference.pytorch_inference_stage.PyTorchInferenceStage` +- Triton Inference Stage {py:class}`~morpheus.stages.inference.triton_inference_stage.TritonInferenceStage` ## Input -- AppShield Source Stage -- Autoencoder Source Stage -- Azure Source Stage -- Cloud Trail Source Stage -- Control Message File Source Stage -- Control Message Kafka Source Stage -- Duo Source Stage -- File Source Stage -- Kafka Source Stage +- AppShield Source Stage {py:class}`~morpheus.stages.input.appshield_source_stage.AppShieldSourceStage` +- Azure Source Stage {py:class}`~morpheus.stages.input.azure_source_stage.AzureSourceStage` +- Cloud Trail Source Stage {py:class}`~morpheus.stages.input.cloud_trail_source_stage.CloudTrailSourceStage` +- Control Message File Source Stage {py:class}`~morpheus.stages.input.control_message_file_source_stage.ControlMessageFileSourceStage` +- Control Message Kafka Source Stage {py:class}`~morpheus.stages.input.control_message_kafka_source_stage.ControlMessageKafkaSourceStage` +- Databricks Delta Lake Source Stage {py:class}`~morpheus.stages.input.databricks_deltalake_source_stage.DataBricksDeltaLakeSourceStage` +- Duo Source Stage {py:class}`~morpheus.stages.input.duo_source_stage.DuoSourceStage` +- File Source Stage {py:class}`~morpheus.stages.input.file_source_stage.FileSourceStage` +- HTTP Client Source Stage {py:class}`morpheus.stages.input.http_client_source_stage.HttpClientSourceStage` +- HTTP Server Source Stage {py:class}`morpheus.stages.input.http_server_source_stage.HttpServerSourceStage` +- In Memory Source Stage {py:class}`morpheus.stages.input.in_memory_source_stage.InMemorySourceStage` +- Kafka Source Stage {py:class}`~morpheus.stages.input.kafka_source_stage.KafkaSourceStage` +- RSS Source Stage {py:class}`morpheus.stages.input.rss_source_stage.RSSSourceStage` ## Output - -- Write To File Stage -- Write To Kafka Stage +- HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` +- HTTP Server Sink Stage {py:class}`~morpheus.stages.output.http_server_sink_stage.HttpServerSinkStage` +- In Memory Sink Stage {py:class}`~morpheus.stages.output.in_memory_sink_stage.InMemorySinkStage` +- Databricks Delta Lake Sink Stage {py:class}`~morpheus.stages.output.write_to_databricks_deltalake_stage.DataBricksDeltaLakeSinkStage` +- Write To Elastic Search Stage {py:class}`~morpheus.stages.output.write_to_elasticsearch_stage.WriteToElasticsearchStage` +- Write To File Stage {py:class}`~morpheus.stages.output.write_to_file_stage.WriteToFileStage` +- Write To Kafka Stage {py:class}`~morpheus.stages.output.write_to_kafka_stage.WriteToKafkaStage` +- Write To Vector DB Stage {py:class}`~morpheus.stages.output.write_to_vector_db.WriteToVectorDBStage` ## Post-process -- Add Classifications Stage -- Add Scores Stage -- Filter Detections Stage -- Generate Viz Frames Stage -- ML Flow Drift Stage -- Serialize Stage -- Timeseries Stage -- Validation Stage +- Add Classifications Stage {py:class}`~morpheus.stages.postprocess.add_classifications_stage.AddClassificationsStage` +- Add Scores Stage {py:class}`~morpheus.stages.postprocess.add_scores_stage.AddScoresStage` +- Filter Detections Stage {py:class}`~morpheus.stages.postprocess.filter_detections_stage.FilterDetectionsStage` +- Generate Viz Frames Stage {py:class}`~morpheus.stages.postprocess.generate_viz_frames_stage.GenerateVizFramesStage` +- ML Flow Drift Stage {py:class}`~morpheus.stages.postprocess.ml_flow_drift_stage.MLFlowDriftStage` +- Serialize Stage {py:class}`~morpheus.stages.postprocess.serialize_stage.SerializeStage` +- Timeseries Stage {py:class}`~morpheus.stages.postprocess.timeseries_stage.TimeSeriesStage` ## Pre-process -- Deserialize Stage -- Drop Null Stage -- Preprocess AE Stage -- Preprocess Base Stage -- Preprocess FIL Stage -- Preprocess NLP Stage -- Train AE Stage - -## Training - -- Training Stage +- Deserialize Stage {py:class}`~morpheus.stages.preprocess.deserialize_stage.DeserializeStage` +- Drop Null Stage {py:class}`~morpheus.stages.preprocess.drop_null_stage.DropNullStage` +- Preprocess AE Stage {py:class}`~morpheus.stages.preprocess.preprocess_ae_stage.PreprocessAEStage` +- Preprocess FIL Stage {py:class}`~morpheus.stages.preprocess.preprocess_fil_stage.PreprocessFILStage` +- Preprocess NLP Stage {py:class}`~morpheus.stages.preprocess.preprocess_nlp_stage.PreprocessNLPStage` +- Train AE Stage {py:class}`~morpheus.stages.preprocess.train_ae_stage.TrainAEStage` From 8816295cf8d0b1d18218e8f72db2b7d67780b313 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 10:14:17 -0800 Subject: [PATCH 06/19] WIP: fix some sphinx API links [no ci] --- docs/source/stages/morpheus_stages.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 77f454d171..dbe4dd0111 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -49,11 +49,11 @@ limitations under the License. - Databricks Delta Lake Source Stage {py:class}`~morpheus.stages.input.databricks_deltalake_source_stage.DataBricksDeltaLakeSourceStage` - Duo Source Stage {py:class}`~morpheus.stages.input.duo_source_stage.DuoSourceStage` - File Source Stage {py:class}`~morpheus.stages.input.file_source_stage.FileSourceStage` -- HTTP Client Source Stage {py:class}`morpheus.stages.input.http_client_source_stage.HttpClientSourceStage` -- HTTP Server Source Stage {py:class}`morpheus.stages.input.http_server_source_stage.HttpServerSourceStage` -- In Memory Source Stage {py:class}`morpheus.stages.input.in_memory_source_stage.InMemorySourceStage` +- HTTP Client Source Stage {py:class}`~morpheus.stages.input.http_client_source_stage.HttpClientSourceStage` +- HTTP Server Source Stage {py:class}`~morpheus.stages.input.http_server_source_stage.HttpServerSourceStage` +- In Memory Source Stage {py:class}`~morpheus.stages.input.in_memory_source_stage.InMemorySourceStage` - Kafka Source Stage {py:class}`~morpheus.stages.input.kafka_source_stage.KafkaSourceStage` -- RSS Source Stage {py:class}`morpheus.stages.input.rss_source_stage.RSSSourceStage` +- RSS Source Stage {py:class}`~morpheus.stages.input.rss_source_stage.RSSSourceStage` ## Output - HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` From 72481be252edeaae9eadb318729845317f4d8d8f Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 10:51:11 -0800 Subject: [PATCH 07/19] WIP [no ci] --- docs/source/examples/index.rst | 1 + docs/source/stages/morpheus_stages.md | 7 ++----- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/examples/index.rst b/docs/source/examples/index.rst index 94fc694655..a7308451a8 100644 --- a/docs/source/examples/index.rst +++ b/docs/source/examples/index.rst @@ -24,6 +24,7 @@ Examples abp_nvsmi_detection/README.md abp_pcap_detection/README.md + doca/README.md gnn_fraud_detection_pipeline/README.md log_parsing/README.md nlp_si_detection/README.md diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index dbe4dd0111..56d57df4bb 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -17,14 +17,11 @@ limitations under the License. # Stages Documentation -## Boundary - -- Linear Boundary Egress Stage {py:class}`~morpheus.stages.boundary.linear_boundary_stage.LinearBoundaryEgressStage` -- Linear Boundary Ingress Stage {py:class}`~morpheus.stages.boundary.linear_boundary_stage.LinearBoundaryIngressStage` +Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API {py:mod}`morpheus.stages`. ## Doca -- Doca Stage {py:class}`~morpheus.stages.doca.doca_source_stage.DocaSourceStage` +- Doca Stage {py:class}`~morpheus.stages.doca.doca_source_stage.DocaSourceStage` A source stage used to receive raw packet data from a ConnectX-6 Dx NIC. This stage is not compiled by default refer to the [Doca Example](../../../examples/doca/README.md) for details on building this stage. ## General From d46759ed0a84b7f16766cc617e167d2a60d15685 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 10:57:12 -0800 Subject: [PATCH 08/19] wip [no ci] --- docs/source/examples/doca/README.md | 1 + docs/source/stages/morpheus_stages.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 120000 docs/source/examples/doca/README.md diff --git a/docs/source/examples/doca/README.md b/docs/source/examples/doca/README.md new file mode 120000 index 0000000000..c3e43df7d9 --- /dev/null +++ b/docs/source/examples/doca/README.md @@ -0,0 +1 @@ +../../../../examples/doca/README.md \ No newline at end of file diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 56d57df4bb..7dccc439fb 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -25,7 +25,7 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos ## General -- Linear Modules Stage {py:class}`~morpheus.stages.general.linear_modules_stage.LinearModulesStage` +- Linear Modules Stage {py:class}`~morpheus.stages.general.linear_modules_stage.LinearModulesStage` Loads an existing, registered, module and wraps it as a Morpheus stage. Refer to [Morpheus Modules](../developer_guide/guides.md#morpheus-modules) for details on modules. - Monitor Stage {py:class}`~morpheus.stages.general.monitor_stage.MonitorStage` - Multi Port Module Stage {py:class}`~morpheus.stages.general.multi_port_modules_stage.MultiPortModulesStage` - Trigger Stage {py:class}`~morpheus.stages.general.trigger_stage.TriggerStage` From 5cce51cc7796e471e5d0be499f2788b5fc0c6b23 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:13:52 -0800 Subject: [PATCH 09/19] wip [no ci] --- docs/source/stages/morpheus_stages.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 7dccc439fb..0fa4c18e7e 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -26,15 +26,15 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos ## General - Linear Modules Stage {py:class}`~morpheus.stages.general.linear_modules_stage.LinearModulesStage` Loads an existing, registered, module and wraps it as a Morpheus stage. Refer to [Morpheus Modules](../developer_guide/guides.md#morpheus-modules) for details on modules. -- Monitor Stage {py:class}`~morpheus.stages.general.monitor_stage.MonitorStage` -- Multi Port Module Stage {py:class}`~morpheus.stages.general.multi_port_modules_stage.MultiPortModulesStage` -- Trigger Stage {py:class}`~morpheus.stages.general.trigger_stage.TriggerStage` +- Monitor Stage {py:class}`~morpheus.stages.general.monitor_stage.MonitorStage` Display throughput numbers at a specific point in the pipeline. +- Multi Port Module Stage {py:class}`~morpheus.stages.general.multi_port_modules_stage.MultiPortModulesStage` Loads an existing, registered, multi-port module and wraps it as a multi-port Morpheus stage. Refer to [Morpheus Modules](../developer_guide/guides.md#morpheus-modules) for details on modules. +- Trigger Stage {py:class}`~morpheus.stages.general.trigger_stage.TriggerStage` Buffer data until the previous stage has completed, useful for testing performance of one stage at a time. ## Inference -- Auto Encoder Inference Stage {py:class}`~morpheus.stages.inference.auto_encoder_inference_stage.AutoEncoderInferenceStage` -- PyTorch Inference Stage {py:class}`~morpheus.stages.inference.pytorch_inference_stage.PyTorchInferenceStage` -- Triton Inference Stage {py:class}`~morpheus.stages.inference.triton_inference_stage.TritonInferenceStage` +- Auto Encoder Inference Stage {py:class}`~morpheus.stages.inference.auto_encoder_inference_stage.AutoEncoderInferenceStage` PyTorch inference stage used for Auto Encoder pipeline mode. +- PyTorch Inference Stage {py:class}`~morpheus.stages.inference.pytorch_inference_stage.PyTorchInferenceStage` PyTorch inference stage used for most pipeline modes with the exception of Auto Encoder. +- Triton Inference Stage {py:class}`~morpheus.stages.inference.triton_inference_stage.TritonInferenceStage` Inference stage which utilizes a [Triton Inference Server](https://developer.nvidia.com/nvidia-triton-inference-server). ## Input From a2544374e00ff1fb3a2c7c812e3110502cc12a82 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:36:18 -0800 Subject: [PATCH 10/19] Created dataframes are actually CuDF dataframes --- morpheus/stages/input/rss_source_stage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/morpheus/stages/input/rss_source_stage.py b/morpheus/stages/input/rss_source_stage.py index e02061f778..6d517eb00f 100644 --- a/morpheus/stages/input/rss_source_stage.py +++ b/morpheus/stages/input/rss_source_stage.py @@ -31,7 +31,7 @@ @register_stage("from-rss") class RSSSourceStage(PreallocatorMixin, SingleOutputSource): """ - Load RSS feed items into a pandas DataFrame. + Load RSS feed items into a DataFrame. Parameters ---------- From 07aa6d657e939bf2d25078932cb6703fd15b6c9b Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:37:03 -0800 Subject: [PATCH 11/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 0fa4c18e7e..79166155bc 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -38,19 +38,19 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos ## Input -- AppShield Source Stage {py:class}`~morpheus.stages.input.appshield_source_stage.AppShieldSourceStage` -- Azure Source Stage {py:class}`~morpheus.stages.input.azure_source_stage.AzureSourceStage` -- Cloud Trail Source Stage {py:class}`~morpheus.stages.input.cloud_trail_source_stage.CloudTrailSourceStage` -- Control Message File Source Stage {py:class}`~morpheus.stages.input.control_message_file_source_stage.ControlMessageFileSourceStage` -- Control Message Kafka Source Stage {py:class}`~morpheus.stages.input.control_message_kafka_source_stage.ControlMessageKafkaSourceStage` -- Databricks Delta Lake Source Stage {py:class}`~morpheus.stages.input.databricks_deltalake_source_stage.DataBricksDeltaLakeSourceStage` -- Duo Source Stage {py:class}`~morpheus.stages.input.duo_source_stage.DuoSourceStage` -- File Source Stage {py:class}`~morpheus.stages.input.file_source_stage.FileSourceStage` -- HTTP Client Source Stage {py:class}`~morpheus.stages.input.http_client_source_stage.HttpClientSourceStage` -- HTTP Server Source Stage {py:class}`~morpheus.stages.input.http_server_source_stage.HttpServerSourceStage` -- In Memory Source Stage {py:class}`~morpheus.stages.input.in_memory_source_stage.InMemorySourceStage` -- Kafka Source Stage {py:class}`~morpheus.stages.input.kafka_source_stage.KafkaSourceStage` -- RSS Source Stage {py:class}`~morpheus.stages.input.rss_source_stage.RSSSourceStage` +- AppShield Source Stage {py:class}`~morpheus.stages.input.appshield_source_stage.AppShieldSourceStage` Load Appshield messages from one or more plugins into a dataframe. +- Azure Source Stage {py:class}`~morpheus.stages.input.azure_source_stage.AzureSourceStage` Load Azure Active Directory messages. +- Cloud Trail Source Stage {py:class}`~morpheus.stages.input.cloud_trail_source_stage.CloudTrailSourceStage` Load messages from a Cloudtrail directory. +- Control Message File Source Stage {py:class}`~morpheus.stages.input.control_message_file_source_stage.ControlMessageFileSourceStage` Recieves control messages from different sources specified by a list of (fsspec)[https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=open_files#fsspec.open_files] strings. +- Control Message Kafka Source Stage {py:class}`~morpheus.stages.input.control_message_kafka_source_stage.ControlMessageKafkaSourceStage` Load control messages from a Kafka cluster. +- Databricks Delta Lake Source Stage {py:class}`~morpheus.stages.input.databricks_deltalake_source_stage.DataBricksDeltaLakeSourceStage` Source stage used to load messages from a DeltaLake table. +- Duo Source Stage {py:class}`~morpheus.stages.input.duo_source_stage.DuoSourceStage` Load Duo Authentication messages. +- File Source Stage {py:class}`~morpheus.stages.input.file_source_stage.FileSourceStage` Load messages from a file. +- HTTP Client Source Stage {py:class}`~morpheus.stages.input.http_client_source_stage.HttpClientSourceStage` Poll a remote HTTP server for incoming data. +- HTTP Server Source Stage {py:class}`~morpheus.stages.input.http_server_source_stage.HttpServerSourceStage` Start an HTTP server and listens for incoming requests on a specified endpoint. +- In Memory Source Stage {py:class}`~morpheus.stages.input.in_memory_source_stage.InMemorySourceStage` Input source that emits a pre-defined list of dataframes. +- Kafka Source Stage {py:class}`~morpheus.stages.input.kafka_source_stage.KafkaSourceStage` Load messages from a Kafka cluster. +- RSS Source Stage {py:class}`~morpheus.stages.input.rss_source_stage.RSSSourceStage` Load RSS feed items into a pandas DataFrame. ## Output - HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` From 87939545725784f61850dc216808ed28aaf023b6 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:45:54 -0800 Subject: [PATCH 12/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 79166155bc..62094471be 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -52,15 +52,19 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos - Kafka Source Stage {py:class}`~morpheus.stages.input.kafka_source_stage.KafkaSourceStage` Load messages from a Kafka cluster. - RSS Source Stage {py:class}`~morpheus.stages.input.rss_source_stage.RSSSourceStage` Load RSS feed items into a pandas DataFrame. +## LLM + +- LLM Engine Stage {py:class}`~morpheus.stages.llm.llm_engine_stage.LLMEngineStage` Execute an LLM engine within a Morpheus pipeline. + ## Output -- HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` -- HTTP Server Sink Stage {py:class}`~morpheus.stages.output.http_server_sink_stage.HttpServerSinkStage` -- In Memory Sink Stage {py:class}`~morpheus.stages.output.in_memory_sink_stage.InMemorySinkStage` -- Databricks Delta Lake Sink Stage {py:class}`~morpheus.stages.output.write_to_databricks_deltalake_stage.DataBricksDeltaLakeSinkStage` -- Write To Elastic Search Stage {py:class}`~morpheus.stages.output.write_to_elasticsearch_stage.WriteToElasticsearchStage` -- Write To File Stage {py:class}`~morpheus.stages.output.write_to_file_stage.WriteToFileStage` -- Write To Kafka Stage {py:class}`~morpheus.stages.output.write_to_kafka_stage.WriteToKafkaStage` -- Write To Vector DB Stage {py:class}`~morpheus.stages.output.write_to_vector_db.WriteToVectorDBStage` +- HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` Write all messages to an HTTP endpoint. +- HTTP Server Sink Stage {py:class}`~morpheus.stages.output.http_server_sink_stage.HttpServerSinkStage` Start an HTTP server and listens for incoming requests on a specified endpoint. +- In Memory Sink Stage {py:class}`~morpheus.stages.output.in_memory_sink_stage.InMemorySinkStage` Collect incoming messages into a list that can be accessed after the pipeline is complete. +- Databricks Delta Lake Sink Stage {py:class}`~morpheus.stages.output.write_to_databricks_deltalake_stage.DataBricksDeltaLakeSinkStage` Write messages to a DeltaLake table. +- Write To Elastic Search Stage {py:class}`~morpheus.stages.output.write_to_elasticsearch_stage.WriteToElasticsearchStage` Write the messages as documents to Elasticsearch. +- Write To File Stage {py:class}`~morpheus.stages.output.write_to_file_stage.WriteToFileStage` Write all messages to a file. +- Write To Kafka Stage {py:class}`~morpheus.stages.output.write_to_kafka_stage.WriteToKafkaStage` Write all messages to a Kafka cluster. +- Write To Vector DB Stage {py:class}`~morpheus.stages.output.write_to_vector_db.WriteToVectorDBStage` Write all messages to a Vector Database. ## Post-process From 46ca5b8ff5d16076fd363c4821f1bf85ea5eb1e8 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:49:41 -0800 Subject: [PATCH 13/19] Fix spelling of MLflow --- morpheus/stages/postprocess/ml_flow_drift_stage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/morpheus/stages/postprocess/ml_flow_drift_stage.py b/morpheus/stages/postprocess/ml_flow_drift_stage.py index 88d45e3541..d8702e43e9 100644 --- a/morpheus/stages/postprocess/ml_flow_drift_stage.py +++ b/morpheus/stages/postprocess/ml_flow_drift_stage.py @@ -34,7 +34,7 @@ @register_stage("mlflow-drift", modes=[PipelineModes.FIL, PipelineModes.NLP, PipelineModes.OTHER]) class MLFlowDriftStage(PassThruTypeMixin, SinglePortStage): """ - Report model drift statistics to ML Flow. + Report model drift statistics to MLflow. Caculates model drift over time and reports the information to MLflow. From 1ffa634a417c871745fd2dc15768dafbe7074d45 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:49:44 -0800 Subject: [PATCH 14/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 62094471be..4f2576b778 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -68,11 +68,11 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos ## Post-process -- Add Classifications Stage {py:class}`~morpheus.stages.postprocess.add_classifications_stage.AddClassificationsStage` -- Add Scores Stage {py:class}`~morpheus.stages.postprocess.add_scores_stage.AddScoresStage` -- Filter Detections Stage {py:class}`~morpheus.stages.postprocess.filter_detections_stage.FilterDetectionsStage` -- Generate Viz Frames Stage {py:class}`~morpheus.stages.postprocess.generate_viz_frames_stage.GenerateVizFramesStage` -- ML Flow Drift Stage {py:class}`~morpheus.stages.postprocess.ml_flow_drift_stage.MLFlowDriftStage` +- Add Classifications Stage {py:class}`~morpheus.stages.postprocess.add_classifications_stage.AddClassificationsStage` Add detected classifications to each message. +- Add Scores Stage {py:class}`~morpheus.stages.postprocess.add_scores_stage.AddScoresStage` Add probability scores to each message. +- Filter Detections Stage {py:class}`~morpheus.stages.postprocess.filter_detections_stage.FilterDetectionsStage` Filter message by a classification threshold. +- Generate Viz Frames Stage {py:class}`~morpheus.stages.postprocess.generate_viz_frames_stage.GenerateVizFramesStage` Write out visualization DataFrames. +- MLflow Drift Stage {py:class}`~morpheus.stages.postprocess.ml_flow_drift_stage.MLFlowDriftStage` Report model drift statistics to MLflow. - Serialize Stage {py:class}`~morpheus.stages.postprocess.serialize_stage.SerializeStage` - Timeseries Stage {py:class}`~morpheus.stages.postprocess.timeseries_stage.TimeSeriesStage` From e61cdb72fde584d063bb04a08c1e2ec9f7ac5b62 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:54:40 -0800 Subject: [PATCH 15/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 4f2576b778..8432ab24bb 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -17,7 +17,7 @@ limitations under the License. # Stages Documentation -Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API {py:mod}`morpheus.stages`. +Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API {py:mod}`morpheus.stages`. In addition to this there are several custom stages contained in the [Examples](../examples.md) and [Developer Guides](../developer_guide/guides.md). ## Doca From 6485467e7668aa34e7ec2114294c913f6004ad17 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:58:09 -0800 Subject: [PATCH 16/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 8432ab24bb..22ab4a1243 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -73,14 +73,15 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos - Filter Detections Stage {py:class}`~morpheus.stages.postprocess.filter_detections_stage.FilterDetectionsStage` Filter message by a classification threshold. - Generate Viz Frames Stage {py:class}`~morpheus.stages.postprocess.generate_viz_frames_stage.GenerateVizFramesStage` Write out visualization DataFrames. - MLflow Drift Stage {py:class}`~morpheus.stages.postprocess.ml_flow_drift_stage.MLFlowDriftStage` Report model drift statistics to MLflow. -- Serialize Stage {py:class}`~morpheus.stages.postprocess.serialize_stage.SerializeStage` -- Timeseries Stage {py:class}`~morpheus.stages.postprocess.timeseries_stage.TimeSeriesStage` +- Serialize Stage {py:class}`~morpheus.stages.postprocess.serialize_stage.SerializeStage` Include & exclude columns from messages. +- Timeseries Stage {py:class}`~morpheus.stages.postprocess.timeseries_stage.TimeSeriesStage` Perform time series anomaly detection and add prediction. ## Pre-process -- Deserialize Stage {py:class}`~morpheus.stages.preprocess.deserialize_stage.DeserializeStage` -- Drop Null Stage {py:class}`~morpheus.stages.preprocess.drop_null_stage.DropNullStage` -- Preprocess AE Stage {py:class}`~morpheus.stages.preprocess.preprocess_ae_stage.PreprocessAEStage` -- Preprocess FIL Stage {py:class}`~morpheus.stages.preprocess.preprocess_fil_stage.PreprocessFILStage` -- Preprocess NLP Stage {py:class}`~morpheus.stages.preprocess.preprocess_nlp_stage.PreprocessNLPStage` -- Train AE Stage {py:class}`~morpheus.stages.preprocess.train_ae_stage.TrainAEStage` +- Deserialize Stage {py:class}`~morpheus.stages.preprocess.deserialize_stage.DeserializeStage` Partition messages based on the pipeline config's `pipeline_batch_size` parameter. +- Drop Null Stage {py:class}`~morpheus.stages.preprocess.drop_null_stage.DropNullStage` Drop null data entries from a DataFrame. +- Preprocess AE Stage {py:class}`~morpheus.stages.preprocess.preprocess_ae_stage.PreprocessAEStage` Prepare Autoencoder input DataFrames for inference. +- Preprocess FIL Stage {py:class}`~morpheus.stages.preprocess.preprocess_fil_stage.PreprocessFILStage` Prepare FIL input DataFrames for inference. +- Preprocess NLP Stage {py:class}`~morpheus.stages.preprocess.preprocess_nlp_stage.PreprocessNLPStage` Prepare NLP input DataFrames for inference. +- Train AE Stage {py:class}`~morpheus.stages.preprocess.train_ae_stage.TrainAEStage` Train an Autoencoder model on incoming data. + From 0990bcdd75b9458ffcf6c92c8ecaf74107953581 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 11:58:18 -0800 Subject: [PATCH 17/19] WIP [no ci] --- docs/source/stages/morpheus_stages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 22ab4a1243..3a6ce7380f 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -84,4 +84,3 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos - Preprocess FIL Stage {py:class}`~morpheus.stages.preprocess.preprocess_fil_stage.PreprocessFILStage` Prepare FIL input DataFrames for inference. - Preprocess NLP Stage {py:class}`~morpheus.stages.preprocess.preprocess_nlp_stage.PreprocessNLPStage` Prepare NLP input DataFrames for inference. - Train AE Stage {py:class}`~morpheus.stages.preprocess.train_ae_stage.TrainAEStage` Train an Autoencoder model on incoming data. - From a7348ec4181e68a624f3a68fa4e1be2e45f733eb Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 12:03:57 -0800 Subject: [PATCH 18/19] Add TOC --- docs/source/stages/morpheus_stages.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/source/stages/morpheus_stages.md b/docs/source/stages/morpheus_stages.md index 3a6ce7380f..19063abd2d 100644 --- a/docs/source/stages/morpheus_stages.md +++ b/docs/source/stages/morpheus_stages.md @@ -19,6 +19,17 @@ limitations under the License. Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API {py:mod}`morpheus.stages`. In addition to this there are several custom stages contained in the [Examples](../examples.md) and [Developer Guides](../developer_guide/guides.md). +## Table of Contents +- [Doca](#doca) +- [General](#general) +- [Inference](#inference) +- [Input](#input) +- [LLM](#llm) +- [Output](#output) +- [Post-process](#post-process) +- [Pre-process](#pre-process) + + ## Doca - Doca Stage {py:class}`~morpheus.stages.doca.doca_source_stage.DocaSourceStage` A source stage used to receive raw packet data from a ConnectX-6 Dx NIC. This stage is not compiled by default refer to the [Doca Example](../../../examples/doca/README.md) for details on building this stage. From c7b034180f4730025d6e53a13bea160691d67253 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 20 Nov 2023 12:16:09 -0800 Subject: [PATCH 19/19] Update getting started --- docs/source/getting_started.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/getting_started.md b/docs/source/getting_started.md index e23a492fca..7667dfab0a 100644 --- a/docs/source/getting_started.md +++ b/docs/source/getting_started.md @@ -272,7 +272,9 @@ This indicates that the `to-file` stage cannot accept the input type of `morpheu #### Pipeline Stages -A complete list of the pipeline stages will be added in the future. For now, you can query the available stages for each pipeline type via: +The CLI allows for an easy way to query the available stages for each pipeline type. Refer to [Morpheus Stages](./stages/morpheus_stages.md) for more extensive documentation of the stages included in Morpheus. + +> **Note**: While most stages are available from the CLI, a small subset of stages, and configuration options for stages are unavailable from the CLI and can only be used via the Python API. ``` $ morpheus run pipeline-nlp --help