Merge pull request #6254 from wazuh/13577-custom-logs-bucket-subscriber

New Custom Logs Buckets documentation

source/cloud-security/amazon/services/prerequisites/S3-bucket.rst

@@ -3,8 +3,6 @@
 .. meta::
   :description: Learn how to configure an Amazon S3 bucket, an object storage service that delivers scalability, data availability, security, and performance.
 
-.. _s3_bucket:
-
 Configuring an S3 Bucket
 ========================
 

source/cloud-security/amazon/services/prerequisites/considerations.rst

@@ -157,6 +157,8 @@ Below there is an example of different services configuration:
   </wodle>
 
 
+.. _using_non-default_aws_endpoints:
+
 Using non-default AWS endpoints
 -------------------------------
 

source/cloud-security/amazon/services/prerequisites/credentials.rst

@@ -88,6 +88,8 @@ To use the *prod* profile in the AWS integration you would use the following buc
     <aws_profile>prod</aws_profile>
   </bucket>
 
+.. _iam_roles:
+
 IAM Roles
 ^^^^^^^^^
 

source/cloud-security/amazon/services/supported-services/custom-buckets.rst

@@ -0,0 +1,135 @@
+.. Copyright (C) 2015, Wazuh, Inc.
+
+.. meta::
+  :description: Learn how to configure Amazon Custom Logs Buckets fetching.
+
+.. _amazon_custom_logs:
+
+Custom Logs Buckets
+===================
+
+.. versionadded:: 4.7.0
+
+`Amazon Simple Queue Service (Amazon SQS)  <https://aws.amazon.com/sqs/>`_ is a fully managed message queuing service. It offers secure, durable, and available hosted queues to decouple and scale software systems and components. It allows sending, storing, and receiving messages between software components at any volume, without losing messages or requiring other services to be available. These features make it an optimal component to associate with Amazon S3 buckets to consume any type of log.
+
+Combining Amazon SQS with Amazon S3 buckets allows Wazuh to fetch JSON, CSV, and plain text logs from any custom path. The origin of these logs don't even need to be AWS.
+
+.. note::
+
+   To properly process CSV logs, they must include column headers.
+
+To set up the Wazuh integration for Custom Logs Buckets, you need to do the following:
+
+   #. Create an AWS SQS Queue.
+   #. Configure an S3 bucket. For every object creation event, the bucket sends notifications to the queue.
+
+AWS configuration
+-----------------
+
+Amazon Simple Queue Service
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Set up a *Standard* type SQS Queue with the default configurations.  You can apply an Access Policy similar to the following example, where ``<region>``, ``<account-id>``, and ``<s3-bucket>`` are the region, account ID, and the name you are going to provide to the S3 bucket.
+
+   .. code-block:: json
+
+     {
+     "Version": "2012-10-17",
+     "Id": "example-ID",
+     "Statement": [
+       {
+         "Sid": "example-access-policy",
+         "Effect": "Allow",
+         "Principal": {
+           "Service": "s3.amazonaws.com"
+         },
+         "Action": "SQS:SendMessage",
+         "Resource": "arn:aws:sqs:<region>:<account-id>:<s3-bucket>",
+         "Condition": {
+           "StringEquals": {
+             "aws:SourceAccount": "<account-id>"
+           },
+           "ArnLike": {
+             "aws:SourceArn": "arn:aws:s3:*:*:<s3-bucket>"
+           }
+         }
+       }
+     ]
+     }
+  
+   You can make your access policy to accept S3 notifications from different account IDs and to apply different conditions. More information in `Managing access in Amazon SQS <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-overview-of-managing-access.html>`_. 
+
+Amazon S3 and Event Notifications
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To configure an S3 bucket that reports creation events, do the following.
+
+#. Configure an S3 bucket as defined in the :doc:`Configuring an S3 Bucket <../prerequisites/S3-bucket>` section. Provide the name you decided in the previous section.
+#. Once created, go to **Event notifications** inside the **Properties** tab. Select **Create event notification**. 
+#. In **Event Types**, select **All object create events**. This generates notifications for any type of event that results in the creation of an object in the bucket.
+#. In the **Destination** section, select the following options:
+
+   -  **SQS queue**
+   -  **Choose from your SQS queues**
+#. Choose the queue you created previously.
+
+Wazuh Configuration
+-------------------
+
+.. warning::
+
+   Every message sent to the queue is read and deleted. Make sure you only use the queue for bucket notifications.
+
+#. Edit the ``/var/ossec/etc/ossec.conf`` file. Add the SQS name and your `Configuration parameters`_ for the buckets service. Set this inside ``<subscriber type="buckets">``. For example:
+
+   .. code-block:: xml
+      :emphasize-lines: 6,7
+
+      <wodle name="aws-s3">
+          <disabled>no</disabled>
+          <interval>1h</interval>
+          <run_on_start>yes</run_on_start>
+          <subscriber type="buckets">
+              <sqs_name>sqs-queue</sqs_name>
+              <aws_profile>default</aws_profile>
+          </subscriber>
+      </wodle>
+
+   Check the :doc:`AWS S3 module </user-manual/reference/ossec-conf/wodle-s3>` reference manual to learn more about the available settings.
+
+   .. note::
+
+      The amount of notifications present in the queue affects the execution time of the AWS S3 module. If the ``<interval>`` value for the waiting time between executions is too short, the :ref:`Interval overtaken <interval_overtaken_message>` warning is logged into the ``ossec.log`` file.
+
+#. Restart the Wazuh manager to apply the changes.
+
+   .. include:: /_templates/common/restart_manager.rst
+
+Configuration parameters
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Configure the following fields to set the queue and authentication configuration. For more information, check the :ref:`subscribers` reference.
+
+Queue
+~~~~~
+
+-  ``<sqs_name>``: The name of the queue.
+-  Optional – ``<service_endpoint>``: The AWS S3 endpoint URL for data downloading from the bucket. Check :ref:`using_non-default_aws_endpoints` for more information about VPC and FIPS endpoints.
+
+Authentication
+~~~~~~~~~~~~~~
+
+The available authentication methods are the following:
+
+-  :ref:`IAM Roles <iam_roles>`
+-  :ref:`Profiles <aws_profile>`
+
+These authentication methods require using the ``/root/.aws/credentials`` file to provide credentials. You can find more information in :ref:`Configuring AWS credentials <amazon_credentials>`.
+
+The available authentication configuration parameters are the following:
+
+-  ``<aws_profile>``: A valid profile name from a Shared Credential File or AWS Config File with the permission to read logs from the bucket.
+-  ``<iam_role_arn>``: ARN for the corresponding IAM role to assume.
+-  Optional – ``<iam_role_duration>``: The session duration in seconds.
+-  Optional – ``<sts_endpoint>``: The URL of the VPC endpoint of the AWS Security Token Service.
+

source/cloud-security/amazon/services/supported-services/index.rst

@@ -53,6 +53,8 @@ The next table contains the most relevant information about configuring each ser
 +--------------+----------------------------------------------------------+-----------------------+----------------+------------------------------------------------------------------------------------------------------------------+
 | Amazon       | :ref:`Amazon Security Lake <amazon_security_lake>`       | subscriber            | security_lake  |                                                                                                                  |
 +--------------+----------------------------------------------------------+-----------------------+----------------+------------------------------------------------------------------------------------------------------------------+
+| Amazon       | :ref:`Custom Logs Buckets <amazon_custom_logs>`          | subscriber            | buckets        |                                                                                                                  |
++--------------+----------------------------------------------------------+-----------------------+----------------+------------------------------------------------------------------------------------------------------------------+
 | Cisco        | :ref:`Umbrella <cisco_umbrella>`                         | bucket                | cisco_umbrella | <bucket_name>/<prefix>/<year>-<month>-<day>                                                                      |
 +--------------+----------------------------------------------------------+-----------------------+----------------+------------------------------------------------------------------------------------------------------------------+
 
@@ -74,4 +76,5 @@ The next table contains the most relevant information about configuring each ser
     ecr-image-scanning
     cisco-umbrella
     elastic-load-balancing/index
-    security-lake
+    security-lake
+    custom-buckets

source/cloud-security/amazon/services/troubleshooting.rst

@@ -252,3 +252,7 @@ Error codes reference
     +-----------+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
     | 21        | Failed fetch/delete from SQS                                      | Check that no more instances of the wodle are running at the same time.                                                                              |
     +-----------+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | 22        | Invalid region                                                    | Check the provided ``region`` in the ``ossec.conf`` file.                                                                                            |
+    +-----------+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | 23        | Profile not found                                                 | Check the provided ``aws_profile`` in the ``ossec.conf`` file.                                                                                       |
+    +-----------+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

source/user-manual/reference/ossec-conf/wodle-s3.rst

@@ -709,23 +709,28 @@ It is necessary to specify the type as an attribute of the ``subscriber`` tag to
 
 		</subscriber>
 
-The currently available type is: ``security_lake``.
+The currently available types are: ``security_lake`` and ``buckets``.
 
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | Options                                | Allowed values                                              | Mandatory/Optional                            |
 +========================================+=============================================================+===============================================+
-| :ref:`subscriber_sqs_name`             | Any valid SQS name                                          | Mandatory for Amazon Security Lake            |
-|                                        |                                                             | Subscription                                  |
+| :ref:`subscriber_sqs_name`             | Any valid SQS name                                          | Mandatory                                     |
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | :ref:`subscriber_iam_role_arn`         | Valid role ARN                                              | Mandatory for Amazon Security Lake            |
 |                                        |                                                             | Subscription                                  |
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | :ref:`subscriber_external_id`          | Valid external ID                                           | Mandatory for Amazon Security Lake            |
 |                                        |                                                             | Subscription                                  |
+|                                        |                                                             | (not available for Custom Logs Buckets)       |
++----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
+| :ref:`subscriber_aws_profile`          | Valid profile name                                          | Optional                                      |
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | :ref:`subscriber_iam_role_duration`    | Number of seconds between 900 and 3600                      | Optional (if set, it requires an iam_role_arn |
 |                                        |                                                             | to be provided)                               |
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
+| :ref:`subscriber_discard_regex`        | A regex value to determine if an event must be discarded    | Optional                                      |
+|                                        |                                                             | (only available for Custom Logs Buckets)      |
++----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | :ref:`subscriber_sts_endpoint`         | Any valid VPC endpoint URL for STS                          | Optional                                      |
 +----------------------------------------+-------------------------------------------------------------+-----------------------------------------------+
 | :ref:`subscriber_service_endpoint`     | Any valid endpoint URL for S3                               | Optional                                      |
@@ -783,6 +788,54 @@ A valid number of seconds that defines the duration of the session assumed when
 | **Allowed values** | Number of seconds between 900 and 3600   |
 +--------------------+------------------------------------------+
 
+.. _subscriber_aws_profile:
+
+aws_profile
+^^^^^^^^^^^
+
+A valid profile name from a Shared Credential File or AWS Config File with the permission to access the service.
+
++--------------------+--------------------+
+| **Default value**  | N/A                |
++--------------------+--------------------+
+| **Allowed values** | Valid profile name |
++--------------------+--------------------+
+
+.. _subscriber_discard_regex:
+
+discard_regex
+^^^^^^^^^^^^^
+
+A regular expression to determine if an event must be discarded. JSON and CSV logs require a mandatory ``field`` attribute. The regex is applied to the event field specified with this attribute.
+
++--------------------+----------------------------------------+
+| **Default value**  | N/A                                    |
++--------------------+----------------------------------------+
+| **Allowed values** | Any regex or sregex expression         |
++--------------------+----------------------------------------+
+
+Attributes:
+
++-----------+-----------------------------------------------------------------+
+| **field** | The event field where to apply the regex                        |
+|           +------------------+----------------------------------------------+
+|           | Default value    | N/A                                          |
+|           +------------------+----------------------------------------------+
+|           | Allowed values   | A str containing the full field name path    |
++-----------+------------------+----------------------------------------------+
+
+Usage examples:
+
+.. code-block:: console
+
+    <discard_regex field="data.configurationItemStatus">REJECT</discard_regex>
+
+Usage example only for plain text logs:
+
+.. code-block:: console
+
+    <discard_regex>.*Log:.*</discard_regex>
+
 .. _subscriber_sts_endpoint:
 
 sts_endpoint
@@ -873,5 +926,9 @@ Example of configuration
         <external_id>wazuh-external-id-value</external_id>
         <iam_role_arn>arn:aws:iam::010203040506:role/ASL-Role</iam_role_arn>
       </subscriber>
+      <subscriber type="buckets">
+        <sqs_name>sqs-custom-logs-queue</sqs_name>
+        <aws_profile>dev</aws_profile>
+      </subscriber>
   </wodle>