[filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism #36690

ShourieG · 2023-09-27T14:16:49Z

Type of change

Enhancement
Docs

Proposed commit message

Added support for the following features as part of a planned enhancement update:

Root level splitting for JSON arrays, similar to GCS input.
File selectors support similar to aws-s3 input, but only supports regex matching atm.
A date filter for blobs based on unix epoch (seconds) format.
Expand event list from field similar to aws-s3 input.
Removed partial save mechanism for now due to concurrency issues.

These additions help make the input more robust for further use. Also added a couple of debug logs for better troubleshooting.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Meta] Improve Azure blob storage input #35126 Filebeat Azure Blob Storate input blobs paths list #35186

Use cases

Screenshots

Logs

…ting

…storage

… field

mergify · 2023-09-27T14:17:55Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-09-27T14:21:18Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

…storage

elasticmachine · 2023-09-27T14:27:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-29T08:34:05.170+0000
Duration: 75 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3193
Skipped	174
Total	3367

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6

Initial look. I will have a deeper read tomorrow.

x-pack/filebeat/input/azureblobstorage/input.go

x-pack/filebeat/input/azureblobstorage/job.go

efd6 · 2023-09-28T06:57:43Z

x-pack/filebeat/input/azureblobstorage/job.go

+	if j.offset == 0 {
+		r, j.isRootArray, err = evaluateJSON(bufio.NewReader(r))
+		if err != nil {
+			return fmt.Errorf("failed to evaluate json for blob: %s, with error: %w", *j.blob.Name, err)


Is j.blob.Name always non-nil?

yes, a file/blob name will always exist, unless there is some unexpected issue from azure's end.

Can we check that at the site that the value becomes ours?

Once we fetch the blob pager, this value becomes localized on the machine.

x-pack/filebeat/input/azureblobstorage/job.go

x-pack/filebeat/input/azureblobstorage/mock/data_random.go

x-pack/filebeat/input/azureblobstorage/mock/testdata/array-at-root.json

bhapas

Added few nits..

Overall the unit testing seems to be limited. How is this tested otherwise?

x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc

x-pack/filebeat/input/azureblobstorage/input_test.go

x-pack/filebeat/input/azureblobstorage/job.go

mergify · 2023-09-28T10:22:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/azure-blob-storage upstream/feature/azure-blob-storage
git merge upstream/main
git push upstream feature/azure-blob-storage

…storage

bhapas

LGTM. But please wait for @efd6 approval

ShourieG · 2023-09-28T13:32:00Z

@efd6 @bhapas addressed all of the suggestions

x-pack/filebeat/input/azureblobstorage/input_stateless.go

x-pack/filebeat/input/azureblobstorage/job.go

x-pack/filebeat/input/azureblobstorage/input.go

x-pack/filebeat/input/azureblobstorage/job.go

x-pack/filebeat/input/azureblobstorage/scheduler.go

andrewkroh · 2023-09-29T00:02:31Z

x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc

+[float]
+==== `timestamp_epoch`
+
+This attribute can be used to filter out files/blobs which have a timestamp greater than the specified value. The value of this attribute should be in unix `epoch` (seconds) format. The timestamp value is compared with the `LastModified Timestamp` obtained from the blob metadata. This attribute can be specified both at the root level of the configuration as well at the container level. The container level values will always take priority and override the root level values if both are specified.


v.Properties.LastModified.Unix() < *s.src.TimeStampEpoch { continue

The description doesn't match up with the code I saw. It seems like it filters blobs that are older than the value.

What is the use case for filtering based on a static value? I've seen inputs offer a relative time based filter when user specifies a duration since now like ignore_older: 168h or since: -168h.

Have updated the description. The use case here is that some customers in the past specially asked for the ability to filter out based on a specific date time, hence added this

andrewkroh · 2023-09-29T00:14:40Z

x-pack/filebeat/input/azureblobstorage/input.go

+// isValidUnixTimestamp checks if the timestamp is a valid Unix timestamp
+func isValidUnixTimestamp(timestamp int64) bool {
+	// checks if the timestamp is within the valid range
+	return minTimestamp <= timestamp && timestamp <= maxTimestamp


What determines the valid range? I can understand validating that the value is non-negative. But why is there an upper range?

The only reason I can think of for checking the upper value is as a sanity check. It probably does not make sense to configure any value that is in the future.

The bound checking can be added to the struct tag and ucfg will enforce it. Like validate:"min=1" for >=1. https://pkg.go.dev/github.com/elastic/go-ucfg#Config.Unpack

Due to this being an optional pointer value, struct tags validation causes a nil pointer deference. And yes this is used as a sanity check for permissible timestamp values.

x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc

x-pack/filebeat/input/azureblobstorage/config.go

…storage

ShourieG · 2023-09-29T08:34:59Z

@efd6 updated the PR with the removal of the partial saves

…emoved partial save mechanism (elastic#36690) * added new debug logs and updated deprecated library methods * added support for path prefix, date filter and root level array splitting * added support for fileselectors, timestamp filtering and expand event field * updated asciidoc, updated comments * updated changelog * addressed PR suggestions * addressed linting errors * addressed further PR suggestions * updated FileSelectors assignment condition * partial save mechanism removed for now due to concurrency issues * updated changelog

ShourieG added 5 commits September 21, 2023 13:15

added new debug logs and updated deprecated library methods

cb93f29

added support for path prefix, date filter and root level array split…

4164ba1

…ting

Merge remote-tracking branch 'upstream/main' into feature/azure-blob-…

09c7252

…storage

added support for fileselectors, timestamp filtering and expand event…

6488a4c

… field

updated asciidoc, updated comments

0e12e55

ShourieG requested a review from a team as a code owner September 27, 2023 14:16

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 27, 2023

mergify bot assigned ShourieG Sep 27, 2023

updated changelog

e69e998

ShourieG added the Team:Security-External Integrations label Sep 27, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 27, 2023

ShourieG added needs_team Indicates that the issue/PR needs a Team:* label 8.11-candidate labels Sep 27, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 27, 2023

ShourieG added Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label labels Sep 27, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 27, 2023

ShourieG added the enhancement label Sep 27, 2023

ShourieG requested a review from efd6 September 27, 2023 14:21

Merge remote-tracking branch 'upstream/main' into feature/azure-blob-…

37cc054

…storage

ShourieG added the backport-skip Skip notification from the automated backport with mergify label Sep 27, 2023

efd6 reviewed Sep 28, 2023

View reviewed changes

bhapas reviewed Sep 28, 2023

View reviewed changes

ShourieG added 3 commits September 28, 2023 18:58

addressed PR suggestions

2dc4212

Merge remote-tracking branch 'upstream/main' into feature/azure-blob-…

79175d4

…storage

Merge remote-tracking branch 'upstream/main' into feature/azure-blob-…

b08440a

…storage

bhapas approved these changes Sep 28, 2023

View reviewed changes

addressed linting errors

5f53d05

efd6 reviewed Sep 28, 2023

View reviewed changes

andrewkroh reviewed Sep 29, 2023

View reviewed changes

ShourieG added 4 commits September 29, 2023 12:32

addressed further PR suggestions

72503ec

Merge remote-tracking branch 'upstream/main' into feature/azure-blob-…

448d1c1

…storage

updated FileSelectors assignment condition

2d4a469

partial save mechanism removed for now due to concurrency issues

6154259

ShourieG changed the title ~~[filebeat][azure-blob-storage] - Added support for new features.~~ [filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism Sep 29, 2023

updated changelog

7a3f072

ShourieG merged commit ee55f1d into elastic:main Oct 2, 2023
8 checks passed

ShourieG deleted the feature/azure-blob-storage branch October 2, 2023 04:10

ShourieG mentioned this pull request Oct 2, 2023

[filebeat][azure-blob-storage] - Cleaned up documentation errors & fixed a minor bug. #36714

Merged

4 tasks

ShourieG mentioned this pull request Oct 10, 2023

Update Azure Blob Storage integration to incorporate latest [azure-blob-storage] input updates elastic/integrations#8153

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism #36690

[filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism #36690

ShourieG commented Sep 27, 2023 •

edited

Loading

mergify bot commented Sep 27, 2023

elasticmachine commented Sep 27, 2023

elasticmachine commented Sep 27, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 left a comment

efd6 Sep 28, 2023

ShourieG Sep 28, 2023 •

edited

Loading

efd6 Sep 28, 2023

ShourieG Sep 29, 2023

bhapas left a comment

mergify bot commented Sep 28, 2023

bhapas left a comment

ShourieG commented Sep 28, 2023

andrewkroh Sep 29, 2023

ShourieG Sep 29, 2023 •

edited

Loading

andrewkroh Sep 29, 2023

ShourieG Sep 29, 2023

ShourieG commented Sep 29, 2023

[filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism #36690

[filebeat][azure-blob-storage] - Added support for new features and removed partial save mechanism #36690

Conversation

ShourieG commented Sep 27, 2023 • edited Loading

Type of change

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Sep 27, 2023

elasticmachine commented Sep 27, 2023

elasticmachine commented Sep 27, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 left a comment

Choose a reason for hiding this comment

efd6 Sep 28, 2023

Choose a reason for hiding this comment

ShourieG Sep 28, 2023 • edited Loading

Choose a reason for hiding this comment

efd6 Sep 28, 2023

Choose a reason for hiding this comment

ShourieG Sep 29, 2023

Choose a reason for hiding this comment

bhapas left a comment

Choose a reason for hiding this comment

mergify bot commented Sep 28, 2023

bhapas left a comment

Choose a reason for hiding this comment

ShourieG commented Sep 28, 2023

andrewkroh Sep 29, 2023

Choose a reason for hiding this comment

ShourieG Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

andrewkroh Sep 29, 2023

Choose a reason for hiding this comment

ShourieG Sep 29, 2023

Choose a reason for hiding this comment

ShourieG commented Sep 29, 2023

ShourieG commented Sep 27, 2023 •

edited

Loading

elasticmachine commented Sep 27, 2023 •

edited by jenkins-beats-ci bot

Loading

ShourieG Sep 28, 2023 •

edited

Loading

ShourieG Sep 29, 2023 •

edited

Loading