Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Azure eventhub support #58

Merged
merged 9 commits into from
Oct 17, 2023
Merged

Conversation

bhapas
Copy link
Contributor

@bhapas bhapas commented Aug 24, 2023

Add support for Azure eventhub output.

For Credentials : https://github.com/Azure/azure-sdk-for-go/blob/6b6f76ebe0d2334c83e8b6f89af4fe9d0b1ce631/sdk/azidentity/README.md?plain=1#L156-L187

Test Execution

workspace/stream - (azure-eventhub-output) >
 ./stream log main.go 
-p=azureeventhub 
--azure-event-hub-connection-string="Endpoint=sb://test-eventhub-stream-seis.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SharedAccessKey"


{"level":"debug","ts":"2023-08-28T10:18:02.777+0200","caller":"output/util.go:28","msg":"Connecting...","address":""}
{"level":"info","ts":"2023-08-28T10:18:02.777+0200","caller":"output/util.go:41","msg":"Connected","address":""}
{"level":"debug","ts":"2023-08-28T10:18:02.777+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":1}
{"level":"debug","ts":"2023-08-28T10:18:04.775+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":2}
{"level":"debug","ts":"2023-08-28T10:18:05.101+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":3}
{"level":"debug","ts":"2023-08-28T10:18:05.422+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":4}
{"level":"debug","ts":"2023-08-28T10:18:05.560+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":5}
{"level":"debug","ts":"2023-08-28T10:18:05.699+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":6}
{"level":"debug","ts":"2023-08-28T10:18:05.831+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":7}
{"level":"debug","ts":"2023-08-28T10:18:05.969+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":8}
{"level":"debug","ts":"2023-08-28T10:18:06.108+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":9}
{"level":"debug","ts":"2023-08-28T10:18:06.243+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":10}
{"level":"debug","ts":"2023-08-28T10:18:06.385+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":11}
{"level":"debug","ts":"2023-08-28T10:18:06.547+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":12}
{"level":"debug","ts":"2023-08-28T10:18:06.689+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":13}
{"level":"debug","ts":"2023-08-28T10:18:06.826+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":14}
{"level":"debug","ts":"2023-08-28T10:18:07.028+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":15}
{"level":"debug","ts":"2023-08-28T10:18:07.227+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":16}
{"level":"debug","ts":"2023-08-28T10:18:07.381+0200","caller":"command/log.go:81","msg":"Sending log line.","address":"","log":"main.go","line_number":17}
{"level":"info","ts":"2023-08-28T10:18:07.516+0200","caller":"command/log.go:94","msg":"Log data sent.","address":"","log":"main.go","total_bytes":337,"total_lines":17}
workspace/stream - (azure-eventhub-output) > 

Screenshots

Messages in Azure portal
image

@elasticmachine
Copy link

elasticmachine commented Aug 24, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-16T12:12:58.083+0000

  • Duration: 6 min 59 sec

Test stats 🧪

Test Results
Failed 0
Passed 22
Skipped 0
Total 22

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

Copy link
Member

@P1llus P1llus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At a quick glance, the main parts that I noticed is that since we don't emulate eventhub locally or in docker in any way, we need to add some more credential options.
Usually the Connection String or some sort of credentials file option would be great, and default back (with a log entry) when no credentials are found, that its going to use default or environment variable instead.

Check the golangci-lint output as well :)

@bhapas bhapas marked this pull request as ready for review August 28, 2023 10:10
@bhapas bhapas requested review from P1llus, marc-gr and a team and removed request for marc-gr August 28, 2023 10:10
@marc-gr
Copy link
Contributor

marc-gr commented Aug 29, 2023

It looks nice!

I wonder if we could add some tests for it if possible and to add some docs about how to use it.

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs docs and probably at least one example configuration to help users understand how to hold it.

pkg/output/lumberjack/lumberjack_test.go Outdated Show resolved Hide resolved
@bhapas
Copy link
Contributor Author

bhapas commented Oct 13, 2023

Raised Azure/azure-sdk-for-go#21758 for adding support for emulator in Azurite for Event Hub.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
command/root.go Show resolved Hide resolved
pkg/output/azureeventhub/azure_event_hub.go Outdated Show resolved Hide resolved
pkg/output/azureeventhub/azure_event_hub.go Outdated Show resolved Hide resolved
pkg/output/azureeventhub/azure_event_hub.go Outdated Show resolved Hide resolved
pkg/output/lumberjack/lumberjack_test.go Outdated Show resolved Hide resolved
pkg/output/azureeventhub/azure_event_hub.go Show resolved Hide resolved
pkg/output/azureeventhub/azure_event_hub.go Outdated Show resolved Hide resolved
@bhapas bhapas requested a review from efd6 October 16, 2023 09:38
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bhapas bhapas merged commit bd3c4f9 into elastic:main Oct 17, 2023
6 checks passed
@bhapas bhapas deleted the azure-eventhub-output branch October 17, 2023 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants