Skip to content

Commit

Permalink
Upgrade azure-eventhub to input v2 API (#39511)
Browse files Browse the repository at this point in the history 
* Rename client to pipelineClient for clarity

* Rename azure to eventHubMetadata for clarity

* Move remaining code out of eph.go

* Remove erroneous error check

* Update tests

Switch from the v1 Outlet to the v2 PipelineClient

* Simplify run()

It seems we only need to wait for the context to be done here.

* Update NOTICE

* Clarify handler tradeoffs

The existing input version does not handle publishing acks from the
Beats pipeline. The input API v1 does not seem to offer this feature.

With the transition to the input API v2, we have acks management.
However, the legacy event hub SDK internally updates the checkpoint
info after a successful handler call, and does not seem to offer hooks
for acks management.

Since the new modern SDK offers better checkpoint management, we keep
the current behavior intact, and we'll implement ACKs in the
event hub input v2.

---------

Co-authored-by: Tiago Queiroz <me@tiago.life>
  • Loading branch information
@zmoog @belimawr
zmoog and belimawr authored Jun 28, 2024
1 parent 8bbcfcc commit 68e2b8b
Show file tree
Hide file tree
Showing 13 changed files with 565 additions and 576 deletions.
104 changes: 50 additions & 54 deletions NOTICE.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1742,11 +1742,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/azcore
Version: v1.9.0
Version: v1.11.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.11.1/LICENSE.txt:

MIT License

Expand All @@ -1773,11 +1773,11 @@ SOFTWARE

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/azidentity
Version: v1.4.0
Version: v1.5.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.4.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.2/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -1954,11 +1954,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Version: v1.0.0
Version: v1.3.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.3.2/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -17511,11 +17511,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : github.com/golang-jwt/jwt/v5
Version: v5.0.0
Version: v5.2.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/v5@v5.0.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/v5@v5.2.1/LICENSE:

Copyright (c) 2012 Dave Grijalva
Copyright (c) 2021 golang-jwt maintainers
Expand Down Expand Up @@ -29001,11 +29001,11 @@ Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-pipeline-g

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/internal
Version: v1.5.0
Version: v1.7.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.5.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.7.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -29122,13 +29122,43 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage
Version: v1.5.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage@v1.5.0/LICENSE.txt:

MIT License

Copyright (c) Microsoft Corporation. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/go-amqp
Version: v1.0.0
Version: v1.0.5
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/go-amqp@v1.0.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/!azure/go-amqp@v1.0.5/LICENSE:

MIT License

Expand Down Expand Up @@ -30826,11 +30856,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/AzureAD/microsoft-authentication-library-for-go
Version: v1.1.1
Version: v1.2.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.1.1/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.2.2/LICENSE:

MIT License

Expand Down Expand Up @@ -36984,40 +37014,6 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


--------------------------------------------------------------------------------
Dependency : github.com/dnaeon/go-vcr
Version: v1.2.0
Licence type (autodetected): BSD-2-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/dnaeon/go-vcr@v1.2.0/LICENSE:

Copyright (c) 2015-2016 Marin Atanasov Nikolov <dnaeon@gmail.com>
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer
in this position and unchanged.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


--------------------------------------------------------------------------------
Dependency : github.com/dnephin/pflag
Version: v1.0.7
Expand Down Expand Up @@ -45608,11 +45604,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/joho/godotenv
Version: v1.3.0
Version: v1.5.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/joho/godotenv@v1.3.0/LICENCE:
Contents of probable licence file $GOMODCACHE/github.com/joho/godotenv@v1.5.1/LICENCE:

Copyright (c) 2013 John Barton

Expand Down Expand Up @@ -49585,11 +49581,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : github.com/pkg/browser
Version: v0.0.0-20210911075715-681adbf594b8
Version: v0.0.0-20240102092130-5ac0b6a4141c
Licence type (autodetected): BSD-2-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/pkg/browser@v0.0.0-20210911075715-681adbf594b8/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/pkg/browser@v0.0.0-20240102092130-5ac0b6a4141c/LICENSE:

Copyright (c) 2014, Dave Cheney <dave@cheney.net>
All rights reserved.
Expand Down Expand Up @@ -49899,11 +49895,11 @@ Contents of probable licence file $GOMODCACHE/github.com/prometheus/client_golan

--------------------------------------------------------------------------------
Dependency : github.com/rogpeppe/go-internal
Version: v1.11.0
Version: v1.12.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/rogpeppe/go-internal@v1.11.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/rogpeppe/go-internal@v1.12.0/LICENSE:

Copyright (c) 2018 The Go Authors. All rights reserved.

Expand Down
17 changes: 9 additions & 8 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,14 +180,14 @@ require (
cloud.google.com/go v0.110.8
cloud.google.com/go/compute v1.23.0
cloud.google.com/go/redis v1.13.1
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.2
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/consumption/armconsumption v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.8.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.1.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2
github.com/Azure/go-autorest/autorest/adal v0.9.21
github.com/apache/arrow/go/v14 v14.0.2
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1
Expand All @@ -209,7 +209,7 @@ require (
github.com/elastic/toutoumomoma v0.0.0-20221026030040-594ef30cb640
github.com/foxcpp/go-mockdns v0.0.0-20201212160233-ede2f9158d15
github.com/go-ldap/ldap/v3 v3.4.6
github.com/golang-jwt/jwt/v5 v5.0.0
github.com/golang-jwt/jwt/v5 v5.2.1
github.com/google/cel-go v0.19.0
github.com/googleapis/gax-go/v2 v2.12.0
github.com/gorilla/handlers v1.5.1
Expand Down Expand Up @@ -240,15 +240,15 @@ require (
github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect
github.com/Azure/azure-amqp-common-go/v4 v4.2.0 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.0 // indirect
github.com/Azure/go-amqp v1.0.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.7.0 // indirect
github.com/Azure/go-amqp v1.0.5 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.1.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/andybalholm/brotli v1.0.5 // indirect
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
Expand Down Expand Up @@ -320,6 +320,7 @@ require (
github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/karrick/godirwalk v1.17.0 // indirect
Expand Down Expand Up @@ -348,7 +349,7 @@ require (
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/pierrec/lz4 v2.6.0+incompatible // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/prometheus/client_golang v1.11.1 // indirect
github.com/rootless-containers/rootlesskit v1.1.0 // indirect
Expand Down
Loading

0 comments on commit 68e2b8b

Please sign in to comment.