Capture EventHub to Azure Blob Storage Container #146
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merge in master
jananivMS
reviewed
Aug 28, 2019
There was a problem hiding this comment.
Stephen - I reviewed the code and it looks goodl As we talked yesterday I opened the following issue to track replacing the blob storage creation with the blob storage helper/operator - Issue #156
That said, the tests seem to be failing for this PR, so you may need to look into that before merging this. So not approving but adding as a comment.
|
Thanks @jananivMS |
|
There may be some race conditions. Might be worth trying to test locally using the |
Thanks for the suggestion - I am seeing the following race condition running locally - though it is not specifically related. |
|
yep, this is likely unrelated, on a side note, I have a test refactor PR that aims to simplify things a bit... |
|
Sorry for the slow review on my part. I will be sure to submit one in the morning if it isn't merged by then. |
|
Thanks a lot @frodopwns |
|
Looks like I can't pull the code due to permissions issues. I can look at it I guess. |
|
Looks reasonable. Could you grant me Read access to your fork so I can test the code locally when you get a chance? Forking is great for open source projects, but while they are private it can be tricky! |
Pr 22 merge (#158)
incorporated storages modulefixed setup and teardown of storage tests
|
@frodopwns I added you to my fork. |
frodopwns
approved these changes
Aug 30, 2019
This was referenced Aug 31, 2019
jananivMS
added a commit
that referenced
this pull request
Sep 3, 2019
* feat: implement keyvault controller * Ace's KV changes with updates * Added an event for the final successful provisioning * Updated changes based on the PR comments * removing unwanted file * making resource group name the one in the keyvault yaml * need to handled unexpected error types...like validation.error (#111) * refactor tests (#90) * improve tests with parallel execution and rm sleep * fix the tests to run on kindcluster * Updates to KV controller from Ace (#80) (#112) * feat: implement keyvault controller * Ace's KV changes with updates * Added an event for the final successful provisioning * Updated changes based on the PR comments * removing unwanted file * making resource group name the one in the keyvault yaml Co-authored-by: Ace Eldeib <alexeldeib@gmail.com> * Test update (#115) * this needs to exist in the reconciler in order to use controllerutil createorupdate * Feat/add consumer group kind (#117) * add consumer group kind * update tests for consumer group * fix isbeingdeleted * Updates to README - steps for onboarding (#114) * cluster additions * updated docs * Update azure-pipelines.yaml (#119) * Update azure-pipelines.yaml * fix tests (#140) * revert back // +kubebuilder:subresource:status changes - fix broken tests * Devcontainer to Help Onboard New People (#142) * add dev conatiner - wip * DevContainer up and running. * Removed `sleep 80` and replaced with `kubectl wait`. * Run `make set-kindcluster` from docker-compose. * Set timeout on wait. * Added `install-test-dependency` to makefile and dockerfile. * Update README - Create SP with contribution rights. * Updated README with details on using devcontainer. * Stuff that wanted me to commit. * Reverted changes made to `docker-build` in Makefile. * pass future where possible instead of bool (#121) * first commit on Amanda's branch * first * before properties * test not tested * test works * unit tests work, needs firewall rules * addresses feedback * erin's feedback * janani's change, pass future * async works much better now * janani feedback * screwed up interface prototype * randomize the resources names used in tests (#152) * Ability to Set SecretName When Creating Event Hub (#151) * Updated eventhub_types - Added `secretName`. * Added `secretName` to sample manifest. * Set secret name to `secretName` if set, otherwise use eventhub name. * Updated Makefile to update Azure Operator. Also added the ability to rebuild image without cache. * Updated README on how to update the Azure Operator. * Updated CRD with SecretName description. * Added tests to ensure `SecretName` was being used if present. * Fix test. * Pr 22 merge (#158) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Pr 22 merge (#158) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Pr 22 merge (#158) (#165) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Capture EventHub to Azure Blob Storage Container (#146) * added eventhub with and without capture * create, delete and get properties for storage manager * capture eventhub tests * added storage tests to make tests * configured location to default set by environment variable * synchronised test setup and teardown * incorporated storages module * fixed setup and teardown of storage tests * fixed storage container tests * Camelcase EventHub (#176) * Removed ports from docker-compose. * Updated CRD - camelcase over lowercase. * Updated example manifests. * Role thing. * Camelcase new changes to EventHub types. * Camelcase example. * Removed old file. * Fixing issues #173 and #174 (#175) * Updated controllers to use `azure.microsoft.com` over `service.azure`. * Updated webhooks to point to `azure.microsoft.com`. * Updated caninject to point to `azure.microsoft.com`. * Regenerated role.yaml. * Point kustomization.yaml in CRD to right base CRDs. * Updated demo. * Role update. * Update group from service to azure in PROJECT. * Increased Partition Count Minimum in EventHub to 2 (#178) * Increase minimum partition count to 2. * Updated the CRD. * Updated eventhub example. * Changed resource group example. * Increased test partition count to 2. * Updated tests.
jananivMS
added a commit
that referenced
this pull request
Sep 4, 2019
…ver (#184) * feat: implement keyvault controller * Ace's KV changes with updates * Added an event for the final successful provisioning * Updated changes based on the PR comments * removing unwanted file * making resource group name the one in the keyvault yaml * need to handled unexpected error types...like validation.error (#111) * refactor tests (#90) * improve tests with parallel execution and rm sleep * fix the tests to run on kindcluster * Updates to KV controller from Ace (#80) (#112) * feat: implement keyvault controller * Ace's KV changes with updates * Added an event for the final successful provisioning * Updated changes based on the PR comments * removing unwanted file * making resource group name the one in the keyvault yaml Co-authored-by: Ace Eldeib <alexeldeib@gmail.com> * Test update (#115) * this needs to exist in the reconciler in order to use controllerutil createorupdate * Feat/add consumer group kind (#117) * add consumer group kind * update tests for consumer group * fix isbeingdeleted * Updates to README - steps for onboarding (#114) * cluster additions * updated docs * Update azure-pipelines.yaml (#119) * Update azure-pipelines.yaml * fix tests (#140) * revert back // +kubebuilder:subresource:status changes - fix broken tests * Devcontainer to Help Onboard New People (#142) * add dev conatiner - wip * DevContainer up and running. * Removed `sleep 80` and replaced with `kubectl wait`. * Run `make set-kindcluster` from docker-compose. * Set timeout on wait. * Added `install-test-dependency` to makefile and dockerfile. * Update README - Create SP with contribution rights. * Updated README with details on using devcontainer. * Stuff that wanted me to commit. * Reverted changes made to `docker-build` in Makefile. * pass future where possible instead of bool (#121) * first commit on Amanda's branch * first * before properties * test not tested * test works * unit tests work, needs firewall rules * addresses feedback * erin's feedback * janani's change, pass future * async works much better now * janani feedback * screwed up interface prototype * randomize the resources names used in tests (#152) * Ability to Set SecretName When Creating Event Hub (#151) * Updated eventhub_types - Added `secretName`. * Added `secretName` to sample manifest. * Set secret name to `secretName` if set, otherwise use eventhub name. * Updated Makefile to update Azure Operator. Also added the ability to rebuild image without cache. * Updated README on how to update the Azure Operator. * Updated CRD with SecretName description. * Added tests to ensure `SecretName` was being used if present. * Fix test. * Pr 22 merge (#158) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Pr 22 merge (#158) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Pr 22 merge (#158) (#165) * kubebuilder init --domain azure --license none * kubebuilder create api --group service --version v1alpha1 --kind Storage * kubebuilder create api --group service --version v1alpha1 --kind CosmosDB * Add MIT License * Initial codes to support Azure Storage Account * Add development docs * Remove the storage account name from the spec * Sync additional resources for Azure storage account 1. Create a secret based on storage account credentials 2. Add the global config * Upgrade kubebuilder to 2.0.0-beta.0 and controller-runtime to v0.2.0-beta.4 * Copy pkg in Dockerfile * Update controller-gen and make manifests * Add prefix "Storage" for storage_types * feature: add redis cache service * Ignore the NotFound error when deleting resources * Requeue the request if the deployment is not complete * feature: add cosmosdb service * Refine the logic of updating additional resources and output * Deploy operator on a remote cluster * add a sample app deployment yaml * Generate assets for the templates * Requeue after 30 seconds to avoid too many requests Ignore the NotFound error when deleting cosmosdb * Fix a bug of missing capacity of rediscache template * fix: judge whether resources need to be updated With adding generation in status, we can judge whether resources need to be updated. Co-authored-by: Bin Xia <binxi@microsoft.com> * Add docs to run the demo * Update manager-role to operate secrets Workaround: the rule should be appended. But I don't know how for now. The workaround is to copy config/rbac/role.yaml and add the new rule. Should be fixed in future. * fix(Makefile): rename the target from "generate" to "generate-template" to avoid conflict * Refactoring data focused operators. Storage currently working though it needs cleanup * Added deepcopy generated code * CosmosDB deploy working * Detailing current implementation of CosmosDB Create parameters * Removing TestTags * Redis cache now deploys * Cleaned up code and removed references to v1alpha1 * Updating controllers logging calls Co-authored-by: Chris Risner <github@chrisrisner.com> Co-authored-by: Bin Xia <binxi@microsoft.com> * Capture EventHub to Azure Blob Storage Container (#146) * added eventhub with and without capture * create, delete and get properties for storage manager * capture eventhub tests * added storage tests to make tests * configured location to default set by environment variable * synchronised test setup and teardown * incorporated storages module * fixed setup and teardown of storage tests * fixed storage container tests * Camelcase EventHub (#176) * Removed ports from docker-compose. * Updated CRD - camelcase over lowercase. * Updated example manifests. * Role thing. * Camelcase new changes to EventHub types. * Camelcase example. * Removed old file. * Fixing issues #173 and #174 (#175) * Updated controllers to use `azure.microsoft.com` over `service.azure`. * Updated webhooks to point to `azure.microsoft.com`. * Updated caninject to point to `azure.microsoft.com`. * Regenerated role.yaml. * Point kustomization.yaml in CRD to right base CRDs. * Updated demo. * Role update. * Update group from service to azure in PROJECT. * Increased Partition Count Minimum in EventHub to 2 (#178) * Increase minimum partition count to 2. * Updated the CRD. * Updated eventhub example. * Changed resource group example. * Increased test partition count to 2. * Updated tests. * Changes to wire Sql firewall in the controller * Addressed PR comments
Porges
pushed a commit
that referenced
this pull request
May 11, 2021
* Fix usage of ** in glob
- Add new dependency becuase golang filepath.glob doesn't support
recursive matching.
* Allow output directory to be configured
- This fixes #144.
- Improved configuration yaml file property casing.
* Add makefile target for running k8sinfra-gen
* Little comment type fixup
* Allow `make gen-arm` to be rerun by deleting the generated code
Since the generated code doesn't currently build it would prevent
`make gen-arm` from being rerun until the apis directory was
removed.
Hit a bug in codegen.isFileGenerated that would fail to close
files - move the deferred close to before the early return.
* Add clarifying comment to Makefile
Co-authored-by: Christian Muirhead <christian.muirhead@canonical.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
refs As a user, I can to enable event logging to a storage container using EventHub Capture #145
Special notes for your reviewer:
If applicable: