Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add native Azure Blob support to BucketController #513

Closed
wants to merge 56 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
04219b8
Add native support to Azure Blob
laozc Dec 3, 2021
774cf9a
Remove calling deprecated MSI function
laozc Dec 3, 2021
e89ef08
Update CRD
laozc Dec 4, 2021
8e86b57
Introduce more explicit Condition types
hiddeco Jul 30, 2021
c50a170
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
65dd61e
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
c278501
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
2d88469
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
e7f0406
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
a722ec7
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
0ae7758
Fixes PR comments
laozc Dec 7, 2021
f1302d8
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
795b83d
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
f82a906
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
6f34ea8
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
317cd44
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
4814abc
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
625f850
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
6931fed
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Dec 11, 2021
4dd7799
Update go.sum
laozc Dec 11, 2021
e9d5353
Introduce more explicit Condition types
hiddeco Jul 30, 2021
10a8cc4
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
c1668ef
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
e5806c3
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
adb131c
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
2b7a308
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
e82a472
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
b65e9bf
Add internal packages error and result
darkowlzz Dec 19, 2021
a956216
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
27455c4
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
009acb6
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
421f50d
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
066f74a
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
18fd284
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
bd0ee6e
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
878856c
bucket: Add more reconciler design improvements
darkowlzz Dec 20, 2021
f912cbf
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Jan 6, 2022
c2793ef
Introduce more explicit Condition types
hiddeco Jul 30, 2021
d7e0faa
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
7d752b7
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
d9ad873
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
1c25c8c
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
f927c76
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
34b5dcf
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
45484ef
api: Embed runtime.Object in Source interface
darkowlzz Jan 19, 2022
c3e2087
Add internal packages error and reconcile
darkowlzz Dec 19, 2021
8908d80
Use runtime-events-fixes branch temporarily
darkowlzz Jan 16, 2022
991692f
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
34b27a0
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
3105ba9
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
cf7dca0
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
af946e7
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
8dd51ab
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
4a1818d
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
a1e5067
bucket: Add more reconciler design improvements
darkowlzz Dec 20, 2021
f73df89
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Jan 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion api/v1beta1/bucket_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ const (
// BucketSpec defines the desired state of an S3 compatible bucket
type BucketSpec struct {
// The S3 compatible storage provider name, default ('generic').
// +kubebuilder:validation:Enum=generic;aws;gcp
// +kubebuilder:validation:Enum=generic;aws;gcp;azure
// +kubebuilder:default:=generic
// +optional
Provider string `json:"provider,omitempty"`
Expand Down Expand Up @@ -86,6 +86,7 @@ const (
GenericBucketProvider string = "generic"
AmazonBucketProvider string = "aws"
GoogleBucketProvider string = "gcp"
AzureBlobProvider string = "azure"
)

// BucketStatus defines the observed state of a bucket
Expand Down
96 changes: 74 additions & 22 deletions config/crd/bases/source.toolkit.fluxcd.io_buckets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.5.0
controller-gen.kubebuilder.io/version: v0.4.1
creationTimestamp: null
name: buckets.source.toolkit.fluxcd.io
spec:
Expand Down Expand Up @@ -35,28 +35,42 @@ spec:
description: Bucket is the Schema for the buckets API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: BucketSpec defines the desired state of an S3 compatible bucket
description: BucketSpec defines the desired state of an S3 compatible
bucket
properties:
accessFrom:
description: AccessFrom defines an Access Control List for allowing cross-namespace references to this object.
description: AccessFrom defines an Access Control List for allowing
cross-namespace references to this object.
properties:
namespaceSelectors:
description: NamespaceSelectors is the list of namespace selectors to which this ACL applies. Items in this list are evaluated using a logical OR operation.
description: NamespaceSelectors is the list of namespace selectors
to which this ACL applies. Items in this list are evaluated
using a logical OR operation.
items:
description: NamespaceSelector selects the namespaces to which this ACL applies. An empty map of MatchLabels matches all namespaces in a cluster.
description: NamespaceSelector selects the namespaces to which
this ACL applies. An empty map of MatchLabels matches all
namespaces in a cluster.
properties:
matchLabels:
additionalProperties:
type: string
description: MatchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
description: MatchLabels is a map of {key,value} pairs.
A single {key,value} in the matchLabels map is equivalent
to an element of matchExpressions, whose key field is
"key", the operator is "In", and the values array contains
only "value". The requirements are ANDed.
type: object
type: object
type: array
Expand All @@ -70,7 +84,10 @@ spec:
description: The bucket endpoint address.
type: string
ignore:
description: Ignore overrides the set of excluded patterns in the .sourceignore format (which is the same as .gitignore). If not provided, a default will be used, consult the documentation for your version to find out what those are.
description: Ignore overrides the set of excluded patterns in the
.sourceignore format (which is the same as .gitignore). If not provided,
a default will be used, consult the documentation for your version
to find out what those are.
type: string
insecure:
description: Insecure allows connecting to a non-TLS S3 HTTP endpoint.
Expand All @@ -85,12 +102,14 @@ spec:
- generic
- aws
- gcp
- azure
type: string
region:
description: The bucket region.
type: string
secretRef:
description: The name of the secret containing authentication credentials for the Bucket.
description: The name of the secret containing authentication credentials
for the Bucket.
properties:
name:
description: Name of the referent
Expand All @@ -99,7 +118,8 @@ spec:
- name
type: object
suspend:
description: This flag tells the controller to suspend the reconciliation of this source.
description: This flag tells the controller to suspend the reconciliation
of this source.
type: boolean
timeout:
default: 20s
Expand All @@ -114,20 +134,24 @@ spec:
description: BucketStatus defines the observed state of a bucket
properties:
artifact:
description: Artifact represents the output of the last successful Bucket sync.
description: Artifact represents the output of the last successful
Bucket sync.
properties:
checksum:
description: Checksum is the SHA256 checksum of the artifact.
type: string
lastUpdateTime:
description: LastUpdateTime is the timestamp corresponding to the last update of this artifact.
description: LastUpdateTime is the timestamp corresponding to
the last update of this artifact.
format: date-time
type: string
path:
description: Path is the relative file path of this artifact.
type: string
revision:
description: Revision is a human readable identifier traceable in the origin source system. It can be a Git commit SHA, Git tag, a Helm index timestamp, a Helm chart version, etc.
description: Revision is a human readable identifier traceable
in the origin source system. It can be a Git commit SHA, Git
tag, a Helm index timestamp, a Helm chart version, etc.
type: string
url:
description: URL is the HTTP address of this artifact.
Expand All @@ -139,23 +163,45 @@ spec:
conditions:
description: Conditions holds the conditions for the Bucket.
items:
description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, type FooStatus struct{ // Represents the observations of a foo's current state. // Known .status.conditions.type are: \"Available\", \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
description: "Condition contains details for one aspect of the current
state of this API Resource. --- This struct is intended for direct
use as an array at the field path .status.conditions. For example,
type FooStatus struct{ // Represents the observations of a
foo's current state. // Known .status.conditions.type are:
\"Available\", \"Progressing\", and \"Degraded\" // +patchMergeKey=type
\ // +patchStrategy=merge // +listType=map // +listMapKey=type
\ Conditions []metav1.Condition `json:\"conditions,omitempty\"
patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`
\n // other fields }"
properties:
lastTransitionTime:
description: lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
description: lastTransitionTime is the last time the condition
transitioned from one status to another. This should be when
the underlying condition changed. If that is not known, then
using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: message is a human readable message indicating details about the transition. This may be an empty string.
description: message is a human readable message indicating
details about the transition. This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
description: observedGeneration represents the .metadata.generation
that the condition was set based upon. For instance, if .metadata.generation
is currently 12, but the .status.conditions[x].observedGeneration
is 9, the condition is out of date with respect to the current
state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
description: reason contains a programmatic identifier indicating
the reason for the condition's last transition. Producers
of specific condition types may define expected values and
meanings for this field, and whether the values are considered
a guaranteed API. The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
Expand All @@ -168,7 +214,11 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase. --- Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be useful (see .node.status.conditions), the ability to deconflict is important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
description: type of condition in CamelCase or in foo.example.com/CamelCase.
--- Many .condition.type values are consistent across resources
like Available, but because arbitrary conditions can be useful
(see .node.status.conditions), the ability to deconflict is
important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand All @@ -181,14 +231,16 @@ spec:
type: object
type: array
lastHandledReconcileAt:
description: LastHandledReconcileAt holds the value of the most recent reconcile request value, so a change can be detected.
description: LastHandledReconcileAt holds the value of the most recent
reconcile request value, so a change can be detected.
type: string
observedGeneration:
description: ObservedGeneration is the last observed generation.
format: int64
type: integer
url:
description: URL is the download link for the artifact output of the last Bucket sync.
description: URL is the download link for the artifact output of the
last Bucket sync.
type: string
type: object
type: object
Expand Down
Loading