Add proposal for new API v1alpha1.ScyllaDBDatacenter

[zimnx]

This proposal aims to introduce new API called v1alpha1.ScyllaDBDatacenter that will eventually replace existing v1.ScyllaCluster. The new API offers the same features as existing one, but it will contain fixes for mistakes we did in the past, as well as several UX improvements.

This PR is a place for discussion. If you find something wrong or something that could be improved within current API please raise it here.

Resolves #1978

[zimnx]

Marking as WIP after offline discussion to prepare API for new changes.

[zimnx]

No longer WIP.

[rzetelskik]

Left a couple of comments.
Please review the code snippets for inconsistent indentations, there's a couple of places where they're off.

Also I haven't noticed any mentions of a deprecation plan. Is the existing CRD going to be deprecated at some point? If so, how long are we going to keep it, when are we going to get rid of the conversion controller etc. Otherwise it should be mentioned in the doc either way imo.

[tnozicka]

Thanks for sending the proposal. A few comments to start with, I'll continue tomorrow.

[tnozicka]

racks/zones could have different machine types or hardware - maybe this should be on the rack level - i guess a similar point can be made for some other fields on this level

[zimnx]

I don't think sysctls should be part of ScyllaDatacenter at all. These should belong to NodeConfig which sets up the nodes. I added them to be backward compatible.

Maybe we should get rid of it, the same way we got rid of Manager tasks and until a better alternative is available, have a support via annotation? WDYT?
We need to have any support because Scylla usually requires aio-max-nr to be increased.

[tnozicka]

I haven't look deep enough to see whether some sysctls make sense per container, but I see that the majority should be per node. So maybe this would go away with #749. I agree that internal annotation let's us defer the decision. (Btw. it would be good to add all those reasons to the annotation const definitions, so we know when we can drop it.)

[tnozicka]

I suppose we should file issues to track deprecation of fields that you are moving to annotation, unless they already have one like hostNetworking.

[zimnx]

Once we have everything finalized I will create issues for missing parts

[tnozicka]

last nits / typos, rest lgtm

thanks for the updates

[tnozicka]

lgtm
/assign rzetelski

[scylla-operator-bot]

@tnozicka: GitHub didn't allow me to assign the following users: rzetelski.

Note that only scylladb members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

lgtm
/assign rzetelski

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rzetelskik]

/lgtm
Thanks!

[scylla-operator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rzetelskik, tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment