feat: dataplane self-registration

[ndr-brt]

What this PR changes/adds

Give the data-plane "self registration" capabilities, through:

• data-plane-self-registration extension, that will obtain the supported src/dest/transfer types and call the data plane selector api
• data-plane-selector-control-api extension, that offers endpoints to permit the data-planes to interact with the control-plane, particularly, they offer "registration" and "unregistration" (that will provided in a subsequent issue/PR)

some diff is caused by moving stuff around and replacing the swagger plugin string with the reference to the version catalog.

Why it does that

simplify data-plane management

Further notes

• now the destination type is not used anymore to select dataplanes, because transferType already represent the handled types and it's already mandatory. in fact, the destination types field could be removed from DataPlaneInstance (upcoming issue)
• addDataplane endpoint on the management API has been deprecated (already removed usage from E2E tests)
• what was turnCount used for? now it isn't used by anyone, I deprecated it.
• signaling api context looks to be not necessary, their api should be exposed on the control one (upcoming issue).

Linked Issue(s)

Closes #2534

Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 77.27273% with 25 lines in your changes are missing coverage. Please review.

Project coverage is 75.04%. Comparing base (7f20ba5) to head (64fdf03).
Report is 254 commits behind head on main.

Files	Patch %	Lines
...plane/selector/RemoteDataPlaneSelectorService.java	50.00%	8 Missing and 5 partials ⚠️
...ipse/edc/junit/extensions/EdcRuntimeExtension.java	0.00%	3 Missing ⚠️
...ane/selector/DataPlaneSelectorClientExtension.java	87.50%	1 Missing ⚠️
...lector/control/api/DataPlaneInstanceValidator.java	87.50%	1 Missing ⚠️
...ector/control/api/DataplaneSelectorControlApi.java	0.00%	1 Missing ⚠️
...rol/api/DataplaneSelectorControlApiController.java	94.11%	1 Missing ⚠️
...trol/api/DataplaneSelectorControlApiExtension.java	85.71%	1 Missing ⚠️
...gistration/DataplaneSelfRegistrationExtension.java	94.73%	1 Missing ⚠️
.../dataplane/api/DataPlaneSignalingApiExtension.java	0.00%	1 Missing ⚠️
...JsonObjectFromDataFlowStartMessageTransformer.java	80.00%	0 Missing and 1 partial ⚠️
... and 1 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4166      +/-   ##
==========================================
+ Coverage   71.74%   75.04%   +3.29%     
==========================================
  Files         919     1012      +93     
  Lines       18457    20547    +2090     
  Branches     1037     1160     +123     
==========================================
+ Hits        13242    15419    +2177     
+ Misses       4756     4620     -136     
- Partials      459      508      +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[paullatzelsperger]

the turnCount was originally intended to track how often each data plane was selected to serve a transfer, to implement a round-robin-like selector strategy.
Together with the turnCount I would put the entire selection strategy idea up for discussion - it seems dataplanes are selected by capability much more than by some high-level load-balancing strategy

[paullatzelsperger]

how about returning a Result<R> here and evaluating that in client code, e.g. the addInstance method?

[ndr-brt]

totally agree, given the size of the PR I added that to my TODO list, but let's see if doing that will help me solve the other 2 comments (definitely it's not gonna help me out 😅 )

[jimmarino]

the turnCount was originally intended to track how often each data plane was selected to serve a transfer, to implement a round-robin-like selector strategy. Together with the turnCount I would put the entire selection strategy idea up for discussion - it seems dataplanes are selected by capability much more than by some high-level load-balancing strategy

Yes, from a control plane standpoint, that type of load balancing should be performed by the data plane "instance" itself. In other words, the control plane assumes a particular data plane instance is capable of handling those conditions.

[ndr-brt]

@paullatzelsperger solved two comments, but those tests need to be changed, because the integration scope is definitely too broad (they depend to the actual controller class!), I added that to my TODO list will do it in the next PRs.

@jimmarino @paullatzelsperger so definitely the turnCount can be removed here, also because it is a value that can only be set at creation (that currently is made on the management-api), so it's definitely not useful.

[paullatzelsperger]

@jimmarino @paullatzelsperger so definitely the turnCount can be removed here, also because it is a value that can only be set at creation (that currently is made on the management-api), so it's definitely not useful.

yep, makes sense

[mhellmeier]

To be honest, I wonder about this pull request. We volunteered to create such a DP self-registration extension. We followed your CONTRIBUTING guidelines and started different discussions on GitHub (see here or here) and on Discord (see here).
The result of all these discussions with @paullatzelsperger , @ndr-brt and @jimmarino was that such an extension should not be implemented in this core repository due to specific deployments:

TL;DR: either of those two solutions could be viable solutions, but due to their specific nature are not fit for general use, thus I don't think they should go in upstream EDC.

Source: @paullatzelsperger -- #2198 (reply in thread)

Since this project is an Eclipse open source project which should follow the open source best practices and the Eclipse Foundation Project Handbook: Where is this change in decision discussed and documented? Why isn't it discussed in the specific discussions linked above?

[paullatzelsperger]

First of all, the discussions you mentioned are (as of today) a year and a half old, and in the world of Software Development this can mean a drastic change of landscape which is exactly what happened here. Furthermore, you can't seriously expect us to remember and relate all current goings-on to age-old discussions.

Second, could you please provide any reference, which forces the maintainers of a project to get in contact with and align with the original authors of a previously discarded discussed idea, if said idea became viable at any time?

Third, this is quite a delicate are of code, so I'm sure you understand that we want this to be implemented by be someone, who is intimately familiar with the code base and has a proven track record of stellar contributions, for example, a committer. So the chances of getting something like this accepted for someone without a track record of any sort are slim, to say the least.

Lastly as I'm sure you know, since you mentioned the EF Handbook and our Contributing guidelines, it generally remains at the discretion of the committers to accept or reject any contribution based on a variety of reasons.

So from my point of view, all this is perfectly aligned with OSS practices, even if you seem offended by this PR. I'm also going to lock this discussion now, because this PR is merged, so we're really beating a dead horse at this point.

[jimmarino]

Sorry, I need to chime in here.

First, you (@mhellmeier) mischaracterized what three committers (@jimmarino, @ndr-brt, and @paullatzelsperger) stated in the referenced discussions. The consensus among the committers was that any solution would need to be generalized and reconciled with work underway in the DSP specification (a.k.a. IDS). There were also requirements for idempotent behavior, which DPF was not equipped to support at the time.

In the roughly year and a half since DSP has been updated to include more explicit state transitions and data formats, and the Data Plane Signaling API was introduced. Moreover, we are close to drafting a Data Plane API standard based on the latter. All of these were prerequisites to this PR, all were extensively discussed, and all have been vetted by implementation experience. This is in line with our positions expressed in the above-referenced discussions.

I also don't understand how this project fails to follow Eclipse procedures. This PR was authored by a committer and was approved by two other committers. I would also have approved it, but I did not have time to finalize my review (I had no issues with it). Furthermore, the PR was the product of extensive discussions and collaboration among the committers. That's precisely how Eclipse projects should be run.

Given the volume of requests and work associated with this project, it is not reasonable to expect the committers to reach out or explain their actions to every potential contributor, interested party, or person who may have raised a question in the past. In order to stay informed and up-to-date, the best approach is to attend EDC office hours, monitor discussions, and read decision records.

[ndr-brt]

I think @jimmarino and @paullatzelsperger already covered the point extensively, I just wanted to add that the related issue (#2534) remained open for like more than a year (read: IT ages :) ), so the idea to have this feature implemented was never discarded in any way, at the end it made it, so I guess it could be considered an happy ending for everyone (a committer contribution is typically less expensive in terms of time both on the "code" side and on the "review" side).