Automatize Maven repository management

[agazzarini]

There are some modules, mainly those related with the search platform bindings, where we provide a lot of versions (one for each version of the target platform).

All those versions need to be available in the maven repository and, most important, when a change (e.g. structural change, bug fix) occurs in one version, all the others must be updated and re-uploaded in the repository.

This is a process that at the moment needs to be done manually:

• loop through all the supported versions of a given artefact (e.g. maven-solr-archetype, maven-solr-platform)
• change the version
• build
• check
• install
• move the generated artefact in the mvn-repo branch

the last step is done through the Github plugin which is already configured (note it's working only if you have a valid Sease account) and cannot be used massively because Github stops requests that exceed a given size.

So at the moment the steps above are repeated for all versions (locally); once all the artefacts (the artefacts of all versions) are built, the generated files are grouped in small chunk and uploaded in the mvn-repo.

This is done manually so it requires a looot of time; we should investigate how to automatise the whole task (or a part of it).

Any feedback is warmly welcome

[agazzarini]

This issue became blocking, because a lot of other open issues are related with it.
Basically we'd need something which automatically takes care about updating all the search-platform related modules.

The latest issue related with that topic is #91: if we add a new lifecycle method on the SearchPlatform interface and we use that in the Engine main controller loop, we will need to update all the ElasticSearch & Solr concrete bindings (all versions!) for including that method implementation, otherwise if someone downloads a past version (without that method), he will get a NoSuchMethodError

@mattoscln any idea about how to automatise this process?

[mattflax]

Hi Andrea,

I've done some investigation, and think I have worked out an approach that should work.

• change the search-platform-impl versions to match the project version, rather than the search engine version.
• change the archetype-resources/pom.xml to include the search engine client libraries in the plugin dependencies:

<build><plugins><plugin>
  <groupId>io.sease</groupId>
  <artifactId>rre-maven-external-solr-plugin</artifactId>
  <version>1.0</version>
  <dependencies>
    <dependency>
      <groupId>org.apache.solr</groupId>
      <artifactId>solr-solrj</artifactId>
      <version>${solrVersion}</version>
    </dependency>
  </dependencies>
  ... everything else ...
</plugin></plugins></build>

• change the scope of the client library dependency in the search-platform-impl pom.xml to provided.

Obviously there are lots of version properties that need changing, but I tried this out locally and it worked, building the external Solr archetype for Solr 7.7.0. It removes the need to maintain one client library version per search engine release, so if you do go ahead and change the search platform interface, you can just change the project version and release a single set of libraries, rather than one for each search engine client version.

How does that sound as an approach?

[agazzarini]

Wow @mattoscln, that sounds really reasonable! Thanks a lot for your input

[agazzarini]

@mattoscln I've created a PR (which includes at the moment only the "Solr" side). When and if you've time, could you please have a look?

#99

[mattflax]

Looks mostly good. However, I can't get it to build at the moment - the SolrClientManagerTest is failing when I build using Maven (it's fine running it directly in IntelliJ).

[mattflax]

I've got a fix - see this gist: https://gist.github.com/mattflax/f6cb42f29875cca8d458d800167e6d1f
(It also includes a default log4j config file to go in src/test/resources to avoid complaints).

One thing that is a concern - the Maven build output states that both LATEST and RELEASE are deprecated and may be removed in a future version. Maybe we should tie the project to the latest current SolrJ release, rather than using LATEST?

[agazzarini]

Probably it has to do with the Maven version. I'm using Maven 3.3.9 and it works fine; but I think applying your patch is a good idea.

About the Solr version: ok, I'll hard-code 8.2.0

[agazzarini]

Just applied: unfortunately I think we cannot support all Solr version with just one build.
Changes that have been applied to solr-core between Solr4 and Solr5 (and some Solr6 version) make this approach impossible.

I have been able to get a green build using:

• Solr 4 - 5: failure in "RRE - Embedded Apache Solr platform binding" due to incompatible method signature
• Solr 6.0.0 - 6.4.0: build failure due to incompatible method signature
• Solr 6.5.0 or greater: OK
• Solr 7.x: OK, sometimes, very rarely, I got an intermittent test failure (connection timeout)
• Solr 8.x: OK

Now, considering Solr 7 will go in EOL very soon, I think we could say we are not supporting the versions listed in the first two points above.

[mattflax]

I think so long as the external Solr connector keeps working (and it seems to, having done a quick check against Solr 5.3.1), this is a good approach. I imagine there are plenty of people out there running old versions of Solr that they don't want to upgrade, but if they can test against an existing installation, it should be fine.

I suspect Elasticsearch will bring a new set of problems, when you get to it! :-)

[agazzarini]

Last update: on Solr side the current approach works at build/compile time, but unfortunately it has several problems (NoSuch*Error) at runtime. I'm trying to figure out a way to combine everything

[agazzarini]

@mattflax at the end we decided to build a branch for each ES/Solr version. The drawback is the redundancy introduced in shared modules (which need to be kept aligned, with some annoying cherry-picking stuff in case of change).
At the moment the maven repository has been re-uploaded from scratch; we have, as I was saying, one branch for each major version of Solr and Elasticsearch and last, the master use the latest Solr and Elasticsearch.

This approach requires a bit of work in case of fix (especially on shared modules) but at the end it seems it's the quicker way to manage multiple versions without going crazy with the dependencies intersection.