Solution for unstable update sites

[hohwille]

Actually the design of the plugin ecosystem of most tools or IDEs (esp. Eclipse) is a disaster:

• Plugins are provided decentralized via URLs
• When the plugin is installed at a specific point in time it is loaded from that URL
• The maintainer of that URL is in full control of the availability and the content behind that URL.
• If the service gets broken, obviously the plugin can not be downloaded.
• Even worse if existing content gets updated, the result is not reproducible anymore.
• The major problem is that exactly the latter happens all the time.
• All famous IDEs do not have a clear versioning concept: While the plugin artifact itself is typically versioned, the request to install a plugin can not specify a version but always downloads the "latest" version. If at all some kind of versioning for stability is in place it happens via the URL itself so for different major versions different URLs are provided.
• A concrete example are the JBoss Plugins available at this update site: http://download.jboss.org/jbosstools/updates/webtools/photon/
• While at some point of time the plugin installation worked, at a later point in time all the plugins had been updated and stopped working with the configured eclipse version of that project.
• Even worse after fixing this in the project settings it worked again for some time and now is entirely broken again as RedHad published new plugin versions that are entirely broken, had expired certificates, etc.
• As a result the entire project with hundreds of developers was blocked and it was impossible to rebuild the IDE with Eclipse and the required projects from scratch.
• This is just an example and we had the same story also for other Eclipse Plugins and also for other tools like SonarQube, IntelliJ, or VS Code.

The perfect solution would obviously to change all these tools to have a reliable and reproducible plugin installations:

• Ensure that all plugins get maintained in a central repository (like maven)
• Ensure that every artifact in this repo is strictly versioned and once a version gets deployed it can never be changed anymore.
• Ensure that a plugin (group) installation can specify an exact version (if omitted the latest version is used). So if a specific version is given you always get the same reproducible result at any point in time in the future.

However, the sad fact is chances to make this happen are practically zero!

There is already some kind of workaround offered by devonfw-ide to address the problem:

We already offer ability for a project specific custom repo. This can be used to archive a fully build IDE including all required plugins into a *.tar.gz file and provide it on your own server in such a custom repo. Then devonfw-ide would not download the IDE vanilla release and then install all the plugins but instead simply download and unpack that complete archive with IDE + plugins.

This approach has several drawbacks:

• You would need to create the archive *.tar.gz package for each OS and architecture what is quite a lot of effort.
• Whenever you want to update the IDE version itself or change any plugin, you have to restart and rebuild all the archives (as a new version) again.

Therefore the goal of this ticket is to offer a new solution:

It should be possible to create your own backup mirror of a plugin update site (this story will only address Eclipse update sites to get started):

• download all contents from the update site to some folder
• simply upload that to your own project specific webserver to a folder path containing the name of the plugin and a version (e.g. timestamp of the day of the backup download).
• edit the plugin *.properties file for the plugin and change the plugin update site URL to that one on your own webserver.

As a result the plugin should not change anymore and therefore the installation shall be stable and reproducible (as long as you as a maintainer of that webserver do your job properly).

Steps to proceed and acceptance criteria:

1. manually try to find a way to do the above steps for a specific plugin update site of eclipse
2. create a new guide (*.asciidoc documentation file) for devonfw-ide describing the process from 1.
3. where suitable consider to use as much automation as possible (e.g. using a crawler to download the update site recursively, create the folder structure with the plugin and date/timestamp of today in the script, maybe even extend eclipse commandlet with an option to do the download automatically such as devon eclipse download-plugin «plugin-id|update-site-url»).
4. See DoD

[alfeilex]

Takeaways from this issue:

• the plugin_mirror_url doesn't work with default plugins in eclipse/plugins/ (e.g. devon add-plugin checkstyle). The reason is that the url (${2}) is set manually in the script. See

  ide/scripts/src/main/resources/scripts/command/eclipse

  Lines 175 to 208 in 61e4e0a

  	function doAddPlugin() {
  	if [ -z "${2}" ]
  	then
  	if [ "${1}" = "startexplorer" ]
  	then
  	doInstallEclipsePlugin "de.bastiankrol.startexplorer.feature.feature.group" "https://basti1302.github.io/startexplorer/update/"
  	elif [ "${1}" = "cobigen" ]
  	then
  	doInstallEclipsePlugin "com.devonfw.cobigen.eclipse.feature.feature.group,com.devonfw.cobigen.tempeng-freemarker,com.devonfw.cobigen.htmlplugin,com.devonfw.cobigen.javaplugin,com.devonfw.cobigen.jsonplugin,com.devonfw.cobigen.openapiplugin,com.devonfw.cobigen.propertyplugin,com.devonfw.cobigen.textmerger,com.devonfw.cobigen.tsplugin,com.devonfw.cobigen.tempeng-velocity,com.devonfw.cobigen.xmlplugin" "https://devonfw.com/cobigen/updatesite/stable/"
  	elif [ "${1}" = "terminal" ]
  	then
  	doInstallEclipsePlugin "org.eclipse.tm.terminal.feature.feature.group,org.eclipse.tm.terminal.view.feature.feature.group,org.eclipse.tm.terminal.control.feature.feature.group,org.eclipse.tm.terminal.connector.ssh.feature.feature.group,org.eclipse.tm.terminal.connector.telnet.feature.feature.group" "http://download.eclipse.org/tm/terminal/marketplace"
  	elif [ "${1}" = "anyedit" ]
  	then
  	doInstallEclipsePlugin "AnyEditTools.feature.group" "https://raw.githubusercontent.com/iloveeclipse/plugins/latest/"
  	elif [ "${1}" = "regexutil" ]
  	then
  	doInstallEclipsePlugin "com.ess.regexutil.feature.group" "http://regex-util.sourceforge.net/update/"
  	elif [ "${1}" = "templatevariables" ]
  	then
  	doInstallEclipsePlugin "net.sf.mmm.eclipse.templatevariables.feature.feature.group" "https://m-m-m.github.io/eclipse-templatevariables/latest"
  	elif [ "${1}" = "spotbugs" ]
  	then
  	doInstallEclipsePlugin "com.github.spotbugs.plugin.eclipse.feature.group" "https://spotbugs.github.io/eclipse/"
  	elif [ "${1}" = "checkstyle" ]
  	then
  	doInstallEclipsePlugin "net.sf.eclipsecs.feature.group" "https://checkstyle.org/eclipse-cs-update-site"
  	elif [ "${1}" = "github" ]
  	then
  	doInstallEclipsePlugin "org.eclipse.mylyn.github.feature.feature.group" "https://download.eclipse.org/egit/github/updates/"
  	elif [ "${1}" = "soapui" ]
  	then
  	doInstallEclipsePlugin "com.eviware.soapui.soapui_feature.feature.group" "http://www.soapui.org/eclipse/update/"
  	else

• Some functions throw errors, but do not indicate that the execution was successful. Maybe we could insert a doSuccess there and generalize this approach. E.g. here

  ide/scripts/src/main/resources/scripts/command/eclipse

  Lines 34 to 40 in 61e4e0a

  	if [ "${result}" != 0 ]
  	then
  	echo
  	doEchoAttention "Installation of plugin ${plugin} failed. Please try to install manually or rerun setup (devon eclipse setup)."
  	echo
  	fi
  	return "${result}"