#832: Solution for unstable update sites

CHANGELOG.asciidoc

@@ -7,6 +7,7 @@ This file documents all notable changes to https://github.com/devonfw/ide[devonf
 Release in process:
 
 * https://github.com/devonfw/ide/issues/751[#751]: Documentation of functions
+* https://github.com/devonfw/ide/issues/832[#832]: Solution for unstable update sites
 
 The full list of changes for this release can be found in https://github.com/devonfw/ide/milestone/32?closed=1[milestone 2022.08.002].
 

documentation/eclipse.asciidoc

@@ -22,6 +22,7 @@ You may also supply additional arguments as `devon eclipse «args»`. These are
 |`ws-re[verse]`           |reverse merge changes from workspace into settings
 |`ws-reverse-add`         |reverse merge adding new properties
 |`create-script`          |create launch script for this IDE, your current workspace and your OS
+|`mirror «id» [«url»]`    |mirror the content of an update-site
 |=======================
 
 There are link:variables.asciidoc[variables] that can be used for Eclipse. These are explained by the following table:
@@ -41,13 +42,15 @@ In your link:settings.asciidoc[settings] git repository create a folder https://
 Here you can create a properties file for each plugin. This is an example https://github.com/devonfw/ide-settings/blob/master/eclipse/plugins/tmterminal.properties[tmterminal.properties]:
 ```
 plugin_url=http://download.eclipse.org/tm/terminal/marketplace
+plugin_mirror_url=https://my-server.com/terminal-2022-09-13
 plugin_id=org.eclipse.tm.terminal.feature.feature.group,org.eclipse.tm.terminal.view.feature.feature.group,org.eclipse.tm.terminal.control.feature.feature.group,org.eclipse.tm.terminal.connector.ssh.feature.feature.group,org.eclipse.tm.terminal.connector.telnet.feature.feature.group
 plugin_active=true
 ```
 
 The variables are defined as follows:
 
 * `plugin_url` defines the URL of the Eclipse update site of the plugin
+* `plugin_mirror_url` defines the URL of the Eclipse mirrored update site of the plugin. See xref:mirroring update sites[Mirroring Eclipse Update Site] 
 * `plugin_id` defines the feature group ID(s) to install. To install multiple features/plugins provide a coma-separated list of IDs. If you want to customize `devonfw-ide` with new plugins you can first install them manually and then go to `About Eclipse > Installation Details` then you can filter for your newly installed plugin and find the values in the `Id` column. Copy & paste them from here to make up your own custom config.
 * `plugin_active` is an optional parameter. If it is `true` (default) the plugin will be installed automatically during the project link:setup.asciidoc[setup] for all developers in your team. Otherwise, developers can still install the plugin manually via `devon eclipse add-plugin «plugin-name»` from the config file `settings/eclipse/plugins/«plugin-name».properties`. See the `settings/eclipse/plugins` folder for possible values of «plugin-name».
 
@@ -70,6 +73,29 @@ ECLIPSE_PLUGINS=("anyedit" "" "regexutil" "")
 ```
 Of course you may also mix plugin IDs with fully qualified plugins.
 
+=== mirroring update sites
+
+A common problem with eclipse plugins is that they are provided decentralized as so-called update sites via URLs.
+The maintainer of that URL is in full control of the availability and the content behind that URL.
+If the service gets broken, obviously the plugin can not be downloaded.
+Even worse, if existing content gets updated, the result is not reproducible anymore.
+
+While the plugin artifact itself is versioned, the request to install a plugin can not specify a version but always downloads the "latest" version from the update site.
+If at all some kind of versioning for stability is in place it happens via the URL itself so for different major versions different URLs are provided.
+A possible solution is to mirror the update site locally and then make it available on your own webserver.
+This way you always have access to the plugin version you need as a developer without being dependent on the plugin provider.
+Below is the process to mirror a plugin update-site.
+
+To mirror with only one paramter, you just need the `«plugin-id»` of an existing plugin in `${DEVON_IDE_HOME}/settings/eclipse/plugins` (e.g. checkstyle).
+Open any CLI in `${DEVON_IDE_HOME}` and run the following command.
+
+`devon eclipse mirror «plugin-id» [«url»]`
+
+This command will automatically mirror the content of an update site to a specific directory named by `«plugin-id»` together with the current date in `${DEVON_DOWNLOAD_DIR}/update-sites/` (e.g. checkstyle-2022-09-14).
+Afterwards, the folder can be uploaded to your own webserver and the URL can be put manually in `«plugin_mirror_url»` in the `«plugin-id».properties` file.
+This only works if a valid `plugin_url` is already set in the properties for the given `plugin_id` (see xref:plugins[plugins]).
+If you want to mirror an update site independently of `«plugin-id».properties`, you can enter an update site URL for the optional `«url»` parameter (e.g. https://checkstyle.org/eclipse-cs-update-site).
+
 == dictionary
 
 Eclipse already comes with a build-in spellchecker. This is very helpful when writing comments. The default settings of `devonfw-ide` ship with a project specific https://github.com/devonfw/ide-settings/blob/master/eclipse/project.dictionary[dictionary file] and according configurations to enable spellchecking and configuring this dictionary.

scripts/src/main/resources/scripts/command/eclipse

@@ -33,9 +33,7 @@ function doInstallEclipsePlugin() {
   result=${?}
   if [ "${result}" != 0 ]
   then
-    echo
     doEchoAttention "Installation of plugin ${plugin} failed. Please try to install manually or rerun setup (devon eclipse setup)."
-    echo
   fi
   return "${result}"
 }
@@ -95,22 +93,23 @@ function doSetup() {
 function doAddPluginFromConfig() {
   if [ -f "${1}" ]
   then
-    plugin_id=""
-    plugin_active="true"
-    plugin_url=""
-    doLoadProperties "${1}"
-    if [ -z "${plugin_id}" ] || [ -z "${plugin_url}" ]
+    doCheckProperties "${1}"
+    local url
+    if [ -n "${plugin_mirror_url}" ]
     then
-      doWarning "Invalid eclipse plugin config: ${file}"
-    elif [ "${plugin_active}" = "true" ] || [ -n "${2}" ]
+      url="${plugin_mirror_url}"
+    else
+      url="${plugin_url}"
+    fi
+    if [ "${plugin_active}" = "true" ] || [ -n "${2}" ]
     then
       # check if plugin successfully installed
       local plugin_fileprefix="${1/*\//}"
       plugin_fileprefix="${plugin_fileprefix/.properties/}"
       local install_file="${ECLIPSE_HOME}/.devon.${plugin_fileprefix}.installed"
       if [ ! -e "${install_file}" ]
       then
-        doInstallEclipsePlugin "${plugin_id}" "${plugin_url}"
+        doInstallEclipsePlugin "${plugin_id}" "${url}"
         result_rc=$?
         if [ "${result_rc}" = 0 ]
         then
@@ -254,6 +253,53 @@ function doImportEclipse() {
   "${ECLIPSE_HOME}/eclipse" -nosplash -vm "${ECLIPSE_JAVA_HOME}/bin/java" -application org.eclipse.ant.core.antRunner -buildfile "${DEVON_IDE_HOME}/scripts/lib_script/import.xml" -data "${WORKSPACE_PATH}" -keyring ~/.eclipse/.keyring -vm "${ECLIPSE_JAVA_HOME}/bin/java" -vmargs -DdevonImportPath="${importPath}" -DdevonImportWorkingSet="${importWorkingSets}" ${ECLIPSE_VMARGS} 
 }
 
+# $1: id
+# $2: optional plugin URL (update site)
+function doMirrorUpdateSite() {
+  if [ -z "${1}" ]
+  then
+    doFail "Parameter \"id\" is missing."
+  fi
+  local url
+  if [ -n "${2}" ]
+  then
+    url="${2}"
+  else
+    doCheckProperties "${1}"
+    url="${plugin_url}"
+  fi
+  local timestamp
+  timestamp=$(date +'%Y-%m-%d')
+  "${ECLIPSE_HOME}/eclipse" -nosplash -vm "${ECLIPSE_JAVA_HOME}/bin/java" -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source "${url}" -destination "${DEVON_DOWNLOAD_DIR}/update-sites/${1}-${timestamp}" -vmargs "${ECLIPSE_VMARGS}"
+  "${ECLIPSE_HOME}/eclipse" -nosplash -vm "${ECLIPSE_JAVA_HOME}/bin/java" -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source "${url}" -destination "${DEVON_DOWNLOAD_DIR}/update-sites/${1}-${timestamp}" -vmargs "${ECLIPSE_VMARGS}"
+  local result="${?}"
+  if [ "${result}" != 0 ]
+  then
+    doEchoAttention "Mirroring of the update-site ${url} failed. Please check the URL or rerun setup (devon eclipse setup)."
+  fi
+  return "${result}"
+}
+
+function doCheckProperties() {
+  local file="${1}"
+  if [ ! -f "${1}" ]
+  then
+    file="${SETTINGS_PATH}/eclipse/plugins/${1}.properties"
+  fi
+  if [ -f  "${file}" ]
+  then
+    plugin_id=""
+    plugin_active="true"
+    plugin_url=""
+    plugin_mirror_url=""
+    doLoadProperties "${file}"
+    if [ -z "${plugin_id}" ] || [ -z "${plugin_url}" ]
+    then
+      doFail "Invalid eclipse plugin config: ${file}"
+    fi
+  fi
+}
+
 # CLI
 if [ "${1}" = "-h" ] || [ "${1}" = "help" ]
 then
@@ -269,6 +315,7 @@ then
   echo " ws-reverse-add                  reverse merge adding new properties"
   echo " create-script                   create eclipse-${WORKSPACE} script if not already exists"
   echo " import «path» [«workingsets»]   import projects in «path» into current workspace. «workingsets» comma-separated list of workingsets (no spaces between ,), eg: ./myproject ws1,ws2"
+  echo " mirror «id» [«url»]             mirror the content of an update-site"
   exit
 fi
 ECLIPSE_HOME="${DEVON_IDE_HOME}/software/eclipse"
@@ -318,6 +365,11 @@ do
   then
     doImportEclipse "${2}" "${3}"
     exit ${?}
+  elif [ "${1}" = "mirror" ]
+  then
+    shift
+    doMirrorUpdateSite "${@}"
+    exit ${?}
   else
     doFail "Unknown argument: ${1}"
   fi