Provide support for custom plugin repositories

[piotrtomiak]

Building Vue.j plugin from sources on master using gradle requires IDEA nightly builds. It was really cumbersome to get everything to compile, so here is PR with support for nightly builds. Example configuration:

repositories {
  mavenCentral()
  maven("https://jetbrains.bintray.com/intellij-third-party-dependencies")
  maven("https://jetbrains.bintray.com/jediterm")
  maven("https://cache-redirector.jetbrains.com/www.myget.org/F/rd-snapshots/maven")
}

intellij {
  pluginName = "Vue.js"
  type = "IU"
  updateSinceUntilBuild = false
  nightlyBuild = "https://..."
  setPlugins("JavaScriptLanguage", "JSIntentionPowerPack", "JavaScriptDebugger", "CSS", "HtmlTools",
             "org.jetbrains.plugins.sass", "org.jetbrains.plugins.less", "org.jetbrains.plugins.stylus",
             "org.intellij.plugins.postcss:${ideaDependency.buildNumber}",
             "com.jetbrains.plugins.Jade:${ideaDependency.buildNumber}",
             "intellij.prettierJS:${ideaDependency.buildNumber}")
}

dependencies {
  testImplementation("com.jetbrains.intellij.javascript:javascript-test-framework:${intellij.ideaDependency.buildNumber}")
  testImplementation("com.jetbrains.intellij.copyright:copyright:${intellij.ideaDependency.buildNumber}")
}

You can of course fix the URL to a particular nightly build. Tested runIde target, which works perfectly.

[piotrtomiak]

@zolotov I haven't bumped plugin version nor added documentation. Let me know if any of these is needed.

[zolotov]

Thanks. It looks like a solution for #15

I have concerns about the API, though:

1. it adds yet another option to intellij extension. ideally I'd get rid of the extension completely, I believe all the settings should be configured in particular tasks or in the dependencies block
2. it reveals intellij build server specifics and is not discoverable or usable for external users
3. there is also intellij.pluginsRepo settings which is somehow related to the changes

Probably, it makes sense to convert intellij.pluginsRepo into an array setting and automatically detect whether it's an plugins.xml-based repository or maven based (e.g. by downloading plugins.xml files every time).

[zolotov]

I haven't bumped plugin version nor added documentation. Let me know if any of these is needed.

No problem, I usually do that right before publishing. I also think that feature should go with 0.5.0

[piotrtomiak]

@zolotov I understand all of your concerns, especially point #2. The intellij.pluginsRepo requires Maven repository, that's why I didn't want to reuse it. Let me know if you think we should change something or abandon the idea.

[zolotov]

Let me know if you think we should change something or abandon the idea.

I've posted the suggestion :)

Probably, it makes sense to convert intellij.pluginsRepo into an array setting and automatically detect whether it's an plugins.xml-based repository or maven based (e.g. by downloading plugins.xml files every time).

[piotrtomiak]

intellij.pluginsRepo into an array setting and automatically detect whether it's an plugins.xml-based repository or maven based

OK, sounds good and more generic! That would however solve just a half of the problem. The other is downloading the latest IDE build. Should we do the same and simply detect if we are dealing with nightly?

[zolotov]

The other is downloading the latest IDE build. Should we do the same and simply detect if we are dealing with nightly?

Sorry, didn't get the problem. Could you elaborate,please?

[piotrtomiak]

@zolotov Nevermind. I replied too quickly. I'll look into applying your suggestions next week. Do you have an example of a corporate repo layout, so that I can support that as well?

[zolotov]

@piotrtomiak as far as I understand, artifacts of nightly teamcity builds is an example of corporate repo layout, so it's just an updatePlugins.xml file and plugin distribution nearby.

[piotrtomiak]

@zolotov I've been trying to figure out the best way to keep backward compatibility and provide support for the corporate layout. How about something like this:

intellij {
  ...
  pluginsRepo {
     marketplace() // maven("https://cache-redirector.jetbrains.com/plugins.jetbrains.com/maven")
     maven("https://corpo.net/intellij/maven")
     custom("https://corpo.net/plugins/intellij/updatePlugins.xml")
     custom("https://other-corpo.org/plugins/ij/plugins.xml")
  }
  ...
}

Now, the question is about intellijRepo property and nightlies. Should we require an url to a nightly build snapshot, or should we figure that out from the list of plugins repos. In the first case, an example configuration for nightly would be something like this:

val nightlyBuildUrl = "https://..."
intellij {
  ...
  intellijRepo = "${nightlyBuildUrl}"
  pluginsRepo {
     maven("${nightlyBuildUrl}/maven-artifacts")
     custom("${nightlyBuildUrl}/IU-plugins/plugins.xml")
  }
  ...
}

Let me know what you think!

[zolotov]

How about something like this:

Love it

Should we require an url to a nightly build snapshot

I don't think url should be required. It's possible to use local directory as an intellij build without a nightly build. Probably I misunderstood the suggestion.

UPD: ah, you want to download Installers right from the buildserver too. Is there a need for that? From the JB network you can access to them without patching an intellijRepo url.

[piotrtomiak]

Love it

Awesome! I'll go with it than :)

[zolotov]

LGTM, would be nice to have a test for custom repository. see org.jetbrains.intellij.DownloadIntelliJPluginsSpec

[piotrtomiak]

LGTM, would be nice to have a test for custom repository. see org.jetbrains.intellij.DownloadIntelliJPluginsSpec

I'll need to include an xml and single jar in a test directory, unless you are aware of a public and stable custom plugin repository. Sounds good?

[zolotov]

I'll need to include an xml and single jar in a test directory, unless you are aware of a public and stable custom plugin repository. Sounds good?

@piotrtomiak sure

[zolotov]

Thank you very much! Merged. I’ll publish 0.5 as soon as I investigate problems with kotlin-serialization 1.4

[Undin]

Awesome!
@piotrtomiak thank you!
Сan't wait for these changes in the release

[rachelcarmena]

Hi! Please, is this feature deployed? Thanks in advance!

[zolotov]

it's in 0.5-SNAPSHOT only so far, will publish as soon as #537 gets fixed

[zolotov]

here is the instruction on how to use snapshot version https://github.com/JetBrains/gradle-intellij-plugin#snapshot-version

[rachelcarmena]

Awesome, thank you so much @zolotov 🙌

Gradle works and accepts the new DSL however It's not finding the plugin.

I'm following this schema: https://jetbrains.org/intellij/sdk/docs/basics/getting_started/update_plugins_format.html#format-of-updatepluginsxml-file

though the test is using this schema: https://github.com/piotrtomiak/gradle-intellij-plugin/blob/9e6031e05d9ad5656c934410c497eba5aab88c7f/src/test/resources/custom-repo/plugins.xml

Please, did I check the wrong documentation for the private repository? Thanks in advance!!

[zolotov]

Please, did I check the wrong documentation for the private repository?

That is the right documentation.

Probably running gradle with --info will give some clues. Please attach the output.

cc @piotrtomiak

[rachelcarmena]

Thank you so much @zolotov !

No clues in the console output with the different Gradle options.

After:

[gradle-intellij-plugin :ide-plugin] Configuring plugin io.arrow-kt.arrow:1.4.10-SNAPSHOT

it appears:

Could not determine the dependencies of task ':ide-plugin:prepareSandbox'.
> Cannot resolve plugin io.arrow-kt.arrow version 1.4.10-SNAPSHOT

and I think the reason is looking for download-url:

        String downloadUrl = pluginsXml.category."idea-plugin"
                .find { it.id.text().equalsIgnoreCase(id) && it.version.text() == version }
                ?."download-url"?.text()

and nothing happens if it's not found:

        if (downloadUrl == null) return null

download-url doesn't exist in the schema that I followed: https://jetbrains.org/intellij/sdk/docs/basics/getting_started/update_plugins_format.html#format-of-updatepluginsxml-file

Thanks again!!

[rachelcarmena]

And <idea-plugin> doesn't exist but <plugin>.

[piotrtomiak]

@rachelcarmena It turns out that I was following a different schema. I will update the code to support both.

[rachelcarmena]

That would be awesome @piotrtomiak , thank you so much!! 👏 💃