VS Code marketplace rate limit error

[garagatyi]

Description

VS Code marketplace impose rate limiting on access to the extensions binaries.
PR eclipse-che/che-plugin-broker#35 makes broker do retries and send pretty error to Che but still doesn't prevent blocking of workspace start because of rate limiting.
This often happens when user sets many VS Code extensions in his workspace configuration or several users share public IP which gets blocked when access rate is too high.
It seems VS Code itself bypasses this problem, but I haven't found how. We need to investigate how exactly it is done.
Another possible improvement is not to download extensions when workspace plugins list doesn't change, so provisioning could be cached.

Related issues:
#12831
#12762

Reproduction Steps

OS and version:

Diagnostics:

[garagatyi]

@l0rd it is problem we discussed today.
@ibuziuk @skabashnyuk @benoitf @evidolob FYI

[l0rd]

Full exception as reported by @benoitf:

{
"$id":"1",
"innerException":null,
"message":"Anonymous usage from IP address ***** has exceeded our rate limits and is being blocked. Anonymous access from this IP address will be unblocked within the next 1 minute. You can also consider logging in to get unblocked. Logged in users have higher limits. Learn more: https://go.microsoft.com/fwlink/?LinkId=823950.",
"typeName":"Microsoft.TeamFoundation.Framework.Server.RequestBlockedException,Microsoft.TeamFoundation.Framework.Server",
"typeKey":"RequestBlockedException",
"errorCode":0,
"eventId":3000
}

cc @slemeur

[l0rd]

The possible solution for this problem can range from a lazy caching mechanism (once we request one vs code extensions we don't request it anymore) to a vs code extension mirror (we pre-fetch all the extensions).

In the short term, to mitigate the problem, we could look at these solutions:

• embed the vs code extensions that are currently in the che-plugin-registry in their sidecar image
• embed the extensions in the che-plugin-registry docker image and change the meta.yml to point to it.
• look at some already existing alternative URLs to download the extension (i.e. the artifact produced by the CI/CD of the extensions)

[garagatyi]

Additional details for the assignee:
We have 10 VS Code plugins in the registry that are useful (I do not treat sonarlint extension as such).
4 of them are published by Red Hat.
We should check if any of plugins (especially non-redhat owned ones) are publishing github releases or host archive binaries anywhere else.

[apupier]

I also hit this issue having just 1 or 2 plugins (I reproduced several times with a mix of Camel, Java and XML) in my stack. this is quite annoying and it was not easy to find the root cause.

What about an even shorter-term solution which is reporting a better error message to user, inviting her/him to retry later pointing that there is a rate limit reached?

[fbricon]

The latest vsix have been "released" on github:

• vscode-yaml: https://github.com/redhat-developer/vscode-yaml/releases/download/0.4.0/redhat.vscode-yaml-0.4.0.vsix
• vscode-xml: https://github.com/redhat-developer/vscode-xml/releases/download/0.5.1/redhat.vscode-xml-0.5.1.vsix

[ibuziuk]

@skabashnyuk @garagatyi @l0rd guys, I'm not very fluent with plugin versioning update process. Could you please clarify the policy? e.g. both xml and yaml plugins used in our registry are obsolete. Should we create dedicated folders with versioning? So, far I created eclipse-che/che-plugin-registry#113 just for url switch

[garagatyi]

@ibuziuk problematic part is that we do not have proper ways for a user to use newer version of plugins in a workspace. So, we usually update just URL to the binary of the plugin.
But since VS Code plugins are properly versioned this seems a bad practice.
We should definitely invest in #12937

At this point of time I'm +1 to create new version for yaml and xml plugins.

[ibuziuk]

At this point of time I'm +1 to create new version for yaml and xml plugins.

@garagatyi @l0rd so are you ok if UD would display 'n' versions of plugin atm once we add latest versions to the registry?

[l0rd]

@ibuziuk yes that's ok. That's a UD problem and we are removing the list of plugins and editors from the create workspace panel in UD #12896 and moving it inside che-theia #12905.

[l0rd]

@ibuziuk since you have done the first PR can you add the newest version of JDT LS as well? Hence adding new meta.yaml.

[ibuziuk]

@l0rd sure thing will do
@fbricon @gorkem could you please give a hand with redhat-developer/vscode-java#892

[ibuziuk]

Some updates before updating plugin registry on prod:

• vscode-kubernetes-tools - PR sent Downloading vsix for kubernetes tools extention from GH release in order to avoid marketplace rate limit error che-plugin-registry#125 / issues Only preview releases contain vsix files (regular releases do not) vscode-kubernetes-tools/vscode-kubernetes-tools#527
• ms-python.python (old version uses marketplace / new version uses github release) - che #12840 Adding the latest version of python plugin. Downloading vscode extention directly from the github release page to avoid marketplace rate limmit issue che-plugin-registry#119
• vscode.go PR sent Using GH release instead of marketplace for go extension to avoid marketplace rate limit issues che-plugin-registry#131
• vscode.node-debug (still uses marketplace) - Releases do not contain vsix extention microsoft/vscode-node-debug#191
• vscode.node-debug2 (node 8 < )(still uses marketplace) - Releases do not contain vsix extention microsoft/vscode-node-debug2#232
• vscode.typescript !!! uses che-incubator !!! https://github.com/che-incubator/ms-code.typescript/releases/download/v1.30.2/che-typescript-language.vsix
• vscode-redhat.java PR sent che #12840 Adding latest 0.43.0 java vscode extention. Using 'vsix' from GH releases to avoid marketplace rate limit error che-plugin-registry#123
• vscode-java-debug - https://github.com/microsoft/vscode-java-debug/releases / Provide new version of redhat-java plugin che-plugin-registry#157
• vscode-sonarlint - uses GH release - che #12840 Downloading sonarlint extention directly from the github release page che-plugin-registry#118 (but looks like extention does not work atm with the latest release 1.6.0)
• openshift-connector - uses GH release che #12840 Adding latest version of openshift-connector plugin. Switching plugin download url from marketplace to github release che-plugin-registry#117
• vscode-xml - uses GH release - che #12840 Adding latest versions of yaml (0.4.0) and xml (0.5.1) vscode plugins che-plugin-registry#114
• vscode-yaml - uses GH release - che #12840 Adding latest versions of yaml (0.4.0) and xml (0.5.1) vscode plugins che-plugin-registry#114

[fbricon]

vscode-yaml 0.3.0 and vscode-redhat.java recent builds have now been attached to github releases

[apupier]

Is usage of Github release artefact a temporary hack?
We are loosing the metrics having them.
it is forcing to add an extra step to configure jobs to push the artefact too.

[ibuziuk]

@apupier yes, this might be temporary if we find a better way to avoid rate limit issues, so far there was no straight forward solution found

[l0rd]

@ibuziuk @apupier this is a temporary solution to mitigate a blocking problem. After Che 7 GA is released we should start working on a lazy caching mechanism or a VS Code Marketplace mirroring as described #12840 (comment).

[ibuziuk]

@l0rd atm there are only 2 plugins that still use marketplace (node-debug / node-debug2), so I sent PR for updating prod (also includes naming convention changes) - openshiftio/saas-openshiftio#1255

Once merged I believe we can stop working on this issue at least till GA since the error should not appear with the updated registry. It might make sense to add some section in README to discourage referencing marketplace directly for the time being WDYT?

[l0rd]

@ibuziuk thanks makes sense

[ibuziuk]

Sometimes hit this problem during installation of java debugger - microsoft/vscode-java-debug#583

[benoitf]

@ibuziuk
would these links work ? AFAIK there is no limit rate on the assets
https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/java/0.44.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage

[ibuziuk]

@benoitf I haven't tried but I believe any link with valid vsix would work

[benoitf]

I mean, these links are generics, it's easy to have links for any vs code extension and it's like the final redirect of marketplace download so there is no limit rate

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.