Alternative download location

[ariya]

Let's explore a different place to host our downloads!

Requirement: good analytics. Minimum requirement: accumulated download counts.

[ariya]

Our current host, Bitbucket, is known to apply rate limiting. In some cases, it's too aggressive.

[ariya]

Bintray has a free plan for FOSS project. Last time I checked, the stats are very minimal (last 30 days only, or something like that). On top of that, we don't know if there is a similar throttling or not.

[ariya]

While we can always use Github Releases feature, there is zero analytics available. Also, someone should investigate the rate limit.

[vitallium]

What about Amazon S3?
Or a static page with Google Analytics linked as downloads.phantomjs.org on Azure? (I have a Bizspark subscription there, so we can do it free)

[ariya]

We may have a CDN sponsoring a download space soon. Stay tuned.

[vitallium]

@ariya Great!

[peixotorms]

What kind of analytics are you looking forward too? I think that any 1 Gbit cheap vps host will do (http://vpsdime.com/), with cloudflare in front of it. It works as cdn and you can see their free analytics as well as be protected against ddos. https://www.cloudflare.com/features-analytics/ But of course, free sponsor is better.

[ariya]

@peixotorms DIY is not an alternative. We couldn't afford any extra time for maintenance.

[siboulet]

Stumbled into this issue coming from #13951. Installing randomly fails due to bitbucket returning HTTP code 429 (rate limit). Did you try to contacting bitbucket to have the limit removed on your account? They seems to be quite happy at helping open source projects and they may very well lift the limit for phantomjs.

[chrylis]

That last would be excellent. I've had close to a 50% failure rate on jobs recently because of the BitBucket 429.

[wimdeblauwe]

Same problem here. We are using the frontend-maven-plugin to build everything and I also get 429 errors all of the time.

[jamiechapman]

Also having this issue too with Heroku.

[shuber]

Running into this issue as well 👍

[ariya]

@siboulet I'm rather swamped right now. If you want to help and talk to Bitbucket to see what they could do, that will be helpful.

Also a reminder to everyone: there's no need for a stream of "me, too" comment. You can use the Subscribe button if you want to get notified of any further change or progress.

[jamiechapman]

@ariya Thanks for the update. My reply was to mark the severity of the issue more than anything rather than getting notified about updates, since the rate limiting causes deployment issues.

[siboulet]

@ariya I have opened a support issue with Bitbucket, reference BBS-28344.

[siboulet]

@ariya here is the response I received. Time to look for a new home for hosting the PhantomJS binaries.

Jesse Yowell [Atlassian]19/Feb/16 2:30 PM
Simon,

The rate limiting issue isn't something we can change here. We limit the number of downloads per repository based on this article: https://confluence.atlassian.com/bitbucket/rate-limits-668173227.html

This repository also seems like a hosting mirror, which is against our Terms of Service:

1. Storage Rules. We do not set pre-defined storage limits on Your Data in Bitbucket. However, we do enforce parameters described in the Bitbucket documentation to ensure that you do not use Bitbucket in a way that consumes a disproportionate amount of system resources (CPU’s, memory, disk space, bandwidth, etc.) or that would adversely impact the performance or operation of Bitbucket for other Bitbucket users. Similarly, since Bitbucket is designed to be used as a source code repository, we reserve the right to remove any other content (such as music or video), particularly if the content is consuming a disproportionate amount of storage. Please note that, since we do not maintain access to your repositories, any removal of Your Data under Section 7.8 (Removals and Suspension) of the Agreement means removal of the entire repository in which the offending data resides, not just the offending portions.

We apologize, but there isn't much we can do here.

Thanks,
Jesse

[chrylis]

I just noticed the issue of download limits on Bintray--if you're using JCenter (not sure if that now includes the non-Java hosting or whether that's branded separately), the whole point is to provide a public mirror, and they won't have rate limiting. No idea about analytics, though I know they provide at least basic download counts.

[ariya]

We are trying to get a working setup with a CDN so that the downloads can be provided by the CDN.

[blaise-io]

Temporary workaround: PHANTOMJS_CDNURL=http://cnpmjs.org/downloads npm install -g phantomjs2

[paladox]

You can add binarys to GitHub and it is fast to download. You can also edit your 2.1.1 release to include the binarys in GitHub.

[Download]

What about RawGit.com?

They are backed by MaxCDN and every time you tag a release, it's contents become automatically available. In my projects I just commit the build to Git (I know some people frown upon it), and so I automatically have CDN urls for direct download.

E.G: check out the download links on this project page: http://download.github.io/picolog/
You'll find they point to rawgit. It works perfectly for me, but of course my projects don't get many downloads at all, unlike the massive popularity of Phantom JS, so I'm not sure how much load they can take and how reliable they are in such scenarios. But the wording in the text under 'Use this URL in production' makes it sound to me that it might work well for Phantom JS as well:

No traffic limits or throttling. Files are served via MaxCDN's super fast global CDN.

[redgeoff]

There is a lot going on in this thread ;). Thanks everyone for their help! The solution I implemented is actually pretty simple and I posted it at bitbucket rate limiting phantomjs

[ariya]

@Download You probably missed the Requirement paragraph. Beside, RawGit doesn't provide uptime guarantee at all.

[damm]

Rawgit is free; the lack of support and uptime guarantee should be acceptable. Reviewing https://github.com/rgrove/rawgit/wiki/Frequently-Asked-Questions#i-need-guaranteed-100-uptime-should-i-use-cdnrawgitcom

It's free so completely acceptable; it costs money (a lot) for 100% uptime (let alone a guarantee) most IaaS providers really don't even give you that. I've gotten 99.99% from Internap but that still gives you a few hours per month

Lastly.

Let's make sure we're realistic in our goals; and also uptime guarantee's generally are a cash return (or credit). It really doesn't mean your going to be up 100% of the time. It means they are going to try but will pay out for downtime.

No I'm not a contributor I just am trying to make sure that most people understand that 100% uptime isn't cheap nor easy.

[piit79]

@damm I think you're missing the point - the main requirement for the download server was good analytics, not 100% uptime (see the first post in this thread). I haven't found any mention of even simple download counters on RawGit...

[Download]

May I ask what part of the analytics you are interested in? Because if you are interested in 'real downloads' (meaning, by people installing PhantomJS vs Travis CI et al downloading it), then a middle ground solution might be a simple Download page in the static site (gh_pages branch) with a download link with an onclick handler that fires an event to GA. Because people will use that link and not the 'raw' rawgit URL (which looks daunting). But this would not count the automated build downloads.

Whichever way you cut it, if the main download location is down often, what you will see is that people will start to create own mirrors... Which also means they will no longer be counted by analytics. What exactly does that information about the number of downloads of the binary give you that cannot be simply derived from stats counters on the page and download link?

Technically, unless you are doing server-side counting (e.g. with log analyzers etc), you can't really track downloads. Not with say Google Analytics at least.

So is this the requirement? A host that supports file downloads and that will report analytics on it?

[damm]

Analytics can be nice and you can definitely do that with Akamai,Fastly and most CDN's.

Also like @Download said having a download page maybe helpful. Rubygems is hosted on S3 however you fetch get the links from rubygems.org for this 'Analytics' effect.

It's also a SPOF (Single Point of Failure)

[yellongbulla]

New user <24hrs...

Haven't managed to get phantomJS to download or build properly and i discover this thread.

8 months?! Pull your finger out guys... wanders off to find an alternative

[Download]

@yellongbulla

I get that it can be a showstopper when trying a new lib. It was for me.

On the other hand, this is an open source project, so anyone could come up with a solution. Including you and me.

The problem is that the people having the issue don't care about the analytics, whereas ariya (apparently) doesn't care so much about the downtime issue...

I was prepared to create a PR to help place the file on CDN when I wrote my comment, but the requirements are too high for me to meet. So I too have given up on phantomjs. It's apparently a very good library but alas.

[yellongbulla]

Counting downloads that just aren't happening... Makes complete sense to me!

A simple link to a mirror on the main download page would be nice...
... did your analytics count that download? ;)

[Ch4s3]

Hi @ariya have you given any more thought to the various CDN options listed above?

[zwetan]

not sure if anyone noticed but if you use http:// instead of https:// for the bitbucket download link, it just works

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
end up with

HTTP request sent, awaiting response... 403 Forbidden
2016-11-18 00:59:54 ERROR 403: Forbidden.

tried numerous times

but a regular non-https download does work even though it mention it does use https
wget http://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

eg. here the output

URL transformed to HTTPS due to an HSTS policy
--2016-11-18 01:01:16--  https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
Resolving bitbucket.org (bitbucket.org)... 2401:1d80:1010::150, 104.192.143.2, 104.192.143.3, ...
Connecting to bitbucket.org (bitbucket.org)|2401:1d80:1010::150|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://bbuseruploads.s3.amazonaws.com/fd96ed93-2b32-46a7-9d2b-ecbc0988516a/downloads/396e7977-71fd-4592-8723-495ca4cfa7cc/phantomjs-2.1.1-linux-x86_64.tar.bz2?Signature=%2FKsqkCHIikt36WrJ43X7SRL6HBM%3D&Expires=1479428415&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=null&response-content-disposition=attachment%3B%20filename%3D%22phantomjs-2.1.1-linux-x86_64.tar.bz2%22 [following]
--2016-11-18 01:01:16--  https://bbuseruploads.s3.amazonaws.com/fd96ed93-2b32-46a7-9d2b-ecbc0988516a/downloads/396e7977-71fd-4592-8723-495ca4cfa7cc/phantomjs-2.1.1-linux-x86_64.tar.bz2?Signature=%2FKsqkCHIikt36WrJ43X7SRL6HBM%3D&Expires=1479428415&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=null&response-content-disposition=attachment%3B%20filename%3D%22phantomjs-2.1.1-linux-x86_64.tar.bz2%22
Resolving bbuseruploads.s3.amazonaws.com (bbuseruploads.s3.amazonaws.com)... 54.231.33.211
Connecting to bbuseruploads.s3.amazonaws.com (bbuseruploads.s3.amazonaws.com)|54.231.33.211|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23415665 (22M) [application/x-tar]
Saving to: ‘phantomjs-2.1.1-linux-x86_64.tar.bz2’

[dogancelik]

Github counts downloads (visible in API) and downloads are not rate-limited AFAIK.

Example: Medium's fork of PhantomJS, Windows .zip release for 2.1.1:
Taken from: https://api.github.com/repos/Medium/phantomjs/releases/2489362
Downloaded 1.731.773 times since March 2016.

{
  "url": "https://api.github.com/repos/Medium/phantomjs/releases/assets/1471657",
  "id": 1471657,
  "name": "phantomjs-2.1.1-windows.zip",
  "label": null,
  "uploader": { /* … */ },
  "content_type": "application/zip",
  "state": "uploaded",
  "size": 18193653,
  "download_count": 1731773,
  "created_at": "2016-03-25T15:29:18Z",
  "updated_at": "2016-03-25T15:29:21Z",
  "browser_download_url": "https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-windows.zip"
}

---

You can see example use of the API in my project website above (loaded with AJAX). Public API is rate-limited but downloads are not, so you need to proxy/cache the API request if you want to use it on the website.

In case you want more analytics:

• You can use Google Analytics to track clicks on download buttons,
  <a href="…" onclick="ga('send', 'event', …)">Download for Windows</a>
• Or use a custom download route that redirects to corresponding Github download URL
  phantomjs.org/download/windows
  ⬇️ (302 redirect) ⬇️
  github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-windows.zip
• Use a CDN (like CloudFlare) on phantomjs.org to track everything.

[haroon-sheikh]

@ariya bintray looks promising? E.g. https://github.com/getgauge/gauge are using bintry for their nightly builds also, it's working really nice for them.

[damm]

bintray's foss 1tb download limit would likely exceeded pretty easily no?

[ariya]

This is a moot point while we're suspending the development (#15344).
We will revisit it again at some point in the future. Thank you!