Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

End of service for Bintray in May, 2021 #6348

Closed
imprakharshukla opened this issue Feb 4, 2021 · 140 comments
Closed

End of service for Bintray in May, 2021 #6348

imprakharshukla opened this issue Feb 4, 2021 · 140 comments
Labels
information Informational post

Comments

@imprakharshukla
Copy link

Problem description
Jfrog's Bintray which Termux uses as the primary hosting, is ending support in May of 2021. Here is the official announcement from the VP of dev relations for Jfrog https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/.

Bintray/JCenter users should start migrating to a new hosting solution.

What new hosting solution will Termux switch to?

@RossComputerGuy
Copy link
Contributor

I've noticed Linode is a cheap solution for running a VPS, ~$5/mo but traffic may cause the price to go up.

@imprakharshukla
Copy link
Author

@RossComputerGuy $5 isn't the actual price because Bintray had allocated around 32 TB / month, which would account to being approximately $350 in Linode's outbound transfer. This doesn't come with a CDN as opposed to Bintray which makes it terribly slow.

Using a dedicated object storage like S3, DO Spaces, wasabi or blackblaze will require an app update, therefore will render the Play Store version useless. Plus it will shoot the prices well beyond a $1000.

@ghost
Copy link

ghost commented Feb 4, 2021

In Feb. 28:

No more submissions will be accepted to Bintray, JCenter, GoCenter, and ChartCenter
The GoCenter and ChartCenter websites will be disabled (client requests will still work)

So this means that package updates will be stopped in Feb 28?

@imprakharshukla
Copy link
Author

@wmcb-tech That is very confusing. They could also mean that no new user would be able to register for these services or just that all the rest and client requests will be blocked. I think former. How unprofessional of them to give such a short window for the migration.

@ghost
Copy link

ghost commented Feb 4, 2021

Termux has more than 1000+ packages available though, well it's not a big deal since we have mirrors

@kcubeterm kcubeterm added the information Informational post label Feb 4, 2021
@kcubeterm
Copy link
Member

By the way I got some good results with ipfs at some extent but don't know what happen if lots of user will be there. So I will share links once I managed to get atleast 10 node.

@ghost
Copy link

ghost commented Feb 4, 2021

So this means that package updates will be stopped in Feb 28?

Yes, this means any data submissions (package uploads) will be stopped.

By the way I got some good results with ipfs at some extent but don't know what happen if lots of user will be there.

@kcubeterm I can give a "stress test" to your setup by sending traffic from https://main.termux-mirror.ml to IPFS repo for some time.

So I will share links once I managed to get atleast 10 node.

Hopefully these 10 nodes are not 10 (free) accounts on Pinata/Temporal... not saying that this is (likely) against their ToS but also effectively only 2 nodes.

@ghost
Copy link

ghost commented Feb 4, 2021

Ok, then I will manage to upload all latest available updates and then disable Github Actions uploads on next week to prevent any potential inconsistent state on Bintray repo (so it will continue to be available in RO mode until shutdown). We still have mirrors as shown in https://github.com/termux/termux-packages/wiki/Mirrors, so hosting by itself is not a problem currently but package submission workflow likely will be switched to manual.

@a1batross Are you continue hosting Termux mirror at https://termux.mentality.rip? I have noticed that last sync was on 26 Dec 2020.

@ghost ghost pinned this issue Feb 4, 2021
@a1batross
Copy link

@xeffyr sorry, I forgot to move mirror.list config from old server so I rewrote it from scratch. Now it's mirroring {termux,game,science,unstable,x11}-packages. :)

@insign
Copy link

insign commented Feb 4, 2021

Maybe we (community) can seed like a p2p network, but or create public mirrors based on checksum verifications?

@Harshiv-Patel
Copy link

Well if every node needs to have the whole 20-50gigs worth of data, the torrent like p2p would be quite expensive for end users, Though I'm interested in the idea. I guess it'd have to be something where people can join and leave seeding parts of it from time to time.

The whole thing looks far more complicated the more I think about it.

@a1batross
Copy link

@Harshiv-Patel termux repos are less than 10GB.

The problem is not the distribution though but publishing. Bintray was pretty good at this.

@kcubeterm

This comment has been minimized.

@ghost

This comment has been minimized.

@ghost
Copy link

ghost commented Feb 6, 2021

Those who want to host few or whole repo can pin content from here.

And will need to re-pin it any time the repo is changes.

@kcubeterm
Copy link
Member

Repository inconsistency: few deb files missing

Most probably, this issue should be fixed after fresh Installation. I have published newer cid on same ipns

Accessibility (stability?) issues: traffic has been reduced by 70% which means users have problems with downloading packages.

Not sure about it, I have installed many packages and there's no issues, latency is also better than before.

Grimler91 added a commit that referenced this issue Feb 7, 2021
Use https://grimler.se directly since bintray repositories can no
longer be updated at the end of February, see
#6348

Same change as already done for x11-repo and unstable-repo.
@ghost

This comment has been minimized.

@ghost

This comment has been minimized.

@kcubeterm
Copy link
Member

Alright, I am aware of that tutorials, but what can I do here, since it's your key only you could publish unless you share that key. From now I will only pin repo with your CID (since it's change always so first my node will resolve ipns)
So ensure don't change your ipns link next time otherwise I have to manually put it into all nodes.

@ghost
Copy link

ghost commented Feb 10, 2021

@a1batross Your mirror looks messed up. There were reports about checksum mismatch from some users and some repositories are empty, e.g. https://termux.mentality.rip/termux-root-packages-24/

If you are using apt-mirror try to remove its cache (like /var/spool/apt-mirror/*, not repository data) and force re-sync.

@a1batross
Copy link

a1batross commented Feb 10, 2021

@xeffyr just did that and changed all unix permissions/ownership.

I joined #termux IRC channel, so if there will be any problems you can ping me there.

Will try to set up mirroring from IPFS this evening or tomorrow. Never worked with IPFS before.

ghost pushed a commit to termux/whatprovides that referenced this issue Feb 11, 2021
@finagolfin
Copy link
Member

Honestly, this is a good thing, as I never liked that Termux was so dependent on that company for free bandwidth. I agree with @insign, we should be able to set up a p2p network between the app installs, seeded by a single cheap VPS that provides the freshly built packages and checksums. The only question is whether we can easily repurpose some existing apt/torrent code out there for this or if it will require some custom code written by us.

@kcubeterm
Copy link
Member

@buttaface We have tested ipfs and got good results. Right now(from yesterday) whole termux traffic are hosting from ipfs and no one reports any error regarding this. We have 12 nodes right now. And that is sufficient but since it's P2P more contributions are welcome.

@Luflosi
Copy link

Luflosi commented May 10, 2021

Note that you can reduce the bandwidth and CPU usage of the IPFS daemon by a lot by setting Routing.Type to dhtclient, see https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#routingtype.
Running an IPFS daemon and changing this setting is still better than using the gateways since then the files can be transferred from any IPFS note that has the files (that did not set this setting to dhtclient) instead of just one gateway.

@SDRausty

This comment was marked as spam.

@ghost
Copy link

ghost commented May 10, 2021

1 even 1/1000 of users will use a local gateway

It is not easy to tell everyone what and why they should configure. Especially if everyone will be warned that IPFS daemon eats battery charge and traffic.

I can make a script and add it to $PREFIX/etc/profile.d supplied with apt package. Script will ask user for confirmation on application start to automatically configure and run IPFS node once done. But I'm very sure there would be lots of complaints about "unknown thing" running in background and eating resources. Just understand that 5 minutes of running IPFS daemon is not enough. It should run for hours and bootstrapping would take significant time.

@iBug
Copy link

iBug commented May 10, 2021

What about routing users to mirror sites by default, as CentOS yum does? This should drastically reduce load on the "official (authoritative) origin".

This seems very feasible to me. We officially maintain a "mirror list" and ask users to choose from one of them. This could be done at setup or periodically so we can have updated mirror lists.

Debian and Ubuntu also provide "mirror selection" during installation, particularly in server setup.

@SDRausty

This comment was marked as spam.

@finagolfin
Copy link
Member

It is not bittorrent, it is a distributed CDN.

Leaving aside the fact that bit torrent can be considered a kind of distributed CDN too, what I said appears to be true, "similar to bit torrent, so we don't know who is donating the majority of the bandwidth." It is worse than bit torrent though, because any torrent swarm only shares content they choose to seed, whereas it appears these ipfs gateways have no choice and simply mirror all ipfs content? Or is there some way for ipfs gateways to decide which data they want to cache and mirror?

In Termux repository case, P2P network is used only to deliver data on gateways which cache it, so trying to scale it with adding a lot of "seeders" is pointless.

That is not what I'm suggesting: I'm saying 90+% of our package traffic should be delivered by old-fashioned web mirrors that we pay for, like grimler.se, though we could still use ipfs for some small percentage and for overages. That way, we aren't depending on the charity of others' ipfs gateways, who may get overloaded like bintray and decide to stop giving us free bandwidth one day.

An even better solution would be to setup a torrent client in every Termux app and go to true p2p distribution between Termux users for packages, but that would require more coding work. To avoid user complaints, we could set it up so that the torrent bandwidth limits are reasonable and easily configured by the Termux user. I would love for this to be the way we go, and would technically contribute my time to such a solution.

I'm actually agnostic on the technical solution chosen, what I'm against is relying on external funders of our bandwidth use, whether bintray or unknown ipfs gateways.

@SDRausty

This comment was marked as spam.

@SDRausty

This comment was marked as spam.

@ghost
Copy link

ghost commented May 10, 2021

@SDRausty I don't know whether you are understanding me or not, but issue here is to ensure that every user or some part (say 30%) of Termux community have read the instructions, understood them properly and implemented all steps as expected. - That's not possible, considering the experience that only very few people actually read the posts marked as "README" or "INFO" and rest just ignores them, opening a duplicate issue on Github, posting on Reddit, asking in chat or leaving a bad review on Google Play, which is understandable as not everyone may know that Termux is open source and is available on Github for some reasons (typically not reading the descriptions and motd). A such example would be #6726 which also linked on Reddit.

So we have for that 2 solutions:

  1. Implement a pop-up notification broadcast system in Termux app or something like a dynamic motd downloaded from our Github repo during shell startup.
  2. Do all significant work ourselves, e.g. as I suggested make a script and put into $PREFIX/etc/profile.d and then only prompt user for something like Do you want to configure and run ...<put description with all pros/cons here>...? (yes/no): .

Disclaimer: both variants are just "what could be done". None of them are planned to be implemented currently because IPFS is not a final solution for hosting of Termux repositories in a long term.

@SDRausty

This comment was marked as spam.

@ghost
Copy link

ghost commented May 10, 2021

@SDRausty Short term solution is what we use currently, i.e. available Termux mirrors.

Long term solution is a host dedicated to Termux project and managed by Termux team members. It can be a VPS or SaaS platform like Bintray was. Whatever it will be, members of @termux org should have right to upload packages on it. This is what is missing on mirrors where only their owner has right to upload packages and only if has access to GPG private key for signing metadata.

Please follow the full discussion to understand what is going on.

@agnostic-apollo
Copy link
Member

So basically to move forward now, we know that IPFS is not a long termux solution and we have to move to VPS/SaaS considering

  1. We are relying on unknown people who may not be available in future and ethically not right to divert large traffic to them without their consent (if not being done).
  2. There is a single point of failure, i.e @xeffyr and his PC, (may they both live long or as much they want ;))
  3. There are outages everyday during certain times of the day for some ipfs mirrors like xeffyr's: "Primary node can be shutdown during night times (22:00 - 09:00 UTC).". This fails apt commands, and requires users to wait or change repos, which is just a bad design, and won't work (easily) for user's automated background scripts and opens new Github issues we have to deal with.

I agree with @buttaface as well about fundraising, Termux should be free (beer and freedom) for users but users who can afford to donate, should do so and should be requested, at least for repository hosting costs. We already have $100 pledged on patreon without in-app or in-apt links, surely it could get bumped a little if its more apparent to users that we need donations with bintray shutdown, specially when its explicitly asked. @Grimler91 is right too about donations decreasing slowly in future after an initial boost, but maybe the final stable baseline that is set is enough to pay for basic hosting. It has stayed at $100 for a while after all, it could stay at ~$300 too. Should ask on Twitter as well, developers around the world are using Termux for their jobs, its time they contribute back in its time of need.

And considering @fornwall has said he's willing to hand over access, we can get the patreon access and use that for fundraising, maybe set up paypal/cryto too, since everybody may not want to use patreon or may want to donate anonymously or every month. It remains to decide who will manage the finances, @xeffyr seems to be reluctant on that phase, whatever his reasons may be, but is understandable, but if hes fine with it, then great. There is indeed an issue of getting "stuck with bills" if donations falls, then whoever is managing the finances may be "forced" to cover the costs (at self sacrifice), specially thinking it as a "responsibility" to millions of people. We should put something in-place that prevents that situation from happening, even if the finance manager can afford to cover the costs. Maybe we can choose a provider which has some lower costs plans as well that are throttled, that can be shifted to temporarily in low fund times, that keeps the repositories running, but at low speeds. If @xeffyr doesn't want to manage the finances, who else can? @buttaface can you? Someone has to now, if not fornwall...

@xeffyr can choose the right provider that works for Termux needs and he and other @termux members can manage it. What are the ideal ones at the moment? Some fundraising may also be beneficial if we choose a provider that supports open source work, we will be paying for the service, but if there is a low fund time, they might be willing to provide support for a month or so.

As for IPFS as an alternative solution, sure, that can be looked into. Maybe we can have an in-app consent page, which if accepted, can export an environmental variable, which profile.d checks first, instead of directly prompting users and "confusing" them and then them opening issues "Help! What is this?". It can be in TermuxSettings and hopefully mainly only informed users who understand what IPFS is and its risks and data/battery usage, will accept it. I am also thinking that there should be some News page in Termux app, that should get messages for users from Termux github to inform them of important messages from maintainers.

@finagolfin
Copy link
Member

If @xeffyr doesn't want to manage the finances, who else can? buttaface can you?

Nowadays you don't really have to, as there are transparent crowdfunding options, like this one by the Linux Foundation, where all money raised is publicly displayed and all expenses are publicly listed, so there is really nothing to "manage" or anybody to trust (an example project using that software can be seen here). Of course, somebody has to set that up, which I could do if the main Termux team leaders don't want to.

@Grimler91
Copy link
Member

Thanks for the write-up @agnostic-apollo.

Regarding 2. we have two points basically, since grimler.se is updated straight from CI (when that works..) and not mirrored from xeffyrs repos.

If we use the funding to setup an official vps and use that as main repo then we could ask some of the big mirrors if they could mirror those repositories. My closest mirror site, acc.umu.se for example, indicate that they are willing to help open source projects (if we provide information, try to foresee the future, and implement some geo-based mirror selecting system).

If we have an official vps I imagine that all members (that want it) can get user access to it, and thereby possibility to manually add debs and check the status. Preferably we would have some safe way to handle interactions with the mirror so that everything cannot be accidentally deleted..

If we ask for donations, and start using the patreon money to help the project, then it might be nice to keep a public ledger of incomes and expenses as well so that donating people know what the money is used for. @buttaface's example above looks like a good example (more serious/official looking than I had in mind).

@agnostic-apollo
Copy link
Member

Nowadays you don't really have to, as there are transparent crowdfunding options,
If we ask for donations, and start using the patreon money to help the project, then it might be nice to keep a public ledger of incomes and expenses as well so that donating people know what the money is used for.

Yes, definitely a public ledger should be there. And Linux foundation crowdfunding looks pretty good actually. They accept both credit cards and invoiced payments, so should be fine for most part. My only concern is that account creation is a huge turn off, specially would be when donating (I still haven't created one for termux wiki! No judging please, any day now... ;)). Basically, specially creating an LFX account just for Termux would put people off who may want to donate and reduce the amount of donations we get, which already are low. Patreon partially also has the same issue, but a lot of people who donate have an account on it now. We can have the main crowdfunding management on LFX or related, but should also provide alternate payment links so extra account creation isn't needed for donors or if anonymity or payment method issues are a concern for someone.

Of course, somebody has to set that up, which I could do if the main Termux team leaders don't want to.

Yeah, setup would definitely be required. VPS receipts will have to handled if using LFX. Otherwise for manual case, collection and redirection of funds to VPS would be needed. Someone has to step up and take responsibility. People here are being "careful", understandably so, but works needs to be done to move forward and solve the current issues, already been a few months.

Thanks for the write-up @agnostic-apollo.

You are welcome :)

Regarding 2. we have two points basically, since grimler.se is updated straight from CI (when that works..) and not mirrored from xeffyrs repos.

Yeah, I should have been more clear, I am aware your mirror and non-IPFS ones can continue working without requiring xeffyr's.

ask some of the big mirrors if they could mirror those repositories... My closest mirror site, acc.umu.se for example, indicate that they are willing to help open source projects

Yup, that would be even better. So, we mainly just need a stable base repository (servers) that other stuff can be built on/refer to. There seem to be quite a few open source helpers out there.

Preferably we would have some safe way to handle interactions with the mirror so that everything cannot be accidentally deleted..

Yeah, that would be a concern. But we should be able to set up some SOPs and write up some safe scripts, etc and automated backups since repo sizes are pretty low.

I guess we should decide what's the right provider for us and see what will be our costs and then move forward on how/where to set up crowdfunding.

@finagolfin
Copy link
Member

My only concern is that account creation is a huge turn off, specially would be when donating

Yes, that is a big problem with LFX, I was just citing that as one example. This other fundraising site doesn't require an account for one-time donations (it does for recurring donations) and seems to raise money for a lot more OSS projects, but charges 10%. I think that would be worth it for not requiring account creation.

People here are being "careful"

Makes sense any time money is involved, but that's also why this isn't like open source, where anybody can step up and contribute. You need someone trusted to handle money matters, which is why it probably has to be @fornwall or someone he delegates.

@agnostic-apollo
Copy link
Member

Well, www.oscollective.org looks good too. Even F-Droid is hosted there. Great that non-account payments are accepted too credit card/paypal/bank transfer and 10% should be worth it. Their financial management compared to LFX would have to be checked as well, hopefully is fine too, specially considering there are charging for it, which is more sustainable.

But not sure if we will be able to meet their hard requirements

If you are an open source project with at least 100 stars on GitHub and at least two contributors, you will likely be immediately approved.

Makes sense any time money is involved... You need someone trusted to handle money matters

Yeah, that's why I said understandable, can be a big responsibility, but as discussed, if crowdfunding site is managing most of the finances, then may not be too big.

which is why it probably has to be @fornwall or someone he delegates.

Yeah, that would be ideal.

@fornwall
Copy link
Member

For short-term, perhaps we just need a beefy enough VPS (or something like a Hetzner dedicated server), and hand out ssh access to those who takes on administering it? If so I can set that up.

If we're hosting packages ourself I still think sponsorship from a CDN like Cloudflare would be ideal, if possible:

  1. It saves so much time and energy to get a stable solution up and running compared to setting up and administering a global mirroring system.
  2. It's really convenient for users from different parts of the world (Termux is big in e.g. Indonesia and Nigeria) with stable and quick access.

Of course, if we can't get sponsorship from a CDN provider it's just out of the question (since paying a good CDN for XX TB is so expensive), but I think it's still worth exploring (I previously reached out to Cloudflare, but didn't reach the right person there to have any conversation, and there are other possible CDN sponsors).

For longer-term, I think it's important to settle if "packages via APK:s" is the way forward (to overcome termux/termux-app#1072).

If so, it would mean that Google Play will serve a majority of the bandwidth - the bandwidth requirements (for e.g. a custom F-Droid repo for non-Play users) will go down considerably and make things much easier to manage. And we need to focus on reaching a working solution there.

@a1batross
Copy link

or something like a Hetzner dedicated server

My mirror runs on Hetzner, the bandwidth is good for me: server uploaded around 10TB data last month. It's a cheapest Ryzen server, located in Finland.

@selurvedu
Copy link

Have you considered FOSSHost? I think Termux may be eligible for their services.

https://fosshost.org/
https://docs.fosshost.org/en/home/application
https://docs.fosshost.org/en/home/mirrors-as-a-service

@ghost
Copy link

ghost commented May 21, 2021

@selurvedu We don't need mirror-as-a-service. As I wrote here - we need either VPS or SaaS platform for hosting packages (similar to Bintray).

Mirrors are good, but please understand that they are mostly back-ups. We don't have primary server for further replication of repositories on mirrors and have to do that on our own (https://grimler.se/termux/ and IPFS nodes) and that has result of 2 branches of Termux repositories.

According to application form, they seem to not provide x86 servers (though it is possible to request AArch64 VPS as requested type of service):

Thank you for your interest in Fosshost. We are currently not accepting any new applications for x86 Virtual Machines.

@kcubeterm said that he works on solution. It is currently available at https://dl.kcubeterm.me/termux-packages-24/ though it works in a mirror mode currently and I don't know how long term it would be.

@iBug
Copy link

iBug commented May 21, 2021

@xeffyr Why is this issue closed? Do I assume you already have a long-term solution (IPFS)? It seems to me like a antipattern to close a ticket before the real issue has been resolved.

@ghost
Copy link

ghost commented May 21, 2021

@iBug This issue has been about Bintray support finish and finding a solution to prevent service interruption. This has been solved for few months, since Feb 2021.

I will open a new issue regarding hosting options.

@selurvedu
Copy link

@xeffyr I see. You could use an AArch64 host for uploading packages and a mirror for distributing them. Also it is not directly stated how their mirroring works, maybe it's simply accessible via (S)FTP.

I'm sorry that I didn't read this whole thread, so I kindly ask you to clarify what are the exact requirements for hosting a repository, i.e., do you need any specific software, or do you simply need a webhook for uploading files via HTTP from a CI? Is some kind of artifact version management needed? Do you want a feature to auto-update the repo lists when packages get added/updated/deleted, like Open Build Service does? etc.

@SDRausty

This comment was marked as spam.

@agnostic-apollo
Copy link
Member

A new issue #6846 has already been opened with the details.

@D4RKH0R1Z0N
Copy link

Termux could use GitHub pages

@insign
Copy link

insign commented May 31, 2021

Termux could use GitHub pages

GH has many kinds of limits since it is not for that.

@ghost ghost locked and limited conversation to collaborators Jun 1, 2021
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
information Informational post
Projects
None yet
Development

No branches or pull requests