Support for channels distributed over SFTP

[andrsd]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

I would like to add capability that would enable to serve channels over SFTP, i.e. instead of having https://pkg.channel.org/, I could have a private channel on privately owned server a serve the packages over sftp. The reason to use SFTP is simply because users who would be accessing the channel would have SSH keys on the server.

Why is this needed?

This would simplify our life where we have privately developed software that we can build as conda packages. They are already depending on other libraries that we distribute using conda channels. So the software can be deployed in the same manner and users can use conda commands like install/update, etc. to get these private packages.

What should happen?

Users would add a channel into .condarc like so:

channels:
  - sftp://pkg@conda.somewhere.com/pkgs

The pkg user here is an example of a setup, where it has the public keys of everybody accessing the machine and serves the packages to the users.

Additional Context

I already wrote code that does the above. It adds a dependency on pysftp which also depends onparamiko. Not all SSH key types are supported in paramiko yet, but I was able to set up the above and distribute the packages using RSA.

If there is an interest in this capability, I can open a PR to show the code. Being a first-time contributor to conda I may have easily overlooked something. I based my work on the existing ftp adapter and modify it so it works with pysftp.

[travishathaway]

Hi @andrsd,

Thanks for this well articulated feature request. We are currently considering adding a plugin hook in conda to enable plugin authors to extend or override our networking code. I believe this use case is a good candidate for that.

Here's a link to that CEP (Conda Enhancement Proposal) and the discussion surrounding it:

• Conda Fetch Plugin Hook ceps#44

Given that we want to allow plugin authors to extend networking behavior, this is unlikely to be added to conda anytime soon. Regardless, we will keep this issue open to foster a discussion for how best to add these capabilities to conda.

[andrsd]

Hi @travishathaway ! Thanks for getting back to me and the info. I will subscribe to the CEP so I am aware of what is going on there. At some point I plan on porting my code into the plugin design to see how that would work. I will ping the PR when I get to that (if it is still open). And I may need some help to get me started (still a conda noob :-))

Also, feel free to close this issue if this is not going to go into conda code base. I am perfectly fine with this being a plugin.

[jaimergp]

As a workaround, how about mounting the SFTP dir in the filesystem via sshfs or similar? conda should work with local channels à la -c /opt/my-custom-sftp-channel, so maybe it works?

[andrsd]

I did consider such a setup, but that could be only a temporary solution for us. It is much better to have a SFTP transport layer in conda, because then everything is transparent for users and they do not have to think about "is my conda channel mounted or not". Also, some users are on macOS and the sshfs support was not the easiest to deal with (differs with the version of the OS).
And as I said, I already have the code that does the SFTP transport layer, so I just need to be able to deploy that in some standard manner.

[jaimergp]

Agreed on all points. I was only suggesting a temporary workaround while things settle in the code base to allow for a SFTP plugin.

[andrsd]

Got you and thanks for your help. I appreciate it!

What I tried to do was (1) create my own version of the conda package. Basically I took the code and applied my patches on a released version and conda build it with my own version number. This sort of worked, but obviously has the problem, that it cannot be served over the SFTP channel (chicken and egg problem). (2) install conda the usual way and then patch the installed package. This works but it is not good way for a regular user (obviously).

These were just attempts to see if the code works and could be easily deployed. Ideally, it would be distributed via an open channel - and the plugin approach is totally fine for our needs. Is there some rough time line when you guys plan of having it done. Or is ti more "it is gonna be done when it is done" kind of a deal?

[jaimergp]

Honestly no clue about time estimations, I'll let @conda/conda-core give you more info there.

In the meantime, you could serve your patched conda via a custom channel at anaconda.org?

[andrsd]

In the meantime, you could serve your patched conda via a custom channel at anaconda.org?

I like this solution. How would I go about it? I am fairly new to the conda development, so I would need a help with this for sure. And getting another set of eyes on the development would be great so that the package gets deployed correctly.

[jaimergp]

This repository contains a recipe/ directory. Amend the source section in meta.yaml so it points to your fork and run conda build. The resulting artifact(s) can be uploaded to your anaconda.org channel with the anaconda-client CLI package, or via the web UI.

[andrsd]

I finally found some time to get to this again. I successfully built the conda package with sftp support locally. I did not have to change the source in meta.yaml - it now points to ../ so it is using the local git repo.

However, I have a question regarding the anaconda.org channel. Are you suggesting to upload the artifacts to the sftp channel? Because that would not really work - cannot get to the sftp channel without the sftp support (chicken and egg problem). So, maybe you were proposing to host 2 channels? One public with the patched conda package and then the sftp channel? I am assuming you did not mean the public anaconda hosted at https://repo.anaconda.com/pkgs, since this way anybody could deploy their packages without any review - such packages might be broken and create lots of headache to your users...

[jaimergp]

maybe you were proposing to host 2 channels? One public with the patched conda package and then the sftp channel? I am assuming you did not mean the public anaconda hosted at https://repo.anaconda.com/pkgs

Correct. I would create a new custom channel at anaconda.org (e.g. andrsd), and your users would get / update their conda with conda install -n base andrsd::conda. After that they can create their environments from the SFTP channel.

[andrsd]

I did not know that custom channels were a thing - that's pretty neat 👍 . Followed your advice and all is working now.

Thanks a lot for your help with this!

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!