Tor onion integration #37
Comments
|
👍 this would be pretty cool! |
|
OK I've written the go-multiaddr Tor onion address decoding: |
|
This sounds like a good plan for location-anonymous tor users to have a persistent identity and announce themselves to peers, but it doesn't cover the use case of anonymously (unlinkably?) accessing data from ipfs. For that we'll need a mode that doesn't transmit any linkable identifiers (which includes not announcing pieces to the DHT). |
|
@leif, Good point. The only "pieces" we'll be announcing to the DHT is .onion addresses. I suspect it will be very easy for us to create a readonly client mode for non-onion using Tor users to access any IPFS storage server without announcing any identity at all. |
|
Today I wrote a TorDialer and OnionListener; This repo also includes examples programs which can be used as an integration test with your system Tor. However it should soon have more ways to use Tor other than just the control port protocol with the system tor:
https://github.com/david415/oniondialer Although |
|
I've been looking closely at the source for Whereas the "readonly Tor-ified" multiaddr looks like this: or should the /tor/ multiaddr not encapsulate and instead look like this? What do IPFS developers think of all this? |
|
from chat:
Similar to A more explicit, but very bloated option for |
|
maybe: |
|
hi. I chatted with Juan last night on #ipfs and we decided that onion multiaddrs will look like this:
Tor doesn't transport IP or TCP... but is a realiable stream transport. We don't need DNS. Tor clients use a local SOCKS proxy; therefore the remote end does the DNS resolution. I hope I've cleared up the confusion regarding this. If not then I'm afraid this ticket will grow too big with my explainations. Perhaps find me on irc instead of commenting here with queries. For the Tor exit node case I think it should be represented like so:
So I think this sort of breaks the multiaddr parser model... and so I made my multiaddr/net dev branches do this:
That is the colon separates the embedded onion port number. |
|
we'll already have things like: if we made the restriction that only one component be used, why not then: o/ this is actually harder to impl and does not layer well on a FS approach like 9P or linux proc files, which the former one does |
|
thats not quite what I was getting at, i'm not saying that we require only one component. I'm saying that we combine the onion address and port into one as they both are given meaning by the |
|
Any suggestions for changes to go-multiaddr-net? |
|
@david415 right, What we should do is:
(i unfort dont have time to work directly on this until after Sep 15. but can CR) |
However... later it would be nice to have more granular policy for operating heterogeneous ipfs nodes/gateways/proxies. |
|
I'm looking at go-multiaddr-net and go-ipfs to see how things work... and it makes sense for thin-waist protocols... but this type of thing isn't going to work for the OnionListener use case: The onion multiaddr isn't enough information to create an onion listener. We need to know the tor process control port in order to create the onion service... then after we auto-create the onion service key material the control port tells us the onion address as well. Perhaps in this case we need an onion + multiaddr helper function to create a new onion service and it's respective multiaddr? But how to plug it into the normal usage of go-multiaddr-net where the API user expects a multiaddr Listener to be created by only feeding it the multiaddr? |
|
@david415 maybe the listen onion multiaddr could be different, could include more information, like the proxy: but it still should be advertised to others as:
yeah that's definitely doable. harder to make things plug in nicely. the other option is to create a "listener-maker" and mount that onto multiaddr, like: but this is kind of "magical" in a way i dont like (i.e. static), probably makes much more sense to rework how multiaddr-net works into something like this: n := manet.NewNetwork()
n.Add(manet.IP4)
n.Add(manet.IP6)
n.Add(manet.TCP)
n.Add(manet.UDP)
n.Add(onion.Onion)
l, err := n.Listener("/ip4/1.2.3.4/tcp/1234")
c, err := n.Dial("/onion/<onionaddr>")
// with shortcuts like:
n := manet.NewThinWaistNetwork()
n.Add(onion.Onion)"network" is a bit odd name, but does give the idea that these networks are stacked and makes more sense why mulriaddrs work the way they do. "transport" is another possible name, but may not capture the full meaning of "network". |
|
Sorry, let's move multiaddr pluggable-ness discussion to multiformats/go-multiaddr-net#7 |
|
hey @david415 -- @whyrusleeping has made transport changes to go-ipfs libp2p and to https://github.com/jbenet/go-multiaddr-net/ -- chances are we can resume the tor work! |
|
cool! i'll take a look. |
|
I can put some effort toward this in feb too.
|
|
does running an ipfs server as a tor hidden service work by default? |
|
No. IPFS does not "work by default" as a Tor onion service. It is not difficult to tell the state of the IPFS + Tor integration because it is utterly non-existent; That's why we have this ticket! |
|
To reiterate what @leif posted: It might work for dialing an IPFS node from a censored location but you better have a custom ipfs build that doesn't scream your identity to everybody in it's vicinity. I hope we can see different flavors of ipfs nodes before 1.0. A privacy preserving client is much needed otherwise every content you share and spread becomes a tracking cookie. I personally already have that barbed-wire in my head stopping me from using it to share music with my family and that's only a 1st world problem. |
|
I just posted a $100 bounty expiring in six months. Also paid to get it tweeted from @bountysource to hopefully get it some more attention. If there are blocker issues, let me know and maybe I can post bounties for those as well. https://www.bountysource.com/issues/26255361-tor-onion-integration |
|
Any updates on this issue? |
|
@david415 if you were willing to enumerate the reasons for your opinion, maybe people who agree with you but are closer to development decisions than we are could include them in future plans some day (Edit: it is obviously dangerous to use ipfs for content that anyone with a lot of power or money wants to remove or track, without using tor, as you can be identified by your hosting ip address.) |
|
It's too late for this discussion. IPFS has failed to embrace the concept of free Tor integration from volunteer developers. That having been said, anonymity is a synonym for traffic analysis resistance; that is to say, even encrypted traffic can be analyzed for the metadata it leaks. Tor is the very weakest of the existing designs for anonymous communication networks however it is the most widely used whereas the other designs from academia have not had much field testing; such as: mix networks, dcnets, verified shuffles and other things can be used to form anonymous communication networks such as private information retrieval, oblivious ram, multi party computation etc. Tor is trivially broken by any sufficient global adversary by means of timing correlation whereas mixnets are not. There are many other ways to break Tor. Anonymity aka traffic analysis resistance is not yet a popular security feature because these designs are in some respects ahead of their time... just like not every software project embraces deterministic builds. Just because your white middle class platitude doesn't allow you to understand why people in high risk situations might need these things doesn't mean they are not needed. In fact, in dealing with such folks I find the easiest way to impart the importance to them is to describe military scenarios, e.g. if you were in the military, overseas, you might actually be interested in traffic analysis resistance. Think about a future brighter than Tor! Think about mixnets, hybrid networks, dcnets and so on. Monoculture is death. Why is Tor the only successful anonymity network? And to a lesser degree I2p? Although the I2p observation is less valid because it's design is so similar to Tor in that it can easily be broken by timing correlation from a sufficiently global adversary. |
|
So what you're saying is that if something can be broken by a "sufficient global adversary" it's not worth using? This seems like a silly reason not to provide protection against casual snoopers, criminals, and copyright trolls.
You are welcome to pur your own efforts into a mixnet or whatever. But please don't go telling others they can't have any protection at all just because their requirements don't match yours.
…On Fri, Sep 4, 2020, at 14:49, David Stainton wrote:
It's too late for this discussion. IPFS has failed to embrace the
concept of free Tor integration from volunteer developers.
That having been said, anonymity is a synonym for traffic analysis
resistance; that is to say, even encrypted traffic can be analyzed for
the metadata it leaks. Tor is the very weakest of the existing designs
for anonymous communication networks however it is the most widely used
whereas the other designs from academia have not had much field
testing; such as: mix networks, dcnets, verified shuffles and other
things can be used to form anonymous communication networks such as
private information retrieval, oblivious ram, multi party computation
etc.
Tor is trivially broken by any sufficient global adversary by means of
timing correlation whereas mixnets are not. There are many other ways
to break Tor.
Anonymity aka traffic analysis resistance is not yet a popular security
feature because these designs are in some respects ahead of their
time... just like not every software project embraces deterministic
builds. Just because your white middle class platitude doesn't allow
you to understand why people in high risk situations might need these
things doesn't mean they are not needed. In fact, in dealing with such
folks I find the easiest way to impart the importance to them is to
describe military scenarios, e.g. if you were in the military,
overseas, you might actually be interested in traffic analysis
resistance.
Think about a future brighter than Tor! Think about mixnets, hybrid
networks, dcnets and so on. Monoculture is death. Why is Tor the only
successful anonymity network? And to a lesser degree I2p? Although the
I2p observation is less valid because it's design is so similar to Tor
in that it can easily be broken by timing correlation from a
sufficiently global adversary.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABI56RPO7RM2SGL6PHVYS3SEFOGDANCNFSM4BOOQVOA>.
--
https://www.literati.org/~seanl/
This is a temporary email address I use for outbound emails. I will be changing it as it starts to receive spam. If you'd like a permanent address to use, please ask for one.
|
|
It sounds like there are newer networks that would be even better to integrate into services than Tor? Are any to or near the point of usability? |
|
Just to add, I suspect the reason that the state of public anonymity tools is not stronger is that the existing international powerholders, whose power could be reduced by widespread accessible anonymity, take diverse action to slow the release and hinder the effective use of the research. The way to make things change would be for people like us to agree to work together on forging one right thing in a development community, and use tools of both interpersonal mediation and software development to bring the result to happen by force of collective determination. It might help if everyone kept themselves more anonymous, collaborated in private as well as in public, and supported people who ran into personal issues so as to resist disruption and keep the work moving forward. |
|
I don't think there's any need to posit a conspiracy to explain this. Strong anonymity is extremely difficult to implement, and not enough people both understand and care about it to make it actually happen.
The best proposal I've seen so far is Chaum's. But there is no stronger system than Tor or I2P that's in a usable state. Personally I trust Tor more than I2P because despite its government funding it has received far more attention from security researchers, and because the compromises have tended to be either against high-value, large targets or large numbers of lower value targets. Large targets are an obvious weakness in Tor against an adversary with wide visibility into the network, and the mass compromises have relied on more traditional means, in particular attacks against the browser.
Burning 0days is costly, so unless it's going to result in the neutralization of a high-value target or a lot of lower-value targets, it seems relatively safe to rely on Tor unless you're engaging in some behavior that's going to get the attention of some well-funded adversary. And if you are, you probably shouldn't be using IPFS either, since it could have its own vulnerabilities that would make compromising Tor unnecessary.
…On Sat, Sep 5, 2020, at 10:00, xloem wrote:
Just to add, I suspect the reason that the state of public anonymity
tools is not stronger is that the existing powerhouses, whose power
could be reduced by accessible anonymity, take diverse action to slow
the release and hinder the effective use of the research. The way to
make things change would be for people like us to agree to work
together on forging the right thing in a development community, and use
tools of both interpersonal mediation and software development to bring
the result to happen by force of collective determination. It might
help if everyone kept themselves more anonymous, collaborated in
private as well as in public, and supported people who ran into
personal issues so as to keep the work moving forward, so as to resist
disruption.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABI56RGGGWSL4NVBOFIB43SEJVETANCNFSM4BOOQVOA>.
--
https://www.literati.org/~seanl/
This is a temporary email address I use for outbound emails. I will be changing it as it starts to receive spam. If you'd like a permanent address to use, please ask for one.
|
Let's try to speak concisely and directly, and find ways to support each other. There are a lot of people taking action for a lot of different reasons, but I have no need to call any of it a conspiracy. My experience is that working on democracy software can become dangerous to the project if it has global publicity, and my knowledge is that acting on this can help sustain other projects. Giving warning prominently, early, and relevently, is an easy way to turn danger into just a conspiracy theory. I'm observing that David worked very hard for Tor here and it didn't pan out. For example, discussion of newer protocols, or discussion encouraging a return to Tor, both could take focus, energy, time, and that developer excitement that can make hours of work seem like nothing, away from a potential developer, replying to someone but possibly disagreeing. If someone were working and took the safety of collaborating among people already in agreement, they could have had more man-hours in their project. I'm not sure anybody here is working on this at the moment, but that's not always the case. If David did most of the work on this earlier, it could give reason to support him.
I'm always looking to find those obscure implementations nobody talks much about. One of them will be the future Tor. It does seem to me that Tor already has a codebase and community, and forking can be a quick way to keep working through disagreements; I haven't looked into the code. |
|
Dear @seanlynch you are belligerent. I am in fact not saying that Tor or I2p aren't worth using. Come off your platitude for a few minutes and think about people who may be in a more high risk situation than yourself. For those people Tor isn't good enough because their adversary may well be the NSA and the FBI and GCHQ and so on. I am welcome to put forth my efforts into mix networks but not because you say so speaking from a place of belligerence. I am a fan of Tor and I do not go around telling people they can't have any protection. Your delusional caricature of myself is offensive and alarming. Check yourself before you wreck yourself. |
|
It's great to see people passionate about adding some level of anonymizing support to IPFS implementations. Just a reminder that discourse in the IPFS repositories is guided by the code of conduct https://github.com/ipfs/community/blob/master/code-of-conduct.md. As for the state of things here I agree with the suggestion above #37 (comment) that the best way to move this forward is for some work to be done improving the plugin system (e.g. for go-ipfs ipfs/kubo#7652 and ipfs/kubo#7653) and then for someone motivated by this to make a plugin for Tor support. If this approach interests any of you then feel free to comment on ipfs/kubo#7653 to voice support for making the libp2p transports pluggable. |
|
I'm not sure of what is the current status about this issue, but @berty we are working on a tor transport. This is a beta and will not protect your IP !Basicaly I need to add a dns resolving service (all dns resolution are still made through your real ip, not through tor) It's using a lot of CGO and CI magic to builtin a tor node for a seemless experience (the CI prebuild tor in go a package so it's CGO seemless (just import it and it works, other alternatives require manualy building a make file to produce a valid go package)). |
|
If I recall correctly openbazaar uses our Tor transport we've written for IPFS: You should not be making separate DNS queries. The Tor exit node handles this. If you do this the Tor integration will be meaningless for traffic analysis resistance because the network intermediaries such as the client's ISP will be able to identify the connection destinations via the DNS queries, obviously. Just my two cents. I'm not clear on why exactly our work on the Tor integration hasn't been acceptable but recent comments suggest a plugin system is desired and is a prerequisite. This to me seems more like a political decision and it would've been nice to know this upfront from Juan instead of years later after our efforts have largely gone unnoticed and for the most part, ignored. Additionally the fact that IPFS and the associated commercial money making enterprises have chosen not to pay for the Tor integration themselves speaks volumes about their priorities. (we make money not privacy) Is it against your code of conduct to be frustrated with the way our volunteer code contributions have been treated? |
|
Really, until we have something like https://onionshare.org/ or https://www.tribler.org/ for ipfs, I wouldn't consider this issue of privacy solved, considering the ipfs dht doxes you. It seems all the big ipfs vc money is going to centralised ipfs cloud providers, and the imitations of them. Would be nice if some went to decentralised privacy. In the end, ipfs just seems a tool for big data companies to save some coin on automatic redundancy replication and deduplication, however the need to encrypt absolutely everything on ipfs for minimal privacy (still doxed by the dht though) nearly completely eliminates such benefit, making ipfs just a slower alternative to existing cloud providers. For alternatives, Sia and Blockstack are worth keeping an eye on, and BitTorrent v2 already works in production and has most of the advantages and use cases of IPFS. IPFS marketing makes it out as if they are the only players in town. |
@balupton To conclude : DHT have advantages but have drawbacks like other solution (central server and blockchain), IPFS's devs thinks DHT is the best tradeoff but nothing stop you from hosting your own central solution (like we do) or write a blockchain rendezvous driver. PS: I just noticed, the DHT is not used on berty right now because we were having problems with it, but it's gonna be bring up later. |
|
Thinking about this more. What is actually needed here for adoption, is not a client that does tor/onion routing for all files, but for only the files/dirs one considers private. For instance, an interface like ipfs desktop, onionshare, tribler, or dropbox - they are all equivalent in this proposal - which when you drag a file into, or select a file in a add file dialog, should prompt, before it is added, which privacy mode you want for the file:
So the transport should be specific to the file/directory's privacy setting. The UI can distinguish the modes via colours and symbols for the file's row in the file listing, similar to browser's incognito mode. Perhaps a moniker between these can be used: private, public, open, confidential, transparent, closed. So private vs transparent for tor vs no-tor, and closed vs open for encrypted vs non-encrypted. For the private mode, many may be suitable with a openvpn or wireguard or cloudflare-warp transport, rather than necessarily a tor transport. VPN transports could also be a source of revenue from affiliate sales. |
|
@balupton The issue with opt-in privacy and encryption is the anonymity set is very small. It is better if privacy is opt-out. We don't want a file or user to be targeted as especially important due to the fact it's marked private. If only a small number of files are marked as private, then those will look especially delicious to those seeking compromising or otherwise personal information. It also makes it a trap for people who are configuring data stores and either are unaware that all the files are public or else forget to toggle the correct setting, and invites for bugs and oversights that make things that were once private not so when some recent change to a script that forgets to explicitly enable privacy restores the default. |
|
For clarification-- your idea does not preclude privacy by default, it just points out that not all files have to be encrypted. That's true, but I'd like to suggest it should be the default that they are. |
|
Has anyone made any progress since this discussion last year? |
|
I'm looking forward to this. It will add tons of value in terms of anonymization of the network |
|
No progress has been made and this is not a priority. Unfortunately, it's a large undertaking with a high risk of going poorly. To make this happen (by default), we'd need to:
|
This comment was marked as spam.
This comment was marked as spam.
|
Who decided that this is not a priority? You have to solve the tor connectivity first if you want to then address corporate proxies. Has it been decided that having IPFs run in a corporate environment is not a priority? By who? |
|
@emdee-net IPFS is an opensource project, we don't think it's a priority means we wont work on it for now. But if you disagree just make it yourself, or pay someone to do it, There is already at least two PoC you can pickup and finish (most of what there is to do is fixing IP leaks: dns, ...). (and fingerprinting but it's a later task, at the start we can just say one node = one identity) If we want to work on other things instead, we work on other things instead. |
|
For what it's worth, protocol labs is actively looking to fund people to work on content routing privacy, so folks looking to add something like TOR could start there. https://research.protocol.ai/blog/2022/new-open-problems-in-private-data-retrieval/ |
and others
chatting with Juan Bennet at c-base, Berlin about Tor onion service integration
here's what we've identified as necessary for proper Tor integration:
/onionto go-multiaddr -/onion/<onion-key>/ipfs/<ipfs-key>/oniondialing to go-multiaddr-net/onionI know plenty of people who'd be willing to run some Tor onion bootstrap nodes.
(edited by @jbenet for links)
The text was updated successfully, but these errors were encountered: