hybrid-tx-building #46
Conversation
|
Thanks @lehnberg, hopefully this RFC will get the conversation moving in the right direction. Following up from the keybase chat, I want to throw out the idea here as well of borrowing PGP's ASCII-Armor technique for encoding Grin slates (similar to the blobs described in this RFC). https://tools.ietf.org/html/rfc4880#section-6.2 Using ASCII-Armor, transaction slates become more resilient during transport across a variety of channels and can improve usability by allowing easy copy/pasting of the slates, even with arbitrary message lengths. This does not fully address end-to-end encryption, but it could be easily facilitated in this model by adding a new Armor Header like With robust armored Grin slates, we might be able to avoid the core code having an opinion about https transport (which can easily violate the privacy of the user) or Tor transport (which is a large technical and cultural barrier for many centralized services). A brief technical description borrowed from its RFC and loosely adapted to fit Grin slates: When Grin encodes slate data into ASCII-Armor, it puts headers around the encoded slate data so that the data can be later reconstructed. Concatenation of the following data creates Grin's ASCII-Armor:
Armor Headers are key/value pairs of strings that give the user some information about how to use the message. Armor Header Keys:
Example: I'm not convinced A next step in determining viability might be to determine the number of armored messages that would be required for normal and large sized Grin slates to see how close we are to reality with the number of messages required (is it 1-2 or 3-5; this makes a major usability difference). I'm not certain this is the right direction to go in but wanted to share it anyway for others to consider. |
|
Thanks for sharing this @j01tz!
Are you saying that one slate (i.e. the payload of one trip) would consist of multiple armored messages? How would that work in practice? |
Yes, since we have to support slates of arbitrary sizes and since there are limits on the size of messages in places where these armored slates will be transported (like clipboards, in email etc.), we have to allow for the possibility of multiple armored messages for a single transaction slate. Note: I have no idea if we get close to reaching this "practicality" limit with armored slates or not- we may find that there is no need to support this.
In practice it would be extremely clunky if multiple messages per slate were required outside of the existing minimum messages for normal transactions. If that ends up being the case, this is probably not the way to go. However even if 99% of transactions will never use more than one message, we do need to be able to support exceptions when they occur. Imagine a spend with many inputs which creates a slate too large to be managed in a text input form once armored. To account for this we would use multiple slate messages. The ascii-armor parser handling the input would detect that multiple messages are used and expect more to follow. Note: MessageID should only used in cases that require multi-part messages and should be the same for all parts of a multi-part armored slate, unique, and deterministically computed from the slate. The above could be an example of a slate consuming a ton of inputs. In this case it may make sense to break it up into smaller chunks to make sure our armored slates aren't too big for a clipboard or email attachment. Both parties would need awareness and capability to handle the slate that has been "chunked" and "armored". The size to chunk messages can just be an adjustable parameter tuned for Grin, PGP uses Again I hope that this does not become too relevant for our usecase, as we lose too much of our potential usability win if support for multiple message parts would be commonly needed for Grin armored transactions. |
|
Thanks for that clarification @j01tz. Once there's a clearer picture of how much fits in an armored slate, it will be clearer whether this is a good fit or not. |
|
It's great. I hope it will implement quickly. And "User withdraws grin from mining pool" method is something like what i mentioned on keybase chat:
And change file to blob info format is much better, which is more friendly on mobilephone. so I can not wait to see it to became real :) |
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| This RFC establishes a baseline for transaction building in Grin between users and services, through the introduction of a new 'hybrid' method. Transactions are initiated used two standardized formats, **QR code** and **blob**, and can then be completed over Tor or clearnet using https. |
There was a problem hiding this comment.
Keeping https will just mean users continue to use MITM services like ngrok, grinplusplus.com, and hedwig.im.
By offering QR code and blob, you've now switched from a truly standardized format (JSON - currently used for all of our slates), to 2 different "standardized" formats.
There was a problem hiding this comment.
Thanks for the review.
Keeping https will just mean users continue to use MITM services
We cannot prevent users from running third party services, so that's definitely possible, but also unrelated to the hybrid transaction method that is proposed here.
What do you suggest? Not keeping https? In favour of Tor-only, I assume?
If so:
- How do services/users in regions where Tor is blocked transact?
- How do services/users that are prohibited culturally/legally from running Tor transact?
- How do mobile wallets transact?
- How does the Grin network transact if/when Tor is down or unavailable?
To me, this would be like shooting ourselves in the foot when it comes to increasing adoption and ease of use, and it introduces a critical external dependency on a third party. The added friction might also lead to... MITM services being created, that act as relays between clearnet users and the Tor network. So I'm not sure it would solve your concern about MITM services.
That being said, since the proposed approach would support Tor-only as well, the matter of whether or not to deprecate https is independent of what is being proposed here, and should probably be handled separately (even if both proposals may end up being implemented at the same time). You're welcome to make a proposal, I believe @j01tz has also been thinking along those lines so you might want to sync.
By offering QR code and blob, you've now switched from a truly standardized format (JSON - currently used for all of our slates), to 2 different "standardized" formats
JSON is not being replaced by QR code or blob. We might end up serializing the slate in some other way than JSON, but it is the output of that serialization that will be encoded into a blob or QR code, or used in NFC, or whatever other ways of exchanging the 1st trip message that we come up with in the future.
It could also be that the blob ends up being what we encode into the QR code, and it might end up being what is passed around in all the other methods as well, replacing file for example.
There was a problem hiding this comment.
How do services/users in regions where Tor is blocked transact?
How do services/users that are prohibited culturally/legally from running Tor transact?
File
How do mobile wallets transact?
Tor or file
How does the Grin network transact if/when Tor is down or unavailable?
In that extraordinarily rare scenario, file.
There was a problem hiding this comment.
We cannot prevent users from running third party services, so that's definitely possible
But we can and should make predictions about what's likely to happen based on our experience over these past 14 months.
There was a problem hiding this comment.
File, Tor, or Tor or File as options implies that services and wallets need to give equal weight in their UIs to two transaction methods that are very different from each other, or serve a user different methods depending on which region they are connecting from, or from which device. One where you enter an address and transact, the other that's an sequential: save / drag and drop file, save/export response, and upload response.
Users need to learn both methods and form factors, increasing the cognitive load.
The model proposed in this RFC has the identical experience regardless of whether Tor is supported or not by the service or end user. It can be carried out through the identical UI and flow, and is flexible to support multiple completion methods.
Similar to how a typical bitcoin tx flow contains both an address in text and the same address as a QR code, I would expect services to provide the same slate as a QR code and a blob, and users would either copy/paste or scan with their phone, depending on which device they are on and on which device they have their wallet.
Services could provide both a clearnet Tor response URLs in the same slate payload, leaving it up the user to respond with whatever they can support on their end.
It reduces friction significantly compared to requiring two distinctly different transaction flows.
There was a problem hiding this comment.
leaving it up the user to respond with whatever they can support on their end.
And what if they can only support file? Then wallets and services still need to know how to support tor and file. I promise I'm not trying to be obtuse here, but I really can't quite grasp how this makes the situation any better at all.
There was a problem hiding this comment.
Under what circumstance would a user be able to interact with a service but not be able to support making an outbound https request?
They'd have to be offline, right? If they are offline, then I guess they are in the same physical location? If so, they might be able to initiate over NFC? That's still tbd as it seems like a minor use case for service <> user at this stage.
For user <> user, they can still interact with file, file is not disappearing?
There was a problem hiding this comment.
Right, so for user <> user, wallets still need to support file. That's my point. Wallets still "need to give equal weight in their UIs to two transaction methods that are very different from each other"
There was a problem hiding this comment.
I see now what you mean. Yes, sure. Wallets would. They probably ought to support direct Tor transacting as well. Services on the other hand, would not. And that is the point of this RFC, to improve the Service <> User transaction flows, to the point of establishing a good baseline. I don't see how offering Tor, or Tor + File, does this in a better way, and I don't understand how offering the hybrid method has an impact on the ability to transact using Tor. It's supported!
I hint at User <> User flows and File in the unresolved questions section of the RFC. It's interesting to see how the armored slate idea will proceed, it might be that we could merge File and Blob, and then hybrid becomes an optional response mechanism, primarily intended to be used when transacting with services. Not sure.
There was a problem hiding this comment.
Tor only is easier to use and provides payment proofs. How would payment proofs work in your tor + file scenario?
|
|
||
| To support this, the transaction slate data at each step in the process is redesigned and minified. | ||
|
|
||
| The introduction of this method improves security as it does not support unsafe communication methods, and it improves usability as friction and potential drop-off points are reduced in the transaction building flow. |
There was a problem hiding this comment.
I don't think this improves security much at all. Unless we're adding e2e encryption, security will be minimally impacted.
There was a problem hiding this comment.
The hybrid method enforces https so communication would be secured over TLS/SSL. This improves security over plain http.
As discussed in Keybase, the hybrid method can support e2e encryption, but it does add more friction as key or password exchange is then required. I think it would be a worth while optional feature to have, but I kept it out of the scope of this RFC. Similarly to the matter of https deprecation, encrypted slates is an independent matter - it can be added with or without the hybrid method.
There was a problem hiding this comment.
"That being said, since the proposed approach would support Tor-only as well, the matter of whether or not to deprecate https is independent of what is being proposed here, and should probably be handled separately"
How is depreciating http in scope, but deprecating https is not?
There was a problem hiding this comment.
@DavidBurkett I'm putting together an RFC for deprecating both http and https. I think I want to advocate for using armored slates and being as transport agnostic as possible in the core wallet, only supporting methods that have acceptable amounts of security and privacy (which leaves importing files/slates directly and Tor). I'll ping you when I get the outline together if you are interested, would like your input in some places before I go too far.
There was a problem hiding this comment.
How is depreciating http in scope, but deprecating https is not?
I'm probably unclear in my writing. Can you point me to the passage that suggested http will be deprecated as part of this RFC, so I can correct it?
TL;DR: grinboxIntegrating grinbox in mimblewimble/grin solves more transaction woes than compacting slates for hybrid transaction via QR codes / text blobs. Question:Are we able to choose either QR codes or text blobs seeing as they both encode the same information? Introduction:QR codes have the advantage that they allow transmitting information through the camera and text blobs have the advantage that they are easily shared via the copy/paste functionality. Design use cases:
Conclusion:QR codes and text blobs each have their own advantages, however, because copy/pasting images isn't well supported on android both should be supported for now to ensure optimal user experience cross platform. Thoughts:Even with compact slates, there's still a lot of messaging required that needs to be routed through 3rd party messaging services. This is exactly where grinbox succeeds.Imagine a UX future with light clients becoming common place and by default connecting to nodes with grinbox builtin. Creating the potential to have asynchronous transactions with human readable addresses to send and receive grin. Replacing the current transaction flow (file exchange / wallet listening / QR code slate scanning) with grinbox simplifies wallet onboarding UX. Two separate UX flows emerge:Wallet UX:
Node UX: (docker?)
Having separate UX flows removes the technical barriers to entry for people that would like to participate in the ecosystem while allowing privacy / security minded users the flexibility to meet their threat model. @yeastplume @jaspervdm @lehnberg @DavidBurkett @xiaojay @i1skn @j01tz |
|
I was able to do an extremely rough hack to see how big armored slates would be in reality. The initial results are somewhat promising, especially with @yeastplume's compact slate improvements. One major advantage that armored slates have over plain json is that a checksum is included which makes them more resilient for transport, regardless of the medium. They can also be intuitively extended to support encryption. My intent is to follow PGPs specification as in RFC4880 as close as possible as it is a well-used and established implementation of ascii-armor. The transport details in the header can be moved to the slate in theory and the advantages and disadvantages are probably worth discussing. Finally, these slates will all be smaller in practice because I didn't want to spend a ton of time trying to make things efficient and perfectly accurate if we weren't even close to having a manageable armored slate size. Let's see how we did... Below is an armored V2 Grin slate with two participants: This is still manageable when copy/pasting but would quickly get out of hand with many outputs. I was not able to generate a static, black and white QR code. Next we have a V4 compact slate with two participants. This size is much more manageable but we still need to do more testing to see how bloated things will get in practice.
Finally we have an even smaller v4 initial compacted slate. This was easily able to fit in a QR code. With some further slimming down I think it would be reasonable to expect consistent QR availability for the initial slates (which might be the most important anyway?). Before I spend much more time on this direction I'd be curious to have input from others. I'm used to this UX flow from PGP but I realize PGP is known for notoriously bad UX so I don't want to force us too hard in that direction. |
What's the flow like for transacting between a couple of mobile wallets?Opening ports requires users root/jailbreak and hosting a tor listener locally on Android/iOS is a development nightmare. |
Mobile to mobile could work with a sequence of QR code exchanges using the mobile cameras as the transport method. You would encode the contents of the armored slates into a QR and exchange via mobile camera. The viability of fitting all slates into standard static black and white QR codes to facilitate this isn't determined yet- it will work with the most simple transactions but I need to get a better idea of the limits of both theoretical and practical slate sizes.
The above mobile to mobile via QR method would not require any transport channels outside of QR/camera so would prevent challenges of managing ports and supporting Tor. |
|
@tenthousandlakesmn as co-creator of grinbox your suggestion warms my heart 🙂. It also fills my head with some questions:
|
|
@j01tz: The QR code handshake only works when the phone are in the same place. How will you buy me a beer when you're in Minneapolis at Target Field for the Green Day show and I'm in Duluth at the Fest of Sail.
Grinbox functionality should ship with each node so grinbox relay's go 1:1 with nodes on the network.
Both http/https are built native in grin, add grinbox.
The end to end encryption of the slate data in grinbox prevents the relay from knowing anything other than the IP's and grinbox addresses of the transacting users. Integrating with tor handles IP exposure.
Look at electrum in bitcoin.
Requiring a small PoW proof before accepting a slate is one option. |
Good question. Unfortunately the best answer I have for now would be to use signal, keybase, email etc. to exchange armored slates. They all have varying risks depending on your threat model but it would be left up to the user rather than the wallet forcing a potentially undesirable transport method. |
|
I am not really sure how this is truly different from file slates, just another mode of transferring information that file slates provide. It does help one trip of the communication but not the other. It only addresses transacting with service providers (or anyone with domains and certificates for https). It does not address user-to-user exchanges at all. How would one user send grin to another user? The "default" method of sending and receiving grin cannot be one that relies on service providers (or relies on domains and certs) |
|
@johndavies24 what's different from current implementation is that:
These three things make it quite different than how the slate used in file transfer works today. Not night and day different, but substantially different enough to require an RFC, something David seems to agree with. |
|
In the linked comment, the RFC I was talking about was just to determine which fields belong in the slate. I was not expecting a change to workflow in any way, but if that's our goal, I agree this is a good time to discuss it. |
|
NFC can push ~424kbit/s so it doesnt need any compression, although it would only help. Also, you made multiple statements that are not accurate in your description. It is not difficult for different devices to share files, they dont need to be same make or even same OS/platform. It is not difficult to have HTTP listener on mobile, beam has it for both platforms and it works in the background on android. TOR is also easy on mobile, both blockstream green wallet and verge wallets use it on both platforms. I think you've made a lot of assumptions that are demonstrably false and are spending a lot of time working on something that at best solves one problem and rather than solve it you may just end up pissing off exchanges as a GUI that is more or less the same for every other coin becomes more complicated for them to deploy. You still never explained how user-to-user would work without domains or certs or how they would even share the QR code remotely, etc etc. #3 is false as you can do that with file slates already, you're not actually adding anything except for support for primitive 2D barcodes. The reason it needs an RFC is because of how slates are changing, not any other aspect, which as it seems will not actually be provided by grin core. This is a whole lot of effort with fingers crossed that someone will build out for it when "default" should be supported and provided by reference applications. |
You do not need to root/jailbreak to have http listener and there are multiple wallets that use tor on both android and iOS. There are multiple opensource wallets for both platforms that provide examples of how to do this. |
What about mobile to mobile in the real world.... when you arent standing right next to each other? |
You are literally suggesting to use the mobile wallet you have to leave the wallet and enter another app? You guys cant be serious with the system you guys are working on. D.O.A. and even more-so considering grin core refuses to provide cross-platform reference wallets that are usable and refuses to assist the devs who provide cross-platform wallets that are usable. |
To clarify, I'm suggesting that the core wallet might consider not having a strong opinion here that forces users and developers into a transport method that all users may not be able to access depending on their location/threat model (Tor, even if easy to support for mobile still suffers from the fact that it will be blocked at the packet level in some countries without extra steps and https will give out your IP address without taking steps). I expect we may end up with a few varying RFCs on the topic and eventually converge on one or a combination of some to ultimately address the transaction building issue. From my perspective (which may not be the right one!) I think starting with the minimum requirements for a private, secure and guaranteed to work method, and leaving opinions that sacrifice those to varying degrees in the name of usability to other wallets built on top of the core wallet is an approach worth considering. I expect down the road some wallets will be geared to be more user friendly but maybe not have the best privacy protections and some wallets will be built for privacy at the sacrifice of some user convenience- the important thing is that the core wallet provides the tools necessary to do both of these things without imposing one on the other IMO. |
|
It seems like core deciding to minimize slate size might be appropriate both as something for the core to do and in use of time. But the core coming up with ways for 3rd party developers to do things seems out of place. You guys are spending a lot of time doing what the core tends to claim is the job of 3rd party devs to build out. The core should focus building out things that it actually intends on delivering the entire set of tools (or the minimal necessary to open the door for others to build the tools). |
|
Just my 2 cents:
After all, I see this RFC as positive for me, cause I would be able to improve UX for users with doing less, as of today. |
|
It might be useful to come up with a unique file extension (e.g. *.grin) to help with any instance or use that involves a file slate. Part of the issue I've had with file slates is either no extension or .json, either way certain platforms dont know what to do with them. This may be beyond the scope of this RFC, but this could be considered as one means of improving the file slate situation |
|
Over the weekend I've been thinking about this RFC, and in particular @DavidBurkett's comment above[1]. There's been a lot of activity, which is great. While David (Grin++) seem skeptical to the proposal and motivation behind it, comments from @xiaojay (Niffler) and @i1skn (Ironbelly) are more positive. This to me says it's at least worth while to try to refine the topics touched on here further. Now how to go about that? The discussions are all over the place: Tor vs https, user<>user vs user<>service, Grinbox vs QRcodes, encrypted vs non-encrypted slates, armored slates exploration by @j01tz, and so on. There's no discussion about serialization or about the actual slate contents itself, the latter of which is what @yeastplume is actively working on. This leads me to believe that the RFC is a bit too big and unfocused at the moment. I took this approach because @yeastplume was asking for a motivation for why we ought to compact slates in the first place. I think he's still asking for the motivation for why we should serialize slates differently than what we do today. If we can agree that there's an underlying motivation to proceed with 'compact slates' for the benefits of 'further research into transaction building improvements', then I think it may make sense to close this RFC in favour of several smaller ones: Immediately:
Closely followed by:
And at some point thereafter, pending further research and deliberation:
All of these efforts could be accepted/rejected individually and do not need to stand/fall together. As for QR codes, @i1skn had thought this could be handled on the wallet implementation level. For some reason at the time, I thought it would be important to have a single "standard" around the QR code design and content. I'm not so sure any more. As long as it's clear what goes into the QR code (be it blob, armored slate, or something else), it might not need to be a "standard". Some wallets or services might execute a different QR design and encoding approach. That's their problem. As long as the encoded content is the same all across, we would achieve interoperability, which is what is desired. As it's not clear at the moment what that encoded content is, I don't think it should hold up progress on other RFC work that can happen in parallel. So what do you think, does it make sense then to close this PR and split the RFCs up? [0] #46 (comment) |
|
So with support expressed in the wallet_dev chat and in the dev meeting for closing this in favour of several smaller, more focused RFCs, I'm closing this pull request.
...are all targeted for v4.0.0 through individual RFCs, you can follow progress here: mimblewimble/grin-pm#248 The rest will hopefully follow. Thanks for all the comments and valuable feedback! While this means the death of the RFC, our thoughts will live on in spirit in the many sprouts of RFCs that are springing from its ashes. 😹 Feel free to point to this closed PR in the motivation section to justify why it's worth while doing the smaller chunks. |
Rendered link to document
Partial first draft. I think it's got enough meat on it for some early discussion (see open questions) and to gauge if there's enough general support for this direction to pursue further, if a change in direction is needed, or whether the whole thing should be abandoned.
Current status:
Pinging usual Tor & Wallet suspects @yeastplume @j01tz @DavidBurkett @jaspervdm.