Crates should allow private impl of external traits for external structs #493
Comments
|
I'm very much in favor of this. It should be possible to extend existing types without having to wrap them in a unit struct, or define a new trait where it doesn't make sense to (e.g. adding a single helper method). |
|
I agree that the current restrictions enforcing coherence (essentially global uniqueness for Having globally unique instances is something of a devil's bargain: convenient at a small, local scale, but leads to a number of problems in the long run, seriously limiting how the language can evolve. This is the situation Haskell is in now, having embraced coherence: see here, and here. I'm a little concerned Rust is making a similarly unfortunate (and unnecessary) choice… Distinguishing the definition of an Aside from helping to restore some modularity now and retain flexibility going forward, explicit canonicalization opens to the door to more general and more powerful mechanisms for implementation resolution. For example, see the work on explicit unification hints here subsuming type classes, canonical structures, etc. Another well-known but somewhat less general approach using ML-style modules is modular type classes. |
|
On the other side of this debate, of course, is Edward Kmett. He was going to give a talk about this, but it got cancelled - I hope it'll be rescheduled at some point! |
|
@glaebhoerl I want to point out that I'm not advocating "implicits" or "ML-modules" per-se as an alternative to type-classes, so in that sense I don't think what I'm proposing is on the "other side" at all, but something else entirely. I would advocate an approach that does not require global uniqueness but would still allow the current behavior to more or less be recovered in a controlled fashion and as a special case without sacrificing modularity. I don't think Kmett addresses this possibility. Having said that, I think Kmett has some great ideas and has done some really amazing things with Haskell but where coherence is concerned, I don't find his arguments convincing. To me, it basically boils down to "coherence is really convenient, and besides look at all we've managed to accomplish with Haskell type classes". What I find lacking is a principled response to the alternatives, e.g., canonical structures, unification hints or other more general mechanisms, or the differences in the way type-classes are implemented in various theorem provers. Part of the reason for this is that it's difficult to compare the alternatives in a practical sense given differences in languages, libraries, and other factors. But that doesn't make the argument in favor of Haskell's approach any more convincing. I also feel such arguments ignore most of the collateral complexity of embracing global uniqueness and trying to work around the consequences elsewhere: several random extensions relating to type-classes, issues with the proposal for the module system (backpack), generalized newtype deriving (with all its quirks), roles, etc. Global uniqueness only simplifies some things from a particular perspective. As a general language design principle, it's hard to justify (especially in retrospect!) and I believe definitely a case of "worse is better". There are very real and well known pain points associated with embracing coherence, as the op points out. I guarantee these will only become more of a thorn in the side of Rust users as the language grows. It would be a shame for Rust to simply adopt the status quo on this rather than reconsider some of the decisions Haskell has made while it's still within the realm of possibility. Furthermore, it seems to me (and I could be off base), that one of the principles behind Rust is to avoid making things implicit when having them explicit offers greater flexibility and control. The current approach ties two different concepts together, definition and canonicalization (globally), where the latter is implicit and doesn't need to be. |
|
+10000 Changes like this would draw me back to the language. |
|
@darinmorrison couldn't agree with you more. Need to look at Edward's comments more, but I am not sure where he could have gotten that opinion. By separating cannonicalization and definition, one can fine tune the coherence rules to exactly what they think they should be (which seems to very from person to person) without sacrificing the expressiveness of the language. Does anything more need to be said? (As an aside, I started writing an RFC https://github.com/Ericson2314/rfcs/blob/master/active/0000-combine-mod-trait.md, which would combine traits and modules, around the time of the associated items RFC. I was told that this was too much to do for 1.0 -- even more obvious at this point in time than then. To reiterate, I do see plenty of advantages to separating cannonicalization and definition, even if traits and modules are kept separate.) |
|
@Ericson2314 interesting… I remember seeing comments about ML style modules somewhere but didn't realize anyone had started writing up an RFC. Even aside from this issue, it would be pretty awesome to have them in Rust if it were feasible. (Higher-rank polymorphism and existentials would also be great with HKTs but I agree it seems tricky with the monomorphization.) I would be willing to help with an implementation effort somehow. /cc @jonsterling |
|
You can get a correct local version of the reasoning that Edward Kmett takes as the main advantage of type classes with the Modular Type Classes approach, albeit with some changes to the version described in the paper. The original paper models type classes using ML modules with generative functors (meaning that a functor applied to the same arguments gives fresh abstract type components in the resulting module) rather than applicative functors (meaning that each distinct application gives identical abstract types in the resulting module). This implies that instance functors are required to be fully transparent, with no abstract type components. If they were modeled with applicative functors instead, you could use ordinary type abstraction to enforce Edward Kmett's desired guarantees from type classes locally. An email to the TYPES mailing list and some slides explain this in a bit more detail. It's unfortunate that this has never appeared in a real programming language, since it really seems like the correct way to combine modules and type classes. There is another issue that comes up with modeling type classes using ML modules that wouldn't really be a problem for Rust. Haskell type classes allow you to write programs that have an unbounded number of instances at runtime: f :: Eq a => Int -> a -> Bool
f 0 x = x == x
f n x = f (n-1) [x]
main = do n <- getLine; print (f (read n) ())Encoding this example with ML modules would require first-class modules, but with Rust you already encounter ad-hoc failure during monomorphization for examples like this. [Edit: @jonsterling pointed out that I swapped generative and applicative in describing the original MTC paper, even though it was clear what I meant from later context.] |
|
@zwarich Interesting! Furthermore, Since Rust doesn't allow impure My draft RFC proposes much of the functionality discussed in the slide show (arguments on functions being lifted onto the projection, etc), but I only hoped it would work out. Nice to learn from the slide show that it actually does (with pure applicative functors)! |
|
@Ericson2314 If you want to see the semantics for an ML module system with both generative and applicative functors worked out in detail, the F-ing Modules paper by Rossberg, Russo, and Dreyer is interesting. They express all module system features by translation into System F_omega. If you want recursive modules with type abstraction, then you have to extend System F_omega to something like Dreyer's RTG. |
|
Might be good to see if we can get @RobertHarper to weigh in on this, particularly on the question of whether or not generative functors make sense in a Rust-like language. (Bob, if you're too busy and don't want to receive notifications on this thread, there's an unsubscribe button on the right side of the screen that you can hit.) |
|
I haven't a clue about the context, but if someone wishes to summarize I could comment or make suggestions. I see that Modular Type Classes are being discussed, which I think are a good idea, of course, and cleaner and more general than the over-evolved mess in Haskell. In particular instance declarations in Haskell are wrong-headed; one should separate the instance itself, which is a functor, from its activation for use during type inference. And then one should be able to instantiate explicitly rather than always rely on the inference mechanism to trip over the thing you want, another problem area. In my view modeling type classes as run-time records is wrong-headed; instances are static, because they are inferred during elaboration. If you wan to pass actual run-time generated records or existentials, please do so, but that's not what type classes are for. It may be helpful to look at our old paper on separate compilation for Standard ML. We thought long and hard about this, but no one paid any attention afaict, for largely "meta" reasons of the timing. But maybe none of this is relevant to your discussion, so forgive me if not, and please fill me in if I can be of help. |
|
@zwarich Can you elaborate on how type abstraction can be used to enforce Haskell-style “coherence” in certain cases? This is interesting. @RobertHarper Hi Bob, Thanks for popping in. I was wondering about the following: In spite of the fact that Rust has more fine-grained structure than ML wrt mutability & references, I think that generative functors are probably crucial and shouldn't be ignored. It feels like applicative functors aren't actually that useful except when you consider them as enabling certain patterns of use with type classes as Derek's slides point out. Moreover, a module system with type classes can start off with only generative functors and be perfectly useful; then applicative functors can be added later to sort of patch over the areas that were slightly more difficult to use. Bob, do you have any comments or corrections to what I have said above? |
|
hi john, not sure i understand "fine-grained srtucture" in rust, so it's hard for me to answer. i agree that applicative functors are a theoretical curiosity, and a bit of a hack in ocaml (they use an approximation of code equality, violating repind, to distinguish Set(X).t from Set(Y).t. There's a decent theory of it now, done by Russo, Dreyer, and Rossberg (in some order), but they are of marginal use in any language with state, certainly, and even without. The only argument for applicative functors was to have useful higher-order functors, but that never panned out. The only reasonable solution for higher-order functors is essentially what Tofte and MacQueen proposed a long time ago in the yucky Definition style; this may or may not be what was used by Dreyer, et al in their paper (I don't recall and don't have it to hand.) |
|
BTW, I don't understand why generative functors interfere with type classes, can you explain? As long as you propagate that "type t = int", etc then the instance is going to be type-transparent, regardless of the functor being "generative". BTW, I'm still not sure that I like the terminology "applicative vs generative functors", because it's not really about the functors, but about tracking the effects using type abstraction. |
|
Bob Re type classes and gen. functors, I think you are right, and I seem to recall that we had a similar conversation at one point where I raised a concern about generative functors and type classes, but I had forgotten it. Honestly, I'm not really sure of the point of having applicative functors in Rust... |
|
I wouldn't know the designer's motives, of course, but I highly doubt there's any technical justification. Putting it the other way 'round, I'd like to hear the justification for it in Rust. |
|
@RobertHarper Excusing the use of the terms 'generative' and 'applicative', doesn't it make more sense for instance functors in the Modular Type Classes approach to be applicative rather than generative? The paper uses generative functors, and this forces instance functors to be transparent, so that distinct applications of the functor that arise from elaboration produce compatible types. Using applicative functors instead would allow for transparent instance functors. That's what Dreyer argues in those slides I linked, and it makes sense to me. Am I missing something? |
|
Type classes are by nature transparent; they cannot be opaque, because if they were, they'd be useless. (See PFPL for a discussion.) I don't see any advantage to applicative functors for anything, and not for type classes, unless I'm missing something. All that applicative functors do for you is to make abstract types more often equal than they would be without them. I don't see the relevance to type classes at all. |
|
Bob, thank you for coming here. First things first. Rust does not allow any effectful top-level (or impl-level, the closest thing to module-level) definitions. (Functions defined here can of course have effects, but the definition of function themselves is effect free.) Put differently, all static initialization is effect-free. This leads me to believe that were functors added to the Rust of today, they would need to be effect-free, "generative" vs "applicative" semantics aside. Does anybody disagree? |
|
No, that is irrelevant. Even in a totally pure functional language, you want only generative functors, because you do not want to confuse, say, posets ordered two different ways. The one-liner is that type abstraction is an effect. That's why existentials combine open with bind. To reiterate, you never want applicative functors, and never ever these only. Robert Harper
|
|
Mmm sorry, by "they would need to be effect-free" I meant that generative functors in Rust would be able to do only what generative functors do in a "totally pure functional language", i.e. no incrementing of global counters or anything like that. My intention was not to squash all discussion of generative functors. I see that "effect-free" is really the wrong term for what I meant. |
|
well, yes, there would be no storage effects, but there are nevertheless abstraction effects. Robert Harper
|
|
Now that I have a keyboard .... Even if Rust is effect-free at initialization time (but then what does initialization consist of if not effects?), so-called generative functors nevertheless introduce a top-level effect, namely type abstraction. Actually, even structure declarations with a "sealing" effect introduce such effects, so it's not really about functors, it's about abstraction. The only reason the discussion becomes about functors is that it is of paramount importance to be able to distinguish statically between different instances of a functor, even if the type parts are the same, because the instances proxy for the difference of interpretation provided by the code attached to the type. So Posets of natural numbers ordered by LT should be distinguishable from those ordered by GT. This is not a matter of effects, notice, but in the presence of effects it becomes even more important, eg one wishes to distinguish statically between two different instances of a hash table, even if all the parameters about keys and values are the same. This example is about state, of course. Historically, the invention of applicative functors was a dead-end attempt to have a useful notion of higher-order functor, which does not fall out easily from standard type theories of modules. (Nor is there a case for higher-order functors even being very useful.) Even with so-called applicative functors, you still don't get a very useful concept of higher-order modules. Standard ML modules are a local optimum that cannot easily be improved upon without making very large changes and introducing considerable complexity. The main improvements that I can envision are modular type classes (mostly to keep up with the Peyton-Jones's, but they can be handy in limited circumstances), and my re-working of data types to better integrate with modules. The latter is definitely worth doing, for a host of good reasons, and is the most significant improvement that I know about to the SML module system. Does Rust have data types in the sense of SML? If you're doing them like in SML or Haskell or O'Caml, then I claim you are doing them wrong, and I know how to do them right. |
|
Bob, do you have a paper or any literature that summarizes your re-working of data types? If I recall correctly, the idea was to support something like this: signature OPTION = data
type 'a t
con some : 'a -> 'a t
con none : 'a t
endAs a result of the special declarations the signature also exposes pattern matching somehow. My questions: does this signature give rise to a default standard implementation structure |
|
the design is implicit in the harper-stone semantics given in the milner volume, and is sketched explicitly in my ml workshop talk, which is available on my web page. i haven’t taken it further, but between the two i feel confident that it can be made to work out. the main ideas are as follows:
the tag “data” on “structure” makes ‘a S.t available for pattern matching. in 2a it also indicates what it means to have a default implementation (as a recursive type). when you implement it yourself, there is trouble if you use effects, because the pattern compiler re-orders things to optimize pattern matching. a modal distinction would be helpful here, so that pattern-matching implementations could be required to be pure. there are details to work out that will no doubt complicate matters, such as mutually recursive data types, or gadt’s, which are a hack in my view anyway, so maybe i don’t care about that so much. there are opportunities to combine this with modular type classes to get a general form of “deriving” in haskell (one that works more broadly), and for providing special syntax in a non-ad hoc manner. bob
|
|
(Sorry to butt in, via Twitter).
To summarize @zwarich's point, Derek Dreyer agrees that the classical motivation for applicative functors are not interesting, and makes a different point in the links. That point is not part of the paper on F-ing modules (by Rossberg, Russo, Dreyer) which you cited, and it doesn't seem to have been published.
A summary of Dreyer's point follows (as a separate, longer comment), in case it helps. |
|
Dreyer's canonical example is the MkSet functor: it's a natural candidate for being an instance functor, but it's not transparent, so you need to apply it explicitly even to instances you make canonical — to which Dreyer comments:
Here's MkSet from his slides: To answer @RobertHarper's point:
Dreyer agrees, but provides a different solution (which however does appear in the F-ing modules paper, at least in the journal version). If it weren't for decidability, you'd just want to compare the ordering functions with contextual equivalence. To approximate that, instead of adding stamps to the result of (generative) functor application (which IIRC is one way of understanding generative functors), you can attach those to value declarations (at least, "moving the stamp generation" is how I understand this, he doesn't say it), so that applicative functors will distinguish posets ordered in different ways. From the email:
As he points out (in the slides), this only works for pure applicative functors. I think "semantically we'd want contextual equivalence, but can't have it" should be a compelling argument. I'm not sure I like this conservative approximation, but it's certainly less conservative than pure generative functors, and I don't see downsides — Dreyer claims it solves at least all the problems in Ocaml.
By looking at Dreyer's page, that paper must be the already cited F-ing modules (journal version). On terminology, they also write:
|
|
To be extra-clear: I don't mean to be polemic, I'm sincerely interested in @RobertHarper and @jonsterling comments/arguments on Dreyer's design (and everybody else of course, but we seem to already buy Dreyer's points). I'm especially interested because I'd like to use similar designs in other contexts. |
|
Yes, I just woke up and I'm quite groggy. |
|
If you want to depend on a specific version, then you can just "vender the dependency" to use Go speak. In that vein, you could maybe avoid violating separation between cargo and rustc by making the dependency live in your own source tree, and enforce the dependency on a specific version using git submodules. As an aside, I do think Go's policy of expecting people to vender dependencies is kinda antithetical to free software, but this usage sounds reasonable. And git submodules stand out like a sore thumb too. |
|
I probably missed something in the discussion, but if concern is that upstream might add own
|
|
On your first point, the key difference is that there is a fully explicit syntax you just need to transform to to fix the error - On your second point, that would violate the coherence property we're trying to maintain. It would be very bad to violate this property because of something called 'the HashTable problem'. For example: mod foo {
impl Hash for i32 { ... }
fn f(mut table: HashMap<i32, &'static str>) {
table.insert(0, "hello");
::bar::f(&table);
}
}
mod bar {
impl Hash for i32 { ... }
fn f(table: &HashMap<i32, &'static str>) {
assert_eq!(table.get(0), Some("hello"));
}
}You might get a different value out for the same key if you move the HashMap between modules/crates, because it will Hash with a different impl. This would be quite bad. And of course it applies to all kinds of traits, not only Hash. In general, when people write code they assume that if you call a method on the same value, you keep dispatching to the same place. Basically, methods ought to be type scoped, not module scoped. |
|
In fact, there is a safer version of this that you can do in Rust @RReverser so long as all you want is convenience in your own crate. Just |
|
@burdges That's exactly the entire point we're trying to avoid with local derives (which is this issue about) - not having For example, if upstream defines own implementation for |
@withoutboats I'm not sure this is a strong argument, this applies to any languages and to anything that can be named, not just traits - e.g. if I have same-named structure called Not really a trait issue in any way, just normal scope semantics. |
@RReverser There's a misunderstanding about "move". In general, |
|
In addition, you don't get the hashtable problem if names refer to different items in different scopes, you get a type error. e.g. mod foo {
struct Foo;
fn foo() -> u32 {
::bar::bar(Foo)
}
}
mod bar {
pub struct Foo(u32);
pub fn bar(foo: Foo) -> u32 {
foo.0
}
}You don't get some undefined behavior hear, you just get a compiler error. And you can always call But impls don't have names and aren't scoped. In order to scope them, you'd have to give them names, and then pass them around as well ass the types. In order for this to behave well it wouldn't be |
Oh right, thanks for explanation. Although not sure if that's a big problem? Exactly same applies to
Yes, that's exactly what I'm suggesting to change. |
Not if it's a function or variable name or ... E.g. when you have |
|
@RReverser Specialization ( |
|
Namespacing is not the same as what we're talking about. The important thing is that you can't pass something with one shape from one module to another just because they have the same name. |
Ah fair enough.
@withoutboats Here you couldn't either, in both places object needs to implement same trait, even if specific implementations differ (just like with functions where you still must have same arguments&return types, but implementations don't matter). |
|
That's not correct, the implementation is a part of the shape. What you're describing is not lexical scope but a very implicit kind of dynamic scope, because hashing calls inside of HashMap can't call free functions from the module its called in unless you pass them in as arguments the same way it can with methods. Its totally different! Methods must be dispatched based on type and not scope, because they are passed around with the type. |
|
Well I'm open to hear other suggestions. Example: I want all Neither I want to implement custom Note that this isn't specific to If speaking not about lexical scope, would it be possible, for example, to have custom trait implementations bound to owner struct? |
Centril
added
the
T-lang
|
Amazing this issue is still open after all these years. After searching around for a solution, this has led me here and I'd like to share my experience as a new Rust user. This problem keeps coming up again and again for me and it's so frustrating. The design choice seems backward in thinking to me. It made me want to give up on Rust because I'd get stuck for days trying to find a solution of which a lot are discussions of why this can't be done and the end is some sort of workaround. Rust as it is already has a high learning curve, why make people's lives more difficult? When learning a new language or anything for that matter, the experience should be smooth and the user should be able to keep progressing forward. This just stops dead the process. I hope there's a good solution in the pipelines to address this. Thanks. |
|
@kintarowins as you can see, it's a difficult problem, and solving it the way being suggested (allowing private impls) is simply not possible. IMO, the most promising solution is to introduce a way to create a "strong type alias" - ie. a convenience for defining a local new-type wrapper around the foreign type and automatically inheriting some or all trait implementations from that type. So far, nobody has submitted an RFC with a workable solution to "delegating" trait implementations in this way, but the language team have said on previous RFCs that this is something they would be interested in a general solution for. |
|
Could someone concisely explain what the cost is for allowing private impls of traits? My understanding is this:
Did I miss something else? Could someone help me understand why allowing private impls is "simply not possible"? (Said by @Diggsey - thanks for sharing this view btw). I don't disagree, I just want to make sure that I've really understood correctly. Thanks, |
|
I'm not exactly sure which suggestion Diggsey is responding to, as there are many ways to interpret "allow private impls" which are possible (but still non-starters due to other massive downsides). What is true is that you can't "just" allow private impls without precisely specifying what happens in all the other crates. I'm guessing what Diggsey meant is that because of all the other constraints, the only plausible answer is "to other crates these impls effectively don't even exist", and then due to "the hashtable problem" discussed above (i.e. the fact that your trait methods may get invoked in someone else's crate) this wouldn't actually solve most of the reasons people think they want private impls or other orphan rule exceptions, because your private impl wouldn't even get used. See https://github.com/Ixrec/rust-orphan-rules for my best attempt to summarize all the constraints involved. Semi-related, but we do have an RFC for delegation now, and it turned out a large chunk of delegation can be done entirely in proc macros (e.g. ambassador). |
|
I am curious as to the status of this and interest by core three years and several postponed RFCs later. As I understand it, the two issues that cause such a feature to not be a simply-of-course implementation are:
I struggle to see how these are problematic enough to outweigh the usability benefits that such a feature would bring, however, particularly given that these are both problems with ordinary functions outside of traits, and no one seems to run into any issues with it. Perhaps I under-appreciate coherence. But even so, there are syntax proposals that would either keep coherence or make its disregardal extremely explicit. Maybe I'll just get used to it, or maybe I'm just writing weird code. But I run into this issue seemingly in every project I do, in its various forms*, and have to resort to various unseemly workarounds*. Coming from a more free-form language this is probably the
|
|
At a minimum, I would love to see some kind of exception made for transformative traits like Into/Display so that way we can at least have a rust way to consume the type and return something more friendly to the current scope. Not saying a user should be allowed to overwrite a trait, There should only ever be one implementation for a trait per type. |
and others
Thursday Apr 24, 2014 at 02:31 GMT
For earlier discussion, see rust-lang/rust#13721
This issue was labelled with: in the Rust repository
As you know, you can't provide an impl for a trait if neither the type nor the trait are defined in the current crate, you get:
For public implementations, this makes perfect sense, there are issues with collisions between implementations and sheer surprise. However, for a private implementation of the trait, this is a real hindrance.
Consider the case of
std::path::Pathnot implementingstd::fmt::Show. The rationale for closing #13009 is perfectly valid: we don't want people treating paths as strings. However, by refusing to add an implementation ofShow, you have made that decision for all programs everywhere. In my case, I had a large struct with numerousPathelements that I wanted to print for debugging, but#[deriving(Show)]won't work becausePathhas noShowimpl, and now I'm stuck either implementing it tediously from scratch, or switching tostrfor my path names.The perfect compromise would have been to allow my crate to define a private
Showimpl forPath. There is precedent for this in other languages, go allows interfaces to implemented anywhere, for example.The text was updated successfully, but these errors were encountered: