unsafe: add Slice(ptr *T, len anyIntegerType) []T

[mdempsky]

reflect.SliceHeader and reflect.StringHeader are clumsy to use because their Data fields have type uintptr instead of unsafe.Pointer.

This proposal is to add types unsafe.Slice and unsafe.String as replacements. They would be declared just like their package reflect analogs, except with unsafe.Pointer-typed Data fields:

type Slice struct {
    Data Pointer
    Len int
    Cap int
}

type String struct {
    Data Pointer
    Len int
}

Additionally, I suggest that for the purposes of type conversion, we treat that string and unsafe.String have the same underlying type, and also []T and unsafe.Slice. For example, these would be valid:

func makestring(p *byte, n int) string {
    // Direct conversion of unsafe.String to string.
    return string(unsafe.String{unsafe.Pointer(p), n})
}

func memslice(p *byte, n int) (res []byte) {
    // Direct conversion of *[]byte to *unsafe.Slice, without using unsafe.Pointer.
    s := (*unsafe.Slice)(&res)
    s.Data = unsafe.Pointer(p)
    s.Len = n
    s.Cap = n
    return
}

While the same results can be achieved using unsafe.Pointer conversions, by using direct conversions the compiler can provide a little extra type safety.

[ianlancetaylor]

If we do this, we should figure out a way to exempt these new types from the Go 1 compatibility guarantee, so that we can change the representation of strings and slices in the future. I'm not sure how best to do that.

[cespare]

@ianlancetaylor reflect.SliceHeader and reflect.StringHeader already try:

It cannot be used safely or portably and its representation may change in a later release.

but the compat doc itself gives a strong exemption for all of unsafe:

Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs.

ISTM that unsafe.{Slice,String} would already be exempted sufficiently.

[rsc]

Go 2 seems like the time to think about this (and reflect.SliceHeader etc).

-rsc for @golang/proposal-review

[bcmills]

This proposal seems a bit redundant with #13656.

How much of the use-case is "create a string or slice aliasing C memory" vs. "manipulate existing strings and slices by tweaking header fields unsafely"?

[mdempsky]

I'd like to suggest renewing consideration of this proposal for Go 1.14. I think it will be useful for users trying to address issues flagged by -d=checkptr.

I'll also offer a counter-proposal that I think better addresses most end user needs in a more ergonomic manner:

package unsafe

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType
func String(ptr *byte, len int) string

[Edit: As discussed below, I'm now in favor of combining Slice's len/cap parameter into a single parameter.]

This is a little less versatile than exposing the Header types, but I think it will minimize typing for most users, while also providing better type safety.

We could also do both this proposal and my original one, if we want to still offer the full flexibility of the Header types. In that case, I would suggest renaming the types to SliceHeader and StringHeader, and reserve the shorter Slice and String identifiers for the constructor functions.

[bradfitz]

I like that counter proposal API.

[mdempsky]

A few additional thoughts to add to my counter proposal:

1. We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

   Edit: ptr == nil && len > 0 is another case to consider.

   Edit 2: Also, len > MAXWIDTH / unsafe.Sizeof(*ptr).

2. The functions would be builtins; in particular, users can't write f := unsafe.String; f(...).

3. The cap argument to unsafe.Slice can be optional; if it's omitted, the len argument is used. (Just like make([]T, n) is shorthand for make([]T, n, n).)

4. Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows. (I.e., make([]T, uint64(10), int8(20)) is valid, even though uint64 and int8 aren't normally assignable to int.)

5. Since unsafe.String would be a builtin, it could evaluate to an untyped string.

[bcmills]

That API is closer to what I had suggested in https://golang.org/issue/13656#issuecomment-303216308, and we've been using that variant within Google for a couple of years now without complaints.

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

[bcmills]

We should decide what happens when len < 0 or cap < len. I'm leaning towards panic, but maybe we should just leave it unspecified/undefined.

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Perhaps the int parameters should actually follow the same goofy semantics that make([]T, n, m) follows.

That would certainly smooth out the call site in the (overwhelmingly common) case that len and/or cap is a C.size_t.

[mdempsky]

If the type desired for the slice does not match the pointer that the user has (for example, if one is a cgo-generated type and the other is a native Go type), I'm assuming that the caller could do something like:

	var s = unsafe.Slice((*someGoType)(unsafe.Pointer(cPtr)), len, cap)

to set the element type?

Yeah, that's my thought. If a user wants to convert *T into []U, then I think it's reasonable to require an explicit conversion there.

[mdempsky]

I would leave it unspecified, but panic is a fine implementation of “unspecified”.

Ack, though my concern is if we panic by default, then users might come to rely on it panicking and not write their own checking.

It would be easy to put the panic behind -d=checkptr though.

[bradfitz]

func Slice(ptr *ArbitraryType, len, cap int) []ArbitraryType

Can we instead do:

func Slice(ptr *ArbitraryType, len int[, cap int]) []ArbitraryType

... with an optional cap. Where omitting cap means cap == len?

[mdempsky]

@bradfitz Yeah, that's my additional thought #3 above. :)

[bcmills]

if we panic by default, then users might come to rely on it panicking and not write their own checking.

Hmm, good point. We could make it a throw! 😉

Or we could make it a panic in ordinary code but a throw under -race or -d=checkptr. (The important thing, I think, is to vary it just enough that it causes tests to fail in some reasonably-common configuration.)

[yaxinlx]

, it's easy enough for the code to check the pointer for nil before calling unsafe.Slice.

It could, but it is a little inconvenient. And it is some inconsistent.

[mdempsky]

But should it returns a nil slice instead if ptr is nil and len is zero?

I had considered that, and I still haven't ruled it out. But we had to pick something, and I don't think it's obvious returning nil in that case is necessarily better. It's also easier to relax an overly strict API than to restrict an overly lax one.

My intuition is it the inconvenience, if any, will be minor. However, if actual use demonstrates that returning nil for unsafe.Slice((*T)(nil), 0) is what users almost always want, I'm open to relaxing the rules to allow that.

[bcmills]

I don't think it's obvious returning nil in that case is necessarily better.

FWIW, I think it is obviously better — unsafe.Slice constructs a slice with the given pointer and length, and the slice with pointer 0 and length 0 is exactly the nil slice (https://play.golang.org/p/hsIOG1wYBhc).

(Compare Ousterhout ch. 10, although I agree that it's not always clear.)

It's also easier to relax an overly strict API than to restrict an overly lax one.

If that's the case, the documentation for unsafe.Slice should say “ptr must be non-nil and len must be non-negative”, rather than promising to panic. (A panic in Go is a defined control-flow construct, not a programmer error.)

That would leave open the possibility of changing the behavior in the future, whereas promising to panic does not.

[mdempsky]

The specified behavior for unsafe.Slice is that it returns (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:], and the current behavior for evaluating (*[0]T)(nil)[:] is to panic: https://play.golang.org/p/v7UWr1QMR0T

I agree that unsafe.Slice((*T)(nil), 0) returning []T(nil) is what users most likely always want. What's not obvious to me is whether that justifies intentionally diverging from existing language semantics.

If others don't share my hesitancy and there's strong support for changing it to return []T(nil) in Go 1.17, I'm happy to update cmd/compile.

That would leave open the possibility of changing the behavior in the future, whereas promising to panic does not.

I think we can change the semantics as long as it's tied to the -lang used to compile the package where the unsafe.Slice call appeared? I'm also fine with changing the spec/docs wording to instead say ptr must be non-nil to leave the behavior unspecified.

(I agree panicking is defined behavior, but my feeling is we've been somewhat lenient about that when it comes to Go 1 compat. I don't have concrete examples off hand though to point to.)

[gopherbot]

Change https://golang.org/cl/329925 mentions this issue: spec: unsafe.Add/Slice are not permitted in statement context

[twmb]

Is it worth it to add an example for how to use this in the context of converting a string to a slice? I have a feeling people will look at the API and just use

slice := unsafe.Slice((*byte)(unsafe.Pointer(&str)), len(str))

If I'm reading this correctly, the proper use is

slice := unsafe.Slice((*byte)(unsafe.Pointer((*reflect.StringHeader)(unsafe.Pointer(&str)).Data)), len(str))

which is a bit unwieldy. It may be useful to have examples on this function in general. Also, from reading the documentation, (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:] does not read with any context, so is a little confusing in general. What does that line mean to me?

[ianlancetaylor]

Converting a string to a slice is fairly unsafe, and not really what this function is for, so I'm would vote against an example of that.

The (*[len]ArbitraryType)(unsafe.Pointer(ptr))[:] just restates the text in code. What would you recommend to make it clearer?

[twmb]

If this is not for converting to a slice, it may be worth explicitly mentioning that.

I think garbage free string<->slice conversions are pretty valuable from a high-ish-performance perspective, e.g. hashing functions usually operate on slices, and often people have strings as input. In these cases, a library author can document that an input slice is not modified, and then offer corresponding non-copying string functions. In one of my own libraries, I explicitly document that a function uses unsafe and that the slice should not be modified by the user (docs here). I feel like this type of escape hatch is similar to Rust's unsafe code blocks, where a user of a library opts in up front to unsafe behavior: in Rust, users of unsafe functions often annotate their unsafe blocks with SAFETY: to describe how the user is not violating required invariants (example here).

I know I've seen many, many string<->slice implementations in Go libraries since 2012. Given that vet now checks for some safety of the conversion (#40701), and given the existence of go-safer, this conversion is clearly something that we know people do. I personally feel like it's well past time for Go to make this conversion a bit easier, because we know today that this is something people want, try to do, and make mistakes with (but I know this desire has been pushed back against all these years, so I don't really mind in the end).

That said, if this is still a pattern we don't want to support, then documenting that unsafe.Slice is not meant for this use case is valuable.

w.r.t. the code block, now that you mention it's just a restatement in code, it's really obvious. Maybe changing the preceding line from

The function Slice returns a slice whose underlying array starts at ptr and whose length and capacity are len:

to

The function Slice returns a slice whose underlying array starts at ptr and whose length and capacity are len. The function is equivalent to the following: may make that really obvious?

[ianlancetaylor]

If this is not for converting to a slice, it may be worth explicitly mentioning that.

Well, maybe, but after all we don't usually document things to say what they are not for. We document them to say what they are for. I'm not denying that there are valid uses of converting a string to []byte, but that's not really the goal of unsafe.Slice and as you say it's awkward to use for that purpose, and it's considerably less safe than the intended use of unsafe.Slice for cgo code. So it kind of seems like a distraction. We don't to start saying all the things that unsafe.Slice isn't for. That's just my own view, of course.

I sent https://golang.org/cl/332409 with your suggestion, we'll see what other people think. Thanks.

[gopherbot]

Change https://golang.org/cl/332409 mentions this issue: spec, unsafe: clarify unsafe.Slice docs

[twmb]

Thank you, that reads more clearly to me!

Good point on not mentioning what things are not for. I don't think the current docs imply what it is for (cgo?), more just that the function exists, but that's ok with me, I don't think I'm the intended user for the function at this point. :)

[gopherbot]

Change https://golang.org/cl/338949 mentions this issue: compiler, runtime: support unsafe.Add and unsafe.Slice

[gopherbot]

Change https://golang.org/cl/340549 mentions this issue: compiler: support export/import of unsafe.Add/Slice