types as C-structs

[vtjnash]

This is a first pass at giving pointers to julia types easy compatibility with C. This is done by "hiding" the type and GC information before the data, and shifting all references in the julia source by 1 to match. What this would mean for the user is that

type A
  a::Int
end
type B
  a::A
end

can be passed to a C function expecting:

typedef struct _A {
  int a;
} A;
typedef struct _B {
  *A a;
} B;

If the user is receiving data from C (e.g. anything not created by Julia's GC), then they must instead declare it as Ptr{A} and access with unsafe_ref (I'm wondering if it might be more appropriate to introduce another type for this purpose, but thats a separate issue I'll go into later).

If @JeffBezanson approves of this, I'll go back through and clean this up some to make it ready to merge. TODO items include:

• remove some casts that are now no-ops
• figure out what emit_arraysize is doing (and replace with offsetof for better maintainability)
• finish fixing OVERLAP_TUPLE_LEN
• stop over-allocating types by sizeof(void*) bytes in alloc_big, and check if there's double counting anywhere else (or conversely, fully move it to alloc_big)
• document how to use & how to avoid getting burned

[StefanKarpinski]

I like it. We'll need some clear documentation on what becomes a pointer and what is just a field, but that's perfectly doable.

[stevengj]

I agree that this would be a nice feature. In PyCall, for example, I needed precisely this feature of inserting a pointer to one structure in another structure, and I needed to use some hacks since & only works in ccall. Making jl_value_t* pointers compatible with C struct pointers (by offsetting the pointer) is a clean way to eliminate this sort of difficulty.

[StefanKarpinski]

@JeffBezanson: can you please review? I think that's the hold up here.

[vtjnash]

@JeffBezanson Yes, I'm waiting for your input before I finish this. I'm on IRC too if you prefer to discuss there.

[JeffBezanson]

Yes, I certainly approve the concept. Please continue fixing and ping me again when ready.

[StefanKarpinski]

How's this going?

[vtjnash]

It's being held up by a thesis.

[StefanKarpinski]

Boo! :-P

[stevengj]

Any progress on this?

[Keno]

Depends on what you want. #3466 does some of it.

[vtjnash]

I'm waiting for after 0.2 gets tagged to merge both

[stevengj]

@loladiro, I was specifically wondering about the capability to treat a jl_value_t * as a pointer to a C struct (by hiding the Julia stuff before the pointer address).

[JeffBezanson]

Bump.

[stevengj]

Is this waiting on #5227?

[vtjnash]

Yes

[ViralBShah]

As already said by many before, this capability would be really amazing to have.

[vtjnash]

almost done getting this recreated against my jn/ccall3 branch, so it should all be ready-to-merge at the same time: 58d3033

[vtjnash]

it's amazing how much our usage of travis has changed since this PR was first submitted: https://travis-ci.org/JuliaLang/julia/builds/6230711

this is likely now ready to merge, pending any findings from CI or code review

[IainNZ]

Seems to have segfaulted building system image on 32bit linux?

[vtjnash]

yes, although, strangely, on windows it did the reverse (the i686 build succeeded and x86-64 failed). the llvm verifier locally pointed out my invalid pointer -> int64 cast though.

[JeffBezanson]

moved to #10579