rb_class_inherited_p returns Ruby's true/false/nil

[shyouhei]

From: https://github.com/ruby/ruby/blob/6b86549df8f6d48eeab3c7b48b3fd9ee02f744ba/object.c#L1828-L1846

Returns nil if there's no relationship between the two.

[eregon]

Thanks!

[eregon]

@shyouhei I'm curious, which assertion found this bug? Could you point me to it?

[shyouhei]

@eregon The spec was it "sets up the allocator for a subclass of Array" in object_spec.rb

spec/optional/capi/object_spec.rb

Lines 915 to 922 in ddd89c8

	it "sets up the allocator for a subclass of Array" do
	klass = Class.new(Array)
	@o.rb_define_alloc_func(klass)
	obj = klass.allocate
	obj.class.should.equal?(klass)
	obj.should have_instance_variable(:@from_custom_allocator)
	obj.should == []
	end

The then-missing RTEST now triggers at the @o.rb_define_alloc_func(klass) line.

spec/optional/capi/ext/object_spec.c

Lines 348 to 361 in 41bf282

	static VALUE speced_allocator(VALUE klass) {
	VALUE flags = 0;
	VALUE instance;
	if (rb_class_inherited_p(klass, rb_cString)) {
	flags = T_STRING;
	} else if (rb_class_inherited_p(klass, rb_cArray)) {
	flags = T_ARRAY;
	} else {
	flags = T_OBJECT;
	}
	instance = rb_newobj_of(klass, flags);
	rb_iv_set(instance, "@from_custom_allocator", Qtrue);
	return instance;
	}

Here, the first rb_class_inherited_p(klass, rb_cString) never fails (because it returns Qnil). So in spite of the class being a subclass of Array, the created object ended up being T_STRING.

The inconsistency was found by my assertion at rb_ary_array_len, triggered by obj.should == [] line.

CApiObject allocator accessors rb_define_alloc_func
- sets up the allocator for a subclass of Array
Breakpoint 1, unexpected_type (x=93825003028200, xt=5, t=7) at error.c:829
829         const char *tname = rb_builtin_type_name(t);
(gdb) bt
#0  unexpected_type (x=93825003028200, xt=5, t=7) at error.c:829
#1  0x00005555557e1efb in rb_check_type (x=x@entry=93825003028200, t=5, t@entry=7) at error.c:863
#2  0x0000555555741002 in Check_Type (v=v@entry=93825003028200, t=RUBY_T_ARRAY) at include/ruby/3/value_type.h:351
#3  rb_array_len (a=a@entry=93825003028200) at include/ruby/3/core/rarray.h:133
#4  0x000055555574d1dd in rb_ary_equal (ary1=93825003028200, ary2=93825003026160) at array.c:4293
#5  0x0000555555727f9e in vm_call_cfunc_with_frame (ec=0x555555b05650, reg_cfp=0x7ffff7fcb868, calling=<optimized out>, cd=<optimized out>) at vm_insnhelper.c:2538
#6  vm_call_cfunc (ec=ec@entry=0x555555b05650, reg_cfp=reg_cfp@entry=0x7ffff7fcb868, calling=<optimized out>, calling@entry=0x7fffffff9cb0, cd=<optimized out>) at vm_insnhelper.c:2558
(... snip ...)

[eregon]

Thanks for the details, that makes perfect sense!