Ruby 3.1 Support in TruffleRuby #2733

fniephaus · 2022-09-22T13:18:09Z

We are working on Ruby 3.1 support for the next release.

Any help is appreciated whether it is:

implementing one of the Ruby 3.1 changes in TruffleRuby, and ensuring it is tested in specs or MRI tests and otherwise add specs
adding specs for Ruby 3.1 changes, which is best done in ruby/spec directly, see Write specs for new Ruby 3.1 features and changes ruby/spec#923

To make it easier to find good getting-started issues, I annotated items with easy, medium and hard.
There is also pure ruby for changes which should only affect Ruby code and not require any change to Java sources or any Java knowledge.
If there is no pure ruby then some Java code is involved, but it should still be fairly straightforward if easy or medium.
hard typically requires more knowledge about TruffleRuby implementation details.

If you'd like to start on some task please mention it here or edit the description directly for committers (to avoid duplicated work).

Notes

Fiber scheduler changes are not implemented because it seems not worth it until Truffle supports VirtualThread on both Native Image and HotSpot.
Ractor is not implemented because it seems of little value. Thread already run in parallel and are far more compatible.

Full list of changes for Ruby 3.1.2

NOTE: https://rubyreferences.github.io/rubychanges/3.1.html gives more details for many features and changes.

From https://github.com/ruby/ruby/blob/master/doc/NEWS/NEWS-3.1.0.md:

This document is a list of user-visible feature changes
since the 3.0.0 release, except for bug fixes.

Note that each entry is kept to a minimum, see links for details.

Language changes

The block argument can now be anonymous if the block will
only be passed to another method. [Feature #11256]
```
def foo(&)
  bar(&)
end
```

[hard, need pattern matching first] Pin operator now takes an expression. [Feature #17411]

Prime.each_cons(2).lazy.find_all{_1 in [n, ^(n + 2)]}.take(3).to_a
#=> [[3, 5], [5, 7], [11, 13]]

[hard, need pattern matching first] Pin operator now supports instance, class, and global variables.
[Feature #17724]
```
@n = 5
Prime.each_cons(2).lazy.find{_1 in [n, ^@n]}
#=> [3, 5]
```
[hard, need pattern matching first] One-line pattern matching is no longer experimental.
[hard, need pattern matching first] Parentheses can be omitted in one-line pattern matching.
[Feature #16182]
```
[0, 1] => _, x
{y: 2} => y:
x #=> 1
y #=> 2
```
@andrykonchin [hard, java] Multiple assignment evaluation order has been made consistent with
single assignment evaluation order. With single assignment, Ruby
uses a left-to-right evaluation order. With this code:
```
foo[0] = bar
```
The following evaluation order is used:
1. foo
2. bar
3. []= called on the result of foo
In Ruby before 3.1.0, multiple assignment did not follow this
evaluation order. With this code:
```
foo[0], bar.baz = a, b
```
Versions of Ruby before 3.1.0 would evaluate in the following
order
1. a
2. b
3. foo
4. []= called on the result of foo
5. bar
6. baz= called on the result of bar
Starting in Ruby 3.1.0, the evaluation order is now consistent with
single assignment, with the left-hand side being evaluated before
the right-hand side:
1. foo
2. bar
3. a
4. b
5. []= called on the result of foo
6. baz= called on the result of bar
[Bug #4443]
Values in Hash literals and keyword arguments can be omitted.
[Feature #14579]

For example,
- {x:, y:} is a syntax sugar of {x: x, y: y}.
- foo(x:, y:) is a syntax sugar of foo(x: x, y: y).
Constant names, local variable names, and method names are allowed as
key names. Note that a reserved word is considered as a local
variable or method name even if it's a pseudo variable name such as
self.
- [hard, need Ractor first] Non main-Ractors can get instance variables (ivars) of classes/modules
  if ivars refer to shareable objects.
  [Feature #17592]
Support endless method definitions #3038 [hard, parser] A command syntax is allowed in endless method definitions, i.e.,
you can now write def foo = puts "Hello".
Note that private def foo = puts "Hello" does not parse.
[Feature #17398], depends on [Feature #16746]

Command line options

--disable-gems is now explicitly declared as "just for debugging".
Never use it in any real-world codebase.
[Feature #17684]

Core classes updates

Note: We're only listing outstanding class updates.

Array

(@andrykonchin) Array#intersect? is added. [Feature #15198]

Class

(@andrykonchin) Class#subclasses, which returns an array of classes
directly inheriting from the receiver, not
including singleton classes. [Feature #18273]

class A; end
class B < A; end
class C < B; end
class D < A; end
A.subclasses    #=> [D, B]
B.subclasses    #=> [C]
C.subclasses    #=> []

Enumerable

Enumerable#compact is added. [Feature #17312]
Enumerable#tally now accepts an optional hash to count. [Feature #17744]

@horakivo [easy, pure ruby] Enumerable#each_cons and each_slice to return a receiver. [GH-1509]

[1, 2, 3].each_cons(2){}
# 3.0 => nil
# 3.1 => [1, 2, 3]

[1, 2, 3].each_slice(2){}
# 3.0 => nil
# 3.1 => [1, 2, 3]

Enumerator::Lazy

Enumerator::Lazy#compact is added. [Feature #17312]

File

[easy, pure ruby] File.dirname now accepts an optional argument for the level to
strip path components. [Feature #12194]

GC

"GC.measure_total_time = true" enables the measurement of GC.
Measurement can introduce overhead. It is enabled by default.
GC.measure_total_time returns the current setting.
GC.stat[:time] or GC.stat(:time) returns measured time
in milli-seconds. [[Feature #10917]]
GC.total_time returns measured time in nano-seconds. [[Feature #10917]]

Integer

added Integer.try_convert, Ruby 3.1 support #2905 [easy, pure ruby] Integer.try_convert is added. [Feature #15211]

Kernel

(@andrykonchin) Kernel#load now accepts a module as the second argument,
and will load the file using the given module as the
top-level module. [Feature #6210]

Marshal

[medium, pure ruby] Marshal.load now accepts a freeze: true option.
All returned objects are frozen except for Class and
Module instances. Strings are deduplicated. [Feature #18148]

MatchData

@horakivo [easy, pure ruby] MatchData#match is added [Feature #18172]
@horakivo [easy, pure ruby] MatchData#match_length is added [Feature #18172]

Method / UnboundMethod

Method#public?, Method#private?, Method#protected?,
UnboundMethod#public?, UnboundMethod#private?,
UnboundMethod#protected? have been added. [Feature #11689]

Module

[hard, java, to do after fixing super] Module#prepend now modifies the ancestor chain if the receiver
already includes the argument. Module#prepend still does not
modify the ancestor chain if the receiver has already prepended
the argument. [Bug #17423]
@horakivo [medium, use SingleValueCastNode] Module#private, #public, #protected, and #module_function will
now return their arguments. If a single argument is given, it
is returned. If no arguments are given, nil is returned. If
multiple arguments are given, they are returned as an array.
[Feature #12495]

Process

@horakivo [easy, pure ruby, should be an "undefined" method] Process._fork is added. This is a core method for fork(2).
Do not call this method directly; it is called by existing
fork methods: Kernel.#fork, Process.fork, and IO.popen("-").
Application monitoring libraries can overwrite this method to
hook fork events. [Feature #17795]

Struct

(won't do, would makes things slower, more complex, and the code be removed in 3.2) @eregon [easy, pure ruby] Passing only keyword arguments to Struct#initialize is warned.
You need to use a Hash literal to set a Hash to a first member.
[Feature #16806]
(@moste00) [easy, pure ruby] StructClass#keyword_init? is added [Feature #18008]

String

@horakivo [easy, pure ruby] Update Unicode version to 13.0.0 [Feature #17750]
and Emoji version to 13.0 [Feature #18029]
(@andrykonchin) String#unpack and String#unpack1 now accept an offset: keyword
argument to start the unpacking after an arbitrary number of bytes
have been skipped. If offset is outside of the string bounds
ArgumentError is raised. [Feature #18254]

Thread

@horakivo [easy, java] Thread#native_thread_id is added. [Feature #17853]

Thread::Backtrace

[easy, pure ruby] Thread::Backtrace.limit, which returns the value to limit backtrace
length set by --backtrace-limit command line option, is added.
[Feature #17479]

Thread::Queue

Thread::Queue.new now accepts an Enumerable of initial values.
[Feature #17327]

Time

[easy, pure ruby] Time.new now accepts optional in: keyword argument for the
timezone, as well as Time.at and Time.now, so that is now
you can omit minor arguments to Time.new. [Feature #17485]
```
Time.new(2021, 12, 25, in: "+07:00")
#=> 2021-12-25 00:00:00 +0700
```
At the same time, time component strings are converted to
integers more strictly now.
```
Time.new(2021, 12, 25, "+07:30")
#=> invalid value for Integer(): "+07:30" (ArgumentError)
```
Ruby 3.0 or earlier returned probably unexpected result
2021-12-25 07:00:00, not 2021-12-25 07:30:00 nor
2021-12-25 00:00:00 +07:30.
(@andrykonchin) Time#strftime supports RFC 3339 UTC for unknown offset local
time, -0000, as %-z. [Feature #17544]

TracePoint

[medium, java] TracePoint.allow_reentry is added to allow reenter while TracePoint
callback.
[Feature #15912]

$LOAD_PATH

[medium, pure ruby, check if not already the case] $LOAD_PATH.resolve_feature_path does not raise. [Feature #16043]

Fiber Scheduler

- [hard, need Fiber scheduler core first] Add support for Addrinfo.getaddrinfo using address_resolve hook.
  [Feature #17370]
- [hard, need Fiber scheduler core first] Introduce non-blocking Timeout.timeout using timeout_after hook.
  [Feature #17470]
- [hard, need Fiber scheduler core first] Introduce new scheduler hooks io_read and io_write along with a
  low level IO::Buffer for zero-copy read/write. [Feature #18020]
- [hard, need Fiber scheduler core first] IO hooks io_wait, io_read, io_write, receive the original IO object
  where possible. [Bug #18003]
Make Monitor fiber-safe. [Bug #17827]
Replace copy coroutine with pthread implementation. [Feature #18015]

Refinement

New class which represents a module created by Module#refine.
@horakivo [medium, pure ruby] include and prepend are deprecated, and import_methods is added
instead. [Bug #17429]

Stdlib updates

The following default gem are updated.
- RubyGems 3.3.3
- base64 0.1.1
- benchmark 0.2.0
- bigdecimal 3.1.1
- bundler 2.3.3
- cgi 0.3.1
- csv 3.2.2
- date 3.2.2
- did_you_mean 1.6.1
- digest 3.1.0
- drb 2.1.0
- erb 2.2.3
- error_highlight 0.3.0
- etc 1.3.0
- fcntl 1.0.1
- fiddle 1.1.0
- fileutils 1.6.0
- find 0.1.1
- io-console 0.5.10
- io-wait 0.2.1
- ipaddr 1.2.3
- irb 1.4.1
- json 2.6.1
- logger 1.5.0
- net-http 0.2.0
- net-protocol 0.1.2
- nkf 0.1.1
- open-uri 0.2.0
- openssl 3.0.0
- optparse 0.2.0
- ostruct 0.5.2
- pathname 0.2.0
- pp 0.3.0
- prettyprint 0.1.1
- psych 4.0.3
- racc 1.6.0
- rdoc 6.4.0
- readline 0.0.3
- readline-ext 0.1.4
- reline 0.3.0
- resolv 0.2.1
- rinda 0.1.1
- ruby2_keywords 0.0.5
- securerandom 0.1.1
- set 1.0.2
- stringio 3.0.1
- strscan 3.0.1
- tempfile 0.1.2
- time 0.2.0
- timeout 0.2.0
- tmpdir 0.1.2
- un 0.2.0
- uri 0.11.0
- yaml 0.2.0
- zlib 2.1.1
The following bundled gems are updated.
- minitest 5.15.0
- power_assert 2.0.1
- rake 13.0.6
- test-unit 3.5.3
- rexml 3.2.5
- rbs 2.0.0
- typeprof 0.21.1
The following default gems are now bundled gems.
- net-ftp 0.1.3
- net-imap 0.2.2
- net-pop 0.1.1
- net-smtp 0.3.1
- matrix 0.4.2
- prime 0.1.2
- debug 1.4.0
The following gems has been removed from the Ruby standard library.
- dbm
- gdbm
- tracer
[hard, java] Coverage measurement now supports suspension. You can use Coverage.suspend
to stop the measurement temporarily, and Coverage.resume to restart it.
See [Feature #18176] in detail.
Random::Formatter is moved to random/formatter.rb, so that you can
use Random#hex, Random#base64, and so on without SecureRandom.
[Feature #18190]

Compatibility issues

Note: Excluding feature bug fixes.

[note: they are not actually deprecated in headers] rb_io_wait_readable, rb_io_wait_writable and rb_wait_for_single_fd are
deprecated in favour of rb_io_maybe_wait_readable,
rb_io_maybe_wait_writable and rb_io_maybe_wait respectively.
rb_thread_wait_fd and rb_thread_fd_writable are deprecated. [Bug #18003]
[easy, C] rb_io_maybe_wait_readable, rb_io_maybe_wait_writable and rb_io_maybe_wait are added.

Stdlib compatibility issues

ERB#initialize warns safe_level and later arguments even without -w.
[Feature #14256]
lib/debug.rb is replaced with debug.gem
Kernel#pp in lib/pp.rb uses the width of IO#winsize by default.
This means that the output width is automatically changed depending on
your terminal size. [Feature #12913]
Psych 4.0 changes Psych.load as safe_load by the default.
You may need to use Psych 3.3.2 for migrating to this behavior.
[Bug #17866]

C API updates

Documented. [GH-4815]
(@moste00) [easy, C] rb_gc_force_recycle is deprecated and has been changed to a no-op.
[Feature #18290]

Implementation improvements

Inline cache mechanism is introduced for reading class variables.
[Feature #17763]
instance_eval and instance_exec now only allocate a singleton class when
required, avoiding extra objects and improving performance. [GH-5146]
The performance of Struct accessors is improved. [GH-5131]
mandatory_only? builtin special form to improve performance on
builtin methods. [GH-5112]
Experimental feature Variable Width Allocation in the garbage collector.
This feature is turned off by default and can be enabled by compiling Ruby
with flag USE_RVARGC=1 set. [Feature #18045] [Feature #18239]

Debugger

A new debugger debug.gem is bundled.
debug.gem is a fast debugger implementation, and it provides many features
like remote debugging, colorful REPL, IDE (VSCode) integration, and more.
It replaces lib/debug.rb standard library.
rdbg command is also installed into bin/ directory to start and control
debugging execution.

error_highlight

A built-in gem called error_highlight has been introduced.
It shows fine-grained error locations in the backtrace.

Example: title = json[:article][:title]

If json is nil, it shows:

$ ruby test.rb
test.rb:2:in `<main>': undefined method `[]' for nil:NilClass (NoMethodError)

title = json[:article][:title]
            ^^^^^^^^^^

If json[:article] returns nil, it shows:

$ ruby test.rb
test.rb:2:in `<main>': undefined method `[]' for nil:NilClass (NoMethodError)

title = json[:article][:title]
                      ^^^^^^^^

This feature is enabled by default.
You can disable it by using a command-line option --disable-error_highlight.
See the repository in detail.

[hard, won't do while error_highlight depends on RubyVM, we can do this once error_highlight switches to Prism] Investigate how to make it work, currently it's not loaded by default and requiring seems to have no effect.

Miscellaneous changes

[easy, pure Ruby, need to update tool/import-mri-files.sh to copy this file] lib/objspace/trace.rb is added, which is a tool for tracing the object
allocation. Just by requiring this file, tracing is started immediately.
Just by Kernel#p, you can investigate where an object was created.
Note that just requiring this file brings a large performance overhead.
This is only for debugging purposes. Do not use this in production.
[Feature #17762]
[easy, pure Ruby, need to update tool/import-mri-files.sh to copy this file] Reuse ext/objspace/lib/objspace.rb from CRuby
@eregon [easy, java] Now exceptions raised in finalizers will be printed to STDERR, unless
$VERBOSE is nil. [Feature #17798]
ruby -run -e httpd displays URLs to access. [Feature #17847]
Add ruby -run -e colorize to colorize Ruby code using
IRB::Color.colorize_code.
@eregon fiber.so is now required by default, so Fiber.current, transfer available by default.

The text was updated successfully, but these errors were encountered:

andrykonchin · 2023-03-07T12:38:59Z

I've taken "Time.new now accepts optional in: keyword argument".

andrykonchin · 2023-03-08T10:39:47Z

I've taken "Enumerable#tally now accepts an optional hash to count"

andrykonchin · 2023-03-08T12:51:22Z

I've taken "$LOAD_PATH.resolve_feature_path does not raise"

andrykonchin · 2023-03-08T13:57:31Z

I've taken "Thread::Backtrace.limit"

eregon · 2023-03-17T16:55:15Z

Most of it is now done and will be in the 23.0 release. I'll close this issue and we can address the remaining items when it becomes possible or important for compatibility.

fniephaus added this to the 22.3.0 Release (October 25, 2022) milestone Sep 22, 2022

fniephaus assigned eregon Sep 22, 2022

fniephaus added this to GraalVM Community Roadmap Sep 22, 2022

fniephaus moved this to In Progress in GraalVM Community Roadmap Sep 22, 2022

fniephaus modified the milestones: 22.3.0 Release (October 25, 2022), 23.0.0 Release (January 24, 2023) Sep 22, 2022

eregon assigned eregon and andrykonchin and unassigned eregon Oct 26, 2022

eregon added the compatibility label Oct 26, 2022

eregon changed the title ~~Support for Ruby 3.1 in TruffleRuby~~ Ruby 3.1 Support in TruffleRuby Nov 15, 2022

eregon pinned this issue Nov 15, 2022

eregon added the good first issue label Jan 10, 2023

eregon mentioned this issue Jan 18, 2023

Implement Array#intersect? #2831

Closed

graalvmbot mentioned this issue Jan 18, 2023

[GR-40333] Ruby 3.1 Array#intersect? #2834

Merged

eregon mentioned this issue Jan 18, 2023

Implement Array#intersect?. #2832

Merged

eregon closed this as completed Mar 17, 2023

github-project-automation bot moved this from In Progress to Done in GraalVM Community Roadmap Mar 17, 2023

eregon mentioned this issue May 5, 2023

Support endless method definitions #3038

Closed

eregon unpinned this issue May 5, 2023

fniephaus added this to TruffleRuby Roadmap Oct 1, 2024

fniephaus removed this from GraalVM Community Roadmap Oct 1, 2024

fniephaus moved this to Done in TruffleRuby Roadmap Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ruby 3.1 Support in TruffleRuby #2733

Ruby 3.1 Support in TruffleRuby #2733

fniephaus commented Sep 22, 2022 •

edited by eregon

Loading

andrykonchin commented Mar 7, 2023

andrykonchin commented Mar 8, 2023

andrykonchin commented Mar 8, 2023

andrykonchin commented Mar 8, 2023

eregon commented Mar 17, 2023

Ruby 3.1 Support in TruffleRuby #2733

Ruby 3.1 Support in TruffleRuby #2733

Comments

fniephaus commented Sep 22, 2022 • edited by eregon Loading

Notes

Full list of changes for Ruby 3.1.2

Language changes

Command line options

Core classes updates

Stdlib updates

Compatibility issues

Stdlib compatibility issues

C API updates

Implementation improvements

Debugger

error_highlight

Miscellaneous changes

andrykonchin commented Mar 7, 2023

andrykonchin commented Mar 8, 2023

andrykonchin commented Mar 8, 2023

andrykonchin commented Mar 8, 2023

eregon commented Mar 17, 2023

fniephaus commented Sep 22, 2022 •

edited by eregon

Loading