show_source crashes IRB when used on some top-level constants in Ruby 3.3

[st0012]

Description

Run show_source against these constants in Ruby 3.3 crashes IRB:

• Gem
• DidYouMean
• RUBY_DESCRIPTION
• SyntaxSuggest
• ErrorHighlight

/Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:98:in `count': invalid byte sequence in UTF-8 (ArgumentError)

          prev_line += t.tok.count("\n")
                                   ^^^^
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:98:in `block in interpolate_ripper_ignored_tokens'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:86:in `each'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:86:in `interpolate_ripper_ignored_tokens'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:137:in `block in ripper_lex_without_warning'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:47:in `compile_with_errors_suppressed'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/ruby-lex.rb:124:in `ripper_lex_without_warning'
        from /Users/hung-wulo/src/github.com/ruby/irb/lib/irb/source_finder.rb:51:in `find_end'
        ......

This is because their source location is the Ruby binary (e.g. /opt/rubies/3.3.0/bin/ruby), which is of course not meant to be read for the command.

In Ruby 3.2 their source locations were all ruby instead. And because no file is called ruby, they'd be treated as source not available.

It's worth noting that in both Ruby 3.2 and 3.3, bundle exec changes these constants' source location:

$ ruby -e 'puts "Gem source location: #{Object.const_source_location(:Gem)}"'
Gem source location: ["ruby", 0]                                                        
$bundle exec ruby -e 'puts "Gem source location: #{Object.const_source_location(:Gem)}"'
Gem source location: ["/opt/rubies/3.2.2/bin/ruby", 0]

However, in Ruby 3.2, their source location would be ruby if called inside the executable exe/irb. But I couldn't find obvious cause for this from reading PRs and code in CRuby or bundler.

Potential Solutions

Although the root cause if not in IRB, I think we should find a way to mitigate the issue regardless. But I'm not sure what's the best approach to check if a file is readable text instead of binary.

Result of irb_info

Ruby version: 3.3.0
IRB version: irb 1.11.0 (2023-12-19)
InputMethod: RelineInputMethod with Reline 0.4.2
Completion: Autocomplete, RegexpCompletor
.irbrc path: /Users/hung-wulo/src/github.com/ruby/irb/.irbrc
RUBY_PLATFORM: arm64-darwin23
LANG env: en_GB.UTF-8
LC_ALL env: en_US.UTF-8
East Asian Ambiguous Width: 1
=> nil

Terminal Emulator

What's your terminal emulator?

Setting Files

Are you using ~/.irbrc and ~/.inputrc?

[st0012]

Update: @tompng found that this issue would also happen to any constant whose source is from a .bundle file. So even without the Ruby 3.3 behaviour change, IRB should still find a way to prevent the problem anyway.