Skip to content

Commit

Permalink
resize strings after parsing
Browse files Browse the repository at this point in the history
The parser uses `rb_str_buf_new` to allocate new strings.
`rb_str_buf_new` [has a minimum size of 128 and is not an embedded
string](https://github.com/ruby/ruby/blob/9949407fd90c1c5bfe332141c75db995a9b867aa/string.c#L1119-L1135).  This causes applications that parse JS to allocate extra memory when parsing short strings.

For a real-world example, we can use the mime-types gem.  The mime-types
gem stores all mime types inside a JSON file and parses them when you
require the gem.

Here is a sample program:

```ruby
require 'objspace'
require 'mime-types'

GC.start
GC.start

p ObjectSpace.memsize_of_all String
```

The example program loads the mime-types gem and outputs the total space
used by all strings.  Here are the results of the program before and
after this patch:

** Before **

```
[aaron@TC json (memuse)]$ ruby test.rb
5497494
[aaron@TC json (memuse)]$
```

** After **

```
[aaron@TC json (memuse)]$ ruby -I lib:ext test.rb
3335862
[aaron@TC json (memuse)]$
```

This change results in a ~40% reduction of memory use for strings in the
mime-types gem.

Thanks @matthewd for finding the problem, and @nobu for the patch!
  • Loading branch information
tenderlove authored and flori committed Jun 21, 2016
1 parent e16dab5 commit 797ce53
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions ext/json/ext/parser/parser.rl
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,8 @@ static char *JSON_parse_string(JSON_Parser *json, char *p, char *pe, VALUE *resu

if (json->symbolize_names && json->parsing_name) {
*result = rb_str_intern(*result);
} else {
rb_str_resize(*result, RSTRING_LEN(*result));
}
if (cs >= JSON_string_first_final) {
return p + 1;
Expand Down

0 comments on commit 797ce53

Please sign in to comment.