Error: toString failed

[Alchemystic]

• Version: v4.4.5
• Platform: Linux 3.19.0-61-generic Add major, minor, and patch labels to PRs, Issues #69~14.04.1-Ubuntu SMP Thu Jun 9 09:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Node server using socket.io is getting hit by some kind of attack that makes it crash, with the error "toString failed". At first I thought I'm just trying to process some really long string sent via sockets by an user, but it's not, it crashes even without doing anything with the messages coming in from the socket events. The code below is the most barebones I could get the actual server while still having it be vulnerable to whatever this is. The "socketio-wildcard" was only added to this so I could see all the events sent in, but there is none when it's actually crashing, just a new connection opening then it goes down. It's definitely some sort of attack or exploit, cloning the server to a different IP made it stop, until after the new was released to the public (site put back online). There were also some spikes in traffic but they didn't coincide with the actual crashes so I'm gonna assume they were just some failed ddos attempts not related to this issue. Anyone encountered this before?

Code:

var port=3001;

var toobusy = require('toobusy-js');
var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var middleware = require('socketio-wildcard')(); //only using this to catch all events, usually only listening for specific events

var Ddos = require('ddos'); //added this later, didn't seem to help
var ddos = new Ddos;
app.use(ddos.express);

io.use(middleware);

http.listen(port, function() {
    console.log('Socket listening on *:'+port);
});

app.use(function(req, res, next) {
    if (toobusy()) {
        console.log('Server is too busy, returning 503.');
        res.send(503, "I'm busy right now, sorry.");
    } else {
        next();
    }
});

// Add headers
app.use(function (req, res, next) {
    // Website you wish to allow to connect
    res.setHeader('Access-Control-Allow-Origin', 'http://siteurl.com');
    // Request methods you wish to allow
    res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS, PUT, PATCH, DELETE, UPGRADE');
    // Request headers you wish to allow
    res.setHeader('Access-Control-Allow-Headers', 'X-Requested-With,content-type');
    res.setHeader('Access-Control-Allow-Credentials', false);
    next();
});

io.on('connection', function(socket) {
    console.log('new connection: '+socket.id);
    console.log(socket);
    socket.on('*', function(data){
        console.log('new socket message: ');
        console.log(data);
    });
});

process.on('SIGINT', function() {
    console.log('Closing server...');
    toobusy.shutdown();
    process.exit();
});

process.on('exit', function () {
    console.log('About to exit.');
});

Error:

Error: toString failed
    at Buffer.toString (buffer.js:377:11)
    at /home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:477:39
    at Receiver.applyExtensions (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:364:5)
    at /home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:466:14
    at Receiver.flush (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:340:3)
    at Receiver.opcodes.1.finish (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:482:12)
    at Receiver.expectHandler (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:451:33)
    at Receiver.add (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/Receiver.js:95:24)
    at Socket.realHandler (/home/alex/node_modules/socket.io/node_modules/engine.io/node_modules/ws/lib/WebSocket.js:800:20)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)

Full log of the output when crashing:
http://pastebin.com/EvAxGQj6

Couldn't do anymore logging because after some time the "attack" stopped, it used to keep going even after the server crashed, but right now it seems to be more "calculated" so to speak, stopping as soon as the server crashes, making it really difficult for me to debug anything.

[mscdex]

/cc @nodejs/buffer

[rvagg]

@nodejs/security

[MylesBorins]

I'll dig into this today

[indutny]

It sounds like an allocation failure. Any chance there were core dumps on that system?

[indutny]

I think it may be mitigated by using maxPayload option for ws.

[Alchemystic]

Any chance there were core dumps on that system?

I'm not sure. How would I check for that?

I think it may be mitigated by using maxPayload option for ws.

Alright, I'm looking into that right now.

[indutny]

@Alchemystic most likely you don't have them then. Node.js has to be running with --abort_on_uncaught_exception and /etc/sys/limits.conf should be edited (or ulimit -c unlimited should be called for app's session).

[indutny]

@Alchemystic in initial message you mentioned traffic increase, I suspect that it should be around several gigabytes (2-3 or more). Does this match your data?

[Alchemystic]

I think it was more like 20 megabytes per second when our average was around 5.

It just happened again 20 minutes ago so here's the spike from when it happened (it flatlined because we didn't start it back up):
http://i.imgur.com/9VoUcNc.png it's going from 200-300K to 6M then crashes

Here is the "network packets" graph:
http://i.imgur.com/Suhaob8.png only a small spike here

No spikes in disk bytes / cpu utilization.

[indutny]

@Alchemystic this seems to be confirming my hypothesis. How many crashes did you get?

[indutny]

@Alchemystic thank you for the data, btw.

[dihmeetree]

@indutny friend of @Alchemystic here :) Can you give me some insight into the maxPayload thing you mentioned above? I'm guessing you're referring to the clientside js where we declare our socket..and there's some sort of parameters or options we can set along with it right? Mind posting what that would look like if you can?

[Alchemystic]

How many crashes did you get?

Two days ago it would crash every other minute basically (100+ crashes). Been mostly down for the last day just trying to find various fixes for it. None of them worked, and it's been really difficult to try and test any other fixes after it stopped being a non-stop thing to only happening when we go online.

[indutny]

interesting.

[Alchemystic]

thank you for the data, btw.

And thank you for the replies, I guess I've been too focused on trying to relate all the info I have and forgot to say that I'm grateful for you guys getting involved and trying to help.

[indutny]

Having core dumps would certainly help quite a lot.

[Alchemystic]

Node.js has to be running with --abort_on_uncaught_exception and /etc/sys/limits.conf should be edited (or ulimit -c unlimited should be called for app's session).

If you could explain a bit how to do what you've mentioned in your post earlier, I'd be more than happy to do it and post the results.

[Alchemystic]

Also, I'm not getting much by looking up the "maxPayload" option, or maybe I just don't understand how to set it. How would that fit in our current set up (the script in the main post)?

[indutny]

@Alchemystic I have no idea how to propagate maxPayload all the way through the socket.io. See socketio/socket.io#2326

Regarding core dumps, they are pretty easy to enable. Let's say you start application with:

node app.js

Replacing this with:

ulimit -c unlimited && node --abort_on_uncaught_exception app.js

Will make node generate core file in the working directory of app on crashes. We'll figure out how to inspect it from there 😉

[trevnorris]

What version of ws are you using for the stacktrace in the OP? Line numbers don't match up w/ latest.

[Alchemystic]

Socket.io version 1.4.6
"ws" from socket.io/node_modules/engine.io/node_modules/ws/ is 1.0.1

@indutny been running it like that for a while. Still no crash. Waiting for it though.

[indutny]

@Alchemystic I think I figured out the reason, so there is no need for the core dump anymore. Thank you for trying it out, though!

Please stay tuned as we are currently resolving this issue.

[Alchemystic]

Alright thank you. We've found a workaround to stop the crashes but I can't go into detail here until the "real" issue is resolved because it may put us at risk again. The attacker and one other person already asked for money for it saying they won't stop crashing it till we give them what they want, I wouldn't want them or others catch on to it. I will e-mail you about it.

Thanks again for all the help.

[lpinca]

Here is a minimal test case to reproduce the issue on my machine:

const crypto = require('crypto');

const buf = crypto.randomBytes(Math.pow(2, 28));
buf.toString();

[indutny]

Alright, the advisory is public now: https://nodesecurity.io/advisories/120 . The bug is in socket.io/ws, please update!