perf: fix for cluster aggregation performance #631
Conversation
| // adding request id in file path to handle concurrency | ||
| const filename = path.join( | ||
| os.tmpdir(), | ||
| `metrics-${process.pid}-${message.requestId}.json`, | ||
| ); | ||
| fs.writeFile(filename, JSON.stringify(metrics), err => { |
There was a problem hiding this comment.
I'm really hesitant to involve files, since that suddenly involves the page cache, file system, more syscalls and possibly other surprises like filesystem permissions, running out of disk space and cleaning up. How much of an improvement did this yield? Did you benchmark before or after Node.js v18.6.0, which has this optimization? How much disk I/O load did your system have when benchmarking? I think IPC is "supposed" to be faster than involving the filesystem.
There was a problem hiding this comment.
The benchmarks that I have performed are on our own application. The screenshots of the results are shared in the
this Issue Link. The benchmarks were done on a 16 core machine and approximately 250-300 metrics (total size ~500 Kbs). At P99 and above levels we are getting ~10x improvement in app performance. The only disk I/Os we have is logging but that is also not happening on /tmp mounted drive on which we are writing our metrics.
The major choking point was with IPC and creating the map and hashing the object. The detailed bifurcation that I have done on my local machine which is 8 core machine is given In the screenshot below. and the code used is in below given zip file. The zip file contains the node modules as well because I have added some logs in prom-client to get the time data of each parts. Here in screenshot if we see the worst IPC time is always higher. I had run multiple iterations and same/similar results are found. The total aggregation time which is 72ms in this screenshot out of which 68ms is taken for hashing and building the map. I haven't tested with node v18.6.0 yet. will test the same code with that as well and share the results in the same thread
cluster_test.zip
There was a problem hiding this comment.
In these benchmarks (issue link) the scraping interval used was every 5 seconds and the throughput on the app was about 1100-1200 RPS.
There was a problem hiding this comment.
Hi @zbjornson
- I also tried the code in the zip file with node v18.6.0. I am getting the similar results without any improvements in the logged timings.
- I think we shouldn't use IPC for this communication because that can create a problem whenever the sizes of metrics go beyond a certain point and that will start blocking the requests routing at high throughputs.
- We can ask user to provide permissions for /tmp folder. From prom-client performance POV i didn't see much of a difference with files but again I am only hitting it once every 5 seconds.
- If not files then we should think of other ways of communication to solve this completely like workers calling over tcp on master port
There was a problem hiding this comment.
Hi @zbjornson @SimenB we went live in production with these changes about 3 months ago and everything is working fine for us. We are not seeing any additional latencies post this change. We serve more than 100,000 RPS at peak for this service. If required I can share the production results as well.
I understand that introducing file system can pose different challenges for the library, But we can explore communication with HTTP or some other mode of communication (Not IPC) when we have larger size of the metrics as the current solution will cause very high tail latencies (>P99) at high throughput and larger metrics sizes.
Requesting your inputs on this, In terms of how we can take this forward as we don't want to diverge from the library maintaining our own custom version.
|
Any concern in filesystem anymore? Should we consider to release it? |
@mrnonz we haven't faced any issues till now related to filesystem. This change would require the users to provide the access to /tmp path for prom-client |
| const promString = registry.metrics(); | ||
| request.done(null, promString); | ||
| } | ||
| fs.readFile(message.filename, 'utf8', (err, data) => { |
There was a problem hiding this comment.
About IPC
Can we have it as a customizable option and keep the default behavior instead ?
So users could pick if they want file vs ipc communication and there would be no breaking change.
Instead of sending filename to worker, could the worker keep a permanent file (named with its pid) and use IPC only to notify the master to read the file (you could use the 1st line of the file to reconcile requestId) or use fs.watchFile
Instead of file, could using unix socket or named pipes (if available) give better perf ?
About hashing
The master process should be used only to initiate and restart workers and aggregate metrics. So it should not need that much CPU compared to workers that handle the workload. And this will put a little more load to all the workers at constant interval.
I guess you could have bad performance when you have a lot of workers and short scrape interval then.
Thus same for this feature, could it be an option to configure prom-client with hashing either in master or distributed on workers, and by default keep the current behavior to introduce no breaking changes.
The changes include performance improvements for cluster mode aggregations flow. The changes are 3 fold -