Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvement: Pub/sub should pass messages by reference #169

Open
emadum opened this issue Feb 4, 2018 · 3 comments
Open

Performance improvement: Pub/sub should pass messages by reference #169

emadum opened this issue Feb 4, 2018 · 3 comments

Comments

@emadum
Copy link
Contributor

emadum commented Feb 4, 2018

Messaging systems work best with small message payloads. Redis in particular suffers huge performance losses when pub/sub message payloads get large.

Since UMF messages already have a unique message ID, we can cache the message payload in redis when a message is sent and only send the message ID over pub/sub, minimizing the message payload.

@sjmcdowall
Copy link
Contributor

My gut tells me there is a "cutoff" in payload size where doing this makes sense but just keeping the message in the payload makes sense to avoid another trip to Redis for EvERY message. I can't imagine too many messages being passed that the receiver doesn't need the payload for it to actually do work. So, that process would need to fetch the payload from Redis (somehow?)

This is fine for LARGE payloads but I think for smaller ones (TBD but maybe 1K or less?) it maybe more efficient to just pass the payload itself. This is just a hunch from 30+ years of doing things like this. :)

Or -- perhaps I am missing the whole point. LOL

@emadum
Copy link
Contributor Author

emadum commented Feb 16, 2018

Yes you're 100% right. We did a little benchmarking of this PR and that quickly became obvious.

I've been busy lately, but plan on doing some more benchmarking using 1k, 5k and 10k message size cutoffs.

@sjmcdowall
Copy link
Contributor

sjmcdowall commented Feb 16, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants