Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: indefinite function runs inside mutex lock #372

Merged
merged 5 commits into from
Mar 15, 2024

Conversation

Sidddddarth
Copy link
Member

@Sidddddarth Sidddddarth commented Mar 10, 2024

Description

Pulled the indefinite function out of mutex locking.

Two locks:
statsdState.clientsLock
statsdClient.statsdMu

The first one is to lock access to *statsdStats.state.pendingClients, *statsdStats.state.clients and *statsdStats.state.client.

The second is to lock access to *statsdClient.*(statsd.Client).

We attempt to set *statsdStats.state.client during startup, and if that fails, a new goroutine(1) is spawned, which requires us to protect access. All other goroutines(n) only clone(read) from this client to create new clients for stats. The clients created in other goroutines(n) have a nil client if setup hasn't succeeded(which is set appropriately in 1 - which requires write access). The stat triggers then need read access for these cloned clients.

Linear Ticket

Fixes PIPE-885

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@github-actions github-actions bot added the stats label Mar 10, 2024
Copy link

codecov bot commented Mar 10, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 19 lines in your changes are missing coverage. Please review.

Project coverage is 79.41%. Comparing base (4c6d4b6) to head (53c85ec).

Files Patch % Lines
stats/statsd.go 37.50% 15 Missing ⚠️
stats/statsd_measurement.go 71.42% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #372      +/-   ##
==========================================
+ Coverage   78.81%   79.41%   +0.59%     
==========================================
  Files          82       91       +9     
  Lines        6461     6819     +358     
==========================================
+ Hits         5092     5415     +323     
- Misses       1115     1131      +16     
- Partials      254      273      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@fracasula fracasula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain the scope of these two mutexes and how are they used to protect resources?

  • statsdState.clientsLock
  • statsdClient.statsdMu

@Sidddddarth
Copy link
Member Author

Can you please explain the scope of these two mutexes and how are they used to protect resources?

  • statsdState.clientsLock
  • statsdClient.statsdMu

The first one is to lock access to *statsdStats.state.pendingClients, *statsdStats.state.clients and *statsdStats.state.client.
The second is to lock access to *statsdClient.*(statsd.Client).

We only set *statsdStats.state.client during startup, and if that fails, a new goroutine(1) is spawned, which requires us to protect access. All other goroutines(n) only clone(read) from this client to create new clients for stats. The clients created in other goroutines(n) have a nil client if setup hasn't succeeded(which is set appropriately in 1 - which requires write access). The stat triggers then need read access for these cloned clients.

Copy link
Collaborator

@fracasula fracasula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me but I left a couple comments, have a looksie 👍

stats/statsd.go Outdated
@@ -200,7 +211,13 @@ func (s *statsdStats) internalNewTaggedStat(name, statType string, tags Tags, sa
taggedClient = &statsdClient{samplingRate: samplingRate, tags: tagVals}
if s.state.connEstablished {
taggedClient.statsdMu.Lock()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to lock here given that we just created the client and it's not a key on any of the maps, right?

stats/statsd.go Outdated
@@ -297,9 +314,8 @@ type statsdClient struct {
}

// ready returns true if the statsd client is ready to be used (not nil).
//
// statsdMu.RLock should be held when calling this method.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same comment might be useful on skip().

@@ -297,9 +312,8 @@ type statsdClient struct {
}

// ready returns true if the statsd client is ready to be used (not nil).
//
// sc.statsdMu.RLock should be held when calling this method.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't understood, why are we removing lock from here and asking caller to hold lock? Isn't that same?

Copy link
Contributor

@mihir20 mihir20 Mar 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline, along with calling this function in skip() fn call, we are recording stats as well here. we would need to hold lock again before recording new stats. Holding multiple locks multiple time can slow application down so we removed lock from here.

Copy link
Contributor

@mihir20 mihir20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Sidddddarth Sidddddarth merged commit 0c2c23b into main Mar 15, 2024
12 checks passed
@Sidddddarth Sidddddarth deleted the fix.LockAfterIndefiniteReturn branch March 15, 2024 06:30
@Sidddddarth Sidddddarth restored the fix.LockAfterIndefiniteReturn branch March 15, 2024 06:30
@Sidddddarth Sidddddarth deleted the fix.LockAfterIndefiniteReturn branch March 15, 2024 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants