Don't generate a mutex for each Locked<T>, share one per object

[Al2Klimov]

This reduces RAM usage per object by sizeof(mutex)*(FIELDS-1).

The atomic_load(const std::shared_ptr<T>*) idea would, without doubt, also consume less RAM. However, it would scale worse, the more objects you have, with a fixed number of mutexes. Neither one SpinMutex per property, nor one std::mutex per object suffer from this limitation.

[julianbrost]

This misses part of what #10113 asked for:

1. Figure out how much of an effect this has on the total memory use of Icinga 2.

We have a vague "Icinga 2 uses more memory in newer versions", so how much does this affect the total memory use of the whole process? Might this be the answer or is it just a small puzzle piece?

Also, what made you choose adding that m_FieldsMutex and passing it around over what I suggested in #10113? I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

[Al2Klimov]

Also, what made you choose adding that m_FieldsMutex and passing it around over what I suggested in #10113?

sharing the mutex between objects, this would reduce the memory requirements

The more fields, especially objects, share one mutex, the slower they can proceed. I prefer 1+ mutex per separate object.

I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

You're right. One wrong move on the mutex and... 🙈

[julianbrost]

Why do you think a spinlock is a good idea here?

Let me quote cppreference.com for std::atomic_flag:

A spinlock mutex demo can be implemented in userspace using an atomic_flag. Do note that spinlock mutexes are extremely dubious in practice.

And we even had problems in Icinga 2 in the past due to a spinlock, see #8507/#8488. So I have doubts that would be a good idea this time.

[Al2Klimov]

Why do you think a spinlock is a good idea here?

std::mutex is 5 pointers large on x64, whereas std::atomic_flag at most one.

spinlock mutexes are extremely dubious in practice.

But the lock scope here is very small, isn't it? Not to mention how unlikely you'll hit the slow path at all in our case...

[julianbrost]

But the lock scope here is very small, isn't it?

Depends on what's inside. If it's an intrusive pointer, it just needs to increment the reference count, if it's a string though, it has to copy it, including allocating memory for it.

Not to mention how unlikely you'll hit the slow path at all in our case...

How did you conclude it's unlikely? #9364 was done for a reason, there were random crashes because. Crash means concurrent access, so that will go into the slow path. Also note that not every concurrent access resulted in a crash, so there will be more "hitting the slow path" than there were crashes.

[julianbrost]

Is this change hidden in a big rebase push related to this conversation? If so, please comment why you think this resolves it. Otherwise, what was the motivation for that change and how will you address this?

[Al2Klimov]

But the lock scope here is very small, isn't it?

Depends on what's inside. If it's an intrusive pointer, it just needs to increment the reference count, if it's a string though, it has to copy it, including allocating memory for it.

You're right. Especially the latter case can take a few more CPU cycles. So instead of wasting them on pure spinning in a waiting thread, the latter now says "after you" while something's locked. This shall bring the mutex holder in foreground and let it finish and unlock.

[julianbrost]

Please use a search engine of your choice to search for the keywords "userspace" and "spinlock". Seems like the majority isn't too positive about that idea. And as already said, Icinga 2 already had a negative experience with spinlocks itself. What makes you believe that now is a good time to use a spinlock in userspace?

By the way, Linus Torvalds' rant post (the one linked from cppreference.com) even has a follow-up from him on yield.

Apart from that, the current version of the PR seems to restrict the use of that suggested spinlock to just intrusive_ptr, so each string attribute would still use its own std::mutex. So the memory use will be somewhere in the middle (I haven't looked into the distribution of attribute types), but why do it that way if it can be done better?

The atomic_load(const std::shared_ptr<T>*) idea would, without doubt, also consume less RAM. However, it would scale worse, the more objects you have, with a fixed number of mutexes.

However, while there may be a large number of objects, the number of CPU cores and thereby threads we start is typically much smaller. And that actually limits the total contention. There won't be a thread for each single object trying to lock it.

[Al2Klimov]

However, while there may be a large number of objects, the number of CPU cores and thereby threads we start is typically much smaller. And that actually limits the total contention. There won't be a thread for each single object trying to lock it.

This sounds nice in theory, but I'd prefer a solution much more predicable, especially for large setups. I mean my one mutex per object idea.

[Al2Klimov]

I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

I've fixed it. :)