Max operations in transaction

[mwf]

Hi, guys!

You've got a constant in server's code:

	// Max operations per txn list. For example, Txn.Success can have at most 128 operations,
	// and Txn.Failure can have at most 128 operations.
	MaxOpsPerTxn = 128

Is there any strong reason under this value? Why is it so low and why isn't it transfered from code to server's config?

[heyitsanthony]

Some light discussion here: #5641. It seems that either 128 is enough or any limit is too low. How many txn operations do you need?

It can be made into a configuration option since the check is at the API layer, but options like that would open the door for apps needing special etcd flags to work at all.

[xiang90]

@mwf

etcd proceeds txn serially. A huge txn will blocks other txn. Thus, there is a limit. 128 might be too small, but unlimited is dangerous.

[mwf]

etcd proceeds txn serially. A huge txn will blocks other txn. Thus, there is a limit. 128 might be too small, but unlimited is dangerous.

But if a client want to set 1000 kvs he will make 8 transactions at the same time and commit them almost simultaniously. What is the defference for other clients if he makes a single transaction? I think there should be no difference, if they are queued somehow. So the first client adds 8 txns to queue and other clients will wait anyway!
And a single transaction should be a little bit faster then 8 consequent ones.

Setting any hard limit can't cover all usecases. In my case 1024 seems to be enough, but I can't see any reason for it :( Besides that I haven't heard if any SQL-based DB supporting transactions made any limits on transaction length.

Maybe the best option is to make some benchmarks. For example 10 clients want to set N kvs each. And we should set MaxOpsPerTxn to MaxInt64 and modify the transactions sizes on client side. And find some optimum value by measuring and minimizing the total time for all client to complete there jobs.

So then you could move it to config with some recomended optimum value.

[heyitsanthony]

Maybe the best option is to make some benchmarks.

This doesn't address that some applications may need more txn ops that what the benchmarks say are optimum, though.

It's tempting to remove the limit entirely by default and let it be a tunable so people can set a limit if they want to protect their clusters from huge txns...

[mwf]

Yeap, sounds good for me.

[mwf]

@heyitsanthony BTW, any chance getting this in 3.2 release?

[heyitsanthony]

@mwf too late for 3.2 since it's an enhancement. Scheduling for 3.3.

[fanminshi]

@heyitsanthony @xiang90
I am thinking about to pass in a flag to etcd that sets MaxTxnsOp and defaults at un-limited. I am also thinking about have some kinda of warning either on client or server side if txnOps > a recommended one.

[xiang90]

@fanminshi

Default should not be unlimited. Probably keep it the same or increase it a little bit.

/cc @heyitsanthony