Changing equation ordering strongly affects time required to solve

[ShantanuThakoor]

@clazarus and I have been investigating the difference in runtime of the tensorflow and nnet versions of the same network, and reached the conclusion that there is a difference in the ordering of the equations and variables.

To test this, we ran the same nnet query with the equations added to the input query in different orders, using random.shuffle to randomly permute the equations.

The resultant query runs in ~10 seconds in some cases, and in some cases does not terminate at all (tried for ~15 minutes). When the query fails to terminate, there seem to be some issues with "High degradation found".

[guykatzz]

I think it makes sense that changing the orders of things (equations, variables) affects the way we traverse the search space, and consequently the solving time. However, if you run into specific instances which, you think, should be solved quickly and are solved very slowly or not at all, please send me the actual benchmark and I can have a look.

[ShantanuThakoor]

We can close this, I think it is fixed due to #24 . (Right @clazarus , @ibeling ?)

[clazarus]

After the meeting on Monday we realized that @ibeling 's latest commit ( cc096ad ) fixed a sneaky bug that was in part responsible for the huge variance of runtime for the same query with different permutations of the equations. The bug caused an issue when the system was restoring a state but the cost function state was not restored. After incorporating this fix and doing some experiments we have observed:

• the variance of runtime has reduced a lot
• queries seem to be running faster in average
• way fewer permutations take a disproportionately longer to run, we believe this is now only being caused by cycles in outgoing and incoming basic variables and we have just found a reproducible case which we will further investigate

I think we can discard this as an issue for now and focus on cycles as a source of long (or potentially infinite) runtime.