enable gset support

[russelldb]

Adding GSet support to 2.2. It was merged into basho/riak_kv develop, but not a 2.2 branch afaict.

Tests at basho/riak_test#1302
Needs

basho/riak-erlang-client#373
basho/riak-erlang-http-client#70
basho/riak_pb#229
basho/riak_test#1302

[ghost]

This is failing quickcheck on my machine complaining about missing schema files:
rebase-nhs-riak.rdb.bet365.gsets.to.develop-2.2.txt

  %% Unknown error: {abort,
 {setup_failed,
  {error,
   {badmatch,
    {error,
     {bad_return,
      {{riak_core_app,start,[normal,[]]},
       {'EXIT',
        {{badmatch,{error,schema_files_not_found}},
         [{riak_core_cli_registry,load_schema,0,
           [{file,"src/riak_core_cli_registry.erl"},{line,51}]},
          {riak_core_app,start,2,
           [{file,"src/riak_core_app.erl"},{line,104}]},
          {application_master,start_it_old,4,
           [{file,"application_master.erl"},{line,269}]}]}}}}}},
   [{riak_kv_test_util,do_dep_apps_fun,2,
     [{file,"src/riak_kv_test_util.erl"},{line,308}]},
    {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
    {riak_kv_test_util,setup,2,
     [{file,"src/riak_kv_test_util.erl"},{line,186}]}]}}}
undefined                                                                                                                                                                                                                                                                                                            

On a previous run I got timeouts :

rebase-nhs-riak.rdb.bet365.gsets.to.develop-2.2.timedout.txt

And before that fail due to emfile - if this ever gets ported to rebar3 it might be worth adding a ulimit check (I try not to set it to 20000 by default). Not suggesting it's something we change now, just making a note for the future :

merge-nhs-riak.rdb.bet365.gsets.to.develop-2.2.txt

[russelldb]

Thanks for the review @bryanhuntesl, are these problems before or after merge to 2.2? They look like the test failures that were fixed in the 2.2.5 work that's already in 2.2. This is old, old code. Do you need me to rebase it?

[russelldb]

We really need to bring back thumbs bot

[ghost]

If you could try rebasing - would be much appreciated. I'm giving it a fresh run right now.

[ghost]

Ok , so on a clean checkout of develop-2.2 I get a bunch of timeouts (6 cancelled tests) and these warnings from above (see attached file).

wednesday.develop-2.2-attempt.txt

I'll try the rebase + test and separately the merge and test again now.

[russelldb]

Hmm, all those problems in the tests are fixed afaict in develop-2.2. It's a shame we keep doing this.

[russelldb]

WRT the timeouts…are you running in some kind of constrained environment? It's not something I've seen running these tests 100s (maybe 1000s) of times since I merged the test fix branches into 2.2

[ghost]

Nope, this is on a fairly new Mac Pro, latest updates.

…

On Thu, 22 Mar 2018, 10:26 Russell Brown, ***@***.***> wrote: WRT the timeouts…are you running in some kind of constrained environment? It's not something I've seen running these tests 100s (maybe 1000s) of times since I merged the test fix branches into 2.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1665 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Aego9rHkyvrj1vepi_YOKvF-CmJzD-Pgks5tg3xPgaJpZM4SYTxN> .

[russelldb]

the thing is, I don't know how to approach this. Do those same tests behave the same way without this change? If so…unrelated, right? We know we're starting from an imperfect point, so to question that same imperfection at every PR…we know that there are issues with the underlying code. But we can't fix all tech debt before making any forward progress.

Do you think that the failures are related to the PR?

I'm still waiting for them to finish running…the nhs-riak fork passes on 2.2 every time and has this change merged. When they finish I'll run basho-2.2, then this branch (I'd expect this branch to fail thanks to it's original 2.2 base being broken) and then the merge of the two. If it works for me, and continues to not work for you, we're kind of at an impasse, eh?

[russelldb]

Top 10 slowest tests (893.069 seconds, 82.7% of total time):
  riak_kv_memory_backend:eqc_test_/0:851
    120.157 seconds
  riak_kv_eleveldb_backend:eqc_test_/0:1039
    120.081 seconds
  riak_kv_multi_backend:eqc_test_/0:730
    120.050 seconds
  riak_kv_multi_backend:eqc_test_/0:728
    120.031 seconds
  riak_kv_bitcask_backend:eqc_test_/0:965
    120.009 seconds
  riak_kv_hll:eqc_value_test_/0:313
    91.051 seconds
  put_fsm_eqc:eqc_test_/0:85
    60.105 seconds
  riak_object_dvv_statem:eqc_test_/0:72
    60.061 seconds
  get_fsm_eqc:eqc_test_/0:50
    60.001 seconds
  ec_eqc:eqc_test_/0:30
    21.523 seconds

Finished in 1080.491 seconds
330 tests, 0 failures

[russelldb]

Hmm, so there is defo a commit missing from basho-2.2 that is in nhs-2.2…I have to go hunting

[ghost]

excellent ! I'd just started another run of just develop-2.2 in case it was intermittent.

[russelldb]

This seems to only be an issue when you run make test and the deps are pulled into riak_kv/deps. There was a change (OpenRiak@6bcfb1f) not sure if that is all it is

[russelldb]

well let you know

OOPS,WRONG BUTTON

[russelldb]

Soooo…it looks like OpenRiak@6bcfb1f was missing from basho-2.2, so I added it (#1666)

I ran tests with that merged to 2.2, and 2.2 passed for me.
I merged this PR (#1665) to 2.2 and ran, and tests pass for me.

Hopefully that waterfall of joy happens for others too

[ghost]

Tests pass - with that #1666 merged. Beautiful. +1 ship it.

[russelldb]

awesome, thanks for you patience/perseverance!