Skip to content

Latest commit

 

History

History
138 lines (93 loc) · 5.6 KB

setup.md

File metadata and controls

138 lines (93 loc) · 5.6 KB
Raw

Getting started

Server setup

To get a feel for the UI, you can poke around a public dist test server operated by Cloudera at http://dist-test.cloudera.org. However, test submission requires authentication.

You can run your own isolate server, dist_test server, slave, beanstalkd, and MySQL instance to try this out locally.

https://github.com/apache/incubator-kudu/blob/master/build-support/dist_test.py

The first step is to clone and build the required following additional repos we need.

Luci is split into luci-py and luci-go. Luci doesn't have releases, so for reference, here are the commit hashes we use:

luci-py:

    commit 1f8ba359e84dc7f26b1ba286dfb4e28674efbff4
    Author: Marc-Antoine Ruel <maruel@chromium.org>
    Date:   Tue Nov 4 15:55:03 2014 -0500

        Add isolate.py archive --blacklist and remove stale chromium blacklist.

        - Remove stale default blacklist that are not relevant anymore. This further
          cleans up the client code from chromium-specific code.
        - Make isolate.py and isolateserver.py use the same default blacklist. Sanity.

        R=vadimsh@chromium.org
        BUG=chromium:429358

        Review URL: https://codereview.appspot.com/167150043

luci-go:

    commit 69b5913d2932a6c71209a5a15ec6fa3af56bb73f
    Author: Todd Lipcon <todd@cloudera.com>
    Date:   Wed Jun 3 11:23:29 2015 -0700

        Allow non-HTTPS servers outside of appspot

        R=maruel@chromium.org
        BUG=

        Review URL: https://codereview.chromium.org/1155533005

Follow the luci-go README for installation instructions. This requires Go 1.4. 1.5 and higher will fail with an error about "use of internal package not allowed". Due to the lack of version pinning that is typical of Go dependency management, the details are left as an exercise for the reader :)

Once that's complete, start an isolateserver.

Next, create a configuration file at ~/.dist_test.cnf that looks like the following:

    [isolate]
    home=/home/andrew/dev/luci-py/client
    server=http://localhost:4242
    cache_dir=/tmp/isolate-cache

    [aws]
    access_key=<FILL IN HERE>
    secret_key=<FILL IN HERE>
    test_result_bucket=<FILL IN HERE>

    [mysql]
    host=localhost
    user=<FILL IN HERE>
    password=<FILL IN HERE>
    database=<FILL IN HERE>

    [beanstalk]
    host=localhost

    [dist_test]
    master=http://localhost:8081/
    allowed_ip_ranges=0.0.0.0/0

Adjust isolate.home to point within your luci-py repo. The client folder should have a run_isolated.py file that is the key functionality we're using.

Note that as part of this, you need to setup an AWS account to store test results, and also a MySQL instance running with a configured user/password and database.

Start beanstalk:

    $ beanstalkd

Start the master and one or more slaves:

    $ ./bin/server
    $ ./bin/slave

dist_test will create the required tables in MySQL on first run.

Client setup

Now, we are ready to submit our first job. dist_test supports multiple frontends, but the easiest to get started with is grind. grind is the dist_test frontend for Maven + Surefire + JUnit based projects, and includes a sample project that we can build and test.

Call setup.sh to create env/grind.cfg. This defaults to the cloudera.org isolate server and dist_test server; change these to localhost. The paths are autogenerated and will be correct for your installation:

    [grind]
    isolate_server = http://localhost:4242
    dist_test_client_path = /home/andrew/dev/dist_test/bin/client
    dist_test_master = http://localhost:8081
    grind_cache_dir = /home/andrew/dev/dist_test/env/.grind/cache
    grind_temp_dir = /home/andrew/dev/dist_test/env/.grind/temp
    isolate_path = /home/andrew/dev/dist_test/bin/isolate
    dist_test_password = 
    dist_test_user =

Now, perform the following:

    $ export PATH=`pwd`/grind/bin:$PATH
    $ export GRINDCFG=`pwd`/env/grind.cfg
    $ cd grind/python/disttest/test/test-resources/MultiModuleTestProject
    $ mvn clean package -DskipTests
    $ grind -c $GRINDCFG test

If this succeeds, you should see a message like this with progress report:

    INFO:__main__:Calling /home/andrew/dev/dist_test/bin/client submit --name MultiModuleTestProject /home/andrew/dev/dist_test/env/.grind/temp/grind.qSIM9d/run.json
    INFO:dist_test.client:Submitting job to http://localhost:8081/submit_job
    INFO:dist_test.client:Submitted job MultiModuleTestProject.andrew.1461116518.30580
    INFO:dist_test.client:Watch your results at http://localhost:8081/job?job_id=MultiModuleTestProject.andrew.1461116518.30580
    0.5s    0/5 tests complete

Click on the job status page URL to watch progress. It's normal for the test run to fail, since some of the included tests are intended to always fail.

Additional resources

More detailed information about dist_test and grind can be found in the included project documentation:

The Apache Kudu repository also has an example of a C++ / gtest dist_test frontend at build-support/dist-test.py. Kudu's frontend is substantially simpler, since its unit tests are statically linked binaries and thus do not have many dependencies and are simple to invoke.