Skip to content

Commit

Permalink
Merge pull request dmakhno#1 from pycontribs/master
Browse files Browse the repository at this point in the history
merge from hugovk
  • Loading branch information
hugovk committed Jan 12, 2015
2 parents 4d8c539 + f3b6e2e commit 51e8ffd
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 24 deletions.
18 changes: 12 additions & 6 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
language: python
python:
- '2.6'
- '2.7'
- '3.2'
- '3.3'
- '3.4'
- 'pypy'
- 'pypy3'
script:
- curl -o travis_after_all.py https://raw.github.com/dmakhno/travis_after_all/master/travis_after_all.py
- echo "Doing something..."
- echo "Done"
after_success:
- python travis_after_all.py
- export $(cat .to_export_back)
- |
if [ "$BUILD_LEADER" = "YES" ]; then
if [ "$BUILD_AGGREGATE_STATUS" = "others_succeeded" ]; then
echo "All Succeded! PUBLISHING..."
echo "All jobs succeeded! PUBLISHING..."
else
echo "Some Failed"
echo "Some jobs failed"
fi
fi
after_failure:
Expand All @@ -21,10 +27,10 @@ after_failure:
- |
if [ "$BUILD_LEADER" = "YES" ]; then
if [ "$BUILD_AGGREGATE_STATUS" = "others_failed" ]; then
echo "All Failed"
echo "All jobs failed"
else
echo "Some Failed"
echo "Some jobs failed"
fi
fi
after_script:
- echo leader=$BUILD_LEADER status=$BUILD_AGGREGATE_STATUS
- echo leader=$BUILD_LEADER status=$BUILD_AGGREGATE_STATUS
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
travis_after_all
travis_after_all
================

[![Build Status](https://travis-ci.org/dmakhno/travis_after_all.png?branch=master)](https://travis-ci.org/dmakhno/travis_after_all)

This is travis helper to run particular work only once in matrix.
This is Travis CI helper to run particular work only once in matrix.

This is workaround for: https://github.com/travis-ci/travis-ci/issues/929

The main goal of this script to have single publishing when build have several jobs. Currently first job is a leader, meaning a node who will mostly do publishing.
The main goal of this script to have a single publish when a build has several jobs. Currently the first job is a leader, meaning a node that will do the publishing.

Existing .travis.yml show how to ensure that all_succeeded or all_failed
An example .travis.yml shows how to ensure that `all_succeeded` or `all_failed`:

```yaml
#...
script:
- curl -o travis_after_all.py https://raw.github.com/dmakhno/travis_after_all/master/travis_after_all.py
- curl -Lo travis_after_all.py https://raw.github.com/dmakhno/travis_after_all/master/travis_after_all.py
after_success:
- python travis_after_all.py
- export $(cat .to_export_back)
- |
if [ "$BUILD_LEADER" = "YES" ]; then
if [ "$BUILD_AGGREGATE_STATUS" = "others_succeeded" ]; then
echo "All Succeded! PUBLISHING..."
echo "All jobs succeeded! PUBLISHING..."
else
echo "Some Failed"
echo "Some jobs failed"
fi
fi
after_failure:
Expand All @@ -32,9 +32,9 @@ after_failure:
- |
if [ "$BUILD_LEADER" = "YES" ]; then
if [ "$BUILD_AGGREGATE_STATUS" = "others_failed" ]; then
echo "All Failed"
echo "All jobs failed"
else
echo "Some Failed"
echo "Some jobs failed"
fi
fi
after_script:
Expand All @@ -44,6 +44,6 @@ after_script:
Limitations/Todo
----------------
- If other jobs will start late, can be global build timeout. (Think of passing leader role to others, for example who started last is the leader)
- More flexible leader defenition (In Matrix not all jobs can publish)
- Have several leaders, according to flavours (E.g. if in Matrix slices responsible for platform)
- If other jobs will start late, can be global build timeout (think of passing leader role to others, for example who started last is the leader).
- More flexible leader definition (in matrix not all jobs can publish).
- Have several leaders, according to flavours (e.g. if in matrix slices responsible for platform).
22 changes: 16 additions & 6 deletions travis_after_all.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import json
import time
import logging
import requests

try:
import urllib.request as urllib2
Expand All @@ -15,9 +16,11 @@
TRAVIS_JOB_NUMBER = 'TRAVIS_JOB_NUMBER'
TRAVIS_BUILD_ID = 'TRAVIS_BUILD_ID'
POLLING_INTERVAL = 'LEADER_POLLING_INTERVAL'
GITHUB_TOKEN = 'GITHUB_TOKEN'

build_id = os.getenv(TRAVIS_BUILD_ID)
polling_interval = int(os.getenv(POLLING_INTERVAL, '5'))
gh_token = os.getenv(GITHUB_TOKEN)

#assume, first job is the leader
is_leader = lambda job_number: job_number.endswith('.1')
Expand All @@ -44,24 +47,24 @@ def __init__(self, json_raw):
self.is_leader = is_leader(self.number)


def matrix_snapshot():
def matrix_snapshot(token):
"""
:return: Matrix List
"""
response = urllib2.build_opener().open("https://api.travis-ci.org/builds/{0}".format(build_id)).read()
response = urllib2.build_opener().open("https://api.travis-ci.com/builds/{0}?access_token={1}".format(build_id, token)).read()
raw_json = json.loads(response)
matrix_without_leader = [MatrixElement(element) for element in raw_json["matrix"]]
return matrix_without_leader


def wait_others_to_finish():
def wait_others_to_finish(token):
def others_finished():
"""
Dumps others to finish
Leader cannot finish, it is working now
:return: tuple(True or False, List of not finished jobs)
"""
snapshot = matrix_snapshot()
snapshot = matrix_snapshot(token)
finished = [el.is_finished for el in snapshot if not el.is_leader]
return reduce(lambda a, b: a and b, finished), [el.number for el in snapshot if
not el.is_leader and not el.is_finished]
Expand All @@ -72,11 +75,18 @@ def others_finished():
log.info("Leader waits for minions {0}...".format(waiting_list)) # just in case do not get "silence timeout"
time.sleep(polling_interval)

def getToken():
data = {"github_token":gh_token}
headers = {'content-type': 'application/json'}
response = requests.post('https://api.travis-ci.com/auth/github', data=json.dumps(data), headers=headers).json()
token = response.get('access_token')
return token

try:
wait_others_to_finish()
token = getToken()
wait_others_to_finish(token)

final_snapshot = matrix_snapshot()
final_snapshot = matrix_snapshot(token)
log.info("Final Results: {0}".format([(e.number, e.is_succeeded) for e in final_snapshot]))

BUILD_AGGREGATE_STATUS = 'BUILD_AGGREGATE_STATUS'
Expand Down

0 comments on commit 51e8ffd

Please sign in to comment.