Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail pipeline on first exception #379

Merged
merged 2 commits into from
Feb 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .flake8
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[flake8]
max-line-length = 88
max-complexity = 8
ignore = W503,E501
ignore = W503,E501,C901
builtins = unicode
tee = True
exclude = venv,env,.venv,scratch
exclude = venv,env,.venv,scratch
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.2.0] - 2023-02-18

### Added

- Option to specify `fail_pipeline_on_first_exception`, default to `true`

## [1.1.0] - 2023-02-02

### Added
Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[![Actions Status](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/workflows/Lint/badge.svg?branch=main)](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)
[![Actions Status](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/workflows/Unit%20Tests/badge.svg?branch=main)](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)
![Version](https://img.shields.io/static/v1.svg?label=Version&message=0.5.0&color=lightgrey&?link=http://left&link=https://github.com/wayfair-incubator/bigquery-buildkite-plugin/tree/v1.0.0)
![Version](https://img.shields.io/static/v1.svg?label=Version&message=1.2.0&color=lightgrey&?link=http://left&link=https://github.com/wayfair-incubator/bigquery-buildkite-plugin/tree/v1.2.0)
![Plugin Status](https://img.shields.io/static/v1.svg?label=&message=Buildkite%20Plugin&color=blue&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGAAAABgCAMAAADVRocKAAAAk1BMVEX///+DrRik1Cb+/vyErhin1yeAqhfJ5n+dzCO50X2VwiC2zna020vg8LSo1i+q1zTN3qL7/fav2UDT4q33++yXuj3z+eLO6IqMuByJsSPM54XX5bWcvUbH5Xrg6sWiwVHB1ovk7cymxFnq8deOtC2xzG7c6L6/1YfU7JbI2pj2+e+jzzPd763K3J3j8ryRtDbY7aOqCe3pAAACTElEQVRoge3Z11KDQBSAYUpItSRqVGwxEVOwxPd/OndhCbA5BTaLM87sueb83+yMStHz3Lhx48bN/5iw+2VjImy0fDvzQiNCLKVrdjn0nq++QwNCLMy+9uOzc2Y59AZBwF4F55NefxxxyxK4aEvI/C7x/QxglrMTBNxVej6V+QNALh+AhkRY5isAsVwBxFWfM/rnLsu/qnwNQIkawBBy/abMawBCaEAQXGFElt/Evo8CIHEEIESWH9XyAAAQACCIH40A8yBwRICARiB5BNAIBKgQ8tLbCZBHgRqBAgWB5wmgQhCAIt7ekTwJ5AQHSGKC1TkgiM7WIs8A0bBvCETR8H7CA4EhIPP9fgPA7AR53u91BBR53+8EKPPdACLvq3wXQC1vH9DytoGjvF0gGo71vE0gCoC8PQDJ2wJEvgfmbQFo3g5A5HNA3K+eL0yBePdO5AtAEAOUoIB4c+ONiHwBZHddjMCB5DUV6yOqfzgBQWCAzMu9ZoAiHgACBpJdmj3SNAdQAgKSXfFQ1gZQz28PlxxQ5tsCIKED+6/qU2tbQBF3lxgwn9YfitsDR0QVmE9D7bHeBNCIEphf63lToEYUAJQ3BxSxlUQGPD3Cr5DmwIGQJ8DypwHqlXX7gec5oMcAXv5OT31OYU6weuM/9tDv/iSwWnJxfghA5s0+QzUCFi828iiwWNvJI4C9PAg8WcwDAPVLYwGwndeAufV8DZB/bm3nK0A3+RyI81tdF3l1gn1neQlM4qnpp+9ms0xP/P8AP/8778aNGzdu/mh+AQp1NCB/JInXAAAAAElFTkSuQmCC)


Expand All @@ -19,7 +19,7 @@ If the version number is not provided then the most recent version of the plugin
```yaml
steps:
- plugins:
- wayfair-incubator/bigquery#v1.0.0:
- wayfair-incubator/bigquery#v1.2.0:
gcp_project: gcp-us-project
dataset_schema_directory: schemas/gcp-us-project/dataset
```
Expand Down Expand Up @@ -48,14 +48,22 @@ Example: `main`

Default: `master`

### `execute_only_changed_files` (optional, string)
### `execute_only_changed_files` (optional, boolean)

Whether to execute just the files that changed in a directory or not.

Example: `true`

Default: `true`

### `fail_pipeline_on_first_exception` (optional, boolean)

Whether to fail pipeline as soon as there is one failure.

Example: `true`

Default: `true`

## Secret

This plugin expects `GCP_SERVICE_ACCOUNT` is placed as environment variable. Make sure to store it [securely](https://buildkite.com/docs/pipelines/secrets)!
Expand Down
5 changes: 5 additions & 0 deletions hooks/command
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,17 @@ else
changed_files=$(git diff-tree --diff-filter=ABMRT --no-commit-id --name-only -r origin..HEAD | xargs | sed -e 's/ /,/g')
fi

default_fail_pipeline_on_first_exception="true"
fail_pipeline_on_first_exception="${BUILDKITE_PLUGIN_BIGQUERY_DEFAULT_FAIL_PIPELINE_ON_FIRST_EXCEPTION:-$default_fail_pipeline_on_first_exception}"

echo "$gcp_service_account" >"$PIPELINE_FILE"

if [[ ${debug_mode} == "true" ]]; then
echo "Configuration:"
echo "GCP Project: ${gcp_project}"
echo "Dataset Schema Directory: ${dataset_schema_directory}"
echo "Execute Only Changed Files: ${execute_only_changed_files}"
echo "Fail Pipeline on First Exception: ${fail_pipeline_on_first_exception}"
echo "Changed Files: ${changed_files}"
echo "Is Default Branch: ${is_default_branch}"
echo "Docker Image: ${image}"
Expand Down Expand Up @@ -120,6 +124,7 @@ args+=(
"--env" "dataset_schema_directory=$dataset_schema_directory"
"--env" "gcp_project=$gcp_project"
"--env" "execute_only_changed_files=$execute_only_changed_files"
"--env" "fail_pipeline_on_first_exception=$fail_pipeline_on_first_exception"
"--env" "credentials=$(</tmp/service_account.json)"
"--volume" "$BUILDKITE_AGENT_BINARY_PATH:/usr/bin/buildkite-agent"
)
Expand Down
4 changes: 3 additions & 1 deletion plugin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ configuration:
prod_build_branch:
type: string
execute_only_changed_files:
type: string
type: boolean
fail_pipeline_on_first_exception:
type: boolean
custom_image:
type: string
required:
Expand Down
10 changes: 9 additions & 1 deletion plugin_scripts/deploy.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ def _deploy():
execute_only_changed_files = _str_to_bool(
os.environ.get("execute_only_changed_files", "true")
)
fail_pipeline_on_first_exception = _str_to_bool(
os.environ.get("fail_pipeline_on_first_exception", "true")
)

try:
bq = BigQuery(credentials, gcp_project)
Expand All @@ -47,6 +50,7 @@ def _deploy():
dataset_schema_directory,
updated_files,
execute_only_changed_files,
fail_pipeline_on_first_exception,
)
except Exception:
deploy_failed = True
Expand All @@ -61,7 +65,8 @@ def _deploy_from_directory(
dataset_schema_directory: str,
updated_files: List[str],
execute_only_changed_files: bool,
):
fail_pipeline_on_first_exception: bool,
) -> bool:
deploy_failed = False
for root, dirs, files in os.walk(dataset_schema_directory):
dataset = root.split("/").pop()
Expand Down Expand Up @@ -91,6 +96,9 @@ def _deploy_from_directory(
print(f"Failed to deploy to Bigquery: {str(e)}")
deploy_failed = True

if fail_pipeline_on_first_exception:
return deploy_failed

return deploy_failed


Expand Down