.ci/generate_bear_requirements: Changes #2446

nemani · 2018-04-19T17:27:38Z

TODO: break into commits.
This commits updates the script to use ruamel.yaml
which is better fork of PyYaml.
Add Update and Check mode for diffing and updating
Bear-Requirments yaml.

Closes #2444

For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!

Checklist

I read the commit guidelines and I've followed
them.
I ran coala over my code locally. (All commits have to pass
individually. It is not sufficient to have "fixup commits" on your PR,
our bot will still report the issues for the previous commit.) You will
likely receive a lot of bot comments and build failures if coala does not
pass on every single commit!

After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.

Please consider helping us by reviewing other peoples pull requests as well:

pick up any PR at https://coala.io/review
review it (check https://coala.io/reviewing for more info)
if you are sure that it needs work, use corobo mark wip <URL> to get it out
of the review queue.

The more you review, the more your score will grow at coala.io and we will
review your PRs faster!

nemani · 2018-04-19T17:28:28Z

@jayvdb this should be it. :))

nemani · 2018-04-22T00:27:00Z

@jayvdb Can you drop this a small review? I have exams this week, and wont get much time to refine the PR. But all necessary functionality is working IMO.

jayvdb

there are no pypi packages which provide deepupdate/deepdiff ?

jayvdb · 2018-04-22T00:41:00Z

bear-requirements.yaml

@@ -1,3 +1,84 @@
+# This is a comment!
+overrides: coala-build.yaml
+pip_requirements:


can you modify the generator so this doesnt move in the yaml.
It is easier to verify the code is correct than verify the moved yaml is identical

What do you mean? The pip_requirements part? or the overrides part?

jayvdb · 2018-04-22T00:41:13Z

bear-requirements.yaml

@@ -1,3 +1,84 @@
+# This is a comment!


obviously not wanted after WIP

jayvdb · 2018-04-22T00:42:29Z

.ci/generate_bear_requirements.py

+    parser.add_argument('--check', '-c', action='store_true',
+                        help='performs a dry run, and reports differences.')
+    parser.add_argument('--update', '-u',action='store_true',
+                        help='updates "bear-requirements.yaml" instaead of overwriting')


instaead

and 80cpl

jayvdb · 2018-04-22T00:44:18Z

Nice work so far, showing the principle idea will work.
Leave it now, until after the exams.

jayvdb · 2018-06-15T17:17:06Z

.travis.yml

@@ -115,7 +115,8 @@ script:
  - pip install $(ls ./dist/*.whl)"[alldeps]"
  - bash .ci/tests.sh
  # Ensure bear requirements are in sync with the bear PipRequirement
-  - .ci/generate_bear_requirements.py
+  - .ci/generate_bear_requirements.py --check


This should be --check --update in the same invocation to speed up the build.

jayvdb · 2018-06-15T17:18:23Z

.ci/generate_bear_requirements.py

    requirements['gem_requirements'] = get_gem_requirements(gem_reqs)
+    requirements['npm_requirements'] = get_npm_requirements(npm_reqs)
+    requirements['pip_requirements'] = get_pip_requirements(pip_reqs)


keep the order : pip ; npm ; gem

then these lines wont appear as moved.

jayvdb · 2018-06-15T17:18:45Z

.ci/generate_bear_requirements.py

+    requirements['pip_requirements'] = get_pip_requirements(pip_reqs)
+
+    if args.update or args.check:
+        inputFile = open(os.path.join(PROJECT_DIR, BEAR_REQUIREMENTS_YAML),'r')


pep8 variable names please

jayvdb · 2018-06-15T17:18:57Z

.ci/generate_bear_requirements.py

+            target[key] = copy.copy(value)
+
+
+def deepdiff(target, src):


deep_diff

jayvdb · 2018-06-15T17:19:17Z

.ci/generate_bear_requirements.py

+    errors = []
+    for key, value in src.items():
+        if not key in target:
+                errors.append((key, "Missing"))


bad indentation; single quote.

jayvdb · 2018-06-15T17:20:24Z

.ci/generate_bear_requirements.py

@@ -167,6 +176,46 @@ def get_pip_requirements(requirements):

    return pip_requirements

+def deepupdate(target, src):


deep_update

jayvdb · 2018-06-15T17:21:55Z

.ci/generate_bear_requirements.py

@@ -30,6 +31,9 @@
 from dependency_management.requirements.NpmRequirement import NpmRequirement
 from dependency_management.requirements.PipRequirement import PipRequirement

+yaml = YAML(typ="rt")


single quotes is our preferred style now

jayvdb · 2018-06-15T17:22:03Z

.ci/generate_bear_requirements.py

@@ -18,9 +18,10 @@
 import json
 import os
 import sys
-from collections import OrderedDict
+import copy


alpha order

jayvdb · 2018-06-15T17:25:19Z

.ci/generate_bear_requirements.py

+    for key, value in src.items():
+        if isinstance(value, list):
+            if not key in target:
+                target[key] = copy.deepcopy(value)


It is simpler if you always do this, irrespective of the type

if not key in target: target[key] = copy.deepcopy(value)

Then in the else you can do the different update mechanisms.
That makes it more clear that this is like a deepcopy for any new keys, but behaves differently for other keys, and raise an exception if there is an update required for an unknown type as we will need to add support for it whenever that happens.

jayvdb · 2018-06-15T17:27:40Z

test-requirements.txt

@@ -9,3 +9,4 @@ pytest-timeout~=1.2
 pytest-xdist~=1.15
 requests_mock~=0.7.0
 wheel~=0.29
+ruamel.yaml==0.15.37


ruamel.yaml~=0.15.38

jayvdb · 2018-06-20T03:30:49Z

.travis.yml

@@ -150,7 +150,10 @@ script:
  - pip install $(ls ./dist/*.whl)"[alldeps]"
  - bash .ci/tests.sh
  # Ensure bear requirements are in sync with the bear PipRequirement
-  - .ci/generate_bear_requirements.py
+  - .ci/generate_bear_requirements.py --check --update
+  - git clone https://github.com/moremoban/setupmobans ../setupmobans


these three lines were moved to a separate job

jayvdb · 2018-06-22T06:17:48Z

Need to rebase on top of #2542 to get into the queue (current 3 in the queue waiting to be merged)

jayvdb · 2018-06-22T09:27:29Z

@gitmate-bot rebase

gitmate-bot · 2018-06-22T09:27:31Z

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

gitmate-bot · 2018-06-22T09:30:33Z

Automated rebase with GitMate.io was successful! 🎉

jayvdb · 2018-07-05T08:10:58Z

And rebase your PR before uploading!

jayvdb · 2018-07-05T13:33:19Z

.ci/generate_bear_requirements.py

-    requirements = {}
+    requirements = CommentedMap()
+    requirements.yaml_set_start_comment(
+                    "This is an automatically generated file.\n "


indent four spaces only and use single quotes.

and rebase again ;-)

jayvdb · 2018-07-05T23:33:04Z

.ci/generate_bear_requirements.py

+        try:
+            input_file = open(input_file_path, 'r')
+        except FileNotFoundError:
+            print("bear-requirements.yaml not found. "


single quotes.

note #2584 . Stop using me as your personal linter and instead use coala-ci to do that for you :P

jayvdb

There are quite a few places where comments are stripped in "update" mode, especially anywhere around the "foo: true" gems. but we have #2561 for those.

This is good enough to begin with, as we will be able to add comments per requirement and them not disappear.

jayvdb · 2018-07-13T00:07:39Z

ack d253c61

This commits updates the script to use ruamel.yaml which is better fork of PyYaml. Also adds Update and Check mode for diffing and updating Bear-Requirements yaml. Closes coala#2444

jayvdb · 2018-07-13T12:09:18Z

@gitmate-bot ff

gitmate-bot · 2018-07-13T12:09:20Z

Hey! I'm GitMate.io! This pull request is being fastforwarded automatically. Please DO NOT push while fastforward is in progress or your changes would be lost permanently ⚠️

gitmate-bot · 2018-07-13T12:10:30Z

Automated fastforward with GitMate.io was successful! 🎉

gitmate-bot added process/wip size/M area/ci difficulty/medium importance/medium labels Apr 19, 2018

gitmate-bot assigned nemani Apr 19, 2018

jayvdb requested changes Apr 22, 2018

View reviewed changes

nemani force-pushed the ci-yaml branch from e1f91d7 to 1ba813f Compare May 27, 2018 01:01

gitmate-bot added size/S and removed size/M labels May 27, 2018

nemani force-pushed the ci-yaml branch from 1ba813f to 9c7b14e Compare May 28, 2018 23:36

jayvdb requested changes Jun 15, 2018

View reviewed changes

jayvdb mentioned this pull request Jun 17, 2018

cEP 23: Breaking down coala coala/cEPs#140

Open

nemani force-pushed the ci-yaml branch from 9c7b14e to cac6342 Compare June 20, 2018 00:11

jayvdb requested changes Jun 20, 2018

View reviewed changes

nemani force-pushed the ci-yaml branch from cac6342 to b1c2af3 Compare June 20, 2018 22:30

jayvdb mentioned this pull request Jun 22, 2018

Generate languages.yaml #2479

Closed

jayvdb changed the title ~~[WIP] .ci/generate_bear_requirements: Changes~~ .ci/generate_bear_requirements: Changes Jun 22, 2018

gitmate-bot force-pushed the ci-yaml branch from b1c2af3 to 3bc57f9 Compare June 22, 2018 09:29

gitmate-bot added process/pending review and removed process/wip labels Jun 22, 2018

jayvdb requested review from saksham189 and yukiisbored June 24, 2018 11:17

nemani force-pushed the ci-yaml branch from 33525b7 to f1b9fb3 Compare July 5, 2018 09:00

gitmate-bot added process/pending review and removed process/wip labels Jul 5, 2018

jayvdb requested changes Jul 5, 2018

View reviewed changes

jayvdb added process/wip and removed process/pending review labels Jul 5, 2018

nemani force-pushed the ci-yaml branch from f1b9fb3 to ed176c6 Compare July 5, 2018 13:40

gitmate-bot added process/pending review and removed process/wip labels Jul 5, 2018

jayvdb requested changes Jul 6, 2018

View reviewed changes

jayvdb added process/wip and removed process/pending review labels Jul 6, 2018

nemani force-pushed the ci-yaml branch from ed176c6 to d253c61 Compare July 7, 2018 13:37

gitmate-bot added process/pending review and removed process/wip labels Jul 7, 2018

jayvdb approved these changes Jul 13, 2018

View reviewed changes

.ci/generate_bear_requirements: Add Update & Check

9c97d5e

This commits updates the script to use ruamel.yaml which is better fork of PyYaml. Also adds Update and Check mode for diffing and updating Bear-Requirements yaml. Closes coala#2444

nemani force-pushed the ci-yaml branch from d253c61 to 9c97d5e Compare July 13, 2018 06:18

gitmate-bot added process/approved and removed process/pending review labels Jul 13, 2018

gitmate-bot removed the process/approved label Jul 13, 2018

gitmate-bot merged commit 9c97d5e into coala:master Jul 13, 2018

gitmate-bot added process/approved and removed process/approved labels Jul 14, 2018

jayvdb mentioned this pull request Jul 20, 2018

generate_bear_requirements: Don't exit with 0 on error #1550

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.ci/generate_bear_requirements: Changes #2446

.ci/generate_bear_requirements: Changes #2446

nemani commented Apr 19, 2018

nemani commented Apr 19, 2018

nemani commented Apr 22, 2018

jayvdb left a comment

jayvdb Apr 22, 2018

nemani Apr 22, 2018

jayvdb Apr 22, 2018

jayvdb Apr 22, 2018

jayvdb commented Apr 22, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 15, 2018

jayvdb Jun 20, 2018

nemani Jun 20, 2018

jayvdb commented Jun 22, 2018

jayvdb commented Jun 22, 2018

gitmate-bot commented Jun 22, 2018

gitmate-bot commented Jun 22, 2018

jayvdb commented Jul 5, 2018

jayvdb Jul 5, 2018

nemani Jul 5, 2018

jayvdb Jul 5, 2018

jayvdb left a comment

jayvdb commented Jul 13, 2018

jayvdb commented Jul 13, 2018

gitmate-bot commented Jul 13, 2018

gitmate-bot commented Jul 13, 2018

		@@ -167,6 +176,46 @@ def get_pip_requirements(requirements):

		return pip_requirements

		def deepupdate(target, src):

.ci/generate_bear_requirements: Changes #2446

.ci/generate_bear_requirements: Changes #2446

Conversation

nemani commented Apr 19, 2018

Checklist

nemani commented Apr 19, 2018

nemani commented Apr 22, 2018

jayvdb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb commented Apr 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb commented Jun 22, 2018

jayvdb commented Jun 22, 2018

gitmate-bot commented Jun 22, 2018

gitmate-bot commented Jun 22, 2018

jayvdb commented Jul 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb left a comment

Choose a reason for hiding this comment

jayvdb commented Jul 13, 2018

jayvdb commented Jul 13, 2018

gitmate-bot commented Jul 13, 2018

gitmate-bot commented Jul 13, 2018