Use conda-build=3 #5274

isuruf · 2018-02-27T05:10:58Z

cc @conda-forge/core

Add c3i LICENSE
Restore the recipes/examples folder

conda-forge-linter · 2018-02-27T05:11:07Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/example2, recipes/example3) and found it was in an excellent condition.

isuruf · 2018-02-27T05:16:59Z

@msarahan, do you know why circleci works, but not appveyor?

jakirkham · 2018-02-27T06:33:36Z

Because the example gets removed from PRs. Same story with AppVeyor. This would happen on CircleCI; however, as you are running CircleCI on your fork, the example is not removed. If you disabled CircleCI on your fork and pushed a fresh commit, it would have the same issue.

This comes down to this code or similar for the other CIs, which makes sure we don't build recipes in PRs that were merged into master. Expect if you commented that code or changed example2 and example3 to not rely on example, the problem would be resolved.

If you do merge these examples, would double check that this code keeps them from going into feedstocks. Seems like it should though. Might also be worth moving all the example recipes into one big example directory to keep the line noise low.

isuruf · 2018-02-27T17:08:27Z

I've removed the example recipe for now. example2 now depends on example3, but example2 is built first. Is this a conda-build issue or was it never meant to replace conda-build-all?

msarahan · 2018-02-27T17:10:16Z

conda-build can build more than one recipe at a time, but it currently does not do topological sorting for which should be built first. We'd like to add that eventually, but it's not high priority for us. We use conda-concourse-ci internally for this purpose.

jakirkham · 2018-02-27T17:11:40Z

So I think the issue is example was moved to example3. Hence we are running into the same issue as before. Would restore example and just add a brand new copy of example3. Happy to push a commit to your PR if you would like assist.

isuruf · 2018-02-27T17:30:40Z

conda-build can build more than one recipe at a time, but it currently does not do topological sorting for which should be built first.

Okay. I guess I can write a simple python script that uses conda_concourse_ci.compute_build_graph.construct_graph. @msarahan, I don't see conda-concourse-ci in defaults. Is it meant for internal use only?

isuruf · 2018-02-27T17:31:12Z

So I think the issue is example was moved to example3.

I don't think that's an issue here, but feel free to push to this PR.

isuruf · 2018-02-27T18:30:36Z

Here's a script that uses conda_concourse_ci. It uses code only from https://github.com/conda/conda-concourse-ci/blob/master/conda_concourse_ci/compute_build_graph.py. We can even vendor it, since it's just one file. We'll need a new dependency, networkx.

import conda_build.conda_interface
import networkx as nx
import conda_build.api
from conda_concourse_ci.compute_build_graph import construct_graph
import argparse
import os

def build_all(recipes_dir, arch, variant_config_files):
    index = conda_build.conda_interface.get_index(channel_urls=['conda-forge'])
    conda_resolve = conda_build.conda_interface.Resolve(index)

    config = conda_build.api.Config(variant_config_files=variant_config_files)

    worker={'platform': get_host_platform(), 'arch': arch, 'label': get_host_platform()}

    G = construct_graph(recipes_dir, worker=worker, run='build', conda_resolve=conda_resolve,
                        folders=os.listdir(recipes_dir), config=config)

    order = list(nx.topological_sort(G))
    order.reverse()

    print('Building packages')
    print('\n'.join(order))

    for node in order:
        conda_build.api.build(G.node[node]['meta'])


def get_host_platform():
    from sys import platform
    if platform == "linux" or platform == "linux2":
        return "linux"
    elif platform == "darwin":
        return "osx"
    elif platform == "win32":
        return "win"


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('recipes_dir', default=os.getcwd(), help='Directory where the recipes are')
    parser.add_argument('--arch', default='64', help='target architecture (64 or 32)')
    parser.add_argument('-m','--variant-config-files', action='append', help='variant config files')
    args = parser.parse_args()
    build_all(args.recipes_dir, args.arch, args.variant_config_files)

CJ-Wright · 2018-02-27T19:21:04Z

.ci_support/build_all.py

+                        folders=os.listdir(recipes_dir), config=config)
+
+    order = list(nx.topological_sort(G))
+    order.reverse()


Why is the build order reversed? (wouldn't this build the bottom of the graph first?)

Could be that I build the graph backward when adding the nodes. I have considered going back and reversing it, but haven't found time. When I have tried, it ended up exploding in my face.

It depends on how the edge direction is defined. reverse here gives the order that I need

scopatz · 2018-02-27T19:25:50Z

recipes/example3/meta.yaml

@@ -10,7 +10,7 @@
 #  `conda install openssl -c conda-forge``

 package:
-  name: {{ name|lower }}
+  name: simplejson2


Is this on purpose?

Yes, but I'll be reverting the examples once I know this works.

isuruf · 2018-02-27T19:39:26Z

This passes on appveyor in my fork.
I can use some help in getting linux to work. Need to copy the python files in .ci_support to the docker container and use them.

isuruf · 2018-02-28T00:53:20Z

Strange errors in linux and osx. @msarahan, any suggestions?

msarahan · 2018-02-28T01:00:54Z

In both cases, it's something to do with the system compilers and the resulting binary search path:

https://travis-ci.org/conda-forge/staged-recipes/builds/346956154#L302

On circle:

Fixing permissions
patchelf: file: /home/conda/staged-recipes/build_artifacts/simplejson2_1519761708098/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python2.7/site-packages/simplejson/_speedups.so
    setting rpath to: $ORIGIN/../../..
WARNING (simplejson2,_speedups.so): did not find - or even know where to look for: /lib64/libc.so.6
WARNING (simplejson2,_speedups.so): did not find - or even know where to look for: /lib64/libpthread.so.0

I don't know anything more off the top of my head. I'll try to reproduce in a docker container tomorrow. @mingwandroid might have more ideas than me.

isuruf · 2018-02-28T01:07:06Z

I can reproduce this locally on linux.

It tries to test a py2.7 package on a py3.6 environment. definitely something wrong with my script


The following NEW packages will be INSTALLED:

    ca-certificates: 2018.1.18-0  conda-forge                                      
    ncurses:         5.9-10       conda-forge                                      
    openssl:         1.0.2n-0     conda-forge                                      
    python:          3.6.4-0      conda-forge                                      
    readline:        7.0-0        conda-forge                                      
    simplejson2:     3.8.2-py27_0 file:///home/conda/staged-recipes/build_artifacts
    sqlite:          3.20.1-2     conda-forge                                      
    tk:              8.6.7-0      conda-forge                                      
    xz:              5.2.3-0      conda-forge                                      
    zlib:            1.2.11-0     conda-forge

isuruf · 2018-02-28T01:40:50Z

Tests pass. Don't merge till I restore the example folder.

Can I get a review?

msarahan · 2018-02-28T02:10:56Z

What conda version are you using? You should be aware of conda/conda-build#2707. This sounds potentially related.

TLDR: probably better to pin conda to 4.3.x for now. Kale is working on a fix.

msarahan · 2018-02-28T02:12:20Z

.appveyor.yml

@@ -5,13 +5,12 @@ environment:

  # Workaround for https://github.com/conda/conda-build/issues/636
  PYTHONIOENCODING: "UTF-8"
+  CONDA_INSTALL_LOCN: "C:\\Miniconda36-x64"


better to leave this as Miniconda3-x64 - that's a symlink to the current miniconda3 install on appveyor. It'll be better to keep up-to-date that way (fewer updates, potentially)

msarahan · 2018-02-28T02:14:43Z

.ci_support/compute_build_graph.py

@@ -0,0 +1,597 @@
+#!/usr/bin/env python


This is pretty big to vendor. It's probably OK, but it would be good to re-examine this vendoring decision in a few months and see if it's still a good idea, or if it would be better to add a dependency on c3i.

I prefer adding c3i as a dependency, but c3i is not on defaults, so I thought it was meant for internal use only.

There's also a lot of unused stuff here. I'm not sure if you want to remove it, or just leave it in for easier diffs down the line, should there be changes to both c3i and this file.

We can work on getting it onto defaults. It has only seen internal use, but I wouldn't say it's only meant for internal use. It's just much more oriented towards our needs, but could grow more general if there's community use and input (PRs definitely welcome).

Sorry to be late. What was the outcome on this?

anaconda-graveyard/conda-concourse-ci#71 (comment)

This functionality will be split from c3i and make its way into conda-build. No timeline right now, but hopefully soon. I don't think it will take too much work to have it simply work. Having it go fast will take more time.

isuruf · 2018-04-09T21:33:53Z

Is the concern that recipes requiring conda-build 3 get merged before we release conda-smithy 3?

Yes.

How would you recommend we address this? Documentation? Something else?

Release conda-smithy=3 first since anything working on cb2 works with cb3, there will be no issues.

Merge this PR after conda-smithy=3 is released, then recipes with cb3 features will work in both staged-recipes and feedstocks

jakirkham · 2018-04-10T03:05:40Z

Merge this PR after conda-smithy=3 is released, then recipes with cb3 features will work in both staged-recipes and feedstocks

So this gets at my larger question. Are we generally ok with merging this as is (aside from merge conflicts and order of merges)?

scopatz · 2018-04-10T14:59:31Z

@jakirkham - yes, I think everyone I talked to yesterday at the con about his @ocefpaf @CJ-Wright @ericdill @jjhelmus etc are OK with it

mwcraig · 2018-04-10T15:00:43Z

Not there, but I'm in favor of this moving ahead....

scopatz · 2018-04-10T15:06:52Z

Hmm I resolved the conflicts but it looks like it failed...

msarahan · 2018-04-10T16:19:25Z

fail seems like a conda update related thing. Kale is no longer tolerating channels that don't have noarch. I will hack in a fix in a bit.

isuruf · 2018-04-10T18:17:11Z

.appveyor.yml

@@ -3,7 +3,7 @@ skip_commits:

 environment:

-  CONDA_INSTALL_LOCN: "C:\\Miniconda36-x64"
+  CONDA_INSTALL_LOCN: "C:\\Miniconda3-x64"


Please don't. conda/conda#7144

Hmmm... OK. I was probably chasing a red herring with the conda exe thing. I will revert this. Thanks for posting the issue.

Given recent issues this seems to solve, are we ok leaving Miniconda36 here short term? This may already be agreeable at this stage. Just trying to confirm.

scopatz · 2018-04-11T15:35:40Z

Pulling the trigger.

ocefpaf · 2018-04-11T16:17:00Z

Let's release conda-smith ;-) I'll get to it soln if no one beats me to it.

…

On Wed, Apr 11, 2018, 13:02 Isuru Fernando ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In .travis_scripts/create_feedstocks <#5274 (comment)> : > @@ -35,9 +35,7 @@ bash ~/miniconda.sh -b -p ~/miniconda ) source ~/miniconda/bin/activate root -conda install --yes --quiet git=2.12.2 -conda install -n root --yes --quiet jinja2 anaconda-client -conda install --yes --quiet conda-build=2 conda-smithy +conda install --yes --quiet conda-forge-ci-setup=1.* conda-smithy=3.* conda-forge-pinning git=2.12.2 This was supposed to be merged after conda-smithy=3 is released. Now feedstock conversion is broken due to this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6BLynp5IR30EyW5afWBLEiPNY94SZXks5tnikrgaJpZM4SUXVP> .

scopatz · 2018-04-11T16:18:11Z

I agree @ocefpaf :)

isuruf · 2018-04-11T16:20:09Z

Milestone for conda-smithy=3 is not yet complete, https://github.com/conda-forge/conda-smithy/milestone/7

ocefpaf · 2018-04-11T16:31:36Z

Let's change the milestone. I don't think those last two are crucial to our workflow.

…

On Wed, Apr 11, 2018, 13:20 Isuru Fernando ***@***.***> wrote: Milestone for cs3 is not yet complete, https://github.com/conda-forge/conda-smithy/milestone/7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6BLzohcN3McEEC1UBCse_1Yy3tHwRuks5tni06gaJpZM4SUXVP> .

isuruf · 2018-04-11T16:33:17Z

Please test before releasing. I'd urge everyone from core to rerender at least 2 of their favourite recipes and see if it works on CI

ocefpaf · 2018-04-11T16:35:58Z

On it. Thanks @isuruf!

scopatz · 2018-04-11T16:41:53Z

I have tried rerendering a few of the harder packages, such as cyclus, and everything works like a dream

isuruf · 2018-04-11T16:43:04Z

Thanks @scopatz. Did you also try on CI?

scopatz · 2018-04-11T16:44:21Z

Not yet.

isuruf · 2018-04-11T16:51:32Z

Please have a look at https://github.com/conda-forge/conda-smithy/wiki/Release-Notes-3.0.0.rc1 and update if you find any mistakes

jakirkham · 2018-04-12T03:22:12Z

Had been playing with setuptools re-rendering yesterday and today (note that compiles code on Windows specifically). Seems to work fine after ironing out a few bugs earlier today.

xref: conda-forge/setuptools-feedstock#82

scopatz · 2018-04-12T14:18:14Z

Seems to work on cyclus and pyembree

MNT: Re-rendered with conda-smithy 3.0.0.rc3.dev30 and pinning 2018.0… cyclus-feedstock#31
MNT: Re-rendered with conda-smithy 3.0.0.rc3.dev30 and pinning 2018.0… pyembree-feedstock#11

isuruf added 2 commits February 26, 2018 21:41

cb3 update

0187910

Add examples

89fd608

Remove example recipe

6243d84

isuruf force-pushed the cb3 branch 2 times, most recently from 430124b to 8c78177 Compare February 27, 2018 19:12

CJ-Wright reviewed Feb 27, 2018

View reviewed changes

Build all using conda-build=3

3974c84

isuruf force-pushed the cb3 branch from 8c78177 to 3974c84 Compare February 27, 2018 19:25

scopatz reviewed Feb 27, 2018

View reviewed changes

isuruf added 2 commits February 27, 2018 13:56

Use build_all.py in linux

e5900c7

Add networkx in linux

74034b3

isuruf added 2 commits February 27, 2018 19:21

Finalize=True

184c082

Add c3i LICENSE

c64bac8

msarahan reviewed Feb 28, 2018

View reviewed changes

Merge branch 'master' into cb3

60177b7

Update .appveyor.yml

6cbeae8

flake8 in build_all; address win and linux CI failures

a2e7490

isuruf commented Apr 10, 2018

View reviewed changes

msarahan added 2 commits April 10, 2018 13:31

attempt to fix windows conda unrecognized command

2c13096

revert appveyor junction point miniconda path

542fe7e

scopatz merged commit b12f7e7 into conda-forge:master Apr 11, 2018

isuruf deleted the cb3 branch April 11, 2018 16:21

djsutherland mentioned this pull request Apr 14, 2018

move example2 back to example [skip ci] #5653

Merged

This was referenced Apr 23, 2018

Drop R 3.3.2. #5325

Closed

Proposal: Drop support for R 3.3.2 conda-forge/r-base-feedstock#36

Closed

Use conda-build=3 #5274

Use conda-build=3 #5274

Conversation

isuruf commented Feb 27, 2018 • edited Loading

conda-forge-linter commented Feb 27, 2018

isuruf commented Feb 27, 2018

jakirkham commented Feb 27, 2018

isuruf commented Feb 27, 2018

msarahan commented Feb 27, 2018

jakirkham commented Feb 27, 2018

isuruf commented Feb 27, 2018

isuruf commented Feb 27, 2018

isuruf commented Feb 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isuruf Feb 27, 2018 • edited Loading

Choose a reason for hiding this comment

isuruf commented Feb 27, 2018

isuruf commented Feb 28, 2018

msarahan commented Feb 28, 2018

isuruf commented Feb 28, 2018

isuruf commented Feb 28, 2018

msarahan commented Feb 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isuruf commented Apr 9, 2018

jakirkham commented Apr 10, 2018

scopatz commented Apr 10, 2018

mwcraig commented Apr 10, 2018

scopatz commented Apr 10, 2018

msarahan commented Apr 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scopatz commented Apr 11, 2018

ocefpaf commented Apr 11, 2018 via email

scopatz commented Apr 11, 2018

isuruf commented Apr 11, 2018 • edited Loading

ocefpaf commented Apr 11, 2018 via email

isuruf commented Apr 11, 2018 • edited Loading

ocefpaf commented Apr 11, 2018

scopatz commented Apr 11, 2018

isuruf commented Apr 11, 2018

scopatz commented Apr 11, 2018

isuruf commented Apr 11, 2018

jakirkham commented Apr 12, 2018 • edited Loading

scopatz commented Apr 12, 2018

isuruf commented Feb 27, 2018 •

edited

Loading

isuruf Feb 27, 2018 •

edited

Loading

isuruf commented Apr 11, 2018 •

edited

Loading

isuruf commented Apr 11, 2018 •

edited

Loading

jakirkham commented Apr 12, 2018 •

edited

Loading