WIP windows conda build for ska3-win #227

taldcroft · 2019-11-22T02:36:19Z

This is a proof-of-concept for making Windows builds which seems to be working. I had quite a journey getting this working so ended up doing micro-commits to allow backtracking to a configuration that works.

It seems that using python setup.py install in this context is fine since conda is doing the file management and so pip is a bit of overkill. But perhaps more importantly, modern pip has dropped support for eggs and this is needed for namespace packages. So going to python setup.py install should allow upgrading pip to the latest.

It also seems like we can just use the build : script directive in meta.yaml, but maybe there is a reason not to that @jeanconn has discovered.

Next steps would be doing the rest of the arch-specific packages and then dealing with the core packages. I'm guessing that our totally-pinned ska3-core package won't resolve on Windows, but not sure.

There is something a bit unsettling in the build log. I did some googling and got to some relevant pages, but couldn't quite make sense of them.

Packaging chandra.time
INFO:conda_build.build:Packaging chandra.time
INFO conda_build.build:build(1570): Packaging chandra.time
Packaging chandra.time-3.20.3-py36_0
INFO:conda_build.build:Packaging chandra.time-3.20.3-py36_0
INFO conda_build.build:bundle_conda(891): Packaging chandra.time-3.20.3-py36_0
compiling .pyc files...
number of files: 22
   INFO (chandra.time,Lib\site-packages\Chandra.Time-3.20.3-py3.6-win-amd64.egg\Chandra\Time\_axTime3.cp36-win_amd64.pyd): Needed DSO python36.dll found in defaults::python-3.6.9-h5500b2f_0
WARNING (chandra.time,Lib\site-packages\Chandra.Time-3.20.3-py3.6-win-amd64.egg\Chandra\Time\_axTime3.cp36-win_amd64.pyd): Needed DSO api-ms-win-crt-string-l1-1-0.dll found in ['vs2015_runtime']
WARNING (chandra.time,Lib\site-packages\Chandra.Time-3.20.3-py3.6-win-amd64.egg\Chandra\Time\_axTime3.cp36-win_amd64.pyd): .. but ['vs2015_runtime'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (chandra.time,Lib\site-packages\Chandra.Time-3.20.3-py3.6-win-amd64.egg\Chandra\Time\_axTime3.cp36-win_amd64.pyd): Needed DSO api-ms-win-crt-runtime-l1-1-0.dll found in ['vs2015_runtime']
WARNING (chandra.time,Lib\site-packages\Chandra.Time-3.20.3-py3.6-win-amd64.egg\Chandra\Time\_axTime3.cp36-win_amd64.pyd): .. but ['vs2015_runtime'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

... AND a bunch more like that

cc: @mbaski @jskrist

jeanconn · 2019-11-22T19:10:37Z

I think one of the reasons most of the packages are set with separate build.sh files is indeed because I thought we'd plop build.bat files in there later. If the windows, Linux, and OSX builds will be good with the same build process that isn't needed.

taldcroft · 2019-11-22T19:45:29Z

As far as I can tell python setup.py install is sufficient in all cases (Python packages) I have thought about.

taldcroft · 2019-11-22T19:51:12Z

I will say that working in Windows turns out to be at least as frustrating as I had feared. From little thinks like "the instructions you find online for recursively removing a directory simply don't work on my machine", to "conda install ska.engarchive" (from our icxc ska3conda) pretends to install that package, and makes it show up in the package list, but doesn't seem to actually put the code into the environment. I guess when you start climbing Everest you know it's going to be painful, especially if you have never worn crampons before (and that is like me and Windows).

jeanconn · 2019-11-22T19:54:40Z

Regarding the Ska.Engarchive issue; do you have a take-away yet? I haven't looked at this at all so was wondering if you were hitting namespace package issues, or issues installing a conda "generic" noarch package, or if even the noarch packages will need to be rebuilt in some way to keep Windows happy.

taldcroft · 2019-11-23T01:20:08Z

I just noticed that Ska.engarchive has noarch: generic instead of noarch: python (like the other noarch Python packages I looked at). And now the phrase issues installing a conda "generic" noarch package means something to me.

I confirmed that making Ska.engarchive as noarch: python works in a local Windows build. This could be related either to windows not liking that, or conda 4.7 not liking that. In any case, noarch: python it is.

jeanconn · 2019-11-23T01:24:09Z

Ha. So it sounds like we should probably retire #168, and figure out if we will have separate noarch:python packages for Windows or if we will need a different way to use the same set of conda packaegs to install out-of-package files we want for production linux.

taldcroft · 2019-11-23T01:28:25Z

Note also this warning that I got from the Ska.Shell package from the icxc repo:

SafetyError: The package for ska.shell located at C:\Users\aldcroft\Miniconda3\pkgs\ska.shell-3.3.4-py_0
appears to be corrupted. The path 'site-packages/Ska.Shell.pth'
has an incorrect size.
  reported size: 28 bytes
  actual size: 29 bytes

This was also cleared using a local build.

jeanconn · 2019-11-23T01:28:53Z

Oh, and per #166, another option might be to make architecture-specific packages for all of them.

taldcroft · 2019-11-23T01:30:31Z

Gahhh..

jeanconn · 2019-11-23T01:32:00Z

Regarding Ska.Shell, it is odd that if there's a real issue with the tarball that Linux conda happily installed it. I'm not sure if I should use your report of weirdness as a credible bug against the builds for 2019_210 .

taldcroft · 2019-11-23T01:36:28Z

I have an inkling of the problem for Ska.engarchive now that you jogged my memory on the task_schedule.cfg. Basically the Windows Python path structure is completely different, so the "generic" build on MacOS is really not generic since it can't but fit into the Windows paths. Conda seems to unhelpfully just ignore (silently) a package that doesn't work. Or maybe I'm missing something, quite possible.

jeanconn · 2019-11-23T01:46:15Z

Right, my similar-but-not-identical guess would be that conda was installing (untarring and maybe copying?) the files from the noarch:generic package, they just didn't end up in anyplace useful for Windows Python.

So then still a question if we want something to help with files like task_schedule.cfg using:

different conda magic... maybe architecture-specific install directives on a noarch:python package
to go back to Makefile or a manual post-Python-module install step to install task_schedule.cfg
to build architecture-specific packages for everything for everything, which prompted your Gahh, because it would be annoying to need to build everything on a Windows VM if that can be avoided.
pull out "production" files like task_schedule to separate conda packages

taldcroft · 2019-11-23T01:48:09Z

Or not... here is the beginning of the file manifest from the Windows build:

python-scripts/cheta_add_derived
python-scripts/cheta_check_integrity
python-scripts/cheta_fix_bad_values
python-scripts/cheta_sync
python-scripts/cheta_update_server_archive
python-scripts/cheta_update_server_sync
python-scripts/ska_fetch
share/eng_archive/task_schedule.cfg
share/eng_archive/task_schedule_gap.cfg
share/eng_archive/task_schedule_sync.cfg
site-packages/Ska.engarchive-4.47.3-py3.6.egg/EGG-INFO/PKG-INFO
site-packages/Ska.engarchive-4.47.3-py3.6.egg/EGG-INFO/SOURCES.txt
site-packages/Ska.engarchive-4.47.3-py3.6.egg/EGG-INFO/dependency_links.txt

I confirmed that the share/task_schedule.cfg is indeed installed to the correct place and everything looks sensible from the conda docs (https://www.anaconda.com/condas-new-noarch-packages/).

All I can wonder is if there is a mismatch between conda versions 4.3 vs 4.7 in terms of how things work. E.g. if I built on Mac using noarch: python with conda 4.7 , would that install to my Windows. I guess I will have to try that experiment at some point, I really do not want to require Windows builds for everything, that would really be unhelpful for reducing our level of effort.

taldcroft · 2019-11-23T01:53:38Z

OK, here is the file manifest from our "noarch: generic":

bin/cheta_add_derived
bin/cheta_check_integrity
bin/cheta_fix_bad_values
bin/cheta_update_client_archive
bin/cheta_update_server_archive
bin/cheta_update_server_sync
bin/ska_fetch
lib/python3.6/site-packages/Ska.engarchive-4.45.1-py3.6.egg/EGG-INFO/PKG-INFO

So definitely that is not going to fly on Windows since the generic is just saying to drop the files exactly as show (confirming your similar but not identical guess?). So it gets back to why task_schedule.cfg wasn't getting installed in the right place earlier. We just need to solve that.

jeanconn · 2019-11-23T01:54:19Z

Wait, I'm a little confused. The first manifest is from your local install of your Windows locally-built version of Ska.engarchive with noarch:python set with a conda 4.7?

jeanconn · 2019-11-23T01:56:53Z

And regarding task_schedule etc, if we got in a pinch it would be annoying but not crazy to just have accompanying "production" packages intended for Linux (like a ska-engarchive-production-helpers conda package). But we don't have to go there yet.

taldcroft · 2019-11-23T02:32:30Z

I'm not seeing any problem building and installing the Ska.engarchive package on my Mac with conda 4.7.12 using noarch: python. So either I'm doing something differently or the problem has been resolved in the meantime (in conda).

neptune$ ls ~/tmp/miniconda3/envs/cheta/share/eng_archive/
task_schedule.cfg       task_schedule_gap.cfg   task_schedule_sync.cfg

neptune$ head ~/tmp/miniconda3/pkgs/ska.engarchive-4.47.3-py_0/info/files 
python-scripts/cheta_add_derived
python-scripts/cheta_check_integrity
python-scripts/cheta_fix_bad_values
python-scripts/cheta_sync
python-scripts/cheta_update_server_archive
python-scripts/cheta_update_server_sync
python-scripts/ska_fetch
share/eng_archive/task_schedule.cfg
share/eng_archive/task_schedule_gap.cfg
share/eng_archive/task_schedule_sync.cfg

jeanconn · 2019-11-23T02:38:54Z

OK. Sounds like conda 4.7.x is doing something different so all good.

jeanconn · 2019-11-23T14:27:20Z

Well almost all good, as I assume that limits us to only moving forward with a strategy for windows that splits from* the centos5/32bit tree if there is no conda 4.7 for our old systems. But maybe we could build conda itself if pushing on this.

*or waits for retirement of the old platform

taldcroft · 2019-11-23T18:02:47Z

Yes, this may well be the fork (or branch) in the road. I.e. we could have a centos5-32 branch to maintain that legacy and let master be the "bring it all forward!" branch.

taldcroft · 2019-11-23T18:09:02Z

Things could get a wee bit interesting in terms of managing all the new package builds (with conda 4.7 and recent dependencies like cython / numpy), where the version of our package is unchanged. Maybe something like $ska3conda/dev as the new root.

taldcroft · 2019-11-23T18:38:56Z

For the record, here are the ska3 builds that don't agree with conda 4.7 on windows:

SafetyError: The package for ska.file located at C:\Users\aldcroft\Miniconda3\pkgs\ska.file-3.4.1-py_0
appears to be corrupted. The path 'site-packages/Ska.File.pth'
has an incorrect size.
  reported size: 27 bytes
  actual size: 28 bytes

SafetyError: The package for ska.matplotlib located at C:\Users\aldcroft\Miniconda3\pkgs\ska.matplotlib-3.11.2-py_0
appears to be corrupted. The path 'site-packages/Ska.Matplotlib.pth'
has an incorrect size.
  reported size: 34 bytes
  actual size: 35 bytes

SafetyError: The package for ska.dbi located at C:\Users\aldcroft\Miniconda3\pkgs\ska.dbi-4.0-py_0
appears to be corrupted. The path 'site-packages/Ska.DBI.pth'
has an incorrect size.
  reported size: 24 bytes
  actual size: 25 bytes

SafetyError: The package for ska.quatutil located at C:\Users\aldcroft\Miniconda3\pkgs\ska.quatutil-3.3.1-py_0
appears to be corrupted. The path 'site-packages/Ska.quatutil.pth'
has an incorrect size.
  reported size: 31 bytes
  actual size: 32 bytes

SafetyError: The package for ska.sun located at C:\Users\aldcroft\Miniconda3\pkgs\ska.sun-3.5-py_0
appears to be corrupted. The path 'site-packages/Ska.Sun.pth'
has an incorrect size.
  reported size: 24 bytes
  actual size: 25 bytes

SafetyError: The package for ska.shell located at C:\Users\aldcroft\Miniconda3\pkgs\ska.shell-3.3.4-py_0
appears to be corrupted. The path 'site-packages/Ska.Shell.pth'
has an incorrect size.
  reported size: 28 bytes
  actual size: 29 bytes

taldcroft · 2019-11-23T19:12:16Z

I have a working proof of concept now that installs all the latest Ska3 packages and their direct dependencies. This version does not yet install many of our standard "core" packages, e.g. bokeh.
It passes tests for xija and cheta (except the sync tests, that currently require making symlinks). I can see that there will be some work ahead to get package unit tests all working, e.g. the kadi tests fail to run because it assumes existence of a USER environment variable.

conda create -n ska3-win  -c https://icxc.harvard.edu/aspect/ska3-conda/windows \
 -c https://icxc.harvard.edu/aspect/ska3-conda python=3.6 ska3-win

jeanconn · 2019-11-26T14:57:06Z

How about making this a PR against a new dev branch? We'll plan to baseline recipe updates and such in that branch for our 'modern' update.

jeanconn · 2019-11-26T15:00:32Z

Also, did you settle on Python 3.6 or have an explicit reason to freeze it there?

taldcroft · 2019-11-26T15:17:42Z

This is not settled, I was just trying a bit of walking before running / jumping toward 3.8.

jeanconn · 2019-11-26T15:20:32Z

That fine. I just wanted to keep an eye on any limits we explicitly added to the recipes and document as needed.

jeanconn · 2019-11-26T17:30:10Z

For "python setup.py install" do we need or want something like "--single-version-externally-managed --record=record.txt". They are going with that in the current conda docs and hard to figure out if it is a recommendation or just an example.

jeanconn · 2019-11-27T18:55:45Z

ska_builder.py

+        cmd_list.append("--skip-existing")
+
+    print('Running:\n' + ' '.join(cmd_list) + '\n')
+    subprocess.run(cmd_list, check=True, shell=True)


The addition of shell=True seems to break this for me on linux. The conda build command just moves to spitting out its helptext and I'm not quite sure why.

Sorry, yes I noticed that as well, so this needs to be conditional on Windows (shell=<is_windows>).

taldcroft added 12 commits November 21, 2019 22:10

Windows compatibility + linter fixes

f361202

Ignore builds/ and .vscode/ directories

f911eae

Working for cmd-line conda build

68ff0be

Working

954119c

Working, use ska_builder's download

3f08bdd

Use old build strings (working)

44a93c0

Add back perl cmd line option

d469de9

Works

478eed0

Whitespace

f9596ba

Put back preserving egg dir

ee986c2

Use script in meta for install

4627c3c

Add back --numpy flag

983f626

taldcroft force-pushed the windows branch from 453f08c to 983f626 Compare November 22, 2019 03:14

taldcroft added 2 commits November 22, 2019 13:17

Add command line args for test and force build

8df9fe1

Update Ska.Numpy recipe

930947f

taldcroft added 2 commits November 22, 2019 14:34

Add printout of conda build command

358e93b

Update xija build recipe

1eeb1e5

Update recipes for Ska.Shell and Ska.engarchive

fe30f18

More updates for windows builds

95febcb

Add core packages

9f71f39

taldcroft changed the title ~~WIP windows conda build for Chandra.Time~~ WIP windows conda build for ska3-win Nov 26, 2019

jeanconn reviewed Nov 27, 2019

View reviewed changes

jeanconn changed the base branch from master to shiny November 28, 2019 03:44

taldcroft mentioned this pull request Nov 28, 2019

WIP Shiny packages updates (CentOS6 conda 4.7 so far) #230

Closed

taldcroft merged commit d2bd4ac into shiny Dec 18, 2019

taldcroft deleted the windows branch January 30, 2020 19:29

javierggt mentioned this pull request Oct 16, 2020

Update sot/mica to Release 4.22.0 #475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP windows conda build for ska3-win #227

WIP windows conda build for ska3-win #227

taldcroft commented Nov 22, 2019 •

edited

Loading

jeanconn commented Nov 22, 2019

taldcroft commented Nov 22, 2019

taldcroft commented Nov 22, 2019

jeanconn commented Nov 22, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019 •

edited

Loading

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 26, 2019

jeanconn commented Nov 26, 2019

taldcroft commented Nov 26, 2019

jeanconn commented Nov 26, 2019

jeanconn commented Nov 26, 2019

jeanconn Nov 27, 2019

taldcroft Nov 27, 2019

WIP windows conda build for ska3-win #227

WIP windows conda build for ska3-win #227

Conversation

taldcroft commented Nov 22, 2019 • edited Loading

jeanconn commented Nov 22, 2019

taldcroft commented Nov 22, 2019

taldcroft commented Nov 22, 2019

jeanconn commented Nov 22, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019 • edited Loading

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 23, 2019

jeanconn commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

taldcroft commented Nov 23, 2019

jeanconn commented Nov 26, 2019

jeanconn commented Nov 26, 2019

taldcroft commented Nov 26, 2019

jeanconn commented Nov 26, 2019

jeanconn commented Nov 26, 2019

jeanconn Nov 27, 2019

Choose a reason for hiding this comment

taldcroft Nov 27, 2019

Choose a reason for hiding this comment

taldcroft commented Nov 22, 2019 •

edited

Loading

jeanconn commented Nov 23, 2019 •

edited

Loading