Add scripts/get-shapefiles.py

[Ircama]

[Edited summary for this PR, left out on my first push.
Edited the command line options to reflect the merged commit]

The aim of this PR is to replace get-shapefiles.sh with a Python version named get-shapefiles.py and positioned in the scripts directory of openstreetmap-carto.

It generates and populates the data directory with all needed shapefiles, including indexing them through shapeindex. Related features are the same as the original shell script, with the addition of some command line options.

The code exploits an internal dictionary for easy adaptation to future changes, supports Windows, UNIX, Python 2.7, Python 3 and curl when installed (even if curl is not required, for more comfortable Windows support). Installation of shapeindex is a prerequisite to index files, but is not strictly required if -s option is used (old indexes are not automatically removed; it is better to manually remove indexes from the subfolders when updating the data directory without updating indexes).

The script can be run from the scripts directory of openstreetmap-carto, or from its base folder, or through absolute path from any directory. It should allow a default Web proxy and also the configuration of the HTTP_PROXY environment variable. With Windows, PAC and NTLM authentication are not supported.

Documentation has been updated accordingly.

Resolves #43
Resolves #66
Resolves #1233

Basic invocation: scripts/get-shapefiles.py

Command-line options: get-shapefiles.py [-h] [-c] [-d <directory name>] [-e] [-f] [-l] [-n]
                         [-p] [-r] [-s] [-u] [--world-boundaries]
                         [--simplified-land] [--ne-admin] [--land-polygons]
                         [--icesheet-polygons] [--icesheet-outlines]

optional arguments:
  -h, --help            show this help message and exit
  -c, --check           check whether the 'data' directory already exists
  -d <directory name>, --directory <directory name>
                        set the name of the data directory (default: 'data')
  -e, --no-extract      do not populate target directories with the expansion
                        of downloaded data
  -f, --force           force continuing even if project.mml does not exist
  -l, --no-curl         do not use 'curl' even if available
  -n, --no-download     do not download archive if already existing locally
  -p, --pause           pause before starting
  -r, --remove          remove each downloaded archive after its expansion
  -s, --no-shape        do not run shapeindex
  -u, --update          force downloading files even if not newer than the
                        locally existing ones
  --world-boundaries    only process world_boundaries
  --simplified-land     only process simplified-land-polygons-complete-3857
  --ne-admin            only process ne_110m_admin_0_boundary_lines_land
  --land-polygons       only process land-polygons-split-3857
  --icesheet-polygons   only process antarctica-icesheet-polygons-3857
  --icesheet-outlines   only process antarctica-icesheet-outlines-3857

[pnorman]

We aren't creating a bin directory, so this should just be shapeindex

[pnorman]

I wouldn't mind replacing get-shapefiles.sh with a python version, but not adding a second version which is to do the same thing. Switching to a python version would make it easier to download on Windows.

[HolgerJeromin]

It would be nice to have a script which is 2.7 and 3 compatible.
But this can be done later.
http://python3porting.com/differences.html seems to be a good resource for that (But my last porting was a year ago).

[Ircama]

@pnorman: In fact I am trying to manage implications of contributing to openstreetmap-carto through a Windows installation of Tilemill. One aspect that needed rewriting was get-shapefiles.sh

[pnorman]

One of the important features of get-shapefiles.sh is you can re-run it to update

[Ircama]

Added command line arguments.

Run it with scripts\get-shapefiles.py

To allow re-run it to update, use get-shapefiles.py -qu
To get help, get-shapefiles.py -h

usage: get-shapefiles.py [-h] [-q] [-u] [-n] [-e] [-s] [-1] [-2] [-3] [-4]
                         [-5] [-6]

This script generates and populates the 'data' directory with all needed
shapefiles.

optional arguments:
  -h, --help        show this help message and exit
  -q, --quiet       don't print status messages to stdout
  -u, --update      allow updating an already existing 'data' directory
  -n, --nodownload  do not download files if already existing locally
  -e, --noextract   do not populate target directories with downloaded data
  -s, --noshape     do not run shapeindex
  -1                only process world_boundaries
  -2                only process simplified-water-polygons-complete-3857
  -3                only process ne_110m_admin_0_boundary_lines_land
  -4                only process water-polygons-split-3857
  -5                only process antarctica-icesheet-polygons-3857
  -6                only process antarctica-icesheet-outlines-3857

[gravitystorm]

How hard is it to make the script do the following:

• Work on both python 2 and 3
• Only use modules from the standard library

Also, I can see there is inconsistencies in both variable naming (all lower (nodownload), snakecase (update_mode), other (Owps)) and indentation. I'm not a python programmer but best to go with whatever the python conventions are.

Finally, if this is a replacement for get-shapefiles.sh, then the PR should include removing the old version.

[Ircama]

@gravitystorm: OK, I'll try in the next days

[pnorman]

variables like --world-boundaries, --simplified-water, --ne-admin, etc would be easier to remember than numerics

[tilmanb]

option_aip controls the call to icesheet-outlines.
option_aio (the next one) controls the ne110m admin boundaries.
I guess this is not intended.

[Ircama]

@tilmanb: Thanks, I will correct this and will also change the water acronyms to land

[pnorman]

I found the following issues running it

• The documentation needs updating to refer to the new name
• The script needs mode +x

I get the error

$ scripts/get-shapefiles.py

Shapeindex is found and has
   Error. [Errno 2] No such file or directory

• Updating should be the default and not require confirmation
• When I run the script with scripts/get-shapefiles.py -su and rerun it, it downloads all the files again. It should only download those which have changed. This takes about 15 seconds for me. Curl uses the -z option for this.
• The status update is substantially less useful than curl's. This isn't essential, but it'd be nice.

Code and data are intermixed and there's a lot of repeated similar code. The shell script has this too, but it's only 86 lines. I suggest moving the definitions of the files and their locations to the top in an array, dict, or other structure and having code which loops over that.

[Ircama]

@pnorman: thanks for the feedbacks, I will setup a UNIX VM and test it there.

OK for the updates you asked.

I did not use curl to simplify the Windows setup.

The documentation needs updating to refer to the new name

Would you refer to the Scripted download section in INSTALL.md or also something else?

[pnorman]

Would you refer to the Scripted download section in INSTALL.md or also something else?

Anywhere that get-shapefiles.sh is mentioned.

[matthijsmelissen]

@Ircama Is this ready for review?

[Ircama]

@math1985 yes, thanks.

[matthijsmelissen]

@pnorman Do you have time to review this?

[pnorman]

Yes, later this week.

[Ircama]

Memo for me: there is some change to be done to comply to PEP8 style guide. Will do ASAP with a new commit. Also a documentation typo needs fixing.

[pnorman]

I still need to do a code review on this.

[Ircama]

[Unwanted push, just rebased by now]

[Ircama]

@pnorman: code now ready for review, thanks

[Ircama]

Rebased

[Ircama]

Resolves #43

[kocio-pl]

I guess this script resolves also #66 and #1233?

[Ircama]

Rebased so that it can be easily reviewed/merged

[Ircama]

Newly rebased

Resolves #66
Resolves #1233

[HolgerJeromin]

If you place the "resolves" text in the first message of this PR the issues will be closed automatically.

[kocio-pl]

I think you should also delete get-shapefiles.sh script in this PR.

[Ircama]

If you place the "resolves" text in the first message of this PR the issues will be closed automatically.

@HolgerJeromin, @kocio-pl: thanks, I thought that any place within the PR for a comment was enough for the auto closure.

I think you should also delete get-shapefiles.sh script in this PR.

Yes, thanks a lot. I have just fixed it with a new commit.

[kocio-pl]

@pnorman Any problems left with this PR?

[pnorman]

@pnorman Any problems left with this PR?

the -1 to -6 short options.

Aside from that, it seems to work.

I don't like the gain in code complexity, but it's probably worth it for windows support in the download script.

[Ircama]

@pnorman - OK no problem; would you like to have the short options removed (keeping the long ones), or both the short and long options removed (with reference to -1 to -6)?

Just one note, as I see many sites mentioning the bash script, I’d like to double check that its removal is wanted.

[pnorman]

@pnorman - OK no problem; would you like to have the short options removed (keeping the long ones), or both the short and long options removed (with reference to -1 to -6)?

The long options are good to have.

Just one note, as I see many sites mentioning the bash script, I’d like to double check that its removal is wanted.

We don't want to maintain two download scripts.

[nebulon42]

I'm currently having problems with shapeindex on my Ubuntu 16.04. It tries to load the shared libraries of Ubuntu 14.04. So I can't test this right now.

I also think the short options -1 .. -6 are not the best choice.

[kocio-pl]

@Ircama Double check is here, so do you plan to remove the short options? I would like to merge this PR soon, if possible.

[Ircama]

Code updated as requested.

I meanwhile noticed that, when using -s to disable shapeindex (e.g., useful in case shapeindex is not properly installed), old indexes are not automatically removed. I did not have time to add this purge operation in the code now (and perform again all testing with all OSs); could be part of a future update. By now, it is better to manually remove indexes from the subfolders when updating the data directory without updating indexes. (Notice that running shapeindex is the default mode and also the suggested one.)

[kocio-pl]

I just made some basic testing with Python 2/3 and shapeindex/--no-shape under Linux and it works.

I think that 6 months for one script is long enough and fine tuning can be done later. In the meantime we lack portability and the code waiting too long in the queue may need to be keep in sync, which is additional problem. Thanks for your code and patience, @Ircama!