Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to NE 5.1.2 - Add Config for new POVs #2078

Merged
merged 17 commits into from
May 17, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 38 additions & 32 deletions data/assets.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
bucket: nextzen-tile-assets
datestamp: 20220426
datestamp: 20220429
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIt: but 5.1.0 was only officially released on May 4th, so bring this forward?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to most recent (which I think @jeffdefacto found an issue in for the iso countries.zip)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That zero length char definition is really odd. Opens fine in QGIS but PostGIS barfs on it (saying it doesn't support 0 length varchar fields).

Filed mbloch/mapshaper#541 upstream.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With mbloch/mapshaper#541 fixed upstream, I'll update Natural Earth to v5.1.1 with the fix.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Natural Earth 5.1.1 is now out, please update here and see if the problem persists.


shapefiles:

Expand Down Expand Up @@ -55,98 +55,104 @@ shapefiles:
url: http://s3.amazonaws.com/tilezen-assets/curated/admin_areas_20180409.zip

- name: ne_110m_lakes
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_lakes.zip
url: https://naciscdn.org/naturalearth/5.1.0/110m/physical/ne_110m_lakes.zip
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching these to versioned CDN links... much more obvious which version we're on and makes updating NE a deliberate process.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. Much clearer

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These will all need to switch to 5.1.1 to pick up the other change.


- name: ne_50m_lakes
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/physical/ne_50m_lakes.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/physical/ne_50m_lakes.zip

- name: ne_10m_lakes
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_lakes.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/physical/ne_10m_lakes.zip

- name: ne_110m_ocean
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_ocean.zip
url: https://naciscdn.org/naturalearth/5.1.0/110m/physical/ne_110m_ocean.zip

- name: ne_50m_ocean
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/physical/ne_50m_ocean.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/physical/ne_50m_ocean.zip

- name: ne_10m_ocean
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_ocean.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/physical/ne_10m_ocean.zip

- name: ne_50m_playas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/physical/ne_50m_playas.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/physical/ne_50m_playas.zip

- name: ne_10m_playas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_playas.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/physical/ne_10m_playas.zip

- name: ne_50m_urban_areas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_urban_areas.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/cultural/ne_50m_urban_areas.zip

- name: ne_10m_urban_areas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_urban_areas.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_urban_areas.zip

- name: ne_110m_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/110m/physical/ne_110m_land.zip

- name: ne_50m_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/physical/ne_50m_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/physical/ne_50m_land.zip

- name: ne_10m_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/physical/ne_10m_land.zip
tile: true

- name: ne_10m_populated_places
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_populated_places.zip

- name: ne_110m_admin_0_boundary_lines_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip

- name: ne_50m_admin_0_boundary_lines_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_0_boundary_lines_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/cultural/ne_50m_admin_0_boundary_lines_land.zip

- name: ne_10m_admin_0_boundary_lines_land
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_boundary_lines_land.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_boundary_lines_land.zip

- name: ne_10m_admin_0_boundary_lines_map_units
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_boundary_lines_map_units.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_boundary_lines_map_units.zip

- name: ne_50m_admin_1_states_provinces_lines
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_1_states_provinces_lines.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/cultural/ne_50m_admin_1_states_provinces_lines.zip

- name: ne_10m_admin_1_states_provinces_lines
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_1_states_provinces_lines.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_1_states_provinces_lines.zip

- name: ne_50m_admin_0_boundary_lines_disputed_areas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_0_boundary_lines_disputed_areas.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/cultural/ne_50m_admin_0_boundary_lines_disputed_areas.zip

- name: ne_50m_admin_0_boundary_lines_maritime_indicator_chn
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_0_boundary_lines_maritime_indicator_chn.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/cultural/ne_50m_admin_0_boundary_lines_maritime_indicator_chn.zip

- name: ne_10m_admin_0_boundary_lines_disputed_areas
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_boundary_lines_disputed_areas.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_boundary_lines_disputed_areas.zip

- name: ne_10m_admin_0_boundary_lines_maritime_indicator_chn
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_boundary_lines_maritime_indicator_chn.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_boundary_lines_maritime_indicator_chn.zip

- name: ne_10m_roads
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_roads.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_roads.zip

- name: ne_110m_coastline
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_coastline.zip
url: https://naciscdn.org/naturalearth/5.1.0/110m/physical/ne_110m_coastline.zip

- name: ne_50m_coastline
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/physical/ne_50m_coastline.zip
url: https://naciscdn.org/naturalearth/5.1.0/50m/physical/ne_50m_coastline.zip

- name: ne_10m_coastline
url: http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_coastline.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/physical/ne_10m_coastline.zip

- name: ne_10m_admin_0_countries
url: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_countries.zip

- name: ne_10m_admin_0_countries_iso
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_countries_iso.zip

- name: ne_10m_admin_0_countries_tlc
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_countries_tlc.zip

- name: ne_10m_admin_0_map_units
url: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_map_units.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_0_map_units.zip

- name: ne_10m_admin_1_states_provinces
url: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_1_states_provinces.zip
url: https://naciscdn.org/naturalearth/5.1.0/10m/cultural/ne_10m_admin_1_states_provinces.zip

wikidata-queries:
- name: aerodrome_passenger_count
Expand Down
95 changes: 78 additions & 17 deletions data/functions.sql
Original file line number Diff line number Diff line change
Expand Up @@ -996,7 +996,7 @@ $$ LANGUAGE plpgsql IMMUTABLE;
-- returns a JSONB object containing __ne_min_zoom and __ne_max_zoom set to the
-- label min and max zoom of any matching row from the Natural Earth countries,
-- map units and states/provinces themes.
CREATE OR REPLACE FUNCTION tz_get_ne_min_max_zoom(wikidata_id TEXT)
CREATE OR REPLACE FUNCTION tz_get_ne_min_max_zoom(wikidata_id TEXT, place_tag TEXT)
RETURNS JSONB AS $$
DECLARE
min_zoom REAL;
Expand All @@ -1006,28 +1006,30 @@ BEGIN
RETURN '{}'::jsonb;
END IF;

-- first, try the countries table
SELECT
min_label, max_label INTO min_zoom, max_zoom
FROM ne_10m_admin_0_countries c
WHERE c.wikidataid = wikidata_id;
-- if it's a country, only look it up in the iso and tlc tables
IF place_tag='country' OR place_tag='unrecognized' THEN
SELECT
min_label, max_label INTO min_zoom, max_zoom
FROM ne_10m_admin_0_countries_iso i
WHERE i.wikidataid = wikidata_id;

-- if that fails, try map_units (which contains some sub-country but super-
-- state level stuff such as England, Scotland and Wales).
IF NOT FOUND THEN
SELECT
min_label, max_label INTO min_zoom, max_zoom
FROM ne_10m_admin_0_map_units mu
WHERE mu.wikidataid = wikidata_id;
IF NOT FOUND THEN
SELECT
min_label, max_label INTO min_zoom, max_zoom
FROM ne_10m_admin_0_countries_tlc t
WHERE t.wikidataid = wikidata_id;
END IF;

IF NOT FOUND THEN
RETURN '{}'::jsonb;
END IF;
END IF;

-- try states and provinces
IF NOT FOUND THEN
nvkelso marked this conversation as resolved.
Show resolved Hide resolved
SELECT
-- try states and provinces if it's not a country
SELECT
min_label, max_label INTO min_zoom, max_zoom
FROM ne_10m_admin_1_states_provinces sp
WHERE sp.wikidataid = wikidata_id;
END IF;

-- finally, try localities
-- There is no concept of max_zoom for ne_10m_populated_places
Expand All @@ -1051,6 +1053,65 @@ BEGIN
END;
$$ LANGUAGE plpgsql STABLE;

CREATE OR REPLACE FUNCTION tz_get_fclass_and_label_position(wikidata_id TEXT, place_tag TEXT)
RETURNS JSONB AS $$
DECLARE
fclass_iso TEXT;
fclass_tlc TEXT;
label_x REAL;
label_y REAL;
BEGIN
IF wikidata_id IS NULL THEN
RETURN '{}'::jsonb;
END IF;

-- if it's a country, only look it up in the iso and tlc tables
IF place_tag='country' OR place_tag='unrecognized' THEN
SELECT
fclass_iso, fclass_tlc, label_x, label_y INTO fclass_iso, fclass_tlc, label_x, label_y
FROM ne_10m_admin_0_countries_iso i
WHERE i.wikidataid = wikidata_id;

IF NOT FOUND THEN
SELECT
fclass_iso, fclass_tlc, label_x, label_y INTO fclass_iso, fclass_tlc, label_x, label_y
FROM ne_10m_admin_0_countries_tlc t
WHERE t.wikidataid = wikidata_id;
END IF;

IF NOT FOUND THEN
RETURN '{}'::jsonb;
END IF;
END IF;

-- There is no label_x and label_y for the non-countries
SELECT
fclass_iso, fclass_tlc, NULL, NULL INTO fclass_iso, fclass_tlc, label_x, label_y
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusingly called: "latitude" (label_y) and "longitude" (label_x) in the NE admin-1 table. Please update to pull them out.

Copy link
Member

@nvkelso nvkelso May 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(so keep the output as label_{x,y} but the input would be longitude, latitude)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 31a5b4a

FROM ne_10m_admin_1_states_provinces sp
WHERE sp.wikidataid = wikidata_id;

-- finally, try localities
IF NOT FOUND THEN
SELECT
fclass_iso, fclass_tlc, NULL, NULL INTO fclass_iso, fclass_tlc, label_x, label_y
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same deal: This is confusingly called: "latitude" (label_y) and "longitude" (label_x) in the NE populated places table. Please update to pull them out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 31a5b4a

FROM ne_10m_populated_places pp
WHERE pp.wikidataid = wikidata_id;END IF;

-- return an empty JSONB rather than null, so that it can be safely
-- concatenated with whatever other JSONB rather than needing a check for
-- null.
IF NOT FOUND THEN
RETURN '{}'::jsonb;
END IF;
RETURN jsonb_build_object(
'__ne_fclass_iso', fclass_iso,
'__ne_fclass_tlc', fclass_tlc,
'__ne_label_x', label_x,
'__ne_label_y', label_y
);
END;
$$ LANGUAGE plpgsql STABLE;


-- returns a JSONB object containing __ne_pop_min and __ne_pop_max
CREATE OR REPLACE FUNCTION tz_get_ne_pop_min_max(wikidata_id TEXT)
Expand Down
2 changes: 1 addition & 1 deletion queries/ne.jinja2
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{% set ne_languages = ['ar', 'bn', 'de', 'el', 'en', 'es', 'fa', 'fr', 'he', 'hi', 'hu', 'id', 'it', 'ja', 'ko', 'nl', 'pl', 'pt', 'ru', 'sv', 'tr', 'uk', 'ur', 'vi', 'zh', 'zht'] %}
{% set ne_viewpoints = ['iso', 'ar', 'bd', 'br', 'cn', 'de', 'eg', 'es', 'fr', 'gb', 'gr', 'id', 'il', 'in', 'it', 'jp', 'ko', 'ma', 'nl', 'np', 'pk', 'pl', 'ps', 'pt', 'ru', 'sa', 'se', 'tr', 'tw', 'ua', 'us', 'vn'] %}
{% set ne_viewpoints = ['iso', 'tlc', 'ar', 'bd', 'br', 'cn', 'de', 'eg', 'es', 'fr', 'gb', 'gr', 'id', 'il', 'in', 'it', 'jp', 'ko', 'ma', 'nl', 'np', 'pk', 'pl', 'ps', 'pt', 'ru', 'sa', 'se', 'tr', 'tw', 'ua', 'us', 'vn'] %}

{% macro ne_common_properties() %}

Expand Down
4 changes: 3 additions & 1 deletion queries/planet_osm_point.jinja2
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,11 @@ SELECT
THEN jsonb_build_object(
'min_zoom', mz_places_min_zoom
) ||
tz_get_ne_min_max_zoom(tags->'wikidata')
tz_get_ne_min_max_zoom(tags->'wikidata', tags->'place')
||
tz_get_ne_pop_min_max(tags->'wikidata')
||
tz_get_fclass_and_label_position(tags->'wikidata', tags->'place')
END AS __places_properties__,

CASE WHEN mz_poi_min_zoom IS NOT NULL
Expand Down
2 changes: 1 addition & 1 deletion vectordatasource/transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -9295,7 +9295,7 @@ def _list_of_countries(value):
# should have an ISO 3166-1 alpha-2 code, so should be 2 ASCII
# latin characters.
candidate = candidate.strip().lower()
if candidate == 'iso' or match('[a-z][a-z]', candidate):
if candidate == 'iso' or candidate == 'tlc' or match('[a-z][a-z]', candidate):
countries.append(candidate)

return countries
Expand Down
1 change: 1 addition & 0 deletions yaml/boundaries.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ global:

- &ne_localized_kind_properties
'kind:iso': {col: fclass_iso }
'kind:tlc': {col: fclass_tlc }
'kind:ar': {col: fclass_ar }
'kind:bd': {col: fclass_bd }
'kind:br': {col: fclass_br }
Expand Down
2 changes: 2 additions & 0 deletions yaml/places.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ global:
wikidata_id: {col: wikidata}
disputed_by: {col: disputed_by }
'place:ISO': { col: 'place:ISO' }
'place:TLC': { col: 'place:TLC' }
'place:AR': { col: 'place:AR' }
'place:BD': { col: 'place:BD' }
'place:BR': { col: 'place:BR' }
Expand Down Expand Up @@ -83,6 +84,7 @@ global:

- &alternate_fclass
fclass_iso: { col: fclass_iso }
fclass_tlc: { col: fclass_tlc }
fclass_ar: { col: fclass_ar }
fclass_bd: { col: fclass_bd }
fclass_br: { col: fclass_br }
Expand Down