Longitude and Latitude not required for projected coordinates

[marqh]

As discussed in CF meeting 20th June 2018 in Reading, UK

The requirement that if x and y spatial coordinates are supplied as projected coordinates then longitude and latitude coordinates must also be provided is a major source of concern for many data creators and leads to significant file size increases.

The relaxation of this requirement is backwards compatible and would make CF easier to use for many data creators.

This Pull Request includes test to relax the requirement.

A change to the conformance rules and checker would also be required

fixes #179

[marqh]

@JonathanGregory @ajelenak-thg @ethanrd

[erget]

Thanks for this Mark. @ajelenak-thg I think it would be a good idea to modify the note in the Swath proposal which discusses this and provide an example of specifying coordinates for FORs but not for the individual FOVs. Be happy to cook up something appropriate.

[davidhassell]

Hello Mark,

I don't think the new text quite works, as it allows for the possibility of providing projection coordinates without either a grid mapping nor 2-d latitudes and longitudes; in which case the coordinate system is not sufficiently described.

If this proposal were to work, a grid mapping must be mandatory if the lats and lons were missing.

I have understood the purpose of providing the lats and lons to be that calculating them on the fly from the projection coordinates and grid mapping parameters was too much to expect of the dataset user (rather than the dataset creator). Perhaps this is no longer the case? If so, we should note this.

I think that this issue is not independent from external_variables - allowing auxiliary coordinates to be external might, in some circumstances, allow the 2-d lats and lons to remain mandatory whilst addressing the file size issue.

[ghost]

I am in favor of maximum flexibility for data creators. They should produce data with geospatial coordinates that are most relevant to their users. If those are not lat-lon they should not be forced by the convention to include them in their datasets as well.

I also think that geospatial coordinates should always declare their coordinate reference system (datum and projection when applicable) but this is more of a best practice. The convention should probably require the use of grid mapping for all geospatial coordinates that are not simple lat-lon.

[cf-metadata-list]

Aleksander: It is extremely unlikely the satellite ground systems implementer/maintainers will ever agree to coordinate data for every pixel because of the product file sizing issue it creates not just for them, but the weather data related ground enterprises. Thus, we will not be able to bring JPSS , WSF, or METOP data sets under the fold of CF. v/r randy

…

On Jul 25, 2018, at 12:40 PM, Aleksandar Jelenak ***@***.***> wrote: I am in favor of maximum flexibility for data creators. They should produce data with geospatial coordinates that are most relevant to their users. If those are not lat-lon they should not be forced by the convention to include them in their datasets as well. I also think that geospatial coordinates should always declare their coordinate reference system (datum and projection when applicable) but this is more of a best practice. The convention should probably require the use of grid mapping for all geospatial coordinates that are not simple lat-lon. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AfI2gWA_iGDY_CecnPYjW01weL0YNTdCks5uKJ-HgaJpZM4UvWGl>.

_____________________________________ Randy C Horne (rhorne@excaliburlabs.com) Principal Engineer, Excalibur Laboratories Inc. voice & fax: (321) 952.5100 cell: (321) 693.1074 url: http://www.excaliburlabs.com

[hrajagers]

@ajelenak-thg For all real world applications I agree with your statement that "The convention should probably require the use of grid mapping for all geospatial coordinates that are not simple lat-lon." The only common exception that I know of is results of theoretical validation cases which are purely x-y based without any reference to the specific location on the earth. This frequently also holds for lab validation data which is referenced typically only to the geometry of the flume. It would be nice if we could flag such cases one way or the other without a CF requirement to place it explicitly somewhere on the earth.

[ghost]

Fair point, @hrajagers. Would such cases be covered by missing grid mapping information? No grid mapping simply implies that users should interpret such geospatial coordinates in any way they find appropriate.

[ghost]

Hi Randy,

You are talking about another issue. We'll try to tackle it after this one is settled.

[dblodgett-usgs]

Dear all,

I think this is the sentence that we need to work on?

A grid mapping variable is required if, in addition, it is desired to describe the mapping between the given coordinate variables and the true latitude and longitude coordinates, or to describe the figure of the Earth used to define the latitude and longitude coordinates, or to describe another coordinate reference system definition used by some coordinates or auxiliary coordinates.

A change to:

A grid mapping variable is required if the given coordinate variables are relative to a planetary latitude longitude datum, or to describe the figure of the Earth used to define the latitude and longitude coordinates, or to describe another coordinate reference system definition used by some coordinates or auxiliary coordinates.

planetary latitude longitude datum is intended to take care of @hrajagers use case of x/y coordinates that are not relative to the earth (or mars!).

This brings up a long standing beef I have with this. I would also like to suggest this change:

or to describe the figure of the Earth used to define the latitude and longitude coordinates

or to describe the latitude and longitude datum in the case that assumption of any arbitrary datum could lead to misinterpretation of the data,

And, to be honest, I don't know what the third clause of this sentence means. Could it be dropped?

Best,

• Dave

p.s. @ajelenak-thg and Randy, I actually think the whole point of this is so data providers DON'T have to provide coordinate data for every pixel. This will allow us to publish grid mappings that express the projected coordinate systems our data are represented in.

Further, I kind of take issue with this being posed as a data provider/data consumer dichotomy. IMHO, data with 2/d lat/lon arrays that expand to many gigabytes when put in memory become literally unusable. e.g. It doesn't matter if you are a data provider or consumer if the 2/d lat lon is too big to work with the requirement can't be followed anyways.

[steingod]

A grid mapping variable is required if the given coordinate variables are relative to a planetary latitude longitude datum, or to describe the figure of the Earth used to define the latitude and longitude coordinates, or to describe another coordinate reference system definition used by some coordinates or auxiliary coordinates.

A modification like this would be very beneficial and the phrasing above reads well for me.

[ghost]

@dblodgett-usgs This issue is about relaxing the requirement to always provide (planetary) lat-lon coordinates in addition to any other kind of geospatial coordinates already present in the file. My understanding of what Randy asked for, and you seem to support, is what I would describe as subsampled coordinates of any kind. That's a different issue from this one. I am in favor of first completing this issue and then moving on to that one.

[dblodgett-usgs]

Oh. I see. Subsampled or otherwise not completely characterized coordinates such as origin/offset is a separate issue. I misunderstood.

[erget]

As I interpret @marqh 's suggestion, the point is to loosen the requirement for coordinates. The goal is that coordinates for every data point in a data set should be reconstructable, if not explicitly encoded in the data.

To be honest, I think we might be able to do this with the existing text without endangering data integrity. For the use case in question, satellite producers (such as EUMETSAT) would like to be able to:

• Provide coordinates for groupings of views, called fields of regard (FORs). These FORs contain fields of view (FOVs) which represent individual data points.
• Provide enough information to be able to reconstruct the position of each FOV within the FOR, without providing coordinates for each FOV. This would save a lot of space.

Now to the actual requirements (I'm adding emphasis):

• CF 1.7, 5.6, paragraph 1, sentence 1: "When the coordinate variables for a horizontal grid are not longitude and latitude, it is required that the true latitude and longitude coordinates be supplied via the coordinates attribute." If we understand our horizontal grid to refer to the grid of FORs, this is already fulfilled. There's no requirement that every data point have coordinates here.
• CF 1.7, 5, paragraph 2, sentence 1: "Any of a variable's dimensions that is an independently varying latitude, longitude, ... dimension ... must have a corresponding coordinate variable." Here also, we could be weasley with our satellite data by claiming that the FOVs vary dependently of the FORs, and therefore they don't need their own coordinate variable.

Please correct me if this is overly naive, but maybe the better approach would be to provide an example in one of the appendices, rather than change the text of the convention.

[marqh]

in response to #133 (comment)

@erget I don't think that this is sufficient. The satellite case with FOVs and FORs only represents a subset of the problem cases here.

By mandating the provision of latitude and longitude coordinates and providing the grid mapping definition capability, CF is forcing data producers to define detailed information about position twice in many cases.

For example, a simple model grid using the UK's OSGB Transverse Mercator CRS is fully defined by the OSGB parameters, a 1D array of eastings and a 1D array of northings.
However, to meet the CF conformance requirements, then a 2D array of latitudes and a 2D array of longitudes must be provided. If only 1 data variable is provided at the surface, a common use case, then this ~triples the size of the file.
Pretty much any spatially aware application would be able to use the OSGB coordinates and CRS definition to geo-locate this data set, so the latitude and longitude arrays are really not required.

I think it is a better approach to remove the mandate that latitude and longitude values will always exist within the file.

[marqh]

in response to #133 (comment)

@davidhassell

If this proposal were to work, a grid mapping must be mandatory if the lats and lons were missing.

I would not object to an either | or condition here.

I have understood the purpose of providing the lats and lons to be that calculating them on the fly from the projection coordinates and grid mapping parameters was too much to expect of the dataset user (rather than the dataset creator). Perhaps this is no longer the case? If so, we should note this.

It is my assertion that this is not the case. I don't mind if this is deemed worthy of a note

I think that this issue is not independent from external_variables - allowing auxiliary coordinates to be external might, in some circumstances, allow the 2-d lats and lons to remain mandatory whilst addressing the file size issue.

I think that the issue of external variables for auxiliary coordinates is likely to take some time to resolve.
If it were resolved to allow this, it would still put an onus on the data producer to provide all of these latitude longitude values somehow.

The assertion here is that they are not required, they are fully calculable by a wide range of software, and so provision should not be mandated.

That is the position I am adopting within the scope of this ticket

[erget]

Regarding #133 (comment) from @marqh - agreed, I had been looking at it from my somewhat narrowed perspective at EUMETSAT HQ, but you're absolutely right about model data. I'm familiar with many other users who would agree that it's redundant to supply coordinates for all of those.

Regarding #133 (comment), I like either | or as well - the point is that coordinates can be reconstructed, supplying them for gridded data would be more the edge case IMHO. This is in line with the way the rest of the geospatial community does things.

So I would agree with this ticket if as @davidhassell suggests a grid mapping is mandatory if coordinates are not supplied. This way we'd have a guarantee that the data within a CF file could be geolocated. The point is that the user must have some way of reconstructing the positions of the geophysical quantities in space and time.

[marqh]

I have updated the proposed text change to include the conditional requirement that either a grid mapping variable is referenced, or latitude and longitude coordinates are provided.

[marqh]

@hrajagers
I agree that there is a useful case that should be able to be handled here.
As CF now recognises CRS WKT for CRS definitions, then I think this will be neatly handled with the proposed change. CRS WKT provides the capability to define an engineering CRS, independent of the earth or a similar body.

Thus, to meet the requirements in the case where the coordinates are not geographic in nature, specify a grid mapping to a CRS definition in WKT that describes an Engineering CRS.

Is this a reasonable implementation from your point of view?

[hrajagers]

@marqh It's a non-trivial way of specifying that you're in an arbitrary coordinate system, but it might be sufficient. However, for this to work, I would suggest to give some example (and/or references) to build on. The most accessible examples of Engineering CRS that I've found so far are given at http://docs.opengeospatial.org/is/12-063r5/12-063r5.html#78. For a general theoretical x/y coordinate system I believe we should then provide a WKT as something like

 EDATUM[“Arbitrary Cartesian Reference System",ANCHOR[“Reference Point”]],
 CS[Cartesian,2],
   AXIS[“projection_x”,east,ORDER[1]],
   AXIS[“projection_y”,north,ORDER[2]],
   LENGTHUNIT[“metre”,1.0]]

Hm, even this WKT string suggests that X axis is pointing East and Y axis is point North which may be unknown or irrelevant. Not sure, whether we can make it even more generic and neutral.

[dopplershift]

This text in the start of chapter 5 should also be updated:

If the coordinate variables for a horizontal grid are not longitude and latitude, it is recommended that they be supplied in addition to the required coordinates. For example, the Cartesian coordinates of a map projection should be supplied as coordinate variables in addition to the required two-dimensional latitude and longitude variables that are identified via the coordinates attribute. The use of the axis attribute with values X and Y is recommended for the coordinate variables (see Chapter 4, Coordinate Types).

And regardless of whether this proposal goes in, the first sentence needs to be clarified--"they" in that sentence is meant to refer to "coordinate variables", but grammatically they corresponds to "longitude and latitude".

It's also a little odd that the paragraph refers to "required coordinates" before, as far as I can tell, there is any mention of coordinates being required earlier in the document.

[JimBiardCics]

@dopplershift I think the "they" and the "required coordinates" actually refer to "X and Y" and "longitude and latitude", respectively. I read the existing CF conventions as requiring longitude and latitude. In the case where the data is on a projected CRS system, 1D X and Y coordinate variables are optional. The paragraph you included in your comment is saying that it is a good thing to include X and Y along with the required longitude and latitude.

[dopplershift]

@JimBiardCics I agree, that's what the paragraph is intending to say. However, the antecedent of the pronoun "they" in that sentence is "longitude and latitude"--hence the confusion.

My other point is that this is the first reference to any kind of requirement for latitude and longitude in the entire document. That, IMO, is also a problem.

[JimBiardCics]

@dopplershift Latitude and longitude are first mentioned at the beginning of Section 4. It's there that they are declared as quasi-required. There are also numerous mentions of lat and lon in the beginning of Section 5.

[marqh]

@dopplershift
thank you for highlighting the extra text at the start of Chapter 5
#133 (comment)
which I agree also needs addressing

I have drafted an update to this text, in
4df9d18
which is included in this PR

[JimBiardCics]

A grid mapping variable may be referenced by a data variable to describe the mapping between the

[marqh]

I prefer to keep some punctuation, as ths is a long sentence with a number of clauses

[JimBiardCics]

@marqh That comma is grammatically wrong.

[JimBiardCics]

I think we should get rid of the word 'true' here and elsewhere. It is confusing.

[marqh]

i agree

[JimBiardCics]

What is the intent of the 'if required' phrase?

[marqh]

Only that not all applications or users require a latitude longitude pair for each grid location, but that you can derive them using numerous well provisioned software tools if you want to.
If you'd rather it was removed, as superfluous, then I am content to do that.

[erget]

As much as I find the idea of using CF on other planetary bodies cool, we may want to tackle that - if a need is identified - in a separate issue. The phrasing "(or other body)" doesn't really align with terminology used elsewhere in the document, where we talk about geolocation, etc. If we're using a planetocentric or planetographic coordinate system a lot of the assumptions that you can use in Earthbound CRS don't apply. I don't believe this is insurmountable but would argue that it is deserving of further attention outside of this issue if we want to address it.

[marqh]

agreed. This was an issue for another activity i worked on recently, but not relevant here; I think I inserted it by habt, not design

[marqh]

An additional note on Conformance.

The conformance document,
(currently https://github.com/cf-convention/Conformance/blob/master/conformance.adoc#56-grid-mappings-and-projections though due to be moved into this repository, I believe)
does not have any mandate for this condition in the text, that I can see.
Although Chapter 5 makes it clear that there is a mandate, I cannot see a corresponding element within the conformance document for this condition, which we are proposing to relax.
Therefore, no change to the conformance document is currently proposed.

[JimBiardCics]

I think this sentence doesn't represent the intent clearly. How about this language for lines 190-197?

A grid mapping variable may be referenced by a data variable in order to explicitly declare the coordinate reference system (CRS) used for the spatial coordinate values. For example, if the spatial coordinates are latitude and longitude, the grid mapping variable can be used to declare the figure of the earth (WGS84 ellipsoid, sphere, etc.) they are based on. If the spatial coordinates are easting, northing, and elevation in a map projection, the grid mapping variable declares the map projection CRS used and provides the information needed to calculate latitude and longitude from easting and northing. If the spatial coordinates are not georeferenced coordinates, the grid mapping variable can make it clear that this is the case.

If the spatial coordinates are not longitude and latitude and no grid mapping variable is provided, then it is recommended that auxiliary latitude and longitude coordinates be included in the file and referenced via the coordinates attribute. These auxiliary coordinate variables provide the latitude and longitude for each element of the data variable.

[marqh]

Hello @JimBiardCics

If the spatial coordinates are not georeferenced coordinates, the grid mapping variable can make it clear that this is the case.

I'm wary of this statement. I'm not sure that it is possible to make it clear that this is the case.

If the spatial coordinates are not longitude and latitude and no grid mapping variable is provided, then it is recommended that auxiliary latitude and longitude coordinates be included in the file and referenced via the coordinates attribute.

This is a content change, not just a rewording. In the current standard and this proposed change, this is mandated, not recommended, I believe. Is it your intent to change this mandate to a recommends, or did you intend this to remain a mandate?

thank you
marqh

[JimBiardCics]

@marqh The non-georeferenced case is described by @hrajagers in an earlier comment to this PR.

Regarding the content change you noted: I'm OK with making it mandatory to include lat and lon if you haven't otherwise declared the CRS for a pair of X and Y coordinate variables.

[marqh]

thank you for the feedback @JimBiardCics

I agree that the multi-phrase sentence at line 190 is convoluted; I have adopted you suggested text for this.
I don't yet agree that the non-georeferenced case is high profile and should be used as an example. Indeed the earlier comment relies on WKT-CRS, which is at the moment an extension to grid_mappins, and for experts, I suggest.

I have kept the previous text on the conditional mandate of either a grid_mapping, or explicit latitude and longitude, as I think it is clearer.

your continued feedback on this text is very much welcome
marqh

[JimBiardCics]

@marqh The non-georeferenced case can wait until someone specifically requests it, but it is a valid use case that was mentioned in the threads on this change. Let's leave it out if it is going to slow down progress on this change.

[dblodgett-usgs]

Given that we have two reviewers approving this, as the moderator of the proposal, I'm going to start the three week clock on dissent. If we have no objection to unanimous consent after that time, I'll merge.

[JonathanGregory]

Dear Dave
I think this notice and the decision about making this change should be made in the issue, not in this pull request, following the guidelines we've been discussing. Regarding the countdown to acceptance, the rules say, "When three weeks have passed with no contributions following a summary, and providing a test file exists (if appropriate), a decision is reached in one of the following ways: Consensus: Accept the proposal if there is no outstanding objection, and at least three contributors have expressed support for it, including at least two members of the conventions committee. If the moderator personally expresses support, he or she can be counted among the supporters." (http://cfconventions.org/rules.html)
Best wishes, Jonathan

[dblodgett-usgs]

Dear Jonathan -- thanks for the reminder of the details here.

@marqh -- would you be willing to work up a top-level summary over in the issue?

[marqh]

@davidhassell
regarding: #179 (comment)

I have taken the points on board. I have attempted to incorporate these into the existing text flow, which I find a bit clearer and more closely related to the previous text.
Please may you review the latest text and let us know whether the changes made, to explicitly state 'horizontal' and to explicitly stress that both explicit coordinates and grid mappings can be provided meet your requirements?

thank you
marqh

[marqh]

@marqh -- would you be willing to work up a top-level summary over in the issue?

of course, please see
#179 (comment)
and
#179 (comment)

[dblodgett-usgs]

With no further discussion on this for three weeks, I'm going to go ahead and merge.