Nav subcommand

[johannessen]

Replacing all waypoints on the device with --flash --erase already works fine. Appending new waypoints without --eraseing the existing data in the device will be added soon-ish.

Routes are also supported, but round-trip HX870→GPX→HX870 doesn’t work so well, as already discussed in #30 (comment).

[cr]

round-trip HX870→GPX→HX870 doesn’t work so well

Please correct me if I'm wrong, I am writing this from memory:

The device contains two databases, a database of waypoints, and a database of routes which reference lists of waypoints in the waypoint database.

We need to distinguish two types of waypoints, those which are referenced in routes, let's call them routepoints, and those which are not, let's keep calling those waypoints.

On export, GPX can distinguish these by wrapping waypoints as individual <wpt> elements, and routepoints as <rtep> elements in an <rte> route definition.

GPX import then could have two modes, maybe:

• Dumb: just add the new waypoints and route points to the existing waypoint database on device and reference those in tracks, never mind de-duplication.
• Clever: read waypoints from device, cross-reference those with new waypoints and routepoints for de-duplicated merged list, and use merged list for referencing in tracks, replace waypoints on device with merged list.

If Dumb were default, activate Clever mode with a --merge option.

If Clever were default, fall back to Dumb mode with --append.

Am I missing something?

[johannessen]

The device contains two databases, a database of waypoints, and a database of routes which reference lists of waypoints in the waypoint database.

That’s correct: Each route is simply a named and ordered list of references to waypoints.

We need to distinguish two types of waypoints, those which are referenced in routes, let's call them routepoints, and those which are not, let's keep calling those waypoints.

On export, GPX can distinguish these by wrapping waypoints as individual <wpt> elements, and routepoints as <rtep> elements in an <rte> route definition.

Fundamentally, the issue is that (in this terminology) routepoints are a subtype of waypoints. In other words: Every routepoint is also a waypoint. It follows that there are two different kinds of routepoints in the HX870:

• Routepoints that only exist because of a route. These may not need a <wpt> element.
• Routepoints that are primarily waypoints, but are also used in a route. These do need a <wpt> in addition to the <rtept>.

Making a distinction between these two is only possible if we know the user’s intention, which we don’t.

Conceptually, I see three options to deal with this issue:

1. A real fix: Automatically reproduce the device’s memory on import to exactly the state it was in before export.
2. A workaround: Allow the user to specify their intention in some way.
3. Don’t deal with it at all (basically “dumb mode only”).

For starters, I went with option 3, simply because it is easiest. I suppose this technically introduces a bug, because route round-trip doesn’t exactly reproduce the device’s nav database, even though it should. But since handling waypoints is much more important than handling routes, I suggest option 3 will be a good enough improvement over the status quo to allow merging this PR once it’s ready for review.

I don’t like option 2, because it makes hxtool unnecessarily difficult to use. Keeping in mind that the user’s intention may well be different for each individual routepoint, I don’t think one simple CLI switch is enough. In fact, it’s not immediately obvious to me that it would even be possible to solve this with CLI params alone.

Option 1 can be achieved in several ways, some of which I’ve already listed in #30 (comment). At this point, I’d tend towards introducing a custom GPX extension to represent the routepoint references (though I don’t yet know if gpxpy would allow us to do that).

The main problem with option 1 would be that any custom metadata that hxtool added to the GPX file might be lost if the user edits it with GUI tools such as QGIS. But in the end, I think such a problem is likely to be outside of the scope of things for hxtool to worry about.

• Clever [mode]: read waypoints from device, cross-reference those with new waypoints and routepoints for de-duplicated merged list, and use merged list for referencing in tracks, replace waypoints on device with merged list.

Right. Removing all duplicate waypoints is one of the other possible approaches for option 1. While it doesn’t give us an exact reproduction of the device’s memory, it’ll likely come close enough in practice.

[cr]

This sounds perfectly reasonable, thanks for taking the time to explain it!

What I don't understand, yet, is

I suppose this technically introduces a bug, because route round-trip doesn’t exactly reproduce the device’s nav database, even though it should.

Could elaborate where the problem lies?

And regarding waypoints and routepoints...

Making a distinction between these two is only possible if we know the user’s intention, which we don’t.

I might be mistaken about the inner workings of popular GPX editors, but I'd expect that on import we may infer the user's intention from whether a <rtep> also appears as a <wpt> or not (and assuming they hardly ever would). On export from the device I agree that we can't, but in which case do we need to distinguish more than whether a waypoint is referenced in a route or not?

Taking a step back, perhaps we agree that our objectives are to have backup capability and to spare the user from the pain of editing waypoints and routes on the device. The main workflows I see so far are:

1. Create a device backup and restore it as perfectly as (sensibly) possible.
2. Create waypoints and routes in an external editor for the next trip, and then replace or append to what is on in the device.
3. Dump state from device, clean it up or amend it in an external editor, and load it back in.

Am I missing something?

Perfect backup/restore can be achieved with a complete config backup, but I think we can make it work via GPX export as well. Looks like all we need is a way to restore the waypoint and route databases in their original order. Ideas here:

• Use <cmt> or <desc> tags for storing waypoint and route database IDs. I'd even expect most software to retain that data after edits for workflow 3 which would be a distinct advantage over working with a proprietary GPX 1.1 extension. The mechanism also provides advanced users with fine-grained control over the on-device database layout while not getting in the way of those who don't know or care.
• We could export each routepoint both as <rtep> and as <wpt>. De-duplication can happen on import. The major disadvantage is that it would clutter the view in external editors, so this should probably not be the default.

As far as I can see, the main decisions we need to make are

• Export routepoints just as <rtep>, or also as <wpt>?
• How to detect and handle duplicates in the waypoint database on import to avoid clutter?

Both could be controlled with relatively simple command line options as long as the default doesn't get in the way of workflow 1.

[johannessen]

I suppose this technically introduces a bug, because route round-trip doesn’t exactly reproduce the device’s nav database, even though it should.

Could elaborate where the problem lies?

Perhaps an example helps clarifying that: Create a GPX file with no <wpt>s and just one simple route connecting two <rtept>s and flash that to the device using the existing code in this branch (if you wanna try it, use --flash --erase, or it won’t work yet). The HX870 will then have two waypoints to support the route (there’s no other way, as you know). Now --dump that data, and you’ll have a GPX file with more waypoints than you did before.

Repeat this often enough, and the user experience will suffer because of all the duplicates. Eventually --flash will fail because the file contains more waypoints that can be stored in the device.

And regarding waypoints and routepoints...

Making a distinction between these two is only possible if we know the user’s intention, which we don’t.

I might be mistaken about the inner workings of popular GPX editors, but I'd expect that on import we may infer the user's intention from whether a <rtep> also appears as a <wpt> or not (and assuming they hardly ever would).

I can’t speak on popularity, but would expect that you are correct.

However, in order to support round-trip, we need both export and import. And because the HX870’s data store is less expressive than GPX is, export is the more critical step here. Export is where the conversion from “by reference” to “by value” has to take place.

Just to emphasise: Use cases that don’t involve round-trip of routes already work fine.

Taking a step back, perhaps we agree that our objectives are to have backup capability and to spare the user from the pain of editing waypoints and routes on the device.

We do.

The main workflows I see so far are:

1. Create a device backup and restore it as perfectly as (sensibly) possible.
2. Create waypoints and routes in an external editor for the next trip, and then replace or append to what is on in the device.
3. Dump state from device, clean it up or amend it in an external editor, and load it back in.

4. Dump nav data from the device for analysis.
   (For example: Plot MARKed positions, which are stored as waypoints, on a nautical chart.)

5. Combinations of the above.
   (For example 2+3: Initially create some nav data in an external editor, then amend it in the radio {say, by MARKing a new waypoint}, then amend it in the external editor {say, by visually moving the new waypoint slightly to a better position}.)

Perfect backup/restore can be achieved with a complete config backup, but I think we can make it work via GPX export as well. Looks like all we need is a way to restore the waypoint and route databases in their original order. Ideas here: […]

Right. Once again, I have already (albeit very briefly) mentioned these ideas over in #30 (comment).

But I would like to focus on finishing this PR first.

In my view, perfecting route round-trip should come at a later stage, if ever. In fact, I think there is a decent chance “option 3” may even work good enough in practice. I don’t really see routes as a frequently used feature of the radio. Waypoints, sure, there are clear use cases for those with the HX870 and I can see their utility. But entire routes, seriously? I personally think those are a bit of a stretch.

(I’m asking the HX870 mailing list about this. May change my mind depending on what that turns up.)

As far as I can see, the main decisions we need to make are

• Export routepoints just as <rtep>, or also as <wpt>?

In the general case: Both, or else data may be lost.

Which use case, exactly, would a CLI option to control this behaviour support?

• How to detect and handle duplicates in the waypoint database on import to avoid clutter?

Detecting duplicates is but one of the possible approaches to “option 1”. Others avoid duplicates in the first place.

BTW, detecting duplicates would in fact not result in a perfect round-trip in all cases, as the user may already have duplicates before we even start. Those would then be silently filtered by hxtool. Of course, one might argue that such duplicates aren’t useful anyway, but as long as the policy of hxtool is not to filter, ever, I suppose this would technically be a bug.