iD should process and apply data returned by the diff upload

[iandees]

When an iD user saves data, the current graph and its history are cleared. A download of the same area immediately follows to load data from the API back in to the freshly-cleaned graph.

This all happens within a second or two of a successful diff upload, not giving much time for any replication that might happen to complete. Since the OSM API is looking to scale horizontally, replication delay will start to play a factor soon.

iD should consume the response from the server and apply it to the graph.

See also:

• @pnorman's dev@ thread
• my bug/pull request [WIP] Tiled OSM data #1396 that is my work on trying to cache OSM data and won't work because of this

[jfirebaugh]

Agreed, however the current behavior is a simple and decent solution to several issues that are otherwise difficult:

• Merge/conflict resolution. The reload is the only mitigation against conflicts in long multi-save edit sessions.
• Memory management/intelligent data unloading. The reload is the only mitigation against ever-increasing memory consumption in long sessions where lots of data has been loaded but is no longer in view.

[pnorman]

Memory management/intelligent data unloading. The reload is the only mitigation against ever-increasing memory consumption in long sessions where lots of data has been loaded but is no longer in view.

If you were to reload the data then apply the it would work ensure that the data you have contains the changes you uploaded. There are probably other ways to free memory.

Make no mistake, the current behaviour will break when katla is brought online and requests are served from either server.

[jfirebaugh]

What's the timeline for deploying katla?

[pnorman]

I'm not sure. Presumably right after we make iD default :)

cc @Firefishy

[pnorman]

02:12 < pnorman> Firefishy: any idea on a timeline of katla? (e.g. 1 week, 1 month, etc)
02:13 < Firefishy> pnorman: 1 month.
02:13 < Firefishy> ~

[tomhughes]

I'm not sure why you think katla is going to be such an issue? The replication delay we see in the current test environment replicating to smaug is essentially zero...

[pnorman]

It fairly regularly gets above 5 seconds

[ToeBee]

I hit this yesterday. I changed the name of a restaurant and when iD refreshed, the old name was still there. After I did a page refresh in the browser a few seconds later, the new name was there.

[dekstop]

Just to give some more feedback on the impact of this -- we had a big Missing Maps mapathon in London last night with ~100 attendees, most using iD. On several occasions the OSM API was overloaded and slow with processing contributions, with a replication lag of up to 5 minutes.

This means that as soon as people hit "save" their just-mapped buildings seemingly vanished. This affects people's workflows (they have to remember where they already mapped, and may ever miss things or double-map some features), and it's particularly confusing for newcomers.

It is not the first large mapathon where it happened either, and I expect it will become a recurring issue as HOT and Missing Maps grow their communities.

[pnorman]

On several occasions the OSM API was overloaded and slow with processing contributions, with a replication lag of up to 5 minutes.

Just so we have accurate numbers, it hit 4:30 lag once on the Tuesday, and there doesn't seem to have been any elevated load on the API.

That's not to say that you weren't having problems with this bug, excursions past 5s replication lag are normal, just that 100 attendees were not causing any different API behavior.

[mmd-osm]

Another option would be to keep the diff results around for a few seconds, reload the data via /map and then cross check /map response with our diff results.

If diff results indicate that object x should have version y, and /map still returns version y-1 then things are out of sync. In this case we could wait a bit and try another /map call. If all is good simply discard the diff results in memory.