Automatically detect and fix situations, where server accepted upload and JOSM assumes upload failure

[dieterdreist]

Just discovered it is all double. It is around 6200 changes...
i don't know exactly which one it was: changesets
*5140491
*5140246
*5140090
*5139835
I set this to blocker because I expect other users having the same issues due to the downtime recently.

[stoecker]

Probably your uploaded twice as you thought the first upload failed?

[dieterdreist]

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

[skyper]

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

[dieterdreist]

Replying to skyper:

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

Actually I think I failed and didn't because I wasn't aware that I should do (the area is almost empty and I didn't expect other mappers)

[dieterdreist]

Replying to dieterdreist:

Replying to skyper:

Replying to dieterdreist:

Replying to stoecker:

Probably your uploaded twice as you thought the first upload failed?

: yes, that's exactly what I did, but I thought that JOSM would prevent to upload twice. It also seemed to be a slightly different number of changes.

Did you update your data before the second upload ?

Actually I think I failed and didn't because I wasn't aware that I should do (the area is almost empty and I didn't expect other mappers)

It happened again! ​http://www.openstreetmap.org/browse/changeset/5227820
This time there was a conflict reported for one node on upload (this also happened above), but while above I clicked on "synchronize whole ..." now I clicked on "cancel" and did "update data" first. On Update I was reported that approx. 2200 elements seem to be "deleted" on the server (the ones I wanted to upload I guess, before the changeset didn't upload "check your internet connection"). After this update there was 1 conflict (multipolygon-relation, I took "my version" not "their version"). After the conflict was resolved, there were 48 duplicate nodes reported. I selected the error and then on fix. Afterwards tried to upload but: 306 duplicate nodes. Again fixed. Afterwards fixed the multipolygon (1 way was double, 648 vs 650 nodes). Now I am uploading again...

[dieterdreist]

fixed. Afterwards fixed the multipolygon (1 way was double, 648 vs 650 nodes). Now I am uploading again...

There is a mean bug somewhere there. After upload I got 966 duplicated nodes. Resolved them and fixed an area with duplicated nodes. After this (did nothing else) validator reported 36 duplicated ways that weren't reported 10 seconds before (the only thing I did in the mean time was selecting the duplicated nodes of one element (which was reported as area feature not closed) and merge them).

[AM909]

Possibly related, though I'm afraid I don't have as much detail as I'd like.

I downloaded some neighboring bounding boxes, then did some work on one of them and uploaded it. Then, I did some work on the next one, attempted to upload it (579 ways to be modified), and got a complaint about the server having a newer version of one of my ways. I chose to sync that way only and tried again. It complained about another way, so I repeated the process. I did this several more times before biting the bullet and choosing to "sync entire dataset", which took a long time (as expected).

At some point, I looked at the open changeset with ​http://www.openstreetmap.org/browse/changeset/5257539 and it showed that it contained 579 ways.

When I then tried to upload again, I noticed that it said there were only 567 ways to upload. It's entirely possible that the difference (12) is the number of times that I sync'd individual ways, though I can't confirm this.

After the upload succeeded, I saved the dataset, and looked at the changeset again, and it now shows 1146 ways (579+567). If I download the changeset XML, I see that 567 of the ways are listed twice, with two successive version numbers (e.g. ​http://www.openstreetmap.org/browse/way/11234037/history). The version number in the file I saved after the successful upload is the latest one (3 in the example).

Note that, in my case, all the edits are to tags on existing ways - no geometry changes and no object adds/deletes.

HTH

[AM909]

Forgot to mention: this happened running r3329.

[dieterdreist]

Replying to Am909:
It happened again with 3376: Uploaded 1808 changes. Got a message that one way is conflicted. I canceled synchronisation: 11 duplicate nodes. I tried update data. 2 errors happened (see attachments).

[dieterdreist]

Replying to dieterdreist:

Replying to Am909:
It happened again with 3376: Uploaded 1808 changes. Got a message that one way is conflicted. I canceled synchronisation: 11 duplicate nodes. I tried update data. 2 errors happened (see attachments).

Please also note that the conflicting data seems to be more (ca. 1900) than the actual number of edits I tried to upload (1808).

[stoecker]

NOTE: When you have a situation like above: JOSM thinks upload failed, but server accepted upload, then you should drop your whole local dataset and reload it from server (you may load it into a different layer to verify if it really is complete).

There is no need to do conflict resolution, as there aren't any conflicts. Conflict handling for new elements is rather nonexisting, so whatever you do it will produce incorrect results.

JOSM may detect this situation and properly handle it in future, but don't expect this soon.

[stanton]

Yes, this is just the problem: I've had at least two of these cases in which something did get uploaded but I don't know how much. Edits to existing objects aren't a problem as the server will reject the new upload due to the version number not matching, but new objects will get created a second time. Verifying how many of these actually made it to the server is a tedious chore when the number of new objects is large - so I believe this case should get handled somewhere, even if only to prevent garbage (duplicate objects) from being uploaded to the server. Frankly, I believe this should go not in the editor but in the API as the "choke point" - I had opened ticket 3521 on OSM but it was closed as "wontfix".

We may need to discuss this among a wider audience - other editors may at times encounter similar issues (I remember about the TIGER import creating massive amounts of dupes), hence this potentially affects all editors and the API.

[stoecker]

Ticket #7321 has been marked as a duplicate of this ticket.

[skyper]

I am not sure if it was #5783 without exception or this bug but I just stumbled over an upload which started to repeat its conflicts where new objects on the server had already an id but JOSM had them still listed as new objects (id=0). All I know is that I had three real conflicts in the beginning (2 releations + one node still in use) and I was uploading ~1000 objects but with more than 100 BBOXs and always tried to upload to one changeset which is still open (changeset: 18559678).

I did save the data after I got more and more conflicts and did restart. After careful inspecting all modified object and downloading/merging I let validator auto fix all the duplicates I ended up without any modified object left and only some deletions to upload which had been uploaded to server already.

[planemad]

A recent example of this issue causing a major data disaster in Uzbekistan: ​http://www.openstreetmap.org/user/PlaneMad/diary/35762

[bastiK]

Ticket #12601 has been marked as a duplicate of this ticket.

[bastiK]

Link to osm ticket menitioned above: #osm3521.

I think there are pragmatic ways to detect this situation. Most simple would be to download a tiny bounding box around one of the nodes to upload. If the server returns a node with the same coordinates and tags, issue a warning or perform more thorough checks.

When the changeset contains modifications or deletions, the server will reject duplicate upload, so this is only necessary for changesets with additions only.

[mmd]

We're currently discussing potential changes to the OSM API to make the changeset upload idempotent. It means that you could re-upload the same changeset after some network failure without risking duplicates on the database, and at the same time still receive the original diffResult message.

Please feel free to join the discussion on ​https://github.com/openstreetmap/openstreetmap-website/issues/2201

There's even a small prototype in that issue to demonstrate the use of an Idempotency-Key for JOSM changeset upload.