JOSM uploads modified data twice

[wolfgang8]

What steps cause (will reproduce) the problem?

1. Download data and notes from area around "Deinzendorf".
2. Modify data of "Deinzendorf" in JOSM.
3. Upload modified data with cs-comment "add+update buildings in Deinzendorf".

​https://www.openstreetmap.org/changeset/130297692
​https://resultmaps.neis-one.org/osm-change-viz?c=130297692#18/48.69581/15.92224

4. Close note ​https://www.openstreetmap.org/note/3145739 in JOSM and upload it.

5. Download data and notes from area around "Kattau" in the same JOSM layer as the data of "Deinzendorf".
6. Modify data of "Kattau" in JOSM.
7. Upload modified data with cs-comment "add+update buildings in Kattau".

​https://www.openstreetmap.org/changeset/130298264
​https://resultmaps.neis-one.org/osm-change-viz?c=130298264#13/48.6861/15.8661

8. Close note ​https://www.openstreetmap.org/note/2896187 in JOSM and upload it.

What is the expected result?

In cs 130297692 the added/changed objects of "Deinzendorf" will be uploaded and in cs 130298264 the added/changed objects of "Kattau".

What happens instead?

In cs 130297692 the added/changed objects of "Deinzendorf" are uploaded correctly. But in cs 130298264, additionally to the added/changed objects of "Kattau" the changed objects of "Deinzendorf" (only the changed - not the added) are also included in this changeset.

To me, it seems, that JOSM didn't flag the changed objects as uploaded after the first upload and therefor they are also included in the second upload.

For example

• changed objects in cs 130297692:

​https://www.openstreetmap.org/way/553575107/history
​https://www.openstreetmap.org/node/5359759450/history

• added objects in cs 130297692:

​https://www.openstreetmap.org/way/1123677452/history
​https://www.openstreetmap.org/node/10276183488/history

Please provide any additional information below. Attach a screenshot if possible.

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2022-10-31 17:29:20 +0100 (Mon, 31 Oct 2022)
Revision:18583
Build-Date:2022-11-01 02:30:58
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (18583 de) Windows 10 64-Bit
OS Build number: Windows 10 Home 2009 (19045)
Memory Usage: 486 MB / 2048 MB (87 MB allocated, but free)
Java version: 11.0.17+8, Eclipse Adoptium, OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920×1080 (scaling 1.25×1.25)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
System property file.encoding: Cp1252
System property sun.jnu.encoding: Cp1252
Locale info: de_AT
Numbers with default locale: 1234567890 -> 1234567890
Dataset consistency test: No problems found

Plugins:
+ AddrInterpolation (36011)
+ FastDraw (35978)
+ FixAddresses (36011)
+ OpeningHoursEditor (35924)
+ PicLayer (1.0.2)
+ RoadSigns (36011)
+ alignways (36011)
+ apache-commons (36034)
+ apache-http (35924)
+ austriaaddresshelper (1597341117)
+ buildings_tools (36011)
+ changeset-viewer (v0.0.6)
+ ejml (35924)
+ geotools (36028)
+ graphview (36011)
+ jackson (36034)
+ jaxb (35952)
+ jna (36005)
+ jts (36004)
+ opendata (36025)
+ reltoolbox (35976)
+ reverter (36011)
+ tageditor (36011)
+ turnlanes-tagging (v0.0.5)
+ undelete (36011)
+ utilsplugin2 (36011)
+ waydownloader (36011)
+ wikipedia (605)

Tagging presets:
+ https://josm.openstreetmap.de/josmfile?page=Presets/ParkingLanes&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/NewTags&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/NewParkingFeatures&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Crafts&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Historical_Objects&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Heritage&zip=1
+ https://raw.githubusercontent.com/yopaseopor/traffic_signs_preset_JOSM/master/A.zip
+ https://josm.openstreetmap.de/josmfile?page=Presets/BuildingPreset&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/MobilePhoneBaseStations&zip=1
+ https://raw.githubusercontent.com/OpenNauticalChart/josm/master/Presets_Hafen.xml
+ <josm.pref>\xmas.xml
+ https://josm.openstreetmap.de/josmfile?page=Presets/Golf_Course&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Healthcare&zip=1

Map paint styles:
- https://github.com/simon04/coloured-addresses.mapcss/raw/master/dist/coloured-addresses.mapcss
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Surface-DataEntry&zip=1
- https://www.dropbox.com/s/qo3ai47fpv241jf/Styles_Fixme_and_Notes.zip?raw=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Streets&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Postcode&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_buildings&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Modified&zip=1
- <josm.pref>\Kartenstile\addr_unit.mapcss
+ <josm.pref>\Kartenstile\ChangeFontSize.mapcss
- https://josm.openstreetmap.de/josmfile?page=Styles/Maxspeed&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/SidewalksAndFootways&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Landcover&zip=1
- <josm.pref>\Kartenstile\applepaintstyles-main\main_street.mapcss

Last errors/warnings:
- 00017.951 E: Fehler beim Laden des Bildes 'xmas_event.png'
- 00017.952 W:  WeihnachtsEvent: Could not get presets icon xmas_event.png
- 00017.953 E: Fehler beim Laden des Bildes 'xmas_pyramid.png'
- 00017.954 W:  Weihnachtspyramide: Could not get presets icon xmas_pyramid.png
- 00291.518 E: Fehler beim Laden des Bildes 'https://www.basemap.at/images/logo_basemap.jpg'
- 00294.721 E: Fehler beim Laden des Bildes 'https://www.basemap.at/images/logo_basemap.jpg'
- 02248.548 W: java.net.SocketTimeoutException: Read timed out. Ursache: java.net.SocketTimeoutException: Read timed out
- 02885.744 E: Fehler beim Laden des Bildes 'https://www.basemap.at/images/logo_basemap.jpg'
- 14510.776 E: Fehler beim Laden des Bildes 'https://www.basemap.at/images/logo_basemap.jpg'
- 14513.385 E: Fehler beim Laden des Bildes 'https://www.basemap.at/images/logo_basemap.jpg'

[taylor.smock]

It looks like something is going wrong when we update the local data post-changeset. We are not (fortunately) uploading duplicate data, at least. We are just marking nodes as modified when they aren't, unless I misunderstood the ticket.

Before I do a deep dive, did you use any tool which may have modified the geometry of the building, even slightly (e.g., squaring the building/parts of the building).

[skyper]

Does not happen for me with josm-latest and fresh preferences:

• ​changeset/130340724
• ​changeset/130340894
• ​changeset/130341047

all uploaded within the same session and using same data layer.

[skyper]

Replying to taylor.smock:

It looks like something is going wrong when we update the local data post-changeset. We are not (fortunately) uploading duplicate data, at least. We are just marking nodes as modified when they aren't, unless I misunderstood the ticket.

Not only nodes but also ways. E.g. version 2 and 3 of ​osmwww:way/553575141 are identical.

Before I do a deep dive, did you use any tool which may have modified the geometry of the building, even slightly (e.g., squaring the building/parts of the building).

Does not make sense with ways, does it ?

[taylor.smock]

Replying to skyper:

Does not make sense with ways, does it ?

Nope. I was just looking at the nodes.

If it is happening to ways, it is still a problem (slightly bigger), but not world ending. Yes, it is leading to some version inflation, but it isn't adding duplicates.

Does not happen for me with josm-latest and fresh preferences:

@wolfgang8: It might be a very specific set of steps that leads to this outcome. It shouldn't be plugin related, since I don't think plugins can do anything about the process of matching the uploaded data to the data in the layer.

Can you try to reproduce and give very explicit steps?
Example:
1) Start JOSM
2) Using the Download button, download the bounding box <bounding box> (you can get the bounding box from the advanced preference osm-download.bounds)
3) Add a node at position <lat lon> (you can get the lat lon of a node by using ctrl+shift+c) in way <id> (you can get the id of an osm object by using ctrl+c) using Improve Way Accuracy
4) Upload
5) Add a node at position <lat lon> of way <other id> using Draw
6) Upload

[wolfgang8]

Thanks for your reply

I tried to reproduce it:

1) Start JOSM
2) Using the Download button, download the bounding box <48.0996146;15.3338671;48.101628;15.3380835>
3) Add a node at position <48.1003587, 15.3377914> in way <way 29769023> using Improve Way Accuracy
4) Upload
5) Add a node at position <48.100215, 15.3367336> of way <way 135746188> using Draw
6) Upload

Changesets:

​https://www.openstreetmap.org/changeset/130347037
​https://www.openstreetmap.org/changeset/130347079

In this case, it looks good to me. It seems, that very specific conditions are needed for this issue.

[taylor.smock]

Thank you for trying to reproduce. Best guess: OsmDataLayer#cleanupAfterUpload isn't getting called.

• 02248.548 W: java.net.SocketTimeoutException: Read timed out. Ursache: java.net.SocketTimeoutException: Read timed out

I don't know if the SocketTimeoutException occurred during the upload or not, but that might be the problem. Something needs to be preventing us from getting to the cleanupAfterUpload call, anyway.

[anonymous]

Replying to taylor.smock:

... Best guess: OsmDataLayer#cleanupAfterUpload isn't getting called.

Interesting that only changed objects are affected - e.g. ​https://www.openstreetmap.org/way/553575107/history - but not new created ones - e.g. ​https://www.openstreetmap.org/way/1123677452/history.

Maybe, the issue is located within OsmDataLayer#cleanupAfterUpload?

[wolfgang8]

The last comment is mine.

[taylor.smock]

Replying to wolfgang8:

Maybe, the issue is located within OsmDataLayer#cleanupAfterUpload?

I doubt it -- the only way for it to be an issue is if the method isn't called, or the server didn't give us data back. At which point, we probably should start having duplicate objects.

Stepping through the code for OsmDataLayer#cleanupAfterUpload:

        // return immediately if an upload attempt failed
        if (Utils.isEmpty(processed)) // If we had duplicate objects uploaded, this would be my guess
            return;

        UndoRedoHandler.getInstance().clean(data); // This can only throw if a `null` command is added, which should only be possible if someone is monkeying around with a debugger

        // if uploaded, clean the modified flags as well
        data.cleanupDeletedPrimitives(); // This should never throw an exception
        data.update(() -> {
            for (OsmPrimitive p: data.allPrimitives()) {
                if (processed.contains(p)) {
                    p.setModified(false); // And now everything should be set to unmodified, assuming that everything was in `processed`. Not necessarily the case, if the SocketTimeoutException is the root cause. But then we don't know what the new id of the object is, so we should have duplicate new objects.
                }
            }
        });