Purging a section of data from the active data set lost line entries in a superroute relation.

[SekeRob]

Mapping a superroute in 43 stages, each stage an entry. While building decided to purge data from the southern half while working northwards on the lower numbered stages of ​https://appenninobiketour.com/percorso/ (JOSM started to slow down with 160MB as a saved dataset. When done on stage 11 and adding to the superroute relation realised that stages 21-29 had disappeared which happened to be in the purged data zone. Pre-Upload the relation looked gapless from 12 to 31. Went about refetching the route stages 21-29 and relinked to save now showing in history view with stages 11-31. Not having worked with superroutes before this loss of the entries in the purged zones was quite surprising. Is this to be expected or have I stumbled on a bug?

For completeness the current config report of 19096

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2024-06-03 18:08:14 +0200 (Mon, 03 Jun 2024)
Revision:19096
Build-Date:2024-06-07 01:31:15
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (19096 en) Windows 10 64-Bit
OS Build number: Windows 10 Home 22H2 (19045)
Memory Usage: 4084 MB / 4084 MB (1971 MB allocated, but free)
Java version: 21.0.3+9-LTS, Azul Systems, Inc., OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 1920×1080 (scaling 1.00×1.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
System property file.encoding: UTF-8
System property sun.jnu.encoding: Cp1252
Locale info: en_US
Numbers with default locale: 1234567890 -> 1234567890
VM arguments: [-Dicedtea-web.bin.location=%UserProfile%\AppData\Local\Programs\OpenWebStart\javaws, -Djava.util.Arrays.useLegacyMergeSort=true, --add-reads=java.base=ALL-UNNAMED,java.desktop, --add-reads=java.desktop=ALL-UNNAMED,java.naming, --add-reads=java.naming=ALL-UNNAMED,java.desktop, --add-exports=java.desktop/sun.awt=ALL-UNNAMED,java.desktop, --add-exports=java.desktop/javax.jnlp=ALL-UNNAMED,java.desktop, --add-exports=java.base/com.sun.net.ssl.internal.ssl=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.security.action=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.security.provider=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.security.util=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.security.validator=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.security.x509=ALL-UNNAMED,java.desktop, --add-exports=java.base/jdk.internal.util.jar=ALL-UNNAMED,java.desktop, --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED,java.desktop, --add-exports=java.desktop/sun.awt.X11=ALL-UNNAMED,java.desktop, --add-exports=java.desktop/sun.applet=ALL-UNNAMED,java.desktop,jdk.jsobject, --add-exports=java.naming/com.sun.jndi.toolkit.url=ALL-UNNAMED,java.desktop, --add-modules=java.scripting,java.sql,javafx.controls,javafx.media,javafx.swing,javafx.web, --add-exports=java.desktop/com.apple.eawt=ALL-UNNAMED, --add-exports=java.desktop/com.sun.imageio.spi=ALL-UNNAMED, --add-exports=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED, --add-exports=javafx.graphics/com.sun.javafx.application=ALL-UNNAMED, --add-exports=jdk.deploy/com.sun.deploy.config=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.desktop/javax.imageio.spi=ALL-UNNAMED, --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED, --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED, -Djava.security.manager=allow]

Dataset consistency test:
[NO DATASET] {Relation id=7164320 version=21 VT [way 1040183355, way 1040183354, way 1040183356, way 487214778, way 487214771, way 1040183351, way 1040183352, way 1040183350, way 1040183348, way 1040183346, way 1040183345, way 1040183347, way 487219254, way 487219263, way 487219241, way 487219235, way 457058700, way 1207114786, way 1207114787, way 487219201]} is referenced by {Relation id=7125614 version=30 VT [relation 7125613, relation 7125612, relation 7126567, relation 7126566, relation 7127284, relation 7127283, relation 7127282, relation 7127281, relation 7128760, relation 7129522, relation 7137939, relation 7153521, relation 7153522, relation 7153520, relation 7154524, relation 7154673, relation 7159129, relation 7159128, relation 7160159, relation 7160158, relation 7160157, relation 7160934, relation 7162469, relation 7162545, relation 7162818, relation 7162817, relation 7164320, relation 7164319, relation 7164318, relation 9266264, relation 9270352, relation 9270471]} but not found in dataset
[NO DATASET] {Relation id=7164319 version=16 VT [way 487234093, way 943919860, way 487234083, way 943919858]} is referenced by {Relation id=7125614 version=30 VT [relation 7125613, relation 7125612, relation 7126567, relation 7126566, relation 7127284, relation 7127283, relation 7127282, relation 7127281, relation 7128760, relation 7129522, relation 7137939, relation 7153521, relation 7153522, relation 7153520, relation 7154524, relation 7154673, relation 7159129, relation 7159128, relation 7160159, relation 7160158, relation 7160157, relation 7160934, relation 7162469, relation 7162545, relation 7162818, relation 7162817, relation 7164320, relation 7164319, relation 7164318, relation 9266264, relation 9270352, relation 9270471]} but not found in dataset
[NO DATASET] {Relation id=7164318 version=20 VT [way 144431071, way 378149301, way 487234076, way 378149418, way 666539838, way 666539836, way 666539835, way 666540787, way 487234070, way 143885724, way 487234080, way 943921612, way 943921613, way 666540785, way 487234079, way 487234077]} is referenced by {Relation id=7125614 version=30 VT [relation 7125613, relation 7125612, relation 7126567, relation 7126566, relation 7127284, relation 7127283, relation 7127282, relation 7127281, relation 7128760, relation 7129522, relation 7137939, relation 7153521, relation 7153522, relation 7153520, relation 7154524, relation 7154673, relation 7159129, relation 7159128, relation 7160159, relation 7160158, relation 7160157, relation 7160934, relation 7162469, relation 7162545, relation 7162818, relation 7162817, relation 7164320, relation 7164319, relation 7164318, relation 9266264, relation 9270352, relation 9270471]} but not found in dataset
[NO DATASET] {Relation id=14683394 version=10 VT [way 487234093, way 943919860, way 487234083, way 943919858]} is referenced by {Relation id=17307598 version=9 VT [relation 14654347, relation 14654443, relation 15023181, relation 15023180, relation 14656884, relation 14682387, relation 14654581, relation 14654602, relation 14656964, relation 14656971, relation 14656993, relation 14682166, relation 14703796, relation 14703865, relation 14703916, relation 14703925, relation 14703941, relation 14703947, relation 14683394, relation 14683759, relation 14683653, relation 14683673, relation 14654549, relation 14703721, relation 14703764]} but not found in dataset
[NO DATASET] {Relation id=14683759 version=5 VT [way 666539840, way 487234082, way 1298100960, way 72796678, way 1041751143, way 1102880852, way 666542852, way 143885726, way 666542851, way 143885668, way 1102880850, way 1102880849, way 610565733, way 1102880847, way 487234049, way 487234050, way 666770143, way 666770142, way 519404762, way 487484842, way 666771927]} is referenced by {Relation id=17307598 version=9 VT [relation 14654347, relation 14654443, relation 15023181, relation 15023180, relation 14656884, relation 14682387, relation 14654581, relation 14654602, relation 14656964, relation 14656971, relation 14656993, relation 14682166, relation 14703796, relation 14703865, relation 14703916, relation 14703925, relation 14703941, relation 14703947, relation 14683394, relation 14683759, relation 14683653, relation 14683673, relation 14654549, relation 14703721, relation 14703764]} but not found in dataset


Plugins:
+ HouseNumberTaggingTool (36226)
+ KartaView (490)
+ Mapillary (2.2.4)
+ OpeningHoursEditor (36258)
+ RoadSigns (36258)
+ apache-commons (36273)
+ buildings_tools (36226)
+ easypresets (1623509627)
+ graphview (36258)
+ gridify (1718663815)
+ libphonenumber (8.13.39)
+ measurement (36256)
+ notesolver (0.5.0)
+ phonenumber (1.1.1)
+ pt_assistant (637)
+ reltoolbox (36280)
+ reverter (36256)
+ routing (36226)
+ scripting (v0.3.1)
+ tageditor (36258)
+ terracer (36205)
+ todo (137)
+ turnlanes (36206)
+ turnlanes-tagging (0.0.5)
+ turnrestrictions (36226)
+ utilsplugin2 (36241)

Tagging presets:
+ <josm.pref>/EasyPresets.xml
+ https://josm.openstreetmap.de/josmfile?page=Presets/OneClick&zip=1
+ https://raw.githubusercontent.com/osmlab/name-suggestion-index/main/dist/presets/nsi-josm-presets.min.xml
+ https://josm.openstreetmap.de/josmfile?page=Presets/Golf_Course&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/AdvertisingPreset&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/Maxspeed-zones&zip=1
+ https://raw.githubusercontent.com/yopaseopor/traffic_signs_preset_JOSM/master/IT.zip
+ https://josm.openstreetmap.de/josmfile?page=Presets/Manholes&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/LaneAttributes&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/TurnLanes&zip=1
+ https://github.com/kendzi/Simple3dBuildingsPreset/releases/download/0.9_2018-05-08/s3db-preset.zip

Map paint styles:
+ https://josm.openstreetmap.de/josmfile?page=Styles/Potlach2_access&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Admin_Boundaries&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/PublicTransport&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/PublicTransportV2&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/ColorWays&zip=1
- https://josm.openstreetmap.de/josmfile?page=Rules/IncompleteObjectWarnings&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_buildings_en&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Enhanced_Lane_and_Road_Attributes&zip=1
+ https://gitlab.com/cartocite/josm-style-traffic-signs-orientation/-/raw/main/traffic_sign_orientation_style.zip
- https://josm.openstreetmap.de/josmfile?page=Styles/Modified&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/MapWithAI&zip=1
+ https://raw.githubusercontent.com/yopaseopor/traffic_signs_style_JOSM/master/Styles_Traffic_signs_EUR_OC.zip
- https://josm.openstreetmap.de/josmfile?page=Styles/Maxspeed&zip=1
- https://raw.githubusercontent.com/OpenNauticalChart/josm/master/european-waterways-classification-style/CEMT-style.mapcss
- https://josm.openstreetmap.de/josmfile?page=Styles/Waterways&zip=1

Last errors/warnings:
- 27462.939 W: java.net.SocketTimeoutException: Connect timed out
- 28480.588 W: Conflict in data - <html>This relation has been changed outside of the editor.<br>You cannot apply your changes and continue editing.<br><br>Do you want to create a conflict and close the editor?</html>
- 28491.763 W: Conflict in data - <html>This relation has been changed outside of the editor.<br>You cannot apply your changes and continue editing.<br><br>Do you want to create a conflict and close the editor?</html>
- 28510.636 W: Conflict in data - <html>This relation has been changed outside of the editor.<br>You cannot apply your changes and continue editing.<br><br>Do you want to create a conflict and close the editor?</html>
- 28526.274 W: Conflict in data - <html>This relation has been changed outside of the editor.<br>You cannot apply your changes and continue editing.<br><br>Do you want to create a conflict and close the editor?</html>
- 29350.040 W: Conflict in data - <html>This relation has been changed outside of the editor.<br>You cannot apply your changes and continue editing.<br><br>Do you want to create a conflict and close the editor?</html>
- 35573.485 W: java.net.SocketException: Socket closed. Cause: java.net.SocketException: Socket closed
- 35573.486 E: java.net.SocketException: Socket closed. Cause: java.net.SocketException: Socket closed
- 37003.428 W: java.net.SocketTimeoutException: Connect timed out
- 37045.505 E: Region [kartaview-private_BLOCK_v2] : Problem loading keys for file kartaview-private_BLOCK_v2

[GerdP]

What makes you think that you encountered a bug in JOSM?

[SekeRob]

I don't say it's a bug, I ask if it is to be expected purging an area of data (Ctrl+Shft+P9) that had no modifier flags yet the superroute being effected by it loosing the members that were in that unmodified area.

[GerdP]

Sorry, I cannot follow. Please describe step by step how to reproduce.

[SekeRob]

A data set growing and growing containing 20, now 22 stages in the Appennino Bike Tour, link in OP. The superroute relation contained 20 line entries(members) for the stages completed. After being done and JOSM getting slower and slower, last evening intolerable, decided to purge sections from the loaded data set (150MB in saved size) covering the area of stages 21 to 29 as that part was done-done, leaving 30-31 as 32-43 still needed to be mapped in the south. That helped tremendously and continued on stage nr 11, (the 21st stage for which a bike relation was created. After completing that stage opened the superroute relation to add nr 11, seeing to my horror that line entries for stages 21-29 were gone from the list. Fortunately just from the superroute, not from the data on the server as is the purpose of Ctrl+Shft+P, just forget the offline data as if never there, so went about refetching the data for the missing stages and re-add them to the superroute.

So in steps:

Before: Superroute has entries for stages 12-31. All data add/changes were upload, 'up' toolbar button grey.
Purge data area for 21-29, recover working speed.
Add completed stages 11 to Superroute and see 21-29 missing.

Notably there was slim hope the refetching of the 21-29 stage data would restore the lines in the superroute, it did not.

Maybe I could have refetched the superroute relation from the server with the 'download object' function as between the purge and mapping/adding stage 11 there were no new uploads from the active data set. Suspect though that flags were set to say the superroute relation was modified and therefor a refetch would just ignore the copy on the server as older.

At any rate if there's a superroute in the active data one might unwittingly mangle it purging areas.

[GerdP]

Is it possible that you downloaded the data from Overpass API instead of OSM and that Overpass didn't have your latest changes?

[SekeRob]

No, the way I get data with JOSM is through the toolbar download button. The gpx traces I pull in with open location (only need to see it in background without causing snap issues). Occasionally use download object. I'm positive there were no uploads in between since the sole purpose of the dataset is to map the Apennines Bike Tour (just happen to stumble on an image near Florence with a new brown direction sign i.e. was on track. Not added up the 43 stage kms but think it will hit about 2000km meandering down the Italian booth.)

If I find this superroute to be a satisfactory solution my idea is to break up a couple of monster relations like the Nazionale Adriatica SS16 with over 2000 members. Touch it, it's near unavoidable mapping along the coast, and one gets the 'incomplete' relation warning and down it comes again to verify there's no new gaps not to speak of the Flixbus monstrosities such as from Zurich to Taranto that use the SS16 and A14 extensively. Break it up and gap fixing becomes a PoP and sure as heck, this cycle route will see many mapper made gap issues in the years to come especially since roundabouts pop up like mushrooms all over the place.

[SekeRob]

Ah, did not register the Overpass API tab on the JOSM download page. Certainly do not know how to compile an Overpass query.

[GerdP]

I see no way to reproduce with your description. If you used overpass to download outdated data you should have seen cnflicts later. Your status report contains several lines about such conflicts, but overpass is not the only possible cause. Another reason could be that you loaded older data from file.
If you can reproduce please report the exact steps.

[anonymous]

Not able to reproduce following the steps I remember, pull all stages from server, superroute complete, purge large segments, superroute remains complete. Positively not in my waking hours have used Overpass. At any rate it's so specific that the 21-29 stages were purged somehow affecting to cause the action to have them go awol from the super relation. Been building it from central Italy north and southbound without gaps. On a next test I'll purge stage data in the middle rather than the ends and see if that causes the bork.

(JOSM really slows down when one has 1500 km of roads and paths in a data set, sum total of the now 25 stages completed probably exceeds 3000 members. Since the 'recommended' max members for a single relation is 300, it would actually be great if the relation editor shows a counter somewhere in the header area. Also a column in the parent/superroute relation view showing each child relation with the member count... turn the field(s) red when more that the recommended max... yes I like bells and whistles)

Stumbled on a 'Ciclovia da Sole' with almost 3300 members and just loading that turned JOSM into a zoom/in/out navigation snail so expanding mapper awareness of supersizes would be good).

[skyper]

Maybe you hit #15194. You might want to take a look at the advanced preferences regarding purge.
For sure you will loose new objects (id=0) which have not been uploaded to server when purging.

Regarding performance, make sure to disable additional map paint styles and use the wireframe mode as much as possible. Sometimes using filters can help, too. For a complete sort of the superroute you need all child relations and their members but I tend to work in smaller sections like sorting each child relation first. For the continuity check in the superroute you only need the first and last way member of the child relations. Additionally, I often use additional data layers where I merge the objects I want to work on and completely disable the huge data layer. If I am done with that part, I upload my changes and merge the layer back into the huge one (in wireframe mode).