"org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset" bug popup sometimes

[mnalis]

What steps will reproduce the problem?

1. closed data layer after updating
2. create a new node
3. try to mark it as restaurant via F3

What is the expected result?

normal editing continues

What happens instead?

"You have encountered a bug in JOSM" window pops up.

Please provide any additional information below. Attach a screenshot if possible.

Note that node that it complains about ( ​https://www.openstreetmap.org/node/10722374641 ) has been created and uploaded successfully before data layer was removed and new one opened. Trying to ignore the error and again use F3 on that node brings up error window again with same error.

It didn't happen in previous testing JOSM (few weeks back),but now has happened several times

(perhaps related to #22662 ?)

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2023-03-01 15:17:42 +0100 (Wed, 01 Mar 2023)
Revision:18678
Build-Date:2023-03-02 02:30:57
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (18678 en) Linux Debian GNU/Linux 11 (bullseye)
Memory Usage: 1610 MB / 2988 MB (700 MB allocated, but free)
Java version: 11.0.18+10-post-Debian-1deb11u1, Debian, OpenJDK 64-Bit Server VM
Look and Feel: javax.swing.plaf.metal.MetalLookAndFeel
Screen: :0.0 1920×1080 (scaling 1.00×1.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→16×16, 32×32→32×32
Environment variable LANG: hr_HR.UTF-8
System property file.encoding: UTF-8
System property sun.jnu.encoding: UTF-8
Locale info: en_US
Numbers with default locale: 1234567890 -> 1234567890
Desktop environment: ICEWM
Java package: openjdk-11-jre:amd64-11.0.18+10-1~deb11u1
libcommons-compress-java: libcommons-compress-java:-
fonts-noto: fonts-noto:all-20201225-1
liboauth-signpost-java: liboauth-signpost-java:-
Dataset consistency test: No problems found

Plugins:
+ DirectUpload (35951)
+ FixAddresses (36011)
+ MapRoulette
+ PicLayer (1.0.2)
+ apache-commons (36034)
+ apache-http (35924)
+ buildings_tools (36011)
+ conflation (0.6.9)
+ ext_tools (35893)
+ imagery_offset_db (35978)
+ jna (36005)
+ jts (36004)
+ openqa (v0.3.3)
+ pt_assistant (1ff2e15)
+ reverter (36043)
+ todo (123)
+ turnrestrictions (36011)
+ utilsplugin2 (36011)
+ wikipedia (605)

Tagging presets:
+ https://<user.name>.com/prezentacije/OpenStreetMap/josm-preset.xml
+ https://raw.githubusercontent.com/OpenNauticalChart/josm/master/INT-1-preset.xml

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/PublicTransport&zip=1
+ http://bezdomni.net/Styles_HighlightedCapesBays-style.mapcss
+ https://josm.openstreetmap.de/josmfile?page=Styles/ColorWays&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Streets&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Fixme&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Modified&zip=1

Validator rules:
+ https://josm.openstreetmap.de/josmfile?page=Rules/KeepRight&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Rules/QAToolInspiredValidations&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Rules/OSMLint&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Rules/MissingWeekdays&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Rules/SourceObjectTag&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Rules/IndoorRules&zip=1

Last errors/warnings:
- 00005.009 W: Plugin MapRoulette requires Java version 17. The current Java version is 11. You have to update Java in order to use this plugin.
- 00005.234 E: Failed to locate image 'bus.png'
- 00111.744 E: Invalid setting (Icon missing): org.openstreetmap.josm.plugins.fixAddresses.FixAddressesPreferences
- 00118.255 E: Failed to locate image '[[Image(pond.png)]]'
- 00118.260 E: Failed to locate image '[[Image(oil-palm.png)]]'
- 00142.805 E: Invalid setting (Icon missing): org.openstreetmap.josm.plugins.fixAddresses.FixAddressesPreferences
- 06967.481 W: java.net.SocketException: Socket closed
- 06967.482 E: java.net.SocketException: Socket closed
- 22790.543 E: Handled by bug report queue: org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=10722374641 version=1 VT lat=44.04921097061714,lon=15.312376568276045}



=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: AWT-EventQueue-0 (20) of main
org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=10722374641 version=1 VT lat=44.04921097061714,lon=15.312376568276045}
	at org.openstreetmap.josm.data.osm.OsmPrimitive.checkDataset(OsmPrimitive.java:198)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.referrers(OsmPrimitive.java:702)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.getReferrers(OsmPrimitive.java:728)
	at org.openstreetmap.josm.data.osm.OsmPrimitive.getReferrers(OsmPrimitive.java:734)
	at org.openstreetmap.josm.data.osm.FilterModel.getAffectedPrimitives(FilterModel.java:442)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetValidation.applyChangedTags(TaggingPresetValidation.java:81)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetValidation.validateAsync(TaggingPresetValidation.java:45)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPreset.lambda$createPanel$2(TaggingPreset.java:452)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetItemGuiSupport.lambda$fireItemValueModified$1(TaggingPresetItemGuiSupport.java:182)
	at org.openstreetmap.josm.tools.ListenerList.fireEvent(ListenerList.java:155)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetItemGuiSupport.fireItemValueModified(TaggingPresetItemGuiSupport.java:182)
	at org.openstreetmap.josm.gui.tagging.presets.items.MultiSelect.lambda$addToPanel$0(MultiSelect.java:97)
	at java.desktop/javax.swing.JList.fireSelectionValueChanged(JList.java:1804)
	at java.desktop/javax.swing.JList$ListSelectionHandler.valueChanged(JList.java:1818)
	at java.desktop/javax.swing.DefaultListSelectionModel.fireValueChanged(DefaultListSelectionModel.java:219)
	at java.desktop/javax.swing.DefaultListSelectionModel.fireValueChanged(DefaultListSelectionModel.java:199)
	at java.desktop/javax.swing.DefaultListSelectionModel.fireValueChanged(DefaultListSelectionModel.java:246)
	at java.desktop/javax.swing.DefaultListSelectionModel.removeIndexInterval(DefaultListSelectionModel.java:715)
	at java.desktop/javax.swing.plaf.basic.BasicListUI$Handler.intervalRemoved(BasicListUI.java:2763)
	at java.desktop/javax.swing.AbstractListModel.fireIntervalRemoved(AbstractListModel.java:184)
	at java.desktop/javax.swing.DefaultListModel.clear(DefaultListModel.java:497)
	at org.openstreetmap.josm.gui.tagging.presets.items.MultiSelect.addToPanel(MultiSelect.java:48)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPreset.createPanel(TaggingPreset.java:434)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPreset.showDialog(TaggingPreset.java:604)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPreset.showAndApply(TaggingPreset.java:516)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPreset.actionPerformed(TaggingPreset.java:506)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetSearchDialog.buttonAction(TaggingPresetSearchDialog.java:57)
	at org.openstreetmap.josm.gui.ExtendedDialog$1.actionPerformed(ExtendedDialog.java:381)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967)
	at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308)
	at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405)
	at java.desktop/javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262)
	at java.desktop/javax.swing.AbstractButton.doClick(AbstractButton.java:369)
	at java.desktop/javax.swing.plaf.basic.BasicRootPaneUI$Actions.actionPerformed(BasicRootPaneUI.java:254)
	at java.desktop/javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1810)
	at java.desktop/javax.swing.JComponent.processKeyBinding(JComponent.java:2900)
	at java.desktop/javax.swing.KeyboardManager.fireBinding(KeyboardManager.java:311)
	at java.desktop/javax.swing.KeyboardManager.fireKeyboardAction(KeyboardManager.java:253)
	at java.desktop/javax.swing.JComponent.processKeyBindingsForAllComponents(JComponent.java:2993)
	at java.desktop/javax.swing.JComponent.processKeyBindings(JComponent.java:2985)
	at java.desktop/javax.swing.JComponent.processKeyEvent(JComponent.java:2862)
	at java.desktop/java.awt.Component.processEvent(Component.java:6412)
	at java.desktop/java.awt.Container.processEvent(Container.java:2263)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5011)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4843)
	at java.desktop/java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1950)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:870)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1139)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:1009)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:835)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4892)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2772)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4843)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:772)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:743)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:117)
	at java.desktop/java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:190)
	at java.desktop/java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:235)
	at java.desktop/java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:233)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.desktop/java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:233)
	at java.desktop/java.awt.Dialog.show(Dialog.java:1070)
	at java.desktop/java.awt.Component.show(Component.java:1716)
	at java.desktop/java.awt.Component.setVisible(Component.java:1663)
	at java.desktop/java.awt.Window.setVisible(Window.java:1031)
	at java.desktop/java.awt.Dialog.setVisible(Dialog.java:1005)
	at org.openstreetmap.josm.gui.ExtendedDialog.setVisible(ExtendedDialog.java:462)
	at org.openstreetmap.josm.gui.ExtendedDialog.showDialog(ExtendedDialog.java:258)
	at org.openstreetmap.josm.gui.tagging.presets.TaggingPresetSearchDialog.showDialog(TaggingPresetSearchDialog.java:46)
	at org.openstreetmap.josm.actions.TaggingPresetSearchAction.actionPerformed(TaggingPresetSearchAction.java:33)
	at java.desktop/javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1810)
	at java.desktop/javax.swing.JComponent.processKeyBinding(JComponent.java:2900)
	at java.desktop/javax.swing.KeyboardManager.fireBinding(KeyboardManager.java:311)
	at java.desktop/javax.swing.KeyboardManager.fireKeyboardAction(KeyboardManager.java:266)
	at java.desktop/javax.swing.JComponent.processKeyBindingsForAllComponents(JComponent.java:2993)
	at java.desktop/javax.swing.JComponent.processKeyBindings(JComponent.java:2985)
	at java.desktop/javax.swing.JComponent.processKeyEvent(JComponent.java:2862)
	at java.desktop/java.awt.Component.processEvent(Component.java:6412)
	at java.desktop/java.awt.Container.processEvent(Container.java:2263)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5011)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4843)
	at java.desktop/java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1950)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:870)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1139)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:1009)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:835)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4892)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2772)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4843)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:772)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:743)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

[mnalis]

Um, actually it did happen before too https://josm.openstreetmap.de/ticket/21829

[skyper]

Maybe even #21654.
Interesting that the node is even not selected or do I misinterpret your report?
Once again I find org.openstreetmap.josm.gui.tagging.presets.TaggingPresetValidation.validateAsync in the log. Disabling the option in Tagging Presets preferences should at least quiet JOSM and allow to continue the work.
Does it happen with plain JOSM and without additional validator rules?
Is it reliable reproducible? I would be happy about a data or session file which triggers the exception all the time. Right now, we try to find the needle in the hay stack.

[mnalis]

Interesting that the node is even not selected or do I misinterpret your report?

perhaps I was too terse, sorry about that. Here is more detailed version:

• I do a long working session in JOSM (at least an hour). During that session I try uploading several times and fix things validators warn about and try uploading etc.
• when I clean up all upload warnings, I finally do upload the changes to OSM (in Changeset 133458374)
• after that I CLOSE existing data layer (so only aerial imagery / Notes / gpx layers remain)
• after that I download new data layer to create new session
• then I press "a" to create new node (which is thus selected)
• then I press "F3" (in attempt to mark new node as a restaurant), but the bug window pops up

what I found interesting and tried to emphasize is that the topmost error message "org.openstreetmap.josm.data.osm.DataIntegrityProblemException: Primitive must be part of the dataset: {Node id=10722374641 version=1 VT lat=44.04921097061714,lon=15.312376568276045}" references "Node id=10722374641",

And that specific node at ​https://www.openstreetmap.org/node/10722374641 is (I verified) the node I was working on in *previous* session in data layer which has been *CLOSED*. So nothing in JOSM should have any idea about that specific node id, IMHO.
I would expect any validators to run/complain only about *currently open* data layer and nodes I'm editing in it (which would be new node with id=0 or whatever new ID it was supposed to get, but *not* 10722374641).

So to me it seems like validator somehow try to validate the node that is no longer existing in current data layer. I have no idea on architecture of JOSM, but random guess would be that maybe there might be same cache of elements used by validators which is not cleared correctly when data layer is closed?

Once again I find org.openstreetmap.josm.gui.tagging.presets.TaggingPresetValidation.validateAsync in the log. Disabling the option in Tagging Presets preferences should at least quiet JOSM and allow to continue the work.

Um, disabling which node exactly? I can try disabling it of cource, but where is it? In F12 / "Tagging Presets" I only have checkboxes for "Run data validator on user input" and "Sort presets menu alphabetically". Is it one of those? (I've also tried looking in "Advanced preferences" for "async", but only found "asynchronous.upload=true").

Does it happen with plain JOSM and without additional validator rules?

I don't know. It happens quite rarely after a lot of working hours in it, and while I understand that being able to reproduce it in clean environment would be helpful, I would really regret having to work without extra bells and whistles for so long.

Is it reliable reproducible?

Unfortunately not. I seems to happen at random (or at least I was unable to find the pattern nor reproduce at will) with about several weeks in between occurrences (so far).

I would be happy about a data or session file which triggers the exception all the time. Right now, we try to find the needle in the hay stack.

I know the feeling. Thanks for the tip, I'll try to save session next time and see if I can reproduce it from that.

[skyper]

Replying to mnalis:

Once again I find org.openstreetmap.josm.gui.tagging.presets.TaggingPresetValidation.validateAsync in the log. Disabling the option in Tagging Presets preferences should at least quiet JOSM and allow to continue the work.

Um, disabling which node exactly? I can try disabling it of cource, but where is it? In F12 / "Tagging Presets" I only have checkboxes for "Run data validator on user input" and "Sort presets menu alphabetically". Is it one of those?

Oh sorry, yes, it is "Run data validator on user input".