Small issue with precision

[naoliv]

Open the attached file, where we have 2 layers: Uploaded (already in OSM) and Before uploading

Validate both: on Before uploading we see only an informational message Building with an almost square angle while at Uploaded (already in OSM) we see a Building duplicated node error, a Crossing buildings warning and a new Overlapping Areas at other.

Together with a friend we were trying to understand why this was happening and we found that the precision of the nodes was changing.

For example, a node that locally was at -23.51108285, -46.489264256 is now at -23.5110829, -46.4892643 in OSM.

Shouldn't the validation tests (and maybe other parts of JOSM?) round the coordinates to avoid inserting data with errors like this?

JOSM:

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2019-08-23 22:53:24 +0200 (Fri, 23 Aug 2019)
Revision:15315
Build-Date:2019-08-24 01:31:00
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (15315 en) Linux Debian GNU/Linux bullseye/sid
Memory Usage: 321 MB / 2048 MB (123 MB allocated, but free)
Java version: 13-ea+30-Debian-1, Debian, OpenJDK 64-Bit Server VM
Screen: :0.0 1920x1080
Maximum Screen Size: 1920x1080
Java ATK Wrapper package: libatk-wrapper-java:all-0.35.0-3
libcommons-compress-java: libcommons-compress-java:all-1.18-2
libcommons-logging-java: libcommons-logging-java:all-1.2-2
VM arguments: [-Dawt.useSystemAAFontSettings=gasp]
Program arguments: [--language=en]
Dataset consistency test: No problems found

Plugins:
+ EasyPresets (1537621333)
+ FastDraw (34977)
+ OpeningHoursEditor (34977)
+ SimplifyArea (34977)
+ apache-commons (34908)
+ buildings_tools (34982)
+ ejml (35049)
+ geojson (124)
+ geotools (34908)
+ http2 (35062)
+ jaxb (35014)
+ jogl (1.2.3)
+ jts (35064)
+ log4j (34908)
+ measurement (35051)
+ opendata (35102)
+ reverter (35084)
+ tageditor (34977)
+ tagging-preset-tester (34908)
+ todo (30306)
+ turnlanes-tagging (281)
+ turnrestrictions (34977)
+ undelete (34977)
+ utilsplugin2 (35098)
+ wikipedia (v1.1.1)

[naoliv]

#14053 is partially related with this

[simon04]

The rounding in JOSM and on the OSM server yields slightly different results (-23.5110829 vs. -23.5110828):

Input  -23.51108285,  -46.489264256
OSM DB -23.5110829,   -46.4892643
JOSM   -23.5110828,   -46.4892643    via org.openstreetmap.josm.data.coor.LatLon#getRoundedToOsmPrecision
JOSM*  -23.511083,    -46.489264     new via org.openstreetmap.josm.data.validation.tests.DuplicateNode.NodeHash#roundCoord

Input  -23.511082861, -46.489264251
OSM DB -23.5110829,   -46.4892643
JOSM   -23.5110829,   -46.4892643    via org.openstreetmap.josm.data.coor.LatLon#getRoundedToOsmPrecision
JOSM*  -23.511083,    -46.489264     new via org.openstreetmap.josm.data.validation.tests.DuplicateNode.NodeHash#roundCoord

Let's reduce the precision used by the duplicate node test by one decimal digit.

The advanced preference value validator.duplicatenodes.precision now defaults to 1e-6 (previously 1e-7).

[simon04]

In 17612/josm:

fix #18074 - Duplicate Node: detect errors caused by different rounding in JOSM and on OSM server

Reduce the precision used by the duplicate node test by one decimal digit

The advanced preference value validator.duplicatenodes.precision now defaults to 1e-6 (previously 1e-7).

[simon04]

Documentation: Help/Preferences/Validator?action=diff&version=60

[GerdP]

Hmm, the default for validator.duplicatenodes.precision was 0.0, not 1e-7
I don't see how this will fix a problem with crossing lines or overlapping areas. It just relaxes the duplicate node test, right?

[simon04]

Replying to GerdP:

Hmm, the default for validator.duplicatenodes.precision was 0.0, not 1e-7

The default comes from ​https://github.com/openstreetmap/josm/blob/094d5522ac6f9ce00fece7fe64ac8a6bc5c84d52/src/org/openstreetmap/josm/data/validation/tests/DuplicateNode.java#L70

I don't see how this will fix a problem with crossing lines or overlapping areas. It just relaxes the duplicate node test, right?

Yeah.

[simon04]

In 17624/josm:

see #18074 - Javadoc warnings

[GerdP]

I think this should be reverted.
I get lots of false positives from this test now. Try eg ways w43266057 w88667177. Distance between the nodes is 4cm, not much, but JOSM clearly shows a gap.

[GerdP]

BTW: Using the value 0.0 for validator.duplicatenodes.precision results in precision being 1/0.0 = Double.infinite and completely weird results.

[simon04]

Thanks for your extensive testing. I'll revert.

[simon04]

In 17665/josm:

see #18074 - Revert "Duplicate Node: detect errors caused by different rounding in JOSM and on OSM server"

This reverts commit r17624.

[smootheFiets]

I ran into what looks like the same issue yesterday. I've uploaded self-intersecting building geometries. I did run the JOSM validator before uploading, it didn't flag the issue. After deleting the data layer and re-downloading an overlapping area, the JOSM validator duly warned "self crossing ways." Expected behaviour: JOSM validator should've given that warning before uploading.

Two examples: ​https://www.openstreetmap.org/way/1003961672 (revert to v1 to see the self-intersection) and ​https://www.openstreetmap.org/way/587543281 (v2 shows the self intersection)

I encounter the same issue with other geometry errors, e.g. building duplicate nodes or building overlaps: no warning before uploading, warning after a fresh download. I can provide examples on request.

I suspect, like naoliv, that the issue is with the finite numerical precision of floats (float32 vs float64?). I don't see what else can go wrong between uploading and re-downloading the same area. In the case of the self-intersections I give above, it's enough to move one node by << 1mm to make the issue go away. So, we're really talking edge cases. But still, I get them on a regular basis. I import building outlines from the Dutch cadaster, a couple thousand in an average week. I get ~a handful of false negatives per week that slip through the cracks of the JOSM validator (which is really good at finding most issues in the data, truth be told).

I don't know to which precision the OSM servers store floats. But surely that can be found out. Would it be possible to round all coordinates in JOSM to that same precision before executing any validator checks? That should make validator results more reproducible, I guess.