[Patch] Using the ignore option on certain validation items does not work

[anonymous]

What steps will reproduce the problem?

1. Download this area ​https://www.openstreetmap.org/#map=20/53.74478626711375/-1.4791523935342819
2. Perform validation (Shift+V)
3. A validation error will show for Similarly named ways (Swithens Grove vs Swithens Drive)
4. Select the validation error and click Ignore
5. Perform validation again.
6. The same validation error reappears.

What is the expected result?

The validation error should not be shown when subsequently performing validation, as it has been explicitly ignored.

What happens instead?

The validation error shows every time validation is performed, despite it having been ignored.

Please provide any additional information below. Attach a screenshot if possible.

This happens for more than this type of error, it also happens for "way end near other highway" amongst others. Ignoring whole groups works as expected, and ignoring instances of some other error types does work. This seems to have been happening since the option of "Ignore for now" was added.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2019-06-03 00:05:30 +0200 (Mon, 03 Jun 2019)
Build-Date:2019-06-02 22:10:24
Revision:15155
Relative:URL: ^/trunk

Identification: JOSM/1.5 (15155 en_GB) Windows 10 64-Bit
OS Build number: Windows 10 Home 1803 (17134)
Memory Usage: 522 MB / 910 MB (354 MB allocated, but free)
Java version: 1.8.0_60-b27, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Screen: \Display0 1600x900
Maximum Screen Size: 1600x900
Dataset consistency test: No problems found

Plugins:
+ PolygonCutOut (v0.7)
+ buildings_tools (34982)
+ imagery_offset_db (34908)
+ splinex (34908)
+ turnrestrictions (34977)
+ utilsplugin2 (34977)

Tagging presets:
+ D:\My Documents\Miscellaneous\JosmPresets.xml

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/Lane_and_Road_Attributes&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Maxspeed&zip=1

Last errors/warnings:
- W: Bing: attribution data is not yet loaded.
- W: java.io.IOException: Attribution is not loaded yet
- W: Bing: attribution data is not yet loaded.
- W: java.io.IOException: Attribution is not loaded yet
- W: Bing: attribution data is not yet loaded.
- W: java.io.IOException: Attribution is not loaded yet
- W: Bing: attribution data is not yet loaded.
- W: java.io.IOException: Attribution is not loaded yet
- W: Bing: attribution data is not yet loaded.
- W: java.io.IOException: Attribution is not loaded yet

[Arthtoach]

Screenshot of example validation error

[Arthtoach]

Crossing waterway/highway is also affected. Possibly anything involving multiple ways...?

[Famlam]

Probably everything with a quote or other special character (':[]) in it, rather than just plain (alphabetical) text

[mdk]

I got the same problem with ignoring "whole group" for "Preset do not contain property key" for the keys "swisstopo:BEZIRKSNUM", "swisstopo:BFS_NUMMER", "swisstopo:KANTONSNUM", "swisstopo:OBJEKTART" and "swisstopo:SHN".
The problem is: when I ignore the "whole group" of any of this validation messages, they DON'T appear in the "Validator Ignore List Management" and so on next validation the messages appear again.
To get a more detailed picture, I manually removed all entries in this list with "don't ignore". Then I start again with the empty list and - supprise - it works. Now I can add "whole group" again, they are shown in the list and the exclusion also works for next validation.
I think there is a problem with old ignore rules in the list. I think the list is persisted and read on startup. The problem is, that I now have a working list, so I can't send you my old persisted ignore list, btw. where is this information stored? Perhaps the user created this issue still has his old file and may attache it!

[mdk]

I forgott that before clearing the list I could'n add "carpenter=framer" (which now works). So I think there is not a direct problem with special characters, as Famlan assumed.

[GerdP]

Obviously a problem introduced with #17268, but I cannot reproduce it right now.

The old "ignored error" list was stored in a file named ignorederrors, located in sub directory validator.
I kept a copy in
c:\Users\Gerd\AppData\Roaming\JOSM\validator\ignorederrors_
with a content like

3000_way[highway=unclassified][!name][noname'NEQ'yes]
303
3701
614

The new list is stored in the preferences, search for validator.ignorelist.

Special case: If you ignore an error that concerns a new object the information is not stored in the list.

I suggest to create a backup of the complete JOSM directory (the one that contains the preferences.xml) before doing any experiments.

[mdk]

Good news: I found a way to reproduce the problem.

I have an older JOSM version, where I found the ignorederrors_ file. It looks like this file is converted into validator.ignorelist some time ago, because list and preferences are not compleatly identical. In JOSM the values from preference are shown, so the old file seams to be ignored.
In the preference.xml I found

  <maps key='validator.ignorelist'>
    <map>
      <tag key='111' value='Ignore list'/>
      <tag key='113' value='Ignore list'/>
      <tag key='120' value='Ignore list'/>
      <tag key='1260_maxspeed without highway' value='Ignore list'/>
      <tag key='1270_street name contains ss' value='Ignore list'/>
      <tag key='2201' value='Ignore list'/>
      <tag key='2301' value='Ignore list'/>
      <tag key='3000_*[name&apos;REGEX&apos;(?i).*Strasse.*]' value='Ignore list'/>
    </map>
  </maps>

I replaced the ignorelist in my actual preference.xml with the above part. As a result, I can't add new ignore groups for 1201 like:

<tag key='1201_Key &apos;swisstopo:BEZIRKSNUM&apos; not in presets.' value='Presets do not contain property key'/>

I tried a little bit around and find out, that when I removed

<tag key='120' value='Ignore list'/>

from the list, I could add 1201 again.
Could it be, that key=120 is no longer a valid rule? Does it is important, that key=1201 should be inserted in this list after key=120? Hope this help you.

[mdk]

The problem about ignoring crossing waterway/highway from Arthtoach (see comment:2:ticket:17837) looks like it have a differnt cause, so I create a new issue #18223 for that.

[GerdP]

Patch to make sure that two entries have the same error code, so that "120" is not treated as group for an entry starting with "1201_Key ". There is probably a simpler way to express this with regex patterns, but I couldn't find one.

[mdk]

A greedy matcher would looks like this: (the pattern can be prepared once)

    Pattern pattern = Pattern.compile("\\d*");

    Matcher matcher = pattern.matcher(input);
    matcher.find();
    String code = matcher.group();

This could replace String extractCodeFromIgnoreKey(String key) in your patch.

[GerdP]

I found something similar but did not like it. I think the small loop is easier to understand, so I'll probably keep that solution unless the work on #18223 requires a different approach. Up to now I think they can be fixed separately.

[GerdP]

In 15459/josm:

fix #17837 and #18223, add unit tests

TODO:

• Will havily blow up size of preferences.xml when user ignores lots of "single elements" instead of the group.
• Maybe ignore entries in preferences with many (how many?) elements, as those probably were caused by error #18223 and are meaningless