MapCSSRendererTest fails on Ubuntu 18.04

[Don-vip]

As I feared with #16688, org.openstreetmap.josm.gui.mappaint.MapCSSRendererTest does not behave the same on Ubuntu 16.04 and Ubuntu 18.04.

https://josm.openstreetmap.de/jenkins/job/JOSM/lastCompletedBuild/testReport/

I don't know why, but this is problematic as we need it to run on both versions now we have updated to 18.04 but Travis CI is still using 16.04.

Test	Reference	Result	Difference
area-text
eval
node-text
node-text2
way-text

[michael2402]

Do we know which JVM versions were used in both cases? I think I noticed differences between SUN and OpenJDK some time in the past.

[ris]

I've always had problems with these tests locally FWIW.

[michael2402]

@ris Wich JVM are you using and which OS?

The problem seems to be that they differ in rendering and the tests are doing a hard comparison of the images. We could add a threshold but judging visual similarity using a simple algorithm is difficult.

[Don-vip]

JVM is the same:
​https://packages.ubuntu.com/xenial/amd64/openjdk-8-jre
​https://packages.ubuntu.com/bionic/amd64/openjdk-8-jre

I think it comes from another package.

[michael2402]

I will test it

Does anyone know the ant equivalent to the gradle --tests parameter (that only executes specific tests)?

[ris]

OpenJDK 8, linux.

@michael2402 incidentally I added a tool to the testutils recently for making lenient checks on simple image content - ImagePatternMatching: ​https://github.com/openstreetmap/josm/blob/46e9e88b6de60be69bba0e8615d93c85222fef89/test/unit/org/openstreetmap/josm/testutils/ImagePatternMatching.java

Essentially allows you to make assertions about strips of the image using regular expressions.

[Don-vip]

Replying to michael2402:

Does anyone know the ant equivalent to the gradle --tests parameter (that only executes specific tests)?

It's a bit complicated right now, see wiki/DevelopersGuide/Compiling

It used to be easy from Eclipse (right click on test, "run as Junit") but currently more difficult, see #17012

[Don-vip]

@team does anyone have a lead to fix the issue? The build is failing since we updated the server to latest version of Ubuntu.

[stoecker]

When the test differs for Java version why not add different reference test data after verifying that the new data is only different, but not wrong?

[michael2402]

We can just re-generate all data using the new configuration. Then visually check if it looks OK. Those are just problems with anti-aliasing and some rounding issues that are one pixel of.

The test is mostly there to be able to see if some map features just got "lost" during a refactoring (e.g. labels not being rendered, ...). For that, we don't need to be able to run it in every environment, although it would be nice.

[Don-vip]

In 14491/josm:

fix #16973 - update mapcss reference images for text to match output we get on Ubuntu 18.04

[ris]

I don't think it's ever a good idea to expect pixel-perfect results from a test unless you're in control of the whole rendering process. In this case we defer the font rendering to swing, which in turn defers it I imagine to a platform-dependent font renderer, in this case freetype.

Tying the test to bit-perfect results will also make a test platform-dependent and probably making local results or travis results rather valueless. I've done this using my ImagePatternMatching tool, but in this case it might be more useful to use a "perceptual" diff or hash like e.g. ​http://pdiff.sourceforge.net/ or ​https://www.phash.org/