[PATCH] Enhanced GPS data rendering - heatmaps instead of dozens of lines

[transmit-receive]

Problem: If many GPS tracks are available for a region, dozens of lines are rendered on top of each other. It is hard to discern where the density of lines is highest, i.e. where the track most likely is centered.

Solution: Draw a heatmap of the GPS track data instead of many lines (something similar to e.g. ​http://labs.strava.com/heatmap/#6/-120.90000/38.36000/blue/bike)

Proposed Algorithm:

• Create array of dimension x*y as big as current field of view
• For each line segment of GPS tracks in view, check which pixels of the current field of view are touched by this line segment, increase counter in these array cells by one
• add semitransparent colormapped layer to display from array where 0=transparent, [1 .. max(array)] = color

Additional thoughts:

• Use a perceptionally uniform colormap (e.g. like 'inferno' in matplotlib ​http://matplotlib.org/users/colormaps.html) instead of misleading 'jet' or 'rainbow' colormap (​https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/)
• this may be slow and my proposed algorith may be crap. feel free to improve

[Klumbumbus]

Can you give an example region, please?

[stoecker]

A German "Autobahn" and neraby area, e.g. in the Ruhrgebiet should fit the requirements :-)

[transmit-receive]

See e.g. here ​http://imgur.com/a/sKDPC
A heatmap would alleviate this mess of lines.

Maybe calculate a density of lines heatmap with e.g. 1m resolution for all the GPS traces after downloading them and store that in cache, so the calculation has not to be done every time the window is moved.

[kidelo]

GPX Heat Map Extension by kidelo

[kidelo]

Happy New Year JOSM-Developers,

during my holiday at the end of last year, i had some hours / days left for realization of the requested feature, see the attached patch file.

If it looks acceptable, please merge it into the JOSM development branch.

Regards Kidelo

Features

• HeatMap for GPS traces
  • dynamic adjustment of GPS trace visibility based on zoom level
  • block size depends on zoom level and line width ( normal adjustment via line width possible )
  • background becomes visible for higher zoom levels automatically ( borders of block are drawn only )
  • 4x color maps and 1x user adjustable color map
  • can't combined with GPX Alpha Blending mode
• GPX Alpha Blending mode
  • Another alpha blending mode as alternative to the heat map or normal layer transparency
  • use the normal GPX color settings
  • can be used with bigger GPS trace points and with direction arrows
  • Only in "Expert Mode"

ColorMap

User : Derived Color Map based on user color ( use same layer / global color for GPS traces )
Inferno : Similar to "Inferno" Color Map ( more brighter values )
Viridis : Similar to "Viridis" Color Map ( more brighter values )
Wood : Brown-Green Color map
Heat : Red-White-Blue Color map

Example images

GPX View Normal:

Heatmap - Colormap: Inferno - Zoom 400m

[stoecker]

Looks really cool.

Comments about the code from the others?

I see that JavaDoc is missing (should also been done properly for the private functions), but otherwise the code seems good on first view.

[Don-vip]

waw this looks awesome!
The code is very clean, I'm going to test and apply the patch right now.
Thank you for this gift!

[Don-vip]

The patch doesn't work nice with the attached sample:

Then some remarks before applying the patch:

1. the color map modes take a lot of lines, it would be cleaner to extract them to resource text files in a new folder data/gpx (data/gpx/inferno.txt, etc.) and load them at runtime
2. Some code conventions:

• don't use default values: replace private boolean heatMapEnabled = false; by private boolean heatMapEnabled;, private int heatMapDrawColorTableIdx = 0; by private int heatMapDrawColorTableIdx;, etc.
• use diamond operator, replace new ArrayList<Integer>() by new ArrayList<>()
• use proper javadoc format for added methods

3. drawHeatMap() is huge (221 lines), it could probably be refactored into smaller methods

[kidelo]

Please check the width of line in configuration settings. The method drawHeatMap() does nothing when line width is 0.
I will harmonize the behavior in next step.

[kidelo]

Feedback to remarks:

1. a good idea, i will check it
2. ok, i must just something learn about your style.
3. yes i know. I think it is possible to move the processing parts of code into separated methods

[kidelo]

GPX Heat Map Extension by kidelo, version 2

[kidelo]

I have reworks some parts of the patch, see attached version 2

• add javadoc to new and old functions
• fix some issues with coding style
• move lookup tables for color maps from class code to resources /data/gpx/
• Remove of some debug logs
• restructure code of drawHeatmap()
• adjust behavior for line width = 0

[kidelo]

Now its works with default configuration settings:

[Don-vip]

In 11431/josm:

fix #13124 - Enhanced GPS data rendering: add new heatmap mode (patch by kidelo)

[Don-vip]

It's perfect! Thanks a lot for this really good stuff :)

[stoecker]

Played with it a bit. Really, really nice.

Two things I noticed:

a) The heatmap ends before the window border resulting in an strange effect at the border when moving the window. Probably it should take a bit larger area into account for the calculations? Tried with a German motorway, where lots of data exists.

b) When you zoom in you loose the overview. E.g. properly separated lanes get a bunch of tracks again. Maybe a minimum setting would be good idea below which the automatic does no longer gets more detailed? For me that was when about 200m was displayed as zoom factor for a German highway and 50m for a third level road. Or a "keep zoom" (or better a "no deeper zoom") feature like for WMS?

[kidelo]

GPX Heat Map Extension by kidelo, version 3

[kidelo]

Some news:

a.) There was several small bugs. I have fixed them, now it works as expected.
b.) The idea behind the automatic algorithm is to get similar results at any zoom layer.

The amount of details for each zoom layer can be adjusted by the normal GPX line width parameter.

• line width = 1 : more separate lines in zoom window. This is useful if you like to separate traffic lanes from each other.
• line width = 2..4 : the average area becomes bigger therefore the tracks are merged.
• line width = > 5 : more overview, very few details

In addition i have enabled an feature again, that was lost during refactoring. If the rectangles become very large with a higher zoom level, only the contour is drawn. So it is possible to see behind" the heat map, with an additional zoom-in. See the example pictures
Last but not least, i have slightly adjusted the parameters of the blending algorithm.

line width 1 / line width 2

line width 3 / line width 5

line width 5 + Zoom in / line width 5 + extra layer

[Don-vip]

In 11451/josm:

fix #13124 - Heat Map Extension (patch by kidelo)

[Don-vip]

thanks! :)

[aceman]

Is the space after "(alpha blending)" needed?

[bastiK]

Pretty impressive!

Some issues I see: Even after [11451], the heat map will not fill the entire mapview. In particular, the call

final Rectangle screenBounds = g.getDeviceConfiguration().getBounds();

returns the same value throughout a session, even when the window size is changed.

Another thing is that in

        final boolean imageRecalc = heatMapCacheVisibleSegments != visibleSegments.size() ||
                                    heatMapCacheZoomScale != zoomScale ||
                                    heatMapCacheLineWith != globalLineWidth;

the comparison heatMapCacheLineWith != globalLineWidth is always true, because in the default settings heatMapCacheLineWith and lineWidth is 0 but globalLineWidth is 1. This means imageRecalc will also be true each time.

How it is done with MapView.offscreenBuffer might be worth a look.

[kidelo]

Replying to aceman:

Is the space after "(alpha blending)" needed?

Not really, we are free to change it. Some other opinions ?

[bastiK]

See #14244, #14247.

[kidelo]

Replying to bastiK:

Pretty impressive!

Some issues I see: Even after [11451], the heat map will not fill the entire mapview. In particular, the call

final Rectangle screenBounds = g.getDeviceConfiguration().getBounds();

returns the same value throughout a session, even when the window size is changed.

Another thing is that in

        final boolean imageRecalc = heatMapCacheVisibleSegments != visibleSegments.size() ||
                                    heatMapCacheZoomScale != zoomScale ||
                                    heatMapCacheLineWith != globalLineWidth;

the comparison heatMapCacheLineWith != globalLineWidth is always true, because in the default settings heatMapCacheLineWith and lineWidth is 0 but globalLineWidth is 1. This means imageRecalc will also be true each time.

How it is done with MapView.offscreenBuffer might be worth a look.

some changes:

1.) fixed, i have found an additional algorithm error where the gray images are mapped to the target image
2.) globalLineWidth != lineWidth -> fixed

[kidelo]

GPX Heat Map Extension by kidelo, version 4

[bastiK]

In 11469/josm:

see #13124 - fix some Heat Map issues (patch by kidelo)

[bastiK]

In 11470/josm:

see #13124 - better cache invalidation

Fixes a problem when zoomed out, so all gpx data is in the center of the mapview.
Then moving the map did not update the gpx layer.

[bastiK]

Thanks, I've changed the cache invalidation a bit.

One more issue is that it doesn't seem to work in WGS84 projection. You may find mv.getDist100Pixel() useful to replace mv.getScale() in the calculations.

[kidelo]

Replying to bastiK:

Thanks, I've changed the cache invalidation a bit.

equalsInWindow(MapViewState other) -> really the better solution to ask the mapview. it knowns it when it was changed;

One more issue is that it doesn't seem to work in WGS84 projection. You may find mv.getDist100Pixel() useful to replace mv.getScale() in the calculations.

i will investigate it in next time

[kidelo]

GPX Heat Map Extension by kidelo, version 5

[kidelo]

new patch that fix the problem with the different projections, thanx to bastiK for the hint.

Projection Merkator

Projection WGS84

[Don-vip]

In 11482/josm:

see #13124 - fix the problem with the different projections (patch by kidelo)

[bastiK]

Ticket can be closed?

[Don-vip]

I think so :)

[michael2402]

In 12157/josm:

See #13124: Refresh the heat map on every invalidation (visible tacks changed, ...)