Modify ↓

Opened 16 years ago

Last modified 21 months ago

#2173 reopened enhancement

Show the Progress of preparing Data after downloading an Area

Reported by: Benedikt.L Owned by: team
Priority: trivial Milestone:
Component: Core Version: latest
Keywords: progress parsing Cc: taylor.smock

Description (last modified by skyper)

es wäre nützlich den Fortschritt anzuzeigen (in Prozent und als Balken).

Attachments (0)

Change History (7)

comment:1 by anonymous, 16 years ago

Summary: Anzeige des Fortschritts der vorbereiteten Daten beim Downloaden der Karte → Show the Progress of preparing Data after downloading an Area

It would be nice to have that feature specially for big areas.
Im often waiting for some mintutes don't now whats going on.
It was in the commandbox in a old version of JOSM but suddenly in a later version it vanished.

comment:2 by xeen, 16 years ago

Currently the parse operation is not threaded, so it hangs the main thread. The scrollbar actually should be intermediate (like when it says "connecting…") but it looks stalled because of this.

comment:3 by skyper, 13 years ago

Description: modified (diff)
Owner: changed from framm to team

comment:4 by jttt, 13 years ago

Is this still necessary? I did some optimizations there so it shouldn't take too long anymore.

comment:5 by gaben, 21 months ago

Resolution: → worksforme
Status: new → closed

14 years later, I think it can be closed. Hardware became fast enough for everyday people and JOSM evolved much, including optimisations.

comment:6 by gaben, 21 months ago

Cc: taylor.smock added
Keywords: progress parsing added
Resolution: worksforme
Status: closed → reopened

After closer inspection of the source code, I still found room for threading improvements in the parsing part as mentioned in comment:2.

@taylor, I saw your work around the pbf reader. Do you have any idea how threaded parsing can be implemented? The issue is that the progress bar is intermediate type, but does not move at all while doing the Rendering data set... task.

in reply to:  6 comment:7 by taylor.smock, 21 months ago

In context, I am assuming you are talking about doing the parsing on a separate thread. When I got the notification, I was thinking about multi-threaded parsing, which is a lot harder (we have to worry about concurrent modifications of collections).
For PBF, it would be a lot easier to do multi-threaded parsing (the PBF is split into "chunks" which can be processed semi-independently).

But the problem you are talking about with the Rendering data set... task is "frozen" since the rendering happens on the EDT thread; the progress monitor cannot update until it gets called on the EDT thread again.

Here are a few stack traces that I took during the freeze in Rendering data set...:

"AWT-EventQueue-0@3166" tid=0x29 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at jdk.internal.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
	  at java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
	  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
	  at sun.awt.SunToolkit.awtLock(SunToolkit.java:238)
	  at sun.java2d.pipe.RenderQueue.lock(RenderQueue.java:112)
	  at sun.java2d.metal.MTLSurfaceData.flush(MTLSurfaceData.java:472)
	  at sun.awt.image.VolatileSurfaceManager.flush(VolatileSurfaceManager.java:442)
	  at java.awt.Image.flush(Image.java:266)
	  at javax.swing.RepaintManager.getVolatileOffscreenBuffer(RepaintManager.java:1109)
	  at javax.swing.RepaintManager$PaintManager.paint(RepaintManager.java:1556)
	  at javax.swing.RepaintManager.paint(RepaintManager.java:1336)
	  at javax.swing.JComponent._paintImmediately(JComponent.java:5266)
	  at javax.swing.JComponent.paintImmediately(JComponent.java:5076)
	  at javax.swing.RepaintManager$4.run(RepaintManager.java:878)
	  at javax.swing.RepaintManager$4.run(RepaintManager.java:861)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:861)
	  at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:834)
	  at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:784)
	  at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1897)
	  at java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:318)
	  at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:-1)
	  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	  at java.awt.EventQueue$4.run(EventQueue.java:720)
	  at java.awt.EventQueue$4.run(EventQueue.java:714)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	  at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:117)
	  at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:191)
	  at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:236)
	  at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:234)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:319)
	  at java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:234)
	  at java.awt.Dialog.show(Dialog.java:1080)
	  at java.awt.Component.show(Component.java:1728)
	  at java.awt.Component.setVisible(Component.java:1675)
	  at java.awt.Window.setVisible(Window.java:1036)
	  at java.awt.Dialog.setVisible(Dialog.java:1016)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doBeginTask$3(PleaseWaitProgressMonitor.java:255)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor$$Lambda$1477/0x00000008016feec8.run(Unknown Source:-1)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doInEDT$0(PleaseWaitProgressMonitor.java:113)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor$$Lambda$1478/0x00000008016ff0e0.run(Unknown Source:-1)
	  at java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:318)
	  at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:-1)
	  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	  at java.awt.EventQueue$4.run(EventQueue.java:720)
	  at java.awt.EventQueue$4.run(EventQueue.java:714)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	  at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	  at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	  at java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

And then another one 13s later

"AWT-EventQueue-0@3166" tid=0x29 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at jdk.internal.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
	  at java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:461)
	  at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:651)
	  at java.util.concurrent.ForkJoinPool.invoke(ForkJoinPool.java:2822)
	  at org.openstreetmap.josm.data.osm.visitor.paint.StyledMapRenderer.paintWithLock(StyledMapRenderer.java:1676)
	  at org.openstreetmap.josm.data.osm.visitor.paint.StyledMapRenderer.render(StyledMapRenderer.java:1645)
	  at org.openstreetmap.josm.gui.layer.OsmDataLayer.paint(OsmDataLayer.java:543)
	  at org.openstreetmap.josm.gui.layer.AbstractMapViewPaintable$CompatibilityModeLayerPainter.paint(AbstractMapViewPaintable.java:27)
	  at org.openstreetmap.josm.gui.MapView.paintLayer(MapView.java:469)
	  at org.openstreetmap.josm.gui.MapView.drawMapContent(MapView.java:584)
	  at org.openstreetmap.josm.gui.MapView.paint(MapView.java:491)
	  at javax.swing.JComponent.paintChildren(JComponent.java:961)
	  - locked <0x3024> (a java.awt.Component$AWTTreeLock)
	  at javax.swing.JComponent.paint(JComponent.java:1137)
	  at javax.swing.JComponent.paintChildren(JComponent.java:961)
	  at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1034)
	  at javax.swing.JComponent.paint(JComponent.java:1137)
	  at javax.swing.JComponent.paintChildren(JComponent.java:961)
	  at javax.swing.JComponent.paint(JComponent.java:1137)
	  at javax.swing.JComponent.paintChildren(JComponent.java:961)
	  at javax.swing.JComponent.paint(JComponent.java:1137)
	  at javax.swing.JComponent.paintChildren(JComponent.java:961)
	  at javax.swing.JComponent.paint(JComponent.java:1137)
	  at javax.swing.JComponent.paintToOffscreen(JComponent.java:5318)
	  at javax.swing.RepaintManager$PaintManager.paintDoubleBufferedImpl(RepaintManager.java:1656)
	  at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(RepaintManager.java:1631)
	  at javax.swing.RepaintManager$PaintManager.paint(RepaintManager.java:1569)
	  at javax.swing.RepaintManager.paint(RepaintManager.java:1336)
	  at javax.swing.JComponent._paintImmediately(JComponent.java:5266)
	  at javax.swing.JComponent.paintImmediately(JComponent.java:5076)
	  at javax.swing.RepaintManager$4.run(RepaintManager.java:878)
	  at javax.swing.RepaintManager$4.run(RepaintManager.java:861)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:861)
	  at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:834)
	  at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:784)
	  at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1897)
	  at java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:318)
	  at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:-1)
	  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	  at java.awt.EventQueue$4.run(EventQueue.java:720)
	  at java.awt.EventQueue$4.run(EventQueue.java:714)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	  at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:117)
	  at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:191)
	  at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:236)
	  at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:234)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:319)
	  at java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:234)
	  at java.awt.Dialog.show(Dialog.java:1080)
	  at java.awt.Component.show(Component.java:1728)
	  at java.awt.Component.setVisible(Component.java:1675)
	  at java.awt.Window.setVisible(Window.java:1036)
	  at java.awt.Dialog.setVisible(Dialog.java:1016)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doBeginTask$3(PleaseWaitProgressMonitor.java:255)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor$$Lambda$1477/0x00000008016feec8.run(Unknown Source:-1)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor.lambda$doInEDT$0(PleaseWaitProgressMonitor.java:113)
	  at org.openstreetmap.josm.gui.progress.swing.PleaseWaitProgressMonitor$$Lambda$1478/0x00000008016ff0e0.run(Unknown Source:-1)
	  at java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:318)
	  at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:-1)
	  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	  at java.awt.EventQueue$4.run(EventQueue.java:720)
	  at java.awt.EventQueue$4.run(EventQueue.java:714)
	  at java.security.AccessController.executePrivileged(AccessController.java:778)
	  at java.security.AccessController.doPrivileged(AccessController.java:400)
	  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	  at java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	  at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	  at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	  at java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

"styled-map-renderer-26@12316" tid=0x1d0 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	 blocks styled-map-renderer-25@12315
	 blocks styled-map-renderer-27@12317
	 blocks styled-map-renderer-28@12318
	 blocks styled-map-renderer-29@12319
	 blocks styled-map-renderer-30@12320
	 blocks styled-map-renderer-31@12322
	  at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1730)
	  - locked <0x3026> (a java.util.concurrent.ConcurrentHashMap$Node)
	  at org.openstreetmap.josm.gui.mappaint.StyleCache.intern(StyleCache.java:94)
	  at org.openstreetmap.josm.gui.mappaint.StyleCache.put(StyleCache.java:52)
	  at org.openstreetmap.josm.gui.mappaint.ElemStyles.getStyleCacheWithRange(ElemStyles.java:221)
	  - locked <0x3027> (a org.openstreetmap.josm.data.osm.Node)
	  at org.openstreetmap.josm.gui.mappaint.ElemStyles.get(ElemStyles.java:150)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.add(ComputeStyleListWorker.java:158)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.visit(ComputeStyleListWorker.java:138)
	  at org.openstreetmap.josm.data.osm.Node.accept(Node.java:215)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.acceptDrawable(ComputeStyleListWorker.java:129)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.computeDirectly(ComputeStyleListWorker.java:116)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.compute(ComputeStyleListWorker.java:93)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.compute(ComputeStyleListWorker.java:34)
	  at java.util.concurrent.RecursiveTask.exec(RecursiveTask.java:110)
	  at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:387)
	  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:-1)
	  at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	  at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	  at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

"styled-map-renderer-25@12315" tid=0x1cf nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
	 waiting for styled-map-renderer-26@12316 to release lock on <0x3026> (a java.util.concurrent.ConcurrentHashMap$Node)
	  at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1726)
	  at org.openstreetmap.josm.gui.mappaint.StyleCache.intern(StyleCache.java:94)
	  at org.openstreetmap.josm.gui.mappaint.StyleCache.put(StyleCache.java:52)
	  at org.openstreetmap.josm.gui.mappaint.ElemStyles.getStyleCacheWithRange(ElemStyles.java:221)
	  - locked <0x3025> (a org.openstreetmap.josm.data.osm.Node)
	  at org.openstreetmap.josm.gui.mappaint.ElemStyles.get(ElemStyles.java:150)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.add(ComputeStyleListWorker.java:158)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.visit(ComputeStyleListWorker.java:138)
	  at org.openstreetmap.josm.data.osm.Node.accept(Node.java:215)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.acceptDrawable(ComputeStyleListWorker.java:129)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.computeDirectly(ComputeStyleListWorker.java:116)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.compute(ComputeStyleListWorker.java:93)
	  at org.openstreetmap.josm.data.osm.visitor.paint.ComputeStyleListWorker.compute(ComputeStyleListWorker.java:34)
	  at java.util.concurrent.RecursiveTask.exec(RecursiveTask.java:110)
	  at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:387)
	  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:-1)
	  at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	  at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	  at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

So, in order to fix the freeze at that step, we would have to do the rendering in a non-EDT thread, which is contraindicted by the Swing threading model (all painting of components must occur on the EDT thread).

We could probably work around that (a bit) by treating the mapview as tiles, and we render each tile to image in a background thread, and then when the tile is ready trigger a repaint of that tile (essentially make the data layer another imagery layer).

That would be a lot of work though. I don't know if anyone has the time to do that. There are probably better uses of time (for example, it would be a lot easier to add something to mapcss that would indicate the "importance" of an object, and then only paint items with an importance > x at certain zoom levels, especiallh when there is a lot of painting to do). Another thing that would help would be to reduce the number of vertices painted (right now, a way with more than one node in a pixel will draw all nodes in that pixel -- this is more important when at low zoom levels, when a way may have all of its nodes in the same pixel).

With that said, the bits that are taking the longest are occuring in separate threads; we are just blocking the EDT while we wait for the background processing to happen. Ideally, we would paint only the objects we have computed styles for, but this is typically only a problem with large datasets/when at low zoom levels.

As a sanity check, I profiled Washington D.C. (loaded from a geofabrik pbf file), and panned the map around.
The ComputeStyleListWorker (the reason for the block on the Rendering data set... progress monitor) task took ~11% of the CPU time. 38% was in paintRecord, 3.7% was spent looking for nodes, and 3.5% was spent drawing virtual nodes. ~20% of the CPU time was spent in GC.

Looking at the recorded timeline, it appears there were ~13 paints during the 43 seconds I was profiling (based off of styled-map-renderer thread activity). Those took relatively little time (<1s). This is due to caching done earlier.

When I profiled first load, the styled-map-renderer threads were busy for ~7s (min) to ~10s (max) and ~20% of the CPU time was spent in GC.

Something that we can try doing is a progressive style computation. So we process however many objects we can in 10ms (60fps would be 16.6ms/frame), and then wait for the next paint call.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as reopened The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user. Next status will be 'new'.
Next status will be 'needinfo'. The owner will be changed from team to Benedikt.L.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.

Add Comment
E-mail address and name can be saved in the Preferences .
Attachments ↑ Description ↑
 
Note: See TracTickets for help on using tickets.