Modify

Opened 13 years ago

Closed 13 years ago

Last modified 3 years ago

#7159 closed defect (fixed)

Layer merging performance

Reported by: Don-vip Owned by: team
Priority: normal Milestone:
Component: Core Version: latest
Keywords: Cc:

Description (last modified by Don-vip)

The "merge layer" action can be slow as hell. With these two data sets, including only new primitives:

  • A: 30.000 highways, 80.000 nodes, 9.000 relations associatedStreet
  • B: 65.000 addr:housenumber

The loading of each data set takes only few seconds. However if I try to merge them, I'm forced to cancel the operation after several minutes of processing without any visual indication of the remaining time.

Two things to do here:

  • Speed up the process, it shouldn't take so long (I have added some traces to measure performances, JOSM merges nodes at the incredible speed of 40 nodes per second !)
  • Indicate in a progress bar the advancement (currently, there is no progress bar at all)

Attachments (1)

7159.patch (10.4 KB ) - added by Don-vip 13 years ago.
patch awaiting end of stabilization phase

Download all attachments as: .zip

Change History (8)

comment:1 by Don-vip, 13 years ago

Description: modified (diff)

by Don-vip, 13 years ago

Attachment: 7159.patch added

patch awaiting end of stabilization phase

comment:2 by Don-vip, 13 years ago

The merging speed bumped in my examples from 40 to 300 nodes per second just by:

  • using a XOR in OsmPrimitive.hasEqualSemanticAttributes()
  • checking if arrays are null or of the same size before creating and comparing the key maps in AbstractPrimitive.hasSameTags()
  • looping on initial target layer primitives instead of current target layer primitives (there's no sensr to check for current primitives as they include primitives we have already merged from the source, right ?)

comment:3 by Don-vip, 13 years ago

In [4684/josm]:

see #7159 - Layer merging performance

comment:4 by Don-vip, 13 years ago

Resolution: fixed
Status: newclosed

comment:5 by taylor.smock, 3 years ago

In 18464/josm:

Extract equalsEpsilon from LatLon into ILatLon

This significantly reduces allocations from DataSetMerger#merge.
In testing, it went from ~400 MiB allocations to ~19 MiB allocations.

The allocations largely came from checking if two nodes had equal coordinates.
The code used LatLon#getCoor followed by equalsEpsilon, when all the
required information was already present in Node, but LatLon had the method.

As an additional enhancement, equalsEpsilon was somewhat generified, in that
there is an additional method where we can specify a different precision.

This fixes #22104, see #7159.

comment:6 by Don-vip, 3 years ago

Nice!

in reply to:  6 comment:7 by taylor.smock, 3 years ago

I'm taking a quick look through other methods in LatLon right now as well, looking for stuff that can/should be moved to ILatLon. I would not be surprised if some of them are being used like so node.getCoor().greatCircleDistance(otherNode.getCoor()).
looks at Geometry#getDistance, Geometry#getDistanceSegmentSegment (my fault)

Current candidates (I'll open a separate ticket for this):

  • greatCircleDistance
  • bearing
  • interpolate (maybe -- sticking point is the return value, I'd want to see how it was being used)
  • getCenter (see above)

EDIT: Ticket opened (#22115).

Last edited 3 years ago by taylor.smock (previous) (diff)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.