Modify

Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#22032 closed defect (fixed)

[PATCH][RFC] Various memory enhancements for MVT tiles

Reported by: taylor.smock Owned by: team
Priority: normal Milestone: 22.06
Component: Core Version:
Keywords: performance Cc:

Description (last modified by taylor.smock)

JOSM start arugment: --download=39.0637818,-108.5670233,39.0660809,-108.5620022
Plugins: Mapillary (shift+, to download MVT tiles)

Memory allocations over 50s for r18430, limited to threads with protobuf related methods:

  • Run 1: 8,979,920,352 bytes
  • Run 2: 8,773,894,032 bytes

Memory allocations over 50s for r18431, limited to threads with protobuf related methods

  • Run 1: 7,653,697,192 bytes
  • Run 2: 7,133,589,400 bytes
  • Run 3: 7,532,364,336 bytes

I'll go ahead and merge this patch on Monday, absent objections.

Memory allocations over 50s for r18431 + patch, limited to threads with protobuf related methods:

  • Run 1: 823,974,256 bytes
  • Run 2: 850,025,712 bytes
  • Run 3: 756,248,936 bytes

Most of the performance enhancements come from:

  1. Avoiding various ArrayList#grow calls
    • A lot is from passing around a `ByteArrayOutputStream
  2. Caching bboxes in VectorNodes (I'm kind of ambivalent on this -- I don't want to keep the bboxes around, but a significant number of allocations came from QuadBuckets#search, of which a significant number of allocations came from VectorNode#getBBox, ~1,400,000,000 bytes, or ~20%)
  3. Caching nodes in a Map<ILatLon, VectorNode> map to avoid calling QuadBuckets#search when possible

The patch additionally fixes a bug with VectorRelation#setMembers and adds a default method putAll for Tagged.

Flamegraph (green == removed allocations, red == added allocations)

Attachments (2)

22032.patch (55.9 KB ) - added by taylor.smock 3 years ago.
MVT performance improvements
flamegraph_22032.png (404.2 KB ) - added by taylor.smock 3 years ago.
Flamegraph (green == removed allocations, red == added allocations)

Download all attachments as: .zip

Change History (8)

by taylor.smock, 3 years ago

Attachment: 22032.patch added

MVT performance improvements

comment:1 by taylor.smock, 3 years ago

Description: modified (diff)

by taylor.smock, 3 years ago

Attachment: flamegraph_22032.png added

Flamegraph (green == removed allocations, red == added allocations)

comment:2 by taylor.smock, 3 years ago

Description: modified (diff)

comment:3 by taylor.smock, 3 years ago

r18465 removed the need to cache bboxes in VectorNodes.

comment:4 by taylor.smock, 3 years ago

Resolution: fixed
Status: newclosed

In 18473/josm:

Fix #22032: Various memory enhancements for MVT tiles

There is about a 88% decrease in memory allocations due to MVT tiles.

The majority of changes came from the following:

  1. Avoiding ArrayList#grow calls by using a ByteArrayOutputStream that gets passed around
  2. Caching nodes to avoid QuadBuckets#search calls

This additionally fixes a bug where VectorRelation#setMembers does not
work properly and adds a putAll method in Tagged (there is a default
method).

comment:5 by taylor.smock, 3 years ago

In 18477/josm:

Fix tests broken in r18473, see #22032

This was caused by vector tiles not always having points,
then ways, then polygons. The code now ensures that we
add points first, followed by ways, and finally by polygons.

comment:6 by taylor.smock, 3 years ago

Milestone: 22.06

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.