[PATCH] Add PBF reading support to JOSM

[taylor.smock]

This is a followup of #6886.

Major differences:

• It uses the JOSM Protobuf parsing code instead of Google's Protobuf parser
• It is not based off of any of the code from ​https://github.com/openstreetmap/OSM-binary except for the proto files. I'm unsure what the licensing situation would require.
• ~1500 added lines
• Performance for PBF reading when compared against the PBF plugin
  • ​colorado-latest.osm.pbf was used for profiling. Note: Takes ~8.8 GB of memory to load the file, no clue how much is needed to render it.
  • PBF plugin (max memory in use: 8779 MB):
    • Time spent loading PBF: 49.5s (3 runs with same time spent)
    • Memory allocations: 30,155,126,741 (average across three runs, ~500 MB spread)
    • CPU cyles: 3061 (average across 3 runs, ~600 CPU cycle spread)
  • Patch (max memory in use: 8783 MB):
    • Time spent loading PBF: 75s (average across 3 runs, min 62s, max 89s)
    • Memory allocations: 30,098,125,773 (average across three runs, ~900 MB spread)
    • CPU cyles: 4650 (average across 3 runs, min 3862, max 5356)

[taylor.smock]

Ticket retargeted after milestone closed

[taylor.smock]

In 18695/josm:

Fix #22603: Add PBF reading support

[gaben]

The difference between this and the plugin, that the latter can export as well (plus has dependency on the official protobuf)?

(Noticed a few since xxx)

[taylor.smock]

Replying to gaben:

The difference between this and the plugin, that the latter can export as well (plus has dependency on the official protobuf)?

The primary difference is that no plugin is required for PBF reading. Moving forward, something that we will probably be able to do is allow users to use a local PBF file as a "download" source instead of the OSM server (mostly for people who want to work with largish areas).

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

(Noticed a few since xxx)

FML. I ran ./scripts/since_xxx.py. Probably the entirely new files, since those get put in the actual default changelist, and then IntelliJ does some stuff to make it appear that they are in a different changeset. And I only ran it against the changeset that I was applying.

[gaben]

Replying to taylor.smock:

The primary difference is that no plugin is required for PBF reading. Moving forward, something that we will probably be able to do is allow users to use a local PBF file as a "download" source instead of the OSM server (mostly for people who want to work with largish areas).

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

I see, that's a quite useful use case.

Also just noticed, r18695 introduced some ErrorProne [MissingSummary] warnings, so nothing serious.

[taylor.smock]

Replying to gaben:

Replying to taylor.smock:

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

I see, that's a quite useful use case.

I really want corporate validation teams to use the same tools as editors for validation purposes, instead of writing their own. It means that (a) any validation check they write will be usable by editor users so that (b) fewer issues are introduced into the OSM database and (c) more feedback can be given on the validations.

As far as the missing summary warnings go, I'll try to fix them this week -- I'm also working on updating the Tasking Manager dependencies, and hopefully I'll finish those in the next day or two.

[nkamapper]

#23030