#23550 closed defect (fixed)
[PATCH] Error when deserializing PBF Blob
| Reported by: | Owned by: | team | |
|---|---|---|---|
| Priority: | normal | Milestone: | 24.03 |
| Component: | Core | Version: | |
| Keywords: | PBF | Cc: |
Description
When deserializing a Blob from OSM PBF file the current version is relying on the serialization order. This crashes when deserializing OSM PBF files generated with Rust Prost library
The attached fix removes the dependency on the serialization order and initializes explicitly the inflator stream.
Attachments (2)
Change History (11)
by , 8 weeks ago
| Attachment: | pbf-deserialization-fix.diff added |
|---|
comment:1 by , 7 weeks ago
@giora.kosoi: Can you attach a small sample PBF that exhibits the problem? Either that, or write a test that fails without the patch applied and passes with the patch applied.
by , 7 weeks ago
| Attachment: | niue-230109.osm.pbf added |
|---|
a sample osm.pbf of niue island that exibits the problem. osmosis --read-pbf ./niue-230109.osm.pbf --write-pbf ./osmosis-niue.osm.pbf works fine
comment:2 by , 7 weeks ago
The attached niue-230109.osm.pbf reproduces the problem. It was generated by the Rust osm-io library that uses the prost library for PBF serialization.
Osmosis and osmium tools consume this file (niue-230109.osm.pbf) without errors. Tests on data content after consumption also indicate correct processing.
The current implementation of PBF deserialization in JOSM relies on the data field in Blog message to be packed after the raw_size field, while the PBF spec recommends not to assume any specific order.
The attached patch fixes the issue.
comment:3 by , 7 weeks ago
Just to mention: The tool osmconvert (version 0.8.10) also fails to read that file:
C:\Users\Gerd\Downloads>osmconvert niue-230109.osm.pbf > xxx.osm osmconvert Error: block raw size expected at: 0x1A.
comment:4 by , 7 weeks ago
At a glance looks that osmconvert has a similar issue. The read loop expects fields to be in a predetermined order, contrary to PBF protocol recommendation.
Please see osmconvert.c version 0.8.11 line 4355 at http://m.m.i24.cc/osmconvert.c
comment:6 by , 5 weeks ago
It can be. I just need to find the time to write a non-regression test first. Which I'll be using attachment:niue-230109.osm.pbf for.
comment:8 by , 5 weeks ago
| Milestone: | → 24.03 |
|---|
comment:9 by , 4 weeks ago
| Summary: | [PATCH] Error when deserializing PBF Blog → [PATCH] Error when deserializing PBF Blob |
|---|
Fix for deserialization error