map download gets outdated cache content

[maarten]

As discussed on the talk mailinglist from ​http://lists.openstreetmap.org/pipermail/talk/2013-July/067693.html

It appears that JOSM 6060 does not always close a changeset at the end of the session leaving it to think data is unchanged. Data in OSM is in fact changed but when you download it in JOSM again, you get the old data. When looking at the history of an object (ctrl-H) you will see the proper history with changes.

I have no idea how to reproduce it. It only happens occasionally.

[bastiK]

I'm not completely certain what happened here, but there doesn't seem to be a problem with JOSM but with the server you are downloading from. If it's a mirror, this would quite normal, if it's the main API then it's a little odd.

[skyper]

We have #5783 still open, where JOSM does not get important information from the server.

@mdeen:
Do you remember if the changeset was closed and JOSM did not get this information or was the changeset still open ?

[maarten]

@bastiK:
I am using the default OSM server URL (checkbox next to that field) and that is set tot the main api.

@skyper:
The changeset in OSM says it was closed: <​http://www.openstreetmap.org/browse/changeset/17057508>
But in JOSM when I did "Close open changesets" (some 2 hours later than when OSM says it was closed) it informed me that there was one changeset open. So JOSM thought it was open. I did not get an exception or closed JOSM abnormally.

[bastiK]

Even if the changeset is still open, you can always download the new version of the objects immediately after upload. If this does not work and you know the upload was successful, then this must be a server problem.

[skyper]

So we need to fill a ticket at ​https://trac.openstreetmap.org ?

[maarten]

@Batik:
When I downloaded from an area, I got the old version. When I downloaded an individual item after that using download object I got a conflict.
So there was a difference in handling downloads by area and downloads by id.

[bastiK]

Replying to mdeen@…:

@Batik:
When I downloaded from an area, I got the old version. When I downloaded an individual item after that using download object I got a conflict.
So there was a difference in handling downloads by area and downloads by id.

The osm server alway delivers the most recent version of any object, no matter what download method you choose. If this was broken at a certain time (very strange) then there is still nothing we can do about it in JOSM.

[maarten]

I don't know where it was broken, but JOSM still thought the changeset was open. That seems to be a JOSM problem.

[bastiK]

If JOSM get the open/closed state of a changeset wrong, this is a minor issue and cannot have the consequences you describe.

But you are right, this is probably a JOSM bug and if would be helpful if you can reproduce the problem (or try to remember the steps next time it happens).

[anonymous]

I have no idea where the bug is, but I have the same issue again, only JOSM now does not see a closed changeset.
I have uploaded changeset ​http://www.openstreetmap.org/browse/changeset/17269345 and the changes are also visible on the map. But when I download the same area in JOSM, I see the old version of items. When I change something and upload it, JOSM says there is a validation issue (which is good).

I have closed JOSM between these editing session.

I don't know if this might be related, but I also have had occasions that I uploaded all my changes and when I want to exit JOSM, it ways I need to upload my changes. When I do so, the upload box shows no changes and JOSM closes.

Could this maybe be a persistent cache somewhere between me and the OSM server? In JOSM I have "no proxy" checked.

[maarten]

It did download the correct versions after I downloaded and uploaded items in a different area. Which could point to a possible bug: what if this only happens when you do not change the download area in the download box? Now I downloaded a different area, next time I encounter this phenomenon I'll just change the download area.

[bastiK]

For every download, JOSM prints something like this

GET http://api.openstreetmap.org/api/0.6/map?bbox=6.4623213,50.8064229,6.4663124,50.8094331

on the command line.

Next time the bug occurs, please download this URL manually with a browser (or wget/curl), save as map.osm and open in JOSM. If you still get the old version, then JOSM can be excluded as the source of error.

[maarten]

It does not do that in the webstart version on Windows.

[Don-vip]

You can enable the Java console even with Webstart:
​http://confluence.concord.org/display/CCTR/Enabling+Windows+Java+WebStart+Console

[maarten]

Perfect. And it did happen again. I'm starting to think this is reproducable. In the java console I see this:

GET ​http://api.openstreetmap.org/api/0.6/map?bbox=5.3000021,51.4044531,5.3081346,51.4077593
CacheEntry​http://api.openstreetmap.org/api/0.6/map?bbox=5.3000021,51.4044531,5.3081346,51.4077593: updateAvailable=false,lastModified=Thu Jan 01 01:00:00 CET 1970,length=-1

Note the "CacheEntry". Does JOSM cache things? When I get the area with wget I get the correct data. The "CacheEntry" line is not there when I change the download area and then I get the correct data also.

[bastiK]

Thanks, this brings us much closer to the real problem. The next step for diagnosis would be to run wireshark when JOSM fetches the outdated data. Basically what I'd like to see is, if there is any connection to the OSM server at all or if it uses the cache file from the hard-drive without further validation. Could you add a status report, please?

See ​http://stackoverflow.com/questions/17181129/java-1-7-applet-cacheentry-preventing-dynamic-updates

The webstart version of Java seems to cache static resources like a browser. In principle, it should recognize dynamic content by looking at the http header, but this seems to go wrong here.

The osm server sends the following header:

wget -S 'http://api.openstreetmap.org/api/0.6/map?bbox=5.3000021,51.4044531,5.3081346,51.4077593'
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Fri, 09 Aug 2013 06:38:38 GMT
  Server: Apache/2.2.22 (Ubuntu)
  Content-Disposition: attachment; filename="map.osm"
  Content-Encoding: identity
  Cache-Control: private, max-age=0, must-revalidate
  Vary: Accept-Encoding
  Content-Type: text/xml; charset=utf-8
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Transfer-Encoding: chunked
Length: unspecified [text/xml]

Before the ticket is closed as "othersoftware", we should find out, how many users are affected by this. If it's a common problem, we can think about a workaround (e.g. dummy parameter with random value).

[Don-vip]

We could try this:
​http://docs.oracle.com/javase/7/docs/technotes/guides/javaws/enhancements6.html

The "cache-control: no-cache" HTTP header is now supported. "no-cache" means cached contents cannot be used without re-validation. When the no-cache directive is used , an update check is always made to make sure the cached content is the same as the server copy pointed to, by the resource URL. If so, cached copy can be used. If not, server copy will be downloaded. Expiration field is ignored, which means that even if the cached resource is not expired yet, update check will still be performed.

[Don-vip]

Yep, this is a change in JDK7u2 and later:
​http://docs.oracle.com/javase/7/docs/technotes/guides/jweb/enhancements-7.html#javase7u2

Caching enabled by default: Caching of network content for application code running in Web Start mode is now enabled by default. This allows application improved performance and consistency with applet execution mode. To ensure the latest copy of content is used, the application can use ​URLConnection.setUseCaches(false) or request header Cache-Control values no-cache/no-store.

[bastiK]

Replying to Don-vip:

Yep, this is a change in JDK7u2 and later:
​http://docs.oracle.com/javase/7/docs/technotes/guides/jweb/enhancements-7.html#javase7u2

Caching enabled by default: Caching of network content for application code running in Web Start mode is now enabled by default. This allows application improved performance and consistency with applet execution mode. To ensure the latest copy of content is used, the application can use ​URLConnection.setUseCaches(false) or request header Cache-Control values no-cache/no-store.

Alright, ​URLConnection.setUseCaches(false) sounds promising...

(Could not reproduce on my system:

icedtea-web 1.3.2 (1.3.2-1ubuntu0.12.10.2)
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.10.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

)

[Don-vip]

In 6123/josm:

fix #8904 - Disable Java HTTP caching, enabled by default with Java Web Start in 7u2 and later

[ScM]

Same issue with changeset 19815334 in josm version 6502 with java 1.7.0_45.
I uploaded the changeset and the changes are visible on
​http://www.openstreetmap.org/changeset/19815334#map=15/10.7609/123.0549&layers=H
and already (partially) rendered on
​http://www.openstreetmap.org/#map=16/10.7608/123.0500
and are shown in iD, but if I try to redownload the area in JOSM the outdated pre-changeset version appears. I tried to restart josm, to no avail. I also tried to change the download area, without solving the issue.