#8566 closed defect (fixed)
Improper handling of percent-encoded urls towards Overpass-API
Reported by: | tyr_asd | Owned by: | team |
---|---|---|---|
Priority: | normal | Milestone: | 14.01 |
Component: | Core | Version: | tested |
Keywords: | overpass api | Cc: | roland.olbricht |
Description (last modified by )
When using the import command of the RemoteControl protocol, JOSM doesn't properly handle already percent-encoded URLs: (This is a regression over what JOSM did a few weeks ago, when I last checked.)
Lets say, I want to load the following URL:
http://www.overpass-api.de/api/interpreter?data=node%5B%22foo%22%5D%3Bout%20meta%3B
Using this URL as the parameter "url" of the import command means that I have to url-encode the whole thing (because "url" is a HTTP-GET parameter):
http://localhost:8111/ GET /import?url=http%3A%2F%2Fwww.overpass-api.de%2Fapi%2Finterpreter%3Fdata%3Dnode%255B%2522foo%2522%255D%253Bout%2520meta%253B
but instead of grabbing the original URL, JOSM does this:
GET http://www.overpass-api.de/api/interpreter?data=node%255B%2522foo%2522%255D%253Bout%2520meta%253B
This looks like JOSM is either doing one superfluous url-encoding step or is missing one internal url-decoding step. I found out that after bug #8148 the code was altered slightly - maybe that could be the cause of this?
Attachments (0)
Change History (13)
comment:1 by , 11 years ago
Cc: | added |
---|---|
Keywords: | overpass api added |
comment:2 by , 11 years ago
Description: | modified (diff) |
---|
comment:3 by , 11 years ago
Component: | Core remotecontrol → Core |
---|---|
Summary: | RemoteControl: import command doesn't properly handle percent-encoded urls → Improper handling of percent-encoded urls towards Overpass-API |
comment:4 by , 11 years ago
OK, confirming that the additional URLEncoder.encode() added in r5782 is actually the bad boy.
If you really want to use such a workaround to circumvent charset problems, you'd at least have to decode the url first.
comment:5 by , 11 years ago
Unfortunately, "loadUrl" is called quite straight from the "Open Location ..." dialogue. And people may input there URLs that would break on decoding, e.g. as part of the URL a regular expression that contains a literal plus sign.
Removing the encoding code won't help either, because the JRE behaves when calling URLs platform-dependend and breaks unencoded queries on Windows.
I would suggest one of the following solutions, ordered from most prospective to least prospective:
- Encode only if the URL contains a non-ASCII character. This lets pass URLs like the one above but would work around the platform dependend encoding of the JRE. False positives are URLs with only ASCII special characters. However, the Overpass server handles them properly. Thus the main risk would be an accidential encoding by the JRE. Such a behaviour was not observed yet.
- Encode only if "loadURL" is called from the download dialogue. I'm not sure whether this is possible in the code with reasonable effort.
- Decode URLs if they are valid in percent encoding, i.e. if the percent sign is the only special character (ASCII or non-ASCII) that is contained in the URL. This may affect regularly entered URLs by surprise, so I strongly suggest not to do this. In particular, this may cause a silent rewriting of regular expressions, and that ultimately ends in very hard to debug misbehaviour of that query.
comment:6 by , 11 years ago
This is causing troubles, again: https://github.com/tyrasd/overpass-ide/issues/31 :(
Here is the problem:
- When calling JOSM's remote control, I cannot url-encode the url because of this bug
- Because JOSM fails to load url's containing (non url-encoded) newlines, I have to strip those
- But Overpass-QL queries can contain single line comments (
// …
). - When a query contains such a single line comment, I cannot strip newlines because they have syntactical meaning.
- Deadlock :(
As JOSM's mirrored download plugin now has its own "Download from Overpass API" tool, I guess nobody still uses the loadURL tool to manually enter a non-escaped overpass query? So, can we – please – revert changeset r5782 now?
comment:10 by , 11 years ago
Milestone: | → 14.01 |
---|
comment:11 by , 9 years ago
This is causing troubles again, I think.
Overpass generates a URL which gets rejected by JOSM:
Cannot open URL
'http://overpass-api.de/api/interpreter?
fixme=JOSM-ticket-8566
&data=%2F*%0AThis%20has%20been%20generated%20b'...
The following download tasks accept the URL patterns shown:
comment:12 by , 9 years ago
Here is the corresponding ticket on github: https://github.com/tyrasd/overpass-turbo/issues/158
Any idea how to fix this?
comment:13 by , 9 years ago
Please do not reuse very old tickets, but open a new one and refer to the other one. This eases to deal with milestone handling …
→ #11356
r5782 (#8505) may be the cause of this behaviour.