[Patch] Encoding of URLs

[roland.olbricht]

Under Windows 7 with Java 6, when typing an URL into "File > Open address ...", this URL is sent enocded as Windows-1252. It should be sent encoded as UTF8.

Example:
​http://overpass-api.de/api/interpreter?data=way[name=Kornstraße];out+meta;
is sent with a single byte 0xDF (which looks like Windows-1252).

I haven't tested it with other systems, so I could be both platform dependend or platform independend.

[Don-vip]

How do you test this ? There is currently no suitable download task for this kind of URL ?

[Don-vip]

In 5772/josm:

see #8505 - strip of trailing and leading whitespace characters of input URL (proper solution than String.trim(), see Utils.strip()) + unit test

[Don-vip]

I have tried to change some internal mechanisms of JOSM, now I have this URL:
​http://overpass-api.de/api/interpreter?data%3Dway%5Bname%3DKornstra%C3%9Fe%5D%3Bout%2Bmeta%3B
It raises some parser errors on Overpass API side, is it feasible to fix them ?

[roland.olbricht]

First of all, thank you for fixing the bug. It now works for me with the version from svn, with URL
​http://overpass-api.de/api/interpreter?data=(way[name="Kornstraße"];>;);out%20meta;

I'm sorry for the confusion. There are two different problems in this query:

• Kornstraße has no quotation marks around the word. The syntax parser can be fixed to handle the string also without quotation marks.
• The equation sign after daata is URL encoded. I would prefer not to fix that. This should be a proper equation mark to make to whole URL properly URL encoded like a GET sent web form.

[Don-vip]

I have not changed that much :) If it works for you now, can we close this ticket ?

[roland.olbricht]

Sorry, it still doesn't work. From the logfile of the server:

[13/Mar/2013:18:29:42 +0100] query string: /api/interpreter?data=(way[name=\"Kornstra\xc3\x9fe\"];>;);out%20meta;, user agent: JOSM/1.5 (5772 SVN de)
[14/Mar/2013:10:12:35 +0100] query string: /api/interpreter?data=(way[name=\"Kornstra\xdfe\"];>;);out%20meta;, user agent: JOSM/1.5 (5772 SVN de)

This is the same JAR file, executed from an USB drive. The first line is from a run under Linux, the second from a run under Windows.

[roland.olbricht]

I suggest the following patch:

Insert into class DownloadOsmTask, in method loadUrl immediately after the begin:

try
{

if (url.matches(PATTERN_OVERPASS_API_URL))
{

int pos = url.indexOf("/interpreter?data=") + 18; encode only the part after the equal sign
url = url.substring(0, pos) + URLEncoder.encode(url.substring(pos), "UTF-8");

}
else if (url.matches(PATTERN_OVERPASS_API_XAPI_URL))
{

int pos = url.indexOf("/xapi?") + 6; encode only the part after the equal sign
url = url.substring(0, pos) + URLEncoder.encode(url.substring(pos), "UTF-8");

}

}
catch (UnsupportedEncodingException e)
{
}

The used classes for import are:
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;

This encodes the URL properly and platform independend in percent encoding.

I've just added a corresponding patch to the mirrored_download plugin.

[Don-vip]

In 5782/josm:

fix #8505 - Encoding of overpass API URLs (based on patch by roland.olbricht)