Opened 14 years ago
Last modified 14 years ago
#5404 reopened defect
OSM-API-close and HTTP-Request-Pipelining don't work over some proxies - JOSM must be restarted for further edits
| Reported by: | Fabi2 | Owned by: | Fabi2 |
|---|---|---|---|
| Priority: | normal | Milestone: | |
| Component: | Core | Version: | latest |
| Keywords: | Cc: |
Description
Now had the time to further analyze the bug in ticket #3970.
After the following HTTP-Request was sent to the proxy, JOSM seems to wait forever for an reply, which never is sent.
PUT http://api.openstreetmap.org/api/0.6/changeset/5656938/close HTTP/1.1 Authorization: Basic Removed= Content-type: text/xml User-Agent: JOSM/1.5 (3503 de) Java/1.6.0_21 Host: api.openstreetmap.org Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Proxy-Connection: keep-alive Content-Length: 0
This means the connection is stalled, as you can see on the wireshark screenshot,
The proxy (Winroute Pro) seems to have this bug with HTTP-request-pipelining, when the Content-Length is 0 bytes.
The request-pipeling is not used by JOSM, as the connection is always closed after each HTTP-request. So "Proxy-Connection: keep-alive" should be "Proxy-Connection: close" or the header should be removed.
The buggy proxy send the request to the outside OSM-API server, but then the connection hangs on the inside.
The changeset is closed on the server, but even when I check manually for it's status (Ctrl-Alt-Q) it is not noticed by JOSM and I must restart JOSM to get rid of the already uploaded changeset. :-(
HTTP-Request-Pipelining for webservers is still turned off by default in Firefox (see about:config), even when many webservers should be better written then some proxy server such as Winroute Pro.
So please turn the pipelining off, at least if an proxy is configured.
System: Linux x86_64
$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Attachments (1)
Change History (19)
by , 14 years ago
| Attachment: | bug-wireshark.png added |
|---|
comment:1 by , 14 years ago
| Summary: | OSM API close and HTTP-Request-Pipelining don't work over some proxies - JOSM must be restarted for further edits → OSM-API-close and HTTP-Request-Pipelining don't work over some proxies - JOSM must be restarted for further edits |
|---|
follow-up: 3 comment:2 by , 14 years ago
| Owner: | changed from to |
|---|---|
| Status: | new → needinfo |
comment:3 by , 14 years ago
Replying to stoecker:
I did a bit googling and omitting the header seems not so easy. Would it help when we e.g. send a single return and thus increase size to 1?
I googled and then looked at the Java-API ( http://www.innovation.ch/java/HTTPClient/api/HTTPClient/HTTPConnection.html#getDefaultHeaders() ) a bit and if you can get even the Prxy Headers with getDefaultHeaders() JOSM with HTTP-Proxy configured should look for the "Proxy-Connection:" header and if found, it should be always turned into a "Connection: close" which means, that the proxy and Webserver must close the connection after the response of the first request, but which is never send in this case (see RFC 2068 page 42 for discussion of persistent connections). HTTP/1.1 requires a Connection:-header (RFC2616 e.g. http://www.rfc-editor.org/rfc/rfc2616.txt pape 45), so it shouls not be removed. An other method is to force the client to HTTP/1.0 where the RFC 2616 states on page 44:
"A proxy server MUST NOT establish a persistent connection with an HTTP/1.0 client."
Then I found in RFC2616 p. 171:
19.6.2 Compatibility with HTTP/1.0 Persistent Connections Some clients and servers might wish to be compatible with some previous implementations of persistent connections in HTTP/1.0 clients and servers. Persistent connections in HTTP/1.0 are explicitly negotiated as they are not the default behavior. HTTP/1.0 experimental implementations of persistent connections are faulty, and the new facilities in HTTP/1.1 are designed to rectify these problems. The problem was that some existing 1.0 clients may be sending Keep-Alive to a proxy server that doesn't understand Connection, which would then erroneously forward it to the next inbound server, which would establish the Keep-Alive connection and result in a hung HTTP/1.0 proxy waiting for the close on the response. The result is that HTTP/1.0 clients must be prevented from using Keep-Alive when talking to proxies. However, talking to proxies is the most important use of persistent connections, so that prohibition is clearly unacceptable. Therefore, we need some other mechanism for indicating a persistent connection is desired, which is safe to use even when talking to an old proxy that ignores Connection. Persistent connections are the default for HTTP/1.1 messages; we introduce a new keyword (Connection: close) for declaring non-persistence. See section 14.10. The original HTTP/1.0 form of persistent connections (the Connection: Keep-Alive and Keep-Alive header) is documented in RFC 2068. [33]"
This seems to be the reason for this bug, the old shitty Winroute proxy ignores the "Proxy-Connection:"-header (this is a non-standardized header from which is telled that it's a netscape hack). But then, the OSM-API-Server has compatibility-code for this header and turns it into a "Connection: keep-alive" internally, before the control is passed on to the CGI-scripts. But as the proxy don't know about it, it waits for a close of the connection from the webserver, which never will happen. This is why I don't see any respone code on the inside connection to the proxy.
But I wonder why it works for the first OSM-API-requests, maybe they always do a close() on the incoming connections when response data are sent, so the disconnect will be forced in this cases by the webserver, even if a "Connection: keep-alive" header is present. This behaviour maybe also an OSM-API bug. As the don't return anything besides a "200 OK" on close, they maybde don't do any close and the connection then hangs after the the "200 OK"-response, waiting for the next requests from the Winroute proxy.
So it will be meaningless (have not tested it yet), to change the "Content-Length:"-header.
The best solution is to replace the "Proxy-Connection:"-header with a "Connection: close"-header.
The next best is to force a HTTP/1.0 connection, but maybe the webserver will it keep also open, becauise of the "Proxy-Connection:"-header.
The last workaround-idea is to use setTimeout() to set a timeout for this connection or all connections with setDefaultTimeout(), and if it is reached, then do an automatic "has this changeset really closed"-request as it is uses with Ctrl-Alt-Q (this works after that) and remove the old changeset on success.
comment:5 by , 14 years ago
Some more information on the "Proxy-Connection:"-header:
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/web-proxy-connection-header.html
comment:6 by , 14 years ago
Now, also reported this an an maybe api bug. http://trac.openstreetmap.org/ticket/3241
comment:7 by , 14 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | needinfo → closed |
Now I fixed the headers with privoxy, and as you can see, the Connection still hangs, so at least it doesn't help in this case.
"Sep 23 22:58:28.668 7f6fd9e76710 Connect: The client closed socket 6 while the server socket 8 is still open."
Socket 6 was JOSM wich I closed. :-(
ep 23 22:52:41.967 7f6fd9e76710 Header: Adding: Connection: close Sep 23 22:52:41.967 7f6fd9e76710 Connect: Complete client request received. Sep 23 22:52:41.968 7f6fd9e76710 Header: New HTTP Request-Line: PUT http://api.openstreetmap.org/api/0.6/changeset/5857713/close HTTP/1.1 Sep 23 22:52:41.968 7f6fd9e76710 Request: api.openstreetmap.org/api/0.6/changeset/5857713/close Sep 23 22:52:41.968 7f6fd9e76710 Connect: via [10.0.0.3]:3128 to: api.openstreetmap.org Sep 23 22:52:41.990 7f6fd9e76710 Connect: Connected to 10.0.0.3[10.0.0.3]:3128. Sep 23 22:52:41.990 7f6fd9e76710 Connect: Created new connection to api.openstreetmap.org:80 on socket 8. Sep 23 22:52:41.990 7f6fd9e76710 Connect: to api.openstreetmap.org successful Sep 23 22:57:42.291 7f6fd9e76710 Connect: The connection on server socket 8 to api.openstreetmap.org isn't reusable. Closing. Sep 23 22:57:52.300 7f6fdbe36700 Connect: accepted connection from ::1 Sep 23 22:57:52.300 7f6fdbe36700 Connect: Listening for new connections ... Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: PUT http://api.openstreetmap.org/api/0.6/changeset/5857713/close HTTP/1.1 Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Authorization: Basic Removed= Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Content-type: text/xml Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: User-Agent: JOSM/1.5 (3555 de) Java/1.6.0_21 Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Host: api.openstreetmap.org Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Proxy-Connection: keep-alive Sep 23 22:57:52.301 7f6fd9e76710 Header: scan: Content-Length: 0 Sep 23 22:57:52.301 7f6fd9e76710 Header: crumble crunched: Proxy-Connection: keep-alive! Sep 23 22:57:52.301 7f6fd9e76710 Header: addh: Connection: close Sep 23 22:57:52.301 7f6fd9e76710 Header: Adding: Connection: close Sep 23 22:57:52.301 7f6fd9e76710 Connect: Complete client request received. Sep 23 22:57:52.301 7f6fd9e76710 Header: New HTTP Request-Line: PUT http://api.openstreetmap.org/api/0.6/changeset/5857713/close HTTP/1.1 Sep 23 22:57:52.301 7f6fd9e76710 Request: api.openstreetmap.org/api/0.6/changeset/5857713/close Sep 23 22:57:52.301 7f6fd9e76710 Connect: via [10.0.0.3]:3128 to: api.openstreetmap.org Sep 23 22:57:52.323 7f6fd9e76710 Connect: Connected to 10.0.0.3[10.0.0.3]:3128. Sep 23 22:57:52.323 7f6fd9e76710 Connect: Created new connection to api.openstreetmap.org:80 on socket 8. Sep 23 22:57:52.323 7f6fd9e76710 Connect: to api.openstreetmap.org successful Sep 23 22:58:28.668 7f6fd9e76710 Connect: The client closed socket 6 while the server socket 8 is still open. Sep 23 22:58:28.668 7f6fd9e76710 Connect: The connection on server socket 8 to api.openstreetmap.org isn't reusable. Closing.
follow-up: 9 comment:8 by , 14 years ago
| Resolution: | worksforme |
|---|---|
| Status: | closed → reopened |
Can you compile JOSM yourself? We should at least try to get a "Connection: Close" out of JOSM for the changeset close request. This may fix your problem.
comment:9 by , 14 years ago
Replying to stoecker:
Can you compile JOSM yourself? We should at least try to get a "Connection: Close" out of JOSM for the changeset close request. This may fix your problem.
Yes, if I get some description on how to do it, some time and the code over the proxy, but as I already tried a "fix simulation" with letting privoxy modifiing the headers as a 2nd proxy before the local proxy (log if it is above), I don't expect that it helps for me. I can do an additional simulation where I don't remove the "Proxy-Connection:"-header.
At least the Proxy-Connection-header is a historic HTTP-protocol missuse and it should be better "Connection: close" instead of "Proxy-Connection: keep-alive". So this is a bug but it wouldn't sadly fix the problem for me.
As the api now makes no reuse of the connections. Sadly a least it wouldn't fix this local here, as even the privoxy-corrected-headers give my a hang here. This it at least also not a bug in the API code, wich is really high-level, as the ruby web framework just automatically outputs a "page" template and fills in detected variables by calling provided functions. As the administrators here should better not called so, i now expect that this is maybe a bug in the the lokal proxy (even if I don't know why the other requests, which are nearly the sam, go through there without problems) or it is a bug in the underlaying software of the api. Yes, this may be a bug in mod_rails and the ruby interpreter/framework which it runs. The TomH says they also use a proxy (maybe mod_proxy) for some requsts to the OSM-API. But I don't try to understand the things behind the OSM-API now.
To fix this, the Proxy-Connection:-header should be removed when Connection:close is added, as they mean the same and say the opposite to the server, the it knows both of them. So it will be better to force the protocol to HTTP/1.0, when you can now remove the Proxy-Connection:-header, as in HTTP/1.1 it will be expected from the server to always ignore the "Proxy-Connection: keep-alive"-header to be compliant, even when the server would understand it, so the connection should be always closed.
comment:10 by , 14 years ago
Only a short question: Did you try using josm.openstreetmap.de as API URL? There is an Apache redirect in between. Maybe it solves your problem as a side effect :-)
comment:11 by , 14 years ago
Result with using josm-redirector:
Sep 26 00:17:50.651 7f335aff4710 Connect: Done reading from server. Content length: 1924 as expected. Bytes most recently read: 476. Sep 26 00:17:50.651 7f335aff4710 Connect: The connection on server socket 8 to josm.openstreetmap.de isn't reusable. Closing. Sep 26 00:17:50.686 7f335bfb2700 Connect: accepted connection from 127.0.0.1 Sep 26 00:17:50.686 7f335bfb2700 Connect: Listening for new connections ... Sep 26 00:17:50.687 7f335aff4710 Header: scan: PUT http://josm.openstreetmap.de/api/0.6/changeset/5874991/close HTTP/1.1 Sep 26 00:17:50.687 7f335aff4710 Header: scan: Authorization: Basic Removed= Sep 26 00:17:50.687 7f335aff4710 Header: scan: Content-type: text/xml Sep 26 00:17:50.687 7f335aff4710 Header: scan: User-Agent: JOSM/1.5 (3561 de) Java/1.6.0_21 Sep 26 00:17:50.687 7f335aff4710 Header: scan: Host: josm.openstreetmap.de Sep 26 00:17:50.687 7f335aff4710 Header: scan: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Sep 26 00:17:50.687 7f335aff4710 Header: scan: Proxy-Connection: keep-alive Sep 26 00:17:50.687 7f335aff4710 Header: scan: Content-Length: 0 Sep 26 00:17:50.687 7f335aff4710 Header: crumble crunched: Proxy-Connection: keep-alive! Sep 26 00:17:50.687 7f335aff4710 Header: addh: Connection: close Sep 26 00:17:50.687 7f335aff4710 Header: Adding: Connection: close Sep 26 00:17:50.687 7f335aff4710 Connect: Complete client request received. Sep 26 00:17:50.687 7f335aff4710 Header: New HTTP Request-Line: PUT http://josm.openstreetmap.de/api/0.6/changeset/5874991/close HTTP/1.1 Sep 26 00:17:50.687 7f335aff4710 Request: josm.openstreetmap.de/api/0.6/changeset/5874991/close Sep 26 00:17:50.687 7f335aff4710 Connect: via [10.0.0.3]:3128 to: josm.openstreetmap.de Sep 26 00:17:50.709 7f335aff4710 Connect: Connected to 10.0.0.3[10.0.0.3]:3128. Sep 26 00:17:50.710 7f335aff4710 Connect: Created new connection to josm.openstreetmap.de:80 on socket 8. Sep 26 00:17:50.710 7f335aff4710 Connect: to josm.openstreetmap.de successful Sep 26 00:18:25.150 7f335aff4710 Connect: The client closed socket 6 while the server socket 8 is still open. Sep 26 00:18:25.151 7f335aff4710 Connect: The connection on server socket 8 to josm.openstreetmap.de isn't reusable. Closing.
Normal OSM-API with HTTP-version downgrade:
Sep 26 00:37:15.562 7f44879b0710 Connect: The connection on server socket 8 to api.openstreetmap.org isn't reusable. Closing. Sep 26 00:37:15.618 7f448896e700 Connect: accepted connection from 127.0.0.1 Sep 26 00:37:15.618 7f448896e700 Connect: Listening for new connections ... Sep 26 00:37:15.627 7f44879b0710 Header: scan: PUT http://api.openstreetmap.org/api/0.6/changeset/5875057/close HTTP/1.1 Sep 26 00:37:15.627 7f44879b0710 Header: scan: Authorization: Basic Removed= Sep 26 00:37:15.627 7f44879b0710 Header: scan: Content-type: text/xml Sep 26 00:37:15.627 7f44879b0710 Header: scan: User-Agent: JOSM/1.5 (3561 de) Java/1.6.0_21 Sep 26 00:37:15.628 7f44879b0710 Header: scan: Host: api.openstreetmap.org Sep 26 00:37:15.628 7f44879b0710 Header: scan: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Sep 26 00:37:15.628 7f44879b0710 Header: scan: Proxy-Connection: keep-alive Sep 26 00:37:15.628 7f44879b0710 Header: scan: Content-Length: 0 Sep 26 00:37:15.628 7f44879b0710 Header: crumble crunched: Proxy-Connection: keep-alive! Sep 26 00:37:15.628 7f44879b0710 Header: Adding: Connection: close Sep 26 00:37:15.628 7f44879b0710 Connect: Complete client request received. Sep 26 00:37:15.628 7f44879b0710 Header: New HTTP Request-Line: PUT http://api.openstreetmap.org/api/0.6/changeset/5875057/close HTTP/1.0 Sep 26 00:37:15.628 7f44879b0710 Request: api.openstreetmap.org/api/0.6/changeset/5875057/close Sep 26 00:37:15.628 7f44879b0710 Connect: via [10.0.0.3]:3128 to: api.openstreetmap.org Sep 26 00:37:15.650 7f44879b0710 Connect: Connected to 10.0.0.3[10.0.0.3]:3128. Sep 26 00:37:15.650 7f44879b0710 Connect: Created new connection to api.openstreetmap.org:80 on socket 8. Sep 26 00:37:15.650 7f44879b0710 Connect: to api.openstreetmap.org successful Sep 26 00:37:32.254 7f44879b0710 Connect: The client closed socket 6 while the server socket 8 is still open. Sep 26 00:37:32.254 7f44879b0710 Connect: The connection on server socket 8 to api.openstreetmap.org isn't reusable. Closing.
Must be a local proxy issue. But the "admins" here run also a windows 2003 dns which is also broken (throws away records received, from the upstream dns).
follow-up: 13 comment:12 by , 14 years ago
You ask why CLOSE fails and the others not. I'm not 100% sure, but maybe this depends on the number of result bytes the server sends. Probably the proxy waits for more bytes due to a bit dumb programming.
Does this issue also happen when using netcat to initiate the connection?
- Record a valid request
- Use netcat to send the request to server
- see results
In this case you can try if you find a combination of headers which work. This is much easier than modifying a software before each test.
The isClosed() after failure check should be implemented in JOSM nevertheless.
comment:13 by , 14 years ago
Replying to stoecker:
You ask why CLOSE fails and the others not. I'm not 100% sure, but maybe this depends on the number of result bytes the server sends. Probably the proxy waits for more bytes due to a bit dumb programming.
That was I also assumed first in the older postings: a blocking read() on the socket, but a you can see in the wireshark screenshot, there is not a single byte received, after JOSM sends the close request to the server. With a blocking read() the response code should be received from the server before the hang.
Does this issue also happen when using netcat to initiate the connection?
- Record a valid request
- Use netcat to send the request to server
- see results
I don't know wher is the difference to insert or remove headers with privoxy or look with wireshark what will happen. Even with netcat I would get no response. But can maybe a problem with the "Content-Length:"-header. Let's try:
# cat osm-proxy-nc.tct PUT http://api.openstreetmap.org/api/0.6/changeset/5875057/close HTTP/1.0 Host: api.openstreetmap.org Content-Length: 1 root@pong # nc 10.0.0.3 3128 <osm-proxy-nc.tct HTTP/1.1 401 Authorization Required Date: Sun, 26 Sep 2010 16:04:54 GMT Server: Apache/2.2.14 (Ubuntu) X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15 WWW-Authenticate: Basic realm="Web Password" X-Runtime: 3 Cache-Control: no-cache Content-Length: 25 Status: 401 Vary: Accept-Encoding Content-Type: text/html; charset=utf-8 Connection: close
The additional return with is the 1 byte of "Content-Length: 1" helped, to get a error response back! :-)
# nc 10.0.0.3 3128 <osm-proxy-nc.tct
HTTP/1.1 409 Conflict
Date: Sun, 26 Sep 2010 16:10:09 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15
X-Runtime: 42
Cache-Control: no-cache
Error: The changeset 5875057 was closed at 2010-09-25 22:37:13 UTC
Set-Cookie: _osm_session=abac609e4ade4991480d6dd506f20599; path=/; HttpOnly
Content-Length: 59
Status: 409
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Connection: close
The changeset 5875057 was closed at 2010-09-25 22:37:13 UTC[root@pong #
Now with full original headers, but "Content-Length: 1" and an "\n" as payload:
nc 10.0.0.3 3128 <osm-proxy-nc.tct HTTP/1.1 409 Conflict Date: Sun, 26 Sep 2010 16:17:51 GMT Server: Apache/2.2.14 (Ubuntu) X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15 X-Runtime: 16 Cache-Control: no-cache Error: The changeset 5875057 was closed at 2010-09-25 22:37:13 UTC Set-Cookie: _osm_session=6b46da11cb8dd9408a039656efcccd51; path=/; HttpOnly Content-Length: 59 Status: 409 Vary: Accept-Encoding Content-Type: text/html; charset=utf-8 Connection: close The changeset 5875057 was closed at 2010-09-25 22:37:13 UTC
When I change the same request to "Content-Length: 0" the connection will hang! So this ia the bug.
In this case you can try if you find a combination of headers which work. This is much easier than modifying a software before each test.
So sending a e.g. newline as payload (updating the header alone doesn't helps) with the close-Request should at least work around this bug in Winroute Pro.
The isClosed() after failure check should be implemented in JOSM nevertheless.
At least the manual changeset status check (Ctrl-Alt-Q), should force update the state of what is already closed and what is stinn open, when it succeeds, as this also doesn't work in this situation.
comment:14 by , 14 years ago
That was I also assumed first in the older postings: a blocking read() on the socket, but a you
can see in the wireshark screenshot, there is not a single byte received, after JOSM sends the
close request to the server. With a blocking read() the response code should be received from
the server before the hang.
You can't assume this. Sorry. I developed a standard used for GNSS data transfer based on HTTP. We had and have lots of trouble with proxies and assumptions like yours usually fail for at least one software. You can't assume anything, there are too many ways you can do wrong in a networking software. :-)
follow-up: 16 comment:15 by , 14 years ago
follow-up: 17 comment:16 by , 14 years ago
comment:17 by , 14 years ago
Replying to anonymous:
Will try it, when the new version is available. Hopefully this will nor cause any harm to the OSM-API, so this workaround maybe should only enabled if any proxy is configured.
The workaround works here as expected.
comment:18 by , 14 years ago
This reason for these (#4216, #5138) bugs seems to be the same, but with the capabilities-api-request affected. The reason why this doesn't block here, is that the additional false at the end:
sendRequest("GET", "capabilities", null,monitor, false);
sendRequest("PUT", "changeset" + "/" + changeset.getId() + "/close", null, monitor);
at least here will supress the "Content-Length:"-header completely, which works fine.
Seems so, that the Kerio Winroute programmer depend on the assumtion, that if there is an "Content-Length:"-header, some data must be transmitted to the server.
I did a bit googling and omitting the header seems not so easy. Would it help when we e.g. send a single return and thus increase size to 1?