Josm do not start in remote control from osm.org in https

[anonymous]

What steps will reproduce the problem?

1. use firefox or chromium (Ubuntu 14.04)
2. visit osm main page
3. try to edit a area from menu...

What is the expected result?
Open edi area as usual

What happens instead?
No error and josm do not open and download the edit area... (josm is opened without any data loaded)

Please provide any additional information below. Attach a screenshot if
possible.

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-05-16 01:37:38
Last Changed Author: Don-vip
Revision: 7134
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-05-15 20:03:47 +0200 (Thu, 15 May 2014)
Last Changed Rev: 7134

Identification: JOSM/1.5 (7134 it) Linux Ubuntu 14.04 LTS
Memory Usage: 169 MB / 455 MB (83 MB allocated, but free)
Java version: 1.7.0_55, Oracle Corporation, Java HotSpot(TM) Server VM
VM arguments: [-Xmx512M]

Plugin: AddrInterpolation (30416)
Plugin: CommandLine (30436)
Plugin: FastDraw (30416)
Plugin: FixAddresses (30416)
Plugin: ImportImagePlugin (30416)
Plugin: OpeningHoursEditor (30416)
Plugin: PicLayer (30436)
Plugin: SimplifyArea (30416)
Plugin: buildings_tools (30416)
Plugin: conflation (0.1.6)
Plugin: continuosDownload (28565)
Plugin: dataimport (30416)
Plugin: download_along (30416)
Plugin: geotools (30416)
Plugin: jts (30416)
Plugin: junctionchecking (30416)
Plugin: log4j (30416)
Plugin: mapdust (30416)
Plugin: merge-overlap (30416)
Plugin: mirrored_download (30436)
Plugin: namemanager (30416)
Plugin: notes (v0.9.2)
Plugin: opendata (30436)
Plugin: pbf (30416)
Plugin: poly (30416)
Plugin: print (30416)
Plugin: proj4j (30417)
Plugin: reltoolbox (30416)
Plugin: reverter (30436)
Plugin: scripting (30604)
Plugin: tageditor (30416)
Plugin: tagging-preset-tester (30416)
Plugin: terracer (30416)
Plugin: turnrestrictions (30416)
Plugin: undelete (30416)
Plugin: utilsplugin2 (30419)
Plugin: walkingpapers (30416)
Plugin: waydownloader (30416)
Plugin: waypoint_search (30416)
Plugin: wikipedia (30449)

[Don-vip]

Indeed it seems at least chrome doesn't like our autosigned certificate. But we can't get a real one for localhost :D we need to find a way to make browser accept it.

[Don-vip]

Good news, we can access system keystore from Java and add our own certificate: ​http://stackoverflow.com/a/5510555/2257172

At least for Windows, I don't know yet for Linux. Even less for Mac :)

[Don-vip]

In 7206/josm:

see #10033 - allow remote control to work from osm.org in https on Windows systems by adding updated JOSM localhost certificate to Windows Root Certificates keystore

[bastiK]

URL for testing: ​https://127.0.0.1:8112/load_and_zoom?left=8.19&right=8.20&top=48.605&bottom=48.590&select=node413602999

General info: ​http://www.chromium.org/Home/chromium-security/root-ca-policy

There is a KeyStore called PKCS11-NSS, but just copying your code doesn't work...

[Don-vip]

Replying to bastiK:

There is a KeyStore called PKCS11-NSS, but just copying your code doesn't work...

Yes it requires some configuration ​described here.

I have not yet read everything but as I understand, we need to build a configuration file like

name=<anything>
nssLibraryDirectory = <path to libnss>
nssSecmodDirectory = <path to nssdb> (/etc/pki/nssdb)
nssDbMode = readWrite

[Don-vip]

@Lesath: Can you please tell me if the bug also affects Safari on Mac OSX? If yes, does this piece of code fix the problem if we add it in tools/PlatformOSX?

    @Override
    public void setupHttpsCertificate(KeyStore.PrivateKeyEntry privateKeyEntry)
            throws KeyStoreException, NoSuchAlgorithmException, CertificateException, IOException {
        KeyStore ks = KeyStore.getInstance("KeychainStore", "Apple");
        ks.load(null, null);
        Enumeration<String> en = ks.aliases();
        while (en.hasMoreElements()) {
            String alias = en.nextElement();
            Certificate c = ks.getCertificate(alias);
            if (ks.isKeyEntry(alias) && c.equals(privateKeyEntry.getCertificate())) {
                // JOSM certificate found, return
                return;
            }
        }
        // JOSM certificate not found, install it
        Main.info("Adding JOSM localhost certificate to Apple KeychainStore");
        ks.setEntry("josm_localhost", privateKeyEntry, new KeyStore.PasswordProtection("josm_ssl".toCharArray()));
    }

[Lesath]

Hi Don-vip,

in Safari it works Out-Of-The-Box somewhat - Safari warns the user about a not trusted certificate for "127.0.0.1" with/without the patch. If you click on "Continue" in Safari it will open the connection.

With Firefox in won't work - it says that the Remote Control Mode is not activated with/without the patch. I'm not sure that I've imported the right classes - so here is the code that compiles in my Mac:

    @Override
    public void setupHttpsCertificate(KeyStore.PrivateKeyEntry privateKeyEntry)
            throws KeyStoreException, NoSuchAlgorithmException, CertificateException, IOException {
        KeyStore ks = null;
        try {
            ks = KeyStore.getInstance("KeychainStore", "Apple");
        } catch (NoSuchProviderException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        ks.load(null, null);
        Enumeration<String> en = ks.aliases();
        while (en.hasMoreElements()) {
            String alias = en.nextElement();
            Certificate c = ks.getCertificate(alias);
            if (ks.isKeyEntry(alias) && c.equals(privateKeyEntry.getCertificate())) {
                // JOSM certificate found, return
                return;
            }
        }
        // JOSM certificate not found, install it
        Main.info("Adding JOSM localhost certificate to Apple KeychainStore");
        ks.setEntry("josm_localhost", privateKeyEntry, new KeyStore.PasswordProtection("josm_ssl".toCharArray()));
    }

I've debugged the code, and it adds the certificate, but I don't know why it doesn't work - currently I've got no time to debug it.

[Don-vip]

Move tickets where work remains to next milestone

[Don-vip]

In 7335/josm:

see #10230, see #10033 - big rework of HTTPS support for Remote Control:

• HTTPS disabled by default, must be enabled in remote control preferences
• Old certificate and private key removed from jar and Windows keystore if found, even if remote control disabled
• New certificate generated at runtime with critical X509 extensions BasicConstraints (non-CA certificate), ExtendedKeyUsage (usage restriction for TLS server sessions)
• New passwords generated at runtime (but stored in clear in user preferences)
• Private key is no longer stored in Windows keystore (only certificate)

[Don-vip]

In 7336/josm:

see #10230, see #10033 - fix unit test

[Don-vip]

In 7337/josm:

see #10230, see #10033 - SAN tweaks + fix unit test (for real?)

[Don-vip]

In 7338/josm:

see #10230, see #10033 - JDK8 compatibility

[Don-vip]

I won't be able to finish the entire subject on this release.

[Don-vip]

In 7342/josm:

see #10230, see #10033 - fix certificate detection

[Don-vip]

In 7343/josm:

see #10230, see #10033 - add "Install/uninstall certificate" buttons in remote control preferences (Windows only)

[bastiK]

Replying to Don-vip:

I won't be able to finish the entire subject on this release.

I think it's no problem to delay it for a week or so, if that helps ...

[Don-vip]

I will need far more than a week :) Concerning Firefox (all platforms) and Linux for example, I fear we need to create a plugin :( Plus, I'm taking some days off this week, so the release is tonight (thank you Simon for the i18n!)

[bastiK]

If all the bugs are sorted out, then we can release of course.

[Don-vip]

I think I finally understand the situation with IE.

IE seems to ignore IP Addresses in SAN and suggest to use a DNS entry instead:
​https://connect.microsoft.com/IE/feedback/details/814744/the-ie-doesnt-trust-a-san-certificate-when-connecting-to-ip-address

The issue is (shamely) closed as wontfix. No progress to expect on this side.

Well, no problem, I tried to use a new entry dns:127.0.0.1. It could work... if there wasn't a Java bug that forbids that as well:
​https://bugs.openjdk.java.net/browse/JDK-8016345

Maybe we can extend or replace DNSName to remove this check:
​http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/tip/src/share/classes/sun/security/x509/DNSName.java

[Don-vip]

In 7347/josm:

see #10033 - add new entry dns:127.0.0.1 to make it work with IE

[Don-vip]

So, here's the certificate backend used by each browser per platform, and our status:

	Chrome	Firefox	IE	Safari
Windows	Windows	Own NSS	Windows	Windows
Linux	Shared NSS	Own NSS	N/A	N/A
Mac OSX	Keychain	Own NSS	N/A	Keychain

NSS support will likely require a plugin containing NSS and JSS libraries. The Keychain support however must be possible in core.

@Lesath: I have tweaked the certificate to match IE behaviour and it looks like it also pleases Safari on Windows, now. Can you please try again to add this code to PlatformOSX and install the certificate? I think it should allow to make the warning disappear completely:

    @Override
    public boolean setupHttpsCertificate(String entryAlias, KeyStore.TrustedCertificateEntry trustedCert)
            throws KeyStoreException, NoSuchAlgorithmException, CertificateException, IOException {
        KeyStore ks = KeyStore.getInstance("KeychainStore", "Apple");
        ks.load(null, null):
        // Look for certificate to install
        String alias = ks.getCertificateAlias(trustedCert.getTrustedCertificate());
        if (alias != null) {
            // JOSM certificate found, return
            Main.debug(tr("JOSM localhost certificate found in {0} keystore: {1}", tr("Apple Keychain"), alias));
            return false;
        }
        // JOSM certificate not found, warn user
        StringBuilder message = new StringBuilder("<html>");
        message.append(tr("Remote Control is configured to provide HTTPS support.<br>"+
                "This requires to add a custom certificate generated by JOSM to the Apple Keychain.<br><br>"+
                "You are now going to be prompted by OSX to confirm this operation.<br>"+
                "To enable proper HTTPS support, <b>please click Yes</b> in next dialog.<br><br>"+
                "If unsure, you can also click No then disable HTTPS support in Remote Control preferences."));
        message.append("</html>");
        JOptionPane.showMessageDialog(Main.parent, message.toString(), tr("HTTPS support in Remote Control"), JOptionPane.INFORMATION_MESSAGE);
        // install it to Apple Keychain, used Chrome and Safari, but not by Firefox
        Main.info(tr("Adding JOSM localhost certificate to {0} keystore", tr("Apple Keychain")));
        ks.setEntry(entryAlias, trustedCert, null);
        return true;
    }

Can you also tell me if there's any UI warning shown by OSX when installing the certificate (like Windows), or not? If not, the JOSM information message can be removed.

[mkoniecz]

see also ​https://github.com/openstreetmap/openstreetmap-website/issues/803

[Don-vip]

I'm not sure they can do anything from their side. Your issue is likely to be closed.
@Lesath: any news please?

[stoecker]

What they could do a osm.org is setup a osm.org https-josm-link, which redirects to a osm.org http-josm-link and this one then calls JOSM, so you get a JumpOutOfTLS function. But I don't know if this is a wise idea at all - Breaks the idea of HTTPS.

[Don-vip]

Move complicated/risky tickets to next milestone.

[Don-vip]

Not enough time/resources for these tickets this month.

[Don-vip]

won't have any time soon for this.

[Don-vip]

Ticket #11170 has been marked as a duplicate of this ticket.

[james2432]

Seems to be a browser safety more than an OS problem, I have installed the cert in JOSM(by clicking on the install button)
The browser when browsing to ​​https://127.0.0.1:8112/load_and_zoom?left=8.19&right=8.20&top=48.605&bottom=48.590&select=node413602999 gave me a security exception page in firefox. I have to accept the security exception (adds the cert to the firefox certificates) and the links now work.

[Stereo]

#14397 has another idea for macOS if ticket:10033#comment:22 doesn't work.

@Don-vip, I could test your code if you can create a build for me?

[Stereo]

​https://stackoverflow.com/questions/24333480/how-to-add-a-trusted-certificate-autority-to-firefox-with-jss has code to add certificates to the Firefox NSS. I can also test that :)

[Stereo]

Ticket #14397 has been marked as a duplicate of this ticket.

[Don-vip]

In 12458/josm:

see #10033 - remove workaround for IE (not needed anymore for IE11 on Windows 10, works also for Edge)

[nicolas17]

​Since Firefox 55 (​ticket) and Chrome 53 (​ticket), an HTTPS page loading http://127.0.0.1:port/ will work and will not give a mixed-content error. Note this only works for the IP address literals 127.0.0.1 and ::1, not for localhost.

Let's stop messing with self-signed certs.

[Stereo]

That's great news nicolas17, thanks!

This leaves us only with Safari on macOS then, although I would imagine that Safari users would tend to be iD users. I've tried the latest Safari, 11.0.2, and it indeed blocks e.g. <iframe src="http://127.0.0.1:8111/version" /> with "[blocked] The page at about:blank was not allowed to display insecure content from ​http://127.0.0.1:8111/version."

[nicolas17]

There is also a draft spec for ​CORS and RFC1918 which would add a variant of CORS for any requests going from public webpages to private addresses (like localhost), even if they are plain old GETs, even from an <img> or <iframe>.

As far as I know, no browser implements this yet. But when/if browsers implement it, then 1. JOSM needs to handle the OPTIONS preflight request and return the correct Access-Control-Allow-* headers, and 2. the website doing the request must be HTTPS.

[Don-vip]

Great news! We should be able to completely drop https support then, and it should work out of the box.

[Stereo]

Please re-open, Safari on macOS still needs this, unfortunately. And does IE/Edge not need https still?

[Don-vip]

It will be fixed by Safari, see ​https://bugs.webkit.org/show_bug.cgi?id=171934
And it has already been fixed in Edge: ​https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/11963735/

[stoecker]

Replying to Don-vip:

We should be able to completely drop https support then, and it should work out of the box.

There is no reason for this. Https support may have other reasons/uses, e.g. non-localhost access.

[Stereo]

I've never heard of anyone running josm non-locally you? The certificate wouldn't match anyway, and openstreetmap.org calls localhost, not an arbitrary host. If someone is willing to go through the trouble to fiddle with the josm certificate store, they're more than willing to nc -l 8111 | nc josm.example.com 8111

[Don-vip]

Replying to stoecker:

Https support may have other reasons/uses, e.g. non-localhost access.

I don't think it works, as we issue the certificate for "localhost". Even if it worked, there would be little added value of using https on a local network for this, as there is no credentials or personal information exchanged.

What I see however is the burden of maintaining this feature. We have a lot of platform-dependent code, use of internal Sun APIs, Java 9 command line arguments...

[Don-vip]

what it looks like if we remove HTTP support for remote control

[bastiK]

Props to Vincent for making it work in the first place, but I would very much welcome the application of this patch...

[Stereo]

​https://github.com/openstreetmap/openstreetmap-website/pull/1707 has some browser statistics - probably the closest we can get to reality.

Can anyone make ​https://stereo.lu/cors.html load in IE 11 while JOSM is running without getting warnings, maybe by adding ​http://127.0.0.1:8111 to the trusted sites?

[Don-vip]

OK so we have only about 40% of visitors using a compatible browser. Let's wait some months to see how the situation evolves:

• Next version of Edge will be released in March (with Windows 10 1803)
• Firefox ESR 60 will be released in May
• Safari: no idea, but there is at least a new version every year

@Stereo: don't waste your time for IE11 now. We'll see when the new version of Edge is released if IE11 benefits from this feature (IE11 and Edge share some code on Windows 10)

[stoecker]

what it looks like if we remove HTTP support for remote control

Hmm, I still don't like dropping HTTPS support much. I'm currently in the process of implementing TLS in several tools and dropping support sounds like the wrong way to me. Wouldn't dropping all the auto-gen-stuff and replacing it by loading a cert/key file instead also remove all strange dependencies?

[Don-vip]

Yes it would remove sun imports and java 9 command line arguments, and become an expert option.

[Don-vip]

Some (bad) news:

• WebKit surprised me by closing their ​bug as Wontfix. So Safari will never work: macOS users will have to use another browser for this feature if Webkit team persists to not follow a W3C standard, unlike all other browsers.
• OSM operation team ​suddenly switched to https only. So the feature does not work anymore for a lot of users I guess. ​This PR must be merged asap.

[Don-vip]

Replying to Don-vip:

• Next version of Edge will be released in March (with Windows 10 1803)

Windows 10 1803 is out, but Edge still doesn't work. I have asked Edge team about that.

[Don-vip]

Answer from Edge team received today:

Microsoft Edge Team

Thanks all for the feedback on this. We reviewed again, this time using jQuery for an XHR request and we can see that over the new Fetch networking stack with XHRs, this is indeed still an open issue. Coding is happening now for the next release and we will review for back level servicing also.

[Don-vip]

stats April 2018

[Don-vip]

I've made some stats for last month (April 2018):

This shows the number of hits on the JOSM website for significant known desktop browsers (are excluded: mobile clients, SVN, APT, JOSM itself, unknown or insignificant browsers). Edge does not appear because our awstats is too old. It is either counted as Chrome or Safari because of its user-agent. But I don't think it is used that much.

• Green: It works right now without enabling https support
• Orange: Users have to upgrade their browser. Given that Firefox 60 ESR has been released today, everyone should be able to in the short of medium term. I marked IE in orange as it will likely work on Windows 10 once bug mentioned above is fixed in Edge (the two browsers share a lot of code)
• Red: It will probably never work

I think the numbers are good enough to consider we could remove https support for remote control. I'm still in favour to drop it for the reasons explained earlier.

[Don-vip]

Patch updated. Dirk, no objection to apply it?

[stoecker]

As said: I don't like dropping it completely. I'd like to have a version where user does provide a crt and key, like all other applications which do provide HTTPS (like Apache, mail servers, ...).

[anonymous]

On the openstreetmap.org side, we've decided to set the http/https remote URL based on the user agent: ​https://git.osm.org/rails.git/commitdiff/50f8524

According to ​https://stats.openstreetmap.org/cgi-bin/awstats.pl?config=www.openstreetmap.org&framename=mainright&output=browserdetail, about 15% of openstreetmap.org visitors would be affected. On the other hand, according to openstreetmap.org's piwik, ff 52 is currently 1.6% and non-mobile safari is 2.4%.

How about leaving the code as vestigial for now, not removing it but not fixing it if it breaks? I'm afraid we'd encourage people to stick with old JOSM versions if we removed functionality too early.

[Don-vip]

Replying to stoecker:

As said: I don't like dropping it completely. I'd like to have a version where user does provide a crt and key, like all other applications which do provide HTTPS (like Apache, mail servers, ...).

Can you please make this change then?

[stoecker]

See #15748.

[stoecker]

@Vincent:

If I forget this (which is very likely) and you see it please give me a reminder mid of June. Otherwise it will have to wait until August.

[Don-vip]

OK :)

News of the day:

• Edge ​fixed the bug today. Now we must wait for release (date unknown)
• WebKit faces ​harsh criticism from Arduino for intentionally not following W3C specs. It's a pleasure to read :D

[stoecker]

Replying to Don-vip:

It's a pleasure to read :D

Indeed.

[Don-vip]

Keeping the sun.security imports makes it harder to compile with latest version of Ant with Java 10+, see #16451

[Don-vip]

​Microsoft news for Edge:

Based on flighting information, this fix has been approved for both our April 2018 and October 2017 releases of Windows 10.

Failing any issues we are yet to find, users on the April update who check for updates manually will get this fix the first week of August. Users on the October 2017 update who manually check will be sent the fix in the last week of August.

This will also be included in a mandatory security update in September for users on both releases.

EDIT: tested and approved! It works flawlessly with Edge now.

[Don-vip]

Replying to stoecker:

@Vincent:

If I forget this (which is very likely) and you see it please give me a reminder mid of June. Otherwise it will have to wait until August.

We're in August :)

[Don-vip]

Dirk, do you still plan to work on this, or is there a chance I convince you dropping https for remote control is fine?

[EdLoach]

If you do remove https for remote control then taginfo will need the JOSM button to know to use http rather than https as I see here for example: ​https://taginfo.openstreetmap.org/tags/amenity=meeting_point

[simon04]

Replying to EdLoach:

If you do remove https for remote control then taginfo will need the JOSM button to know to use http rather than https

This is the relevant code in Taginfo: ​https://github.com/taginfo/taginfo/commit/0e97f9b3773585d8fb76659dc710726a888c4851

Here are a few other usages of :8112, ​https://github.com/search?q=localhost%3A8112+JOSM+-repo%3Asidjy%2Fsidjy.github.io+-path%3Asrc%2Forg%2Fopenstreetmap%2Fjosm%2F&type=Code

Overpass Turbo: ​https://github.com/tyrasd/overpass-turbo/blob/c7ea2474b3120fa6f7e702f7014c632e45fbba01/js/ide.js#L2329-L2330

[anonymous]

That github search misses the clever cookies who used 127.0.0.1.

​https://github.com/search?o=desc&q=8112+JOSM+-repo%3Asidjy%2Fsidjy.github.io+-path%3Asrc%2Forg%2Fopenstreetmap%2Fjosm%2F&s=indexed&type=Code

[anonymous]

It also looks like Apple is having a change of heart for Webkit:

​https://bugs.webkit.org/show_bug.cgi?id=171934#c36

[marc_marc]

my josm v15322 stop listening on 8112 but overpass turbo seems to still using it ​https://github.com/tyrasd/overpass-turbo/blob/master/js/ide.js#L2329-L2330

[Don-vip]

Replying to marc_marc:

my josm v15322 stop listening on 8112

I'm still in favor of dropping https support, but didn't change anything. If you josm stops listening on this port, there's a problem somewhere: current josm should still be able to accept requests there.

[marc_marc]

Replying to Don-vip:

If you josm stops listening on this port, there's a problem somewhere

indeed :

Cannot start IPv4 remotecontrol https server on port 8112: Keystore was tampered with, or password was incorrect

keystore delete -> port listening
+ https certif readded into FF for overpass-turbo

[Stereo]

Overpass Turbo now calls JOSM over http: ​https://github.com/tyrasd/overpass-turbo/pull/417

I've also created PR on anything I could find on github that didn't look like an abandoned project.

Could we set a milestone for this maybe?

[Don-vip]

Dirk, do you plan to work on it or not? Otherwise we can simply drop https support and move on.

[stoecker]

Still planning to do something on it. Only I need to get in the mood again to do private software development. :-(

[Stereo]

Taginfo now calls JOSM over http. As far as I can tell, there are no significant projects that call josm over httpS anymore, and the https code is now vestigial that will never be called anymore. If someone wants to play with it, it could be a plugin? But I don't see the use case.

[Klumbumbus]

It seems ​https://www.keepright.at/report_map.php?zoom=14&lat=48.20808&lon=16.37221 still uses https.
current state for me:

• Firefox - doesn't work
• Opera - works
• Edge - works
• IE - can't test as the popups don't stay open

I guess keepright is not really maintained as the popups are glitched for years.

[Don-vip]

Safari developers ​changed their mind (or sacked the stubborn guy who refused to follow standards?). The browser is finally joining the other ones to accept 127.0.0.1 and ::1. It was the only remaining major browser where http mixed content does not work, and the only potential use case I saw.

[stoecker]

Aaaaah. Drop it. I wont have time for this soon.

[Don-vip]

ok, thanks!

[Don-vip]

In 15469/josm:

fix #10033, fix #15748, fix #17097 - drop remote control https support

Rationale: all modern browsers (including next version of Safari) allow mixed-content to localhost.

Cross-platform / cross-browser HTTPS support is a pain to maintain, was never completed, and is no longer needed.

[taylor.smock]

Replying to Don-vip:

In 15469/josm:

fix #10033, fix #15748, fix #17097 - drop remote control https support

Rationale: all modern browsers (including next version of Safari) allow mixed-content to localhost.

Cross-platform / cross-browser HTTPS support is a pain to maintain, was never completed, and is no longer needed.

Quick update: WebKit never got around to applying the patch (as of 2021-12-28).