loadAttributionText Throws NumberFormatException which gets swallowed by thread

[jhuntley]

The exception is thrown in BingAerialTileSource and gets swallowed by the thread making it difficult to detect. However, if you select the bing tab in the sample program provided with JMapViewer, you can tell something is wrong. All the tiles pop up with the 'X' images. I had to add an additional exception catch statement to view the actual exception:

        } catch (SAXException e) {
            System.err.println("Could not parse Bing aerials attribution metadata.");
            e.printStackTrace();
        } catch (ParserConfigurationException e) {
            e.printStackTrace();
        } catch (XPathExpressionException e) {
            e.printStackTrace();
        } catch (Exception e) {
        	e.printStackTrace();
        }

While the current implementation is cleverly using the Xpath parse techniques. I would like to propose a slightly more simple approach with more error checking to ensure data loads correctly. I have also introduced a TileLoadException to throw when any tile loading issues are encountered. I'm attaching source contributions in this ticket. Please review and contribute upon approval. Thanks!

[simon04]

While the current implementation is cleverly using the Xpath parse techniques. I would like to propose a slightly more simple approach with more error checking to ensure data loads correctly.

Just being curious: Why do you consider your approach simpler than using XPaths? I mean, when the concept of XPaths is known, the code can easily be read. Independently, some error checking can be implemented.

Using a special exception for tile loading makes sense to me.

[jhuntley]

Why do you consider your approach simpler than using XPaths?

Actually, I guess I didn't describe that quite correctly. I don't mean more simple, as in fewer lines and more readable. Xpath approach was simple, elegant, etc... I guess the DOM parse approach I've written is a step down or lower level. I inspect the document a little more closely, inspect each node level vs searching the doc with explicit paths or wildcarded paths. Explicit path search seems to be a little more delicate and break anytime the doc changes from the source. EX:

imageryZoomMax = Integer.parseInt(xpath.compile("//ImageryMetadata/ZoomMax/text()").evaluate(document));

The above example fails to parse the the int value from the xml doc. It actually throws the exception which gets swallowed by the thread. Breaking the statement into segments reveals the string value=null before the parse:

            XPathExpression exp=xpath.compile("//ImageryMetadata/ZoomMax/text()");
            String strValue=exp.evaluate(document);
            imageryZoomMax = Integer.parseInt(strValue);

When you pass null into parseInt, you'll get a NullPointerException which is typical for the parse methods in Integer,Double, etc... I'm sure your aware of this already.

I tinkered with the XPath syntax a bit and just couldn't get XPath to return the proper node as you expect, only null. Plus having the call parse all in one line made it difficult to debug. I kept having to separate out all the calls to get to the root of the problem, like in the example above. After some time, i just gave up and decided to go with an approach i know worked, and I could step through easily in my debugger. I imagine there may be something else like an i18n issue or some other character parse issue going on here. Someone with more xpath experience could probably find an equal fix to this as well, with the existing code.

I just like being able to check if a specific node exist before parsing the value, possibly eventually returning an error message related to exact problematic nodes while parsing doc. The version i've supplied will also continue to parse the remaining doc even if some of the nodes have been removed. Of course all will fail if you can't get the actual grid coordinates.

Anyways, that's my reasoning behind it all. Pass or fail, I won't be offended if you choose to update the existing code base with a different solution, as I hope none of you were offended by my proposal. All i care about is getting the issue resolved and making sure I can continue to use your precious code base. Let me know if there's anything else I can do to help. Thanks!

[bastiK]

Good to know, that exceptions in loadAttributionText() can get lost. We should probably catch "Exception" in the BingAerialTileSource constructor.

Agree with simon04, it should be possible to integrate these additional checks into the existing code. Instead of XPath you created your own meta-language in XMLTileParseUtil. (I don't understand the problems with XPath, maybe a small standalone example could demonstrate the issue?)

[jhuntley]

(I don't understand the problems with XPath, maybe a small standalone example could demonstrate the issue?) 

That's what I mentioned in my initial comment regarding. I spent a considerable amount of time trying to get the following to work:

            XPathExpression exp=xpath.compile("//ImageryMetadata/ZoomMax/text()");
            String strValue=exp.evaluate(document);

For whatever reason, it just keeps returning null. The syntax is correct and the path. The raw XML has data for that element, '21'. However, it keeps returning null. I had no intentions of writing XMLTileParseUtil except that it's the only thing i could get to work consistently. I'll go back and try the xpath statement again, but it might help to get another pair of eyes on this.

FYI, went back and verified my development environment to make sure it adheres to guidelines. I'm using the right JDK, 1.6.

I'll check out xpath again and give an update in a few. BTW, you guys have an IRC channel? Didn't see one on the wiki.

[jhuntley]

I guess I should also mention that it's not just that example which returns null. All the exp.evaluate calls return null.

[jhuntley]

So reverting back to the old code. If you open the 'latest' demo app, select bing in drop-down, zoom in. You will encounter the following exceptions as a result of the exp.evaluate calls:

failed loading 4/6/3 null
failed loading 4/8/3 null
failed loading 4/9/3 null
failed loading 4/7/3 null
failed loading 4/10/3 null
java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: ""
	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
	at java.util.concurrent.FutureTask.get(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.getAttributionText(BingAerialTileSource.java:310)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintAttribution(JMapViewer.java:960)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintComponent(JMapViewer.java:591)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintChildren(Unknown Source)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintChildren(Unknown Source)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JLayeredPane.paint(Unknown Source)
	at javax.swing.JComponent.paintChildren(Unknown Source)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintToOffscreen(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paint(Unknown Source)
	at javax.swing.RepaintManager.paint(Unknown Source)
	at javax.swing.JComponent._paintImmediately(Unknown Source)
	at javax.swing.JComponent.paintImmediately(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.seqPaintDirtyRegions(Unknown Source)
	at javax.swing.SystemEventQueueUtilities$ComponentWorkRequest.run(Unknown Source)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.loadAttributionText(BingAerialTileSource.java:108)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.access$0(BingAerialTileSource.java:92)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:54)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
failed loading 4/9/7 null
failed loading 4/8/7 null
failed loading 4/7/7 null
failed loading 4/10/6 null
failed loading 4/6/7 null
failed loading 4/10/4 null
failed loading 4/10/5 null
failed loading 4/5/7 null
failed loading 4/5/6 null
failed loading 4/5/5 null
failed loading 4/5/4 null
failed loading 4/5/3 null
failed loading 4/11/3 null
failed loading 4/11/4 null
failed loading 4/11/5 null
failed loading 4/11/6 null
failed loading 4/4/6 null
failed loading 4/4/5 null
failed loading 4/4/7 null
failed loading 4/4/3 null
failed loading 4/4/4 null
failed loading 4/10/7 null
failed loading 4/11/7 null
java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: ""
	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
	at java.util.concurrent.FutureTask.get(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.getAttributionText(BingAerialTileSource.java:310)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintAttribution(JMapViewer.java:960)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintComponent(JMapViewer.java:591)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintToOffscreen(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paint(Unknown Source)
	at javax.swing.RepaintManager.paint(Unknown Source)
	at javax.swing.JComponent._paintImmediately(Unknown Source)
	at javax.swing.JComponent.paintImmediately(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.seqPaintDirtyRegions(Unknown Source)
	at javax.swing.SystemEventQueueUtilities$ComponentWorkRequest.run(Unknown Source)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.loadAttributionText(BingAerialTileSource.java:108)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.access$0(BingAerialTileSource.java:92)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:54)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: ""
	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
	at java.util.concurrent.FutureTask.get(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.getAttributionText(BingAerialTileSource.java:310)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintAttribution(JMapViewer.java:960)
	at org.openstreetmap.gui.jmapviewer.JMapViewer.paintComponent(JMapViewer.java:591)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintToOffscreen(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(Unknown Source)
	at javax.swing.RepaintManager$PaintManager.paint(Unknown Source)
	at javax.swing.RepaintManager.paint(Unknown Source)
	at javax.swing.JComponent._paintImmediately(Unknown Source)
	at javax.swing.JComponent.paintImmediately(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
	at javax.swing.RepaintManager.seqPaintDirtyRegions(Unknown Source)
	at javax.swing.SystemEventQueueUtilities$ComponentWorkRequest.run(Unknown Source)
	at java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at java.lang.Integer.parseInt(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.loadAttributionText(BingAerialTileSource.java:108)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource.access$0(BingAerialTileSource.java:92)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:54)
	at org.openstreetmap.gui.jmapviewer.tilesources.BingAerialTileSource$1.call(BingAerialTileSource.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Actually, I've identified the problem. I'll write an example up for you.

[jhuntley]

I've taken the bing response and save it to disk so I can parse through it for UTF sequenses. I was able to trim down the response to a specific location in the Copyright and various other places. Just loading this cause xpath to fail:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Copyright>Copyright © 2011 Microsoft and its suppliers. All rights reserved. This API cannot be accessed and the content and any results may not be used, reproduced or transmitted in any manner without express written permission from Microsoft Corporation.
</Copyright>
</Response>

Response:

org.xml.sax.SAXParseException: unsupported five or six byte UTF-8 sequence (character code: 0xa9)

I imagine that copyright character is despised by xpath?

Here's all you need to test the code:

File f=new File("C:/projects/workspace/test/AerialTest.htm");

FileInputStream streamIn   = new FileInputStream(f);

DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
Document document = builder.parse(streamIn);

XPathFactory xPathFactory = XPathFactory.newInstance();
XPath xpath = xPathFactory.newXPath();
imageUrlTemplate = (String) xpath.evaluate("/Response/Copyright", document, XPathConstants.STRING);

Trying to load ImageUrl with utf sequence included also fails:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Copyright>Copyright © 2011 Microsoft and its suppliers. All rights reserved. This API cannot be accessed and the content and any results may not be used, reproduced or transmitted in any manner without express written permission from Microsoft Corporation.
</Copyright>
<ImageryMetadata>
<ImageUrl>http://ecn.{subdomain}.tiles.virtualearth.net/tiles/a{quadkey}.jpeg?g=790</ImageUrl>
</ImageryMetadata>
</Response>

imageUrlTemplate = xpath.compile("//ImageryMetadata/ImageUrl/text()").evaluate(document);

Remove the utf sequence and it works:

<?xml version="1.0" encoding="utf-8"?>
<ImageryMetadata>
<ImageUrl>http://ecn.{subdomain}.tiles.virtualearth.net/tiles/a{quadkey}.jpeg?g=790</ImageUrl>
</ImageryMetadata>
</Response>

I'm not sure why xPath balks differently when you decide to include the whole doc. Regardless the calls fail when multibyte characters are included in the doc.

I googled for work arounds and read the java doc for xPath, ​http://download.oracle.com/javase/6/docs/api/javax/xml/xpath/package-summary.html. I didn't find much out there other than crude hacks. For example, in order for the xpath code to work, we would have to strip the response of multibyte characters before passing to XPath, something I don't recommend. It could possibly corrupt the content from the Bing service.

My recommendation, and you can take or pursue a different solution, would be to keep the source I have provided. We should also add comments to the source regarding the UTF and multibyte char support.

All hail the copyright char ©!

[bastiK]

I compiled your example and it works without problems (Ubuntu, Oracle Java 1.6.0_26-b03).

Are you 100% sure, your test file "AerialTest.htm" is encoded in utf-8? Maybe your Java installation is broken. It should be possible to try another version of Java (e.g. 1.7) alongside without installation (i.e. just unpack to a folder).

[simon04]

I'm not very familiar with Java's XML implementation, but perhaps different parsers are returned by the following snipped …

DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();

IMHO, Java's XML implementation doesn't follow the KISS principle at all.

[jhuntley]

Are you 100% sure, your test file "AerialTest.htm" is encoded in utf-8?

The source is straight from the bing url:
​http://dev.virtualearth.net/REST/v1/Imagery/Metadata/Aerial?include=ImageryProviders&output=xml&key=Arzdiw4nlOJzRwOz__qailc8NiR31Tt51dN2D7cm57NrnceZnCpgOkmJhNpGoppU

I actually didn't make it up; i just cropped it out from Bing's response. I'm not sure how MS is encoding the doc, I'm not very familiar with bing, but I believe the copyright encoding for the character is ‘©’.

I'm using JDK version "1.6.0_21" hosted on Windows 7. The dev guidelines say to use a compatible 1.6. It could be a bug in the windows JVM. I'll check.

[simon04]

To check the encoding of the modified file, you can open it in Firefox and take a look at or play around with View » Character Encoding …

[jhuntley]

Replying to simon04:

To check the encoding of the modified file, you can open it in Firefox and take a look at or play around with View » Character Encoding …

I'm aware, thanks. Regardless, you should be able to tell the encoding in the first line of the response, encoding="utf-8". That is if Bing set it correctly along with the character encodings. Just looking at the raw text in the response, there are no real UTF8 character references. The copyright symbols aren't encoded either. We could try manually converting the doc:

DocumentBuilder.parse(new InpputSource(new InputStreamReader(inputStream, "<real encoding>")));

However, you say the the primitive example i provided parses the copyright correctly on your machine, without exceptions?

I upgraded my JDK to the latest 1.6, 1.6.0_27.

[bastiK]

Replying to jhuntley:

Replying to simon04:

To check the encoding of the modified file, you can open it in Firefox and take a look at or play around with View » Character Encoding …

I'm aware, thanks. Regardless, you should be able to tell the encoding in the first line of the response, encoding="utf-8". That is if Bing set it correctly along with the character encodings. Just looking at the raw text in the response, there are no real UTF8 character references. The copyright symbols aren't encoded either.

I don't really understand your reply. Every single character (including the copyright symbol) is encoded when text is saved or transmitted. What do you mean by character references?

We could try manually converting the doc:

DocumentBuilder.parse(new InpputSource(new InputStreamReader(inputStream, "<real encoding>")));

Not a good idea, it is handled automatically. In this case it does not work, but it is probably your fault, not Java's. ;-)

However, you say the the primitive example i provided parses the copywrite correctly on your machine, without exceptions?

Yes. Your OS will return a different value for Charset.defaultCharset() than mine, but it shouldn't matter.

Could you change the first line of the xml document from <?xml version="1.0" encoding="utf-8"?> to <?xml version="1.0" encoding="ISO-8859-1"?> and see if there is a difference?

[jhuntley]

I don't really understand your reply. Every single character (including the copyright symbol) is encoded when text is saved or transmitted. What do you mean by character references? 

I'm referring to entity references:

• ​http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
• ​http://www.copyrightauthority.com/copyright-symbol

Could you change the first line of the xml document from <?xml version="1.0" encoding="utf-8"?> to <?xml version="1.0" encoding="ISO-8859-1"?> and see if there is a difference? 

I'll check it.

[jhuntley]

Sorry to waste your time. This turned out to be a classpath issue. There was a gnujaxp.jar file in the classpath. Wow. Not sure what was using it, but it clearly wasn't needed and was causing this issue.