ELI (Editor Layer Index) does not load

[zelonewolf@…]

See also ​https://github.com/osmlab/editor-layer-index/issues/2337

Adding the ELI to JOSM results in no imagery being listed in the imagery list.
JOSM version 19128

Reproduce:

1. Follow the instructions here to add the ELI URL to JOSM's preferences and remove the JOSM URL:
2. Refresh the imagery list using the square refresh button
3. Result: no imagery layers listed

[taylor.smock]

See #23485.

The problem is that we require an imagery category in the XSD. I don't know the reasoning for that off the top of my head.

The reason why things changed recently is we do more validation of requirements in the XSD that cannot be done via XSD due to technical limitations. See source:trunk/resources/data/maps.xsd@19130:638-640,686-688#L638​ .

Anyway, the "easy" solution would be to add an arbitrary category to each entry. It does look like the ELI has a category property that can be used (and the JOSM category might be based off of that, I don't know).

EDIT:
In other words, ELI is producing a technically invalid xml file. It worked previously because we weren't enforcing all of the validity requirements.

[anonymous]

Is it expected that loading ELI in its current form causes a silent failure? There is no indication that anything failed to load other than the empty imagery list.

[taylor.smock]

I literally just added a logging section (locally) so that the failure isn't totally silent. It still won't show up in the UI though.

[taylor.smock]

In 19131/josm:

See #23785: Log when an imagery layer is removed due to validity reasons

[anonymous]

Thank you!

[stoecker]

Replying to taylor.smock:

The problem is that we require an imagery category in the XSD. I don't know the reasoning for that off the top of my head.

We use the category to decide if entries are displayed automatically or not, thus a category isn't optional.

P.S. @ticket author: Any reason why you want to load the ELI list?

[zelonewolf@…]

I want to load the ELI list because it is consistently up to date with imagery in my area (Rhode Island, USA) and currently lists 4 separate imagery layers from 2021 to 2023 when I edit with iD. The JOSM list only has a single imagery layer from 2019 and it has been in a broken state for a very long time. I thought it would just be easier to load in the ELI since someone seems to be regularly adding imagery in my area to ELI.

[taylor.smock]

As an FYI, you can add those imagery layers to the JOSM list. You just have to do a bit of fiddling.

https://josm.openstreetmap.de/wiki/Maps/USA%20States

Looking at wiki:ImageryCompare, it does appear that we are missing the updated layers from Rhode Island.

I would highly recommend adding them to the JOSM index instead of using the ELI. Specifically because the JOSM index is the default, and this will help other people editing in your area.

Anyway, it looks like someone should go through the new ELI urls. And maybe fix mismatches between the ELI and our index.
*** URLs found in ELI but not in JOSM (432): ***

I'll do that if I have a spare hour or two sometime. And remember it.

[stoecker]

We stopped the ELI→JOSM synchronization as it was always a one-way process. ELI never attempted to fix their errors or make synchronisation easier. Seeing that ELI is a clone of the JOSM imagery storage, sadly that's the fact from the very beginning.

@zelonewolf:
As Taylor says you can edit the WIKI yourself. This means adding, removing, changing entries. In case adding entries is to complicated for you: The main task is always checking what is new and better and where ELI is old and outdated - so please post the exact names of missing valid entries in a ticket (like here) and I or another one can copy them from ELI. The scripts in the SVN generate output from ELI, which can be copied 1:1 to the wiki.

@Taylor:
I fear your time is better spent for other tasks... Checking each and every entry is a lot of work and to get it right you have to fix both datasets which simply makes little sense (or the ignore list grows a lot, current revision number 1465 for that list gives an indication). As soon as you stop it will go out of sync again as it seems there is no cooperation possible.

[taylor.smock]

Replying to stoecker:

@Taylor:
I fear your time is better spent for other tasks...

Kind of why I said "if I have a spare hour or two sometime." Which is unlikely. I might look into adding one missing entry every day or two though.

[stoecker]

Replying to taylor.smock:

Replying to stoecker:

@Taylor:
I fear your time is better spent for other tasks...

Kind of why I said "if I have a spare hour or two sometime." Which is unlikely. I might look into adding one missing entry every day or two though.

Is there the iD dataset available somewhere in easy accessible form (best a single URL)? Syncing iD against JOSM would be much more useful (iD is mainly an ELI subset).

[taylor.smock]

It looks like they parse it; ​https://github.com/facebook/Rapid/blob/main/package.json#L37 -> ​https://github.com/facebook/Rapid/blob/main/data/imagery.json . Or ​https://github.com/openstreetmap/iD/blob/develop/data/imagery.json for iD.

TBH, I'd rather do Rapid than iD since it is easier for me to poke them than the iD developers.

[stoecker]

I'll see if I can add sync for these two links ;-)

As far as I know changes are still done at ELI, the scripts should only decide what they present and what not.

[zelonewolf@…]

Great, sounds like you guys have a handle on this. If it's useful I can make a new ticket pointing out the missing Rhode Island imagery layers (basically all four that appear at ​https://osmlab.github.io/editor-layer-index/) but it sounds like you have things sorted.

[stoecker]

Rhode Island changes: Maps/USA States

[taylor.smock]

I was just thinking it might not be a bad idea to extend the imagery-xml-bounds plugin to make editing the wiki pages easier (think "load existing imagery", "add new imagery", etc.). The big problem with the wiki pages is that there is a lot of text people have to go through, and editors need to semi-understand XML.

@zelonewolf: I typically try to get people looking for new imagery to add it to the wiki themselves. I do this in an effort to extend/improve the knowledge of other people, reduce the bus factor for new imagery, etc. Yes, we (the JOSM core developers) can add imagery entries to the wiki pages, but is it the best use of our time? I'd rather help someone learn how to add new imagery entries than do it myself, since hopefully that means they will add other entries they find of interest.

[zelonewolf@…]

I would add them myself, but I don't really understand how it works and I can barely stumble through getting a manual entry to load, and all the various WMS, TMS, etc formats don't mean anything to me and there's no obvious documentation on those wiki links. I can edit a complex land cover multipolygon, but these imagery sources are over my head.

Look at it from my perspective. This is my starting point:
​https://maps.edc.uri.edu/rigis/rest/services/IMG/RI_202403_RGB_3in_web/ImageServer

I have no idea how to stick what's in there into that complex XML format and no idea where I'm supposed to get the bounds polygon from. That's why I want to load ELI where someone who knows who this stuff works has already done the work for me. So like I said, if it's useful to the project to make tickets here when I find one missing, I'm happy to do that -- but if that's just making busy work for someone, I won't.

[taylor.smock]

OK. In that case, from your perspective, would it help if there was a plugin that would ask questions and help you add the imagery source to our list?

I would assume so, but assumptions can be wrong.

---

Synchronizing imagery entries isn't busy work, since it is actually useful to someone. It just isn't something that is "high" priority for me (otherwise I'd take time to go through the list).

I really only look at that stuff when someone makes a ticket, since that is the only way I know that the entry is actually used.

[zelonewolf@…]

Yes, that would definitely make things easier. Probably the easiest thing would be, if the user can get the imagery layer working in JOSM, to be able to persist that, or get the output of it to paste in somewhere. Because then you can verify on the JOSM side that it actually works correctly before submitting it.

[stoecker]

Replying to zelonewolf@…:

Yes, that would definitely make things easier. Probably the easiest thing would be, if the user can get the imagery layer working in JOSM, to be able to persist that, or get the output of it to paste in somewhere. Because then you can verify on the JOSM side that it actually works correctly before submitting it.

I now added a link to the ELI dataset in JOSM format for Copy&Paste to ImageryCompare page. This wiki has a preview feature which allows to check if the entry works. This isn't 100% feature compatible with JOSM, but works for the majority of entries (TMS, WMS). Also the page validates the XML structure, so when the preview worked it usually will also work in JOSM.

Sync now also shows some information about iD and Rapid to know which entries probably are important.

Like OSM itself JOSM also depends on others to help, this includes the imagery pages as well ;-)

[stoecker]

P.S. Docs are here: wiki:Maps#Documentation (linked on the ImageryCompare page).

If ELI has valid entries, simply copy them. There is no need to redo that, as setting up new entries from scratch is usually a bit overwhelming.

Copying, fixing and deleting (outdated) existing entries is much easier ;-)