Mechanism to modify JOSM settings and store files

[stoecker]

Currently we have a way to access presets and styles, but some of these requires changing configuration. We would need a way (e.g. using a XML format) to modify josm settings:

• adding new entries
• modifying entries
• adding new elements to preferences collections

E.g. Chaning colors, adding new WMS sources or combined presets/mappaint styles would be the target.

Should a bit like the way windows registry edit files work with registry editor.

The system should include a mechanism to store files inside of JOSM's folder structure. Also probably an uninstall mechanism would be a good idea.

[DiverCTH <chunter952@…>]

As a member of the US Local Chapter working group, I second this feature request.

[akks]

Let us try to implement this at last.

If people prefer XML, I propose format like this:

<config>
    <preferences operation="append">
        <tag key='expert' value='true'/>
        <tag key='gui.geometry' value='x=0,y=0,width=1680,height=1050'/>
        <tag key='gui.maximized' value='false'/>
          <!-- josm prferences format here, all preferences replace existing ones-->
          
        <list key='imagery.layers.default'>
            <entry value='http://{switch:a,b,c}.www.toolserver.org/tiles/bw-mapnik/{zoom}/{x}/{y}.png'/>
            <entry value='....'/>
        </list>
        <!-- entries are added to existing list if needed (if the list does not contain them)-->
    </preferences>

    <preferences operation="replace">
        <!-- josm prferences format here, all preferences replace existing ones or are created-->
         
        <list key='imagery.layers.default'>
            <entry value='http://{switch:a,b,c}.www.toolserver.org/tiles/bw-mapnik/{zoom}/{x}/{y}.png'/>
        </list>
        <!--  after this operation list contains only one item - the list was replaced -->
    </preferences>

    <preferences operation="deleteKeys" pattern="cache.*"/>
       <!--  delete whole keys -->

    <preferences operation="list-delete" key="imagery.layers.default" entryvalue="" /> 

    <preferences operation="map-delete" key="some.map.key" entrykey="entry" />
       <!--  delete entries from map -->

    <preferences operation="map-delete" key="some.map.key" entryvalue="entry" />
       <!--  delete entries from map -->

    <preferences operation="maps-delete" key="imagery.entries" index="1" />
       <!--  delete map from maps? how to choose which map? by index? -->
       
    <preferences operation="maps-delete" key="imagery.entries"/>
       <!--  delete map from maps by finding matching map? -->
             <tag key='name' value='OpenStreetMap (Mapnik)'/>
             <tag key='type' value='tms'/>
    </preferences>

    <preferences operation="lists-delete" key="mappaint.style.sources-list" index="1" >
       <!--  delete list from lists? how to choose which list? by index? -->
    
    <preferences operation="lists-delete" key="mappaint.style.sources-list">
          <entry value='http://josm.openstreetmap.de/josmfile?page=Styles/Whitewater&amp;zip=1'/>
          <entry value=''/>
       <!--  delete matching list from lists? -->
    </preferences>


    <plugin operation="install" name="reverter" />

    <plugin operation="install" name="reverter" url="...."/>

    <plugin operation="uninstall" name="reverter" />
    
    <file operation="delete" name="cache/1.txt" />

    <file operation="download" name="cache/1.txt" url="http://...."  />
    
    
</config>

[akks]

Operation with preferences will be performed with standard Main.pref API, not in preferences.xml.

Some operations can be implemented later.

Please propose your simplifications or improvements :)

[stoecker]

Some notes:

a) To "append" to a key usually you will need the default value, which must not be known yet. It may be necessary to ask the user to activate the corresponding action first, which is not easy, as you can't name it - Example: Adding another source list for Imagery data (e.g. my personal one :-)

b) The map/list delete probably is best done using a regexp style pattern as argument.

c) I wouldn't make anything like plugin-operation. This can be done by generic file and key operations. If you need less specific handling you get a much more powerful interface.

d) Probably it would be a good idea to group actions into tasks.

e) Maybe add an <ask> to make tasks optional. F.e. a "Do you want to change colors", ... User visible text should follow the generic josm translation structure then: text="", de.text="..." Ask mya be boolean or provide more "numbered" choices.

[akks]

As I see, append and replace operations need to create separate Preferences object, read XML fragment to it and perform merge or replacement operation using Main.pref .

All other operations (regexp-based deletion) can use their own syntax. Installing plugin may be removed (but an example with download and preferences-modify should be provided).

I do not know how to deal with default values and prefer to ignore them :)

I have chosen chose DOM for reading changes.xml, separate node will be extracted and for preference readers.

Please people give me some advice!

[akks]

Puff, it was not easy :) Append and replace are somehow implemented and seem to work. Please look at the code or try testing this functionality (Advanced preferences - Read file button, ​http://dl.dropbox.com/u/63393258/josm-custom.jar).

I will better wait for comments and corrections before adding more.

[stoecker]

Replying to akks:

I do not know how to deal with default values and prefer to ignore them :)

Check if default is available when required. If not, inform the user, that corresponding software function needs to be called first. Present the config string, so user gets a slight idea what function it may be. Also add a note that activating the wrong function does no harm.

[bastiK]

Replying to akks:

I do not know how to deal with default values and prefer to ignore them :)

If default is unknown, save additions as normal preference, but add a flag "append" to this setting. When the setting is read (Main.pref.getCollection('foo', defaultVal);) return concatenation of provided default value and entries to be appended.

If you want to make this work across sessions, the flag also needs to be saved in preferences.xml:

<maps key='imagery.entries' operation='append'>
    <map>
      <tag key='name' value='ScanEx IRS'/>
      <tag key='type' value='scanex'/>
      <tag key='url' value='IRS'/>
      <tag key='attribution-text' value='IRS'/>
      <tag key='attribution-url' value='http://irs.gis-lab.info/'/>
    </map>
</maps>

This is probably necessary because users might restart JOSM immediately after applying the preference modifications.

Btw., you can try to use getAllSettings and putSetting in order to abstract over different setting types.

[akks]

My brain is melting :) So, for this specific key as an example

• If there is no imagery.entries in the file, some default list of maps is associated with it
• Default in unknown until the first call of Main.pref.getListOfStruct("imagery.entries",someDefaultList)
• If we want to append additional maps to the default, we must take this default value into account. If it is in defaults, collectionDefaults, etc. (not null), we can use it. If there is null there - we have a problem :)
• There is no "default" problem for replace and for single value properties.
• One possible (and more simple) solution is to ask user to call needed software function and then repeat the attempt.
• Another solution is to remember "maps = [possibly unknown default] + [specified maps]" in preferences XML, so user will get correct preferences after restart. Flag should be added (and stored in XML) and all preferences retrieving function need to be modified in Preferences.java.

I do not think I can make proper modification of Preference lists/maps/list getter functions - they are too complex and have some "workarounds to remove in 2012".

• Deletion by key does not have "default" problem - default value will be used for deleted key after restart.
• Deletion by list/map values or pattern cause biggest "default" problems if some part of default values need to be deleted. Remembering preferences (or even patterns) "that should be deleted when they appear" is, hmmm... a very strange thing :) Do we really need such deletion?

Did I understand correctly?

[stoecker]

"maps" is a bad example, as this does not use the default mechanism. There are no JOSM internal map defaults.

One possible (and more simple) solution is to ask user to call needed software function and then repeat the attempt.

I would start with this one and use Paul's better integrated approach when the functionality real gets used more often and users complain about it.

so user will get correct preferences after restart

Not after restart, but after first usage ...

all preferences retrieving function need to be modified in Preferences.java

Probably this can be reduced to the putXXXDefault() functions.

To explain default handling: We only store settings which are different to defaults.

• to save space
• to have the chance to change default values

This means that after loading the information is not known until a call for the corresponding preference settings has at least been done once (i.e. by calling the corresponding function).

[bastiK]

Replying to akks:

My brain is melting :) So, for this specific key as an example

• If there is no imagery.entries in the file, some default list of maps is associated with it
• Default in unknown until the first call of Main.pref.getListOfStruct("imagery.entries",someDefaultList)
• If we want to append additional maps to the default, we must take this default value into account. If it is in defaults, collectionDefaults, etc. (not null), we can use it. If there is null there - we have a problem :)
• There is no "default" problem for replace and for single value properties.
• One possible (and more simple) solution is to ask user to call needed software function and then repeat the attempt.
• Another solution is to remember "maps = [possibly unknown default] + [specified maps]" in preferences XML, so user will get correct preferences after restart. Flag should be added (and stored in XML) and all preferences retrieving function need to be modified in Preferences.java.

I do not think I can make proper modification of Preference lists/maps/list getter functions - they are too complex and have some "workarounds to remove in 2012".

• Deletion by key does not have "default" problem - default value will be used for deleted key after restart.
• Deletion by list/map values or pattern cause biggest "default" problems if some part of default values need to be deleted. Remembering preferences (or even patterns) "that should be deleted when they appear" is, hmmm... a very strange thing :) Do we really need such deletion?

Did I understand correctly?

Yeah, that's it in a nutshell...

A better example would be Main.pref.getCollection("imagery.layers.sites", Arrays.asList(DEFAULT_LAYER_SITES))): You may want to add another source of imagery layers, but better not kill the default one.

All in all it seem to be a lot of work to cover some rare special cases. I can help to adapt the lists/maps/list getter functions, but I'm not too much into this project.

It may be easier, if preference.xml with flags and the new preference modification was essentially the same format. (Like we use *.osm files with action="modify/delete" to save edits.)

[akks]

OK then, I'll try to implement all operations not touching Preferences.java for now, using Preference.readXML() for reading preference fragments. Like it is now, but with defaults handling.

Later we can change format a little (operation='append' for separate lists/maps) and provide automatic merging without bothering user :)

[akks]

I tried to add default handling and warnings, deletion of keys and collection elemnts, downloading files and deleting them.
Please have a look at it when you have some time.

[akks]

I have added unzipping support by invoking "jar xf ..." (tested on Windows).
Complied version: ​http://dl.dropbox.com/u/63393258/josm-custom.jar

Late after review and debugging we want to expose some functions for plugins. Current downloading and unzipping lacks progressbar - do not know how to use it correctly...

Here is updated patch and example XML:

[stoecker]

No external file calls in core except the one exception of calling a web-browser. See e.g. for presets and styles how unzipping is done properly. Is really a simple task without such "broken" workarounds.

[akks]

Replying to stoecker:

No external file calls in core except the one exception of calling a web-browser. See e.g. for presets and styles how unzipping is done properly. Is really a simple task without such "broken" workarounds.

Ok, ok:) Though it is not so simple for archive with folders (patch will be posted tomorrow)

[akks]

Internal unzipping implemented. All methods became static, some of them - public (for simpler usage from different parts of JOSM and plugins).

I would like to ask bastiK to add some support for preference keys exporting to output stream, maybe after tested is released (method for extract specific part of preferences.xml programmatically - it can be used then to save and even share imagery, offsets, plugins, shortcuts, etc.)

What is next? Are there any "must have" features left?

[bastiK]

Replying to akks:

I would like to ask bastiK to add some support for preference keys exporting to output stream, maybe after tested is released (method for extract specific part of preferences.xml programmatically - it can be used then to save and even share imagery, offsets, plugins, shortcuts, etc.)

No need to add support, this is already possible at the moment. Have look at public String toXML(boolean nopass). Do this with one Setting object rather than with all the settings and omit <preferences> element at the beginning and the end.

[akks]

Replying to bastiK:

No need to add support, this is already possible at the moment. Have look at public String toXML(boolean nopass). Do this with one Setting object rather than with all the settings and omit <preferences> element at the beginning and the end.

Thank you! Preferences.java is really big, but its reusing potential is really amazing :) It seems we can finish this ticket without touching it then.

I am adding exporting keys, <task> and <ask>, maybe variables ${{x}} and <if>. I am also separating XML processing from other work in CustomConfigurator.

[akks]

I have added task, ask, messagebox, if/else, variables (<var name="x" value="1"/>) and keys exporting button in Advanced preferences. Also after importing preferences table updates correctly.

I also tried to make progressbar (PleaseWaitRunnable), but it needs bidirectional user interaction and access GUI not from EDT, so I removed it (code is commented).
Plugin installation may be some problem, because it need versioning.

Now I thisk my part is finished and wait for testing and reviews...

Only advanced preferences tab is extendedchanged and new class introduced, so nothing can be broken except introduced actions :)

[akks]

This was last addition - <plugin install="a;b;c" uninstall="d;e;f" /> and messagebox "Do not forget to restart".

@team: Please have a look at it when you have time. Compiled version, as usual: ​http://dl.dropbox.com/u/63393258/josm-custom.jar

[akks]

In last patch version I added support for deleting plugin jars - <plugin delete="A;B" /> makes uninstall & delete.
Also some warnings were added (no preferences to save, etc.) and correct file dialogs were used (XML only, allow multiple files import). Do not forget to test or review it :)

[bastiK]

This is advanced stuff. I'm wondering if there's some kind of framework that we can build on, like XSLT, so we don't have to create all from scratch.

• It may be useful to delete values by pattern (regex) and delete a nonempty directory.

• Someone might want to delete files from cache. It should be expected, that cache folder is not a subdirectory of the normal josm pref directory. (E.g. on another disk, shared cache for multiple profiles, etc.).

• I'd prefer ${x} instead of ${{x}}.

• Maybe some kind of summary of the concrete changes in a final confirmation dialog.

• Change wording in the GUI: It seems as if after loading a xml file all my previous preferences might be dropped.

[akks]

Replying to bastiK:

This is advanced stuff. I'm wondering if there's some kind of framework that we can build on, like XSLT, so we don't have to create all from scratch.

I thought about XSLT, but current format it rather simple and do not rely on preferences.xml format (all modifications are done by Java API calls). If it is really needed, I can add XSLT preprocessing step for <preference> tag body
(<preferences><xslt>...</xslt><fragment>...<fragment> </preferences>).

• It may be useful to delete values by pattern (regex) and delete a nonempty directory.

Thank you, I did not notice that nonempty directories are not deleted. This will be added of course.

Regex is not so hard to implement (if it will continue to use preference parser, not new one...), but may be hard to understand - which maps from map should the following command delete?

   <preferences operation="delete-values" regexp="true">
    <maps key='imagery.entries'>
    <map>
      <tag key='u.*' value='.*openstreetmap.org.*/>
    </map>
    </maps>

• Someone might want to delete files from cache. It should be expected, that cache folder is not a subdirectory of the normal josm pref directory. (E.g. on another disk, shared cache for multiple profiles, etc.).

I agree, this should be possible. But should we allow to delete any folder by absolute path? If not, only 'basedir=prefs / basedir=cache / ... are possible.

• I'd prefer ${x} instead of ${{x}}.

I'd prefer too, but what about possible conflicts with URLs, messages and so on? Though if variable x is not defined, ${{x}} (or ${x}) will not be replaced, so I will switch to ${x}.

• Maybe some kind of summary of the concrete changes in a final confirmation dialog.

It would be good, but I really can not decide what to put there (N keys modified, M keys deleted, P items deleted? Or their list? Or list and values (possibly very long)? Or show scrollable log? Or ask on deletion only?). Some action effect is only known when it is performed (asking user/if).

This needs two pass analyzing reading in some cases but is realizable. We need to decide what we want to ask.

• Change wording in the GUI: It seems as if after loading a xml file all my previous preferences might be dropped.

I agree - message should be changed (do we need to show some summary? plugins were insalled + keys were modified + ... ?).

[bastiK]

Replying to akks:

Replying to bastiK:

This is advanced stuff. I'm wondering if there's some kind of framework that we can build on, like XSLT, so we don't have to create all from scratch.

I thought about XSLT, but current format it rather simple and do not rely on preferences.xml format (all modifications are done by Java API calls). If it is really needed, I can add XSLT preprocessing step for <preference> tag body
(<preferences><xslt>...</xslt><fragment>...<fragment> </preferences>).

• It may be useful to delete values by pattern (regex) and delete a nonempty directory.

Thank you, I did not notice that nonempty directories are not deleted. This will be added of course.

Regex is not so hard to implement (if it will continue to use preference parser, not new one...), but may be hard to understand

like this? <tag key='u.*' key-is-regex='true'/>

• which maps from map should the following command delete?

     <preferences operation="delete-values" regexp="true">
      <maps key='imagery.entries'>
      <map>
        <tag key='u.*' value='.*openstreetmap.org.*/>
      </map>
      </maps>
  

In principle both could be needed: Delete entire map from maps and delete tags from map inside maps.

• Someone might want to delete files from cache. It should be expected, that cache folder is not a subdirectory of the normal josm pref directory. (E.g. on another disk, shared cache for multiple profiles, etc.).

I agree, this should be possible. But should we allow to delete any folder by absolute path? If not, only 'basedir=prefs / basedir=cache / ... are possible.

We should prevent modifications outside the JOSM folders, so the latter makes sense to me.

• I'd prefer ${x} instead of ${{x}}.

I'd prefer too, but what about possible conflicts with URLs, messages and so on? Though if variable x is not defined, ${{x}} (or ${x}) will not be replaced, so I will switch to ${x}.

Then this is a hack and we should use proper escaping instead. (Tricky to implement, though.)

• Maybe some kind of summary of the concrete changes in a final confirmation dialog.

It would be good, but I really can not decide what to put there (N keys modified, M keys deleted, P items deleted? Or their list? Or list and values (possibly very long)? Or show scrollable log? Or ask on deletion only?). Some action effect is only known when it is performed (asking user/if).

This needs two pass analyzing reading in some cases but is realizable. We need to decide what we want to ask.

Personally, I'd feel more comfortable to see a detailed list of all actions, before they are executed.

[akks]

Replying to bastiK:

Personally, I'd feel more comfortable to see a detailed list of all actions, before they are executed.

Without full checking of real values it is easy (delete all keywords matching "....", ).

Showing actual changes before applying them is much harder (map "" will be deleted form "imagery.entries", list "" will be added to list "", ...). Subsequent actions can cause different results, and I do not want to simulate it or implement preferences diff :).

In any case the log can be rather long - how to show it better and what should user do simply apply the commands without scrolling?

[akks]

Replying to bastiK:

like this? <tag key='u.*' key-is-regex='true'/>

This is the worst case - it can not be parsed by Preferences.java :)

In principle both could be needed: Delete entire map from maps and delete tags from map inside maps.

For now all lists from lists and maps from maps are atomic entities. I do not see reasons to introduce huge complexity for cases like renaming unknown imagery layer. (if really needed, simple "replace value" by regexp can be introduced).

[bastiK]

Here is another idea:

Why not have Javascript code in the XML and run it in a sandbox with some API? This would give more power to the user and allow complicated checks. Just hand over all the preferences as a JSON object, let the code modify this object and then apply the changes. For easier copy & paste of preference entries, they could be referenced by XPath expression.

E.g. the following would add scanex imagery, if not already present.

<config>
<script type="text/javascript"><![CDATA[

imagery = API.pref['imagery.entries'];
found = false;
for (i=0; i<imagery.length; i++) {
  if (/scanex/i.test(imagery[i]['name'])) found = true;
}
if (!found) {
  scanex = API.parseMapXPath('/maps[@key='imagery.entries']/map[1]');
  API.pref['imagery.entries'].push(scanex);
}

]]></script>
 <maps key='imagery.entries'>
   <map>
     <tag key='name' value='ScanEx IRS'/>
     <tag key='type' value='scanex'/>
     <tag key='url' value='IRS'/>
      <tag key='attribution-text' value='IRS'/>
      <tag key='attribution-url' value='http://irs.gis-lab.info/'/>
    </map>
 </maps>
</config>

It's a bit crazy, this way we don't need to invent and maintain a huge domain specific language for a very limited purpose.

[akks]

Replying to bastiK:

Here is another idea:

Why not have Javascript code in the XML and run it in a sandbox with some API? This would give more power to the user and allow complicated checks. Just hand over all the preferences as a JSON object, let the code modify this object and then apply the changes. For easier copy & paste of preference entries, they could be referenced by XPath expression.

It's a bit crazy, this way we don't need to invent and maintain a huge domain specific language for a very limited purpose.

If there will be Josm JS API, it should also allow to work with files, plugins, etc. Then people will ask switching modes and drawing...
We have no JS engine in JOSM currently and it will be not me who can add and maintain it :) (there is working Scripting plugin, by the way)

XPath is also not so well-known for writers of configuration files :). I cannot include support for advanced XML processing now, because XML-fragments with preferences are parsed by Preferences class and then all modifications are done for Java objects, not XML. Only some preprocessor can be called (like XSLT for whole preferences.xml or matching tags - say if it is needed), but it can not help in complex cases.

So I tried to make a simple copy-paste-style language with limited number of commands (for partial import/export, not for scripting - I started from shortcut schemas), which I can document and support. "<if>" and var were introduced only to ask "which style dou you prefer?" and similar questions.

I do not see real need for scripting to do simple preference modifications. If some project or plugin need advanced processing (like renaming, numbering, rearranging of layers), why they could not edit preferences manually from their plugin code? :)

[bastiK]

Replying to akks:

Replying to bastiK:

Here is another idea:

Why not have Javascript code in the XML and run it in a sandbox with some API? This would give more power to the user and allow complicated checks. Just hand over all the preferences as a JSON object, let the code modify this object and then apply the changes. For easier copy & paste of preference entries, they could be referenced by XPath expression.

It's a bit crazy, this way we don't need to invent and maintain a huge domain specific language for a very limited purpose.

If there will be Josm JS API, it should also allow to work with files, plugins, etc. Then people will ask switching modes and drawing...
We have no JS engine in JOSM currently

Yes, we have: OpenJDK and Oracle Java ships with a JavaScript engine (Rhino). So executing JS is just 3 lines of code (see ​doc) and should work out of the box for all Java platforms, that are able to run JOSM.

XPath is also not so well-known for writers of configuration files :).

XPath would be optional. Also you should expect a minimum of technical understanding from anyone who messes with preferences of many users. Using an id like in CSS/HTML may be another option.

I cannot include support for advanced XML processing now, because XML-fragments with preferences are parsed by Preferences class and then all modifications are done for Java objects, not XML. Only some preprocessor can be called (like XSLT for whole preferences.xml or matching tags), but it can not help in complex cases.

StAX parsing, it is actually quite elegant, because you can use the parsing code of sub-elements in other parsers. See e.g. osm and osmchange reader. In this case it's a bit messy because of multiple migration steps, but if all fails I would prefer duplication of preference parsing over broken user interface.

So I tried to make a simple copy-paste-style language with limited number of commands (for partial import/export, not for scripting - I started from shortcut schemas), which I can document and support.

I do not see real need for scripting to do simple preference modifications. If some project or plugin need advanced processing (like renaming, numbering, rearranging of layers), why they could not edit preferences manually from their plugin code? :)

I don't consider it simple anymore at this point already, and there will probably be extensions in future. My point is, that the JavaScript solution would be easier for developers and more powerful for users. What I call API is not more than exposing the necessary objects to JavaScript + functions to show a confirmation dialog and to delete files and folders. It is not thought as a full blown JOSM JS API, but just a convenient way to manipulate preferences.

[akks]

I am not going to rewrite the whole thing from scratch, it is not that bad (you can do it if needed) :)
I used xml formats like ant build.xml.
(in fact, preferences/delete, preferences/replace, preference/append and download covers 99% of customizator needs, all other things are optional)

Though we can add scripting for more advanced preferences manipulation, if it is already available in JDK. I imagine how to start JS engine and pass objects to it, but have no idea what "API" it should use with complex functions like "parseMapXPath". I am not so familiar with JavaScript, XPath, etc. , so I can not implement it myself (or even write good scripts for it).

I also can not advice to make all code dependent on preferences.xml format (especially duplicate parsing code, though I like StAX too).

Another problem - if all actions like asking user confirmations, deleting keys, etc. will be based on JS and XPATH only, the code may be shorter and cleaner, but resulting config file will be unreadable and ununderstandable by non-programmers. We need to find good balance between "declarative syntax" (simple, but harder to extend) and "JS API" (universal, but harder to understand). And JS code is hard to mix with declarative XML syntax without even IDs of elements (we do not want another self-made JavaFX 1.0, I hope).

[bastiK]

Replying to akks:

I am not going to rewrite the whole thing from scratch, it is not that bad (you can do it if needed) :)

Ok, let's say it was just a quick idea I found appealing, but your approach is perfectly fine. :)

Another problem - if all actions like asking user confirmations, deleting keys, etc. will be based on JS and XPATH only, the code may be shorter and cleaner, but resulting config file will be unreadable and ununderstandable by non-programmers.
We need to find good balance between "declarative syntax" (simple, but harder to extend) and "JS API" (universal, but harder to understand).

I think it could be combined, if needed.

And JS code is hard to mix with declarative XML syntax without even IDs of elements.

That's what XPath is good for.

[akks]

I have implemented recursive deletion, ${{x}} - ${x} and directory switching (base =prefs/plugins/cache).

So, we need to decide where to put JS ans XPath and if it can replace some of xml-pseudolanguage features.

JS will be placed in <script>, it is OK. We need access to preferences from it (API.pref) and maybe to messageboxes and files/plugins.
This can be done by the same functions (or classes) that process declarative XML, (which I do not want to drop completely :-) ). Any way, this API can be useful for remote control and so on, not only for JS. I agree with you that this should be reusable part of JOSM.

Main problem: we have some copy-pasted preferences fragments in customConfig.xml. Accessing them with XPath is overkill, I think. We can give them ID if needed.

So, for example, we need to find first entry in imagery containing "Mapnik" (or matching some expression) and insert specified entries after it.
I think pure JS is enough to do it, API call is only needed to retrieve fragment for customConfig.xml like

<script> 
...
   API.pref['imagery.entries']
for ....
  idx= ....
...
scanex = API.getFragmentByID(1);
if (API.ask("Do you want to add preferences")) {
   API.pref['imagery.entries'].splice(idx,0,scanex);
...
}
</script>
<fragment id="1"> 
<map>
...
</map>
</fragment>

Only (but not very big) problem is to obtain <preferences id="1"> body as JSON (can you propose good and short DOM/String XML to JSON function to include in JOSM?). And no XPath :) XPath could be useful for searching original (existing) preferences, but it is JSON (?) format. It would be very good if you add some API method to return modifiable part of preferences by XPath (including defaults...). I am not even sure if it is possible... I also do not know, how to implement "API" correctly (see patch).

What do you think about this mixed approach?

[bastiK]

Replying to akks:

I have implemented recursive deletion, ${{x}} - ${x} and directory switching (base =prefs/plugins/cache).

So, we need to decide where to put JS ans XPath and if it can replace some of xml-pseudolanguage features.

JS will be placed in <script>, it is OK. We need access to preferences from it (API.pref) and maybe to messageboxes and files/plugins.
This can be done by the same functions (or classes) that process declarative XML, (which I do not want to drop completely :-) ). Any way, this API can be useful for remote control and so on, not only for JS. I agree with you that this should be reusable part of JOSM.

Main problem: we have some copy-pasted preferences fragments in customConfig.xml. Accessing them with XPath is overkill, I think. We can give them ID if needed.

Doesn't matter, XPath or ID, both should work. Java has good XPath support, and from my point of view it is easy to use and fits the problem.

What do you think about this mixed approach?

Sounds good.

[bastiK]

Java -> JSON

[bastiK]

I've attached a file that demonstrates an alternative way to create the JSON object. After the init script has run, everything should be pure JS. The conversion back to Java is missing, but should work the same.

Unfortunately, the official API seems to be very minimalistic and all the useful Rhino classes are marked "internal". I haven't found an easy way to let the script run in a sandbox, so far. It will probably have access to all the Java classes.

[akks]

Thank you, you sample looks good. Is it OK to use preference maps directly to construct JS objects and call pref.putCollection and so on from JS or is is better way?

And what about XML fragment to Js object conversion? If we read fragment-> maps-> js object, it will be hard to put any fragment to maps, better convert xml->js directly by third party or ours code.

And API will be Java object, not JS? (because it has methods)

[bastiK]

Java -> JS -> Java

[bastiK]

Replying to akks:

Thank you, you sample looks good. Is it OK to use preference maps directly to construct JS objects and call pref.putCollection and so on from JS or is is better way?

I would just convert back to Java (see attachment).

And what about XML fragment to Js object conversion? If we read fragment-> maps-> js object, it will be hard to put any fragment to maps, better convert xml->js directly by third party or ours code.

Sorry, I don't understand, can you explain in more detail?

And API will be Java object, not JS? (because it has methods)

See TestJS2.java, you can also wrap methods, to convert parameters and return values in case they are lists or maps.

[akks]

It seems I understand conversion prinicples now, thank you.

Only question - after we extract all List<Maps<String,String>> from JS object, what should we do with result to change preferences correctly? (Main.pref.putListOfStructs?)

I have an how to read preferences fragment without XML handling: parse it by Preferences class to Preferences object (not single map or list) when JS asks preferecnce fragment, Preferences is converted to JS (using the same method as main preferences). Map or list can be extracted by JS array index:

<script> 
 scanex = API.fragments['fragm1'][0]; 
// fragments are stored before script execution and converted XML->java->JS
</script>
<preferences id="fragm1">  <!-- no "operation" -->
<map>
...
</map>
</fragment>

that would be logical and simple enough to implement and understand.

I'll try to make working prototype based on your sample and listed ideas :)

[simon04]

Maybe this is interesting in the context of user preferences (though, not directly related to this ticket:
From ​http://wiki.openstreetmap.org/wiki/API_v0.6#Preferences

The OSM server supports storing arbitrary user preferences. This can be used by editors, for example, to offer the same configuration wherever the user logs in, instead of a locally-stored configuration.

[akks]

Replying to simon04:

Maybe this is interesting in the context of user preferences (though, not directly related to this ticket:
From ​http://wiki.openstreetmap.org/wiki/API_v0.6#Preferences

Thank you, this may be useful later...

I have just finished bastiK's JavaScript integration. Simple file and dialog API works OK, preferences modification too, but there are some bugs with saving. I will post new patch tomorrow after eliminating them.

[akks]

Replying to simon04:

Maybe this is interesting in the context of user preferences (though, not directly related to this ticket:
From ​http://wiki.openstreetmap.org/wiki/API_v0.6#Preferences

Thank you, this may be useful later...

I have just finished bastiK's JavaScript integration. Simple file and dialog API works OK, preferences modification too, but there are some bugs with saving. I will post new patch tomorrow after eliminating them.

[akks]

Uff, that was not easy :-)

Now

• JavaScript works in <script> element and in embedded ${expression}
• All variables are stored in JS engine, <if> tag was simplified
• API was inroduced, may be extended (see examples)
• Preference fragments from XML (only already parsed) can be referenced as API.fragments[id] from <script> code
• Preferences in Advanced preferences table are colorizedd (like shortcuts) and sorted. User can press "Cancel" and discard preference changes.
• Known default values are given to script, after its work values that are equal to default are skipped.
• Log is shown after importing files. (Messages can be added or changed in code)

Please have a look at it. Custom build: ​http://dl.dropbox.com/u/63393258/josm-custom.jar

[akks]

I noticed JavaScript changed slightly in JDK 1.7, for example JSON object was not available in 1.6. I have commented these lines in scripts.zip sample.

[akks]

minor improvements

[akks]

Asynchronous task for file downloading (and unpacking) was introduced, patch was updated for latest JOSM.

What else do we need to change before committing? (this feature does not change behavior of other components, except advanced preferences colorizing)

[akks]

I think this feature should be committed before stabilization phase starts.

• Do we need restricting javascript - now it can use arbitrary java classes?
• Do we need any confirmations? User can reject prefernces change now, but can not "undo" files downloaded or deleted.

I'll wait for comments before tomorrow.

[akks]

In 5114/josm:

see #4421: Mechanism to modify JOSM settings and store files, advanced preferences dialog modifications
+ JavaScript configuration API
+ Asynchronous file download task DownloadFileTask
+ Function to export arbitrary preference keys to file

[akks]

The only way to get more feedback was to commit :)

Warning: arbitrary Java code execution from imported file (via JavaScript) is possible (like in Scripting plugin?) - do we need sandboxing or customized ClassLoader for JS Engine? Can someone implement it?

We also need some wiki page to describe abilities and configuration file format. Can someone create template in proper place?
Then I can put examples and description there (after file format is stabilized).

Feel free to modify or comment.

[akks]

In 5115/josm:

see #4421: plugin installation fix for JOSM custom configurator

[akks]

So, the mechanism is in the core for 2 weeks but there are still no more comments :)

Do we need any changes before stable release?
What is the best place for wiki page with configuration file format and examples? Could someone create the template?

[akks]

one more example: installing plugins, chaging settings and imagery sources

[simon04]

I guess this is the better ticket to discuss an idea from #7548:

Replying to comment:5:ticket:7548

Concerning plugins, custom styles, custom presets and imagery sources, it would be great to somehow have "bundled versions" which installs the things automatically.

Currently this can be already done by custom configurations, (#4421)
For example, I have settings for drawing forests in Russia: ​http://josm.openstreetmap.de/attachment/ticket/4421/forestMapping.xml

Right, your settings configurator features all this. :-) Do you see a way to automatically apply such a file (upon start of JOSM or better also via webstart)?

[simon04]

Replying to akks:

What is the best place for wiki page with configuration file format and examples? Could someone create the template?

As it is in core, I guess the JOSM wiki is preferred. What about Help/Preferences/ImportExport or similar?

[simon04]

Is it possible to install styles and presets? How is this done?

The background of my questions is to simplify this process ​http://wiki.openseamap.org/wiki/JOSM_and_Plugin (install JOSM, install plugin x, install preset y, install style z, tick this setting) to "Click on this link to launch the JOSM-OpenSeaMap-Editor" (via webstart).

[simon04]

Something that came to my mind: As importing settings may be used to help beginners to setup JOSM for a specific purpose, it is counter-intuitive that the "Read from file" button is only accessible when activating the expert mode. What about placing the button next to next to "OK"/"Cancel"/"Help"?

[akks]

Replying to simon04:

Is it possible to install styles and presets? How is this done?

The background of my questions is to simplify this process ​http://wiki.openseamap.org/wiki/JOSM_and_Plugin (install JOSM, install plugin x, install preset y, install style z, tick this setting) to "Click on this link to launch the JOSM-OpenSeaMap-Editor" (via webstart).

Currently it can be done by downloading file and then modifying preferences (only plugin installation relies on existing JOSM functions). I'll try to make an example for Wiki.

[simon04]

Replying to simon04:

Do you see a way to automatically apply such a file (upon start of JOSM or better also via webstart)?

See attachment:4421_argument.patch​ for an experiment. This allows to use an application argument like --load-preferences=http://josm.openstreetmap.de/raw-attachment/ticket/4421/forestMapping.xml to apply the settings upon start.

[akks]

The patch look nice :)

Currently this is experimental feature and not exposed to novices, but later it can even work through remote control...
Many small buttons like "export/import shortcuts" and so on can be easily introduced if needed.

[akks]

Exporting does not work, sorry :(
I'll fix it.

P.S. It seems I have captured JDK-depedent bug: works on jre 1.7 but not on 1.6.

[akks]

In 5162/josm:

Josm XML-JS configurator, see #4421: bugfixes, JS variables for directories and version

[akks]

Here are examples of preferences and styles manipulation:

[akks]

More complicated script with bundled style and presets files

[simon04]

Replying to akks:

Here are examples of preferences and styles manipulation:

Thank you for this illustrative example. When the command line option is available, it will be really easy to provide a customized JOSM (e.g., "JOSM-OpenSeaMap-Editor"). :-)

[akks]

Replying to simon04:

Replying to akks:

Here are examples of preferences and styles manipulation:

Thank you for this illustrative example. When the command line option is available, it will be really easy to provide a customized JOSM (e.g., "JOSM-OpenSeaMap-Editor"). :-)

But we does not need to change preferences every time JOSM starts :-) Maybe there should be something else for customized JOSM... (however, we can execute sctript every time and have big "if not already applied" in it)

I think you would better check and commit command-line option yourself (do not know - before tested release or after it, it seems to be non-invasive enough).

[simon04]

Replying to akks:

I think you would better check and commit command-line option yourself (do not know - before tested release or after it, it seems to be non-invasive enough).

I guess we should await the upcoming release as there is a new string to be translated …

[simon04]

In 5201/josm:

see #4421 - command line argument --load-preferences= to apply settings on JOSM start

[skyper]

Thanks for your work guys.

It just saved my prefs as I had to remove my preference dir to get the mirror_download working again and importing from file worked perfectly.

[stoecker]

Hmm, I miss documentation of this feature somewhere in the wiki. I wanted to refer to this and found it nowhere. Also commandline option is missing in the Help/CommandLineOptions wiki page.

[skyper]

Replying to stoecker:

Hmm, I miss documentation of this feature somewhere in the wiki. I wanted to refer to this and found it nowhere. Also commandline option is missing in the Help/CommandLineOptions wiki page.

Sadly, I do not have that much time the last months or if I have I am often tired. - There are quite some major enhancement like selecting and zooming within conflict manager which are not yet documented.

[akks]

Replying to skyper:

Replying to stoecker:

Hmm, I miss documentation of this feature somewhere in the wiki. I wanted to refer to this and found it nowhere. Also commandline option is missing in the Help/CommandLineOptions wiki page.

Sadly, I do not have that much time the last months or if I have I am often tired. - There are quite some major enhancement like selecting and zooming within conflict manager which are not yet documented.

Very basic documentation is on page Help/Preferences/Advanced, there is a link to CustomConfiguration on it.
If it is a proper place, I can extend CustomConfiguration.

[stoecker]

I would move it to Help/, otherwise it cannot be part of the online help system.

[akks]

Help/Preferences/CustomConfiguration, Help/CustomConfiguration, Help/Preferences/ImportExport, or somewhere else?

[stoecker]

I renamed it to the last one. When I searched, then I first checked Help/Preferences/.

[akks]

I tried to fill the page Help/Preferences/ImportExport with examples.

Feel free to replace/rework it, especially when more practical applications can be provided :)