Upgrade to Trac 1.6 when released

[stoecker]

Trac 1.4 was released. Upgrade should be done after waiting some time first the bugs to settle after a major version upgrade.

• Change site html to new format ​https://trac.edgewall.org/wiki/TracInterfaceCustomization#SiteAppearance
• switch plugins to 1.4 branches
• iniadminpanel plugin wont work, as the Genshi support seems broken in a major way: ​https://trac.edgewall.org/ticket/13237 - update plugin to get it work, still needs a proper Jinja update for real 1.4 support
• Add a modified CSS to get an acceptable layout back.
• Check if JOSMs simple minded wiki parser survives the layout changes

Plugins:

• AdvancedTicketWorkflowPlugin ok
• BackLinksMacro not used, deleted
• HudsonTrac ok
• NavAdd no longer needed
• TracAccountManager probably ok
• TracAdvParseArgsPlugin not used, deleted
• TracIniAdminPanel ok
• TracSpamFilter ok
• TracStats ok
• Tracticketstats not used, deleted
• TracVote ok
• TracXMLRPC ok
• TranslatedPages ok
• WantedPages not used, deleted

[stoecker]

Ooops, wrong component :-)

[Klumbumbus]

I hope the translation is included properly this time :)

[Don-vip]

1.4 looks very promising:

• Switch to Jinja2 template engine for faster and more memory lenient server-side content generation
• 5 times speed-up when rendering query results, thanks to the migration from Genshi to Jinja2.

Maybe we should nevertheless try this version on another machine, to make sure all bugs impacting us are properly identified and fixed in the next maintenance update. Otherwise I fear some bugs could remain undetected until we switch.

[stoecker]

That "Genshi to Jinja2" was actually the reason why I went away from Trac mailinglists. They wanted to drop Genshi without a time when both are supported and I voiced the opinion that this is the worst sign you can sent to plugin authors. And instead of reacting on what I said they started to blame me for the way I said it (which was relatively harmless, I do worse when really upset :-).

Regarding the speedup: I don't think you will notice it. E.g. everything which is slow in our wiki (like the translation tables) neither uses Genshi nor Jinja and the main speed issue is database access and client side rendering, not the content generating.

[stoecker]

TODO:

• change site.html to new format: ​https://trac.edgewall.org/wiki/TracInterfaceCustomization#SiteAppearance

[stoecker]

Lots of work and little benefit.

Let's hope situation get's better.

[Don-vip]

Why so late? Are you waiting for a new Trac release?

[stoecker]

I either hope that the workload gets reduced or that there comes up a reason why we should upgrade (i.e. a wonderful new feature :-)

[simon04]

May I move this to the longterm goals?

[stoecker]

Ticket #20382 has been marked as a duplicate of this ticket.

[anonymous]

1. Any updates? Is this really that much work? ​https://trac.edgewall.org/wiki/TracUpgrade

2. Well if going Trac 1.4 is so much work can we at lest get Trac 1.2.6 (latest stable)?

[stoecker]

Replying to anonym:

1. Any updates? Is this really that much work? ​https://trac.edgewall.org/wiki/TracUpgrade

A lot. I did this for 2 other (less intensively used) instances and it took some time.

2. Well if going Trac 1.4 is so much work can we at lest get Trac 1.2.6 (latest stable)?

I think so. Have to check.

[stoecker]

There was trouble with trac 1.2.x upgrade as well: See #16651

[stoecker]

Seems trac 1.6 will finally have python 3 support, so that's the next logical step for update. I'd think we skip 1.4 completely.

​https://trac.edgewall.org/wiki/TracDev/ReleaseNotes/1.6

[anonymous]

Quite some time has passed and trac 1.6 is nowhere to be seen. Can we reapproach the question of updating to 1.4.3 or at least 1.2.6?

[gaben]

Python 3.5+ compatibility added with ​Trac 1.5.3.

[stoecker]

Replying to gaben:

Python 3.5+ compatibility added with ​Trac 1.5.3.

I know. There are already people officially using the devel version instead of the release. Probably that's really the way forward.

[anonymous]

So trac 1.6 seems to be ready. We can start the upgrade process or if we don’t want to maintain trac consider moving to GitHub

[stoecker]

Ticket #23224 has been marked as a duplicate of this ticket.

[stoecker]

Replying to anonym:

So trac 1.6 seems to be ready. We can start the upgrade process or if we don’t want to maintain trac consider moving to GitHub

Who is "we" and why should "we" consider moving to GitHub?

If "we" will start the update process, then "we" could start by adapting AdvancedTicketWorkflowPlugin, BackLinksMacro, HudsonTrac, NavAdd, TracAccountManager, TracAdvParseArgsPlugin, TracIniAdminPanel, TracSpamFilter, TracStats, Tracticketstats, TracVote, TracXMLRPC and WantedPages plugins to Trac 1.6. I doubt most of them will work from scratch.

If the plugins work it still will take me several days of work to get a working Trac 1.6 instance including updating the internal plugins, the configuration and other stuff. I'm surely not doing this two weeks after a release which is a bit dubious, as not even the trac wiki reflects that release yet.

[anonymous]

Moving to GitHub would considerably lower the workload needed to maintain Trac. Imagine a scenario where Trac has a vulnerability it is fixed but you haven’t updated trac here. As a result someone can insert unauthorized code into JOSM. That would be bad, and since we all know that trac updating has been neglected well it may not be as far fetched as one may think

[anonymous]

Keeping one’s workspace updated is not something that should be questioned. However we all can see that this has been grossly neglected. As such there are two ways out of this debt: keep trec up to date or outsource this task by moving to GitHub

[stoecker]

Discussing with an "anonym" one makes not much sense, so this will be my last answer to you.

It seems you actually have little knowledge about the software you're talking about because compromising Trac will not give you a chance to insert any code in JOSM. You have to compromise the server behind it to do so.

Also you have simply NO understanding of the costs involved to maintain an infrastructre under maintainer control compared to an infrastructure under control of a company which is known to exploit their users. You have no understanding which services this server provides and what the costs of moving them would be.

JOSM has never neglected to update the infrastructure, especially Trac, but we have REJECTED these updates, because the quality of the updated versions was worse than the quality of the running version. Reason for this is probably the toxic environment caused by the political corrections groups which drove away the majority of real Trac contributors.

And last it seems like these groups you're only talk and do nothing, as I don't see much progress with the plugins mentioned above.

[taylor.smock]

Moving to GitHub would considerably lower the workload needed to maintain Trac

See also #16857. We'd also have to move to git from subversion, which would be a PITA. I'd want to do it right, which would mean:

• Contacting old contributors to see what name and email combination they want to be used for attribution
• Writing a script to parse out patch by <foo> so that we have proper patch authorship information
• Rewriting the version code (git rev-list --count HEAD is off with our git mirror -- 18540 versus 18874, a difference of 334)
• Converting svn:externals to git submodules (git submodule add -b <branch> <url> -- if <branch> is ., it matches the current branch of the current repository; the -b <branch> is important, since otherwise we would have to update the submodule constantly)

All of that takes work. If you (@anonymous) want to help out, you can help update the plugins mentioned in comment:27, update ​tracboat to work with current gitlab versions, or look into other self-hosted alternatives.

Anyway, endpoints that are consumed by JOSM and other applications:

• latest
• tested
• plugin
• styles
• presets
• rules
• maps

There are probably a few more endpoints that I don't know about/don't remember.

Then there are some plugins which parse wiki pages. Some of our integration tests parse wiki pages for example.

[stoecker]

AdvancedTicketWorkflowPlugin

[taylor.smock]

• ​BackLinksMacro -- provides [[BackLinks]] and [[BackLinks(pagename)]] macros. I don't know if we use either. It isn't currently maintained. And seems to be partially broken as is.
  • ​TracBacksPlugin
  • ​TrackBackLinkPlugin
• ​HudsonTrac
  • Only needed if we use the build timeline. Otherwise Trac 1.4 allows for custom navigation items.
• TracAccountManager -- Probably ​AccountManagerPlugin. Probably fine (last change 2023-06-08).
• TracAdvParseArgsPlugin -- Probably ​AdvParseArgsPlugin
• TracSpamFilter -- Probably ​SpamFilter. Probably works (has a 1.5.x version).
• TracStats -- Probably ​TracStatsPlugin. But we don't seem to be using it.
• TracTicketStats -- Probably ​TracTicketStatsPlugin. It isn't currently maintained. I think this is a "nice-to-have", but is definitely not critical.
• TracVote -- Probably ​VotePlugin, appears to be used by trac.edgewall.org, so probably works with 1.6 with no changes.
• TracXMLRPC -- Probably ​XmlRpcPlugin. Probably fine (last change on 2023-03-13).
• WantedPages -- Probably ​WantedPagesMacro, but could be ​WantedPagesPlugin (which redirects to WantedPagesMaco]).

[stoecker]

TracSpamFilter -- Probably ​SpamFilter. Probably works (has a 1.5.x version).

Not fine. Major work. Ask the author ;-)

• Spambayes doesn't work properly for python3 - I'm thinking about integrating the relevant parts into the core
  • If somebody could help me with that it would be appreciated. The plugin only uses a small part of spambayes, so stripping the code to minimum and include it seems better than to wait for a proper spambayes for python3. When keeping the interface using the external lib remains an option.
• Only a single page has been converted to Jinja2. For IniAdmin one page took me about 6 hours, so that's quite a task. Gets probably faster now that I know most details.

TracStats -- Probably ​TracStatsPlugin. But we don't seem to be using it.
TracTicketStats -- Probably ​TracTicketStatsPlugin. It isn't currently maintained. I think this is a "nice-to-have", but is definitely not critical.

Some of the stats plugins results are probably only visible for admins. Have to check.

​BackLinksMacro -- provides BackLinks and BackLinks(pagename) macros. I don't know if we use either.
WantedPages -- Probably ​WantedPagesMacro, but could be ​WantedPagesPlugin (which redirects to WantedPagesMaco]).

Have to check the wiki, when not used I'll drop them.

TracVote -- Probably ​VotePlugin, appears to be used by trac.edgewall.org, so probably works with 1.6 with no changes.

trac.edgewall.org runs with 1.4.3. No 1.6 yet (e.g. because of spamfilter plugin).

[stoecker]

BackLinksMacro

[stoecker]

WantedPages

[stoecker]

TracAdvParseArgsPlugin

[stoecker]

Forgot TranslatedPages

[skyper]

Unfortunately, I do not have time for (J)OSM, atm.

Replying to stoecker:

Replying to taylor.smock:

​BackLinksMacro -- provides BackLinks and BackLinks(pagename) macros. I don't know if we use either.

Have to check the wiki, when not used I'll drop them.

Currently, back links are added manually.
If macros are not currently used on any wiki page does not mean that they are not used at all. I used BackLinks quite a lot to find wiki pages which link to the wiki page I was editing and to make sure that I did not break to many links. If possible, please, keep this option.

[taylor.smock]

@skyper: Would either one of the maintained options work for you? I linked to them in comment:34. Both of them are "automatic" backlinks. ​TracBacksPlugin is more for the ticket section, while ​TracBackLinkPlugin adds them at the bottom of a wiki page (as a collapsible Back Links section).

[stoecker]

One problem is, that this only work in some cases. There are so many linking methods, that the backlinks are more a guess than exact results. Also all the TranslatedPages stuff is a black box to all the macros ;-)

[stoecker]

A small progress report: I'm nearly at the stage to replace one of my own instances of Trac with 1.6. Lots of work to upgrade all the stuff, but it seems to be running now. Tomorrow I'll test a lot and if no major errors appear, I'll start the next one (needing Spamfilter plugin).

P.S. I'm now also maintainer of HudsonPlugin ;-)

[skyper]

Replying to taylor.smock:

@skyper: Would either one of the maintained options work for you? I linked to them in comment:34. Both of them are "automatic" backlinks. ​TracBacksPlugin is more for the ticket section, while ​TracBackLinkPlugin adds them at the bottom of a wiki page (as a collapsible Back Links section).

I am fine with ​TracBackLinkPlugin. In my workflow, I only used the macro in "Preview".

Replying to stoecker:

One problem is, that this only work in some cases. There are so many linking methods, that the backlinks are more a guess than exact results. Also all the TranslatedPages stuff is a black box to all the macros ;-)

That is a pity. Anyway, some backlinks are more than no option at all.

[stoecker]

SpamFilter is finished, TranslatedPages probably also, but yet untested.

[stoecker]

TracVote: Ok

[stoecker]

TranslatedPages ok

[stoecker]

TracXMLRPC ok

[stoecker]

TracStats ok.

I'll aim to do wiki update next week. We'll probably need to adapt josm a bit so the WikiReader can strip new page design.

[taylor.smock]

OK. I'll wait until after the wiki update to do a new stable release so we can ensure that it works (looks like MOTD and help pages are the primary users).

[stoecker]

Report(s) 1, 2, 3, 4, 5, 8 could not be upgraded and may need to be manually
edited to avoid an "ambiguous column name" error. See https://trac.edgewall.org/wiki/1.3/TracUpgrade#enum-description-field for more
information.

[taylor.smock]

Oddly enough, 6 and 14 were broken as well. I think I fixed them (t.description from description).

[gaben]

Where should I report post upgrade findings?

[taylor.smock]

Probably here, with a caveat: I think stoecker pulled the trigger ~30 minutes ago, and he might be in the process of fixing some things.

For example, the time before last when I looked at the main page, the JOSMImage and Version macros were broken.

I think the "big" thing he is probably working on right now is Unsupported version control system "svn": No module named 'svn'. I think that will fix most of the remaining issues.

[stoecker]

Jupp. Updated to Jammy. Was easier (I hope) than fixing the the issue on focal ;-)

The majority is through. You can start reporting stuff, thought keep it simple. Only tell what's broken. Details will only matter when it's not ovious.

[taylor.smock]

• https://josm.openstreetmap.de/maps
• https://josm.openstreetmap.de/plugin
• https://josm.openstreetmap.de/styles (probably same issue as /plugin)
• https://josm.openstreetmap.de/rules (see above)
• https://josm.openstreetmap.de/presets (see above)

[stoecker]

Ah. That will probably be there for some time ;-) Python2 to 3 changed handling of bytes and string and the auto-convert doesn't catch this. I have to find and fix each place manually... Starting now ;-)

[gaben]

OK. I don't see the customized "don't be shy" message on https://josm.openstreetmap.de/newticket page.

[taylor.smock]

Replying to taylor.smock:

OK. I'll wait until after the wiki update to do a new stable release so we can ensure that it works (looks like MOTD and help pages are the primary users).

Good news: it looks like the wiki parsing code still works. At least for the MOTD and Help. :)

[stoecker]

All variants of "/maps" should be working now.

[stoecker]

The API endpoints plugin, styles, presets, rules also fixed.

[stoecker]

Replying to stoecker:

Ah. That will probably be there for some time ;-) Python2 to 3 changed handling of bytes and string and the auto-convert doesn't catch this. I have to find and fix each place manually... Starting now ;-)

Jupp. The template format changed completely. I'll first have to check how to fix it.

[taylor.smock]

It looks like /josmfile is broken: https://josm.openstreetmap.de/josmfile?page=Presets/Maxspeed-zones&zip=1 .

[taylor.smock]

Replying to stoecker:

Jupp. The template format changed completely. I'll first have to check how to fix it.

Assuming you are talking about Genshi to Jinja2, ​Trac has a porting page.

[stoecker]

Replying to taylor.smock:

It looks like /josmfile is broken: https://josm.openstreetmap.de/josmfile?page=Presets/Maxspeed-zones&zip=1 .

Fixed.

Assuming you are talking about Genshi to Jinja2, ​Trac has a porting page.

I did that excessively in the last months. ;-) Still I need to find out how to make the new template for the newticket page. For Genshi it was part of the generic template with an "if" clause.

[taylor.smock]

I assume you saw the ​if section.

I'll play with ./ticket/templates/ticket.html on a local instance to see if I can figure it out.

[stoecker]

I don't get any stacktraces in the log anymore, so I think most major issues are gone. I'll get back to reading my book. Will look for this again this evening.

[stoecker]

Old Genshi template

[stoecker]

Replying to taylor.smock:

I assume you saw the ​if section.

I'll play with ./ticket/templates/ticket.html on a local instance to see if I can figure it out.

I attached the old template.

Currently running is a new template site_head.html

<!-- site_head.html: Add site-specific style sheet -->
<link rel="stylesheet" href="/josm.css" />

[taylor.smock]

Thanks for the old template.

[gaben]

Also, everything looks smaller than before on the site. Do you see any difference after a hard reload?

[stoecker]

Replying to gaben:

Also, everything looks smaller than before on the site. Do you see any difference after a hard reload?

Maybe I overdid the customization. I don't like some default settings of Trac 1.6. Changed 10/12px font now to 12/14px. Better?

[gaben]

Yep, much better, but still feels weird. I still like the old instance looks more. Checking the available Trac demo, unfortunately it shows the same behaviour :/

Some further tweaks might be needed. E.g. if I open this ticket, the ticket number and the title is enormous.
For comparison, the old looks feel more consistent: ​https://web.archive.org/web/20220809041648/https://josm.openstreetmap.de/ticket/18092

[taylor.smock]

Add New Bug readme

[taylor.smock]

Replying to gaben:

Yep, much better, but still feels weird. I still like the old instance looks more. Checking the available Trac demo, unfortunately it shows the same behaviour :/

This might be from comment:75. The text sizes appeared to be the same previously when I was comparing two tabs.

For attachment:18092.patch​, it can be applied from the trac root directory. I did make some modifications for i18n (gettext for text without html -- the html seems to mess things up, or I don't understand something).

[stoecker]

it can be applied from the trac root directory.

Ah no. I can't patch the trac installation ;-) I'm sure there still is a way to override this in the trac data part and not the installation.

I'll have a look soon.

Possibly I'll write a short macro to do so. This could replace the ugly JavaScript based translation also.

[taylor.smock]

My bad. It looks like we could copy the ticket template out to trac data, but that is probably the "wrong" thing to do.

[stoecker]

Replying to gaben:

Yep, much better, but still feels weird. I still like the old instance looks more. Checking the available Trac demo, unfortunately it shows the same behaviour :/

Some further tweaks might be needed. E.g. if I open this ticket, the ticket number and the title is enormous.
For comparison, the old looks feel more consistent: ​https://web.archive.org/web/20220809041648/https://josm.openstreetmap.de/ticket/18092

Adapted font sizes a bit.

[stoecker]

Replying to taylor.smock:

My bad. It looks like we could copy the ticket template out to trac data, but that is probably the "wrong" thing to do.

Probably not wrong. But probably not right either ;-)

I'll find a proper solution, but it's not urgent.

[gaben]

Replying to stoecker:

Adapted font sizes a bit.

Thanks! Getting better :D

Another finding; the vote buttons disappeared.

[stoecker]

Replying to gaben:

Replying to stoecker:

Adapted font sizes a bit.

Thanks! Getting better :D

Another finding; the vote buttons disappeared.

Is back. Somehow 2 plugins were installed, but disabled.

[taylor.smock]

Add New Bug readme via site_footer.html and site_head.html

[stoecker]

Created a JS-Only solution based on you example. Not perfect, but OK.

[gaben]

Replying to stoecker:

Is back. Somehow 2 plugins were installed, but disabled.

While tried to upvote a ticket, I received an HTTP 500 in the console and an HTML page saying NameError: name 'unicode' is not defined.

[stoecker]

Fixed locally. Reported upstream: ​https://trac-hacks.org/ticket/14268

[gaben]

Bug reports from JOSM are broken, see the first report #23371.

[stoecker]

Replying to gaben:

Bug reports from JOSM are broken, see the first report #23371.

That's why I like the python 2 to 3 switch so much. Changes which are only found when you use each and every code path are so much fun. There is nothing better to do than to recheck all lines of code you wrote because someone decided such an incompatibility is a good idea.

Fixed.

[taylor.smock]

I do agree that the python2 to python3 transition sucked.

Unfortunately, python2 is no longer supported. On the plus side (as of python 3.5), there is ​type hinting so it should be easier to find those bugs on a going forward basis. Assuming you use type hinting.

[stoecker]

Another Trac bug found in the logs: ​https://trac.edgewall.org/ticket/13687

[stoecker]

Replying to taylor.smock:

I do agree that the python2 to python3 transition sucked.

Unfortunately, python2 is no longer supported. On the plus side (as of python 3.5), there is ​type hinting so it should be easier to find those bugs on a going forward basis. Assuming you use type hinting.

Ah. No. But sounds useful. Will try to remember it.

Issue here is mainly that python cleaned the str/bytes stuff. Internally you always work with str now, but you output bytes and define the encoding. So any send() now NEEDS bytes, where it before accepted str. That's really good. Python 3 design is much better this way - you get a proper error instead of double encoding and similar errors. But the transition is awful...

[GerdP]

For me the textbox (where I type this text) shows several equal icons with the letter I (kursiv). I guess it should show different icons for the text formatting options?

[GerdP]

Now that I wrote comment:93 I got an email that tels me that Klumbumbus has replied to my comment. But I don't see that comment here?

[Klumbumbus]

Replying to GerdP:

Now that I wrote comment:93 I got an email that tels me that Klumbumbus has replied to my comment. But I don't see that comment here?

I accidentaly wrote a comment. As it was empty I deleted it. All fine.

[GerdP]

I've now pressed Ctrl+F5 to refresh the browser and the icons (and comment by Klumbumbus) are shown.

[Klumbumbus]

At the wiki pages heading generate now a horizontal line. I like that as it helps to understand the structure of the page. However Languages list, Table of content and Images now sometimes go over this line which looks a bit weird. Not sure how we should handle this. Examples:

• wiki:/Help/Dialog/TagsMembership
• wiki:/Help/Dialog/MapPaint
• wiki:/Help/EditToolbar
• wiki:/Help/Dialog/LayerList

[stoecker]

Replying to Klumbumbus:

At the wiki pages heading generate now a horizontal line. I like that as it helps to understand the structure of the page. However Languages list, Table of content and Images now sometimes go over this line which looks a bit weird. Not sure how we should handle this. Examples:

In one of my wikis I solved this by moving the box more to the right, so there is no more line right of the box.

[stoecker]

.wiki-toc, #prefs {
  right: 0px;
  position: absolute;
  margin: 0;
}

[gaben]

You can see this behaviour on ​Trac 1.4 demo as well, so it's not JOSM specific, more like a bug.

[Klumbumbus]

Yes if the line doesn't stick out on the right it would be fine.

[Klumbumbus]

Replying to stoecker:

In one of my wikis I solved this by moving the box more to the right, so there is no more line right of the box.

Could you instead limit the line on the right so that it ends earlier?

[stoecker]

Replying to Klumbumbus:

Replying to stoecker:

In one of my wikis I solved this by moving the box more to the right, so there is no more line right of the box.

Could you instead limit the line on the right so that it ends earlier?

You can try playing with css in the developer console of your browser. I'll accept any sensible changes ;-)

Essentially Trac defines standard css and josm.css overrides some elements.

[Klumbumbus]

trac.css line 618 margin-right: -1rem; changing it to 0rem seems to fix this.

[gaben]

Replying to Klumbumbus:

I accidentaly wrote a comment. As it was empty I deleted it. All fine.

Is empty comments a thing?