transfer ET wiki from mediawiki to bitbucket wiki (markdown)

[Roland Haas]

Maintaining the mediawiki server has become burdensome and we are currently stuck with a version of MediaWiki that is hard to update or maintain.

Instead we should transition the wiki content to markdown and use bitbucke’ts wiki facilities. A possible tool to enable this is https://github.com/peterjc/mediawiki_to_git_md which requires an XML dumb of MediaWiki’s database content.

[Roland Haas]

@stevenrbrandt would you mind providing me with such a XML dumb. The required commands are shown on https://github.com/peterjc/mediawiki_to_git_md#user-content-mediawiki-export-to-xml

One could also use pandoc for the mediawiki to markdown conversion though one likely has to handle attachments etc. manually in that case.

[Roland Haas]

Maintaining the mediawiki server has become burdensome and we are currently stuck with a version of MediaWiki that is hard to update or maintain.

Instead we should transition the wiki content to markdown and use bitbucke’ts wiki facilities. A possible tool to enable this is https://github.com/peterjc/mediawiki_to_git_md which requires an XML dumb of MediaWiki’s database content.

[Ian Hinder]

Does anyone have any experience with using a Bitbucket wiki for such a large project? When I used the Bitbucket wiki a few years ago, I found the whole experience very painful. Unless someone has more recent and better experience, I would maybe try this out for a less visible project than the ET. Roland: I know NCSA uses Confluence, but I don’t know if the wikis are the same.

With Mediawiki, we benefit from the fact that there is a huge ecosystem of plugins and extensions and a long history of people adding desired features. It’s also open source. The Bitbucket wiki is, as far as I know, closed source, and almost certainly not as extensible.

Thoughts?

[Roland Haas]

The confluence wiki is very different from bitbucket’s wiki. Confluence uses something else (some XML variant) to internally store its data and has lots of plugins for different things. It is also WYSIWG only.

My main drive to use Bitbucket (and not something else) is that we are already using Bitbucket for the repos so it will not be yet another account.

I have not tried this conversion at all before (other than running a single mediawiki page through pandoc) so it may well not work. I would be very happy if someone who has experienced this (you apparently) could provide some report on the problems encountered.

While in principle we would benefit from the large ecosystem of mediawiki, in practise we do not since the mediawiki instance ins hampered by being quite old and not being easily updatable (has to do with how user accounts are managed apparently). Eg the currently running instance is 1.31.0 which is from 2018-04 .

This is the current list of extensions in our mediawiki: https://docs.einsteintoolkit.org/et-docs/Special:Version#mw-version-ext

We are also experiencing spam issues on the wiki https://docs.einsteintoolkit.org/et-docs/Special:ActiveUsers with pages being created to advertise “stuff”. This is in spite of using Google’s recaptcha test.

The Bitbucket wiki is not open source yet its storage format is just a bunch of markdown files (and the wiki itself is is a git repo) so one can easily grab it and convert to anything else.

So, I am definitely not wedded to Bitbucket but am also trying to divorce myself from having to run and maintain a mediawiki installation.

[Ian Hinder]

Yes, I understand the motivation and agree with all you say. I’m just sounding a note of caution that I found the Bitbucket wiki frustrating to use (many years ago). We probably would be willing to accept a loss of functionality to gain the benefits you cite, but not if it’s really painful to use.

I only used it for a small project. I seem to remember problems such as filenames which couldn’t be entered, requiring me to manually edit the wiki repo (the web editor was doing some sort of nonintuitive transformation on the filenames). I was using Creole as the markup language, not markdown. According to https://confluence.atlassian.com/bitbucket/wikis-221449748.html, Creole has a privileged status, in that “macros” only work on pages that are written in Creole. So if you want to use macros, you have to use Creole.

Bitbucket Cloud wikis support Creole, Markdown, reStructuredText, and Textile syntax. There is special Bitbucket markup that you can use to link to Bitbucket objects from wikis.

In some cases, you can use a Bitbucket macro or the Bitbucket markup to do the same thing. The Bitbucket markup works in all languages; The Bitbucket macros work only with Creole markup.

I found it hard (I think impossible) to generate a site map to get an idea of what pages were present in the wiki. In fact, after creating a page, and navigating away from it, I remember losing that page and having to clone the wiki repository to find it again! (I suppose I could have looked in the browser history, but still…). If Bitbucket wikis have received attention from Atlassian in the last few years, then maybe these usability issues have been resolved. My experience has been, though, that the Bitbucket features I find missing have been requested ages ago, and have long threads of people requesting them and “plus-one”ing the request, and tend to end up with people complaining about the fact that even though the feature is obvious and highly requested, it hasn’t been implemented in many years, and probably Atlassian are trying to push people towards Confluence instead of improving the free tools.

Anyway, I don’t have any recent experience, so it might be better now. I just wouldn’t assume that it will work smoothly without trying it out a bit. Unfortunately, once someone has invested effort into the conversion to a new system, there is a lot of inertia to adopt the new system, even if a proper evaluation hasn’t been made, or has found problems.

[Roland Haas]

We are not really bound to Bitbucket and could also use GitHub, where one can eg resize images by using regular <img> tags in markdown (bitbucket does not let one do that). See https://github.com/rhaas80/wikitest/wiki

[Roland Haas]

Both GitHub and Bitbucket have options to let the wiki be editable by anyone with an account which may increase contributions by non-maintainers who typically do not have accounts on the ET mediawiki wiki and for whom getting an account may be a barrier (it requires email to [e-mail] and that one waits until @Steven R. Brandt or @Roland Haas create the account for them).