move AEIThorns into git repositories

[Roland Haas]

Currently the ET contains the following AEI thorns:

• AEIThorns/ADMMass
• AEIThorns/AEILocalInterp
• AEIThorns/PunctureTracker
• AEIThorns/SystemStatistics
• AEIThorns/Trigger

since the AEI would like to shut down the svn server (or at least change it to read only mode) it would be good to move these thorns into:

EinsteinAnalysis:: ADMMass PunctureTracker CactusUtils:: SystemStatistics Trigger (need change of license to LGPL) Numerical:: AEILocalInterp (license must stay GPL)

For the ET thorns Barry kindly did the conversion using a set of scripts of his. If you were to make the scripts accessible (or give the whole thing a try yourself), then this could be started as soon as the authors (Ian and Frank are main authors if SystemStatistics and Trigger, Erik, Barry and I contributed, I am fine with a license change) agree to the license change where required.

Keyword: ADMMass
Keyword: AEILocalInterp
Keyword: PunctureTracker
Keyword: SystemStatistics
Keyword: Trigger

[Barry Wardell]

I will reuse the script I had for LSUThorns to merge these AEI thorns into their respective arrangements.

In terms of licenses, I'm fine with any code I have contributed being switched to LGPL.

[Frank Löffler]

Since this involved git merges, it would really be great if this would also work across branches (so that a switch to an older ET release also contains the relevant thorns, as in their svn release branches).

[Barry Wardell]

The branches of the merged thorns are preserved by the script, inside a thorn-specific namespace. For example, there would be a ADMMass/ET_2014_11 branch, and similarly for all other branches and tags.

Going beyond this to implement your suggestion is a little more tricky. The problem is that these thorns didn't exist inside those arrangements at the time that the previous branches were created. We could artificially merge the branches to make it appear as if they had been in that arrangement all along, but I don't know whether that is a good idea or not. This step could also be deferred to some point in the future, after the initial merge has been completed.

[Frank Löffler]

I am probably a little naive, but what's the difference between the 'master' and any other branch? Would we need to merge the master branches as well, and could similarly merge other branches? Surely it would change the state of that branch, but only by adding a thorn (which we would be fine with).

[Barry Wardell]

The difference is that with the master branch the point at which the merge happens really does correspond to the point in time when the thorn was moved into the arrangement, whereas on the release branches the thorn existed in some other arrangement at the time of the release.

There's no technical impediment to merging the thorn release branches into the arrangement release branch, it's more of a conceptual thing. For example, it could lead to a situation where the same thorn appears twice: once in the old arrangement, once in the new arrangement.

[Roland Haas]

I believe what Frank is suggesting is to make a combined ET_2015_05 (for example) branch by merging all the THORN/ET_2015_05 branches in a commit. Eg.

git checkout -b ET_2015_05 PunctureTracker/2015_05
git merge ADMMass/2015_05 Trigger/2015_05

which actually requires that all those branches share a single ancestor.

Basically we need to be able to have something like this

!TARGET   = $ARR
!TYPE     = git
!URL      = https://bitbucket.org/einsteintoolkit/aeithorns.git
!REPO_PATH= $2
!REPO_BRANCH = ET_2015_05
!CHECKOUT =
AEIThorns/PunctureTracker
AEIThorns/Trigger

in a thorn list (ie. a single branch that contains the files from all thorns).

[Barry Wardell]

Yes, as I said, actually doing the merge is straightforward. It's not even necessary that they share a single ancestor.

The problem is you would then have a thornlist which is not consistent with the repository layout. For example, you would have two directories: arrangements/AEIThorns/PunctureTracker and arrangements/EinsteinAnalysis/PunctureTracker. We could work around this by updating the thornlists in all of the release branches.

Replying to [comment:6 rhaas]:

Basically we need to be able to have something like this

!TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/einsteintoolkit/aeithorns.git !REPO_PATH= $2 !REPO_BRANCH = ET_2015_05 !CHECKOUT = AEIThorns/PunctureTracker AEIThorns/Trigger

in a thorn list (ie. a single branch that contains the files from all thorns).

Are you suggesting that we have another arrangement repository for AEIThorns (in addition to moving the thorns to other arrangements)?

[Roland Haas]

Ah, I was not quite thinking straight. I had written my comment as if we were actually transitioning the AEIThorns arrangement but instead we are spreading the individual thorns around into other repositories.

In this case I think that the "new" thorns from AEIThorns only need to appear in their target repositories in branches to correspond to releases starting from the upcoming one. SIt will not be possible anyway to update a thornlist by just bumping its ET_BRANCH define at the beginning since that updated thornlist would try to check out a branch of the svn AEIThorns that does not exist.

One could consider having the new AEIThorns thorns appear in branches of the target repo that correspond to old releases. If they did then there would indeed be the situation that an old thornlist would ask for (eg.) ET_2015_05 in both the svn AEIThorn and EinsteinAnalysis and would show for example two ADMMass thorns (though only one would appear in a CHECKOUT statement and be used by Cactus). A "new" thornlist could then be used to transparently go back to an older release, albeit with ADMMass staying in EinsteinAnalysis and not showing up in AEIThorns. This can be useful for regression analysis and we did have user reports about thorns moving between arrangements causing issues. It would be rather coarse grained regression stepping though since one can only step in increments of a release, so the usefulness is not so great.

Even with changing existing branches though, an old thornlist will not be updatable by just changing the ET_BRANCH define, rather a new thornlist would be downgradable in this manner.

[Frank Löffler]

Replying to [comment:8 rhaas]:

One could consider having the new AEIThorns thorns appear in branches of the target repo that correspond to old releases. If they did then there would indeed be the situation that an old thornlist would ask for (eg.) ET_2015_05 in both the svn AEIThorn and EinsteinAnalysis and would show for example two ADMMass thorns (though only one would appear in a CHECKOUT statement and be used by Cactus).

Assuming the svn version would to be used (otherwise, why get it), then the 'other' version of the thorn would be only somewhere inside 'repos', which we don't expect users to traverse anyway. 'repos' is just an unfortunate workaround due to the inability to do partial checkouts.

Even with changing existing branches though, an old thornlist will not be updatable by just changing the ET_BRANCH define, rather a new thornlist would be downgradable in this manner.

Yes. What I was thinking of was the possibility of the svn server going away completely. If that would happen, we would need to change our old release thornlists to point to the new thorn-wise branches in git, instead of a cleaner repository-wide release branch. Of course, then we could do the actual merge as well.

[Barry Wardell]

Replying to [comment:9 knarf]:

Assuming the svn version would to be used (otherwise, why get it), then the 'other' version of the thorn would be only somewhere inside 'repos', which we don't expect users to traverse anyway. 'repos' is just an unfortunate workaround due to the inability to do partial checkouts.

Good point, I guess it wouldn't really be all that much harm since they won't be in arrangements.

Yes. What I was thinking of was the possibility of the svn server going away completely. If that would happen, we would need to change our old release thornlists to point to the new thorn-wise branches in git, instead of a cleaner repository-wide release branch. Of course, then we could do the actual merge as well.

If we are thinking about the svn server going away, then I agree a repository-wide release branch would be more convenient. Either way, the release thornlists will need to be updated, but I guess that isn't really much of a problem.

[Barry Wardell]

I have updated the merged repositories with the merged release branches. They can be found at:

https://bitbucket.org/barrywardell/carpet https://bitbucket.org/barrywardell/cactusutils https://bitbucket.org/barrywardell/cactusnumerical https://bitbucket.org/barrywardell/einsteinanalysis https://bitbucket.org/barrywardell/numerical

OK to push these to the official repositories?

[Ian Hinder]

I have read through the discussion twice (sorry for not paying attention earlier; I was busy with other things). I think I have got the gist of the arguments presented, but I may still be misunderstanding. Sorry! I'm not sure that merging the release branches is the right approach. It may be better to have a separate AEIThorns git repository (or one per thorn) to handle the loss of svn.aei.mpg.de, and separate this issue from the moving of the thorns. I need to think about this a bit more.

[Erik Schnetter]

A single git repository that keeps a faithful record of the history of the former svn repository seems like a good idea, at least for archaeological reasons. This will allow us also to change our mind later.

[Roland Haas]

Erik: are you proposing that we keep the AEIThorns around twice? Once in a repo that serves only to record old history and once in their new location?

Since in svn it looks like one-thorn-per-repo anyway, I think that the history is already captured in the git repos, ie "git log ADMMass" inside of EinsteinAnalysis shows all history. We may want to revive an AEIThorns repo in git for new (or less known public) AEIThorns that have not yet become very widely used and it would be fine to start off that repo as if the former public AEIThorns had lived in there I think. The first new commit would likely be "remove XXX since it has moved to YYY" though.

So far, as far as we can control, the AEI git repo may become read-only, but we are still lobbying to keep the up and running if for no other reason than that the URLs were mentionend in published papers.

[Ian Hinder]

I think you meant "the AEI SVN repo" in your last paragraph.

Barry and I discussed this at length in a call yesterday. We think it is best to separate the two issues: the SVN repository going away, and the thorns moving to new arrangements. For various reasons, we decided to: * Create git copies of all the AEIThorns repositories (they are 1 repository per thorn) which can then be pointed to by the thornlist in the release branches * Merge the current state of those repositories into the relevant ET repositories master branches. (Technically this will be done in reverse, but that's not important for this discussion.) This leads to some duplication, but overall we decided that this was the best strategy, to minimise confusion and make possible everything that is needed.

Barry will implement this, and assuming no major objections, will commit the changes.

[Frank Löffler]

No objections from me. In the end, these are all just little details, and as long as history is preserved in an accessible place, I am for moving forward. Something else I learned in this discussion is that apparently "the AEI svn server going away" is not just a mere possible scenario, but more imminent; good to know.

[Barry Wardell]

As Ian says, we discussed this on Skype yesterday and worked out a solution which should hopefully satisfy all of our desires/requirements. In short, the thorns have been moved and everything is now working the way we want it.

For those who are interested, a more detailed explanation follows. What was done: 1. Merge the master branches of the to-be-moved thorns into the master branches of the arrangement repository they are moving to. 2. Push other branches and tags from the moved thorns' repository into the arrangement repository under their own namespace (so we have, e.g., PunctureTracker/ET_2015_05) 3. Do not merge the release branches of the moved thorns into the arrangement release branches. This would have too many weird side-effects (multiple copies of a thorn in the Cactus tree, not reflecting any true history, having the merged thorns appear on the tips of branches but not being present in any intervening commits between branches, etc.).

This is all we need going forward and will work for all future releases. I have gone ahead and pushed these changes to the official repositories.

This strategy also preserves the full history of the thorns in a faithful way and makes it easy to go back to older versions. For example, if one wants to go back to a previous version of PunctureTracker while keeping it in the new EinsteinAnalysis arrangement, they can just checkout that version using, e.g.,

git checkout origin/PunctureTracker/ET_2011_05 PunctureTracker

This will move the PunctureTracker directory back to the ET_2011_05 release version while keeping other thorns at the current release.

Alternatively, if one wants to move the entire arrangement back to the ET_2011_05 release, they can use

git checkout ET_2011_05

as normal. This will then not have PunctureTracker present, as expected since it wasn't in the EinsteinAnalysis arrangement at the time of the ET_2011_05 release. However, if one really does want to have the ET_2011_05 version of PunctureTracker, then that's easily achieved too:

git checkout origin/PunctureTracker/ET_2011_05 PunctureTracker

I think making this step explicit is better (less confusing) than merging the PunctureTracker/ET_2011_05 branch into the ET_2011_05 branch. It also means that old release thornlists should still work fine without any modification.

A separate issue is that the AEI are keen to retire their svn server. It's running an old version of the OS, is becoming a chore to maintain, and isn't being used by anyone any more. Once it goes offline, old release thornlists will no longer work. To work around this, we can make git versions of the relevant thorn repositories available on BitBucket. In principle, this could be achieved using the existing merged arrangement repositories and just checking out, e.g., the PunctureTracker/ET_2011_05 branch in the EinsteinAnalysis repository, which will only give the PunctureTracker files and not the other files in the arrangement. The only catch is that this would make checkouts larger, since each of these thorns will now have a repository which contains the history of the whole arrangement. To cut down on repository size/download time, the best solution is to make the relevant release branches available in separate read-only repositories, one for each thorn. I have done this for the both AEIThorns and LSUThorns repositories, so we now have, e.g., a [https://bitbucket.org/einsteintoolkit/archivedthorns-puncturetracker PunctureTracker repository], which only contains the necessary release branches. I have also updated all of the old release branch thornlists in the Einstein Toolkit manifest repository, so that new downloads of the ET will use the git versions of the repositories from BitBucket in place of the old svn repositories. I have tested that I can use GetComponents with these updated release thornlists to check out the old releases.

[Barry Wardell]

This has now been done, as described in detail in my previous comment.