Setup Makefiles and utils sub-modules

[David Dickinson]

Currently users have to clone the utils and Makefiles directory in addition to the main code they want to use. We should be able to set these up as git sub-modules which should mean these will be automatically cloned with the parent code.

[Stephen Biggs-Fox]

Agreed. This is the proper thing to do.

[Stephen Biggs-Fox]

I'm trying to build GS2 on YARCC right now but there's no utils or Makefiles. Since this is annoying me right now, I may as well fix it as I'm going along.

[David Dickinson]

Sorry I should have assigned this to myself as I've already got some stuff for this. If you've already done it then fine, but if not don't worry as I can push something soon.

[David Dickinson]

(also I note you changed the status to open -- what was the status before? I thought it was already open).

[Stephen Biggs-Fox]

No worries. I hadn't got very far. And yes, please do push ASAP as I need the sub-modules.

Status was "new".

How have you dealt with the protocol? Because if it's https protocol then developers need to override this with a config option or they won't be able to push changes to the sub-modules but if it's git protocol then non-developers won't be able to download the sub-modules. I think the former is probably the lesser of two evils.

[David Dickinson]

See the branch feature_submodules_for_externals

At least in my experience you shouldn't make changes to the sub-modules directly and then push them, it works much better if you just checkout the repository directly and make your changes there, commit and push. Can then pull changes into sub-module repo -- this then provides a change in the commit to be used which is something that will be committed to the main repo as an "updating version of <sub-module> to be used.

I think anonymous users should be able to clone utils/Makefiles anyway just not commit.

[David Dickinson]

We should perhaps link to https://stackoverflow.com/a/4438292 to describe how to automatically fetch the sub-modules if possible

[Stephen Biggs-Fox]

Add utils and Makefiles as submodules

This fixes issue #7

→ <<cset f0ebd67405d1>>

[David Dickinson]

It looks like you've fixed this as well -- did the above branch not work?

[Stephen Biggs-Fox]

No, sorry, it did not. I'm just testing mine now. Watch this space for a follow-up comment...

[David Dickinson]

@steve_biggs ok thanks, what happens when you try to use it?

[Stephen Biggs-Fox]

I agree that we should provide instructions on how to get the sub-modules. This probably belongs in the README (along with my git config trick - see below).

I've tried the branch feature_submodules_for_externals. It has the problem I was referring to earlier. To reproduce:

• Log on to a machine for which you have not yet uploaded the ssh key to your bitbucket account to simulate you being a user but not a developer
• Clone the repository as follows: git clone -b feature_submodules_for_externals --recurse-submodules https://bitbucket.org/gyrokinetics/gs2.git
• The clones of the sub-modules fail with the error mesasge: "Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists."

The reason for this can be seen as follows:

• cd into the directory you just checked out (or another working copy elsewhere with that branch checked out)
• Run cat .gitmodules
• The urls are listed as: "[e-mail]:gyrokinetics/xxx.git" where xxx is utils or Makefiles

As a result, git tries to checkout the sub-modules using the git protocol which is for developers to be able to push changes back, hence regular non-dev GS2 users do not have access to the repository via this protocol as they are not on the project team within bitbucket. This instead needs to use the https protocol, which is a read-only protocol for anyone to use.

Fixing this is probably achieved more easily in a new branch rather than trying to fix the one above. Therefore, I have done so in a branch called "bugfix/iss-7".

Testing this results in the following behaviour:

(1) Testing as above (i.e. user, not developer) works fine - gs2, utils and Makefiles all checked out successfully via https, and directories populated

(2) Cloning from a developer machine (i.e. where the ssh key has been added to my bitbucket account) works fine - gs2 checked out via git protocol so changes can be pushed back, utils and Makefiles checked out and directories populated but via https. Therefore, a separate git protocol checkout of the sub-projects would be necessary to contribute (push changes). This will be the default behaviour for all developers.

(3) If you set git config --global [e-mail]:gyrokinetics/.insteadOf https://bitbucket.org/gyrokinetics/ and then repeat the clone from a developer machine, this also works fine - gs2, utils and Makefiles all checked out successfully via git protocol, and directories populated. Changes could be pushed to utils or Makefiles projects directly from subdirectories. Separate commits would have to be pushed to the gs2 project to update which commit the sub-modules should be on. I have worked in this fashion on other projects before and it works better than (2) as you have only one copy of gs2 and its sub-projects all in one place with fewer steps required to modify sub-projects and incorporate those changes into the super-project.

NB: For Dave (and anyone else who has checked out feature_submodules_for_externals), you may encounter problems because utils and Makefiles already exist as subdirectories within the working copy and within .git/modules. To fix this, delete the branch feature_submodules_for_externals, delete the subdirectories utils and Makefiles from the working copy and from .git/modules, then check out the branch bugfix/iss-7, and finally run git submodule update --init --recursive.

[David Dickinson]

Thanks, I've fixed the original branch as this already includes some notes in README.

I think having the sub-modules as non-pushable is actually a plus point. I can see the advantages for allowing changes directly in the sub-module directory but keeping them separate helps make things a bit more explicit -- it also forces people to think about the fact that those changes don't just impact gs2, but other codes as well. I guess nothing stops people doing it differently on their own machine.

[Stephen Biggs-Fox]

Add detail to README about gs2 submodules

This addresses issue #7

→ <<cset c95e124b393f>>

[David Dickinson]

I think the PR #6 addresses this, however there are a couple of features that would be nice to include I think.

1. It would perhaps be nice to add a makefile target to update/fetch the submodules so we could do something like make get-submodules (or some other appropriate target name).
2. It would be nice if the sub-modules target where a part of the makefile dependency tree -- i.e. doing something like make depend would first ensure that the sub-modules have been fetched (but not updated to arbitrary newest version) before identifying the source code dependency. As we're also thinking about removing the dependency file, such that any initial attempt to build the code will require the dependency file to be generated, then I think this would be a nice way to ensure users will automatically get the submodules at the point that they first need them (i.e. when trying to build the code).

[Peter Hill]

Fixed in PR #6