develop branch

[Lars Skjærven]

suggesting to add a develop branch to be used in the way we use master today. then let master be updated only for releases - similar to what we discussed some time ago. any thoughts?

[Barry Grant]

This sounds sensible - I drafted a development model figure (attached) in which a new branch would be started for each release. Bug-fixes would then be made to this branch and then merged continuously to master. With all new features starting a new branch from master rather than the release branch. Can't remember what the reason/advantage was at the moment. Either way as long as we keep things consistent we should be fine. Thoughts?

[Xinqiu Yao]

This is very sensible to me to have a release branch. In this figure, MASTER is actually the same to the DEVELOPMENT mentioned by Lars. Right? I remember we have discussed it but finally put it aside. The major concern was: It could be too complicated to have a good control for developers. For example, we have to do bugfix on exactly RELEASE but not MASTER. Otherwise, it may cause a bit trouble, i.e. "conflict" when we merge RELEASE to MASTER.

But I think it is good time to have this new branch now, considering that we are a bit more familiar with Git already and should be fine with the small troubles even if they occur.

[Lars Skjærven]

I find the model you posted some time ago interesting. I guess it's similar to what you have in mind, but I miss the develop branch in your scheme:

When the source code in the develop branch reaches a stable point and is ready to be released, all of the changes should be merged back into master somehow and then tagged with a release number. Therefore, each time when changes are merged back into master, this is a new production release by definition. http://nvie.com/posts/a-successful-git-branching-model/

[Xinqiu Yao]

This strategy is really nice and professional, but to make it work we need to be bound with some roles strictly. For example, we shouldn't do any "developing" commit on master, which I guess is very easily to be broken. Master is the default branch and so some of our Bio3D developers, who may be still not very familiar with git, may mistakenly commit their codes on master, and once it is done I don't know if there is a easy way to reverse on Bitbucket. I think that is why Barry want master be the actual develop and create a release branch for stable production. Am I right?

I think we can first try to push a release branch, which has the HEAD at the current published version (I actually have one on my local repo), and see how it works. What do you think?

[Lars Skjærven]

right - good point. so then master could still be the development-branch as we use it today. a release branch sounds good, but when the release is ready, I think it would be nice to merge the final release onto a branch containing only releases (see the scheme I posted marked as the master branch).

With regard to the release branch, can we now merge the feature_data-frame branch into master, and then make a release branch from there? I've made quite a few updates to the data-frame branch (all NMA vignettes should be working- as well as all the tests (right?)). I understood that Barry wanted the data-frame version pushed out in the next release as well. if you agree you can probably branch them off now- you have more control of the git things than me.

[Xinqiu Yao]

Thanks Lars, I will do that right now. It is definitely good to have a branch with purely production. It is just a question of name. What about "product" or "public"? By the way, I am really not good at naming...

[Xinqiu Yao]

Already done the merge and branching off. I still got test errors:

  Running ‘test-all.R’
 ERROR
Running the tests in ‘tests/test-all.R’ failed.
Last 13 lines of output:
  Mean relative difference: 2

  6. Failure(@test-nma.R#171): NMA -----------------------------------------------
  nowAnm7 not equal to anm7
  Mean relative difference: 2

  7. Failure(@test-nma.pdbs.R#30): eNMA works ------------------------------------
  nowU2 not equal to U2
  Mean relative difference: 2

  Error: Test failures
  In addition: There were 13 warnings (use warnings() to see them)
  Execution halted

Maybe because of my R version?

> sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

Another question: do you know much about feature_visualize_dataframe? I am wondering: Should we merge it for next release, too?

[Lars Skjærven]

thanks for the merge. that's perfect!

names: production sounds good. or how about just releases? also release_history or production_releases.. hmm.

I've updated the test script for the nma function. added a check on the signs of the eigenvectors. apparently there might be changes between platforms (see help(eigen)). can you run the test again?

lets wait for barry with the visualize branch. I have no idea how ready it is.

[Xinqiu Yao]

Thanks! I will try the test and let you know if I found any problem.

Probably a shorter name is better, isn't it? I guess this branch will also be used for downloading. How about we try "production" and, if we want, we can change the name any time.

As for visualize, you are right, Barry knows the progress better and so let's wait a little bit.

[Xinqiu Yao]

I have tested the latest commit and similar errors came up with test-nma.pdbs.R. I simply updated it following your edits in test-nma.R and now seems no problem. Let me know if the changes are not suitable.

[Lars Skjærven]

thanks. lets keep this thread open until barry can comment on the production name and branching discussion.

ps - see also this site: https://www.atlassian.com/git/workflows#!workflow-gitflow

for future, we should probably name the release branches with a version number, e.g. release_2.1-0

[Barry Grant]

I like all the suggestions and discussion here.

If I can try to summarize:

• 1) All minor development will be done on the master branch.

• 2) Major new feature development will be done on separate feature_yourname branches that will be seeded from master and merged to master before deletion. These merges will often concede with new releases.

• 3) When ready, a new release_major.minor.version.number branch will be seeded from master. A major.minor version number tag will also be added to the master branch at this point. Bug fixes for this release should be done on this release branch and then merged to master. These should in principal bump the last patch component of the version number (<major>.<minor>-<patch>)

Point 3 is the only one we do not follow currently. The major advantage (and new practice that we will need to stick to) is that bug fixes will be done primarily on this release branch (perhaps with a patch or minor version number update - see https://bitbucket.org/Grantlab/bio3d/wiki/Versioning%20guidelines ) and then merged back to master. Also, singe new functions (and major new features) will not be added to the release branch. These will be kept on master and feature branches respectively.

The weak point I see is the minor and patch version number updating of the release branch upon bug fixes. Should we automate at lest the patch number updating somehow? Also we need to remember to tag the master branch with major.minor version numbers each release.

Am I missing anything or skipping other important advantages of alternate schemes?

p.s. the feature-visualize-dataframe branch is not yet ready for merging to master, sorry.

[Lars Skjærven]

Sounds good ! We can try this, but I think life would be easier if we have one develop and one master (or production) branch, as discussed above. That way you can tag on the master (or production) without the mess you'll get from the develop branch. With your option you'll keep them intertwined, which will work, but probably will more messy(?). It's also cute to very easily go back on the master (or production) to view the individual releases. For the rest, I agree :)

[Barry Grant]

Are tags not supposed to allow us to step back to these different release points regardless of the new additions on master. Keeping these tags on master also has the nice advantage of allowing any previous version to be downloaded from here on bitbucket under the tags tab.

I am still not sure what separate develop/master branches contribute at this stage but we can certainly have a parent release branch (or probably better called production as you mention) containing just initial releases, which I think is what you are asking for? I could be confused because my master is your develop and production. Ok we probably need a new digram and beer for this.

[Lars Skjærven]

absolutely. I'm referring to this this solution (called "gitflow workflow" in this link, which is the same as the one you pointed out some time ago).

I haven't tried either solution before, but I liked the idea of having two branches to record the history of the project. I think you'll end up with a cleaner history if you don't mix development and and the release history.

[Barry Grant]

Hi folks, I have been thinking more about the proposed branching models and suggest we go for the simpler three branch model (option 1 below) for now and then assess after future releases whether four branch (option 2 below) wold be beneficial.

To recap these two options:

Option 1 entails 3 types of branches master, feature and release. This is very similar to what we do now.

• Basically, master is for minor development and serves as the parent for all branches,
• we have multiple feature branches for major development tasks,
• and also various release branches upon which bugfixes and release oriented tasks should be performed (i.e. no new functions are to be incorporated once a release branch is started). This entails no merging of master and feature into release branches - this is the one big rule for this model.

Option 2, highlighted by Lars, entails 4 types of branches, the 3 above plus a new develop branch. There are some important distinctions here, including:

• The master branch is really a reference with all minor development and other branch seeding/merging happening on develop.
• Both feature and release branches use develop as their parent branch (for both seeding and merging). In other words feature branches should never interact directly with master.
• In contrast, release branches can and SHOULD be merged into develop and master but should not be seeded from master.
• Note that when develop has enough new things incorporated then a release branch should be started so no new features can be added after this point (i.e. release branches are only for release-oriented tasks as described for Option 2.)

Although there are three main extra rules/work practices that will need to be strictly adopted for option 2 to work, it will offer the one big advantage of having a potentially cleaner master reference branch that tracks all current and previous release states.

The question is then, do we need a clean master branch and if so is it worth the extra hassle? One thing that a clean master branch offers is potentially easier tracking of changes between releases (as it is not mixed up with minor development commits). However, I think sticking to our commit message syntax of “NEW: my new feature description” already does this and will hopefully allow us to generate a NEWS file easily. Another potential advantage is being able to step back to previous releases easily. However, using tags properly on the master bracnh of option 1 should also allow for this already.

Given the extra disadvantages of option 2, namely:

• More steps (merges) required in general and hence more potential for mistakes (e.g. extra step of having to merge bugfixes from release into both master and develop).
• Not having minor development on the default master is confusing given current work practices - also potentially bigger barrier for new developers (?)
• Not having master as the parent for feature branches is also potentially confusing and open to mistakes.

We could of course consider a hybrid where we adopt option 1 but also start a release tracking branch (with a similar purpose to the master of option 2). The only problem I see here is that I will likely forget about it. Therefore I think we should begin with Option 1 plus a releases branch and then assess after future releases whether Option 2 wold be more beneficial.

[Lars Skjærven]

Ok. Let's try option 1 for now. We can nonetheless adjust the course if we see that things get messy.

Hmm.. but what do you do in option #1 to deal with development on master while a release branch is open? If I understand it correctly it would go like this:

1. branch off release_2.1 form master
2. bugfixing etc happing on release_2.1
3. Meanwhile, Lars get's a funny idea to implement some strange stuff and add it to master
4. you're now done with bugfixing in release_2.1, and you merge back to master
5. you tag 2.1 at branch master

or do you keep the release_2.1 open, and make a tag at that branch?

[Barry Grant]

The initial brach from master would be the point that tag 2.1 gets created. Every successful bug fix on release_2.1 should in theory bump the last patch number 2.1-1, 2.1-2 etc. and can be merged into the releases branch and master. We don't re-tag master with each patch merge.

Note that we only need the fourth releases branch if we plan to delete individual release_2.X etc. branches. The tags on master are for major.minor versions not patches. Optional tags on release branches indicate patch numbers.

So in your fig the green tag is the first node, where you have the tag is actually 2.1-2 tagged on the release branch. Problems?

[Lars Skjærven]

aha! so release it's not a dedicated branch to prepare the release. that's where I got lost. In my mind the release branch was for preparing the release:

"Once develop has acquired enough features for a release, you fork a release branch off of develop. Once it's ready to ship, the release gets merged into master and tagged with a version number. "

Anyway – thanks for the clarification. I think this is getting somewhere. but we definitively need the diagram. I tried with the beer, but that didn't help

[Xinqiu Yao]

Thanks for the discussion. Once we decided, we can try with current release and see how it works. Btw, have you guys checked the script I wrote a couple weeks ago for bumper version automatically? It is under util/bumpversion.sh. Let me know if it is useful.

[Barry Grant]

It is confusing but worth drinking over - I mean thinking over...

Attached is a quick sketch of the three models.

The third is the simplest (and my current favorite) with one permanent releases branch, which gets seeded from master (where continual minor development happens).

This new releases branch is where bug-fixes and other release only related commits go. Its last point represents our last stable bug-fixed release with all patch number glory included. Tags on master only represent the first release point e.g. 2.1-0 and subsequent points/commits may include new functions not in release.

The difference with option 1 above is that we don't use new release specific branches each time but just do all these things on the one permanent releases branch that I was proposing we use for clean tracking anyway.

Can clean digram up if this all makes any sense to others.

[Lars Skjærven]

got it. I like the idea that the releases branch holds the last bug-fixed release. but that assumes that bugs can be resolved with one commit. can hold for relatively minor patches.. hmm.... ok. I will shut up now

Xinqiu: very good. I recently saw this site containing a more advanced script for doing all the git stuff. we can probably hold off with that though

[Xinqiu Yao]

Thanks, Lars. The gitflow seems interesting: I need time to digest it but it is good to know there are developed ways to do the same task.

About the branch model, I guess we can still have temporary bugfix branch for extended bugfixing period, which should be branched out from release and finally merged into both release (with bumped patch number) and master.

[Barry Grant]

Have you guys seen or tried this: https://github.com/petervanderdoes/gitflow

Although we have now built up some inertia and some of us don't like the git-flow naming conventions, it now looks like we can use whatever names we want and it won't impact our ability to use these nice tools.

When you git flow init you will be prompted to enter the names you want to use for each branch type, with some suggestions, which is then stored in a config file and used by all other git-flow commands.

For example, we can continue using master as our development branch and releases as our production branch, with no confusion and no tool headaches.

Obviously the master branch is the one we use daily, and should be kept for regular (non feature) development tasks. What I am suggesting here is that we can now treat git-flow as a workflow, rather than specific/mandated branch guidelines.

I think this might be worth exploring further.

[Xinqiu Yao]

That's really a cool staff, and I guess Lars mentioned it before but I haven't explored deeply. I like the way they make branches and finish them: it is how we did but with much fewer lines of commands to input. I am curious how the "version bump" works in the releases. Will definitely have a play with it!

[Lars Skjærven]

This could be nice to try out. Perhaps for next release?