There is currently 3 ideas I am having for future versions:
- overlays (patches that go on and come off transparently as you interact with the repository)
- mq-like implementation (when complete this will be a full superset of mq)
- rebasing patches (patches will be rebased automatically and transparently when possible to avoid rejects) - currently being done in trunk
Plans to make attic a superset of mq
The use cases for the existing patch handling extensions could be reduced to two different feature sets / models. One that allows the '''handling and management of patches''' and its conversion into and from commits, and another one that implements a way to '''collaboratively develop patches'''.
The first feature set would merge the capabilities of mq, attic and shelve, while the second one would correspond to pbranch. Naturally, the two extensions implementing these feature sets should be highly integrated with each other.
This proposal tackles the first part, as of right now attic is already a superset of shelve and I (BillBarry) have plans to extend the functionality of attic to become a superset of mq.
General interaction with patches
Interaction with patches can be viewed from different angles:
- from how they interact with the repo history and the working copy contents: they can be '''working copy changes''' or '''mutable changesets'''.
- As '''working copy changes''', patches don't affect the repository history and are just a way to store changes using a name so they can be applied or unapplied later. They're used like that in the existing shelve and attic extensions.
- As '''mutable changesets''', the existing changes are solidified as changesets in the repository history, so they can be rebased, merged, pushed or pulled... This is how Mercurial Queues view patches.
- from how they are created: they can '''consolidate existing changes''' in the working copy or used to '''stash away changes''' those changes.
- Using patches to consolidate changes in the working copy is useful when trying to evolve those changes or when they correspond to a changing base content which needs some adaptation in the form of additional patches, and giving these changes a name is very convenient. This is how Mercurial Queues generate its patches, taking them from existing content to later manage in a convenient way.
- Stashing away changes allows a more convenient workflow, avoiding having to mess with unrelated changes, so they can be temporarily put appart to concentrate on a different aspect and later can be reapplied. The existing shelve extension uses patches like this.
- patches can be a '''single set of changes''' or '''stacked''', either in series, with a given order, or as unrelated changes.
These series allow reordering of patches or partial application, allowing guards.
Once patches are created, there's also the need to refresh its changes, apply or unapply them to the working copy or repository history, and delete them.
Conceptual user interface
- Create named patch and keep related changes to the working copy
- Create named patch and stash away related changes to the working copy
- Apply changes from named patch (or series of patches) to the working copy
- Unapply changes from named patch (or series of patches) to the working copy
- Apply changes from named patch (or series of patches) to the repo history
- Unapply changes from named patch (or series of patches) from the repo history
- Set guards for a patch
- Refresh changes belonging to the named patch
- Transfer content between patches (rename, fold, split, join, rearrange...)
- Delete this patch name or named series of patches
- Status information (changes, guards,...)
Simplified proposed user interface for attic
(none of these commands necessarily have their final names yet)
- Patch creation
store- Create named patch and keep related changes in the working copy:
store --rev imports existing repo changesets (multi-rev-spec, rebase around merges?) --import imports a patch from elsewhere
shelve- Create named patch and stash away related changes to the working copy
shelve --rev imports an existing revision --import imports from elsewhere
- Patch editing
unshelve- Apply changes from named patch to the working copy
shelve- Unapply changes from named patch to the working copy (patch creation is just a special case)
refresh- Refresh changes belonging to a named patch that is applied to repo history
push- Apply changes from named patch (or series of patches) to the repo history
pop- Unapply changes from named patch (or series of patches) from the repo history
goto- pop/push to get a different applied patchset in repo history
- Transfer content between patches
fold- rename, join patches, delete patches
split- split queues
join- join queues
rearrange- open up the attic rearrange-editor
- Status information
diff- get changes that would be used in a following shelve/refresh/store/... command
display- list patches/queues in the attic (knowing the storage model, this would be something like a attic-aware
ls .hg/attic/printing a bit of status info)
status- list info about the current queue (something like
ls .hg/attic/[queuename]/while printing status info)
finish- Delete this patch name or named series of patches
- ''Is this needed?''. It should delete a patch (series of patches) in the attic and would convert mutable changesets to normal ones (without qbase/qtip tags), without changing the working copy state (applied content would remain applied and same for unapplied content, which would disappear).
split --interactive- split a patch
shelve --interactive- shelve part of a patch (leave the rest applied)
guard- adds a guard to a patch (meaning afterwords the patch will be skipped over when pushing/popping/..., unless you have the guard selected)
select- selects a guard
Storage data model for attic
attic stores shelves as patch files in a private directory, and queues are represented as subdirectories with a series file and patches in it. Additional status information, as current patch and stack, applied patches or current guards would be stored in plain text files, preferably starting with a period (conventions dictating not to start patch names with a . are already in place).
attic command set
My current thoughts for the additions to attic with respect to mq are:
mq: strip strip a revision and all its descendants from the repository attic: probably will not include (this will be a separate extension you need to turn on by itself) mq: qclone clone main and patch repository at same time attic: might be considered for ease of users using a host like bitbucket.org mq: qcommit commit changes in the queue repository qinit init a new queue repository qsave save current queue state qrestore restore the queue state saved by a rev attic: will not be included by default mq: qfinish move applied patches into repository history attic: commit (commit will commit all applied patches followed by the current working directory) mq: qfold fold the named patches into the current patch qdelete remove patches from queue qrename rename a patch attic: fold fold all named patches into stdout --output name fold to output file instead (implies --remove) --remove erase named patches --applied fold all applied patches and the current working copy --limit # fold # topmost applied patches and the current working copy commit ops (-U, -D, -m, -e, -l, ...) examples: fold 3 patches into one: hg fold -o finalpatch queue1/patcha queue1/patchb someshelf delete a patch (output to /dev/null unnecessary, but would eliminate console output): hg fold --remove queue1/patcha >/dev/null rename a patch: hg fold -o queue1/patchb queue1/patcha mq: qimport import a patch qnew create a new patch qpop pop the current patch off the stack qpush push the next patch onto the stack attic: will mostly stay the same (name changes maybe?) mq: qgoto push or pop patches until named patch is at top of stack attic: qgoto will now be able to switch between queues as well: hg qgoto mypatch1 - stay in the current queue, goto mypatch1 hg qgoto otherqueue/ - pop all patches in the current queue and apply patches in the other queue until you hit the one you were on before in that queue hg qgoto otherqueue/somepatch - pop all patches in the current queue and go to 'somepatch' in the other queue mq: qguard set or print guards for a patch qselect set or print guarded patches to push attic: haven't thought about these yet mq: qdiff diff of the current patch and subsequent modifications qrefresh update the current patch attic: these will probably stay as shortcuts, both can be translated into fold: hg qdiff == hg fold -l 1 hg qrefresh == hg fold -l 1 -o tmp && hg qpop && mv [tmp] [current patch] && hg qpush mq: qprev print the name of the previous patch qapplied print the patches already applied qnext print the name of the next patch qseries print the entire series file qtop print the name of the current patch qunapplied print the patches not yet applied qheader Print the header of the topmost or specified patch attic: qstatus print the status of the current patch queue --header [name] print the header of the topmost or specified patch --verbose print all patches in the current queue (including patches not in series) --quiet print prev, current, next patches --queue name use queue <name> instead of current patch queue So, the initial commands for attic-queues will be the following to hit mq functionality: fold fold all given named patches into stdout qimport import a patch qgoto push or pop patches until named patch is at top of stack qnew create a new patch qpop pop the current patch off the stack qpush push the next patch onto the stack qguard set or print guards for a patch qselect set or print guarded patches to push qdiff diff of the current patch and subsequent modifications qrefresh update the current patch qstatus print the status of the current patch queue Additionally: split split a named patch into two patches interactively rearrange bring up an editor where you can swap lines to reorder queues, split them and pull patches out qsplit split a queue into two along some given arrangement qjoin append one queue to another There will also be some special names: 'hg qgoto somethingspecial/floor' - will goto the floor of the somethingspecial queue (change the current active queue to 'somethingspecial' and apply no patches) 'hg qgoto somethingspecial/ciel' - will goto the somethingspecial queue and apply all patches (according to guards) 'hg qfold somethingspecial/*' - will fold the entire somethingspecial queue and print the new patch to stdout 'hg qgoto queue/floor' - will go to the floor of the default queue (queue will be the name used if none is provided) 'hg qgoto someshelf/' - will create a queue from a shelf and apply it: move .hg/attic/someshelf to .hg/attic/someshelf/someshelf create .hg/attic/someshelf/series 'hg qgoto nonexistingqueue/' - will create a new queue with the provided name some example commands: 'hg qpush --shelf shelfname' - will push the shelf onto the queue as the topmost applied patch (will move the shelf) 'hg qpop --shelf' - will pop the topmost applied patch into a shelf - will turn a queue into a shelf if there is only one patch 'hg qnew --shelf shelfname' - will turn shelfname into a queue (just like 'hg qgoto shelfname/') 'hg qnew --queue newqueue patchname' - will create a new patch in a new queue (newqueue/patchname will be the new patch) 'hg qjoin queue1 queue2' - will move queue1/* into queue2 and append queue1/series to the end of queue2/series 'hg qsplit --guard "+stable" newqueue' - will split the current queue into: current containing all patches that would be applied if that guard was set newqueue containing all patches that would not be applied if that guard was set 'hg qsplit newqueue' - current contains all patches currently applied, newqueue contains the rest 'hg qsplit --patches p1 p2 p3 newqueue' - current contains all but p1, p2, p3; newqueue contains p1, p2, p3 in that order
Edited #mercurial IRC Log 28-01-2009
* mpm would prefer if there were one non-buggy, easy-to-use union of mq/pbranch/record/shelve/attic. <pachi> after_fallout: I have the new status and default patch working too <after_fallout> mpm: I don't think that will quite happen; mq and attic are entirely different directions and not very similar when you actually use them; they are actually decent compliments of each other <after_fallout> shelve/record/attic maybe could be one <mpm> pbranch appears to be a superset of mq, possibly of attic as well. <mpm> If there's some reason why we need to have more than one way to handle "storing work as patches", then I'm open to it. But not thrilled about it. <tonfa> mpm: I guess we at least need to keep a "quilt-like" mode <mpm> Yes, but there's no reason that we can generalize it to a DAG or whatever pbranch is doing. Or teach it about chunk granularity like record. <tonfa> yeah it seems like attic and pbranch could be combined <mpm> I've not really looked closely at any of them aside from mq, that's just my view from a distance. <tonfa> attic is one-level pbranch with nice interactive mode <after_fallout> I think attic is quite a bit simpler than pbranch too <after_fallout> it isn't meant for long term patch management <mpm> I appreciate that, however.. <muggs> big difference: pbranch work can be pulled/pushed around <mpm> It kind of sucks to have one tool for beginners and a second incompatible tool for experts. <mpm> (And to fix bugs in them both) <after_fallout> I agree, I don't think we will have that though <mpm> No? What happens when a user that starts with attic outgrows it? <tonfa> it sure would be nice to have just one extension that does everything, with one or two commands for beginners <tonfa> and the others for advanced usage <after_fallout> if the reason for having a patch around changes from "I am working on it but it is gonna sit around here for a bit while I do something else" to "I have to manage it for an extended period of time" then it is there for a different reason <pachi> if attic could become pbranch when working in "versioned" patches mode... <after_fallout> yeah something like that <mgeisler> tonfa: especially now that the three extensions have borrows a lot of code from each other <after_fallout> a considerable amount of the shared work can be joined together <after_fallout> attic uses the record code straight out of record <mgeisler> after_fallout: yeah, I'know its not that bad -- it just suggest that things should be merged like mpm says <after_fallout> or that there is some refactoring to do <after_fallout> for example see record.py line 68 <after_fallout> class header(object): <after_fallout> """patch header <after_fallout> XXX shoudn't we move this to mercurial/patch.py ? <after_fallout> """ <muggs> in order to support collaboration like pbranch does, the extension would need some way to extend the wire protocol <tonfa> ideally we should have a hg import --interactive <tonfa> when that happens it means the record stuff is sufficiently refactored :)
Edited IRC log, conversation between parren and pachi
<pachi> after_fallout is working on attic. I just did some minor contributions, but like it very much <parren> I can imagine. It sounds attractive. <parren> See, I think where attic/shelve shine is for quick juggling of several strands of work. <pachi> though it looks like having something as easy to work with as attic, but which gets into pbranch when it gets versioned looks like an ideal solution <pachi> yes <parren> So you want something like a pbranch import from attic? <pachi> but, from reading the pbranch docs I thought that the only "problem" with it is that it's 'in-history' <parren> And that is exactly why pbranch is so much more heavyweight than both mq and attic and shelve. <parren> Why are we discussing this in private? <pachi> one idea (probably stupid, but that's why I'm asking more competent people like you) is if it would be feasible to tag branches that are used in pbranch like the closed branches <pachi> using an extra tag that avoids to push/pull them <pachi> unless tagged to work otherwise <parren> Well, pulling/pushing these is one of the reasons I created pbranch. For when you want to collaborate on patches. <pachi> that would get us some sort of "local-only" branches that don't propagate <pachi> and one could strip them easily <pachi> so, if one wants to share them, one just untags the branch and they work as normal <pachi> but they could be kept as a local only thing meanwhile <parren> Hmm. <pachi> so shelve/unshelve could work like a pbranch without worry of history polution as it could be possible to reset that branch history (folding) <pachi> conceptually it's like having a full clone that you can publish when you want <parren> Are the new closed branches already exempt from push/pull? I doubt it somehow. <pachi> no <pachi> I had a look to what pull/push does, and it uses findincoming to find the tips that lack in local that are in remote, so excluding tagged heads could work <pachi> (or even walking a branch and having a cache) <parren> Yep, I can imagine. <pachi> does it make sense? <parren> Could work. Feels somewhat like git branches. <pachi> but they're like the inverse concept, you don't add remotes but tag "local-only" branches, as, by default, hg shares all of the branches <parren> Yes, of course. <pachi> I think after_fallout is willing to work on something better, and even on record/mq refactoring, but your pbranches look brilliant too <pachi> I know you're really busy, but do you think a common plan could be layed out? <parren> What this does not yet address is mq's guards. pbranch cannot do them. And mq is better at juggling patches freely. Swapping the order of two patches with pbranch is, for example, rather hard with pbranch (which is why you should be starting them as parallel patches in the first place). <pachi> but it looks more like an UI problem, doesn't it (the patchgraph thing IIRC)? <pachi> or is it a problem of how to get the desired order manipulating the DAG? <parren> No. If you have A->B->C, then C contains everything in B. Now try swapping to A->C->B. You have to first rebase C on A, then B on C. And preferably in a non-destructive way. <pachi> I see <parren> And guards are even harder. Every change in guard setup would effectively cause rebases recorded in history. Not likely what people want. <pachi> well, maybe mq still has its uses, but for special cases like those. At least a versioned version of attic/shelve would be good, even without stacked patches <pachi> maybe extracting those bits makes mq cleaner and stacks are implemented as an addition to the stacked patches <pachi> something like pbranch, but with an additional series file <parren> ? <parren> pbranch has the .hg/pgraph file, which is like .hg/patches/series. <pachi> so the problem is how the new patches are created, using overlays instead of disjoint patches? <parren> I like the idea of making attic more akin to starting a series of parallel pbranch patches. <parren> You've lost me there. What do you mean by "overlay"? <pachi> yes, that's the idea (mpm suggested that pbranch is like a superset of mq) <parren> Like, yes. But as I said, mq still has its strong points. <pachi> as I understand it, the problem with pbranch is that you start a patch an you later don't really want to create a new ppatch on top of it if they need to be reordered later ala mq <pachi> s/an/and/ <parren> Right. You should start it independently. Which is a kind of bad premise because you're prone to forget. <parren> It's like doing fixes properly at the root cause so you don't have to backport, only forward-merge. Everybody knows this, but it just does not always happen this way. <pachi> but if mq manipulations had a separate pbranch that gets done and undone transparently but take each of the pieces of the stack from isolated pbranches, then it could perhaps work <pachi> s/done and undone/applied and unapplied/ <parren> Only as long as all the patches are truly independent. Usually, some are, some are not. <pachi> the problem now (as I see it) is that creating new branches that get destroyed are a bad idea because they get shared and then are there forever <pachi> but if they were not independent wouldn't we gave the same problems than with mq? <parren> Yes. When you work with pbranch today, you have to work in a separate clone which you plan to throw away eventually (when the patches have been merged upstream). <pachi> *have <parren> Yes, with mq you then get failed applies. <pachi> if that clone were an 'in-repo but only local' branch, it wouldn't be needed <parren> Correct. So I kind of like this idea. <pachi> the point is... if we had throw away branches that don't get published... would it allow to do things differently? <parren> Yes. Maybe we could even say that while you're only using this by yourself, we don't have to use hg branches at all and use bookmarks instead. And flag some bookmarks as non-push/pullable. <pachi> I had the intutition that it would be so, as hg can do the same as git, but we need clones for strategically hiding branches and that has a cost when trying to work from a given repo, as some operations are problematic without resorting to an external clone <pachi> yes <pachi> all existing syntactic sugar could be applied to it <parren> When you want to share, you migrate them to proper branches. This could be a pbranch command, committing all pbranch bookmarks to pbranch branches. <pachi> yes, that's it <parren> Using bookmarks you also won't run into trouble with colliding branch names (say remote has a new branch that collides with one of your private attic branches). But I think we should define a branch namespace for pbranch et al anyway. <parren> Both bookmarks and pbranch still need the wire protocol change so they can transmit their metadata (bookmark definitions and .hg/pgraph). <pachi> indeed, it would have it's own space (really, like what mq has) to do the bookkeeping <parren> Well, maybe not. In this case it probably suffices if local fs->fs clones copied the stuff. <pachi> but while that doesn't happen we'd only miss how patches relate to each other, wouldn't we? <pachi> I mean... we'd have all the patches (if they were shared) but not how they get merged in a series <parren> pbranch actually recreates a good idea of the relations from the recorded dependencies. See http://arrenbrecht.ch/mercurial/pbranch/collab.htm#pagetoc__1_2 <pachi> that would be still very useful, but more similar to a versioned attic <pachi> I see <parren> And now that we have branch closing, it could recreate an even better idea. <pachi> maybe tonfa knows if that 'local only' or 'locked' branches is feasible. IIRC he worked on findincoming... <parren> I've also dug into it for the shallow clones. I think it can work. After all, it really amounts to the same as `hg push -r x -r y -r z` where xyz are the heads of all non-blocked branches, no? <pachi> yes, and I even thing (from a very superficial understanding) that it could be just removing the 'local' heads <parren> So for push it sounds like even an extension could do it, but wrapping `hg push`. <parren> For pull it's likely harder, since remote has to do the filtering. <pachi> but pulling from it should work the same <pachi> I can imagine that that's where the fun begins (how to get the subgraph) <parren> subgraph meaning? pgraph? <pachi> no, how to 'obliterate' the non shared parts <pachi> the non-shared branches could be useful for many other things I suppose <pachi> besides pbranch <parren> Sure. <parren> But I'd really focus on non-shared heads, not branches. More general, and works with bookmarks. <pachi> fine <pachi> I was thinking in the same semanthics as branch "closing" <parren> So: attic becomes a bunch of independent pbranch patches, but based on bookmarks instead of named branches. And we block them from getting out of the repo. And we make pbranch work with both bookmarks and named branches, with an option to promote bookmarks to branches for sharing. Is that the plan? <pachi> but 'hiding' here instead <pachi> parren that's it :) <parren> Maybe this could be tied into bookmarks directly. Flag a bookmark as hidden. <pachi> well, those would be details that depend on the wire protocol allowing the transfering of some bookkeeping <pachi> even if a named branch had to be used to have a name for patches while that's not available would do <pachi> IMHO <parren> Actually, attic would become just a bunch of hidden heads, with no pbranch meta-info needed. But you can seamlessly upgrade from a patch in the attic to making it a full-blown pbranch patch. <pachi> handling that as metadata or inmutable information is equivalent in some way if we can throw away branches <pachi> having the easiest UI to start from something like attic and go to a full pbranch is key, IMHO <parren> Things is, we're talking about hiding these things. So if the things don't get propagated, then we don't need to propagate meta-info about them either, do we? <pachi> no <pachi> only if they get shared, that's why how storing the patch names isn't important until they get shared, and probably the bookmark wire extension is needed <pachi> but the problem of sharing metadata is not a problem of this particular idea <parren> Well, as soon as we share, we don't them hide anymore, do we? Which is why I say we teach pbranch to work with both bookmarks (non-shared) and named branches (shared). <parren> So once we share, we don't hide, so there is no hiding metadata to share anymore <pachi> I'd use only bookmarks, as mpm probably wants to solve the bookmarks sharing problem anyway <pachi> but, that's not important ATM, IMHO <pachi> at the very least we'd have what pbranch has now ;) <parren> Right. And since pbranch already does named branches and needs to be taught to use bookmarks, we lose nothing. And once bookmarks can be shared we can maybe drop the named branch support in pbranch. So this might be the time to rename pbranch. :) <parren> How about sending a summary as an RFC to the devel list? With this transcript and excerpts from the one tonfa sent appended? <pachi> but otherwise I can try. I can edit text better than coding ;) <parren> I'll see if I can find some time tonight. Got to go now. Kids and dinner time. I enjoyed this discussion. Thanks. <pachi> ok, thanks a lot
Edited IRC log, conversation between after_fallout and pachi
<pachi> hi <pachi> I've been talking to parren, the pbranch extension author <pachi> about an idea to unify pbranch/attic/shelve, as mpm suggested <after_fallout> I had the same thoughts as he does prettymuch with pbranch <after_fallout> looking at pbranch we seem to have a couple of mostly minor issues <after_fallout> 1. you have to know in advance when you want parallel branches <after_fallout> 2. it does things to the history of your repository that get propagated when you push and pull <after_fallout> mq doesn't really have those problems <pachi> 1 and 2 can be solved with the 'local only' branches IMHO <after_fallout> I think you can avoid them in the first place <pachi> well, they can be avoided if some branches can be hidden from sharing <pachi> that's at the end what mq does <pachi> and attic, even if unversioned <pachi> it just hides information from propagating <after_fallout> true <pachi> so you can alter contents without polluting history <pachi> it just adds an outer repo to do its job, but it could be done in a branch (a specialized branch) to avoid the problems of managing it (qclone, qpush -R and so on) <pachi> same with versioned attic <after_fallout> ahh, bookmarks instead of named branches <pachi> :) <after_fallout> that would solve some of the problems <after_fallout> right, in these secondary repos, you don't actually care about the history, only that you have the right version <after_fallout> you need the history to be able to direct a merge, but you don't need it <pachi> as they won't pollute anything, and could even be stripped of folded or anything without disturbing other repos <after_fallout> to see what happened at any given time <pachi> yes <pachi> pbranch has a sort of pseries in the form of pgraph <after_fallout> yeah sorta <pachi> as I see it it's like an out of band DAG <pachi> so the relationships between changesets for ordering are done with that instead of using hg itself, which is used to get the real thing, but not to store their relations initially <after_fallout> more like a set of dags I think <pachi> a set of dags and a way to relate them using another metadag that does the stacking <pachi> well, at least that's what I understand... <after_fallout> the way I understood it, you could start multiple pbranches on different nodes on the repository and never have them relating to each other <pachi> yes, I think it's like that, though they can relate too, but it's not required <after_fallout> a set of connected pbranches is a DAG, but the pgraph contains all the sets of pbranches <after_fallout> which may or may not be all connected <pachi> that's the problem parren saw to replicate mq behaviour, but it can be done using a new 'local' branch for stacking <pachi> essentially, that's what mq does with the main repo it versions <after_fallout> I guess <pachi> use another repo to do the compositing, but we could do that in the same repo using another local branch (that can be stripped and reapplied at will) and still keep patches standalone