Backport changes in preparation for git migration from trunk.

[gnu-emacs] / admin / notes / repo

NOTES ON COMMITTING TO EMACS'S REPOSITORY    -*- outline -*-

* Commit to the right branch

You can view the available Emacs branches at

http://bzr.savannah.gnu.org/r/emacs/

Development normally takes places on the trunk.
Sometimes specialized features are developed on separate branches
before possibly being merged to the trunk.

Development is discussed on the emacs-devel mailing list.

Sometime before the release of a new major version of Emacs
a "feature freeze" is imposed on the trunk.  No new features may be
added after this point.  This is usually some months before the release.

Shortly before the release, a release branch is created, and the
trunk is then free for development.

For example, "emacs-23" for Emacs 23.2 and later, "EMACS_23_1_RC" for
23.1, "EMACS_22_BASE" for 22.x, and "EMACS_21_1_RC" for 21.x.

Consult emacs-devel for exactly what kinds of changes are allowed
on what branch at any time.

** elpa

This branch does not contain a copy of Emacs, but of the Emacs Lisp
package archive (elpa.gnu.org).  See admin/notes/elpa for further
explanation, and the README file in the branch for usage
instructions.

* Install changes only on one branch, let them get merged elsewhere if needed.

In particular, install bug-fixes only on the release branch (if there
is one) and let them get synced to the trunk; do not install them by
hand on the trunk as well.  E.g. if there is an active "emacs-24" branch
and you have a bug-fix appropriate for the next emacs-24.x release,
install it only on the emacs-24 branch, not on the trunk as well.

Installing things manually into more than one branch makes merges more
difficult.

http://lists.gnu.org/archive/html/emacs-devel/2010-03/msg01124.html

The exception is, if you know that the change will be difficult to
merge to the trunk (eg because the trunk code has changed a lot).
In that case, it's helpful if you can apply the change to both trunk
and branch yourself (when committing the branch change, indicate
in the commit log that it should not be merged to the trunk; see below).

* Backporting a bug-fix from the trunk to a branch (e.g. "emacs-24").

Indicate in the commit log that there is no need to merge the commit
to the trunk.  Anything that matches `bzrmerge-skip-regexp' will do;
eg start the commit message with "Backport:".  This is helpful for the
person merging the release branch to the trunk.

http://lists.gnu.org/archive/html/emacs-devel/2010-05/msg00262.html

* Installing changes from your personal branches.

If your branch has only a single commit, or many different real
commits, it is fine to do a merge.  If your branch has only a very
small number of "real" commits, but several "merge from trunks", it is
preferred that you take your branch's diff, apply it to the trunk, and
commit directly, not merge.  This keeps the history cleaner.

In general, when working on some feature in a separate branch, it is
preferable not to merge from trunk until you are done with the
feature.  Unless you really need some change that was done on the
trunk while you were developing on the branch, you don't really need
those merges; just merge once, when you are done with the feature, and
Bazaar will take care of the rest.  Bazaar is much better in this than
CVS, so interim merges are unnecessary.

Or use shelves; or rebase; or do something else.  See the thread for
yet another fun excursion into the exciting world of version control.

http://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00086.html

* Installing changes from gnulib

Some of the files in Emacs are copied from gnulib.  To synchronize
these files from the version of gnulib that you have checked out into
a sibling directory of your branch, type "admin/merge-gnulib"; this
will check out the latest version of gnulib if there is no sibling
directory already.  It is a good idea to run "bzr status" afterwards,
so that if a gnulib module added a file, you can record the new file
using "bzr add".  After synchronizing from gnulib, do a "make" in the
usual way.

To change the set of gnulib modules, change the GNULIB_MODULES
variable in admin/merge-gnulib before running it.

If you remove a gnulib module, or if a gnulib module
removes a file, then remove the corresponding files by hand.

* How to merge changes from emacs-24 to trunk

The following description uses bound branches, presumably it works in
a similar way with unbound ones.

0) (This step is only necessary if using bzr older than 2.4.0.)
Get the bzr changelog_merge plugin:

cd ~/.bazaar/plugins
bzr branch http://bazaar.launchpad.net/~spiv/bzr-changelog-merge/trunk changelog_merge

This plugin should make merging ChangeLogs smoother.  It merges new
entries to the top of the file, rather than trying to fit them in
mid-way through.  Newer versions of the plugin should also be able to
deal with changes to *old* ChangeLog entries, that should not be
floated to the head of the file (see launchpad#723968).

It is included in bzr from 2.4.0 onwards, so remember to delete the
copy in ~/.bazaar if you upgrade bzr.

Maybe the default Emacs behavior without this plugin is better,
though, it's not clear yet.

1) Get clean, up-to-date copies of the emacs-24 and trunk branches.
Check for any uncommitted changes with bzr status.

2) M-x cd /path/to/trunk

The first time only, do this:
cd .bzr/branch
Add the following line to branch.conf:
changelog_merge_files = ChangeLog

3) load admin/bzrmerge.el

4) M-x bzrmerge RET /path/to/emacs-24 RET

It will prompt about revisions that should be skipped, based on the
regexp in bzrmerge-missing.  If there are more revisions that you know
need skipping, you'll have to do that by hand.

5) It will stop if there are any conflicts.  Resolve them.
Using smerge-mode, there are menu items to skip to the next conflict,
and to take either the trunk, branch, or both copies.

6) After resolving all conflicts, you might need to run the bzmerge
command again if there are more revisions still to merge.

Do not commit (or exit Emacs) until you have run bzrmerge to completion.

Before committing, check bzr status and bzr diff output.
If you have run bzrmerge enough times, the "pending merge tip" in bzr
status should be the last revision from the emacs-24 branch, and
bzr status -v should show all the revisions you expect to merge.

(Note that it will also show "skipped" revisions.  This is expected,
and is due to a technical limitation of bzr.  The log data for those
revisions gets merged, the actual changes themselves do not.
http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00609.html )

In particular, check the ChangeLog entries (eg in case too many
entries have been included or whitespace between entries needs fixing).
bzrmerge tries to fix up the dates to today's date, but it only does
this where there are conflicts.  If you used the changelog_merge plugin,
there won't be any conflicts, and (at time of writing) you will need
to adjust dates by hand.  In any case, if someone made multiple
ChangeLog entries on different days in the branch, you may wish to
collapse them all to a single entry for that author in the trunk
(because in the trunk they all appear under the same date).
Obviously, if there are multiple changes to the same file by different
authors, don't break the logical ordering in doing this.

Notes:

1) If a file is modified in emacs-24, and deleted in the trunk, you
get a "contents conflict".  Assuming the changes don't need to be in
the trunk at all, use `bzr resolve path/to/file --take-this' to keep the
trunk version.  Prior to bzr 2.2.3, this may fail.  You can just
delete the .OTHER etc files by hand and use bzr resolve path/to/file.

2) Conflicts in autoload md5sums in comments.  Strictly speaking, the
right thing to do is merge everything else, resolve the conflict by
choosing either the trunk or branch version, then run `make -C lisp
autoloads' to update the md5sums to the correct trunk value before
committing.

* Re-adding a file that has been removed from the repository

It's easy to get this wrong. Let's suppose you've done:

bzr remove file; bzr commit

and now, sometime later, you realize this was a mistake and file needs
to be brought back.  DON'T just do:

bzr add file; bzr commit

This restores file, but without its history (`bzr log file' will be
very short).  This is because file gets re-added with a new file-id
(use `bzr file-id file' to see the id).

Instead of adding the file, try:

bzr revert -rN file; bzr commit

where revision N+1 is the one where file was removed.

You could also try `bzr add --file-ids-from', if you have a copy of
another branch where file still exists.

* Undoing a commit (uncommitting)

It is possible to undo/remove a bzr commit (ie, to uncommit).
Only do this if you really, really, need to.  For example, if you
somehow made a commit that triggers a bug in bzr itself.
Don't do it because you made a typo in a commit or the log.

If you do need to do this, do it as soon as possible, because the
longer you leave it, the more work is involved.

0. First, tell emacs-devel that you are going to do this, and suggest
people not commit anything to the affected branch for the duration.

In the following, replace USER with your Savannah username, and
BRANCH with the name of the branch.
Let's assume that revno 100 is the bad commit, and that there have
been two more commits after that (because nothing is ever easy).

1. Ensure your copy of the branch is up-to-date (for a bound
branch, bzr up; for an unbound branch, bzr pull) and has no local
changes (bzr st).

2. Make a record of the commits you are going to undo:
bzr diff -c 102 > /tmp/102.diff
etc

Also record the commit message, author, and any --fixes information.

3. Most Emacs branches are set up to prevent just this kind of thing.
So we need to disable that protection:

bzr config append_revisions_only=False \
  -d bzr+ssh://USER@bzr.savannah.gnu.org/emacs/BRANCH/

4. Undo the commits:
bzr uncommit -r -4

This will show the commits it is going to undo, and prompt you to confirm.

5. If using an unbound branch:
bzr push --overwrite

6. Now, replay the commits you just undid (obviously, fix whatever it
was in the bad commit that caused the problem):

patch -p0 < /tmp/100.diff
bzr commit --author ... --fixes ... -F /tmp/100.log
etc

7. If using an unbound branch:
bzr push

8. Finally, re-enable the branch protection:
bzr config append_revisions_only=True \
  -d bzr+ssh://USER@bzr.savannah.gnu.org/emacs/BRANCH/

9. Tell emacs-devel that it is ok to use the branch again.
Anyone with local changes should back them up before doing anything.

For a bound branch, bzr up will convert any of the undone commits to a
pending merge. Just bzr revert these away.

For an unbound branch, bzr pull will complain about diverged branches
and refuse to do anything.  Use bzr pull --overwrite.

* Loggerhead

Loggerhead is the bzr tool for viewing a repository over http (similar
to ViewVC).  The central version is at http://bzr.savannah.gnu.org/lh/emacs,
but if you just like the way this interface presents data, then if
you have your own copy of the repository, you can operate your own
Loggerhead server in stand-alone mode, and so help to reduce the load
on Savannah:

  bzr branch lp:loggerhead ~/.bazaar/plugins/loggerhead
  cd /path/to/emacs/bzr
  bzr serve --http

You may need to install some Python dependencies to get this command to work.
For example, on RHEL6 I needed:

  yum install python-paste python-simplejson
  yum --enablerepo=epel install python-simpletal

Then point your web-browser to http://127.0.0.1:8080/ .

* Bisecting

This is a semi-automated way to find the revision that introduced a bug.

First, get the bzr bisect plugin if you do not have it already:

  cd ~/.bazaar/plugins
  bzr branch lp:bzr-bisect bisect

`bzr help bisect' should work now.

It's probably simplest to make a new copy of the branch to work in
from this point onwards.

Identify the last known "good" revision where the relevant issue is
NOT present (e.g. maybe Emacs 24.1).  Let's say this is revision 1000.

  bzr bisect start
  bzr bisect no -r 1000

At this point, bzr will switch to the mid-point of revision 1000 and
the current revision.  If you know that the issue was definitely
present in some specific revision (say 2000), you can use:

  bzr bisect yes -r 2000

Now bzr switches to revision 1500.

Now test whether the issue is present.  You might need to rebuild
Emacs to do this, or if you know the problem is in a specific Lisp
file, you might be able to get away with just loading that one file in
current Emacs.

If the issue is present, use

  bzr bisect yes

If it is not, use

  bzr bisect no

Repeat until you zero-in on the specific revision.

When finished, use

  bzr bisect reset

or simply delete the entire branch if you created it just for this.

* Commit emails

** Old method: bzr-hookless-email
https://launchpad.net/bzr-hookless-email

Runs hourly via cron.  Must ask Savannah admins to enable/disable it
for each branch.  Stores the last revision that it mailed as
last_revision_mailed in branch.conf on the server.  Breaks with bzr 2.6:

http://lists.gnu.org/archive/html/savannah-hackers-public/2013-05/msg00000.html

Fix from https://bugs.launchpad.net/bzr-hookless-email/+bug/988195
only partially works.  Breaks again on every merge commit:

https://lists.ubuntu.com/archives/bazaar/2013q2/075520.html
http://lists.gnu.org/archive/html/savannah-hackers-public/2013-05/msg00024.html

You can force it to skip the merge commit by changing the value for
last_revision_mailed, eg:

bzr config last_revision_mailed=xfq.free@gmail.com-20130603233720-u1aumaxvf3o0rlai -d bzr+ssh://USERNAME@bzr.savannah.gnu.org/emacs/trunk/

** New method: bzr-email plugin
https://launchpad.net/bzr-email
http://lists.gnu.org/archive/html/savannah-hackers-public/2013-06/msg00007.html

Runs on commit.  Projects can enable it themselves by using `bzr
config' to set post_commit_to option for a branch.  See `bzr help email'
(if you have the plugin installed) for other options.

The From: address will be that of your Savannah account, rather than
your `bzr whoami' information.

Note: if you have the bzr-email plugin installed locally, then when
you commit to the Emacs repository it will also try to send a commit
email from your local machine.  If your machine is not configured to
send external mail, this will just fail.  In any case, you may prefer
to either remove the plugin from your machine, or disable it for Emacs
branches.  You can do this either by editing branch.conf in your Emacs
branches, to override the server setting (untested; not sure this
works), or by adding an entry to ~/.bazaar/locations.conf:

  [bzr+ssh://USERNAME@bzr.savannah.gnu.org/emacs/*/]
  post_commit_to = ""

You have to use locations.conf rather than bazaar.conf because the
latter has a lower priority than branch.conf.

* Using git-bzr

** initially

You can use Git locally to talk to the Bazaar repo as a "remote" repo
via git-bzr (aka git-remote-bzr).  Initial clone:

 git clone bzr::bzr+ssh://USER@bzr.sv.gnu.org/emacs/trunk e

This creates the working dir e/ (with subdir .git, etc).  Disk usage
is 13G (as of early 2014), so you will probably want to repack:

 git repack -a -d -f --window=250 --depth=250 --window-memory=N

where N is chosen to avoid swapping.  E.g., given 512MB RAM, N="200m"
results in "du -sh .git" => 559M, about double the smallest reported
value (obtained with "deprecated" command "git gc --aggressive").

** steady-state

Use "fetch", "pull" and other remote-to-local commands as usual.

For "push", the Emacs Bazaar repo is configured with

 append_revisions_only = True

so some versions of git-remote-bzr may raise AppendRevisionsOnlyViolation
(in func do_export) instead of displaying a "non fast-forward" message
and skipping the branch.  See:

 http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00436.html

which includes a provisional patch to git-remote-bzr to do that.