| < Debian Packaging Tools | Russ Allbery > Technical Notes > Debian | Packaging Scripts for Debian > |
Introduction
Initial Configuration
Repository Layout
Creating a New Package
Packaging Workflow
Handling Repacked Upstream
Backporting
Debian and Upstream Combined
Git Cheat Sheet
Converting to Git
There are a bunch of other pages out there on the web, and this doesn't attempt to be a comprehensive guide. Instead, this is an opinionated description of the tools I personally use for Debian packaging with Git, and a description of my workflow. I maintain it partly for my own purposes as a reminder, but you may find it interesting while developing your own workflows.
The Git version control system (VCS) is a distributed VCS originally written by Linus Torvalds for Linux kernel development after moving away from BitKeeper. "Distributed" means that it has strong support for branching, merging, and parallel lines of development, stores the complete repository history locally, and is designed for multiple people working on the same software in parallel and merging in complex ways between their separate trees.
Of the distributed version control systems, Git has the strongest following among free software developers at the moment and is the most popular in Debian for packaging. Before switching to Git, I used Arch (bazaar and tla), bazaar-ng (bzr), and svk, as well as (non-distributed) Subversion. Prior to Debian packaging, I've also used CVS and RCS extensively, and I've used Mercurial at work (although not for Debian packaging). Disclaimer: I think Git is intuitive and natural, and I think Mercurial is awful, confusing, broken, and slow. If this is the exact opposite of your opinion (and that is true for many people), this workflow may not be very comfortable for you.
I used to have getting-started information about how to use Git on this page, but there are now so many resources on-line for learning Git that it didn't seem worthwhile to keep this up-to-date. One starting place if you're unfamiliar with Git basics is the Git user manual. There are lots of others.
First, you will need a build environment for packages. For new systems, I
use sbuild with btrfs, documented on a separate page.
The other common option is pbuilder, with or without
cowbuilder to speed up the chroot setup.
There are a lot of different tools for using Git for Debian packaging with different properties. I won't try to present a comprehensive survey, just the option I use. I do most development with git-buildpackage, but use dgit to make the final upload.
First, set up some global configuration for git-buildpackage. I use the
following settings in ~/.gbp.conf:
[DEFAULT]
builder = sbuild
pristine-tar = True
[buildpackage]
export-dir = ../build-area
tarball-dir = ../tarballs
[import-orig]
dch = False
[dch]
full = True
This enables pristine-tar, which stores information used to
re-create the upstream tarball in the Git repository. Use of this program
is a bit controversial since it doesn't always work properly, but I've had
good luck with it. Just be aware that, if you run into problems, you may
need to download the previous upstream tarball from Debian.
This uses the convention of ../build-area for storing build results
(and ../tarballs for upstream tarballs if you're not using
pristine-tar), rather than cluttering up the parent directory. I
much prefer this, since Debian package builds generate a bunch of messy
files, but be aware that dgit doesn't support this. That's why I use
git-buildpackage for all the intermediate builds and only use dgit for the
final upload build.
For all Debian packages that aren't maintained as part of a larger
packaging group with different conventions, I use the DEP-14 branch layout. The short version is that packaging for the
unstable distribution goes in debian/master, packaging for stable
versions (security updates, for instance) go in debian/<codename>
branches named for the release code name, and upstream tarballs are
imported into upstream/latest.
To indicate that a package uses this branch layout, I put the following in
debian/gbp.conf in the package:
[DEFAULT]
debian-branch = debian/master
upstream-branch = upstream/latest
This is included in the package, not just set locally in my own configuration, so that anyone who clones the package from its Git repository will get the proper configuration.
There are various options if Debian packaging requires changes to the
upstream source. I've eventually converged on using the gbp pq
tool (included in git-buildpackage) to manage a quilt series of patches
against the upstream source while using Git to edit the packages when
needed. More on that below. This means there are no persistent branches
for patches against upstream, just transient branches named things like
patch-queue/debian/master.
If upstream also uses Git, I add the upstream repository as a Git remote
(usually using the remote name upstream) and will sometimes
maintain a local master branch tracking upstream. One special case
of this is when I'm also upstream, which is discussed below. One of the
advantages of the DEP-14 branch layout is that it easily allows mixing
upstream development and Debian patching in the same repository without
mixing Debian packaging files with upstream files.
git-buildpackage can do most of the work of managing the
upstream/latest branch. To start a new Git repository from an
existing package without caring about the history, run:
gbp import-dsc --git-debian-branch=debian/master \
--git-upstream-branch=upstream/latest /path/to/dsc
You'll get a sub-directory of your current directory named after the
package and containing an initialized Git repository with an
upstream/latest and debian/master branch and with (if you
use the above configuration) a pristine-tar branch that holds the
metadata used by pristine-tar. gbp import-dsc will also create an
upstream/* tag for the upstream version and a debian/* tag
for the Debian version of the package.
If you're importing an existing package with source format 1.0 that has
upstream patches mingled with Debian changes, you may need to shuffle
things around a bit. I generally do that by saving off the upstream
patches, reverting all those changes on debian/master, and then
using gbp pq to re-apply the patches in a proper quilt sequence
using the instructions below.
If you're instead starting from scratch with packaging a new package,
create an empty directory, cd into it, and run git init to create
an empty repository. Then rename the upstream tarball to
<package>_<version>.orig.tar.gz (don't repack it unless you have
to, just rename it) and run:
gbp import-orig --git-debian-branch=debian/master \
--git-upstream-branch=upstream/latest /path/to/upstream/tarball
This will create an upstream/latest branch and load the upstream
source and then merge that onto your master branch. You can then create
the debian directory and start creating the packaging.
I always use 3.0 (quilt) as the source package format (configured
in debian/source/format). Maintaining changes from upstream as a
stack of diffs is a bit tedious, but gbp pq makes it simple enough
(described below), and it's very nice to have separated patches to send
upstream when needed.
I also always write a debian/watch file if upstream puts their
releases anywhere that uscan can work with (which is almost
always). This not only allows Debian QA tools to alert me for new
upstream releases, but also allows git-buildpackage to easily download the
new upstream releases for me. (See below.) See the uscan(1) manual page
for information on how to write that configuration file.
Further Debian-specific work can then be done on the debian/master
branch by switching to it with git checkout, modifying and
committing things.
If I need to make changes to the upstream source, I first run:
gbp pq import
to import the current patches and switch to a patch-queue branch. I then
create new commits, and possibly rebase (against debian/master) to
reorder or change existing commits. When done, I run:
gbp pq export
gbp pq drop
git add debian/patches
git commit
to commit the changes to the patch sequence.
When ready for a test build, run gbp buildpackage. The package
will be built and, if successful, put in ../build-area, where I
inspect the package with debc and run lintian on it to look
for any issues.
When a new release is building and tested and ready to upload to Debian, I
make sure that the target distribution in debian/changelog is
correct and the timestamp on the changelog entry has been updated, and
then run:
origtargz
dgit --gbp sbuild
dgit --gbp push
to build the final version of the package. origtargz checks out
the upstream tarball using pristine-tar and puts in the parent directory
where sbuild and dgit expect it. Unfortunately, dgit gbp-build
doesn't work because dgit doesn't support ../build-area and can't
find the correct files.
When there's a new upstream release, run:
git fetch upstream
gbp pq import # only if you have patches against upstream
git checkout debian/master
gbp import-orig --uscan --upstream-vcs-tag=<upstream-tag>
# The rest are only if you have patches against upstream
gbp pq rebase
gbp pq export
gbp pq drop
git add debian/patches
git commit
If you don't have a working debian/watch file or don't want to use
uscan, you can also download the upstream release tarball, omit the
--uscan flag to gbp import-orig, and just provide the path
to the upstream release tarball. I do this when packaging software for
which I'm also upstream, since I generally prepare the Debian package with
a local release tarball that I haven't published yet, and then publish the
release tarball and the Debian package at the same time.
This assumes that upstream is also using Git and tags their releases. If
not, remove the git fetch command and the --upstream-vcs-tag
flag. This fetches the upstream tarball using uscan, merges it
into debian/master, and sets up that commit to merge with the
upstream Git history, if available, so that you can easily look through
upstream changes with git log.
If upstream always uses the same tag pattern for release tags, you can set
that pattern with an upstream-vcs-tag setting in
debian/gbp.conf and then don't have to remember to specify the
flag. See the gbp-import-orig(1) man page for more details.
Note the gbp pq commands, which are necessary to rebase any patches
you have against the upstream source. You can omit them if you're not
carrying any patches.
When making additional changes to the Debian packaging, I still commit
debian/changelog entries along with the corresponding change and
just resolve the conflicts when I cherry-pick or merge. This doesn't
really bother me, but it is some additional work. Some people instead
leave the Debian changelog file alone until the release and then use
gbp dch or some similar tool to generate it and then edit it. At
some point, I'd like to play with that idea.
debcommit's support for determining the commit message from my changelog entry doesn't do what I want, so I write the commit message separately. Your mileage may vary.
gitk is incredibly useful for performing any sort of archaeology. It's particularly useful if you forget to tag uploads and need to figure out what specific revision corresponded to a Debian package upload.
If the upstream source has to be repacked for DFSG reasons, you can just
generate a new *.orig.tar.xz file outside of Git and then import it
as described above. However, Git and git-buildpackage have some
additional features that may make this easier if you're just excluding
certain files from the upstream source.
The easiest approach is when you use a copyright-format
1.0 debian/copyright file and have a working debian/watch
file. In this case, add a Files-Excluded field to the first stanza
of that file, listing the files in the upstream source that must be
omitted. uscan will then automatically do the repacking for you.
See uscan(1) for more information.
If you're not using that format, you can import the upstream distribution selectively using:
gbp import-orig --upstream-vcs-tag=<upstream-tag> --filter='rfc*' \
/path/to/upstream/tarball
for example, adding --filter options for every file in the upstream
source that you need to exclude. (These options can be recorded in
debian/gbp.conf for others working on the package rather than
giving them directly on the command line.) Then, you can generate the
DFSG-free version of the upstream tarball with:
git archive --prefix=<package>_<version>.orig/ upstream/<version> \
| xz > <package>_<version>.orig.tar.xz
pristine-tar commit <package>_<version>.orig.tar.xz
rm <package>_<version>.orig.tar.xz
Whether to use the uscan(1) approach, this approach, or the more
traditional approach of generating the *.orig.tar.xz file
separately and then using gbp import-orig as always is up to you
and depends on which approach you find simpler.
Often backporting a package to stable is as simple as just adding a new changelog entry for the backport and then rebuilding it in a stable chroot instead of an unstable chroot. This document doesn't cover that case, since it can be done without any representation in the version control repository (and normally there's no point in committing such backports anywhere).
If backporting requires more work, such as changing build dependencies or
adjusting the packaging for tools that aren't present in stable, the
easiest way to do this using Git is to create a new branch off of
debian/master and make the necessary changes there. For example,
for a stretch backport:
git checkout -b debian/stretch-backports debian/master
# edit debian/control and other files as needed
# edit debian/changelog to add backport log entry
gbp buildpackage --git-debian-branch=debian/stretch -v<last-version>
The -v option should specify the last version of the package
available in stable, if it was previously in stable, so that the
*.changes file will include all changes since that date. The
version of the backport should follow the conventions of whatever
repository for which it is being prepared; for the backports.debian.org
repository, see the documentation for contributing.
For later backports, you normally will be able to merge the new master branch into the backport branch with:
git checkout debian/stretch
git merge debian/master
The debian/changelog file may have conflicts that have to be
resolved, normally by removing the old backport log entry and adding a new
one for the new backport.
For many of my packages, I'm both the Debian packager and the upstream maintainer. In those cases, I want a combined repository that I can use for both upstream maintenance and for Debian packaging.
Before using Git, I used to do this by maintaining the debian
directory in with the upstream code like any other part of the code, on
the main development branch, and just excluding it from releases. Then,
to build Debian packages, I'd unpack the latest tarball, export the
debian directory from the repository into the unpacked working
tree, and build the result. I had a script that automated much of this
process.
This works, but it has a few drawbacks:
It requires automation and pieces external to the repository, such as the latest upstream tarball. Without my script to automate the builds, setting up the build directory is somewhat tedious and requires explanation. Someone can't just check out the repository and build new Debian packages based on the current version without some extra effort.
It requires that the only changes for Debian packaging be isolated to the Debian directory. If I use a patch system, this is okay, but it limits the possibilities.
After switching to Git, I wanted a more general solution without these limitations that could also leverage pristine-tar so that all the information required to build the package was within the repository.
The wonderful part of the DEP-14 layout plus git-buildpackage's support
for merging with upstream releases when importing upstream tarballs means
that using the above workflow just works. I keep the upstream source on
the master branch, the Debian packaging on the debian/master
branch, generate a tarball as normal for upstream releases, and import it
like I would with any other Git-using upstream. All the right things
happen, including maintaining a proper merged Git history in the Debian
packaging branch.
For upstream releases, I make release/<version> signed Git tags,
and then reference those with --upstream-vcs-tag or, better, the
corresponding configuration in debian/gbp.conf.
Converting an existing repository that combines Debian packaging and the
upstream development is a bit tricky, since you want to create a new
debian/master branch that shares the history of the current
master and an upstream/latest branch that can be merged into
debian/master and used by gbp import-orig, but you also want
to remove the debian/* directory from the master branch.
The first time I tried this, I ended up merging the removal of the
debian/* directory into my debian/master branch and having
to back out of that.
After some experimentation, the following approach seems to work the best:
Before removing debian/* from master, create the
debian/master branch with git branch debian/master
master.
git rm -r debian on the master branch and commit it.
git branch upstream/latest master. Now you have an upstream
branch with the right history and without the debian/*
directory, although it doesn't match the upstream tarball release
(it's missing generated files).
Record in Git that the debian/master branch is already a
correct reflection of the changes relative to the
upstream/latest branch by adding a merge commit. git
checkout debian/master and then git merge -s ours
upstream/latest. The -s ours is the key trick; it records the
merge commit without deleting debian/* as part of the merge.
Now you can run gbp import-orig on the current upstream release
tarball, let it fix up the upstream branch to match the release
tarball by adding all the generated files, and then merge them into
the debian/master branch. You can do this now if you haven't
made any changes since the previous release. If you have made
changes, the easiest thing to do is to wait until you do another
release and then have that be the first true imported release.
I don't bother to go back and try to reconstruct history and put previous
releases in the upstream/latest branch. It's a lot of work and
doesn't seem worth the effort.
Most of the common Git commands will quickly enter your finger memory if you've used Git for very long. Here are a few less obvious ones that are worth knowing.
git commit --amendModify the last commit. I use this all the time, usually because I commit and then remember I need to fix one more thing, or made an error in the commit message. This is a form of history rewriting, so like all history rewriting should be done before you push.
git rebase -i <commit>Do an interactive rebase of your current branch against some other commit (usually the name of a branch, but sometimes it's useful to rebase on top of a specific commit). This means Git will find the common ancestor, and then interactively let you rewrite the commits as if they were applied directly on top of that commit. You'll be given an editor window into which you can enter commands that can do a lot of different things: combine commits (the most common thing I do), reorder them, change their commit messages, or modify them.
git add -p <file>Prompts you for each modified hunk of a file and asks you whether to add it to the index (stage it for the next commit). This is very useful if you have a dirty tree with work that should be in two or more separate patches, since you can select some modifications of a file to be in the next commit while leaving other ones uncommitted.
I also add the following aliases to my personal ~/.gitconfig:
[alias]
update = pull --rebase --stat
where = symbolic-ref HEAD
This lets me run git update to do a git pull --rebase and
quickly rebase local work on top of upstream, something I do constantly
throughout the day. git where shows the current branch extremely
quickly, without doing the slow checks for uncommitted files that
git status does.
I had a bunch of old Debian packaging repositories in Subversion that I had to convert to Git. There are a lot of different ways to do this, but here's the one that works for me. This technique is heavily based on a blog entry from Martin Krafft. I've done this conversion, so am not updating these instructions, but they probably haven't changed.
First, install git-svn if it's not already installed.
My packages all used an svn-buildpackage layout, which meant there was an
upstream directory under branches that had the imports of
the upstream source. I'm usually willing to throw away any other
branches, so I started with:
git svn clone --stdlayout --branches=branches/upstream --no-metadata \
-A /path/to/authors/file <svn-url>
This will create a new sub-directory named after the last component of the
<svn-url> with a new Git repository. This command treats all the upstream
tags and current as branches and ignores the other branches; if you want
to keep all the branches, svn mv the contents of
branches/upstream into branches first and then point git svn
at the whole branches directory.
Note the -A option; use that to specify the path to a mapping from
usernames in the Subversion commits to full names and e-mail addresses for
Git. Without this, your old commits will have ugly identification. See
the git-svn man page for the details on the format.
--no-metadata means that we're doing a one-way conversion and
git svn shouldn't keep the data required to support commits back to
the Subversion repository.
git svn will leave your Subversion trunk and branches as remote
branches in your repository and will create all of your tags as more
remote branches. The first step is to clean that up. Get a full branch
list with git branch -r. Then, create new local branches for the
remote branches that are really branches; at the least, you'll probably
want:
git branch upstream/latest branches/current
git branch -r -d branches/current
Note the removal of the remote tracking branch using git branch -r
-d; that's how you get rid of those. Next, convert each tag into a real
tag. The revision to which the tag should apply will be the revision
prior to the one recorded in the repository, so take advantage of Git's
^ shortcut for previous revision with commands like:
git tag debian/<version> tags/<version>^
git branch -r -d tags/<version>
Similarly for all the tags of the upstream imports, do:
git tag upstream/<version> branches/<version>^
git branch -r -d branches/<version>
Once you finish this, git branch -r should show no remote tracking
branches and only your regular upstream and master branches.
If you used the svn-buildpackage mergeWithUpstream feature to not
store the upstream source in your Git repository, the conversion has a few
more steps. You need to import the upstream source into the repository to
use Git the way that I outline in this document. You should be able to do
that by just importing the most recent upstream tarball from Debian with
gbp import-orig. This will commit and tag the upstream source and
merge it into your master branch, fleshing out your repository.
All of your old tags will still have only modified files, but normally
that's not a serious problem.
If you had to do gbp import-orig as described above, it will take
care of committing the pristine-tar metadata. Otherwise, grab the last
upstream orig.tar.gz (and make sure that it's named properly), put
it somewhere not in the current directory, and run:
pristine-tar commit /path/to/orig.tar.gz
to load it into the repository. If you want, you can also load older
upstream tar.gz files, but I never bother.
You can now delete some additional junk left over from the git svn
conversion process. Edit .git/config and remove the information
about the remote tracking branches for the Subversion repository, and
delete all of .git/svn.
If you're like me, you weren't very good about always tagging releases in
Subversion. You can now much more easily fix this by browsing the
repository using gitk and finding the release points for each
Debian package. When you do, paste the hash (shown nicely in gitk for
you) into a git tag command like:
git tag debian/<version> <hash>
You can add the -s flag to create signed tags.
When you select a revision in gitk, it copies the hash of that
revision into the X cut buffer for you, so you can paste it into a command
without selecting it.
Finally, do the same things as noted above under setting up a Debian
packaging repository (clone master to debian/master, move
upstream changes into a patch series, and so forth).
My old CVS repositories don't use branches or tags, so importing them was much easier and essentially follows the git-cvsimport man page.
First, install the git-cvs package if you haven't already.
Now, run git cvsimport. Since I'm doing one-way conversions and
plan on discarding all the old CVS information afterwards, I use options
like:
git cvsimport -d <cvs-repository> -o master -k -A <authors-file> \
-a -C <project> <module>
where <project> is the name of the new directory you want to create with the imported repository. As with Subversion, you'll need an authors file that maps the usernames stored in CVS commit messages to identities in the new Git repository.
If you're like me, you'll then want to go back through the history with
gitk and tag all the past releases with git tag.
When I did a Google search on migrating a bzr repository to Git, most of
the documentation said to use tailor. However, it didn't work; it got as
far as a changeset that renamed ChangeLog to CHANGES.old and
then bailed because it claimed the git add command on
CHANGES.old failed. When I ran exactly the command that it claimed
to have ran by hand, it worked fine, but I couldn't figure out how to
convince tailor of that.
Thankfully, there's a better alternative, although it's not
well-documented yet. Git and bzr both have support for the fast-import
repository serialization format, although the bzr support is somewhat
hidden away if you're using Debian. Here are the instructions that I used
back when I did this; now, it looks like bzr-fastimport is a
regular Debian package and installs a plugin, so I suspect that just
installing it and using the commands it provides will work.
First, get the bzr-fastimport source tree from Launchpad:
bzr branch lp:bzr-fastimport
Then, create a new directory and empty Git repository for your project:
mkdir <project>
cd <project>
git init
Now, you can import your old bzr repository by serializing it using the
bzr-fast-export.py script that comes with bzr-fastimport:
/path/to/bzr-fastimport/exporters/bzr-fast-export.py \
/path/to/bzr-repository | git fast-import
git checkout master
This will even preserve tags in your bzr repository.
| < Debian Packaging Tools | Russ Allbery > Technical Notes > Debian | Packaging Scripts for Debian > |