Ubuntu Open Week - Packaging 101 - Mon, Nov 27, 2006
06:00 dholbach Hello everybody! We'll move on to Packaging 101 now. ... 06:01 dholbach If you want to become part of the MOTU team, or help out with packaging in general, apart from knowing how to write or debug code, it's good to know the tools the distribution uses. 06:01 dholbach This session won't make you a packaging expert, but afterwards you will have heard of some things, which will make it easier to find your way around. There'll be a MOTU session on Wednesday, 15.00 UTC and Saturday on 16.00 UTC, which will cover what the MOTU team does in more detail. 06:02 dholbach Please run the following command, it will take some time to work in the background. 06:02 dholbach sudo apt-get install pbuilder; sudo pbuilder create 06:03 dholbach MOTUs are the "Master of the Universe", the team of maintainers that takes care of the package in Universe and Multiverse - that's where you start as a package maintainer. http://wiki.ubuntu.com/MOTU 06:03 dholbach Packaging - what does that involve? 06:03 dholbach * adding enough information to the Upstream source, to make it buildable on a minimal system. 06:03 dholbach * split the installed files up into separate packages depending on the target audience 06:04 dholbach - example everybody uses libgtk2.0-0, but not everybody has an interest in libgtk2.0-dev 06:05 dholbach * make the packages work out of the box 06:05 dholbach * add copyright information, nice description, documentation, etc. 06:05 dholbach Package maintenance involves far more than that, Jordan Mantha (Laser_away) will give a talk about that later today at 21.00 UTC and Thursday at 20.00 UTC. 06:05 dholbach First we'll install some tools we'll need for the session: 06:05 dholbach sudo apt-get install devscripts dpkg-dev build-essential 06:06 dholbach I also prepared an example package we'll use also: 06:06 dholbach mkdir 101; cd 101; wget http://daniel.holba.ch/temp/packaging101.tar.gz 06:07 dholbach I learned most from looking at existing packages and trying to fix them, so let's take a look at the libsexy package. It's the one in feisty, I broke a bit to illustrate a few common problems you'll face in the daily work as a package maintainer or Ubuntu developer. 06:07 dholbach If you untar the packaging101 tarball, you'll find the following files: 06:07 dholbach libsexy_0.1.10-1ubuntu1.diff.gz libsexy_0.1.10-1ubuntu1.dsc libsexy_0.1.10.orig.tar.gz 06:07 _tx Some things I write don't work well with ppc or x86 but work well on the x86_64 namely my system. 06:08 dholbach Together they make the source package. The .orig.tar.gz is called the 'original upstream tarball', the '.diff.gz' file is the changes we need to do to make the package buildable and working correctly in Ubuntu. The .dsc provides meta information like checksums, etc. 06:08 dholbach To unpack the source package run 06:08 dholbach dpkg-source -x libsexy_0.1.10-1ubuntu1.dsc 06:08 dholbach If you change into the directory and have a look into it, you'll find a debian/ directory, which contains the packaging information. 06:09 dholbach Let's try to build the package now. You can use dpkg-buildpackage, but running debuild is easier mostly. So to run a build, simply type 06:09 dholbach debuild 06:09 dholbach in the directory. 06:09 dholbach The friendly build script might now ask you to install some build dependencies, after you did that, it should be spitting out some interesting messages. 06:10 dholbach in my case the build is complaining (among other things) about the following: 06:11 dholbach sexy-tooltip.c: In function 'sexy_tooltip_new_with_label': 06:11 dholbach sexy-tooltip.c:143: warning: implicit declaration of function 'gtk_label_new' 06:11 dholbach sexy-tooltip.c:143: warning: assignment makes pointer from integer without a cast 06:11 dholbach sexy-tooltip.c:144: warning: implicit declaration of function 'gtk_label_set_markup' 06:11 dholbach sexy-tooltip.c:144: warning: implicit declaration of function 'GTK_LABEL' 06:11 dholbach It's a problem I added myself. :-) 06:11 dholbach We will need to patch the source to make it build again. 06:11 dholbach If you have a look at debian/rules you will see that it mentions /usr/share/cdbs/1/rules/simple-patchsys.mk - after working with packages for a while that it means you can use cdbs-edit-patch to add a patch. Martin Pitt (pitti) will give a separate session on how to patch packages on [patching source packages will be Tuesday 18:00 and Thursday 17:00 UTC]. 06:12 dholbach First we run fakeroot debian/rules clean to clean up our existing build again. 06:12 dholbach next we run cdbs-edit-patch 01-fix-gtk-breakage 06:13 dholbach as I said before, pitti will explain what happens then :-) 06:14 dholbach (Note: cdbs-edit-patch needs cdbs installed.) 06:14 dholbach Once we're presented with a subshell, in a directory like /tmp/cdbs-new-patch.R29983/libsexy-0.1.10.new , we edit libsexy/sexy-tooltip.c with our editor of choice 06:15 dholbach and add a missing #include <gtk/gtk.h> 06:15 dholbach just below the existing #include 06:15 dholbach now save the file, and hit Ctrl-D 06:16 dholbach ls debian/patches/ should show our shiny new patch 06:16 dholbach Now we run debuild again and see if we fixed the package. 06:17 dholbach It looks like our attempt succeeded. 06:18 dholbach ls ../*.deb will show us the packages we made 06:18 dholbach I'll answer a bunch of questions after the next step. 06:18 dholbach As I explained above, part of the packaging work is trying to make sure that a package builds in a minimal environment also. Why? 06:19 dholbach Because on our build daemons, a minimal system will be created for each package build to make sure it works in every, even a fresh, scenario. 06:19 dholbach We will now generate a new source package with the changes we made earlier. 06:19 dholbach fakeroot debian/rules clean 06:20 dholbach debuild -S -sa 06:20 dholbach should take us there. 06:20 dholbach cd ..; sudo pbuilder build libsexy_0.1.10-1ubuntu1.dsc should give the pbuilder some work and we can take a look at a few questions.
<xeruno> what's cdbs?
- cdbs is a collection of scripts that make packaging, especially writing debian/rules (a Makefile to automate the package build) a lot easier, by simplifiying common tasks.
<TLE> Will the script disregard temporary editor files when it does the patch ?
- I'm not sure what you mean with editor files. It's better to get rid of them. They will show up in the .diff.gz and in case of binary files make the build fail.
<DraxNS> is there a short manual on how to make simple packages? like to create package for krusader from source?
- I will later post some links to help with packaging, but I fear that packaging will always be a bit tough in the beginning.
<amarillion> Will debuild automagically add all patches in the debian/patches directory
- the rules in debian/rules (namely simple-patchsys.mk) takes care of that
<bettsp> It only succeeds if my name is Daniel Holback
I'm sorry, yes you're right. Either disregard the message or run debuild with -k<your keyid>
[bettsp] You forgot to run the S-expression I added in the next line to fix my misspelling of your name :)
- yes, I should have said "please add" (it works too when it can't sign the package. that's just a warning)
<proppy> you refer to 'our build daemon' is that kind of pbuilderd ?
the build daemons ( https://launchpad.net/+builds ) use sbuild, which is very similar
06:26 dholbach Ok, my pbuilder is 'finished' now - we'll get back to questions later again. 06:26 dholbach funnily enough the build fails with: 06:26 dholbach checking for PACKAGE... configure: error: Package requirements (pango >= 1.4.0, glib-2.0 >= 2.4.0, gtk+-2.0 >= 2.4.0, libxml-2.0) were not met: 06:26 dholbach No package 'gtk+-2.0' found 06:26 dholbach This is because libgtk2.0-dev is not installed in the minimal environment I was talking about, but how does pbuilder know what to install? 06:27 dholbach if you edit debian/control you will find the line Build-Depends 06:28 dholbach This line indicates the packages that need to be installed to build the package correctly. We'll just add libgtk2.0-dev 06:28 dholbach after that we'll run debuild -S again 06:28 dholbach please note that this time we don't need to run fakeroot debian/rules clean why? 06:29 dholbach because the build happened in that clean environment, not in the source dir - another good point about using pbuilder 06:29 dholbach let's run sudo pbuilder build libsexy_0.1.10-1ubuntu1.dsc again and see if we get it working now
<levander> What all does fakeroot debian/rules clean do?
- it runs the clean target of the upstream source (so whatever the upstream maintainer thought should be cleaned), plus dh_clean and rules you set up yourself
<TLE> if I edit the source with emacs, emacs creates a temp file, will the script disregard that when it makes the diff for the patch ?
- I'm not sure which file you're referring to. I'm emacs illiterate - sombody else please answer. :-). Emacs will create a sexy-tooltip.c~ backup file, and that will be included in the diff if you don't remove it likewise will gedit by default. if you mean backup files, yes, the "debhelper" system is smartyt enough to not ignore them, but get rid of them when you "clean" the package.
<daxelrod> How would we know from the error message mentioning 'gtk+-2.0' that the package we need is 'libgtk2.0-dev'? Is there a convention for figuring this out?
excellent question. One answer to that is: experience with packaging. But there are also some indicators of where to look: In the configure script it failed, because it tried to look up in /usr/lib/pkgconfig so if the build fails for you in pbuilder, but works if you run on your system, you can easily check that directory and find out, where and why it went wrong apt-cache search <...> or apt-file search <...> can help also. check configure.ac is another solution
06:35 dholbach The pbuilder build worked for me right now, but there are still some things that are weird: 06:36 dholbach If you run dpkg -c /var/cache/pbuilder/result/libsexy-doc_*.deb (which displays the contents of the package), you will find it's nearly empty 06:36 dholbach it merely contains a couple of changelogs and copyright notices - what went wrong? 06:37 dholbach If we look into the debian/ directory, you will see that it contains a couple of *.install files 06:37 dholbach these files mention which directories and files are supposed to be installed into which package 06:38 dholbach debian/libsexy-doc.install is empty 06:39 dholbach if you run debuild (to build the package again) and dh_install --list-missing afterwards, it will present you with a list of files that were not installed into any package 06:40 dholbach once we add debian/tmp/usr/share/gtk-doc/html to debian/libsexy-doc.install it will contain something 06:40 dholbach now we can run debuild binary to not have to go through the whole procedure again, but only rebuild the packages 06:41 dholbach You will find that if you run dpkg -c libsexy-doc*.deb again, it will contain the documentation 06:42 dholbach I introduced some more bugs into the package, but given that we only have 19 minutes left, we should probably move on to some more questions.
<proppy> debian/libsexy-doc.install doesn't exist
- hum, it existed for me - anyway, it was supposed to be an empty file.
<nmsa> install pbuilder is still on, slow connection, will it install a new env. will get all packages from net? is doing validation on retrieved packages ...
- it's not installing all packages, but only a very minimal set, needed to build packages
<cjwatson> which packages are considered by debootstrap?
- build-essential will be contained in any case.
<Florob> Are *.install files cdbs specific?
- no, they're not. They're scanned by dh_install.
<amarillion> which is the preferred format for documentation? Should you provide docs in a certain format if they aren't provided upstream?
- manpages are a good example of documentation that sometimes does not exist. It's always good to push it back to the upstream developers afterwards again. I don't have a link on manpages handy, but I could find you one. I have a template I always use to get started on them.
<rmjb> is the *.install file method the correct way for one source package to have multiple binary targets? like a lib, lib-dev, -doc and so on?
some people use *.dirs files or run dh_install to debian/<binary package name> on their own, I personally prefer the .install file method
<xhaker> how about policy on packages. I read executables on /usr/bin must have manpages.
- manpages are not a strict must have, but they come in very handy and users are gratefule for manpages
<somerville32> Is it possible to have pbuilder setup to work in multiple releases concurrently?
- there's a page on the MOTU wiki about that - if nobody finds it out in the next minutes, I can try after the talk
<xhaker> can i build for any version deboostrap supports?
you can either set up a chroot for yourself (I think that's what most developers do), because that way you can test the resulting binary packages yourself, or you can have multiple differently configured pbuilders. https://wiki.ubuntu.com/PbuilderHowto?highlight=%28pbuilder%29
[sol] Should all packages install to /usr? What's the use of /usr/local?
/usr/local is for software you install yourself (like if you use ./configure && make && sudo make install)
<xhaker> pbuilder extracts the minimal system and then downloads additional dependencies.. does it cache them? for how long? where?
- yes, it does in /var/cache/pbuilder/aptcache/. I think it stores the packages until they're superseded by a newer version
06:52 dholbach If you run into any other packaging related trouble and don't know where to look in the documentation, your first check could be similar packages, then ask the people on #ubuntu-motu or on firstname.lastname@example.org
[proppy] can i chroot inside a pbuilder environment to debug a packaging failure ?
- yes, you can sudo pbuilder login can help with that
[jonh_wendell] do we have to do a clean before every debuild? even if we change just 1 char in some file?
- no you don't - you can run debuild binary to benefit from a previous build and just restart from wherever the build failed. debuild cleans for you unless you use the -nc option