1 [15:59] <@mvo> welcome everyone - and sorry that I could not make it this morning
   2 [16:00] <@mvo> and a special thanks to dholbach for doing a Q&A
   3 [16:00] <dholbach> mvo: no worries :)
   4 [16:01]  * iulian m00s
   5 [16:01] <@mvo> I'm going to talk about how to make package upgrade cleanly
   6 [16:01] <@mvo> I will give a overview on the common problems I see when looking at upgrade failure logs
   7 [16:02] <toabctl> hi
   8 [16:02] <@mvo> on the bright side: dpkg/apt tend to do pretty well - but if stuff goes wrong recoverying is not a great user experience
   9 [16:02] <@mvo> so we should try to make sure that stuff does not get wrong :)
  10 [16:02] <keffie_jayx> o/
  11 [16:02] <@mvo> (and improve our tools to deal better with failures)
  12 [16:03] <@mvo> if you have any questions or special topics that you would like to talk about, just ask here (or in #ubuntu-classroom-chat)
  13 [16:04] <Rinia> hello everybody
  14 [16:04] <@mvo> I prepared some notes, if there are no questions/comments I will just paste a bit into the channel
  15 [16:04] <keffie_jayx> :D
  16 [16:05] <@mvo> = How to make packages upgrade cleanly =
  17 [16:05] <@mvo> Normal deb packages usually upgrade just fine, dpkg takes care of all the heavy lifting and shuffling of files. The most common source of problem on upgrades are:
  18 [16:05] <@mvo> • file overwrite problems (e.g. when a package was split)
  19 [16:05] <@mvo> • maintainer script problems
  20 [16:05] <@mvo> A general good advise is to keep the amount of code in the maintainer script small and clean. Shell is a language that can be tricky and a failure in the script will make the upgrade abort and gives the user a bad experience.
  21 [16:07] <@mvo> the most common source of overwrite problems are caused by package splits
  22 [16:08] <dholbach> mvo: why are package split? does that happen very often? is ther an example? :-)
  23 [16:08] <@mvo> but they can also happen if two package just happen to install the same file into the same location. if that can not be avoided (e.g. by changeing the program to look into a different location) the package can not be installed in parallel and should conflict
  24 [16:08] <@mvo> for a package split we use a different technique
  25 [16:08] <@mvo> dholbach: thanks, I give a example now :)
  26 [16:08] <@mvo> == Package splits ==
  27 [16:08] <@mvo> If a package foo gets split into two packages like foo and foo-common there needs to be a "Replaces" line in the "foo-common" package. The reason is that on upgrade, the new package "foo-common" may get installed when the old "foo" is still installed. Because the foo-common has files that belong to the old foo dpkg will complain if the proper "Replaces" line is missing.
  28 [16:08] <dholbach> woohoo
  29 [16:09] <@mvo> this is pretty common for projects that start out as e.g. update-manager and then later grow new frontends (like udpate-manager-text). then the package typically gets split into update-manager-common, update-manager-gtk, update-manager-text etc
  30 [16:10] <@mvo> Example:
  31 [16:10] <@mvo>  foo 1.0 is a big package
  32 [16:10] <@mvo>  foo 1.1 is split into foo and foo-common
  33 [16:10] <@mvo> foo-common now needs a line:
  34 [16:10] <@mvo> Replaces: foo (<< 1.1)
  35 [16:10] <@mvo> to ensure that if the new foo-common is unpacked before the foo package is upgraded everything still works as expected.
  36 [16:10] <neurobuntu> how does (<< 1.1) differ from (< 1.1) ?
  37 [16:13] <@mvo> neurobuntu: the "<" is a older (and now deprecated way) of expressing the same. it used to mean earlier-or-equal
  38 [16:13] <@mvo> neurobuntu: but its a bit confusing, this is why "<<" and "<=" etc are now used
  39 [16:13] <@mvo> The most common case is something like the following for "foo":
  40 [16:13] <@mvo> Depends: foo-data (= ${source:Version})
  41 [16:13] <@mvo> This means that dpkg needs to unpack foo-data first before it unpacks the new foo. So without the Replaces line above it will not work.
  42 [16:13] <neurobuntu> ok thanks
  43 [16:14] <@mvo> cheers
  44 [16:15] <@mvo> any questions on this particular bit (splitting/overwrite problems)?
  45 [16:15] <@mvo> I should note that we have a system that can check for file conflicts in the archive, but its currently in the need for some bugfixing
  46 [16:16] <dholbach> what is a good way to test if my packages upgrade well? is  dpkg -i *.deb  good enough?
  47 [16:17] <@mvo> dholbach: yes, that is a good way. its a good way of testing before a upload to make sure the current version is installed
  48 [16:17] <@mvo> and then install the new package like this
  49 [16:17] <@mvo> the good news is that for the common case (package just got some new files/some old files got removed) dpkg will do everything
  50 [16:17] <@mvo> and we do not have to worry
  51 [16:18] <@mvo> testing is always good, its pretty anoying if a forgoten "fi" in a maintainer script causes problems later :)
  52 [16:18] <@mvo> which brings me to the next topic:
  53 [16:18] <@mvo> == Maintainer script problems ==
  54 [16:19] <@mvo> We have two classes of issues here
  55 [16:19] <@mvo> one are "just" bugs
  56 [16:19] <@mvo> a forgoten "fi"
  57 [16:19] <@mvo> the use of bash syntax (like "==")
  58 [16:20] <@mvo> instead of POSIX (which expects "=")
  59 [16:20] <@mvo> etc
  60 [16:20] <@mvo> (we use dash as /bin/sh so we need to be careful with that, some other distros do not have this problem)
  61 [16:22] <@mvo> the other class of problems is that the system is in a flux during a complex upgrade (like a upgrade from 6.06 -> 8.04 :)
  62 [16:22] <dholbach> mvo: what are maintainer scripts? is there a good way to test them? :)
  63 [16:22] <@mvo> dholbach: good point, sorry for using jargon :)
  64 [16:22] <dholbach> (I hope everybody else feels free to pester mvo with questions too and it's not just me. :-))
  65 [16:23] <@mvo> maintainer scripts are the scripts known as: "prerm, postrm, preinst, postinst"
  66 [16:23] <@mvo> they are run as part of the package installation
  67 [16:23] <toabctl> mvo, are they makefiles? or just shellscripts?
  68 [16:23] <@mvo> they can contain any shell code and are used for stuff like "ldconfig"
  69 [16:23] <@mvo> toabctl: usually shell scripts, some contain perl too (notable stuff that uses debhelper)
  70 [16:24] <neurobuntu> when upgrading a package, are the prerm and postrm run (almost like the old package is removed before the new is installed)?
  71 [16:25] <@mvo> neurobuntu: yes they are. I have a example for this in a bit, it can become relatively complex
  72 [16:25] <@mvo> a maintaner script may alter the system state in any way, this is why we do not support rollbacks or downgrades in a general way. because it may simply be not possible to undo (or know) what a maintainer script did
  73 [16:25] <@mvo> (but that is just a side note :)
  74 [16:26] <dholbach> mvo: is there any way to test them or do I just re-install the package to see if it worked?
  75 [16:26] <@mvo> dholbach: they can be tested by running them with the approprate arguments (simulating the way they are called by dpkg). this can be very helpful, especially when run with "sh -x" to see line by line what is actually happening
  76 [16:27] <@mvo> I will give a example for this
  77 [16:27] <@mvo> Problems here are usually caused by the fact that the system is in a pretty unusual state during a upgrade. Some packages are half-installed, some are not available etc. Maintainer scripts should not make too many assumptions about the system because during a upgrade those may well be wrong. A script should gracefully whenever possible.
  78 [16:27] <@mvo> Its useful to know how maintainer scripts get called:
  79 [16:27] <@mvo> http://debid.vlsm.org/share/Debian-Doc/debian-policy/ch-maintainerscripts.html has all the details in Section "6.5".
  80 [16:27] <@mvo> The way apt works is that it splits the changes into dpkg runs that get called with "--unpack" and then "--configure". On a typical upgrade, this is:
  81 [16:27] <@mvo> Unpack:
  82 [16:27] <@mvo> •  old-prerm upgrade new-version
  83 [16:27] <@mvo> • new-preinst upgrade old-version
  84 [16:27] <@mvo> • files from the new package are now in place
  85 [16:27] <@mvo> • old-postrm upgrade new-version
  86 [16:27] <@mvo> Configure:
  87 [16:27] <@mvo> • postinst configure most-recently-configured-version
  88 [16:28] <@mvo> neurobuntu: I hope the table below anser the earlier question
  89 [16:28] <neurobuntu> yes it does
  90 [16:28] <@mvo> so quite a bit of script that are run :)
  91 [16:28] <@mvo> and if something goes wrong, it gets even more complext, if e.g. the old-prerm fails, the new-prm is called
  92 [16:28] <neurobuntu> what is the logic behing running the new preinst and copying the new files before running the old-postrm
  93 [16:28] <@mvo> this can be useful when recovering from a mistake in a maitainer script
  94 [16:32] <@mvo> neurobuntu: a good question. I currently do not have a good example, generally the postrm is there to do final cleanup
  95 [16:33] <@mvo> useful for e.g. update-menu or something similar that updates the debian menu and needs the new menu files to get reliable data
  96 [16:34] <@mvo> (not the best example in the world I guess :)
  97 [16:34] <@mvo> The time between unpack and configure can be pretty long (on release upgrades) because apt tries to put as much work into a single dpkg run as possible (to speed things up, dpkg invocation is slow).
  98 [16:34] <Rail> FYI, detailed information about maintainer scripts can be found at http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
  99 [16:36] <@mvo> Rail: thanks! especially section 6.6 is interessting here
 100 [16:36] <@mvo> This can be tricky if critical functionality relies on the work that is done in configure. A example is python when the symlinks get removed in preinst and re-created in postinst. This means that the package python functionality is not available during a release upgrade (people can not play certain gnome-games ;) Its also common for daemons that stop in preinst and start again in postinst). Often this is unavoidable, but it should be kept
 101 [16:36] <@mvo>  in mind when thinking about the package maintainer scripts.
 102 [16:37] <@mvo> generally I would recommend writing small and simple maintainer scripts if at all possible, the less complexity the less likely are bugs :)
 103 [16:38] <dholbach> mvo: is there a way to avoid maintainer scripts easily?
 104 [16:38] <@mvo> debhelper takes care of a lot of this
 105 [16:38] <@mvo> e.g. the ldconfig scripts are all auto generated
 106 [16:39] <@mvo> or the python magic
 107 [16:39] <@mvo> or init stuff
 108 [16:39] <dholbach> or is there a way around having to write (really good) shell scripts? :)
 109 [16:39] <@mvo> and thats good, auto-generated code is nice because it needs to be fixed only in one place if a errorr is found
 110 [16:40] <@mvo> another new development is the increased use of triggers
 111 [16:40] <neurobuntu> triggers?
 112 [16:40] <@mvo> triggers are a way to express a interessted in a certain directory for example
 113 [16:41] <@mvo> its a relatively new concept
 114 [16:41] <neurobuntu> ok i'll be sure to read up on them
 115 [16:41] <@mvo> a package like e.g. "man-db" tells dpkg that its interessed in changes in /usr/share/man
 116 [16:41] <@mvo> and everytime a file is installed there, dpkg will notice that and mark the trigger as pending
 117 [16:42] <@mvo> then dpkg will run the pending triggers at the end
 118 [16:42] <@mvo> (or when needed)
 119 [16:42] <@mvo> so it runs the man-db.postinst with "triggered" as argument
 120 [16:42] <@mvo> and the man-db package then processes the new man-pages and builds a db from them
 121 [16:43] <neurobuntu> so man-db says its interested in /usr/share/man I install package foo which installs /usr/shar/man/foo.man and then dpkg runs the man-db.postinst?
 122 [16:43] <@mvo> this is a very useful concept because it means that mandb can be run when everything else got installed (needs to be run only once)
 123 [16:43] <@mvo> neurobuntu: yes
 124 [16:43] <neurobuntu> wow thats really slick!
 125 [16:44] <@mvo> neurobuntu: re "configure" (from #ubuntu-classroom-chat")
 126 [16:45] <@mvo> the configure here is something different, it just shares the same name with the autotools configure
 127 [16:45] <@mvo> dpkg roughtly splits a package into "unpacked" and "configured"
 128 [16:45] <@mvo> unpack means the files are in place
 129 [16:45] <@mvo> configured means that the package is actually usable
 130 [16:45] <@mvo> the switches are "dpkg --unpack $long_list_of_packages"
 131 [16:46] <@mvo> and dpkg --configure $long_list_of_packages"
 132 [16:46] <@mvo> in a way, this is a performance optimization to avoid having to call dpkg too often
 133 [16:47] <@mvo> each time dpkg is run it builds a (internal) database of all installed files and the relations
 134 [16:47] <@mvo> that can be time consuming
 135 [16:47] <@mvo> so the less times dpkg needs to be called, the better
 136 [16:47] <neurobuntu> ok that makes sense...  forgive me if you answered this earlier but after the files are in place (dpkg --unpack),  preinst gets run?
 137 [16:48] <neurobuntu> or are all the preinst scripts run before the unpack happens and then all the postinst scripts are run?
 138 [16:49] <@mvo> neurobuntu: the preinst script gets run before the new files are in place. this can cause bugs, we had a package that tried to start a daemon in preinst, but the daemon was not on disk yet, so the package was not installable (or rather, installed, but always gave a error message)
 139 [16:49] <@mvo> ia: thanks for your question - this is a tricky one
 140 [16:50] <neurobuntu> ok thanks
 141 [16:50] <neurobuntu> this is making sense
 142 [16:50] <@mvo> ia: the python problem we had were hard to reproduce because the system is in so much of a flux during a upgrade. its tricky to get this right. in my experience jaunty-final did pretty well and the cases of failure were rare
 143 [16:50] <@mvo> but its a good example why complexity in maintainer scripts is generally something that may not work so well
 144 [16:51] <@mvo> because its hard to predict the environment in which the script is run, that may depend on the installed packages on the system that is upgraded etc
 145 [16:51] <@mvo> very hard to reproduce
 146 [16:52] <@mvo> what we can  do now is upgrade-testing on real HW
 147 [16:52] <@mvo> there is a update-manager --sandbox switch
 148 [16:52] <@mvo> for jaunty->karmic upgrades
 149 [16:53] <@mvo> that will perform a upgrade into a writeable filesystem snapshot in /tmp - but it keeps the real system unchanged (or rather, restores it on the next reboot)
 150 [16:53] <@mvo> this is quite useful to test real world upgrades
 151 [16:53] <@mvo> chroots (e.g. pbuilder login) or other tests are useful too
 152 [16:54] <@mvo> we plan to have a auto-upgrade-tester package for karmic that can be used to test system upgrades of any given input packages, lets see how this goes :)
 153 [16:54] <@mvo> I guess the summary is: the best way for us (as a distro) to ensure clean upgrades is lots of testing (on a wide range of setups)
 154 [16:55] <@mvo> for the maintainer: write good, clean, small maintainer scripts :)
 155 [16:56] <ia> mvo: ok, thanks :-)
 156 [16:56] <@mvo> (for the specific python problem we plan to change the way the python packages are done to invlove a little bit less magic :)
 157 [16:56] <@mvo> for karmic
 158 [16:57] <neurobuntu> mvo thanks this has been really helpful
 159 [16:58] <@mvo> thanks for comming and listening!
 160 [16:58] <@mvo> and asking good questions of course :)
 161 [17:00] <@mvo> ia: the auto-upgrade-tester is a project that can perform a full upgrade in a virtual environment
 162 [17:00] <@mvo> it can use chroot, kvm and ec2 (the later is still a bit experimental)
 163 [17:00] <@mvo> kvm is really the most useful one
 164 [17:01] <@mvo> you give it a base image that can be any kvm image (e..g build via ubuntu-vm-builder) and then it will perform a release upgrade from the installed version of ubuntu to the next using update-manager in a non-interactive session
 165 [17:01] <@mvo> that takes ~1h on my realatively slow system
 166 [17:02] <@mvo> after that I can see if the given image upgrades cleanly or if there are issues
 167 [17:02] <@mvo> its very useful as a QA tool
 168 [17:02] <@mvo> its currently only available via bzr, but we want to package it
 169 [17:02] <@mvo> so that everyone can test and play with it
 170 [17:03] <@mvo> the base-image is not touched, so when a failure happen a fix can be tested as well with it
 171 [17:04] <@mvo> e.g. when python-foo does not upgrade cleanly (but only in a complex dist-upgrade), a fix can be uploaded to a test PPA and the upgrade can be pointed to that PPA to test if the fix is really good
 172 [17:04] <@mvo> does that answer the question? or should I expand on a particular are more?
 173 [17:04] <@mvo> area
 174 [17:05] <ia> mvo: yep, thanks for answer :-)
 175 [17:05] <@mvo> :)
 176 [17:10] <@mvo> if there are no further question I say thanks for comming and have a good day
 177 [17:10] <Rail> mvo: thank you
 178 [17:10] <Rail> and others for questions


Packaging/Training/Logs/2009-06-04-2 (last edited 2009-06-04 17:04:04 by i59F71954)