Blogs

Packaging Drush and Dependencies for Debian

Lately I have been trying to avoid non packaged software being installed on production servers. The main reason for this is to make it easier to apply updates. It also makes it easier to deploy new servers with meta packages when everything is pre packaged.

One tool which I am using a lot on production servers is Drupal's command line tool - drush. Drush is awesome it makes managing drupal sites so much easier, especially when it comes to applying updates. Drush is packaged for Debian testing, unstable and lenny backports by Antoine Beaupré (aka anarcat) and will be available in universe for ubuntu lucid. Drush depends on PEAR's Console_Table module and includes some code which automagically installs the dependency from PEAR CVS. The Debianised package includes the PEAR class in the package, which is handy, but if you are building your own debs from CVS or the nightly tarballs, the dependency isn't included. The auto installer only works if it can write to /path/to/drush/includes, which in these cases means calling drush as root, otherwise it spews a few errors about not being able to write the file then dies.

A more packaging friendly approach would be to build a debian package for PEAR Console_Table and have that as a dependency of the drush package in Debian. The problem with this approach is that drush currently only looks in /path/to/drush/includes for the PEAR class. I have submitted a patch which first checks if Table_Console has been installed via the PEAR installer (or other package management tool). Combine this with the Debian source package I have created for Table_Console (see the file attached at the bottom of the post), you can have a modular and apt managed instance of drush, without having to duplicate code.

I have discussed this approach with anarcat, he is supportive and hopefully it will be the approach adopted for drush 3.0.

Update The drush patch has been committed and should be included in 3.0alpha2.

Upcoming Book Reviews

Packt Publishing seem to have liked my review of Drupal 6 Javascript and jQuery, so much so they have asked me to review another title. On my return from linux.conf.au and Drupal South in New Zealand, a copy of the second edition of AJAX and PHP was waiting for me at the post office. I'll be reading and reviewing the book during February.

I will cover LCA and Drupal South in other blog posts once I have some time to sit down and reflect on the events. For now I will just gloat about winning a spot prize at Drupal South. I walked away with Emma Jane Hogbin and Konstantin Käfer's book, Front End Drupal. I've wanted to buy this title for a while, but shipping from the US made it a bit too pricey even with the strong Australian Dollar. I hope to start reading it in a few weeks, with a review to follow shortly after.

Got a book for me to review? I only read books in dead tree format as I mostly read when I want to get away from the screen. Feel free to contact me to discuss it further.

Ads don't Belong on your Business Site

Back in the late 90s there was a range of free website hosting options - geocities, angelfire and tripod are the big 3 I remember straight off the top of my head. The business model was pretty simple, you got a free site, albeit with a pretty crappy url, and the host got to inject ads into the page. The first "site" ever I ever built was hosted by tripod and is still up, I have forgotten the login details so it hasn't been updated for 11 years.

Of the 3 stars of this business model, angelfire and tripod are still offering an ad supported version along with ad free, fee for service upgrades, but geocities is dead. Today, the business model has evolved, you can get a free but ad supported blogs (see Blogger.com or wordpress.com), email services (see gMail, Yahoo or Microsoft) or project hosting (see sourceforge, xp-dev or CodePlex) along with many other online services. For personal stuff I think this is fine, and the same goes for small not for profit organisations. On the other hand if you run a business and want to appear professional, profitable and "up with technology", then you don't want your email address to be [email protected], or your website to be acme-inc.freehostingco.com. It could be worse, you could be using an email address or hosting supplied by your ISP such as [email protected] and myisp.com.au/acme-inc. In the case of email, you can use google apps for domains and still look professional.

It is different if you are solely providing free (as in beer) content, such as video, news or a professional blog. This is a clear business model, fund free content via advertising, it has been used by print news, radio and television for decades. I also think it is fine for community based free/open source software projects to use it to get some additional revenue. It is different if you are a profit making business.

Not only does running ads on your site look unprofessional, you could be promoting the competition. Google targets their ads based on the content of the page. For example if you are a small shop and you have a page listing the types of products you offer, Google is likely to serve up ads on that page for those products. Do you really want an ad from a competitor showing up with "Cheap [item], free next day delivery"? How many high volume paying customers will you lose for that extra few dollars a month in Ad Sense revenue?

I find it more shocking on large corporate sites. Yes, they attract a lot of eye balls, but I pay my phone company enough money and they make large enough profits, that I shouldn't have to be subjected to ads on their corporate home page. It makes them look cheap. The same goes for smaller businesses.

In the case of a business blog, it should be part of your business website. If people find your blog and they like what they see, they are likely to click around your site to find out more about you. If you have your blog on blogger they are only likely to find other blog posts, and if you have a link to your business website, they will probably stop clicking once they hit your business site as it is completely different from your blog. If on the other hand it's all nicely integrated, your readers are able to move from your blog posts to your business content seamlessly - and so are more likely to become a customer, rather than another bounced visitor.

What are your options? Many hosts offer one click installers for setting up drupal, wordpress or other content management systems. With a bit of help from an online tutorial or 2 you should be able to get drupal up and running with a basic site and a theme from contrib. Sure it will look a bit cheap, but no worse than something on blogger. If you were to host it with dreamhost it is going to cost you around 120USD/130AUD for one year. Even if you have to pay someone to help you setup your CMS site, it will probably cost you less than 500USD in the first year for a basic setup. A basic setup will allow your business to project a professional image to the world. Add a professionally designed custom theme and site build for another 1850USD/2000AUD or so and you are set for a few years. Of course you will spend more if you want some to help with designing your information architecture, help with SEO, produce or proof content or suggest images etc. The investment is likely to pay for itself over that time in increased sales.

Updating all of your Drupal Sites at Once - aka Lazy Person's Aegir

Aegir is an excellent way to manage multi site drupal instances, but sometimes it can be a bit too heavy. For example if you have a handful of sites, it can be overkill to deploy aegir. If there is an urgent security fix and you have a lot of sites (I am talking 100s if not 1000s) to patch, waiting for aegir to migrate and verify all of your sites can be a little too slow.

For these situations I have a little script which I use to do the heavy lifting. I keep in ~/bin/update-all-sites and it has a single purpose, to update all of my drupal instances with a single command. Just like aegir, my script leverages drush, but unlike aegir there is no parachute, so if something breaks during the upgrade you get to keep all of the pieces. If you use this script, I would recommend always backing up all of your databases first - just in case.

I keep my "platforms" in svn, so before running the script I run a svn switch or svn update depending on how major the update is. If you are using git or bzr, you would do something similar first. If you aren't using any form of version control - I feel sorry for your clients.

So here is the code, it should be pretty self explanatory - if not ask questions via the comments.

#!/bin/sh # Update all drupal sites at once using drush - aka lazy person's aegir # # Written by Dave Hall # Copyright (c) 2009 Dave Hall Consulting http://davehall.com.au # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # Alternatively you may use and/or distribute it under the terms # of the CC-BY-SA license http://creativecommons.org/licenses/by-sa/3.0/ # Change this to point to your instance of drush isn't in your path DRUSH_CMD="drush" if [ $# != 1 ]; then     SCRIPT="`basename $0`"     echo "Usage: $SCRIPT path-to-drupal-install"     exit 1; fi SITES_PATH="$1" PWD=$(pwd) cd "$SITES_PATH/sites"; for site in `find ./ -maxdepth 1 -type d | cut -d/ -f2 | egrep -v '(.git|.bzr|.svn|all|^$)'`; do     if [ -f "${site}/settings.php" ]; then         echo updating $site         $DRUSH_CMD updatedb -y -l $site     fi done # Lets go back to where we started cd "$PWD"

OK, so my script isn't any where as awesome as aegir, but if you are lazy (or in a hurry) it can come in handy. Most of the time you will probably still want to use aegir.

Notes:

Make sure you make the script executable (hint run chmod +x /path/to/update-all-sites)

If you don't have drush in your path, I would recommend you add it, but if you can't then change DRUSH_CMD="drush" to point to your instance of drush - such as DRUSH_CMD="/opt/drush/drush".

Thanks to Peter Lieverdink (aka cafuego) for suggesting the improved regex.

DRBD on Ubuntu Karmic

Ubuntu 9.10 (aka karmic koala) has a frustrating packaging bug. Even though the stock server kernel includes the DRBD module, the drbd8-utils package depends on drbd8-source. drbd8-source uses DKMS to build the drbd module to match the installed kernel/s. As I stated in the bug report (lp:474660), "really don't like having build-essential installed on production net facing servers, and where possible any productions servers".

As side from personal opinion on whether the module should be bundled or not, the fact is that is bundled and so there is no need for the dependency on drbd8-source. As a work around I have added a meta package to provide drbd8-source, so I don't need to install build-essential and build the module every time a new kernel is installed.

After a quick test it is working well. Here is the

DEBIAN/control

file I used to make it all happen.

Package: dhc-drbd8-source-hack
Version: 0.1
Section: meta
Priority: optional
Architecture: all
Provides: drbd8-source
Installed-Size:
Maintainer: Dave Hall <[email protected]>
Description: Package to hack around drbd8-source dependency for drbd8-utils

If you are unsure how to use the control file above, see my recent blog post on building meta packages for ubuntu and debian.

Setting up a private package repository is outside the scope of this post. If you want to set one up, I would recommend Sander Marechal's slightly dated howto - Setting up and managing an APT repository with reprepro. With a few changes I found it worked well.

If you have your own repository running you can simply run

sudo apt-get install dhc-drbd8-source-hack drbd8-utils

, if you don't you can run the following commands

sudo dpkg -i /path/to/dhc-drbd8-source-hack*.deb && sudo apt-get install drbd8-utils

. Either way you should now have drbd8-utils installed on ubuntu karmic without having to install the redundant drbd8-source package.

To take it a step further you could build a meta package to install both drbd8 packages and allow you to have a potentially smoother upgrade to lucid. The meta package would contain the following line

Depends: dhc-drbd8-source-hack drbd8-utils

This is similar to what I now have in my HA server meta package.

<?php print t('hello world'); ?>

My blog is now syndicated on Planet Drupal. I am very excited about this - thanks Simon.

For the last 8 years or so I have been running my own IT consulting business, focusing on free/open source software and web application development. My clients have range from micro businesses up to well known geek brands like SGI. Until recently I lead the phpGroupWare project.

My Drupal profile doesn't really give much of a hint about my involvement with Drupal. My biggest regret is not signing up for a d.o account sooner. I forget when I started using Drupal 4.7, but I liked it straight away. It was the first CMS which worked the way I thought a CMS should work.

Over time I have learned how to get Drupal to do what I want it to do. Due to the massive range of contrib modules I haven't got my hands very dirty hacking on Drupal - yet.

This year I have been involved in a major Drupal project which involves hosting around 2100 sites. Aegir has made a lot of this painless, especially with our 3,000 line install profile. Over the Christmas period I hope to find the time to blog about the setup, parts of it are pretty crazy.

I'll get around to upgrading my site to Drupal 6 one of these days when I get some time, that should coincide with a visual and content refresh. Feel free to check out some of my older Drupal related posts.

Drupal 6 JavaScript and jQuery

I have just finished reading Matt Butcher's latest book, Drupal 6 JavaScript and jQuery, published by Packt Publishing - ISBN 978-1-847196-16-3. It is a good read. It is one of those books that arrived at the right time and left me inspired.

I have always leaned towards Yahoo's YUI toolkit when I need an Ajax framework, while the rest of the time I just bash out a bit of JS to get the job done. The more I use Drupal, the more I have been wanting to find time to get into jQuery. This book has got me motivated to play with jQuery - especially in combination with Drupal.

The book is logically structured and flows well from chapter to chapter. I find Matt's writing style easy to read, he even brought a smile to my face a few times. Matt assumes a basic knowledge of JS and Drupal, but he also provides links so the reader is able to get additional information if their knowledge is lacking. However, a couple of times Matt seemed to switch quite abruptly from assuming a good level of knowledge on a particular topic to explaining what seemed to me to be basic or simple concepts in great detail.

In the first chapter, entitled Drupal and JavaScript, Matt covers the basics of Drupal, its relationship with JavaScript and recommends some essential items for any serious Drupal developer's toolbox. This chapter provides a nice introduction of what is to come in the rest of the book and allows the reader to become acquainted with Matt's style.

Working with JavaScript in Drupal covers the basics of the Drupal coding standards and why sticking to the standard is important. It then moves onto a quick overview of Drupal's theme engine, PHPTemplate, and integrating JS with Drupal themes. I felt that the development practices part of this chapter could have been expanded a bit more and turned into its own chapter. Understanding the basics of theming is critical for being able to follow the rest of the book, but again I think this half of the chapter could have been developed into a separate chapter. Regardless of how the chapter was arranged, the content is well written and provides solid and practical examples.

In jQuery: Do More with Drupal, Matt gives a detailed overview of jQuery and how it is used in Drupal. Although the code sample has limited real world usefulness, it provides the reader with a very clear idea of the power of jQuery and how easy it is to use with Drupal. By the end of this chapter I was left feeling like I wanted to get my hands dirty with jQuery, unfortunately it was after 1am and I had to work the next day.

In Chapter 4, we move onto Drupal's Behaviors, which is covered in great detail. Behaviors are a key part of Drupal's JS implementation and essentially provide an events based hooks system in JavaScript. Once again Matt spends a lot of time explaining this feature, how it works, how to use it and where to learn more. Matt's description of this feature had me thinking "OMG, Drupal behaviours are awesome" throughout the chapter.

Lost in Translations, is the name of a good movie starring Scarlett Johansson and Bill Murray, which I enjoyed watching a few years ago, oh and is also the fifth chapter of the book. I suspect that I am like many English speaking Drupal developers in that I use the basics of the Drupal translation engine, but pay very little attention to how it works as my target audience is English speaking like me. Not only does Matt explain how Drupal's translation system works in both PHP and JavaScript, he makes it clear why all Drupal developers should understand and use the system - regardless of their native/target language/s.

The JavaScript Themeing chapter was a bit of a surprise for me. I was expecting Drupal to have a JS equivalent to PHPTemplate and for this chapter to outline it and provide some code samples. Instead I learn that Drupal has a very simple, and easy to use, JS themeing system. Matt spends some time discussing best practice for themeing content in JS and goes on to provide the code for his own simple yet powerful jQuery based themeing engine for Drupal.

In AJAX and Drupal Web Services, we learn about JSON, XML and XHR in the context of Drupal. Once again Matt demonstrates the ease of using Drupal and jQuery for quickly building powerful functionality.

Chapter 8 is entitled, Building a Module, and covers the basics of building a JS enabled module for Drupal. Matt also discusses when JS belongs in a theme and when it should be part of a module. The cross promotion of his other book Learning Drupal 6 Module Development ramps up a couple of notches in this chapter. I found the plugs a bit irritating (especially as I own a copy of the book), but overall the chapter is loaded with useful information.

The final chapter, Integrating and Extending, leaves the reader with a solid understanding of what can be done to make jQuery even more useful. This chapter provides a nice motivational finish to the book.

At the start of each chapter Matt recaps what has been covered and outlines where the chapter is heading which makes it easy to get back into the book after putting it down for a few days.

This book is definitely not for the copy and paste coder, nor the developer who just wants ready made solutions they can quickly hack into an existing project. Some may disagree, but I think this is a real positive of this book. Matt uses the examples to illustrate certain concepts or features which he wants the reader to understand. I found the examples got me thinking about what I wanted to use JS and jQuery for in my Drupal sites. Although some of the code samples run to several pages, Matt then spends a lot of time explaining what is happening in bit sized chunks, which makes it easy to understand. I also appreciated the links to documentation so I could get the information I'd need to write my own code for my projects.

One thing which always annoys me about Packt books is the glossy ink they use. In some lighting conditions it is too shiny, which makes it annoying to read, especially with a bed side lamp. On the positive side, the paper is solid and easy to turn.

Sprinkled through the book is some cross promotion of other Packt titles, which I have no issue with, it is a good opportunity to try to grab some additional sales. In a couple of the later chapters it becomes a bit too much. I think once or twice per chapter is reasonable.

I really enjoyed reading Drupal 6 JavaScript and jQuery, it is easy to read and the chapters are a size which lend themselves to being read in a session. I think any Drupal developer who wants to get into using JS in their sites/projects would benefit from reading this book. I finished it feeling like I wanted to start doing some hacking. I plan to update this site in the next few months, and now jQuery enabled effects is on the requirements list. I hope I can bump into Matt Butcher at a DrupalCon or somewhere else in my travels so I can buy him a beer to thank him for putting together a quality book.

Disclaimer Packt Publishing gave me a dead tree copy of this book to review it and keep. I'm glad they gave me a good title to review.

Are sploggers getting smarter?

Some time ago I switched from using Akismet to Mollom. When I made the switch I was preparing to (finally) migrate to Drupal 6, but that project has stalled. I also though it would be a better idea to use something developed by acquia, rather than the compeition.

After switching to Mollom I noticed the number of comments I was getting went through the floor, even though I kept the module up to date.

After some complaints from users I disabled caching and allowed Mollom approved comments to be posted without moderation.

There has been a big problem with my recent changes - the sploggers are winning. It seems if a splogger quotes a paragraph or 2 from the item they are commenting on, they're getting through. I haven't been able to find a way of telling if it is humans entering the CAPTCHAs or if it is bots exploiting a weakness in Mollom.

Either way I think this is a problem that needs to be sorted. No I don't have any suggestions for solving it, I am busy solving other issues - so please don't label me an OSDB. Well given only the Mollon client is FOSS and I suspect the problem is in the (proprietary) backend, I can duck that accusation :)

Work, Business, Travel and Family

Financially the last 12 months has been pretty good for DHC. What financial crisis? This has come at quite a cost.

Once again I have ended up having an extended period of time away from home for business. Today is day 37, country number 5. I am sitting in a hotel room on the evening of my only day off this week wondering why am I doing this. I have a family, friends, clients and a life back in Australia.

Others have blogged about hitting the travel wall - for me it kicks in after about 2 weeks and I keep fighting it until I get home. This year I have been away for 103 of 311 days - that doesn't include the days spent in Melbourne or visiting family. I really understand what Dries meant in a recent tweet.

I am starting to realise humans aren't designed to work like this.

Sure there is a lot of stuff that can be done via email, irc, chat or on the phone. All of the technology in the world can't replace face to face meetings. Just as technology can't provide the opportunity to exchange a funny story or joke in the corridor, it certainly can't make up for just wanting to be at home.

I know a few developers like myself who have ideas kicking around in the back of their mind most of the time, then it clicks - bang solution! This background noise is a blessing and curse. It never goes down too well when I am suppose to be spending time with the family, but while pushing one of the kids on the swing the answer to a problem from yesterday comes to me, ring the client to discuss the solution and make sure they are happy with it - damn junior has fallen over while I was distracted.

The background noise also has another significant downside, it is very hard to shut off. I find alcohol works pretty well at giving me some "mental space", while also being legal and readily available. The downside is the harder I work the more I need that mental space and so drink more. Earlier today, I found a site which claims to guestimate how much alcohol you would have to consume in 3 hours before you were dead, I tweeted my results, I thought it was amusing. The other thing I came across was a blog post from Stephan Hermann (aka @ubuntuworker on identi.ca) on his fight with alcohol. Stephan's post was a timely reminder of how things can slide.

Most of us in IT want to see the system delivered, the product shipped, the customer happy, and most of us will put 120% into making that happen. When it is all done, we go for a beer or 10. Just about every IT shop I have worked for had beers on a Friday night, after a project was delivered or sometimes just when we came up with a lame excuse.

I don't think many people outside of the IT industry understand how stressful it can be and drinking, or other substance (ab)use, is considered an acceptable way of dealing with that stress.

Earlier this year I was in Beijing and some client data was lost due to a mix up (mostly caused by me not being able to read Chinese and someone else's lack of English). I felt awful. Around the same time a guy working for Foxconn a new iPhone prototype threw himself out of the office after being accused of stealing one of the units. This was an extreme case but I so felt for the guy, he was just doing his job, something stuffs up and he feels powerless.

There are many more stories I could link to and discuss, but I won't.

In Australia and other places there is a lot of talk about making work environments more family friendly. Sadly I don't see that happening in our industry. I see the opposite. The global financial crisis is used by some to make their staff work harder and longer for less money. As a contractor I get to set a price and run the meter, but I find myself saying yes to projects or demands that I should not be accepting.

When I return to Australia I plan to give my liver a break, spend more time with my family and develop strategies for "switching off" more often. Life is too short to be a slave to work. I can't continue to work like this claiming I am building a future for my children when I am undermining my own future, and so theirs. Time to deal with my issues, make a plan and be successful, like another Dave.

Sorry for the rant but I felt that this all needed to be said.

Building Debian (and Ubuntu) Meta Packages

Over the last few weeks I have been building a bunch of Debian packages (aka debs) for a new Ubuntu server roll out. Most of the packages are either updates to existing packages or meta packages. Building meta packages is pretty easy, once you know how.

I will outline how to build a simple package which pulls in a couple of useful packages.

First off we need to create the directory structures and files. I do all of my packaging work in /home/$USER/packaging, with my meta packages living in a directory called meta.

For your first package run the following command

$ mkdir -p /home/$USER/packaging/meta/my-meta/DEBIAN

The key to creating meta packages is the "control" file. I have a very generic package I use for all of my servers, called dhc-base . This pulls in what I consider to be the minimum dependencies needed for a basic server. My

~/packaging/meta/dhc-base/DEBIAN/control
file looks something like this:

Package: dhc-base
Version: 0.1
Section: main
Priority: standard
Architecture: all
Depends: dhc-archive-keyring, fail2ban, iptables, openssh-server, screen, shorewall, ubuntu-minimal
Maintainer: Dave Hall <EMAIL-ADDRESS>
Description: Base install meta package for boxes administered by Dave Hall Consulting

The fields should all be pretty self explanatory. The key one is "Depends" which lists all of the packages which you want your package to pull in. I try to keep the list alphabetical so it is easier to manage.

In my example I pull in some basic things which I use all the time as well as the the gpg signing key for my packages, which I have also packaged - I may blog how to do that one day too.

Now we are ready to build the package. simply run

$ dpkg-deb -b /home/$USER/packaging/meta/my-meta

and wait for dpkg-deb to work its magic. Now you should have a shiny new deb called my-meta.deb sitting in /home/$USER/packaging/meta

If you have a bunch of meta packages to build, it can become tedious to have run the command over an over again, and each time the packages will overwrite the previous version. To save me some effort I wrote a little shell script which build a package, and gives it a nice version number too.

#!/bin/bash # # build-meta - dpkg-deb wrapper script for building meta packages # # Developed by Dave Hall Consulting # # Copyright (c) 2009 Dave Hall Consulting - http://davehall.com.au # # You may freely use and distribute this script as long as the copyright # notice is preserved # function usage { SCRIPT=`basename $0` echo Usage: $SCRIPT package-path output-path } if [ $# != 2 ]; then usage $0 exit 1 fi DIR=$1 OUT=$2 DPKG_DEB=dpkg-deb PKGNAME=`basename $DIR` BUILDREV=`date +%Y%m%d$H%I%S` VERSION=`cat $DIR/DEBIAN/control | awk '$1~/^Version:/{print $2}'` echo "Building $PKGNAME" $DPKG_DEB -b $DIR $OUT/${PKGNAME}_$VERSION-${BUILDREV}_all.deb

The script it pretty simple. It takes to arguments, the path for the package and directory to put the final package in, it will even read the version number from the control file.

To process all of the meta packages at once, simply run:

$for pkg in `find /home/$USER/packaging/meta -maxdepth 1 -type d | egrep -v '(.bzr|.svn|.git)'`; do /path/to/build-meta $pkg /home/$USER/packaging/built; done

Now you should have a nice collection of meta packages to deploy.

If you want to setup your own debian repository for your meta packages, I would recommend reading Ian Lawrence's reprepro howto.

I have found meta packages really simplify the tedious task of setting up common package sets - especially for server roll outs.

Update: If you are storing your meta packages under version control, as you should be, there is a problem. If you build the debs direct from a subversion checkout the .svn directory is included - so make sure you svn export your meta packages. Same principle applies for other version control systems.