drupal

First Impressions Motorola Dext and Drupal Editor for Android

Today I purchased a Motorola Dext (aka Cliq) from Optus. Overall I like it. It feels more polished than the Nokia N97 which I bought last year. The range of apps is good. Even though the phone only ships with Android 1.6, 2.1 for the Dext is due in Q3 2010.

The apps seem to run nice and fast. The responsive touch screen is bright and clear. I am yet to try to make a call on it from home, but the 3G data seems as fast as my Telstra 3G service, so the signal should be ok.

The keyboard is very functional, albeit cramped with my fat thumbs. The home screen is a little cluttered for my liking too, but it won't take much to clean that up. I will miss my funambol sync, which is only available for Android 2.x

I started writing this post using the Drupal Editor for Android app, which is pretty nice. The GPL app uses the XML-RPC and Drupal core's Blog API module. Overall it feels like a stripped down version of Bilbo/Blogilo. Drupal Editor is an example of an app which does one thing and does it simply but well. The only thing I haven't liked about it was when originally writing this post. I bumped the save button and published an incomplete and poorly written post. Next time I will untick the publish checkbox until I am ready to really publish it.

I would still like a HTC Desire, but Telstra is only offering them on a $65 plan with no value. The Nokia N900 was off my list, due to the USB port of death and Nokia's spam policies. The Nexus One was on the list too, but a local warranty was a consideration.

Solr Replication, Load Balancing, haproxy and Drupal

I use Apache Solr for search on several projects, including a few using Drupal. Solr has built in support for replication and load balancing, unfortunately the load balancing is done on the client side and works best when using a persistent connection, which doesn't make a lot of sense for php based webapps. In the case of Drupal, there has been a long discussion on a patch in the issue queue to enable Solr's native load balancing, but things seem to have stalled.

In one instance I have Solr replicating from the master to a slave, with the plan to add additional slaves if the load justifies it. In order to get Drupal to write to the master and read from either node I needed a proxy or load balancer. In my case the best lightweight http load balancer that would easily run on the web heads was haproxy. I could have run varnish in front of solr and had it do the load balancing but that seemed like overkill at this stage.

Now when an update request hits haproxy it directs it to the master, but for reads it balances the requests between the 2 nodes. To get this setup running on ubuntu 9.10 with haproxy 1.3.18, I used the following /etc/haproxy/haproxy.cfg on each of the web heads:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    maxconn 4096
    nbproc 4
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    maxconn 2000
    balance roundrobin
    stats enable
    stats uri /haproxy?stats

frontend solr_lb
    bind localhost:8080
    acl master_methods method POST DELETE PUT
    use_backend master_backend if master_methods
    default_backend read_backends

backend master_backend
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check

backend slave_backend
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

backend read_backends
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

To ensure the configuration is working properly run

wget http://localhost:8080/solr -O -
on each of the web heads. If you get a connection refused message haproxy may not be running. If you get a 503 error make sure solr/jetty/tomcat is running on the solr nodes. If you get some html output which mentions Solr, then it should be working properly.

For Drupal's apachesolr module to use this configuration, simply set the hostname to localhost and the port to 8080 in the module configuration page. Rebuild your search index and you should be right to go.

If you had a lot of index updates then you could consider making the master write only and having 2 read only slaves, just change the IP addresses to point to the right hosts.

For more information on Solr replication refer to the Solr wiki, for more information on configuring haproxy refer to the manual. Thanks to Joe William and his blog post on load balancing couchdb using haproxy which helped me get the configuration I needed after I decided what I wanted.

Check Drupal Module Status Using Bash

When you run a lot of drupal sites it can be annoying to keep track of all of the modules contained in a platform and ensure all of them are up to date. One option is to setup a dummy site setup with all the modules installed and email notifications enabled, this is OK, but then you need to make sure you enable the additional modules every time you add something to your platform.

I wanted to be able to check the status of all of the modules in a given platform using the command line. I started scratching the itch by writing a simple shell script to use the drupal updates server to check for the status of all the modules. I kept on polishing it until I was happy with it, there are some bits of which are a little bit ugly, but that is mostly due to the limitations of bash. If I had to rewrite the it I would do it in PHP or some other language which understands arrays/lists and has http client and xml libraries.

The script supports excluding modules by using a extended grep regular expression pattern and nominating a major version of drupal. When there is a version mismatch it will be shown in bolded red, while modules where the versions match will be shown in green. The script filters out all dev and alpha releases, after all the script is designed for checking production sites. Adding support for per module update servers should be pretty easy to do, but I don't have modules to test this with.

To use the script, download it, save it somewhere handy, such as

~/bin/check-module-status.sh
, make it executable (run
chmod +x ~/bin/check-module-status.sh
). Now it is ready for you to run it -
~/bin/check-module-status.sh /path/to/drupal
and wait for the output.

Packaging Drush and Dependencies for Debian

Lately I have been trying to avoid non packaged software being installed on production servers. The main reason for this is to make it easier to apply updates. It also makes it easier to deploy new servers with meta packages when everything is pre packaged.

One tool which I am using a lot on production servers is Drupal's command line tool - drush. Drush is awesome it makes managing drupal sites so much easier, especially when it comes to applying updates. Drush is packaged for Debian testing, unstable and lenny backports by Antoine Beaupré (aka anarcat) and will be available in universe for ubuntu lucid. Drush depends on PEAR's Console_Table module and includes some code which automagically installs the dependency from PEAR CVS. The Debianised package includes the PEAR class in the package, which is handy, but if you are building your own debs from CVS or the nightly tarballs, the dependency isn't included. The auto installer only works if it can write to /path/to/drush/includes, which in these cases means calling drush as root, otherwise it spews a few errors about not being able to write the file then dies.

A more packaging friendly approach would be to build a debian package for PEAR Console_Table and have that as a dependency of the drush package in Debian. The problem with this approach is that drush currently only looks in /path/to/drush/includes for the PEAR class. I have submitted a patch which first checks if Table_Console has been installed via the PEAR installer (or other package management tool). Combine this with the Debian source package I have created for Table_Console (see the file attached at the bottom of the post), you can have a modular and apt managed instance of drush, without having to duplicate code.

I have discussed this approach with anarcat, he is supportive and hopefully it will be the approach adopted for drush 3.0.

Update The drush patch has been committed and should be included in 3.0alpha2.

Upcoming Book Reviews

Packt Publishing seem to have liked my review of Drupal 6 Javascript and jQuery, so much so they have asked me to review another title. On my return from linux.conf.au and Drupal South in New Zealand, a copy of the second edition of AJAX and PHP was waiting for me at the post office. I'll be reading and reviewing the book during February.

I will cover LCA and Drupal South in other blog posts once I have some time to sit down and reflect on the events. For now I will just gloat about winning a spot prize at Drupal South. I walked away with Emma Jane Hogbin and Konstantin Käfer's book, Front End Drupal. I've wanted to buy this title for a while, but shipping from the US made it a bit too pricey even with the strong Australian Dollar. I hope to start reading it in a few weeks, with a review to follow shortly after.

Got a book for me to review? I only read books in dead tree format as I mostly read when I want to get away from the screen. Feel free to contact me to discuss it further.

Updating all of your Drupal Sites at Once - aka Lazy Person's Aegir

Aegir is an excellent way to manage multi site drupal instances, but sometimes it can be a bit too heavy. For example if you have a handful of sites, it can be overkill to deploy aegir. If there is an urgent security fix and you have a lot of sites (I am talking 100s if not 1000s) to patch, waiting for aegir to migrate and verify all of your sites can be a little too slow.

For these situations I have a little script which I use to do the heavy lifting. I keep in ~/bin/update-all-sites and it has a single purpose, to update all of my drupal instances with a single command. Just like aegir, my script leverages drush, but unlike aegir there is no parachute, so if something breaks during the upgrade you get to keep all of the pieces. If you use this script, I would recommend always backing up all of your databases first - just in case.

I keep my "platforms" in svn, so before running the script I run a svn switch or svn update depending on how major the update is. If you are using git or bzr, you would do something similar first. If you aren't using any form of version control - I feel sorry for your clients.

So here is the code, it should be pretty self explanatory - if not ask questions via the comments.

#!/bin/sh # Update all drupal sites at once using drush - aka lazy person's aegir # # Written by Dave Hall # Copyright (c) 2009 Dave Hall Consulting http://davehall.com.au # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # Alternatively you may use and/or distribute it under the terms # of the CC-BY-SA license http://creativecommons.org/licenses/by-sa/3.0/ # Change this to point to your instance of drush isn't in your path DRUSH_CMD="drush" if [ $# != 1 ]; then     SCRIPT="`basename $0`"     echo "Usage: $SCRIPT path-to-drupal-install"     exit 1; fi SITES_PATH="$1" PWD=$(pwd) cd "$SITES_PATH/sites"; for site in `find ./ -maxdepth 1 -type d | cut -d/ -f2 | egrep -v '(.git|.bzr|.svn|all|^$)'`; do     if [ -f "${site}/settings.php" ]; then         echo updating $site         $DRUSH_CMD updatedb -y -l $site     fi done # Lets go back to where we started cd "$PWD"

OK, so my script isn't any where as awesome as aegir, but if you are lazy (or in a hurry) it can come in handy. Most of the time you will probably still want to use aegir.

Notes:

Make sure you make the script executable (hint run chmod +x /path/to/update-all-sites)

If you don't have drush in your path, I would recommend you add it, but if you can't then change DRUSH_CMD="drush" to point to your instance of drush - such as DRUSH_CMD="/opt/drush/drush".

Thanks to Peter Lieverdink (aka cafuego) for suggesting the improved regex.

<?php print t('hello world'); ?>

My blog is now syndicated on Planet Drupal. I am very excited about this - thanks Simon.

For the last 8 years or so I have been running my own IT consulting business, focusing on free/open source software and web application development. My clients have range from micro businesses up to well known geek brands like SGI. Until recently I lead the phpGroupWare project.

My Drupal profile doesn't really give much of a hint about my involvement with Drupal. My biggest regret is not signing up for a d.o account sooner. I forget when I started using Drupal 4.7, but I liked it straight away. It was the first CMS which worked the way I thought a CMS should work.

Over time I have learned how to get Drupal to do what I want it to do. Due to the massive range of contrib modules I haven't got my hands very dirty hacking on Drupal - yet.

This year I have been involved in a major Drupal project which involves hosting around 2100 sites. Aegir has made a lot of this painless, especially with our 3,000 line install profile. Over the Christmas period I hope to find the time to blog about the setup, parts of it are pretty crazy.

I'll get around to upgrading my site to Drupal 6 one of these days when I get some time, that should coincide with a visual and content refresh. Feel free to check out some of my older Drupal related posts.

Drupal 6 JavaScript and jQuery

I have just finished reading Matt Butcher's latest book, Drupal 6 JavaScript and jQuery, published by Packt Publishing - ISBN 978-1-847196-16-3. It is a good read. It is one of those books that arrived at the right time and left me inspired.

I have always leaned towards Yahoo's YUI toolkit when I need an Ajax framework, while the rest of the time I just bash out a bit of JS to get the job done. The more I use Drupal, the more I have been wanting to find time to get into jQuery. This book has got me motivated to play with jQuery - especially in combination with Drupal.

The book is logically structured and flows well from chapter to chapter. I find Matt's writing style easy to read, he even brought a smile to my face a few times. Matt assumes a basic knowledge of JS and Drupal, but he also provides links so the reader is able to get additional information if their knowledge is lacking. However, a couple of times Matt seemed to switch quite abruptly from assuming a good level of knowledge on a particular topic to explaining what seemed to me to be basic or simple concepts in great detail.

In the first chapter, entitled Drupal and JavaScript, Matt covers the basics of Drupal, its relationship with JavaScript and recommends some essential items for any serious Drupal developer's toolbox. This chapter provides a nice introduction of what is to come in the rest of the book and allows the reader to become acquainted with Matt's style.

Working with JavaScript in Drupal covers the basics of the Drupal coding standards and why sticking to the standard is important. It then moves onto a quick overview of Drupal's theme engine, PHPTemplate, and integrating JS with Drupal themes. I felt that the development practices part of this chapter could have been expanded a bit more and turned into its own chapter. Understanding the basics of theming is critical for being able to follow the rest of the book, but again I think this half of the chapter could have been developed into a separate chapter. Regardless of how the chapter was arranged, the content is well written and provides solid and practical examples.

In jQuery: Do More with Drupal, Matt gives a detailed overview of jQuery and how it is used in Drupal. Although the code sample has limited real world usefulness, it provides the reader with a very clear idea of the power of jQuery and how easy it is to use with Drupal. By the end of this chapter I was left feeling like I wanted to get my hands dirty with jQuery, unfortunately it was after 1am and I had to work the next day.

In Chapter 4, we move onto Drupal's Behaviors, which is covered in great detail. Behaviors are a key part of Drupal's JS implementation and essentially provide an events based hooks system in JavaScript. Once again Matt spends a lot of time explaining this feature, how it works, how to use it and where to learn more. Matt's description of this feature had me thinking "OMG, Drupal behaviours are awesome" throughout the chapter.

Lost in Translations, is the name of a good movie starring Scarlett Johansson and Bill Murray, which I enjoyed watching a few years ago, oh and is also the fifth chapter of the book. I suspect that I am like many English speaking Drupal developers in that I use the basics of the Drupal translation engine, but pay very little attention to how it works as my target audience is English speaking like me. Not only does Matt explain how Drupal's translation system works in both PHP and JavaScript, he makes it clear why all Drupal developers should understand and use the system - regardless of their native/target language/s.

The JavaScript Themeing chapter was a bit of a surprise for me. I was expecting Drupal to have a JS equivalent to PHPTemplate and for this chapter to outline it and provide some code samples. Instead I learn that Drupal has a very simple, and easy to use, JS themeing system. Matt spends some time discussing best practice for themeing content in JS and goes on to provide the code for his own simple yet powerful jQuery based themeing engine for Drupal.

In AJAX and Drupal Web Services, we learn about JSON, XML and XHR in the context of Drupal. Once again Matt demonstrates the ease of using Drupal and jQuery for quickly building powerful functionality.

Chapter 8 is entitled, Building a Module, and covers the basics of building a JS enabled module for Drupal. Matt also discusses when JS belongs in a theme and when it should be part of a module. The cross promotion of his other book Learning Drupal 6 Module Development ramps up a couple of notches in this chapter. I found the plugs a bit irritating (especially as I own a copy of the book), but overall the chapter is loaded with useful information.

The final chapter, Integrating and Extending, leaves the reader with a solid understanding of what can be done to make jQuery even more useful. This chapter provides a nice motivational finish to the book.

At the start of each chapter Matt recaps what has been covered and outlines where the chapter is heading which makes it easy to get back into the book after putting it down for a few days.

This book is definitely not for the copy and paste coder, nor the developer who just wants ready made solutions they can quickly hack into an existing project. Some may disagree, but I think this is a real positive of this book. Matt uses the examples to illustrate certain concepts or features which he wants the reader to understand. I found the examples got me thinking about what I wanted to use JS and jQuery for in my Drupal sites. Although some of the code samples run to several pages, Matt then spends a lot of time explaining what is happening in bit sized chunks, which makes it easy to understand. I also appreciated the links to documentation so I could get the information I'd need to write my own code for my projects.

One thing which always annoys me about Packt books is the glossy ink they use. In some lighting conditions it is too shiny, which makes it annoying to read, especially with a bed side lamp. On the positive side, the paper is solid and easy to turn.

Sprinkled through the book is some cross promotion of other Packt titles, which I have no issue with, it is a good opportunity to try to grab some additional sales. In a couple of the later chapters it becomes a bit too much. I think once or twice per chapter is reasonable.

I really enjoyed reading Drupal 6 JavaScript and jQuery, it is easy to read and the chapters are a size which lend themselves to being read in a session. I think any Drupal developer who wants to get into using JS in their sites/projects would benefit from reading this book. I finished it feeling like I wanted to start doing some hacking. I plan to update this site in the next few months, and now jQuery enabled effects is on the requirements list. I hope I can bump into Matt Butcher at a DrupalCon or somewhere else in my travels so I can buy him a beer to thank him for putting together a quality book.

Disclaimer Packt Publishing gave me a dead tree copy of this book to review it and keep. I'm glad they gave me a good title to review.

Drupal Book Review

OK, I am not reviewing a book today, but I soon will be. Packt Publishing have asked me to review Matt Butcher's new book Drupal 6 JavaScript and jQuery. The book looks pretty interesting. Alhtough it isn't on the same scale, being asked to review a serious Drupal developer book, is a bit like Obama winning the noble peace prize - ok maybe I am exaggerating a little there.

I really like YUI, but Drupal has made me interested in jQuery. One of the things awesome things about Drupal is that you can use jQuery without ever having to touch jQuery. This has made me lazy about learning jQuery - especially in the context of Drupal. It look like I have run out of excuses.

The book should arrive in France by the end of the week, but I won't be back in France for a couple of weeks, I have heading off to China for 10 days or so to catch up with friends and discuss some ideas about doing cool things with Drupal. Watch this space.

Missing Software Freedom Conference Kosova

Today I should be in Prishtina Kosovo running Drupal workshops at the first Software Freedom Conference Kosova. Unfortunately due to work and family commitments I had to decline the invitation. I hope to make it there next year.

I will also be missing out on DrupalCon Paris next week too.

Sometimes it sucks to be in Australia, especially when Europe is so far away and so many cool things happening there too.