The Road to DrupalCon Dublin

DrupalCon Dublin is just around the corner. Earlier today I started my journey to Dublin. This week I'll be in Mumbai for some work meetings before heading to Dublin.

On Tuesday 27 September at 1pm I will be presenting my session Let the Machines do the Work. This lighthearted presentation provides some practical examples of how teams can start to introduce automation into their Drupal workflows. All of the code used in the examples will be available after my session. You'll need to attend my talk to get the link.

As part of my preparation for Dublin I've been road testing my session. Over the last few weeks I delivered early versions of the talk to the Drupal Sydney and Drupal Melbourne meetups. Last weekend I presented the talk at Global Training Days Chennai, DrupalCamp Ghent and DrupalCamp St Louis. It was exhausting presenting three times in less than 8 hours, but it was definitely worth the effort. The 3 sessions were presented using hangouts, so they were recorded. I gained valuable feedback from attendees and became aware of some bits of my talk needed some attention.

Just as I encourage teams to iterate on their automation, I've been iterating on my presentation. Over the next week or so I will be recutting my demos and polishing the presentation. If you have a spare 40 minutes I would really appreciate it if you watch one of the session recording below and leave a comment here with any feedback.

Global Training Days Chennai

Thumbnail frame from DrupalCamp Ghent presentation video

DrupalCamp Ghent

Thumbnail frame from DrupalCamp Ghent presentation video

Note: I recorded the audience not my slides.

DrupalCamp St Louis

Thumbnail frame from DrupalCamp St Louis presentation video

Note: There was an issue with the mic in St Louis, so there is no audio from their side.

Per Environment Config in Drupal 8

One of the biggest improvements in Drupal 8 is the new configuration management system. Config is now decoupled from code and the database. Unlike Drupal 6 and 7, developers no longer have to rely on the features module for moving configuration around.

Most large Drupal sites, and some smaller ones, require per environment configuration. Prior to Drupal 8 this was usually achieved using a combination of hard coding config variables and features. Drupal 8 still allows users to put config variables in the settings.php file, but putting config in code feels like a backward step given D8 emphasis on separating concerns.

For example we may have a custom module which calls a RESTful API of a backend service. There are dev, stage and production endpoints that we need to configure. We also keep our config out of docroot and use drush to import the config at deployment time. We have the following structure in our git repo:

+- .git/
+- .gitignore
+- README.md
+- config/
|  |
|  +- README.md
|  |
|  +- base/
|  |
|  +- dev/
|  |
|  +- prod/
|  |
|  +- stage/
+- docroot/
+- scripts/
+- and-so-on/

When a developer needs to export the config for the site they run drush config-export --destination=/path/to/project/config/base. This exports all of the configuration to the specified path. To override the API endpoint for the dev environment, the developer would make the config change and then export just that piece of configuration. That can be done by runing drush config-get mymodule.endpoint > /path/to/project/config/dev/mymodule.endpoint.yml.

Drupal 8 and drush don't allow you to import the 2 config sets at the same time, so we need to run 2 drush commands to import our config. drush config-import --source=/path/to/project/config/base && drush config-import --partial --source=/path/to/project/config/dev. The first command imports the base config and the second applies any per environment overrides. The --partial flag prevents drush deleting any missing config. In most cases this is ok, but watch out if you delete a view or block placement.

Best practices are still emerging for managing configuration in Drupal 8. While I have this method working, I'm sure others have different approaches. Please leave a comment if you have an alternative method.

Update 11-Mar-2016:Removed --partial from base import. This prevented old configuration being removed during updates.

Leaking Information in Drupal URLs

Update: It turns out the DA was trolling. We all now know that DrupalCon North America 2016 will be in New Orleans. I've kept this post up as I believe the information about handling unpublished nodes is relevant. I have also learned that m4032404 is enabled by default in govCMS.

When a user doesn't have access to content in Drupal a 403 forbidden response is returned. This is the case out of the box for unpublished content. The problem with this is that sensitive information may be contained in the URL. A great example of this the DrupalCon site.

The way to avoid this is to use the m4032404 module which changes a 403 response to a 404. This simple module prevents your site leaking information via URLs.

Managing Variables in Drupal 7

A couple of times recently the issue of managing variables in Drupal 7 has come up in conversation with other developers. This post outlines the various ways of managing variables in Drupal sites. The three things this guide ensures:

  • Sensitive data is kept secure
  • Variables are correct in each environment
  • You are able to track your variables (and when they changed)

The Variables Table

The most common place you'll find configuration variables is in Drupal's variable table (aka {variable}). The values in this table are often managed via admin forms that use system_settings_form(). Users enter the values click "Save configuration" and the data is stored in the database.

If you prefer to manage your configuration via the command line and know the variable you wish to set you can use drush vset. This does exactly the same thing as admin form, without needing to click on a mouse.

$conf Array

While the variables table is great at storing our variables, there are times when you want to enforce a setting. This might be because you want to prevent users from changing it (accidentally or otherwise) or because you need it to be different in each environment. The $conf array in settings.php always overrides any values in the variable table.

Acquia, Pantheon and platform.sh all provide environment variables so you can use different values in your $conf array depending on the environment.

Exporting Variables

In Drupal 7, the common way to export your variables is by using Strongarm with Features. I'm not going to cover how to do this as there is loads of documentation already available on this topic.

If your variable changes on a per environment basis or if it calculated on the fly, then you won't want to use strongarm+features as the exported values are static. You will need to put them in settings.php.

Note to self: I should debug and reroll my patch for adding support in alter hooks strongarm.

My settings.php is Out of Control!

This is a common problem, especially on more complex sites. To avoid this I recommend creating sites/default/settings/settings.[env].php files. Your settings.php file should check for the environment in an environment variable and then include the appropriate settings.[env].php file.

What About Sensitive Data?

You can encrypt variables on a case by case basis using the encrypt module and some custom code similar to what I recently implemented in the Acquia SDK module (see on store and on read examples). Warning: This doesn't encrypt the data if you're using drush vset.

If you are storing sensitive data in your variables table I would recommend you implement hook_sql_sync_sanitize() which will delete the sensitive data from your db when drush sql-sanitize or drush sql-sync --sanitize are run.

How to Decide?

This little code snippet should help you decide.


// Don't try using this code in your Drupal site.

if (!using_version_control()) {
  // Seriously there is no point in doing this without version control.

if (is_data_sensitive($var)) {
  $var = encrypt_var($var);
  if (!we_use_drush_based_workflows()) {
    // I'm serious!

if (is_unique_per_environment($var)) {
else {
  if (!we_use_features_based_workflow()) {
    // I'm serious!

Interacting with the Acquia Cloud API using Python

The Acquia Cloud API makes it easy to manage sites on the platform. The API allows you to perform many administrative tasks including creating, destroying and copying databases, deploying code, managing domains and copying files.

Acquia offers 2 official clients. The primary client is a drush plugin which can only be downloaded from Acquia Insight. The other is a PHP library which states in the README that it is "[n]ot ready for production usage".

On a recent project using WF Tools we needed some pretty advanced deployment scripts for sites hosted on Acquia Cloud. We had tried using a mix of bash and PHP, but that created a maintenance nightmare, so we switched to Python.

I was unable to find a high quality Python library, so I wrote a python client for the Acquia Cloud API. The library implements all of the features that we needed, so there are a few things missing.

Chaining complex commands together is easy because the library implements a fluent interface. An extreme example of what is possible is below:

import acapi

# Instantiate the client
c = acapi.Client([email protected]', 'acquia-token')

# Copy the prod db to dev, make a backup of the dev db and download it to /tmp

Some of the code is "borrowed" from the Python client for Twilio. The library is licensed under the terms of the MIT license.

I am continuing to develop the library. Consider this a working alpha. Improving error handling, creating a comprehensive test suite and implementing the missing API calls are all on the roadmap. Pull requests are welcome.

The code is PEP 8 (coding standards and PEP 257 (documentation standards) compliant and uses the numpydoc for code documentation.

Check out the Python client for Acquia's Cloud API on github.

Automated Security Reviews of Drupal Sites

Most experienced Drupal developers have worked on sites with security issues. Keeping modules up to date is a pretty straight forward process, especially if you pay attention to the security advisories. Coder can find security issues in your code.

Recently I needed to perform a mass security audit to ensure a collection of sites were properly configured. After searching and failing to find a module that would do what I needed, I decided to write my own. The Security Check module for Drupal checks basic configuration options to ensure a site configuration doesn't have any obvious security flaws. This module isn't designed to find all flaws in your site.

Security Check works by checking a list of installed modules, settings for variables and permission assignments. I hope that others in the community will have suggestions for other generic tests that can be implemented in the module. If you have any ideas (or patches), please submit them as issues in the queue.

This module isn't a substitute for a full security audit, which can be conducted in house or by a third party such as Acquia's Professional Service team. Security Check is designed to be run as part of an automated site audit to catch low hanging fruit.

To use Security Checker install it in ~/.drush and then cd to any docroot and run "drush security-check" or if you prefer to use an alias run "drush @example secchk" from anywhere.

Visualising Drupal Development History with Gource

Over the Christmas break I came across gource, a software version control visualization tool. Gource produces really nice visual representations of software projects growing. About 2 years ago David Norman produced a gource video of development of Drupal up to the 7 release. This is pretty cool, but it only shows who committed the patch, not who contributed to it.

After some searching I found the Drupal contribution analyzer sandbox project. This module allows you to produce contributor tag clouds and code swarm videos. This was closer to what I was after, but I had to patch patch drupal_log_generator.py to support the gource custom log format.

The result is a 5:23 minute video showing the growth of Drupal.

The first few years things are pretty consistent and easy to follow. The Drupal 8 development cycle shows how much the community of contributors has grown. Towards the end of last year things look really chaotic.

To produce the video I used a clone of the Drupal 8 branch as at some time on January 1, 2013. The gource command I used was:
gource --log-format custom -i 500 -s 0.0001 -a 0.01 -r 30 --title "Drupal" --highlight-users --disable-progress --hide filenames -1280x720 drupal --bloom-intensity 0.2 --bloom-multiplier 0.2 --stop-at-end /tmp/commit.log -o -| ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libvpx -b 10000K ~/Videos/drupal.webm

I considered writing a script to find and download user avatars from groups.drupal.org but after reviewing the video without them I decided it would be too cluttered.

Can you find your name in the video?

Note: I gave up on trying to embed the video

What LEGO taught me about Community Building (Drupal Version)

This is a Drupalised version of Jive's Deirdre Walsh's blog post entitled "What LEGO taught me about Community Building".

Like most kids, I loved LEGO. I would spend hours building everything from spaceships to crazy robots (true story).

As an adult, building a community has that same sense of awesomeness.

Here is a list of the top 7 things LEGO taught me about building a quality community.

Accessibility. You can find LEGO building blocks anywhere (especially in the cup holders in my car). Social business needs to be the same. A strong community should span internally and externally, across departments, geographies, and devices.

Usability. Unlike Ikea furniture, anybody can pick up a few LEGO blocks, stick them together, and build something amazing. A good community should make it easy for members to go from a newbie to expert in record time, with engaging tutorials and introductory tours.

Fun. LEGO allows people spend hours being creative. Communities should engage users. Every week Drupal events are held which help make this a reality.

Beneficial. LEGOs are more than just an entertaining toy. By playing with LEGOs, kids learn things like simple mechanics. The same should ring true for your community - members should learn through building and sharing. Community members should be free to run, study, redistribute, modify and copy the code.

Next Generational. LEGO has evolved its product offerings. When I had more free time I used to play around with LEGO Mindstorms NXT. This flavor of LEGO allows you to build and program robots - a far advancement from the standard building blocks. A good community will also adopt next-generation technologies, such as enterprise applications, deep webservices integration, html5 and responsive design.

Versatile. By buying a single set of LEGOs you can make several different creations. One day, you'll build a log cabin and the next day a castle. Building a community is similar. With an investment in one strong social business platform, like Drupal, you can build a variety of vibrant communities for areas like customer support, sales and marketing, social intranet, etc.

Leader. Most boxes of LEGOs comes with one of those cool little, plastic people. Like those minifigs, it's key to have a community manager, who can serve as the front-person. Altimeter Research’s Jeremiah Owyang studied community manager job descriptions from 16 different organizations and found four key elements: community advocacy, brand evangelism, savvy communication skills and editorial planning, and liaising between internal decision makers and community members. In the Drupal community we don't need to link to LinkedIn profiles of people who inspire us. All the cool people have accounts on drupal.org - including Dries.

While building a community might not feel like child's play, just remember that it can be fun and the hard work will pay off in the end. Communities are real things that involve people, they are more than a website built using a CMS. As an example look at recent DrupalCons or BADCamp.

Now, if I can only get belly to be as flat as a LEGO minifig's....

Coder Talks Wanted for DrupalCon Sydney

One of the many hats I wear these days is Development and Coding Track Chair for DrupalCon Sydney 2013. As outlined in the track description we are planning on showcasing what is awesome today in Drupal 7 and the cool stuff that is coming in Drupal 8. Given that there are no core conversations in Sydney we are trying to put together a more intermediate-to-advanced level track. I want people to come to these sessions and go away with their heads full of ideas about what they can do better in their next project.

If you have a session that you think fits that brief then please submit something. If you want to ask me anything before submitting your session, feel free to contact me. The decision on which sessions are accepted will be made in late October / early November by the track team, the global track chairs, the content chair and myself in a collaborative decision making process. The accepted sessions will be announced on 13 November.

Although the event won't be as big as a northern hemisphere DrupalCon, it is going to be full of great people. The initial 100 early bird tickets sold out in less than 8 hours!

Please be aware that there is no financial support available for speakers and you will be required to buy a speakers ticket at a cost of 165USD.

Submissions close at 23:59 AEST (UTC+10) on 26 October so submit a session today!

Switching Installation Profiles on Existing Drupal Sites

In my last blog post I outlined how to use per project installation profiles. If you read that post and want to use installation profiles to take advantage of site wide content changes and centralised dependency management, this post will show you how to do it quickly and easily.

The easiest way to switch installation profiles is using the command line with drush. The following command will do it for you:

$ drush vset --exact -y install_profile my_profile

An alternative way of doing this is by directly manipulating the database. You can run the following SQL on your Drupal database to switch installation profiles:

UPDATE variable SET value = 'my_profile' WHERE name = 'install_profile';
-- Clear the cache using MySQL only syntax, when DB caching is used.

Before you switch installation profiles, you should check that you have all the required modules enabled in your site. If you don't have all of the modules required by the new installation profile enabled in your site, your are likely to have issues. The best way to ensure you have all the dependencies enabled is to run the following one liner:

drush en $(grep dependencies /path/to/my-site/profiles/my_profile/my_profile.info | sed -n 's/dependencies\[\]=\(.*\)/\1/p')

Even though it is pretty easy to switch installation profiles I would recommend starting your project with a project specific installation profile.

Edit: Jaime Schmidt picked up a missing step in the instructions above. You need to enable the installation profile in the system table. The easiest way to do that is with this drush one liner:

echo UPDATE system SET schema_version = 0 WHERE name = 'my_profile' | drush sqlc && drush cc all

Further Edit: Marji Cermak picked up a typo in the dependencies one liner. It is one word I use a lot and always misspell.