drupal

Interacting with the Acquia Cloud API using Python

The Acquia Cloud API makes it easy to manage sites on the platform. The API allows you to perform many administrative tasks including creating, destroying and copying databases, deploying code, managing domains and copying files.

Acquia offers 2 official clients. The primary client is a drush plugin which can only be downloaded from Acquia Insight. The other is a PHP library which states in the README that it is "[n]ot ready for production usage".

On a recent project using WF Tools we needed some pretty advanced deployment scripts for sites hosted on Acquia Cloud. We had tried using a mix of bash and PHP, but that created a maintenance nightmare, so we switched to Python.

I was unable to find a high quality Python library, so I wrote a python client for the Acquia Cloud API. The library implements all of the features that we needed, so there is a few things missing.

Chaining complex commands together is easy because the library implements a fluent interface. An extreme example of what is possible is below:


import acapi

# Instantiate the client
c = acapi.Client('user@example.com', 'acquia-token')

# Copy the prod db to dev, make a backup of the dev db and download it to /tmp
c.site('mysite').environment('prod').db('mysite').copy('dev').backups().create().download('/tmp/backup.sql.gz')

Some of the code is library is "borrowed" from the Python client for Twilio. The library is licensed under the terms of the MIT license.

I am continuing to develop the library. Consider this a working alpha. Improving error handling, creating a comprehensive test suite and implementing the missing API calls are all on the roadmap. Pull requests are welcome.

The code is PEP 8 (coding standards and PEP 257 (documentation standards) compliant and uses the numpydoc for code documentation.

Check out the Python client for Acquia's Cloud API on github.

Automated Security Reviews of Drupal Sites

Most experienced Drupal developers have worked on sites with security issues. Keeping modules up to date is a pretty straight forward process, especially if you pay attention to the security advisories. Coder can find security issues in your code.

Recently I needed to perform a mass security audit to ensure a collection of sites were properly configured. After searching and failing to find a module that would do what I needed, I decided to write my own. The Security Check module for Drupal checks basic configuration options to ensure a site configuration doesn't have any obvious security flaws. This module isn't designed to find all flaws in your site.

Security Check works by checking a list of installed modules, settings for variables and permission assignments. I hope that others in the community will have suggestions for other generic tests that can be implemented in the module. If you have any ideas (or patches), please submit them as issues in the queue.

This module isn't a substitute for a full security audit, which can be conducted in house or by a third party such as Acquia's Professional Service team. Security Check is designed to be run as part of an automated site audit to catch low hanging fruit.

To use Security Checker install it in ~/.drush and then cd to any docroot and run "drush security-check" or if you prefer to use an alias run "drush @example secchk" from anywhere.

Visualising Drupal Development History with Gource

Over the Christmas break I came across gource, a software version control visualization tool. Gource produces really nice visual representations of software projects growing. About 2 years ago David Norman produced a gource video of development of Drupal up to the 7 release. This is pretty cool, but it only shows who committed the patch, not who contributed to it.

After some searching I found the Drupal contribution analyzer sandbox project. This module allows you to produce contributor tag clouds and code swarm videos. This was closer to what I was after, but I had to patch patch drupal_log_generator.py to support the gource custom log format.

The result is a 5:23 minute video showing the growth of Drupal.

The first few years things are pretty consistent and easy to follow. The Drupal 8 development cycle shows how much the community of contributors has grown. Towards the end of last year things look really chaotic.

To produce the video I used a clone of the Drupal 8 branch as at some time on January 1, 2013. The gource command I used was:
gource --log-format custom -i 500 -s 0.0001 -a 0.01 -r 30 --title "Drupal" --highlight-users --disable-progress --hide filenames -1280x720 drupal --bloom-intensity 0.2 --bloom-multiplier 0.2 --stop-at-end /tmp/commit.log -o -| ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libvpx -b 10000K ~/Videos/drupal.webm

I considered writing a script to find and download user avatars from groups.drupal.org but after reviewing the video without them I decided it would be too cluttered.

Can you find your name in the video?

Note: I gave up on trying to embed the video

What LEGO taught me about Community Building (Drupal Version)

This is a Drupalised version of Jive's Deirdre Walsh's blog post entitled "What LEGO taught me about Community Building".

Like most kids, I loved LEGO. I would spend hours building everything from spaceships to crazy robots (true story).

As an adult, building a community has that same sense of awesomeness.

Here is a list of the top 7 things LEGO taught me about building a quality community.

Accessibility. You can find LEGO building blocks anywhere (especially in the cup holders in my car). Social business needs to be the same. A strong community should span internally and externally, across departments, geographies, and devices.

Usability. Unlike Ikea furniture, anybody can pick up a few LEGO blocks, stick them together, and build something amazing. A good community should make it easy for members to go from a newbie to expert in record time, with engaging tutorials and introductory tours.

Fun. LEGO allows people spend hours being creative. Communities should engage users. Every week Drupal events are held which help make this a reality.

Beneficial. LEGOs are more than just an entertaining toy. By playing with LEGOs, kids learn things like simple mechanics. The same should ring true for your community - members should learn through building and sharing. Community members should be free to run, study, redistribute, modify and copy the code.

Next Generational. LEGO has evolved its product offerings. When I had more free time I used to play around with LEGO Mindstorms NXT. This flavor of LEGO allows you to build and program robots - a far advancement from the standard building blocks. A good community will also adopt next-generation technologies, such as enterprise applications, deep webservices integration, html5 and responsive design.

Versatile. By buying a single set of LEGOs you can make several different creations. One day, you'll build a log cabin and the next day a castle. Building a community is similar. With an investment in one strong social business platform, like Drupal, you can build a variety of vibrant communities for areas like customer support, sales and marketing, social intranet, etc.

Leader. Most boxes of LEGOs comes with one of those cool little, plastic people. Like those minifigs, it's key to have a community manager, who can serve as the front-person. Altimeter Research’s Jeremiah Owyang studied community manager job descriptions from 16 different organizations and found four key elements: community advocacy, brand evangelism, savvy communication skills and editorial planning, and liaising between internal decision makers and community members. In the Drupal community we don't need to link to LinkedIn profiles of people who inspire us. All the cool people have accounts on drupal.org - including Dries.

While building a community might not feel like child's play, just remember that it can be fun and the hard work will pay off in the end. Communities are real things that involve people, they are more than a website built using a CMS. As an example look at recent DrupalCons or BADCamp.

Now, if I can only get belly to be as flat as a LEGO minifig's....

Coder Talks Wanted for DrupalCon Sydney

One of the many hats I wear these days is Development and Coding Track Chair for DrupalCon Sydney 2013. As outlined in the track description we are planning on showcasing what is awesome today in Drupal 7 and the cool stuff that is coming in Drupal 8. Given that there are no core conversations in Sydney we are trying to put together a more intermediate-to-advanced level track. I want people to come to these sessions and go away with their heads full of ideas about what they can do better in their next project.

If you have a session that you think fits that brief then please submit something. If you want to ask me anything before submitting your session, feel free to contact me. The decision on which sessions are accepted will be made in late October / early November by the track team, the global track chairs, the content chair and myself in a collaborative decision making process. The accepted sessions will be announced on 13 November.

Although the event won't be as big as a northern hemisphere DrupalCon, it is going to be full of great people. The initial 100 early bird tickets sold out in less than 8 hours!

Please be aware that there is no financial support available for speakers and you will be required to buy a speakers ticket at a cost of 165USD.

Submissions close at 23:59 AEST (UTC+10) on 26 October so submit a session today!

Switching Installation Profiles on Existing Drupal Sites

In my last blog post I outlined how to use per project installation profiles. If you read that post and want to use installation profiles to take advantage of site wide content changes and centralised dependency management, this post will show you how to do it quickly and easily.

The easiest way to switch installation profiles is using the command line with drush. The following command will do it for you:

$ drush vset --exact -y install_profile my_profile

An alternative way of doing this is by directly manipulating the database. You can run the following SQL on your Drupal database to switch installation profiles:


UPDATE variable SET value = 'my_profile' WHERE name = 'install_profile';
-- Clear the cache using MySQL only syntax, when DB caching is used.
TRUNCATE cache;

Before you switch installation profiles, you should check that you have all the required modules enabled in your site. If you don't have all of the modules required by the new installation profile enabled in your site, your are likely to have issues. The best way to ensure you have all the depedencies enabled is to run the following one liner:

drush en $(grep depedencies /path/to/my-site/profiles/my_profile/my_profile.info | sed -n 's/depedencies\[\]=\(.*\)/\1/p')

Even though it is pretty easy to switch installation profiles I would recommend starting your project with a project specific installation profile.

Edit: Fellow Technocrat, Jaime Schmidt picked up a missing step in the instructions above. You need to enable the installation profile in the system table. The easiest way to do that is with this drush one liner:

echo UPDATE system SET schema_version = 0 WHERE name = 'my_profile' | drush sqlc && drush cc all

Managing per Project Installation Profiles

Unbeknown to many users, installation profiles are what is used to install a Drupal site. The two profiles that ship with core are standard and minimal. Standard gives new users a basic, functional Drupal site. Minimal provides a very minimal configuration so developers and site builders can start building a new site. A key piece of a Drupal distro is an installation profile.

I beleive that developers and more experienced site builders should be using installation profiles as part of their client sites builds. In Drupal 7 an installation profile is treated like a special module, so it can implement hooks - including hook_update_N(). This means that the installation profile is the best place for controlling turning modules on and off, switching themes or any other site wide configuration changes that can't be handled by features or a module specific update hook.

In an ideal world you could have 1 installation profile that is used for all of your projects and you just include it in your base build. Unfortunately installation profiles tend to evolve into being very project specific. At the same time you are likely to want a common starting point. I like to give my installation profiles unique names, rather than something generic like "my_profile", I prefer to use "[client_prefix]_profile". I'll cover project prefixes in another blog post.

After some trial and error, I've settled on a solution which I think works for having a common starting point for an installation profile that will diverge overtime using a unique namespace. My solution relies on some basic templates, a bash script with a bit of sed. I could have written all of this in PHP and even made a drush plugin for it, but I prefer to do this kind of thing on the command line with bash. I'm happy to work with someone to port it to a drush plugin if you're interested.

Here is a simple example of the templates you could use for creating your installation profile. The version on github is closer to what I actually use for clients, along with the build script.

base.info

name = PROFILE_NAME
description = PROFILE_DESCRIPTION
core = 7.x
dependencies[] = block
dependencies[] = dblog

base.install

<?php
/**
 * @file
 * Install, update and uninstall functions for the the PROFILE_NAME install profile.
 */

/**
 * Implements hook_install().
 *
 * Performs actions to set up the site for this profile.
 *
 * @see system_install()
 */
function PROFILE_NAMESPACE_install() {
  // Enable some standard blocks.
  $default_theme = variable_get('theme_default', 'bartik');
  $values = array(
    array(
      'module' => 'system',
      'delta' => 'main',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'content',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'user',
      'delta' => 'login',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'navigation',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'management',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 1,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'help',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'help',
      'pages' => '',
      'cache' => -1,
    ),
  );
  $query = db_insert('block')->fields(array('module', 'delta', 'theme', 'status', 'weight', 'region', 'pages', 'cache'));
  foreach ($values as $record) {
    $query->values($record);
  }
  $query->execute();

  // Allow visitor account creation, but with administrative approval.
  variable_set('user_register', USER_REGISTER_VISITORS_ADMINISTRATIVE_APPROVAL);

  // Enable default permissions for system roles.
  user_role_grant_permissions(DRUPAL_ANONYMOUS_RID, array('access content'));
  user_role_grant_permissions(DRUPAL_AUTHENTICATED_RID, array('access content'));
}

// Add hook_update_N() implementations below here as needed.

base.profile

<?php
/**
 * @file
 * Enables modules and site configuration for a PROFILE_NAME site installation.
 */

/**
 * Implements hook_form_FORM_ID_alter() for install_configure_form().
 *
 * Allows the profile to alter the site configuration form.
 */
function PROFILE_NAMESPACE_form_install_configure_form_alter(&$form, $form_state) {
  // Pre-populate the site name with the server name.
  $form['site_information']['site_name']['#default_value'] = $_SERVER['SERVER_NAME'];
}

Some developers might recognise the code above, it is from the minial installation profile.

The installation profile builder script is a simple bash script that relies on sed.

build-profile.sh

#!/bin/bash
#
# Installation profile builder
# Created by Dave Hall http://davehall.com.au
#

FILES="base.info base.install base.profile"
OK_NS_CHARS="a-z0-9_"
SCRIPT_NAME=$(basename $0)

namespace="my_profile"
name=""
description="My automatically generated installation profile."
target=""

usage() {
  echo "usage: $SCRIPT_NAME -t target_path -s profile_namespace [-d 'project_descrption'] [-n 'human_readable_profile_name']"
}

while getopts  "d:n:s:t:h" arg; do
  case $arg in
    d)
      description="$OPTARG"
      ;;
    n)
      name="$OPTARG"
      ;;
    s)
      namespace="$OPTARG"
      ;;
    t)
      target="$OPTARG"
      ;;
    h)
      usage
      exit
      ;;
  esac
done

if [ -z "$target" ]; then
  echo ERROR: You must specify a target path. >&2
  exit 1;
fi

if [ ! -d "$target" -o ! -w "$target" ]; then
  echo ERROR: The target path must be a writable directory that already exists. >&2
  exit 1;
fi

ns_test=${namespace/[^$OK_NS_CHARS]//}
if [ "$ns_test" != "$namespace" ]; then
  echo "ERROR: The namespace can only contain lowercase alphanumeric characters and underscores ($OK_NS_CHARS)" >&2
  exit 1
fi

if [ -z "$name" ]; then
  name="$namespace";
fi

for file in $FILES; do
  echo Processing $file
  sed -e "s/PROFILE_NAMESPACE/$namespace/g" -e "s/PROFILE_NAME/$name/g" -e "s/PROFILE_DESCRIPTION/$description/g" $file > $target/$file
done

echo Completed generating files for $name installation profile in $target.


Place all of the above files into a directory. Before you can generate your first profile you must run "chmod +x build-profile.sh" to make the script executable.

You need to create the output directory, for testing we will use ~/test-profile, so run "mkdir ~/test-profile" to create the path. To build your profile run "./build-profile.sh -s test -t ~/test-profile". Once the script has run you should have a test installation profile in ~/test-profile.

I will continue to maintain this as a project on github.

GitList and my TODO List

Months ago I was searching for a good web front end for git for doing code reviews and browsing repos. My short list ended up being Gitweb and GitLab.

Gitweb is a Perl based web front end for git that is a sub project of the official git project. Out of the box Gitweb is pretty ugly and I have never found it to be very user friendly. Even with all of its problems, it does what it does pretty well.

GitLab is a Rails based github clone that is being actively developed and shows a lot of promise. If you want a fully blown ready to go basic github clone to setup and run for your dev team, GitLab looks great. Unfortunately trying to install it on RHEL5 wasn't an easy task. GitLab offered a lot more functionality than I needed and my lack of Ruby experience meant that it was adding more complexity than I wanted.

I ended up reluctantly recommending that we use Gitweb but I have since changed my mind.

Over the last few weeks I've been playing with GitList by Klaus Silveira. GitList is a web front end for browsing git repos, that looks a little like github. Klaus describes GitList as:

GitList is an elegant and modern web interface for interacting with multiple git repositories. It allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history, diffs. It also generates RSS feeds for each repository, allowing you to stay up-to-date with the latest changes anytime, anywhere. GitList was written in PHP, on top of the Silex microframework and powered by the Twig template engine. This means that GitList is easy to install and easy to customize. Also, the GitList gorgeous interface was made possible due to Bootstrap.

It is nice having an open source side project to contribute to that gets me out of the Drupal ghetto. GitList also gives me an excuse to play with Symfony components and twig more, both things that are already in Drupal 8 core. Silex has been on my "to play with list" for some time too.

I have been searching for an alternative to Gitweb that is easy to use, has a nice UI and the basic features I want in a web front end for git - commit history, diffs, blame and branch comparison. GitList ticked all the boxes, except blame was a bit broken and branch comparison wasn't implemented.

I now run GitList on my laptop for offline repo browsing and find it very useful. I will be deploying it for a client in the next few weeks as part of a broader Drupal workflow management system.

This isn't a review of GitList and I won't be providing any feature comparison of GitList vs (Gitweb|GitLab|github). Instead this post is inspired by a discussion on twitter between Klaus (@klaussilveira), Fabien Potencier (@fabpot), Peter Droogmans (@attiks) and myself (@skwashd). I want outline how we are using GitList and the underlying git library, my vision for GitList and what I can do to make it happen.

Missing Features

Based on the project description I think that GitList is almost feature complete. There are a few TODO items in the README. In my time using GitList I haven't found myself going "if only it did X (and Y and Z)" - except branch diff support. I have been working on a branch diff patch, and I hope to clean up that code this weekend and submit a pull request. I think it is good when something does one thing and does it well.

Library and App Separation

As part of the workflow management system for Drupal we're developing we needed a solid git library for PHP. The one that ships with GitList is pretty nice. I've been working with my fellow Technocrat developer, James Harvey, on extracting the git library from GitList. We still have some work to do on this, but it is usable today. We have added some enhancements to the library for our purposes and removed the Silex dependency. Our vision is to have a generic OO git library for PHP 5.3+ and that GitList use this as a sub module.

Although I am currently hosting the code in my github repo, I would love to see Klaus create a new primary repo for the library and that James, myself and others would continue to develop and support his library. It is not my intention to fork the library from GitList and maintain it myself in isolation - that is a waste of energy and resources.

Git Module for Drupal

Following on from our creation of a generic git library for PHP, we have created a git wrapper module for Drupal. We plan to release this module in the next week or so on drupal.org - we've already reserved the project namespace. As part of this change the features git module will now depend on the new library wrapper module. We will release other modules that use this library as they pass QA.

Releases

I have a hacker mentality, I am happy to clone a repo, create a branch or tag and start playing. If I find something that it broken, then there is a good chance that I'll fix it and post a patch. At the same time I understand that not everyone can work that way and that in some environments there is a focus on only using official stable releases. It is also difficult tracking bugs against git commit hashes. GitList is moving quickly, but I think that it could benefit from having official releases.

If the library and app are separated, they can have separate and independent release cycles. I am happy to work with Klaus to work out a plan for GitList releases. One of my first roles in an open source project was as a release manager.

Discussion Space

I have posted this on my blog because I don't know what the real audience is for this conversation. I know that there are 3 people interested in participating in the conversation but are there others? I know others contribute patches to GitList. I could have emailed Klaus directly, but that would have excluded others who are interested. Even though in some ways email is on the way out, some discussions require more than 140 characters. Is it worth setting up a mailing list, google group, a forum or some other channel of communication to discuss these issues?

The Future

I am already dependent on GitList for a significant piece of work. I want to work with Klaus and the rest of the GitList community to make a kick arse web front end for git, I also need a rock solid PHP lib for git. I think GitLib provides a solid foundation for both. Fabien wants to use it as a good Silex example. What do others want to do with GitList? Let the discussion begin ...

Upadte: Klaus has asked that we move the discussion to the recently created GitList mailing list/google group, so I have created a new thread to continue this conversation. I can cross an item off my list - discussion space.

Your Site Should be Full of BEANs*

From: Dave Hall
To: boxes <boxes-module@drupal.org>
Subject: Our Relationship

Dear boxes,

I'm sorry but things just aren't working out between us. It's not you, it's me. I need some time to myself. I need to think things through. I'm not sure what I want. We should spend some time apart. We should try new things. I will miss you, but this is for the best. Let's meet for coffee in a couple of weeks.

Love

Dave
xox

Breaking up is never easy, but recently I broke up with the boxes module. I'd been with boxes for a long time, we'd done a lot of good things together. Over time I'd become oblivious to some of boxes' flaws, such as giving black eyes to sys admins who run features reverts on sites where users can edit boxes. Life was pretty good, but my life still felt somewhat empty and incomplete.

One day I was standing in a crowded room with lots of modules on drupal.org and across the room I spotted BEAN. I feel in love instantly. BEAN was decked out in all the latest Drupal 7 gear - fields, Entity API and CTools. The way BEAN moved make me weak at the knees. After another beer I got my courage up, crossed the room and asked if BEAN would like to come back to my site. I was shocked, BEAN wanted to hang out with me and it wasn't just a one night fling. I've been going steady with BEAN for a couple of months now.

Just before BEAN moved in permanently to my dev environment I had to find a way of removing all the remnants of boxes from my life (aka existing sites). It took me less than an hour. I created a new BEAN type and called it "box", I then wrote a little "drush scr" script which converted all of my boxes to BEANs and even updated my contexts to use the newly created BEANs. I hope neither bean or boxes finds my work on github. I chose github hoping neither of them would find my work and also because I don't want to maintain this as a module on d.o.

On a serious note - BEAN is awesome! If you haven't tried it, you really should. BEAN treats blocks as content, not config which is really useful for most sites. For new sites BEAN is a drop in replacement for core blocks or boxes making it super easy to get started. To learn more check out the docs.

If you're interested in seeing the full power of the BEAN module, and other cool stuff I've been working on recently, please comment on my DrupalCon session proposal - An Enterprise Scale Drupal Workflow. It will be a more technical version of my session at Drupal Business Days a couple weeks ago.

* "Full of beans" is phrase commonly used in Australia to describe energetic children.

Drupal in the Enterprise (aka Vote for my DrupalCon Session)

TL; DR: [spam]Please vote for my DrupalCon Denver proposal on Drupal workflows in the enterprise.[/spam]

For the last few months I've been working for Technocrat on a new Drupal based site for the Insurance Australia Group's Direct Insurance brands. The current sites are using Autonomy Teamsite.

The basics of the build are relatively straight forward, around 1000 nodes, a bunch of views and a bit of glue to hold it all together. Where things get complicated is the workflow. The Financial services sector in Australia is subject to strict control of representations being made about products. The workflow system needs to ensure IAG complies with these requirements.

During the evaluation we found that generally Drupal workflows are based around publishing a single piece of content on the production site. In the IAG case a collection of nodes need to be published as a piece of work, along with a new block. These changes need to be reviewed by stakeholders and then deployed. This led us to build a job based workflow system.

We are using the Features module to handle all configuration, deploy for entities and some additional tools, including Symfony, Jenkins and drush to hold it all together.

I've proposed the session for Drupal Downunder in January and will refine the session based on feedback from there in preparation for Denver. If you want to learn more about Drupal Workflows in the Enterprise, please vote for my session.