devops

The Road to DrupalCon Dublin

DrupalCon Dublin is just around the corner. Earlier today I started my journey to Dublin. This week I'll be in Mumbai for some work meetings before heading to Dublin.

On Tuesday 27 September at 1pm I will be presenting my session Let the Machines do the Work. This lighthearted presentation provides some practical examples of how teams can start to introduce automation into their Drupal workflows. All of the code used in the examples will be available after my session. You'll need to attend my talk to get the link.

As part of my preparation for Dublin I've been road testing my session. Over the last few weeks I delivered early versions of the talk to the Drupal Sydney and Drupal Melbourne meetups. Last weekend I presented the talk at Global Training Days Chennai, DrupalCamp Ghent and DrupalCamp St Louis. It was exhausting presenting three times in less than 8 hours, but it was definitely worth the effort. The 3 sessions were presented using hangouts, so they were recorded. I gained valuable feedback from attendees and became aware of some bits of my talk needed some attention.

Just as I encourage teams to iterate on their automation, I've been iterating on my presentation. Over the next week or so I will be recutting my demos and polishing the presentation. If you have a spare 40 minutes I would really appreciate it if you watch one of the session recording below and leave a comment here with any feedback.

Global Training Days Chennai

Thumbnail frame from DrupalCamp Ghent presentation video

DrupalCamp Ghent

Thumbnail frame from DrupalCamp Ghent presentation video

Note: I recorded the audience not my slides.

DrupalCamp St Louis

Thumbnail frame from DrupalCamp St Louis presentation video

Note: There was an issue with the mic in St Louis, so there is no audio from their side.

Internal Applications: When Semantic Versioning Doesn't Make Sense

Semantic Versioning (or SemVer) is great for libraries and open source applications. It allows development teams to communicate to user and downstream developers the scope of changes in a release. One of the most important indicators in versioning is backwards compatibility (BC). SemVer makes any BC break clear. Web services and public APIs are another excellent use case for SemVer.

As much as I love semantic versioning for public projects, I am not convinced about using it for internal projects.

Libraries, be they internal or publicly released should follow the semantic version spec. Not only does this encourage reuse and sharing of libraries, it also displays good manners towards your fellow developers. Internal API endpoints should also comply with SemVer, but versioning end points should only expose major.minor in the version string. Again this will help maintain good relations with other devs.

End users of your application probably won't care if you drop jQuery in favour of straight JS, swap Apache for nginx or even if you upgrade to PHP 7/Python 3.x. Some users will care if you move a button. They won't care that the underlying data model and classes remain unchanged and so this is only a patch release. In the mind of those users, the button move is a BC break, because you changed their UI. At the end of the day, users don't care about version numbers, they care about features and bug fixes.

Regardless of the versioning system used, changes should be documented in a changelog. The changelog shouldn't just be a list of git commits streamed into a text file. This changelog should be something a non technical user can get their head around. This allows all stakeholders to clearly see the incremental improvement in the system.

In my mind SemVer breaks down when it comes to these types of (web) applications. Without a doubt any of the external integration points of the app should use semantic versioning. However the actual app versioning should just increment. I recommend using date based version numbering, such as 201512010 for the first release on 1 December 2015. The next release on that same day, if required, would be 201512011 and so on. If you release many times per day then you might want to consider a 2 or 3 digit counter component.

The components that make up your application such as the libraries, docker base images, ansible playbooks and so on should be using SemVer. Your build and test infrastructure should be able to create reproducible builds of the full stack if you reference specific versions or dependencies.

Instead of conditioning end users to understand semantic versioning, energy should be expended on teaching users to understand continuous delivery and deployment. The app is going to grow and evolve, tagging a release should be something a script can do. It shouldn't need 5 people in the tea room trying to work out if a feature can go in or not because it might be considered a BC break.

When building an application, get the code written and have people using it. If they're not happy, continue to iterate.

Interacting with the Acquia Cloud API using Python

The Acquia Cloud API makes it easy to manage sites on the platform. The API allows you to perform many administrative tasks including creating, destroying and copying databases, deploying code, managing domains and copying files.

Acquia offers 2 official clients. The primary client is a drush plugin which can only be downloaded from Acquia Insight. The other is a PHP library which states in the README that it is "[n]ot ready for production usage".

On a recent project using WF Tools we needed some pretty advanced deployment scripts for sites hosted on Acquia Cloud. We had tried using a mix of bash and PHP, but that created a maintenance nightmare, so we switched to Python.

I was unable to find a high quality Python library, so I wrote a python client for the Acquia Cloud API. The library implements all of the features that we needed, so there are a few things missing.

Chaining complex commands together is easy because the library implements a fluent interface. An extreme example of what is possible is below:


import acapi

# Instantiate the client
c = acapi.Client('[email protected]', 'acquia-token')

# Copy the prod db to dev, make a backup of the dev db and download it to /tmp
c.site('mysite').environment('prod').db('mysite').copy('dev').backups().create().download('/tmp/backup.sql.gz')

Some of the code is "borrowed" from the Python client for Twilio. The library is licensed under the terms of the MIT license.

I am continuing to develop the library. Consider this a working alpha. Improving error handling, creating a comprehensive test suite and implementing the missing API calls are all on the roadmap. Pull requests are welcome.

The code is PEP 8 (coding standards and PEP 257 (documentation standards) compliant and uses the numpydoc for code documentation.

Check out the Python client for Acquia's Cloud API on github.

Automated Security Reviews of Drupal Sites

Most experienced Drupal developers have worked on sites with security issues. Keeping modules up to date is a pretty straight forward process, especially if you pay attention to the security advisories. Coder can find security issues in your code.

Recently I needed to perform a mass security audit to ensure a collection of sites were properly configured. After searching and failing to find a module that would do what I needed, I decided to write my own. The Security Check module for Drupal checks basic configuration options to ensure a site configuration doesn't have any obvious security flaws. This module isn't designed to find all flaws in your site.

Security Check works by checking a list of installed modules, settings for variables and permission assignments. I hope that others in the community will have suggestions for other generic tests that can be implemented in the module. If you have any ideas (or patches), please submit them as issues in the queue.

This module isn't a substitute for a full security audit, which can be conducted in house or by a third party such as Acquia's Professional Service team. Security Check is designed to be run as part of an automated site audit to catch low hanging fruit.

To use Security Checker install it in ~/.drush and then cd to any docroot and run "drush security-check" or if you prefer to use an alias run "drush @example secchk" from anywhere.