Blogs

First Impressions Motorola Dext and Drupal Editor for Android

Today I purchased a Motorola Dext (aka Cliq) from Optus. Overall I like it. It feels more polished than the Nokia N97 which I bought last year. The range of apps is good. Even though the phone only ships with Android 1.6, 2.1 for the Dext is due in Q3 2010.

The apps seem to run nice and fast. The responsive touch screen is bright and clear. I am yet to try to make a call on it from home, but the 3G data seems as fast as my Telstra 3G service, so the signal should be ok.

The keyboard is very functional, albeit cramped with my fat thumbs. The home screen is a little cluttered for my liking too, but it won't take much to clean that up. I will miss my funambol sync, which is only available for Android 2.x

I started writing this post using the Drupal Editor for Android app, which is pretty nice. The GPL app uses the XML-RPC and Drupal core's Blog API module. Overall it feels like a stripped down version of Bilbo/Blogilo. Drupal Editor is an example of an app which does one thing and does it simply but well. The only thing I haven't liked about it was when originally writing this post. I bumped the save button and published an incomplete and poorly written post. Next time I will untick the publish checkbox until I am ready to really publish it.

I would still like a HTC Desire, but Telstra is only offering them on a $65 plan with no value. The Nokia N900 was off my list, due to the USB port of death and Nokia's spam policies. The Nexus One was on the list too, but a local warranty was a consideration.

ACMA Investigates Nokia for SMS Spam

The ongoing saga of Nokia's txt spam continues.

The bad news is that I received another txt from Nokia today. This comes after being told by Nokia that I would no longer receive any txts from them. The message reads:

Tip: Use less battery power and help conserve energy with a few helpful tips from Nokia. Vist http://environment.nokia.mobi to learn more.

I have an energy saving tip for Nokia, stop sending txt messages I don't want, then I won't waste energy on trying to make them stop.

Now for the good news. A little while ago ACMA told me that they were preparing to launch a formal investigation into Nokia's SMS messages under the Spam Act. It's now official. Late last Friday I received the following email from the ACMA Investigator handling the matter:

Dear Mr Hall

I write with reference to your complaint #XXX concerning allegations of breaches of the Spam Act 2003 (Spam Act). The Australian Communications and Media Authority (ACMA) has now commenced an investigation into Nokia Australia Pty Ltd about potential contraventions of the Spam Act. During the course of this investigation, the ACMA may require you to provide more information about any dealings you have had with Nokia Australia Pty Ltd and potentially complete a witness statement. The Anti-Spam Team will contact you in due course if this is required. The Anti-Spam Team does not provide your personal information to the business apart from the electronic account information (mobile telephone number) you have already provided. As I am sure you can appreciate, the ACMA is not able to disclose details of the investigation with you, but will advise you when an outcome has been determined. On behalf of the ACMA, I appreciate your assistance in this investigation and thank you for your cooperation.

Information about the Spam Act is available on our website at www.spam.acma.gov.au Please contact us if you have any queries.

I wonder how much energy Nokia will put into defending itself to ACMA.

Watch this space for more news.

X Mail Headers on identi.ca

A while ago I submitted a patch for statusnet, the code that powers identi.ca and other microblogging platforms. The patch added X headers to email notification messages, to make it easier to filter messages. The headers look something like this:

X-StatusNet-MessageType: subscribe
X-StatusNet-TargetUser: skwashd
X-StatusNet-SourceUser: username
X-StatusNet-Domain: identi.ca

The patch was included in the 0.9.1 release of statusnet, and is now running on identi.ca. I think this is the highest traffic site running any of my code. I am pretty excited about it.

This is one of the reasons why I love free software cloud services. Instead of just being a passive consumer of the service, you can actively contribute to its development.

You can run it too by downloading statusnet 0.9.1. Enjoy.

Now to setup some mail filters in Zimbra ...

Tricks to Running HAProxy on pfSense Embedded

HAProxy is available as an addon module for pfSense 1.2.3. This makes it really easy to have pfSense control the gateway and load balancing. There are a couple of tricks to getting it all up and running.

Although everything looked good in the webgui HAProxy just wouldn't start. After logging in it seemed that there were 2 problems, firstly as mentioned in the forums the IP addresses must be an interface or CARP addresses not Virtual IPs for HAProxy to work and secondly the file descriptor limits have to be increased. To increase the file descriptor limits run the following commands from a shell on pfSense.

mount -o rw /dev/ufs/pfsense1  /
echo >> /etc/sysctl.conf
echo '# File descriptor limits for HAProxy' >> /etc/sysctl.conf
kern.maxfiles=2000011 >> /etc/sysctl.conf
kern.maxfilesperproc=2000011 >> /etc/sysctl.conf
sysctl kern.maxfiles=2000011
sysctl kern.maxfilesperproc=2000011
mount -o ro /dev/ufs/pfsense1  /

The mount commands are only needed if running on embedded pfSense to make the CF card writeable while we make the changes then make it read only again once we are done. The echo commands add the new limits to /etc/sysctl.conf so the settings persist and the sysctl commands make them apply now.

I haven't tested to see if the file descriptor issue effects the non embedded version of pfSense, feel free to let me (and others know) via the comments.

Making it Easier to Spawn php-cgi on Debian and Ubuntu

Apache is a great web server, but sometimes I need something a bit more lightweight. I already have a bunch of sites using lighttpd, but I'm planning on switching them to nginx. Both nginx and lighttpd use FastCGI for running php. Getting FastCGI up and running on Ubuntu (or Debian) involves a bit of manual work which can slow down deployment.

The normal process to get nginx and php-cgi up and running is to install the spawn-fcgi package, create a shell script such as /usr/local/bin/php-fastcgi to launch it, then a custom init script, after making both of these executable you need to run update-rc.d then finally should be right to go. Each of these manual steps increases the likelihood of mistakes being made.

Instead, I created a deb contains a configurable init script. It is pretty simple, the init script calls spawn-fcgi with the appropriate arguments. All of the configuration is handled in /etc/default/php-fastcgi. The main config options are:

  • ENABLED - Enable (or disable) the init script. default: 0 (disabled)
  • ADDRESS - The IP address to bind to. default: 127.0.0.1
  • PORT - The TCP Port to bind to. default: 9000
  • USER - The user the php scripts will be excuted as. default: www-data
  • GROUP - The group the php scripts will be executed as. default: www-data
  • PHP_CGI - The full path to the php-cgi binary. default: /usr/bin/php5-cgi

The last 3 variables are not in the defaults file as I didn't think many users would want/need to change them, feel free to add them in if you need them.

Once you have set ENABLED to 1, launch the init scipt by executing sudo /etc/init.d/php-fastcgi start. To check that it is running run sudo netstat -nplt | grep 9000 and you should see /usr/bin/php5-cgi listed. Now you can continue to configure your webserver.

The package depends on php5-cgi and spawn-fcgi, which is available in Debian testing/squeeze, unstable/sid, along with Ubuntu karmic and lucid. For earlier versions of ubuntu you can change the dependency in debian/control from spawn-fcgi to lighttpd and disable lighttpd once it is installed so you can get spawn-fcgi . I haven't tested this approach and wouldn't recommend it.

You can grab the http://davehall.com.au/sites/davehall.com.au/files/php-fastcgi_0.1-1_all.deb">binary package and install it using dpkg or build it yourself from the source tarball.

For more information on setting up nginx using php-cgi I recommend the linode howto - just skip the "Configure spawn-fcgi" step :)

Solr Replication, Load Balancing, haproxy and Drupal

I use Apache Solr for search on several projects, including a few using Drupal. Solr has built in support for replication and load balancing, unfortunately the load balancing is done on the client side and works best when using a persistent connection, which doesn't make a lot of sense for php based webapps. In the case of Drupal, there has been a long discussion on a patch in the issue queue to enable Solr's native load balancing, but things seem to have stalled.

In one instance I have Solr replicating from the master to a slave, with the plan to add additional slaves if the load justifies it. In order to get Drupal to write to the master and read from either node I needed a proxy or load balancer. In my case the best lightweight http load balancer that would easily run on the web heads was haproxy. I could have run varnish in front of solr and had it do the load balancing but that seemed like overkill at this stage.

Now when an update request hits haproxy it directs it to the master, but for reads it balances the requests between the 2 nodes. To get this setup running on ubuntu 9.10 with haproxy 1.3.18, I used the following /etc/haproxy/haproxy.cfg on each of the web heads:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    maxconn 4096
    nbproc 4
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    maxconn 2000
    balance roundrobin
    stats enable
    stats uri /haproxy?stats

frontend solr_lb
    bind localhost:8080
    acl master_methods method POST DELETE PUT
    use_backend master_backend if master_methods
    default_backend read_backends

backend master_backend
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check

backend slave_backend
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

backend read_backends
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

To ensure the configuration is working properly run

wget http://localhost:8080/solr -O -
on each of the web heads. If you get a connection refused message haproxy may not be running. If you get a 503 error make sure solr/jetty/tomcat is running on the solr nodes. If you get some html output which mentions Solr, then it should be working properly.

For Drupal's apachesolr module to use this configuration, simply set the hostname to localhost and the port to 8080 in the module configuration page. Rebuild your search index and you should be right to go.

If you had a lot of index updates then you could consider making the master write only and having 2 read only slaves, just change the IP addresses to point to the right hosts.

For more information on Solr replication refer to the Solr wiki, for more information on configuring haproxy refer to the manual. Thanks to Joe William and his blog post on load balancing couchdb using haproxy which helped me get the configuration I needed after I decided what I wanted.

Check Drupal Module Status Using Bash

When you run a lot of drupal sites it can be annoying to keep track of all of the modules contained in a platform and ensure all of them are up to date. One option is to setup a dummy site setup with all the modules installed and email notifications enabled, this is OK, but then you need to make sure you enable the additional modules every time you add something to your platform.

I wanted to be able to check the status of all of the modules in a given platform using the command line. I started scratching the itch by writing a simple shell script to use the drupal updates server to check for the status of all the modules. I kept on polishing it until I was happy with it, there are some bits of which are a little bit ugly, but that is mostly due to the limitations of bash. If I had to rewrite the it I would do it in PHP or some other language which understands arrays/lists and has http client and xml libraries.

The script supports excluding modules by using a extended grep regular expression pattern and nominating a major version of drupal. When there is a version mismatch it will be shown in bolded red, while modules where the versions match will be shown in green. The script filters out all dev and alpha releases, after all the script is designed for checking production sites. Adding support for per module update servers should be pretty easy to do, but I don't have modules to test this with.

To use the script, download it, save it somewhere handy, such as

~/bin/check-module-status.sh
, make it executable (run
chmod +x ~/bin/check-module-status.sh
). Now it is ready for you to run it -
~/bin/check-module-status.sh /path/to/drupal
and wait for the output.

Western Digital My Book World Edition Licensing and the GPL

Earlier today I purchased a Western Digital My Book World Edition (MBWE) 1Tb NAS as I needed a simple NAS. The MBWE seemed like a good option, it runs GNU/Linux, it is hackable and there is a bit of a community around it. I got the thing home and started setting it up through the web gui so I could enable SSH and NFS on it, until I hit the EULA. Yes I am one of those people who does actually read the EULA of most software before installing it. The EULA didn't strike me as a very free/open source software friendly. It states (emphasis mine):

WESTERN DIGITAL TECHNOLOGIES LICENSE AGREEMENT AND WARRANTY DISCLAIMER

NOTICE: By accepting this License Agreement and/or by installing, using or copying the Licensed Software, You are becoming a party to, indicating Your consent to, and agreeing to be bound by the terms of this License Agreement, without modification. If You do not understand and accept all of the following terms and conditions or You respond "NO" or give any other response that indicates you do not accept this License Agreement, then You must not install, use, or copy the Licensed Software.

1. DEFINITIONS.

(a) "Agreement" and/or "License Agreement" shall mean this License Agreement and any and all documents incorporated by reference; (b) "You" and/or "Your" shall mean the individual or legal entity exercising rights under; (c) "Licensed Software" shall mean computer software owned by Western Digital Technologies and may include associated media, printed materials, and "online" or electronic documentation provided by Western Digital Technologies; and (d) "Western Digital Technologies" shall mean Western Digital Technologies, Inc. and its subsidiaries, affiliates, licensees and agents.

2. LICENSE GRANT.

Subject to the terms of this Agreement, Western Digital Technologies hereby grants You a non-exclusive and non-transferable license to and use for personal or internal purposes the Licensed Software. This license is not a sale of the Licensed Software. Western Digital Technologies has no duty to provide further updates or changes to the Licensed Software.

3. RESTRICTIONS.

You acknowledge and agree that You shall not (a) modify, adapt, translate or create any derivative works of the Licensed Software; (b) attempt to disable the Licensed Software by any means or in any manner; (c) attempt to decompile, disassemble, reverse engineer, or otherwise attempt to derive the source code for the Licensed Software; (d) distribute, encumber, sell, rent, lease, sublicense, or otherwise transfer, publish or disclose the Licensed Software to any third-party; or (e) remove or alter any trademark, logo, copyright or other proprietary notices, legends, symbols or labels in the Licensed Software.

4. TITLE.

Title, ownership and all intellectual property rights in and to the Licensed Software shall remain in Western Digital Technologies and/or its suppliers. The Licensed Software is protected by the copyright laws of the United States and international copyright treaties. Title, ownership rights and intellectual property rights in and to the content accessed through the Licensed Software including any content contained in the Licensed Software media demonstration files is the property of the applicable content owner and may be protected by applicable copyright or other law. This license gives you no rights to such content.

5. DISCLAIMER OF WARRANTY.

THE LICENSED SOFTWARE IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, WESTERN DIGITAL TECHNOLOGIES FURTHER DISCLAIMS ALL WARRANTIES INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE, AND FREEDOM FROM INTERFERENCE WITH ENJOYMENT OR FROM NON-INFRINGEMENT OF THIRD PARTY INTELLECTUAL PROPERTY RIGHTS. WESTERN DIGITAL TECHNOLOGIES DOES NOT WARRANT THAT THE LICENSED SOFTWARE WILL MEET YOUR REQUIREEMNTS OR WILL BE ERROR FREE. THE ENTIRE RISK ARISING OUT OF THE USE OR PERFORMANCE OF THE LICENSED SOFTWARE AND DOCUMENTATION REMAINS WITH YOU. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL WESTERN DIGITAL TECHNOLOGIES OR ITS SUPPLIERS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, DIRECT, INDIRECT, SPECIAL, PUNITIVE, OR OTHER DAMAGES WHATSOEVER (INCLUDING, WITHOUT LIMITATION. DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION, OR OTHER PECUNIARY LOSS) ARISING OUT OF THIS AGREEMENT OR THE USE OF OR INABILITY TO USE THE LICENSED SOFTWARE, EVEN IF WESTERN DIGITAL TECHNOLOGIES HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. BECAUSE SOME STATES/JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT APPLY TO YOU.

6. TERMINATION.

This license shall terminate automatically if you fail to comply with the limitations described in this Agreement. No notice shall be required from Western Digital Technologies to effectuate such termination. Upon termination you must destroy all copies of the Licensed Software.

7. U.S. GOVERNMENT RESTRICTED RIGHTS AND EXPORT RESTRICTIONS.

The Licensed Software is provided with RESTRICTED RIGHTS. Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subparagraph (c)(I)(ii) of The Rights in Technical Data and Computer Software clause of DFARS 252.227- 7013 or subparagraphs (c)(i) and (2)of the Commercial Computer Software-Restricted Rights at 48 CFR 52.227-19, as applicable. Manufacturer is Western Digital Technologies, 20511 Lake Forest Blvd., Lake Forest, CA 92630. You acknowledge that none of the Licensed Software or underlying information or technology may be downloaded or otherwise exported or re-exported into (or to a national or resident of) Angola, Cuba, Iran, Iraq, Libya, North Korea, Sudan, Syria, or any other country to which the U.S. has embargoed goods; or anyone on the U.S. Treasury Department's list of Specially Designated Nationals or the U.S. Commerce Department's Table of Denial Orders. By using the Licensed Software, you are agreeing to the foregoing, and are representing and warranting that you are not located in or under the control of a national or resident of any such country or on any such list.

8. INDEMNITY

You hereby agree to indemnify, defend and hold Senvid and Western Digital Technologies from and against any all liabilities, damages, claims, fines and expenses (including reasonable attorney's fees) arising out of any breach of this End User License Agreement and Warranty Disclaimer.

9. MISCELLANEOUS.

(a) This Agreement constitutes the entire agreement between the parties concerning the subject matter hereof; (b) This Agreement may be amended only by a writing signed by both parties; (c) This License Agreement shall be governed by the laws of the State of California, without regard to conflicts of law provisions, and you consent to the exclusive jurisdiction of the state and federal courts sitting in the State of California; (d) This Agreement shall not be governed by the United Nations Convention on Contracts for the International Sale of Goods; (e) If any provision in this Agreement should be held illegal or unenforceable by a court having jurisdiction, such provision shall be modified to the extent necessary to render it enforceable without losing its intent or severed from this Agreement if no such modification is possible, and other provisions of this Agreement shall remain in full force and effect; (f) A waiver by either party of any term or condition of this Agreement or any breach thereof, in any one instance, shall not waive such term or condition or any subsequent breach thereof; (g) The provisions of this Agreement that require or contemplate performance after the expiration or termination of this Agreement shall be enforceable notwithstanding said expiration or termination; (h) You may not assign or otherwise transfer by operation of law or otherwise this Agreement or any rights or obligations herein except in the case of a merger or the sale of all or substantially all of Your assets to another entity; (i) This Agreement shall be binding upon and shall inure to the benefit of the parties, their successors and assigns; (j) Neither party shall be in default or be liable for any delay, failure in performance (excepting the obligation to pay), or interruption of service resulting directly or indirectly from any cause beyond its reasonable control, and; (k) If any dispute arises under this Agreement, the prevailing party shall be reimbursed by the other party for any and all legal fees and costs associated therewith.

IF YOU ACCEPT the terms of this Agreement: I acknowledge and understand that by ACCEPTING the terms of this Agreement, I am agreeing to be bound by the terms, conditions, and limitations of this Agreement.

IF YOU DO NOT ACCEPT the terms of this Agreement. I acknowledge and understand that by refusing to accept these terms, I have rejected this license agreement and therefore have no legal right to install, use, or copy the Licensed Software.

I should probably preface the rest of this post with IANAL and that this does not constitute legal advice and if you need legal advice pay loads of money to a lawyer (I hope that keeps my lawyer happy).

I took a risk and clicked "I Agree" because I couldn't be bothered driving another 130km round trip to take it back and get something different and also I wanted to start hacking it anyway. I also figured that if WD's lawyers wanted to come after me I had the GPL on my side.

I suspect someone in engineering just copied and pasted the text in, just like they do for all of their software. To their credit WD does make the source code for the firmware for the device available on their website, so it doesn't strike me as an intentional attempt to avoid their obligations under the GPL. This seems like a pretty easy problem to fix, get a couple of people from WD's legal and engineering teams in a room and work out what the new text should be and then have engineering roll a new firmware release.

I have also contacted Western Digital's support department about the problem. Any updates will be posted here.

Update 23-Feb-2010: WDC support have responded apologising for any inconvenience caused and thanking me for bringing this to their attention. They also informed me that the information has been passed onto the relevant person. As yet they haven't responded to my follow up question about being informed when the issue is fixed.

Update 16-Mar-2010: After Western Digital closed the issue as resolved, I queried what the resolution was and received the following response, "Thank you for bringing this to our attention. Your issue has been forwarded to our legal and engineering teams and they are working to get our EULA and the terms of the GPL code in sync with one another. We are striving to get this resolved ASAP". It looks like the issue is now with the legal team at Western Digital.

Group Redent Plugin for Status.net / Identi.ca

For a bit over a year now I've been using the free software based microblogging service identi.ca. Unlike twitter, the identi.ca code base is released under the AGPLv3, through the status.net project. Anyone is free to setup their own instance of status.net and can even federate with other instances, using the Open Micro Blogging standard.

Another feature which identi.ca/status.net has over twitter is support for groups, so someone can send a message to a group of people, a bit like how a mailing list works. One down side of this feature is that some people redent (like retweeting, but on identi.ca) without removing the group "bang tag" (aka !). This leads to the same message being sent to the group over an over again which gets annoying. I left !linux and have considered leaving !ubuntu, because of how often this happens.

As status.net is freely available, I decided to write a plugin which handles this problem. If someone redents a message to a group without changing the group tag to a hash tag, the plugin does it for them. It is like the status.net twitter bridge when reposting dents to twitter. If someone really wants to redent something to a group, they simply use !! instead of ! and it will be sent to the group. This is useful, if for example, a dent from someone is relevant to a group, but wasn't posted there initially, it can be redented to the group

I disagree with Bela Hausmann (aka @and3k), I think it should be enabled by default on identi.ca. I hope others consdier it to be a useful feature and ask for it to be added to identi.ca.

If you run your own instance of status.net running, http://davehall.com.au/sites/davehall.com.au/files/GroupRedentCleanerPlugin.php.txt">download the plugin, drop it in

/path/to/statusnet/plugins/GroupRedentCleanerPlugin.php

and add the following to your config.php: addPlugin('GroupRedentCleaner', array()); Have a play!

Making My Nokia SMSes Stop

This is a public service announcement.

Tracy Postill, Corporate Communications Manager at Nokia Australia sent me this response when I enquired as to why it had taken more then 6 months for me to be unsubscribed from the My Nokia service.

There are a number of very simple processes to unsubscribe:

- sending "stop" to +61416906978

- through My Nokia application in phone

- online at www.nokia.com.au/mynokia

- request "unsubscribe" through Australia.mynokia@nokia.com

- Nokia Customer Care (Carline or email support)

This information is presented to consumers in multiple ways:

- after signing in via phone, customer receives a welcome message and information on how to opt-out (SMS)

- ~20% of scheduled SMS contain opt-out information (SMS)

- in every sales pack, there are the T&Cs for MyNokia including all the details of how to opt-out

- My Nokia FAQ on https://my.nokia.com/appmanager/AustraliaEnglishMY_NOKIA_AUSTRALIA/Login?SAMLart=AAQAAYU6%2FZbJFfjRP3cckEAJmxKhSmx8sqF8q2CNtVwZBvZUL6lb%2FBg%2Fmio%3D&RelayState=qh8DL4DVL1Qp9q612ncWTwWvRp6QQKYB52xh1226HJJ1RP14NvQr!-432133733!1266205653496#faq14

My signup text message and terms and conditions were returned to Optus along with everything else that came with my N97. None of the txts I have recieved from My Nokia has contained any opt out information and the FAQ requires a My Nokia account to be able to access it.

I explained to Tracy that I think Nokia has major issues with their processes for unsubscribing from My Nokia, she has "sent this email to our head of care and asked him to follow up on this". I will post a follow up if I receive a response from "head of care". I hope this saves others the hassle of trying to unsubscribe from the service.

Update 25-Feb-2010 The txts have stopped. What an ordeal! Hopefully if you are reading this the options above work for you.