howto

Multi Core Apache Solr on Ubuntu 10.04 for Drupal with Auto Provisioning

Apache Solr is an excellent full text index search engine based on Lucene. Solr is increasingly being used in the Drupal community for search. I use it for search for a lot of my projects. Recently Steve Edwards at Drupal Connect blogged about setting up a mutli core Solr server on Ubuntu 9.10 (aka Karmic). Ubuntu 10.04LTS was released a couple of months ago and it makes the process a bit easier, as Apache Solr 1.4 has been packaged. An additional advantage of using 10.04LTS is that it is supported until April 2015, whereas suppport for 9.10 ends in 10 months - April 2011.

As an added bonus in this howto you will be able to auto provision solr cores just by calling the right URL.

In this tutorial I will be using Jetty rather than tomcat which some tutorials recommend, as Jetty performs well and generally uses less resources.

Install Solr and Jetty

Installing jetty and Solr just requires a simple command

$ sudo apt-get install solr-jetty openjdk-6-jdk

This will pull down Solr and all of the dependencies, which can be alot if you have a very stripped down base server.

Configuring Jetty

Configuring Jetty is very straight forward. First we backup the existing /etc/default/jetty file like so:

sudo cp -a /etc/default/jetty /etc/default/jetty.bak

Then simply change your /etc/default/jetty to be like this (the changes are highlighted):

# Defaults for jetty see /etc/init.d/jetty for more

# change to 0 to allow Jetty to start
NO_START=0
#NO_START=1

# change to 'no' or uncomment to use the default setting in /etc/default/rcS 
VERBOSE=yes

# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
#JETTY_USER=jetty

# Listen to connections from this network host (leave empty to accept all connections)
#Uncomment to restrict access to localhost
#JETTY_HOST=$(uname -n)
JETTY_HOST=solr.example.com

# The network port used by Jetty
#JETTY_PORT=8080

# Timeout in seconds for the shutdown of all webapps
#JETTY_SHUTDOWN=30

# Additional arguments to pass to Jetty    
#JETTY_ARGS=

# Extra options to pass to the JVM         
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true"

# Home of Java installation.
#JAVA_HOME=

# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in /etc/default/jetty). Should contain a list of space separated directories.
#JDK_DIRS="/usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun"

# Java compiler to use for translating JavaServer Pages (JSPs). You can use all
# compilers that are accepted by Ant's build.compiler property.
#JSP_COMPILER=jikes

# Jetty uses a directory to store temporary files like unpacked webapps
#JETTY_TMP=/var/cache/jetty

# Jetty uses a config file to setup its boot classpath
#JETTY_START_CONFIG=/etc/jetty/start.config

# Default for number of days to keep old log files in /var/log/jetty/
#LOGFILE_DAYS=14

If you don't include the JETTY_HOST entry Jetty will only bind to the local loopback interface, which is all you need if your drupal webserver is running on the same machine. If you set the JETTY_HOST make sure you configure your firewall to restrict access to the Solr server.

Configuring Solr

I am assuming you have already installed the Apache Solr module for Drupal somewhere. If you haven't, do that now, as you will need some config files which ship with it.

First we enable the multicore support in Solr by creating a file called /usr/share/solr/solr.xml with the following contents:

<solr persistent="true" sharedLib="lib">
 <cores adminPath="/admin/cores" shareSchema="true" adminHandler="au.com.davehall.solr.plugins.SolrCoreAdminHandler">
 </cores>
</solr>

You need to make sure the file is owned by the jetty user if you want it to be dymanically updated, otherwise change persistent="true" to persistent="false", don't include the adminHandler attribute and don't run the commands below. Also if you want to auto provision cores you will need to download the jar file attached to this post and drop it into the /usr/share/solr/lib directory (which you'll need to create).

sudo chown jetty:jetty /usr/share/solr
sudo chown jetty:jetty /usr/share/solr/solr.xml
sudo chmod 640 /usr/share/solr/solr.xml
sudo mkdir /usr/share/solr/cores
sudo chown jetty:jetty /usr/share/solr/cores

To keep your configuration centralised, symlink the file from /usr/share/solr to /etc/solr. Don't do it the other way, Solr will ignore the symlink.

sudo ln -s /usr/share/solr/solr.xml /etc/solr/

Solr needs to be configured for Drupal. First we backup the existing config file, just in case, like so:

sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.orig.xml
sudo mv /etc/solr/conf/solrconfig.xml /etc/solr/conf/solrconfig.orig.xml

Now we copy the Drupal Solr config files from where you installed the module

sudo cp /path/to/drupal-install/sites/all/modules/contrib/apachesolr/{schema,solrconfig}.xml /etc/solr/conf/

Solr needs the path to exist for each core's data files, so we create them with the following commands:

sudo mkdir -p /var/lib/solr/cores/{,subdomain_}example_com/{data,conf}
sudo chown -R jetty:jetty /var/lib/solr/cores/{,subdomain_}example_com

Each of the cores need their own configuration files. We could implement some hacks to use a common set of configuration files, but that will make life more difficult if we ever have to migrate some of cores. Just copy the common configuration for all the cores:

sudo bash -c 'for core in /var/lib/solr/cores/*; do cp -a /etc/solr/conf/ $core/; done'

If everything is configured correctly, we should just be able to start Jetty like so:

sudo /etc/init.d/jetty start

If you visit http://solr.example.com:8080/solr/admin/cores?action=STATUS you should get some xml that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<response>
	<lst name="responseHeader">
		<int name="status">0</int>
		<int name="QTime">0</int>
	</lst>
	<lst name="status"/>
</response>

If you get the above output everything is working properly

If you enabled auto provisioning of Solr cores, you should now be able to create your first core. Point your browser at http://solr.example.com:8080/solr/admin/cores?action=CREATE&name=test1&i... If it works you should get output similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<response>
	<lst name="responseHeader">
		<int name="status">0</int>
		<int name="QTime">1561</int>
	</lst>
	<str name="core">test1</str>
	<str name="saved">/usr/share/solr/solr.xml</str>
</response>

I would recommend using identifiable names for your cores, so for davehall.com.au I would call the core, "davehall_com_au" so I can easily find it later on.

Security Note: As anyone who can access your server can now provision solr cores, make sure you restrict access to port 8080 to only allow access from trusted IP addresses.

For more information on the commands available, refer to the Solr Core Admin API documenation on the Solr wik.

Next in this series will be how to use this auto provisioning setup to allow aegir to provision solr cores as sites are created.

Building Debian (and Ubuntu) Meta Packages

Over the last few weeks I have been building a bunch of Debian packages (aka debs) for a new Ubuntu server roll out. Most of the packages are either updates to existing packages or meta packages. Building meta packages is pretty easy, once you know how.

I will outline how to build a simple package which pulls in a couple of useful packages.

First off we need to create the directory structures and files. I do all of my packaging work in /home/$USER/packaging, with my meta packages living in a directory called meta.

For your first package run the following command

$ mkdir -p /home/$USER/packaging/meta/my-meta/DEBIAN

The key to creating meta packages is the "control" file. I have a very generic package I use for all of my servers, called dhc-base . This pulls in what I consider to be the minimum dependencies needed for a basic server. My

~/packaging/meta/dhc-base/DEBIAN/control
file looks something like this:

Package: dhc-base
Version: 0.1
Section: main
Priority: standard
Architecture: all
Depends: dhc-archive-keyring, fail2ban, iptables, openssh-server, screen, shorewall, ubuntu-minimal
Maintainer: Dave Hall <EMAIL-ADDRESS>
Description: Base install meta package for boxes administered by Dave Hall Consulting

The fields should all be pretty self explanatory. The key one is "Depends" which lists all of the packages which you want your package to pull in. I try to keep the list alphabetical so it is easier to manage.

In my example I pull in some basic things which I use all the time as well as the the gpg signing key for my packages, which I have also packaged - I may blog how to do that one day too.

Now we are ready to build the package. simply run

$ dpkg-deb -b /home/$USER/packaging/meta/my-meta

and wait for dpkg-deb to work its magic. Now you should have a shiny new deb called my-meta.deb sitting in /home/$USER/packaging/meta

If you have a bunch of meta packages to build, it can become tedious to have run the command over an over again, and each time the packages will overwrite the previous version. To save me some effort I wrote a little shell script which build a package, and gives it a nice version number too.

#!/bin/bash # # build-meta - dpkg-deb wrapper script for building meta packages # # Developed by Dave Hall Consulting # # Copyright (c) 2009 Dave Hall Consulting - http://davehall.com.au # # You may freely use and distribute this script as long as the copyright # notice is preserved # function usage { SCRIPT=`basename $0` echo Usage: $SCRIPT package-path output-path } if [ $# != 2 ]; then usage $0 exit 1 fi DIR=$1 OUT=$2 DPKG_DEB=dpkg-deb PKGNAME=`basename $DIR` BUILDREV=`date +%Y%m%d$H%I%S` VERSION=`cat $DIR/DEBIAN/control | awk '$1~/^Version:/{print $2}'` echo "Building $PKGNAME" $DPKG_DEB -b $DIR $OUT/${PKGNAME}_$VERSION-${BUILDREV}_all.deb

The script it pretty simple. It takes to arguments, the path for the package and directory to put the final package in, it will even read the version number from the control file.

To process all of the meta packages at once, simply run:

$for pkg in `find /home/$USER/packaging/meta -maxdepth 1 -type d | egrep -v '(.bzr|.svn|.git)'`; do /path/to/build-meta $pkg /home/$USER/packaging/built; done

Now you should have a nice collection of meta packages to deploy.

If you want to setup your own debian repository for your meta packages, I would recommend reading Ian Lawrence's reprepro howto.

I have found meta packages really simplify the tedious task of setting up common package sets - especially for server roll outs.

Update: If you are storing your meta packages under version control, as you should be, there is a problem. If you build the debs direct from a subversion checkout the .svn directory is included - so make sure you svn export your meta packages. Same principle applies for other version control systems.

Howto get a Public IP on Vodafone's 3G Network in Australia

Vodafone in Australia offers a pretty good mobile data plan - 5Gb for $39.95 per month. They have recently upped the price to $49.95 p/m.

Unlike 3, vodafone doesn't offer a public IP addresses to their "mobile broadband" customers. Vodafone pitch this as a business product. I don't agree with it, but I can see how you could justify only offering a NAT'd IP address when using your handset to access the internet or maybe even as a tethered modem. Such logic can't be sustained when offering a HSDPA modem as a "mobile broadband" service. If it is mobile "broadband" then it should be similar to a fixed line broadband service.

After discovering VF only offer a handful of gateways for their data customers, I tried finding out about getting a dynamic public IP address.

To cut a long story short, after 4 calls to data support, and about the same to corporate support, I was at a dead end. Consumer data support told me that I needed to talk to Corporate data support, who wouldn't talk to me as I wasn't a corporate customer.

Eventually I gave up and called the TIO, who, as always were great. I then called the Vodafone complaints team who struggled with all the details of broadband, public IPs, gateway IPs, various service acronyms and the terms which I had agreed to.

After a few more phone calls and waits I was finally awarded my dynamic static IP address. They add something to your account to give you access to the full access APN which gives you a public IP and no port restrictions. For the record the APN is "internet", instead of the normal "vfinternet.au", but this won't work unless VF enable it for you. I some how think Vodafone award access as a prize for persistence.

I did a quick check on the vf.au site again tonight and it seems the small print is the same, so if you sign up for the service I think you have good ground for getting a public dynamic IP like I did. It will just take jumping through a few hoops.

Update: The title should have read public not static IP.

Using Gigabyte BIOS Updates on Linux Boxes

Update: Comments suggest that gigabyte are now using 7zip, not rar as their archive format.

Dealing with Gigabyte support can be a frustrating experience. They only offer support via their website. Once they reply to your enquiry which can take several days, you get a response telling you to visit their website to read the response, and you can reply. This process means it can take several weeks to get a clear and final answer.

In my case I was trying to get a fix for what I thought was a flakey BIOS in my Gigabyte GA-M68SM-S2L. Although Gigabyte claim that their QFlash BIOSes can be loaded independently of the OS the box is running, they only supply them as Windows binary self extracting archives. Gigabyte tech support aren't much help, suggesting that users can just extract it on a Windows box. There is an alternative.

The Gigabyte QFlash binaries are simply self extracting rar files. The following steps make it easy to update Gigabyte QFlash BIOSes on a linux box (albeit with non free software).

  • Download the firmware from Gigabyte
  • cd /path/to/gigabyte-fw.exe
  • unrar e gigabyte-fw.exe
  • cp firmware.fw /path/to/usbstick
  • Reboot computer and select flash BIOS from USB

If all goes to plan you should now have a new BIOS and not had to use a Windows machine to do it.

A Virtual Host per Project

Not long before my old laptop got to the end of it usable lifespan I started playing with the Zend Framework in my spare time. One of the cool things about ZF is that it wants to use friendly URLs, and a dispatcher to handle all the requests. The downside of this approach, and how ZF is organised, it works best if you use a Virtual Host per project. At first this seemed like a real pain to have to create a virtual host per project. One Saturday afternoon I worked through the apache docs and found a solution - then I found it fantastic. Rather than bore you with more of my views on Zend Framework, I will explain how to have a virtual host model that requires a little work up front and is very low maintenance.

It gets tedious copying and pasting virtual host config files each time you want to start a new project, so instead I let Apache do the work for me.

I added a new virtual host config file called projects to

/etc/apache2/sites-available
. The file contains

UseCanonicalName Off

LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon

<Directory /home/dave/Projects>
Options FollowSymLinks
AllowOverride All
</Directory>

NameVirtualHost 127.0.0.2
<VirtualHost 127.0.0.2>
	ServerName projects

	CustomLog /var/log/apache2/access_log.projects vcommon

	VirtualDocumentRoot /home/[username]/Projects/%1/application/www
	AccessFileName     .htaccess
</VirtualHost>

The important bit is the VirtualDocumentRoot directive which tells Apache to map a hostname to a path. I use an IP address from the 127.0.0.0/8 range for the virtual host, so they aren't accessible to the outside world and I don't have to worry about it changing every time I check locations.

All of my projects live under ~/Projects and each one gets a directory structure that looks something like this.

[projectname]
  |
  +- notes - coding notes, like grep output when refactoring etc
  |
  +- resources - any reference material or code snippets
  |
  +- application - the code for the project
     |
     +- www - document root for vhost

There are usually other paths here too, but they vary from project to project.

To make this work there are few more steps. First enable the new virtual host

$ sudo a2ensite projects

Don't reload apache yet.

Next you need to add the apache module

$ sudo a2enmod vhost_alias

Time to edit your

/etc/hosts
file so you can find the virtual hosts. Add a line similar to this
127.0.0.2 projects phpgw-trunk.project [...] phpgw-stable.project

Now you can restart apache.

$ sudo /etc/init.d/apache2 reload

This is handy for developing client sites - especially using drupal.

Now my

/var/www/index.html
is just an empty file.

I am getting a bit bored with adding entries to

/etc/hosts
all the time. If I get around to adding dnsmasq with wildcard hosts to the mix, I will post a follow up.

This setup is based on my current dev environment (Ubuntu Hardy), but it also works on older versions of Ubuntu. The steps should be similar for Debian and derivatives. For other distros, it should work, just how to make it work may be a little different. Feel free to post tips for others in the comments.

My New Toy - The Nokia N95

About 7 weeks ago I bought a Nokia N95 and I love it. I considered the Neo 1973 from openMoko, a completely open phone platform was appealing, but at the end of the day it isn't certified for Australia, it doesn't have WIFI or a camera nor does it do HSDPA/3G, all things on must have list. The iPhone was never in the race.

I picked up phone for just over 800AUD via ebay, they have since dropped a little in price. It is an Australia version with full local warranty support. The only downside is that it a 3 branded version, not a generic, but hey it works.

The phone got a real work out during my trip to Norway and it worked well. The GPS is a little slow to lock, but once it gets a lock it is right to go. The wifi works well. It is handy knowing if wifi is available somewhere before booting your laptop. I took a stack of pictures with the 5M pixel camera, the ones in bad light or inside aren't fantastic, but when taking shots outside it works a treat. The "DVD quality video" is pretty good too. It makes better movies than our old DVD based handycam. I am planning to use the phone at the birth of my second child (due any day now).

As I expected the phone "just works" as a standalone device, but the real test is how well it works with a Linux desktop. I can report that with Ubuntu 7.10 (aka Gutsy Gibbon) the N95 works well for all the stuff you really need. Below is a couple of quick mini howtos for a few things that you might want to with your N95. Some of the instructions are generic enough that they may work with little (or no) change with other handsets.

Disclaimer: I accept no responsibility for any data loss or stress caused by you following these instructions. Also you should carefully check your warranty before trying any of this. That said - "I just worked for me".

Pairing

For the bluetooth related stuff below you will need to pair your phone and PC. The quickest and easiest way to do this is using the bluetooth-applet. Here is how to do it:

  • Install the bluez-gnome deb - sudo apt-get install bluez-gnome
  • Once installed you should have a bluetooth icon in your GNOME system tray, if not just run "bluetooth-applet &" from the console
  • Right click on the icon and click "Browse Devices"
  • Your phone should be in the list, click on it and then click the Connect button
  • On your phone enter a 4 digit PIN when prompted - it can be anything you like
  • The bluetooth icon will then flash, click on it and enter the same PIN
  • Now your phone and PC should be paired
  • To make sure they are paired we will connect via obex-ftp
  • Right click on the icon and click "Browse Devices"
  • Your phone should be in the list, click on it and then click the Connect button
  • When prompted on your phone allow the connection
  • Nautilus should now launch and you you 2 folders "C:" (internal phone memory) and "E:" (microSD card)

Your PC and N95 are now paired and should be able to communicate via bluetooth without any problems.

Exchanging Files

Copying files to/from the N95 can be a little slow. Lets go from slowest to fastest.

Bluetooth

Copying files using bluetooth is very simple with a gnome desktop and the N95 using OBEX-FTP. Just install OBEX-FTP support for nautilus - "sudo apt-get install gnome-vfs-obexftp". Anytime you want to access the files on your phone via obexftp, just fire up nautilus and type "obex:///" and wait for a list of devices to be displayed. Double click on your phone and you are right to go. It can be rather slow copying files from your phone to your PC this way, but if you don't want to find cables or card readers it works. By slow I mean 30mins for a 100Mb video to copy.

There is also the gnome-obex-server package, which allows you to push content from your phone onto your PC, but I found this slow and I had to jump through too many hoops on the phone to send a file.

USB Cable

The N95 comes with a USB cable with a mini USB connector for connecting your N95 to your PC. When you plug the cable in the phone asks you which mode you wish to use. If you select Mass Storage it is treated like a usb mass storage device by and gnome-volume-manager, so it is mounted as soon as you plug it in. You are then able to access your microSD card.

I haven't been able to get the "Media Player" mode to work with the desktop music players I have tried it with - Rhythmbox, Banshee and Amarok. When used in "Mass Storage" mode it is possible to use File > Scan removable media in Rhythmbox.

Card reader

This is the fatstest way to read data from the microSD card. On the N95 press the power button for half a second, scroll down the list of options and select "Remove memory card", then remove your card from your phone. Now just put in into the SD card adaptor that came with the phone and use it like a normal SD card. Transfers speeds are quite good using this method.

Sync

Setting up sync with the N95 and opensync was relatively painless. I am syncing contacts with evolution, I have also tried with calendar events, which seemed to work as well, I haven't tried todos as I don't use them.

Start off by installing the bits we need

$ sudo apt-get install multisync-gui opensync-plugin-evolution opensync-plugin-syncml

This should also install all the dependencies needed to make opensync work.

On the N95 go Menu > Tools > Sync > PC Suite > Edit Sync Profile. First start by editing the settings under Applications. Lifeblog, Text messages and Bookmarks aren't supported by opensync, so disable them but setting "Include in sync" to No. You can also disable Calendar, Notes (really ToDos) if you wish.

As I had already transferred contacts from my old SE v600i to the N95 and I wanted to sync with an existing addressbook I had some issues. Namely 2 contacts didn't want to sync - I never found out which ones. So I found the easiest way to setup the sync was to create a new addressbook in evo.

  • Go into contacts ([ctrl]-2)
  • From the menu select File > New > Addressbook
  • Fill in the information - Type: "On This Computer", Name: "Phone" (or something else that makes sense for you
  • Click OK
In your GNOME menu under accessories, select multisync-gui. Now we need to create the sync pair.
  • Click the Add button
  • Give the group a meaningful name, such as "n95-evo" and click apply
  • Click the edit button for your new group
  • Tick the checkboxes for those sources which you don't want to use - you must disabled note as this isn't supported by the N95, this list should match the config on your phone
  • Click the Add Member button
  • Select Evolution 2.x from the list of options and Click Apply
  • Select which addressbook you want to sync with - in our case "Phone" (or which ever one you created above)
  • Click the Add Member button
  • This time select SyncML over OBEX Client
  • This where your XML hacking skills come into it (or you can just use my config
    <config>
    	<bluetooth_address>AD:DR:OF:MY:FO:NE</bluetooth_address>
    	<bluetooth_channel>10</bluetooth_channel>
    	<interface>0</interface>
    	<identifier>PC Suite</identifier>
    	<version>1</version>
    	<wbxml>1</wbxml>
    	<username></username>
    	<password></password>
    	<type>2</type>
    	<usestringtable>1</usestringtable>
    	<onlyreplace>0</onlyreplace>
    	<recvLimit>0</recvLimit>
    	<maxObjSize>0</maxObjSize>
    	<contact_db>Contacts</contact_db>
    	<calendar_db>Calendar</calendar_db>
    	<note_db>Notes</note_db>
    </config>
    
    		
    Change AD:DR:OF:MY:FO:NE to the address of your phone Note: I have only been able to get the N95 to sync with opensync using bluetooth.
  • Click Close
  • Click the "Refresh" and the sync should start
  • You have now synced your Nokia N95 with your Linux desktop! YAY!
  • All of your contacts from your phone should now be added to your evo addressbook
  • In evolution, copy any contacts from other addressbooks to the "Phone" addressbook
  • In multisync-gui click the "Refresh" button again and wait for the sync to complete

After I make changes to either either my N95 or evolution contacts I sync them. If you are also syncing your calendar you probably want to sync at least daily

Occasionally you may get conflicts. The GUI allows you to choose which one you want to keep. Having merge support in the GUI would be nice, but I can live with one taking precedence over the other on a per record basis.

Firmware Upgrades and Installing Maps

You need a real box Windows XP to upgrade the firmware or install full maps on the phone. I quickly tried using the usb support in qemu to connect the phone to a virtual version of XP, but I couldn't get it to work. It would be nice if Nokia offered firmware upgrades "over the air" so you could just use WIFI to upgrade the phone's firmware.

You can load map data as you move around, using mobile data, but this is an expensive way of doing it. You only need the map loader software installed on the Windows machine and then you can use the phone in mass storage mode of the microSD card to load the map data. I might try loading map data via qemu one day, but I have all the maps loaded that I currently need.

Software

In terms of what extra software you might want to install on the phone, here is a list of what I am using:

  • PuTTY - SSH on the phone, it is handy when I really need SSH and I don't have my laptop with me. Goota love a phone running SSH2 with public key authentication.
  • VNC. There is a Symbian sponsored port of TightVNC now available for the N95 and other S60 handsets. I am yet to get working properly
  • Fring is a free (as in beer) VoIP and IM app for mobile devices. It works pretty well. I have some issues using it with NodePhone, but I probably just have to tweak something there
  • I am also running the Lotus Sametime client, so I can keep in touch with the ReSight team out on the road. Pidgin is still the best Sametime (or general IM) client I have found, but there is no symbian port (hint hint)

Where is opera mini? I removed it, as I found the WebKit based browser on the N95 nicer in the longer run. It might use more bandwidth, but that is fine when using WIFI for most of your browsing anyway.

Overall

The Nokia N95 is certainly one nice handset. The battery life sucks - mine goes on charge every evening. The phone will never win an award from the FSF for freedom, but it never set out to do that. It sets out to be an all in one device and I think it achieves that goal pretty well. It is certainly usable under Linux, even if it does have some distance to go before it can really be considered truly Linux friendly.

About 7 weeks ago I bought a Nokia N95 and I love it. I considered the Neo 1973 from openMoko, a completely open phone platform was appealing, but at the end of the day it isn't certified for Australia, it doesn't have WIFI or a camera nor does it do HSDPA/3G, all things on must have list. The iPhone was never in the race.

I picked up phone for just over 800AUD via ebay, they have since dropped a little in price. It is an Australia version with full local warranty support. The only downside is that it a 3 branded version, not a generic, but hey it works.

The phone got a real work out during my trip to Norway and it worked well. The GPS is a little slow to lock, but once it gets a lock it is right to go. The wifi works well. It is handy knowing if wifi is available somewhere before booting your laptop. I took a stack of pictures with the 5M pixel camera, the ones in bad light or inside aren't fantastic, but when taking shots outside it works a treat. The "DVD quality video" is pretty good too. It makes better movies than our old DVD based handycam. I am planning to use the phone at the birth of my second child (due any day now).

As I expected the phone "just works" as a standalone device, but the real test is how well it works with a Linux desktop. I can report that with Ubuntu 7.10 (aka Gutsy Gibbon) the N95 works well for all the stuff you really need. Below is a couple of quick mini howtos for a few things that you might want to with your N95. Some of the instructions are generic enough that they may work with little (or no) change with other handsets.

Disclaimer: I accept no responsibility for any data loss or stress caused by you following these instructions. Also you should carefully check your warranty before trying any of this. That said - "I just worked for me".

Pairing

For the bluetooth related stuff below you will need to pair your phone and PC. The quickest and easiest way to do this is using the bluetooth-applet. Here is how to do it:

  • Install the bluez-gnome deb - sudo apt-get install bluez-gnome
  • Once installed you should have a bluetooth icon in your GNOME system tray, if not just run "bluetooth-applet &" from the console
  • Right click on the icon and click "Browse Devices"
  • Your phone should be in the list, click on it and then click the Connect button
  • On your phone enter a 4 digit PIN when prompted - it can be anything you like
  • The bluetooth icon will then flash, click on it and enter the same PIN
  • Now your phone and PC should be paired
  • To make sure they are paired we will connect via obex-ftp
  • Right click on the icon and click "Browse Devices"
  • Your phone should be in the list, click on it and then click the Connect button
  • When prompted on your phone allow the connection
  • Nautilus should now launch and you you 2 folders "C:" (internal phone memory) and "E:" (microSD card)

Your PC and N95 are now paired and should be able to communicate via bluetooth without any problems.

Exchanging Files

Copying files to/from the N95 can be a little slow. Lets go from slowest to fastest.

Bluetooth

Copying files using bluetooth is very simple with a gnome desktop and the N95 using OBEX-FTP. Just install OBEX-FTP support for nautilus - "sudo apt-get install gnome-vfs-obexftp". Anytime you want to access the files on your phone via obexftp, just fire up nautilus and type "obex:///" and wait for a list of devices to be displayed. Double click on your phone and you are right to go. It can be rather slow copying files from your phone to your PC this way, but if you don't want to find cables or card readers it works. By slow I mean 30mins for a 100Mb video to copy.

There is also the gnome-obex-server package, which allows you to push content from your phone onto your PC, but I found this slow and I had to jump through too many hoops on the phone to send a file.

USB Cable

The N95 comes with a USB cable with a mini USB connector for connecting your N95 to your PC. When you plug the cable in the phone asks you which mode you wish to use. If you select Mass Storage it is treated like a usb mass storage device by and gnome-volume-manager, so it is mounted as soon as you plug it in. You are then able to access your microSD card.

I haven't been able to get the "Media Player" mode to work with the desktop music players I have tried it with - Rhythmbox, Banshee and Amarok. When used in "Mass Storage" mode it is possible to use File > Scan removable media in Rhythmbox.

Card reader

This is the fatstest way to read data from the microSD card. On the N95 press the power button for half a second, scroll down the list of options and select "Remove memory card", then remove your card from your phone. Now just put in into the SD card adaptor that came with the phone and use it like a normal SD card. Transfers speeds are quite good using this method.

Sync

Update: I am no longer using OpenSync. I am using Funambol's S60 SyncML app, which works a treat for syncing with my Zimbra server.

Setting up sync with the N95 and opensync was relatively painless. I am syncing contacts with evolution, I have also tried with calendar events, which seemed to work as well, I haven't tried todos as I don't use them.

Start off by installing the bits we need

$ sudo apt-get install multisync-gui opensync-plugin-evolution opensync-plugin-syncml

This should also install all the dependencies needed to make opensync work.

On the N95 go Menu > Tools > Sync > PC Suite > Edit Sync Profile. First start by editing the settings under Applications. Lifeblog, Text messages and Bookmarks aren't supported by opensync, so disable them but setting "Include in sync" to No. You can also disable Calendar, Notes (really ToDos) if you wish.

As I had already transferred contacts from my old SE v600i to the N95 and I wanted to sync with an existing addressbook I had some issues. Namely 2 contacts didn't want to sync - I never found out which ones. So I found the easiest way to setup the sync was to create a new addressbook in evo.

  • Go into contacts ([ctrl]-2)
  • From the menu select File > New > Addressbook
  • Fill in the information - Type: "On This Computer", Name: "Phone" (or something else that makes sense for you
  • Click OK

In your GNOME menu under accessories, select multisync-gui. Now we need to create the sync pair.

  • Click the Add button
  • Give the group a meaningful name, such as "n95-evo" and click apply
  • Click the edit button for your new group
  • Tick the checkboxes for those sources which you don't want to use - you must disabled note as this isn't supported by the N95, this list should match the config on your phone
  • Click the Add Member button
  • Select Evolution 2.x from the list of options and Click Apply
  • Select which addressbook you want to sync with - in our case "Phone" (or which ever one you created above)
  • Click the Add Member button
  • This time select SyncML over OBEX Client
  • This where your XML hacking skills come into it (or you can just use my config
    <config>
    	<bluetooth_address>AD:DR:OF:MY:FO:NE</bluetooth_address>
    	<bluetooth_channel>10</bluetooth_channel>
    	<interface>0</interface>
    	<identifier>PC Suite</identifier>
    	<version>1</version>
    	<wbxml>1</wbxml>
    	<username></username>
    	<password></password>
    	<type>2</type>
    	<usestringtable>1</usestringtable>
    	<onlyreplace>0</onlyreplace>
    	<recvLimit>0</recvLimit>
    	<maxObjSize>0</maxObjSize>
    	<contact_db>Contacts</contact_db>
    	<calendar_db>Calendar</calendar_db>
    	<note_db>Notes</note_db>
    </config>
    
    		
    Change AD:DR:OF:MY:FO:NE to the address of your phone Note: I have only been able to get the N95 to sync with opensync using bluetooth.
  • Click Close
  • Click the "Refresh" and the sync should start
  • You have now synced your Nokia N95 with your Linux desktop! YAY!
  • All of your contacts from your phone should now be added to your evo addressbook
  • In evolution, copy any contacts from other addressbooks to the "Phone" addressbook
  • In multisync-gui click the "Refresh" button again and wait for the sync to complete

After I make changes to either either my N95 or evolution contacts I sync them. If you are also syncing your calendar you probably want to sync at least daily

Occasionally you may get conflicts. The GUI allows you to choose which one you want to keep. Having merge support in the GUI would be nice, but I can live with one taking precedence over the other on a per record basis.

Firmware Upgrades and Installing Maps

You need a real box Windows XP to upgrade the firmware or install full maps on the phone. I quickly tried using the usb support in qemu to connect the phone to a virtual version of XP, but I couldn't get it to work. It would be nice if Nokia offered firmware upgrades "over the air" so you could just use WIFI to upgrade the phone's firmware.

You can load map data as you move around, using mobile data, but this is an expensive way of doing it. You only need the map loader software installed on the Windows machine and then you can use the phone in mass storage mode of the microSD card to load the map data. I might try loading map data via qemu one day, but I have all the maps loaded that I currently need.

Software

In terms of what extra software you might want to install on the phone, here is a list of what I am using:

  • PuTTY - SSH on the phone, it is handy when I really need SSH and I don't have my laptop with me. Goota love a phone running SSH2 with public key authentication.
  • VNC. There is a Symbian sponsored port of TightVNC now available for the N95 and other S60 handsets. I am yet to get working properly
  • Fring is a free (as in beer) VoIP and IM app for mobile devices. It works pretty well. I have some issues using it with NodePhone, but I probably just have to tweak something there
  • I am also running the Lotus Sametime client, so I can keep in touch with the ReSight team out on the road. Pidgin is still the best Sametime (or general IM) client I have found, but there is no symbian port (hint hint)

Where is opera mini? I removed it, as I found the WebKit based browser on the N95 nicer in the longer run. It might use more bandwidth, but that is fine when using WIFI for most of your browsing anyway.

Overall

The Nokia N95 is certainly one nice handset. The battery life sucks - mine goes on charge every evening. The phone will never win an award from the FSF for freedom, but it never set out to do that. It sets out to be an all in one device and I think it achieves that goal pretty well. It is certainly usable under Linux, even if it does have some distance to go before it can really be considered truly Linux friendly.

pfSense and Routed Subnets

I have a few clients running IPCop firewall appliance boxes, but for more complex setups (such as multiple WAN connections) I use pfSense. pfSense is a FreeBSD based firewall appliance. pfSense comes in 2 flavours, one of which is designed to run on low spec embedded hardware, such as that sold by Yawarra Information Appliances. I know that I could just use a bash scripts or Shorewallbut not all my clients are command line ninjas, and I have better things to do with my time.

Until recently in Austalia, "residential grade" ADSL connections used PPPoA/PPPoE (aka Layer 3), while "business grade" services were almost exclusively RFC 1483 bridged connections (aka Layer 2). Earlier this year, Telstra Wholesale have stopped offering Layer 2 connections, and are they are now in the process of migrating all resellers' customers to Layer 3 services. For customers with a single usable static IP address this is unlikely to mean any real change. For customers with larger IP allocations (say /29s or larger) they will switch from an IP block being available from the modem to PPPoE with additional IPs being available via a routed subnet.

After some discussion and playing, I found out there are 2 ways to get a routed subnet working with a pfSense box.

Option A - Firewall handles PPPoE and subnet used on DMZ

This is the solution I went for recently for a new connection setup for a client.

  • Configure ADSL modem/router to run in fully bridged mode
  • Configure pfSense's WAN interface to use PPPoE and fill in the appropriate information.
  • Configure the DMZ to use the routed subnet
  • Assign the first usable IP address to the DMZ interface (usually OPT1) on the pfSense box
  • Allocate the remaining IPs to the boxes in the DMZ
  • Setup your rules appropriately

Option B - Modem handles the PPPoE and subnet used on WAN

This method seems to make more sense for people moving from Layer 2 to Layer 3 connections. Please be aware that I haven't tested this, but I am told it should work.

  • Configure ADSL modem/router to work as router connecting via PPPoE
  • Configure the Ethernet port on the modem/router to use the first usable IP address from the routed subnet range
  • Configure pfSense's WAN interface to use a "static" connection and fill in the appropriate information, with the second usable IP address being assigned to the interface.
  • Assign any left over IP addresses as "Proxy ARP" addresses under Virtual IPs
  • Setup your rules and NATing appropriately

I hope someone find this useful.

Unlocking a Novatel Merlin U530 under Linux

Or another reason why I am glad that I don't use 3 anymore.

I should have posted this some time ago, but forgot and so it has been sitting in my Drivel drafts folder for a bit.

You might want to try these instructions under windows too, if you get a dud unlock. Just use another terminal emulator instead of minicom.

Skip the history and read how to unlock it

I recently switched from 3 mobile to virgin for my UMTS data. I already had a Novatel Merlin U530 card from three. The card isn't fantastic but it works pretty well at upto 384/64Kbps which is good enough for mail and feed reading (my main 2 activities on the train) or ssh.

When trying to get the card unlocked, 3 was less than helpful initially, which is pretty normal for them. Initially the operator told me that my card could not be unlocked, when I asked why I was told it was in my contract. When I read the contract I had signed to the operator I was put on hold. She came back and told me that it couldn't be unlocked, I told her it could as I had read online that it could. Then I hear in the background "just keep telling him it can't be unlocked". When I asked to speak to the person in the background telling her what to say she told me that there was no one telling her what to say. I told her I could hear the guy and that I wanted to either speak to the person or get their name so it could be included in a report to the TIO. I was again put on hold. About 1 minute later I was told that I would be transferred to data services.

After spending a little while on hold I got to speak to someone in data services. First they wanted to know why I wanted my card unlocked, I explained I could get a better deal elsewhere, to which the response was that I could talk to sales to see if there were any new offers available - I declined. Next I had to play 20 questions to make sure I was the account holder and that I was out of contract. After reading the IMEI code off the bottom of the card I was given the unlock code and told that I would be emailed the software - a zip file containing some Windows only application.

I had my content filtering too high so amavis bounced the message the first time. When I rung 3 back, at first 3 told me that I need to talk to my ISP as it was very strange that the message was bounced, I told them I would just use another account - my gmail account. The guy then told me that gmail bounces their messages, so we used my hotmail account.

So now I had the software, so I tried using XP under qemu to talk to the card for the unlocking, no luck. A few days later I borrowed a XP laptop from a friend. By now I had misplaced the unlock code. Back on the 3 merri go round, this time data services was happy to hand over the code. The operator waited on hold while I unlocked the card. I started the application, inserted the card, plugged in the unlock code and got the message confirming that the card was unlocked.

Later that day I tried using a vodafone SIM, no go. Another vodafone SIM no go. My Virgin SIM didn't work either. I tried using the unlock software which kept on erroring.

I thought going to a 3 store might get it fixed quickly. Basically I was told that as the card was out of warranty and I wasn't using a 3 SIM they wouldn't provide any support. The best I could do was pay for a firmware reload from a non authorised service centre.

Back to the 3 call centre merry-go-round. I explained that I was extremely close to lodging a TIO or Consumer Affairs complaint as the card useless and not working as advertised. I spoke to a helpful operator who told me he would escalate the job.

Over the weekend I spent a fair bit of time searching for answers. Eventually I ended up finding a list of AT+C modem commands. In the list was

AT+CLCK

which is for facility lock. I played with it a bit and found that network personalisation was still active, but no other locks were on. This seemed strange to me. I figured this is where my problem was.

This is how I unlocked my card. You must have a 3 (or locking carrier) SIM inserted in the card while attempting this.

I connected to the card using minicom by running the following in a terminal (I didn't need to be root).

minicom /dev/ttyS2
Minicom should then load and output something similar to this
Welcome to minicom 2.2

OPTIONS: I18n 
Compiled on Mar  7 2007, 15:10:03.
Port /dev/ttyS2

               Press CTRL-A Z for help on special keys
                                                     
AT S7=45 S0=0 L1 V1 X4 &c1 E1 Q0                     
OK                                                         

I then entered

AT+CLCK=?

which should return

+CLCK: ("AB","AC","AG","AI","AO","IR","OI","OX","PN","SC")

OK

This is a list of locks available on the card.

Enter the following

AT+CLCK="PN",2

Which means, for network personalisation (PN), query the status (2), which should output

+CLCK: 1

OK

The 1 indicates locked.

You will need your unlock code from your carrier - I do not know how to get it other than via your carrier. This is not a "how to unlock your card with the unlock code" howto. So replace <CODE> with your unlock code.

AT+CLCK="PN",0,<CODE>

The 0 is for unlock. If it works should get OK or something, I forget exactly the response, but ERROR means it failed.

Now your card is unlocked and should work with any carrier.

Use [ctrl] x to quit minicom.

Enjoy!

If you need the settings details for using a Merlin U530 with Virgin Mobile in Australia under Linux try the following

Quick favicon.ico Creation

An easy way of creating favicons using the linux CLI - too easy