mirror

Internode adds SourceForge.net Mirror

Yesterday Internode annouced that they will be hosting the Australian SourceForge.net mirror. Internode has been a long term supporter of FOSS. They are one of the few ISPs who officially support Linux. They offer their massive mirror, which has terabytes of FOSS, to the world.

The new SourceForge mirror will be available to the world. The Australian mirror was previously hosted by Optus. Optus hosts other mirrors for FOSS projects including ubuntu. Unfortunately their mirrors are not as reliable as many users would like. The Australian sf.net mirror should be rock solid if Internode's past performance is anything to go by.

Like the other content Internode mirrors, the sf.net mirror will be unmetered for node's ADSL customers. This is in addition to a large amount of other unmetered content, such as ABC content, almost 100 streaming radio stations and other services..

I use and recommend internode to anyone who is interested in high quality ADSL services. The support for FOSS is a secondary consideration, as it is pointless having access to it all, but the connection being down all the time.

Member's Areas and wget

Earlier this evening I was discussing mirroring restricted areas of sites with wget on #ubuntu-au. The solution is pretty simple.

  1. Install the web developer extension for firefox
  2. Login to the target site
  3. On the webdev toolbar select Cookies > View Cookie Information
  4. For each of the cookie entries add the following to a file called wget-cookies.txt which should be saved in your home directory
<.target.domain.name>[tab]FALSE[tab]/[tab]FALSE[tab]1496836642[tab]<key>[tab]<value>

This is what it all means

  • <.target.domain.name> the domain of the site
  • TRUE the domain wide flag, if the domain starts with . this should be TRUE
  • / the path the cookie applies to
  • FALSE is the cookie secure (or available via HTTPS)
  • 1496836642 the expiry of the cookie (i am using 11:57:22 UTC on 7-Jun-2017)
  • <key> the name of the cookie
  • <value> the value of the cookie

If you just want to pull down a single page use the following command:

wget  --load-cookies ~/wget-cookies.txt <target-url>

Then you should have the target page

If you want to mirror the whole site as an authenticated user try something like:

wget --mirror -w 2 -p --convert-links  --load-cookies ~/wget-cookies.txt <target-url>

I tested this with a couple of my own sites and it seems to work well.

Before doing something like this, check the term of service and the license of the content to ensure that you are not in violation of either.