Update: I updated the script with --remote-silent and --servername based on a suggestion in the comments.

Basic Script

The first thing to decide is location and naming convention for the files. As there is going to be one file per week, more or less indefinitely, I’m going to need some form of date in the file name so that when listed alphabetically, the files will also be sorted chronologically. For location, I’m using a folder inside Dropbox which is an easy way for the notes to be automatically backed up and synced to my various computers.

I would like to name each file log_yyyy-mm-dd.txt where yyyy-mm-dd is the date of the first weekday (Monday) for that week, but the difficulty is that the script needs to be able to create a new, properly named file when run on any day of the week, not just on Monday. OS X ships with the BSD version of the date command, which allows date manipulation via the -v parameter.

date -v-monday +log_%Y-%m-%d.txt

This will take the current date and rewind to the most recent Monday (-v-monday), then reformat that date as a string suitable for a file name. I put the year first so that earlier dates will sort correctly before later ones. You should enclose the format parameter (+log_%Y-%m-%d.txt) in quotes if your format string includes spaces or any characters such as * or ? which the shell would interpret incorrectly.

This is enough to write a basic launcher script:

#!/bin/bash -e

LOG_DIR=$HOME/Dropbox/work/worklog
LOG_FILE=$(date -v-monday '+log_%Y-%m-%d.txt')

gvim $LOG_DIR/$LOG_FILE

If for some reason your favorite text editor is not MacVim, you can replace the last line as necessary. I save the file to ~/bin/edit_worklog.sh and make it executable. Upon running the script, I get a new MacVim window with the empty file ~/Dropbox/work/worklog/log_2011-12-05.txt open in it. So far so good.

Improvements

Now I’d like to make some improvements. When I make my log entries, I like to separate each day with a header line that includes the date, like this:

Monday, 12/05

 * did
 * some
 * stuff

Tuesday, 12/06

 * continued work on stuff
 * meeting with Mr. X

... etc

It would be nice if the launcher script could create the 5 weekday headers for me when creating the new log file. I already have the date for Monday for the name of the file, but now I need dates for Tuesday through Friday. These are simply offset by one or more days from the Monday value. For this I can use -v+1d to add days as needed. If you are using GNU date, you can use something like -d'monday+1 days'. Here’s the whole week:

# prints Monday, 12/05
date -v-monday '+%A, %m/%d'

# prints Tuesday, 12/06
date -v-monday -v+1d '+%A, %m/%d'

# prints Wednesday, 12/07
date -v-monday -v+2d '+%A, %m/%d'

# prints Thursday, 12/08
date -v-monday -v+3d '+%A, %m/%d'

# prints Friday, 12/09
date -v-monday -v+4d '+%A, %m/%d'

Now it would be nice to have the script open the file with the cursor position at the end and to control the size of the editing window. These can be accomplished by adding + and --cmd 'set lines=25 columns=60' (respectively) to the gvim command.

Lastly, if I run the script while the file is already open for editing, I’ll get the usual vim warning about the swap file being already in use. For this project, I would rather bring the existing editor window to the foreground. For this we can use vim’s remote editor capabilities. By adding --remote-silent we ask vim to edit the file in an existing window (the remote part) and create a new editor window if one doesn’t already exist (the silent part). We should also add --servername worklog to give the remote window a name so that it doesn’t replace any existing remote sessions.

Here’s my full script with all improvements and some handy comments.

#!/bin/bash
#
# edit_worklog.sh - script to edit (and create if necessary)
#                   a rotating weekly work log file
#
# Author: Barry Stump

# this is the path to the weekly log folder
LOG_DIR="$HOME/Dropbox/work/worklog"

# the files are named by the first day of the week (most recent Monday)
LOG_FILE="$LOG_DIR/"$(date -v-monday '+log_%Y-%m-%d.txt')

# does today's file not exist yet?
if [ ! -e $LOG_FILE ]; then

    # create new log file with a line for each weekday...
    # (this version is for the BSD date command that
    # comes with OS X. For GNU date, use something like
    # -d'monday+2 days')

    # Monday
    date -v-mon '+%A, %m/%d' > $LOG_FILE
    echo >> $LOG_FILE
    # Tuesday
    date -v-mon -v+1d '+%A, %m/%d' >> $LOG_FILE
    echo >> $LOG_FILE
    # Wednesday
    date -v-mon -v+2d '+%A, %m/%d' >> $LOG_FILE
    echo >> $LOG_FILE
    # Thursday
    date -v-mon -v+3d '+%A, %m/%d' >> $LOG_FILE
    echo >> $LOG_FILE
    # Friday
    date -v-mon -v+4d '+%A, %m/%d' >> $LOG_FILE
    echo >> $LOG_FILE

fi

# open up the file
# --cmd "set lines=25 columns=60"
#       (resize window to 60x25)
#
# +     (move cursor to end of file)
#
# --servername worklog --remote-silent
#       (bring window to foreground if already open)
#
gvim --cmd "set lines=25 columns=60" + \
     --servername worklog --remote-silent $LOG_FILE

Keyboard Shortcut

Now that I have a handy script to launch my work log, I would like to bind it to a hotkey combination. In OS X, I accomplished this by using the Automator utility to create a service workflow and then binding a hotkey to it. These directions are for OS X 10.6 Snow Leopard. Older or newer versions of OS X may require some tweaks.

First, open Automator (from the Applications folder) and choose the Service template in the dialog that appears.

For “Service receives selected” choose “no input”. Now drag “Run Shell Script” from the library listing on the left to the empty section on the right. Replace the default command cat with the path and name of the saved script from above. Here I’m using $HOME/bin/edit_worklog.sh.

When done, save the workflow with ⌘S and enter a name in the popup. If you ever need to edit the workflow again later, it is located at ~/Library/Services. I called my workflow “View Worklog”. This adds an entry “View Worklog” to the Services menu of every application. The final step is to associate a system-wide keyboard shortcut to this service. Open System Preferences, choose Keyboard, and then click the Keyboard Shortcuts pane. Click “Services” from the category list on the left, then scroll down to “General”. Double click to the right of your newly created service name to assign a keyboard shortcut. I used ⇧⌘` which appears to be unused on my system.

Close System Preferences to save the settings, and you should be able to pull up the weekly log window just by using the new keyboard shortcut combination.

WebSocket Handshake

To establish a WebSocket connection, the browser makes a HTTP GET request to the server with an offer to upgrade to WebSocket. In the current websocket spec, that request looks like this:

Client Request

GET /resource HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: example.com:8080
Origin: http://www.example.com
Sec-WebSocket-Key1: W  1529l9 6i4b{{a 9 z55
Sec-WebSocket-Key2: F13 O05   q  q6 - 4k7D   97

-#&nD[fU

The first line is a HTTP/1.1 GET request. The next two lines offer to upgrade the connection to the WebSocket protocol. The Host header lists the server name and port the client wishes to connect to (necessary when the server hosts multiple domains on the same IP address). The Origin header indicates where the request originates from, not unlike the HTTP Referer header. The last two headers and the 8 bytes at the end are random tokens which the server will use to construct a challenge response.

If the server accepts the upgrade offer (based on the resource, host, and origin), it returns a response like this:

Server Response

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Origin: http://www.example.com
Sec-WebSocket-Location: ws://example.com:8080/resource

8jKS'y:G*Co,Wxa-

The first three lines indicate that the server is upgrading the connection to the WebSocket protocol. The Sec-WebSocket-Origin header echoes the value the client sent, and the Sec-WebSocket-Location header returns the full URI for the websocket connection. The 16 bytes at the end form the challenge response. It is calculated by extracting the numeric digits from the client's first Sec-WebSocket-Key header, concatenating them to form a number, and dividing by the number of spaces. The same process is done with the second client key, the two numbers are then concatenated together along with the final 8 bytes of the client request, and the MD5 sum of the result becomes the server challenge response. Whew.

Here's a snippet from the server code which calculates the security response:

private function handleSecurityKey($key) {
    // extract the numeric digits to form a number
    preg_match_all('/[0-9]/', $key, $number);
    // extract the number of spaces
    preg_match_all('/ /', $key, $space);
    if ($number && $space) {
        // perform the calculation per the websocket spec
        return implode('', $number[0]) / count($space[0]);
    }
    return '';
}

private function getHandshakeSecurityKey($key1, $key2, $code) {
    return md5(
        pack('N', $this->handleSecurityKey($key1)).
        pack('N', $this->handleSecurityKey($key2)).
        $code,
        true
    );
}

At any point during this handshake process, if the client or the server does not receive the expected response, it will refuse the upgrade simply by closing the connection.

WebSocket API

Inside a compatible browser, you create and interact with WebSockets via a WebSocket object. According to the WebSocket API, creating a new WebSocket object causes the browser to attempt a connection to the server. The server address is specified in the same manner as a HTTP URI except “ws” replaces “http” and “wss” replaces “https”. Once created, the WebSocket object exposes four callback functions for handling events, a function for sending messages, another for closing the connection and a readyState property for determining the status of the connection.

// connect to server
var websocket = new WebSocket("ws://example.com:8080/resource");

// called when the connection is established
websocket.onopen = function(evt) { ... };

// called when a message is received from the server
websocket.onmessage = function(evt) { ... };

// called when an error occurs
websocket.onerror = function(evt) { ... };

// called when the connection is closed (by either side)
websocket.onclose = function() { ... };

// send a message to the server
websocket.send(data);

// close the connection
websocket.close();

// status of the connection
// 0 == CONNECTING
// 1 == OPEN
// 2 == CLOSING
// 3 == CLOSED
var state = websocket.readyState;

Data Transport

Once the connection between client and server has been established successfully, it is held open and becomes a two way TCP socket, allowing either side to send messages at will.

Each message consists of a 0x00 byte followed by UTF-8 data followed by a 0xFF byte. Any deviation from this, for example missing start or end byte markers or the data is not UTF-8, results in an error and the termination of the connection. The WebSocket protocol currently does not support binary data streams.

On the client side, each message received triggers the onmessage handler function, allowing client code to react appropriately.

The Client

Now it’s time to dive into the client code. The html is really simple:

<html>
<head>
  <title>HTML5 Chat Demo</title>
<body>

<h3>HTML 5 Chat Demo</h3>

<h4 id="error" style="color:#ff0000"></h4>
<div id="console" style="height:300px; overflow:scroll; border:1px solid black"></div>
<div style="padding-top:10px">
    <label>Handle:</label><input id="handle" size="25" value="">
    <button id="connect">Connect</button>
    <button id="disconnect">Disconnect</button><br>
    <label>Message:</label><input id="message" size="25" value="">
    <button id="send">Send</button>
</div>


</body>
</html>

We have a large div to represent the chat window. Below it is a text box to hold the user’s chat handle, two buttons for connecting and disconnecting to the server, followed by another text box for the user’s current message, and a send button. The actual work is done via javascript.

First we need to see if the browser supports WebSockets, if not, we display an error message:

// test for websocket support
if ("WebSocket" in window) {

    // websockets supported!

} else { // no websocket support
    $('error').innerHTML = "Your browser does not appear to support WebSockets";
}

Once compatibility has been determined, we can attempt to make a connection and hook up the handler functions.

// creating a WebSocket object causes it
// to connect to the server
websocket = new WebSocket(server);

// called once the connection is established
websocket.onopen = function(evt) {
    $('console').innerHTML += "CONNECTED<br>";
};

// called upon receipt of a message
websocket.onmessage = function(evt) {
    $('console').innerHTML += "RECEIVED: "+evt.data+"<br>";
};

// called when an error occurs
websocket.onerror = function(evt) {
    $('console').innerHTML += "ERROR: "+evt.data +"<br>";
};

// called when the connection is closed (by either side)
websocket.onclose = function() {
    $('console').innerHTML += "CLOSED<br>";
};

The Server

The server application needs to listen on a given port for incoming connections, offer to upgrade them from HTTP to WebSockets, perform the handshake correctly, and finally perform any logic specific to your application. In this case, it forwards messages between connected clients. In PHP, this requires a number of calls to the various socket_* functions. This example is pretty light on proper error handling, and uses code adapted from here and here

First we need to create a socket to listen on. This initial socket needs to be saved as it is required for getting the socket objects for any clients that connect.

protected function connectMaster() {
    // create a server socket.  AF_INET = IPv4, SOCK_STREAM + SOL_TCP = TCP/IP
    $this->master = socket_create(AF_INET, SOCK_STREAM, SOL_TCP) or die("socket_create() failed");
    // reuse local addresses
    socket_set_option($this->master, SOL_SOCKET, SO_REUSEADDR, 1) or die("socket_option() failed");
    // bind the IP address to the socket
    socket_bind($this->master, $this->address, $this->port) or die("socket_bind() failed");
    // listen for incoming connections (max of 20 pending)
    socket_listen($this->master, 20) or die("socket_listen() failed");
    // save the socket object
    $this->sockets[] = $this->master;

    $this->log("Websocket server started : ". date('Y-m-d H:i:s'));
    $this->log("Listening on             : ". $this->address ." port ". $this->port);
    $this->log("Master socket            : ". $this->master);
    $this->log("CTRL-C to quit");

    return $this->master;
}

Now we enter the main server run loop. It waits around for any of its open sockets to have data. If it’s the master socket, it means we have a new connection to process. If it’s any other socket, we send the handshake if necessary, otherwise, we pass the message to our callback function.

public function run() {
    // main run loop
    while(true){
        $changed = $this->sockets;
        // wait for any of the sockets to change (have data to read)
        // see http://www.php.net/manual/en/function.socket-select.php
        if (false === socket_select($changed, $write=NULL, $except=NULL,NULL)) {
            die("socket_select() failed: ". socket_strerror(socket_last_error()));
        }
        foreach($changed as $socket){
            if ($socket == $this->master) {
                // new incoming connection, time to accept
                $this->log("Incoming connection request");
                $client = socket_accept($this->master);
                if($client === false){ $this->log("socket_accept() failed: ". socket_strerror(socket_last_error())); continue; }
                else{ $this->connect($client); }
            } else {
                // existing connection has data to read
                $bytes = @socket_recv($socket, $buffer, 2048, 0);
                // close the connection if there was an error or no data to read
                if($bytes == 0) {
                    $this->disconnect($socket);
                } else {
                    $user = $this->getUserBySocket($socket);
                    // perform the WebSocket upgrade HTTP handshake if needed
                    if(! $user->handshake) {
                        $this->doHandshake($buffer, $socket);
                        $user->handshake = true;
                    } else {
                        $this->log("< ". $this->unwrap($buffer));
                        $user->lastAction = time();
                        // call the callback function
                        if ($this->callback) {
                            call_user_func($this->callback, $user, $this->unwrap($buffer), $this);
                        }
                    }
                }
            }
        }
    }
}

Here’s the simple callback function which passes the message along to the other clients so that they can display our chat messages:

// callback function
function process($sourceUser, $msg, $server){

    // echo the data back to all connected users
    foreach($server->getUsers() as $user){
        $server->send($user->socket, $msg);
    }
}

Because PHP is not normally used as an infinitely running process, we have to adjust the time limit config as well as turn off the buffering subsystem to get this to work smoothly without timing out.

// show us all errors
error_reporting(E_ALL | E_NOTICE);
// keep running until script aborts or we're killed
set_time_limit(0);
// flush output buffering
ob_implicit_flush();

And there you have the basics of a simple websocket server and client.

Limitations of WebSockets

Currently, browsers based on recent versions of webkit (Safari & Chrome but not iPhone) support draft76[the latest draft] of the WebSocket spec, but as a draft, it’s a bit of a moving target.

As Greg Wilkins discusses, there are some fundamental difficulties writing a robust application based on the current WebSocket draft specification. In particular, since any problem encountered by client or server results in the connection closing, it is impossible for the client to distinguish errors from failed connections from time outs or transient network issues as all result in a single call to onclose().

Also, as discussed here, even when using port 80, non-WebSocket aware proxies often terminate WebSocket handshakes or connections preventing clients behind the proxy from communicating with the WebSocket server.

Hopefully, future drafts of the spec will address these issues.

Putting It All Together

To see the chat client in action, click here.

The full client and server source code is available as a download:

phpwebsocket.zip[5.4k]

To launch the server, use something like this:

php server/server.php 127.0.0.1 8080

Edit the server address in the client/index.html code to match your own server.

// the websocket object
var websocket;

// our chat handle
var handle = "";

// the address of the websocket server
var server = "ws://chat.dev.weevilgenius.net:1212/chat_demo";


function init()
{

Setup

This week you will need to host your files on a PHP server with curl support enabled. We will be using the following PHP script to act as an ajax proxy when communicating with Twitter. This is done to get around the same origin browser security policy which prevents scripts from accessing a server other than that which hosts the page containing the script. There are other techniques for this which were discussed in some detail in the chat room.

curl.php

<?php
$apiCall = isset($_GET['apiCall']) ? $_GET['apiCall'] : false;
if (!$apiCall) {
    header("HTTP/1.0 400 Bad Request");
    echo "Failed because apiCall parameter is missing";
    exit();
}

$url = urldecode($apiCall);

// Open the Curl session
$session = curl_init($url);
// Don't return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);

// Make the call
$json = curl_exec($session);

// The web service returns JSON
header("Content-Type:  application/json");

echo $json;
curl_close($session);

You can determine whether your server supports the curl extension by looking at the output of this PHP snippet:

<?php phpinfo();

You should see a section titled “curl”.

Section 1

Time to add the javascript necessary to fetch tweets asynchronously. This is standard stuff for current websites and the browser specific details would normally be handled for you by a javascript framework such as jQuery. None of this code is specific to HTML 5.

function pageLoaded() {
    //showProfile('JakeCarter');
    getTweets();
}

function getTweets() {
    //console.log('getTweets()');

    var client = new XMLHttpRequest();
    client.onreadystatechange = function() {
        if (this.readyState == XMLHttpRequest.prototype.DONE && this.status == 200) {
            console.log("Got tweets? Yup!");
            // So far so good.
            // TODO: Add code to convert JSON to JS Object

            // TODO: Loop through results and Create List Items for our list.
        }
    };
    var apiCall = 'http://search.twitter.com/search.json?q=OReillyMedia';
    var requestString = '../../../shared/scripts/curl.php?apiCall=' + escape(apiCall);

    client.open('GET', requestString, true);
    client.send();
}

To view the javascript console, use Firebug in Firefox, or the Web Inspector in webkit browsers (Safari & Chrome/Chromium). Here’s what the page looks like in Chromium.

Section 2

Now we need to actually parse the data returning from Twitter so that we can display it appropriately on our page. Because a JSON string is valid javascipt, we can use the javascript built-in eval() function to convert it directly to the equivalent javascript objects. In any kind of production system, you should not use eval() and instead use a proper JSON parser as otherwise you have a security risk of running untrusted or unintentional code. The rest of the new code simply creates some new DOM elements for displaying the twitter data and adds them to the page.

function getTweets() {
    //console.log('getTweets()');

    var client = new XMLHttpRequest();
    client.onreadystatechange = function() {
        if (this.readyState == XMLHttpRequest.prototype.DONE && this.status == 200) {
            // So far so good.
            var tweetsList = document.getElementById('tweetsList');
            var response = eval('(' + this.responseText + ')');
            for (var resultId in response.results) {
                var result = response.results[resultId];

                // Get info we'll use.
                var name = result.from_user;
                var text = result.text;
                var imageUrl = result.profile_image_url;

                // Create Image Element
                var imageElement = document.createElement('img');
                imageElement.setAttribute('src', imageUrl);
                imageElement.setAttribute('alt', name)
                imageElement.setAttribute('class', 'user-image');
                // imageElement.setAttribute('onClick', "showProfile('" + name + "')");

                // Create username span
                var usernameSpan = document.createElement('span');
                usernameSpan.setAttribute('class', 'username');
                usernameSpan.innerText = name;

                // Create message span
                var messageSpan = document.createElement('span');
                messageSpan.setAttribute('class', 'message');
                messageSpan.innerText = text;

                // Create container div
                var div = document.createElement('div');
                div.appendChild(imageElement);
                div.appendChild(usernameSpan);
                div.appendChild(messageSpan);

                // Create List Item
                var li = document.createElement('li');
                li.appendChild(div);

                // Add List Item to List
                tweetsList.insertBefore(li, tweetsList.firstChild);
            }
        }
    };

Here’s what it looks like on an Android device:

Section 3

Now that we’re getting back real tweets, let’s add a click handler to display detailed profile information. It will make another call to Twitter.

// Create Image Element
var imageElement = document.createElement('img');
imageElement.setAttribute('src', imageUrl);
imageElement.setAttribute('alt', name)
imageElement.setAttribute('class', 'user-image');
imageElement.setAttribute('onClick', "showProfile('" + name + "')");

function getProfileForScreenName(screenName) {
    var client = new XMLHttpRequest();
    client.onreadystatechange = function() {
        if (this.readyState == XMLHttpRequest.prototype.DONE && this.status == 200) {
            var profile = eval('(' + this.responseText + ')');

            // Get HTML Elements.
            var profileImage = document.getElementById('profileImage');
            profileImage.setAttribute('src', profile.profile_image_url);
            profileImage.setAttribute('alt', profile.name);

            var name = document.getElementById('name');
            name.innerText = profile.name;

            var screenName = document.getElementById('screenName');
            screenName.innerText = "@" + profile.screen_name;

            var location = document.getElementById('location');
            location.innerText = profile.location;

            var statusText = document.getElementById('statusText');
            statusText.innerText = profile.status.text;

            var followersCount = document.getElementById('followersCount');
            followersCount.innerText = "Followers: " + profile.followers_count;

            var friendsCount = document.getElementById('friendsCount');
            friendsCount.innerText = "Following: " + profile.friends_count;

            switchToSectionWithId('profile');
        }
    };
    var apiCall = 'http://api.twitter.com/1/users/show.json?screen_name=' + screenName;
    var requestString = '../../../shared/scripts/curl.php?apiCall=' + escape(apiCall);

    client.open('GET', requestString, true);
    client.send();
}

function showProfile(username) {
    clearAllNavItems();
    getProfileForScreenName(username);
}

While we’re at it, we can use a CSS3 transition to animate the display of the profile view. This transition will apply to the opacity of the section elements.

section {
    -webkit-transition-property: opacity;
    -webkit-transition-delay: 0.5s;
    -webkit-transition-duration: 0.5s;
    -webkit-transition-timing-function: ease;
    z-index: 0;
    position: absolute;
    top: 88px;
    width: 100%;
    background-color: #ffffff;
    display: block;
    opacity: 0;
}
section.selected {
    z-index: 100;
    opacity: 1;
}

Here’s what the profile view looks like on the device.

Final Section

Now let’s use a webfont for a custom look. Before webfonts, you could only specify CSS fonts which you hoped the user had installed. Now we can specify a custom font and provide the file for the browser to use. At the moment, this only works in the Android browser, not the iPhone.

@font-face {
    font-family: Chunk;
    src: url('../../../shared/fonts/chunk/Chunk.otf') format('opentype');
}
/*
    Header
*/
header {
    width: 100%;
    height: 44px;
    background-image: -webkit-gradient(
        linear,
        0% 0%,
        0% 100%,
        from(#666666),
        to(#666666),
        color-stop(.5,#333333)
    );
    font-family: Chunk, Helvetica;

    -webkit-box-sizing: border-box;

Here’s the page at the end of week 3 with the custom font.

Useful Links

Class source code download: http://examples.oreilly.com/0636920014225
Class forum: http://forums.oreilly.com/category/104/HTML5-Mobile-Web-Development/

These came up in the chat channel during this week’s session

• http://rumpetroll.com/
• http://html5demos.com/
• JSONP on Twitter
• JSON parser
• Google’s closure minifier
• jQuery Mobile
• jqTouch
• League of Moveable Type
• Font Squirrel
• http://paulirish.com/2009/bulletproof-font-face-implementation-syntax/
• http://github.com/madrobby/zepto

Considerations

There are a number of different things I could do to save a copy of my files:

• Use rsync to create a remote directory copy
• Use cp to create a remote directory copy
• Use tar to create a remote file copy
• Use tar + gzip (or bzip2) to create a remote compressed file copy
• Create a ext2/3/4 filesystem inside a remote file using the loopback device
• Use one of the many backup utilities available for linux

Why did I choose Squashfs? Well, since I’m going to be copying the majority of a running filesystem, any of the methods which create a remote directory copy will be filled with thousands of files and directories. I don’t really like that kind of mess unless the external disk is dedicated. Tar+gzip gets me a single file and compression too, but it’s a pain to search as the whole thing would need to be untarred before use. Creating a read/write filesystem in a file is really useful, but I’m happy to trade write ability for compression. I don’t really need a backup utility since this is a one time operation.

Squashfs is in widespread use as the primary filesystem for many live CD linux distributions, and the linux kernel has included support since version 2.6.29. On Ubuntu, the user level tools can be installed via the squashfs-tools package.

Squash It

Enough talk, time to put this into practice. First, we need to verify that our kernel has the squashfs module available:

$ modinfo squashfs
filename:       /lib/modules/2.6.28-19-generic/kernel/ubuntu/squashfs/squashfs.ko
license:        GPL
author:         Phillip Lougher <phillip@lougher.demon.co.uk>
description:    squashfs 3.2-r2-CVS, a compressed read-only filesystem
srcversion:     7626F567E1FBB5EDCFF9F67
depends:
vermagic:       2.6.28-19-generic SMP mod_unload modversions 586

Good, we’ve got the kernel module. Now we need to install the userland tools.

$ sudo apt-get install squashfs-tools

Now let’s create a little squashfs file to verify that everything works. I’m going to use one the linux kernel source directories as a test.

$ mksquashfs /usr/src/linux-headers-2.6.28-19 test.sqsh
Parallel mksquashfs: Using 2 processors
Creating little endian 3.1 filesystem on test.sqsh, block size 131072.
[============================================================================================] 8848/8848 100%
Exportable Little endian filesystem, data block size 131072, compressed data, compressed metadata, compressed fragments, duplicates are removed
Filesystem size 8196.05 Kbytes (8.00 Mbytes)
        23.39% of uncompressed filesystem size (35034.82 Kbytes)
Inode table size 94740 bytes (92.52 Kbytes)
        29.11% of uncompressed inode table size (325410 bytes)
Directory table size 89099 bytes (87.01 Kbytes)
        57.22% of uncompressed directory table size (155701 bytes)
Number of duplicate files found 202
Number of inodes 10353
Number of files 8842
Number of fragments 284
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 1511
Number of uids 1
        root (0)
Number of gids 0

That was easy. Now let’s mount the file.

$ mkdir ~/squash_test
$ sudo mount -t squashfs -o loop test.sqsh ~/squash_test

Let’s compare the contents to the original source:

$ ls -l ~/squash_test
total 59
drwxr-xr-x 24 root root   358 2010-09-20 09:20 arch/
drwxr-xr-x  2 root root    57 2010-09-20 09:20 block/
drwxr-xr-x  3 root root    50 2010-09-20 09:20 crypto/
drwxr-xr-x 19 root root   223 2010-09-20 09:20 Documentation/
drwxr-xr-x 83 root root   889 2010-09-20 09:20 drivers/
drwxr-xr-x  2 root root    25 2010-09-20 09:20 firmware/
drwxr-xr-x 62 root root   653 2010-09-20 09:20 fs/
drwxr-xr-x 27 root root   351 2010-09-20 09:20 include/
drwxr-xr-x  2 root root    37 2010-09-20 09:20 init/
drwxr-xr-x  2 root root    25 2010-09-20 09:20 ipc/
-rw-r--r--  1 root root  2430 2008-12-24 15:26 Kbuild
drwxr-xr-x  6 root root   134 2010-09-20 09:20 kernel/
drwxr-xr-x  6 root root   131 2010-09-20 09:20 lib/
-rw-r--r--  1 root root 57730 2010-09-16 07:13 Makefile
drwxr-xr-x  2 root root    37 2010-09-20 09:20 mm/
drwxr-xr-x 45 root root   477 2010-09-20 09:20 net/
drwxr-xr-x  6 root root    89 2010-09-20 09:20 samples/
drwxr-xr-x 10 root root  1057 2010-09-20 09:20 scripts/
drwxr-xr-x  6 root root    81 2010-09-20 09:20 security/
drwxr-xr-x 20 root root   205 2010-09-20 09:20 sound/
drwxr-xr-x 21 root root   257 2010-09-20 09:20 ubuntu/
drwxr-xr-x  2 root root    37 2010-09-20 09:20 usr/

$ ls -l /usr/src/linux-headers-2.6.28-19
total 148
drwxr-xr-x 24 root root  4096 2010-09-20 09:20 arch/
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 block/
drwxr-xr-x  3 root root  4096 2010-09-20 09:20 crypto/
drwxr-xr-x 19 root root  4096 2010-09-20 09:20 Documentation/
drwxr-xr-x 83 root root  4096 2010-09-20 09:20 drivers/
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 firmware/
drwxr-xr-x 62 root root  4096 2010-09-20 09:20 fs/
drwxr-xr-x 27 root root  4096 2010-09-20 09:20 include/
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 init/
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 ipc/
-rw-r--r--  1 root root  2430 2008-12-24 15:26 Kbuild
drwxr-xr-x  6 root root  4096 2010-09-20 09:20 kernel/
drwxr-xr-x  6 root root  4096 2010-09-20 09:20 lib/
-rw-r--r--  1 root root 57730 2010-09-16 07:13 Makefile
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 mm/
drwxr-xr-x 45 root root  4096 2010-09-20 09:20 net/
drwxr-xr-x  6 root root  4096 2010-09-20 09:20 samples/
drwxr-xr-x 10 root root  4096 2010-09-20 09:20 scripts/
drwxr-xr-x  6 root root  4096 2010-09-20 09:20 security/
drwxr-xr-x 20 root root  4096 2010-09-20 09:20 sound/
drwxr-xr-x 21 root root  4096 2010-09-20 09:20 ubuntu/
drwxr-xr-x  2 root root  4096 2010-09-20 09:20 usr/

Perfect! How much space did we save?

$ ls -lh ~/test.sqsh
-rwx------ 1 bstump bstump 8.1M 2010-10-18 15:34 /home/bstump/test.sqsh*

$ du -hs /usr/src/linux-headers-2.6.28-19
64M     /usr/src/linux-headers-2.6.28-19

In this example, the source directory is about 64 MB versus only 8 MB for the compressed version. That’s 8:1 compression. However, the linux kernel is highly compressible source code. Typical compression for mixed files is about 2:1.

Unmount the file when we’re done.

$ sudo umount ~/squash_test

Now I’m ready to back up my primary file system to a single archive file in preparation for a system update.

This week starts our construction of “Tweetstr”, a mobile web twitter client. We create and edit the basic HTML and CSS files to display the home screen while exploring some new HTML 5 and CSS 3 tags as well as some settings specific to Apple devices such as the iPhone.

Section 1

We start with some basic html and css files.

index.html

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <meta http-equiv="Content-Language" content="en"/>

        <!-- TODO: Set Viewport -->

        <title>Tweetstr</title>

        <!-- TODO: Add CSS Reset -->

        <link rel="stylesheet" href="style.css" type="text/css" media="screen" charset="utf-8">

        <script type="text/javascript" charset="utf-8">
            function navSelected(navItem) {
                // First we need to clear all other nav items.
                clearAllNavItems();

                // Now give navItem the selected css class.
                event.target.setAttribute('class', 'selected');
            }

            function clearAllNavItems() {
                var navList = document.getElementById('navList');

                for (var i = 0; i < navList.children.length; i++) {
                    var li = navList.children[i];
                    li.setAttribute('class', '');
                }
            }
        </script>
    </head>
    <body>
        <header>
            Tweetstr
        </header>
        <nav>
            <ul id='navList'>
                <li id='liHome' onclick='navSelected()' class='selected' ></li>
                <li id='liMentions' onclick='navSelected()'></li>
                <li id='liFaves' onclick='navSelected()'></li>
                <li id='liMessages' onclick='navSelected()'></li>
                <li id='liSearch' onclick='navSelected()'></li>
            </ul>
        </nav>
        <section id='home'>
            <ul class='tweets'>
                <li>
                    <div>
                        <img class='user-image' alt='JakeCarter'
                             src='user-image.png' />
                        <span class='username'>JakeCarter</span>
                        <span class='message'>I love twitter!</span>
                    </div>
                </li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
                <li></li>
        </section>
    </body>
</html>

style.css

body {
    font-family: "Helvetica";
}
/*
    Header
*/
header {
    width: 100%;
    height: 44px;

    text-align: center;
    padding-top: 10px;

    font-weight: bold;
    font-size: 1.5em;
}

/*
    Nav
*/
nav {
    height: 44px;
    width: 100%;
}
nav ul {
    height: 100%;
    width: 320px;
    margin-left: auto;
    margin-right: auto;
}
nav ul li {
    display: inline-block;
    float: left;
    width: 64px;
    height: 100%;
    border: 1px solid #999999;

    background-repeat: no-repeat;
    background-position: center center;
}

/*
    List
*/
section#home ul.tweets li {
    min-height: 52px;
}

section#home ul.tweets li div {
    padding: 2px;
}

section#home ul.tweets li div img {
    float: left;
    padding-right: 5px;
}

Notice how simple the doctype tag is for HTML 5:

<!-- HTML 4 -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<!-- HTML 5 -->
<!DOCTYPE html>

Also, we’re using some new HTML 5 semantic tags, <header>, <nav>, and <section> instead of generic <div> tags as with HTML 4.

<body>
    <header>
        Tweetstr
    </header>
    <nav>
        <ul id='navList'>
            <li id='liHome' onclick='navSelected()' class='selected' ></li>
            <li id='liMentions' onclick='navSelected()'></li>
            <li id='liFaves' onclick='navSelected()'></li>
            <li id='liMessages' onclick='navSelected()'></li>
            <li id='liSearch' onclick='navSelected()'></li>
        </ul>
    </nav>
    <section id='home'>

By the end of section 1, the application looks like this:

The browser zooms way out, the nav items don’t fit on a single line, and we don’t have much styling yet.

Section 2

Let’s correct that zoom problem by setting the viewport.

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Language" content="en"/>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" />

    <title>Tweetstr</title>

The viewport is a meta tag which Apple introduced for mobile Safari which tells the browser what zoom level (or “scale”) should be shown for the page and whether the user can zoom in or out when viewing that page. You can set width to either a fixed number of pixels or device-width (recommended). Viewport settings are also supported by Android and mobile Firefox.

We’ll also add a HTML 5 compatible reset stylesheet. This “resets” various styles to sane, known values for most of the tags supported by HTML 5. Jake has provided the one from HTML5 doctor.

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Language" content="en"/>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" />

    <title>Tweetstr</title>

    <link rel="stylesheet" href="html5reset.css" type="text/css" media="screen" charset="utf-8" />
    <link rel="stylesheet" href="style.css" type="text/css" media="screen" charset="utf-8">

It needs to be added before the existing stylesheet as they are processed in order from top to bottom.

Here’s how the page looks by the end of section 2:

Now the zoom level is looking correct, and we have sane CSS values for our new HTML 5 tags, but we still have lots of clean up to go.

Section 3

Each of the five nav boxes has been set to a width of 64 pixels in the style.css file. 64 * 5 = 320, which is the width of the basic iPhone screen, so why do they not all fit on a single row like they’re supposed to? It’s because browsers by default uses the “content box” sizing model, where the given width and height are applied to the content with borders and padding added after. This makes our 64 pixel boxes take up more than 64 pixels.

CSS3 adds the ability to specify an alternate box sizing model “border box” where the width applies to the entire visible content, with the content area shrinking to accommodate the space required for borders and padding. See this for a more lengthy discussion on the topic.

Although CSS3 is not a final specification, we can use browser specific CSS styles to control the sizing model.

/* CSS3 official */
box-sizing: border-box;
box-sizing: content-box;

/* webkit (Chrome/Safari) */
-webkit-box-sizing: border-box;
-webkit-box-sizing: content-box;

/* Mozilla (Firefox) */
-moz-box-sizing: border-box;
-moz-box-sizing: content-box;

/* IE 8 */
-ms-box-sizing: border-box;
-ms-box-sizing: content-box;

Let’s fix up our boxes by requesting the border box sizing model:

header {
    width: 100%;
    height: 44px;

    -webkit-box-sizing: border-box;

    text-align: center;
    padding-top: 10px;

    font-weight: bold;
    font-size: 1.5em;
}

nav ul li {
    display: inline-block;
    float: left;
    width: 64px;
    height: 100%;
    border: 1px solid #999999;

    -webkit-box-sizing: border-box;

    background-repeat: no-repeat;
    background-position: center center;
}

Here’s what the page looks like at the end of section 3.

That looks better. All of the nav boxes fit on a single line.

Section 4

Before CSS3, using a gradients on the web meant creating image slices with the required colors and using the CSS background-repeat declaration. Lots of work and more importantly, lots of files for the browser to download. CSS3 allows us to create gradients through pure CSS without involving separate images.

Here Jake adds a gradient to the header.

header {
    width: 100%;
    height: 44px;
    background-image: -webkit-gradient(
        linear,
        0% 0%,
        0% 100%,
        from(#666666),
        to(#666666),
        color-stop(.5,#333333)
    );

    -webkit-box-sizing: border-box;

    text-align: center;
    color: #ffffff;
    padding-top: 10px;

    font-weight: bold;
    font-size: 1.5em;
}

And the nav:

nav {
    height: 44px;
    width: 100%;
    background-image: -webkit-gradient(
            linear,
            0% 0%,
            0% 100%,
            from(#aaaaaa),
            to(#aaaaaa),
            color-stop(.5,#cccccc)
        );
}

And a contrasting gradient for the selected state on each nav button:

#liHome.selected {
    background-image: -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liMentions.selected {
    background-image: -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liFaves.selected {
    background-image: -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liMessages.selected {
    background-image: -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liSearch.selected {
    background-image: -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

Here’s what the page looks like after adding the gradients.

Section 5

Now let’s add the images which will form the face of our nav buttons. Notice how in the selected state, each box has two background images set, one for the gradient and one for the icon. The order is important here.

#liHome {
    background-image: url('../../../shared/images/icons/icons/53-house.png');
}
#liHome.selected {
    background-image: url('../../../shared/images/icons/icons/53-house.png'),
        -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liMentions {
    background-image: url('../../../shared/images/icons/icons/09-chat2.png');
}
#liMentions.selected {
    background-image: url('../../../shared/images/icons/icons/09-chat2.png'),
        -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liFaves {
    background-image: url('../../../shared/images/icons/icons/28-star.png');
}
#liFaves.selected {
    background-image: url('../../../shared/images/icons/icons/28-star.png'),
        -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liMessages {
    background-image: url('../../../shared/images/icons/icons/08-chat.png');
}
#liMessages.selected {
    background-image: url('../../../shared/images/icons/icons/08-chat.png'),
        -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

#liSearch {
    background-image: url('../../../shared/images/icons/icons/06-magnifying-glass.png');
}
#liSearch.selected {
    background-image: url('../../../shared/images/icons/icons/06-magnifying-glass.png'),
        -webkit-gradient(
                linear,
                0% 0%,
                0% 100%,
                from(#dddddd),
                to(#dddddd),
                color-stop(.5,#cccccc)
            );
}

Here are the nav buttons with their nice icons:

Section 7

Another nifty CSS3 feature is the new nth-child selector. Without this, if you wanted to style every nth row of a list or table with a distinctive look, you were forced to apply different classes to them, typically with javascript or a server side language like PHP. CSS3 lets you select and style nth rows directly from your stylesheet.

Here Jake gives every other (odd) row a gradient.

section#home ul.tweets li:nth-child(odd) {
    background-image:  -webkit-gradient(
        linear,
        0% 0%,
        0% 100%,
        from(#aaaaaa),
        to(#aaaaaa),
        color-stop(.5,#cccccc)
    );
}

In addition to the shortcuts “odd” and “even”, you can put in a custom formula. For example, if you wanted to style every 3rd row, starting from row 2, you could use this: :nth-child(3n+2). Powerful stuff.

Here’s what it looks like in the browser:

Final Section

The page is starting to look like a proper mobile application. For the final section of this week, we’re going to add some Apple-specific fields to tell iOS devices some additional information about how the application should behave.

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Language" content="en"/>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" />

    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="black">

    <title>Tweetstr</title>

    <link rel="apple-touch-icon" href="../../../shared/images/tweetstr-icon.png"/>

    <link rel="stylesheet" href="html5reset.css" type="text/css" media="screen" charset="utf-8" />
    <link rel="stylesheet" href="style.css" type="text/css" media="screen" charset="utf-8">

The first one, apple-mobile-web-app-capable determines whether the page can be treated as a full screen application (without the Safari URL bar and navigation buttons) when launched from the home screen. To get a web page to launch from the home screen on iOS devices, open it in mobile Safari, click the “+” bookmark button and choose “Add to Home Screen”. A web page saved this way with the meta tag set will launch full screen. Without it, you get the normal mobile Safari UI. The sugested name of the application short cut on the home screen comes from the title of the web page.

The next Apple specific tag is apple-mobile-web-app-status-bar-style which controls the appearance of the status bar for full screen web apps when launched from the home screen. It can be set to “default”, “black” or “black-translucent”. Here we use black.

The last meta tag we added sets the icon used on the device’s home screen for the web app. Without this tag, iOS will save a miniature screenshot of the web page as the application icon. This version of the tag allows iOS to add rounded corners and that Apple glossy highlight to the icon. If you don’t want this (or your icon already has these effect applied), use apple-touch-icon-precomposed instead.

One additional Apple tag which Jake didn’t cover is apple-touch-startup-image which allows you to specify an image to display as a splash screen while your web application is loading. Because of the varying screen resolutions even across Apple devices, this is not as useful as the other tags.

Here is the Add to Home screen for Tweetstr:

And here is the icon on the home screen:

And here it has been launched as a full screen web application. Notice there is no Safari UI and the title bar is now black.

That’s all until next week.

Useful Links

Class source code download: http://examples.oreilly.com/0636920014225
Class forum: http://forums.oreilly.com/category/104/HTML5-Mobile-Web-Development/

These came up in the chat channel during this week’s session

• mobile bookmark bubble - opensource code created by Google to remind users to add your web app to the home screen
• retina display - how to target the iPhone 4 retina display with a different CSS stylesheet
• Apple web app meta tags - Apple specific meta tags for mobile apps
• Open App Market – marketplace for mobile apps

An easy solution for one or two files is simply to invoke vim with the -n option which tells vim to not use any swap files for that session.

vim -n foo.txt

But what I want is a more permanent solution that doesn’t eliminate swap files but gets them out of the directories of the files I’m editing. To do this, first create an empty directory; I used $HOME/.vim_swap so that it would be in my home directory but hidden from normal directory listings. Open up your .vimrc file and add the following line, adjusting to match the directory you created.

set directory^=$HOME/.vim_swap//   "put all swap files together in one place

The directory setting is a comma separated list of directories where vim should place swapfiles. Each path is tried in order and the first one that doesn’t fail (because it doesn’t exist or for permissions reasons) is where vim will create swapfiles. By default on unix systems, it it set to “.,~/tmp,/var/tmp,/tmp” which means it first tries “.” or the same directory as the file being edited. By using ^=, we tell vim to prepend the given value to those already set (the defaults). This allows vim to fall back to other directories should the specified one fail for some reason (directory doesn’t exist or disk is full for example). The double slash at the end tells vim to form swap file names using the full path of the edited file, replacing / with %, avoiding name conflicts. See :help directory for more details.

Save the .vimrc file and you’re done. All new vim sessions will now attempt to use the new directory for swap files. You can see the new value by typing :set directory? in a new vim session and you can verify that swap files are being put in the right place by doing a directory listing of the new folder.

$ ls ~/.vim_swap
%home%bstump%Dropbox%blog%vim_swapfiles.txt.swp
%home%bstump%.vimrc.swp

Now you have full swap file support in vim without cluttering up source directories and confusing sync programs like Dropbox.

Sample Data

For the examples below, I’m using the “world” database which you can download from the MySQL developer website. It contains city, country, and language data for many parts of the world, and is simple to understand, while having enough data for the query plans to be realistic.

The database is organized into three tables, City, Country, and CountryLanguage which look like this:

City Table (4079 rows)

CREATE TABLE City
(
  ID            INT(11)  NOT NULL AUTO_INCREMENT,
  Name          CHAR(35) NOT NULL,
  CountryCode   CHAR(3)  NOT NULL,
  District      CHAR(20) NOT NULL,
  Population    INT(11)  NOT NULL default '0',

  PRIMARY KEY (ID)
) ENGINE=MyISAM CHARSET=latin1

Country Table (239 rows)

CREATE TABLE `Country` (
  Code              CHAR(3)     NOT NULL,
  Name              CHAR(52)    NOT NULL,
  Continent         ENUM('Asia',
                         'Europe',
                         'North America',
                         'Africa',
                         'Oceania',
                         'Antarctica',
                         'South America'
                    )           NOT NULL DEFAULT 'Asia',
  Region            CHAR(26)    NOT NULL,
  SurfaceArea       FLOAT(10,2) NOT NULL DEFAULT '0.00',
  IndepYear         SMALLINT(6)     NULL,
  Population        INT(11)     NOT NULL DEFAULT '0',
  LifeExpectancy    FLOAT(3,1)      NULL,
  GNP               FLOAT(10,2)     NULL,
  GNPOld            FLOAT(10,2)     NULL,
  LocalName         CHAR(45)    NOT NULL,
  GovernmentForm    CHAR(45)    NOT NULL,
  HeadOfState       CHAR(60)        NULL,
  Capital           INT(11)         NULL,
  Code2             CHAR(2)     NOT NULL,

  PRIMARY KEY (Code)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CountryLanguage Table (984 rows)

CREATE TABLE CountryLanguage (
  CountryCode CHAR(3)       NOT NULL,
  Language    CHAR(30)      NOT NULL,
  IsOfficial  ENUM('T','F') NOT NULL DEFAULT 'F',
  Percentage  FLOAT(4,1)    NOT NULL DEFAULT '0.0',

  PRIMARY KEY  (CountryCode, Language)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

Using EXPLAIN

Type “EXPLAIN” followed by any SELECT type query and the server will return a number of useful pieces of information for each table referenced in the query.

• id – server assigned integer (starting from 1) indicating the order in which the query optimizer decided to process the tables. You can force your own order by using the STRAIGHT_JOIN query hint.
• select_type – The type of query. Will be one of the following:
  • SIMPLE – Vanilla query without unions or subqueries
  • PRIMARY – Outermost SELECT query
  • UNION – Second or later SELECT query of a union
  • DEPENDENT UNION – Second or later SELECT query of a union which is dependent on an outer query
  • SUBQUERY – First SELECT query in a subquery
  • DEPENDENT SUBQUERY – First SELECT query in a subquery, dependent on an outer query
  • DERIVED – SELECT query used as a table in a FROM clause
• table – The table name or alias
• type – The access type (sometimes called the join type). This is the single most important piece of information given, as it indicates how efficiently the server was able to retrieve the requested rows of data. I cover each of the access types in detail below.
• possible_keys – Indexes which the server could possibly use to retrieve the requested rows from this table. Prior to version 5.0, MySQL was only able to use a single index per table in a query. 5.0 added the index_merge access type which allows multiple indexes to be used in certain cases.
• key – The index selected (if any) for this table. The index used is chosen based on the estimated amount of work the query optimizer thinks using each index will require. The current query optimizer attempts to minimize disk reads. A NULL value here means no index will be used, indicating a full table scan. You can use the FORCE INDEX, USE INDEX, and IGNORE INDEX query hints to guide the optimizer.
• key_len – The number of bytes of the chosen index that the server will actually use. Mostly useful for determining how many parts of a multi-part index the server used.
• rows – Estimated number of rows the optimizer thinks the server will have to examine in order to retrieve the requested rows. This number is based on table statistics and index selectivity.
• Extra – Additional information pertaining to this table.
  • Distinct – Server is able to perform an optimization where it will stop searching for more rows for the current row combination after it has found the first matching row
  • Not exists – Server is able to do a LEFT JOIN optimization where it will not examine more rows in this table for the current row combination after it finds one row that matches the JOIN criteria.
  • range checked for each record – Server found no good index to use, but checks indexes for each row. Slow, but faster than a full table scan.
  • Using filesort – Server will need to do an extra pass to sort the rows.
  • Using index – Server is able to return rows directly from the index instead of doing an additional seek to read the actual rows. Generally good news.
  • Using temporary – Server will need to create a temporary table to hold results. Often occurs with differing GROUP BY and ORDER BY columns.
  • Using where – Server is using values in the WHERE clause to limit returned rows.

Access Types

Here in order of fastest to slowest are all of the possible access types that MySQL can use when retrieving rows from a table.

system

This is a special case of the const access type where the data source is a system table.

const

This access type is used when the table has a primary key (or unique index) where each part is compared to a fixed value. Only a single row (at most) can match, and therefore the result values can be treated as constants in the rest of the query. This is extremely fast.

EXPLAIN SELECT * FROM City WHERE ID = 1

eq_ref

This access type is used when a join uses all parts of a unique, non-nullable index. A single row (at most) is read from the table for each combination of rows returned from the other joins. This is the most efficient access type for most joins.

EXPLAIN SELECT City.Name, Country.Name AS Country
        FROM City
        INNER JOIN Country ON Country.Code = City.CountryCode
        WHERE City.Name LIKE 'S%';

In this example, the server is able to locate a single row from the Country table via its primary key for each City row returned.

ref

Identical to eq_ref except that one or more rows are read from the table instead of a single row. This happens when either the join condition uses only the left most part of a multicolumn index or the index is not unique but does not contain NULL values.

EXPLAIN SELECT C.Name, L.Language
        FROM Country C
        INNER JOIN CountryLanguage L ON L.CountryCode = C.Code
        WHERE C.Name LIKE 'P%';

In this example, the CountryLanguage table has a two part primary key on (CountryCode, Language), but we are only matching on CountryCode, so multiple rows can be returned for each matching Country.

fulltext

The join is performed using a FULLTEXT index match. Note that FULLTEXT indexes and operations are not available in all MySQL database engines.

ref_or_null

Identical to ref access type but with an extra search for nullability.

ALTER TABLE Country ADD INDEX ix_Country_IndepYear (IndepYear);
EXPLAIN SELECT Name, IndepYear
        FROM Country
        WHERE IndepYear = 1975
          OR IndepYear IS NULL;
ALTER TABLE Country DROP INDEX ix_Country_IndepYear;

For this example, we have to add an index to a nullable column. Pretty much all queries using this access type have a characteristic OR [COLUMN] IS NULL clause in them.

index_merge

Multiple indexes on the same table are used to resolve multiple range conditions, then the results are merged together. Ordinarily, MySQL will use at most one index per table in a query. Using table and index statistics, it selects a single index from those available which it estimates will help the most with the given table and conditions. MySQL 5 added a new access type which can use multiple indexes under certain conditions.

ALTER TABLE Country ADD INDEX ix_Country_Region (Region);
ALTER TABLE Country ADD INDEX ix_Country_Continent (Continent);
EXPLAIN SELECT Name, Continent, Region
        FROM Country
        WHERE Continent = 'Europe'
          OR Region = 'Nordic Countries';
ALTER TABLE Country DROP INDEX ix_Country_Region;
ALTER TABLE Country DROP INDEX ix_Country_Continent;

In this example, we add an index on Continent and another on Region. Before version 5, the server would have been forced to select only one of the indexes to pull rows, resorting to scanning the results to eliminate rows not matching the other clause. Here, however, MySQL is able to use both indexes and merge (union in this case) the results.

unique_subquery

Similar to ref but for subqueries of the type IN (SELECT ... FROM ... WHERE ...) with a unique, non-nullable index. The subquery is completely replaced with an index lookup function.

EXPLAIN SELECT Name
        FROM City
        WHERE CountryCode IN (SELECT Code FROM Country
                              WHERE Continent='North America');

In this example, we use a subquery to gather all the countries in North America in order to limit the cities. While this produces the desired result, it’s more efficiently rewritten as an INNER JOIN like this:

EXPLAIN SELECT Ci.Name
        FROM City Ci
        INNER JOIN Country Co ON Co.Code = Ci.CountryCode
          AND Co.Continent='North America';

This produces the same result but uses the eq_ref access type which is generally faster.

index_subquery

Similar to unique_subquery except that the index is not unique.

ALTER TABLE Country ADD INDEX ix_Country_Code (Code);
EXPLAIN SELECT Name
        FROM City WHERE CountryCode IN (SELECT Code
                                        FROM Country IGNORE INDEX (PRIMARY)
                                        WHERE Continent = 'North America');
ALTER TABLE Country DROP INDEX ix_Country_Code;

Here we have to add a non-unique index to Country.Code and force the query optimizer to ignore the primary key in order to get the index_subquery access type.

range

This access type is used when an index exists for the given column(s) and a range of values is requested. Possible operators include =, <>, >, >=, <, <=, IS NULL, <=>, BETWEEN, LIKE, and IN. Note that range only occurs with LIKE when the first character is not a wildcard.

EXPLAIN SELECT Name FROM City WHERE ID BETWEEN 10 AND 100;
EXPLAIN SELECT Name FROM City WHERE ID < 10;
EXPLAIN SELECT Name FROM City WHERE ID IN (10,20,30,40);

index

A bit of a misnomer, index access type should probably be called index_scan as it results in a full scan of the index tree. This is usually better than a full table scan because the size of the index is usually smaller than the raw data. It occurs when each of the requested columns is contained in a single index, but none of the other more efficient access types apply. It always results in the Extra column of the EXPLAIN output showing “using index”.

EXPLAIN SELECT CountryCode
        FROM CountryLanguage
        WHERE Language = 'English';

Here, even though we only reference columns in the primary key for the CountryLanguage table, it lists CountryCode before Language, so eq_ref is not available, and the engine has to scan the full index to get the values we’ve requested.

ALL

A full table scan is performed for each combination of rows returned from previous tables. This can occur because no index is available for the conditions given, or no WHERE clause is given, or simply because the query optimizer estimates that a full table scan would be less costly than one of the other access methods above (usually only when the number of rows is small or index selectivity is poor). Typically, seeing the ALL access type indicates poor performance, and you can improve performance by adding appropriate indexes or rewriting the query.

EXPLAIN SELECT Code
        FROM Country
        WHERE Continent = 'Asia';

Here we see the ALL access type because there is no index available on the Country.Continent column, and the server is forced to scan each data record looking for matches. If we add an index on the Continent column, we can bump the access type all the way up to ref.

ALTER TABLE Country ADD INDEX ix_Country_Continent (Continent);
EXPLAIN SELECT Code
        FROM Country
        WHERE Continent = 'Asia';
ALTER TABLE Country DROP INDEX ix_Country_Continent;

Here’s a sample of text with the long row highlighting turned off:

And here’s what it looks like when turned on:

Function

" Highlight rows longer than 80 characters
function ToggleOverLengthHi()
    if exists("b:overlengthhi") && b:overlengthhi
        highlight clear OverLength
        let b:overlengthhi = 0
        echo "overlength hilight off"
    else
        " adjust colors/styles as desired
        highlight OverLength ctermbg=darkred gui=undercurl guisp=blue
        " change '81' to be 1+(number of columns)
        match OverLength /\%81v.\+/
        let b:overlengthhi = 1
        echo "overlength hilight on"
    endif
endfunction

Now you can bind (map in vim lingo) the function to an unused key to toggle the highlight. In this example, I’ve used F1 which is normally mapped to :help

Keybind

map <silent> <F1> <Esc>:call ToggleOverLengthHi()<CR>

To use this in your own vim sessions, add the function and the key binding line to your .vimrc file.

A couple of notes: The highlighting only works if you have the syntax highlighting system turned on, for example with syntax on in your .vimrc file. The long row highlighting is off by default, and the toggle is set per buffer, so if you have multiple files open in the same vim session, you can toggle each one separately.

Making the Connection

Suppose you have a web application and you wish to know how much database activity is generated by requesting a single page. Perhaps page response times are slow, or the application uses an ORM layer which you suspect is overly chatty, or you want to see how well object caching works in different scenarios. Time to set up the proxy.

In this example, I’m going to use the mysql command line as the client, and my server is running locally at 127.0.0.1:3306

Download the latest version of the proxy from the MySQL website and unpack it on a machine which can be reached by network from the application you want to test. Now you have to tell it where to listen for incoming connections, via the --proxy-address=host:port parameter. You can omit the host part to bind to all available ip addresses. Since my local server is using the default port of 3306, I’ll use :3307 for the proxy.

Next, you need to tell the proxy where the real server is. You do this via the --proxy-backend-addresses=host:port parameter. The proxy can support multiple back end hosts, just repeat the parameter and change the host:port values. If you use multiple back end hosts, the proxy will connect to each one in round robin fashion, becoming a simple load balancer. In my case, I’ll just point to my local instance of 127.0.0.1:3306.

Time to test the basic proxy. In one window, I launch the proxy, and leave it running (you can terminate the proxy by typing CTRL-C).

mysql-proxy --proxy-address=:3307 --proxy-backend-addresses=127.0.0.1:3306

I can now open another command window to test the proxy with the mysql client:

$ mysql -h 127.0.0.1 -P 3307 -u root -p -A mysql
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.0.75-0ubuntu10.5 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW TABLES;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| func                      |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| host                      |
| proc                      |
| procs_priv                |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
17 rows in set (0.00 sec)

Note that I had to specify 127.0.0.1 as the host in order to get the client to make a network connection to the server (the proxy, in this case), instead of using the system mysql socket. MySQL proxy only works with a network connection.

Basic Logging

So far, my example is pretty useless. Sure, I can connect to the proxy which then relays commands on to the server, but I can’t do anything useful with it yet. The real power of the proxy is the ability to write scripts to inspect, log, redirect, and even manipulate the queries and responses passing through. You invoke this by writing your script using the Lua programming language, and then telling the proxy to run it with the --proxy-lua-script=/path/to/script.lua parameter.

Here is a script which simply prints out each query sent to the server.

-- dump_queries.lua
-- Prints a line for each query received from the client

function read_query(packet)
    if packet:byte() == proxy.COM_QUERY then
        print("Query: " .. packet:sub(2))
    end
end

I save the file as dump_queries.lua, and restart the proxy:

mysql-proxy --proxy-address=:3307 --proxy-backend-addresses=127.0.0.1:3306 --proxy-lua-script=/home/bstump/mysql_proxy/dump_queries.lua

Now I can connect to the proxy and make some queries:

$ mysql -h 127.0.0.1 -P 3307 -u root -p -A mysql
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 5.0.75-0ubuntu10.5 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SELECT NOW();
+---------------------+
| NOW()               |
+---------------------+
| 2010-07-08 11:07:43 |
+---------------------+
1 row in set (0.00 sec)

mysql> SELECT COUNT(*) FROM user;
+----------+
| COUNT(*) |
+----------+
|        6 |
+----------+
1 row in set (0.00 sec)

In the proxy window, you can see all of the queries, including one at the top which the client sent behind the scenes when connecting to the server.

Query: select @@version_comment limit 1
Query: SELECT NOW()
Query: SELECT COUNT(*) FROM user

Advanced Logging

Here is a more advanced script which prints the query time and the number of rows returned as well as the query. It is a simplified version of a more robust logger script available here.

-- dump_queries_with_times.lua
-- Prints a line for each query received from the client along with times

function read_query(packet)
    if packet:byte() == proxy.COM_QUERY then
        -- inject the query so that we can inspect the results and get the times
        proxy.queries:append(1, packet, { resultset_is_needed = true })

        return proxy.PROXY_SEND_QUERY
    end
end

-- read_query_result() is only called for queries injected via read_query()
function read_query_result(inj)
    local row_count = 0
    local res = assert(inj.resultset)
    local num_cols = string.byte(res.raw, 1)
    if num_cols > 0 and num_cols < 255 then
        for row in inj.resultset.rows do
            row_count = row_count + 1
        end
    end

    local log_string = string.format("Query [%.3f ms]: %s [%d rows]",
        (inj.query_time / 1000),
        inj.query,
        row_count)

    print(log_string)
end

The output looks like this:

Query [0.423 ms]: select @@version_comment limit 1 [1 rows]
Query [0.496 ms]: SELECT DATABASE() [1 rows]
Query [149.016 ms]: SELECT data_source_id, COUNT(*) FROM event GROUP BY data_source_id [5 rows]
Query [94.104 ms]: SELECT artist_id FROM artist [17065 rows]

Other Uses

Because of the ability to write your own scripts, there are lots of interesting uses for MySQL proxy. In addition to the types of logging covered above, people have written scripts to:

• Rewrite queries from one schema to another
• Perform cross tabulations
• Return information and perform operations normally only available to mysqladmin
• Load balance/server fallback/pool connections
• Rewrite/obfuscate connection credentials
• Gather query statistics
• Automatically log warnings and errors if generated
• Split read and write queries to separate servers
• Perform QA operations on database connections
• Measure bandwidth by user or session
• Selectively filter or block certain queries
• Provide some server admin capabilities via a SQL interface

Useful Links

• MySQL Proxy Download
• MySQL Proxy Documentation
• MySQL Proxy Wiki
• MySQL Proxy Launchpad
• MySQL Proxy scripts

In all of the examples below, I will use the sample data from the join post

actor Table:

movie Table:

The general structure of a union query looks like this (the parentheses are optional but help separate the queries visually):

(SELECT query1)
UNION [ALL | DISTINCT]
(SELECT query2)
...

Using our sample data, suppose you wanted a list of movies in which either Béla Lugosi or Carrie Fisher appeared. You could easily write this using a phrase like WHERE A.actor='Béla Lugosi' OR A.actor='Carrie Fisher', but we’re going to use a UNION here:

(
  SELECT
    M.title,
    M.year
  FROM movie M
  INNER JOIN actor A
    ON A.actor_id = M.actor_id
    AND A.actor = 'Carrie Fisher'
)
UNION
(
  SELECT
    M.title,
    M.year
  FROM movie M
  INNER JOIN actor A
    ON A.actor_id = M.actor_id
    AND A.actor = 'Béla Lugosi'
);

results:

As you can see, the results are just the individual query results glued together. How is this better than the single query version? It isn’t in this example, but the union allows us to do things that would be difficult or impossible to do with a single query. For example, you can sort the results within each query separately (add an ORDER BY clause inside the parentheses) or sort the entire combined result (add an ORDER BY clause outside the parentheses).

For a more involved example, suppose we wanted to see the most represented actors in the 1980′s versus the 2000′s. Splitting each decade into a separate query yields the following union:

(
  SELECT
    '2000''s' AS decade,
    A.actor,
    COUNT(*) AS movies
  FROM actor A
  INNER JOIN movie M ON M.actor_id = A.actor_id
  WHERE M.year BETWEEN 2000 AND 2009
  GROUP BY A.actor
)
UNION
(
  SELECT
    '1980''s' AS decade,
    A.actor,
    COUNT(*) AS movies
  FROM actor A
  INNER JOIN movie M ON M.actor_id = A.actor_id
  WHERE M.year BETWEEN 1980 AND 1989
  GROUP BY A.actor
)
ORDER BY decade, movies DESC;

results:

Although our sparse data set does not yield very interesting results, the UNION provides a nice way to organize the queries and control the sorting of the various parts.

UNION ALL vs. UNION DISTINCT

By default, MySQL removes duplicate entries from the merged result set before returning the results. This is equivalent to using UNION DISTINCT. If you need all results returned, whether or not they are duplicates, use UNION ALL.

Using UNION to optimize OR conditions

Before version 5.0, MySQL was unable to use more than one index per table in a single query. One side effect of this is that trying to use an OR expression on two different columns, such as WHERE col1='foo' OR col2='bar' could perform very poorly because MySQL could only utilize an index to limit results on one or the other column, but not both. By converting the query to a UNION, it is possible to get the server to use two indexes and then merge the results. MySQL 5 added the index_merge query plan which basically does this internally.