James Andres's Tips & How-To's

Back in February I released a Ruby script called DGEDeploy . We use DGEDeploy internally at Donat Group to help manage several of our project deployments.

Besides being a misnomer, DGEDeploy had one major weak spot: actually pushing code to servers. That, and a few other bugs, are fixed now in this latest release. Heck, lets call it version 2.

The latest version adds a new script, DGEPush, that acts as a simple RSync front-end. When using SSH keys it’s a breeze to merge a development branch into staging and push to the server.

DGEPush features list: * Rsync’s code using the -C option to skip .svn and CVS directories. This saves server space, and is a better security practice. * Can use SSH keys for better security and ease of use. * Can deploy to multiple servers in one shot.

Here’s how you get it running:

Create a dgepush user, on your remote server

On Ubuntu / Debian this is as simple as:

# Log into your remote server
ssh remote-server.com

# Create an account called 'dgepush'
sudo adduser dgepush

If you like, set up key based password-less logins for the remote user:

# On 'local' machine, ie: your laptop
sudo adduser dgepush
sudo su dgepush
cd ~

# Create the SSH keys
mkdir -m 700 .ssh
cd .ssh
ssh-keygen -t rsa -f dgepush-key
# !! Don't enter a passphrase, just hit Enter and Enter.

# Copy the key to the remote machine, this syntax only works on Linux.
# Possibly only on Debian / Ubuntu Linux.
ssh-copy-id -i ~/.ssh/dgepush-key.pub dgepush@remote-server.com

Test your SSH keys ssh -i /home/dgepush/.ssh/dgepush-key dgepush@remote-server.com should log right in without a password.

Here’s some other tutorials on the subject: * Password-less logins with OpenSSH * ssh-keygen: password-less SSH login

Update your dgedeploy-projects.conf

Adding a ‘servers’ section to any workflow enables you to get started with DGEPush.

/etc/dgedeploy/dgedeploy-projects.conf

# YAML 1.1
# Settings for example-project deployment
example-project:
  # The workflow field is required, the order of the parameters specify which
  # branch merges with which branch.  eg:
  #     development --> staging --> live
  workflow:
    - development
    - staging
    - live
  development:
    # The order of the urls parameter is important.  The first in the list will
    # be used as the default choice whenever necessary.  Each url should be,
    # a directory inside a Subversion repository, of course.  All urls should
    # be in the SAME Subversion repository.
    urls:
      - http://example.com/svn/example-project/trunk
  staging:
    urls:
      - http://example.com/svn/example-project/tags/staging
    servers:
      # This server uses key based password-less logins.
      - "-e 'ssh -i /home/dgepush/.ssh/dgepush-key' dgepush@stage.com:/var/www/stage.com/"
      # You can push to multiple servers by adding more lines!
      - "dgepush@mirror-stage.com:/var/www/mirror-stage.com/"
  live:
    urls:
      - http://example.com/svn/example-project/tags/live
    servers:
    # This server will require a password every time DGEPush is run.
    - "dgepush@live.com:/var/www/live.com/"

DGEPush in action

If you have already got DGEDeploy up and running using DGEPush is one more step. Here’s a basic example:

# First merge trunk into staging branch
sudo dgedeploy example-project http://example.com/svn/example-project/trunk
# Next push the staging branch to the server
sudo -u dgepush  dgepush example-project staging

Important points: * sudo -u dgepush runs the command as the dgepush user * dgepush example-project staging copies the contents of the dgedeploy cace for ‘example-project/staging’ to remote server. In other words, DGEPush won’t work unless you have initialized your project with DGEDeploy at least once.

Download DGEDeploy-v2.0.tar.gz

Wheew, it’s been a crazy summer. Excuses for my absence include:

I got married!

I moved!

Ahhh glad to be back.

Linux’s Music Player Daemon, MPD, is an interface-less jukebox. It’s like iTunes, Winamp, or Rhythm Box, except there is no built in GUI. Think of it as the Apache or MySQL of the music player world.

MPD is great for remotely controlling your Linux powered entertainment system or, as in my office’s case, to allow 15 staff to each act as work-day DJs.

Our office music server runs Ubuntu 7.10, but the default mpd package for Ubuntu doesn’t have M4A (AAC) support for MPD. Here’s what we did to re-compile MPD.

This tutorial will set you up with a new MPD with support for OSS/ALSA that will play the following formats:

• MP3
• AAC
• OGG Vorbis
• FLAC
• WAV

First, MPD requires libfaad2-dev and libmp4v2-dev to compile with AAC support. I’m using the debian-multimedia repositories to get these packages and they don’t have versions of libfaad2-dev and libmp4v2-dev suitable for Ubuntu 8.10.

No worries, this is a good excuse to learn how to build packages “gentoo style” using apt to build packages from source.

You’ll need the appropriate repositories enabled. Check your /etc/apt/sources.list file to ensure you have at least universe and multiverse enabled. Then add the debian-multimedia repositories. Append the below lines to your /etc/apt/sources.list.

## Debian Multimedia
deb http://www.debian-multimedia.org stable main
deb-src http://www.debian-multimedia.org stable main
deb http://www.debian-multimedia.org testing main
deb-src http://www.debian-multimedia.org testing main
deb http://www.debian-multimedia.org unstable main
deb-src http://www.debian-multimedia.org unstable main

Update your apt package database with an apt-get update. Next, we need to install the basic dependencies that will allow you to compile the software.

sudo apt-get install build-essential dpkg-dev \
     debhelper libasound2-dev libid3tag0-dev \
     ccache libxvidcore4-dev libid3-3.8.3-dev \
     libgtk2.0-dev libesd0-dev libsdl1.2-dev \
     liba52-dev liblame-dev libxt-dev \
     nasm libx264-dev dpatch \
     autotools-dev libfaad-dev libvorbis-dev \
     libimlib2-dev libfaac-dev texi2html \
     libtheora-dev libgsm1-dev libxvmc-dev \
     libamrnb-dev libamrwb-dev libsamplerate0-dev \
     libflac-dev fakeroot checkinstall \
     libmp3lame-dev quilt libdc1394-22-dev \
     libdirac-dev libschroedinger-dev

Once that has finished coming down the wire we’re almost ready to start compiling. Let’s get the MPD source code, and some initial dependencies.

sudo mkdir -p /usr/local/src
cd /usr/local/src
sudo wget http://www.musicpd.org/uploads/files/mpd-0.13.2.tar.gz
sudo tar zxvf mpd-0.13.2.tar.gz
sudo chown -R "$USER":admin mpd-0.13.2*

NOTE: The above snippet download’s mpd version 0.13.2. This is up to date as of November 12, 2008.

Time to compile! We’re using apt-get -b source PACKAGE-NAME to download and compile the source packages. This is a good way to do things since it will create DEB packages automatically, plus it’s good Debian form. I recommend learning how to use this feature of apt.

sudo apt-get -b source libavcodec-dev libavutil-dev
sudo dpkg -i *.deb
sudo apt-get -b source libmp4v2-0
sudo dpkg -i *.deb

Lastly we compile and install mpd. Run the following commands.

cd mpd-0.13.2
make clean
./configure
make
sudo checkinstall -D make install
sudo ln -s /usr/local/bin/mpd /usr/bin/mpd

If you haven’t received any errors, then you’re good to go!

The MPD Wiki is a good resource for help with the rest of the set up and configuration.

DGEDeploy is a Ruby script that helps make subversion merges a little easier. I wrote it at Donat Group Enterprises for use as a light weight tagging and release control solution.

Why?

I wrote DGEDeploy to:

• Make tedious ‘svn merge’ commands easier to use.
• Reduce the number of typos when running ‘svn merge’ commands.
• Be extremely simple, its not meant to replace Capistrano, CruiseControl, ...
• To improve my Ruby scripting skills

There are a lot of similar scripts out there, but if you find this one useful please let me know :-).

How does it work?

The script uses two configuration files to keep track of vital settings. The first, ‘dgedeploy.conf’, stores settings for the script itself (ie: subversion credentials). The second, ‘dgedeploy-projects.conf’ stores information about your codebase and projects.

All settings are stored in the YAML format.

Example: /etc/dgedeploy/dgedeploy-projects.conf

# YAML 1.1
# Settings for example-project deployment
example-project:
  # The workflow field is required, the order of the parameters specify which
  # branch merges with which branch.  eg:
  #     development --> staging --> live
  workflow:
    - development
    - staging
    - live
  development:
    # The order of the urls parameter is important.  The first in the list will
    # be used as the default choice whenever necessary.  Each url should be,
    # a directory inside a Subversion repository, of course.  All urls should
    # be in the SAME Subversion repository.
    urls:
      - http://example.com/svn/example-project/trunk
  staging:
    urls:
      - http://example.com/svn/example-project/tags/staging
  live:
    urls:
      - http://example.com/svn/example-project/tags/live

So, as shown in the configuration example above, the script is based on workflows. Merges always happen forward and must hit every level in the chain. This ensures your development code doesn’t go straight to live without being merged into staging first.

Make a merge

The syntax for the script is fairly straightforward. Once you have everything installed (see README.txt for details) you initialize your working copies like so …

dgedeploy example-project initialize

After the working copy caches are setup a merge can be performed using the same script. This example merges all of trunk into staging.

dgedeploy example-project \
  http://example.com/svn/example-project/trunk

#
# Translates to ...
#   svn merge \
#   http://example.com/svn/example-project/tags/staging \
#   http://example.com/svn/example-project/trunk \
#   /var/cache/dgedeploy/example-project/tags/staging
#

DGEDeploy will also merge just specific subdirectories. To send only a certain subdirectory from staging to live run a command like this …

dgedeploy example-project \
  http://example.com/svn/example-project/tags/staging/sub-dir

#
# Translates to ...
#   svn merge \
#   http://example.com/svn/example-project/tags/live/sub-dir \
#   http://example.com/svn/example-project/tags/staging/sub-dir \
#   /var/cache/dgedeploy/example-project/tags/live/sub-dir
#

Enjoy!

James.

Download DGEDeploy.tar.gz

I use Drupal a lot. This has caused me to become atuned to some of it’s foibles. Hook ordering, the order in which hooks/callbacks are called, is a one issue that agrevates me quite often. There just doesn’t seem to be a Right Way ^TM to get out of a hook ordering jam.

The community has taken a few stabs at the hook ordering problem. Sometimes the concept of a “hook registry” has been used. Here’s two threads I found on the issue (if anyone knows of some newer ones please let me know!):

• Hook ordering
• Eliminate hooks entirely in Drupal 6

I haven’t seen any commited code relating to hook orderig or a hook registry in Drupal 6, though I think it may have found new life in the updated menu system.

Alas, I was really hoping hook ordering of some form would make it into Drupal 6. So over the next few paragrahs I’ll lay out my grand vision for hook ordering.

Here lies James’ receipe for hook ordering in Drupal

First, the links above contain some fairly drastic ideas on how to implement a hook ordering system. I propose something a little simpler.

Reasons why, I think, my solution is simpler:

• It requires the addition of only one hook
• It requires only a few lines of extra, unobtrusive, code to the core
• It won’t affect any existing contrib or core modules

Also, please feel free to steal my ideas for something else if you like, I got the idea from the Compiz project anyway—namely their plugin dependancy system.

Addition of the hook_depends()

Sure, it sounds like reeling in a pair of adult diapers instead of trout. The API needs more humor anyway.

Ex:

function mymodule_depends() {
  return array(
    'hook_nodeapi' => array(
      'before' => array(
        'taxonomy_nodeapi',
        'audio_nodeapi'
      ),
      'after' => array(
        'image_nodeapi'
      ),
    ),
    'hook_form_alter' => array(
      'before' => array(
        'taxonomy_form_alter'
      ),
    ),
  );
}

The hook_depends is implemented by any module that cares about what order it’s hook(s) get called in. It should only be used when there is an issue with the standard hook ordering (more on that later). The idea should be quite straightforward from the example: the “mymodule” module needs its hook_nodeapi() run before a few other modules and after one, it also needs its hook_form_alter() run before one module.

How the core orders the hooks based on hook_depends()

When a new Drupal module is installed the core invokes all instances of hook_depends(). These should, in a perfect world, only exist in contrib modules because the core should take care of it’s own issues in a cleaner manner.

So, once a list of modules and their requested dependancies has been gathered the system can pour over them to figure out which order each hook should be run.

I’m sure someone else has a better algorithm than me, but here’s a shot.

• Find all implementations of the hook in question (from all modules, including ones that implement hook_depends().
• Split the list of hooks into two groups: (1) Hooks that don’t care when they are run, (2) hooks that care when they are run.
• Keep making consecutive passes over the hooks in group (2) trying to fit them into group (1) as they desire to be placed.
• If a cyclic relationship occurs then throw a nasty error (more on this later).

Once the ordered lists have been made for each hook cache the list to the database. I’m unsure if this should be done using cache_set or by keeping track of it in a seperate table entirely. Either way the result is the same.

Modifications to module_implements

Since Drupal is quite good at keeping it’s core code squeaky clean, adding hook ordering is a simple matter. Just modify module_implements() to grab the cached list of ordered hooks and everything should cascade from there.

What to do if there is a collision or cycle?

Nothing. Well throw an error, I guess. The real point is that while cyclic dependancies are easy to create it’s up to contrib module developers to avoid them. As long as the hook_depends() is used sparingly and only when there is a bonified reason cyclic dependancies shouldn’t happen.

Reasons to use hook_depends(), aka common reasons hook ordering is needed:

• Some module up the food-chain from my module is messing up my data, and I want it to stop.
• I want to mess with the data being supplied to some module down the food-chain.

Why not have ‘after all’ or ‘before all’ ordering classes?

Because somebody has to go first, and two hooks can’t both be first. It’s better to encourage developers to place their hooks directly before the module they intend to superceed. Also, wasn’t this what hook_init() an hook_exit() were created for?

All thoughts and feedback are welcome!

James.