Along with the glamour and excitement of designing and developing good looking and functional sites, the web industry also has it’s share of work we’d rather not do: the tedium of changing every reference of Widgets to The Widget Company — filling the future’s Christmas stockings™ and the dangerous work of performing surgery on live sites. These tasks are perfect candidates for automation and scripting: have too large an impact when they fail, are too big, too boring, or too error prone to do manually, but just right for our computerised slaves.

While making simple changes to the files and databases that comprise most web-sites is easy making the same change, however simple, to 1000 files is mind numbingly boring. Luckily there are plenty of tools available that can make this sort of work the matter of minutes. If you work on Mac OS X, Linux or another UNIX-like operating system these tools should already be installed. Windows users should probably switch to something better.

Finding things

Commands like grep and find make it trivial to find files that need updating: grep allows you to scan for files that contain a certain pattern and find allows you to find files that match certain conditions (things like permissions, update time, and name). Using them you can quickly find the files you need to modify. Here are two examples:

  1. To find all the files that contain the string /old/path/ (perhaps you’re moving the a section of your site?) you can simply run grep -lr '/old/path/' *.

  2. To find all the files called something like * you can use find . -type f -iname '*'.

There are a wealth of other tools that can be used to locate other details on a system. The apropos command searches the manual built-in to most UNIX-like operating systems for the specified words, which locates the file that will be run when you execute a given command, and locate scans through a list of every file on your system for those with a matching name.

If there’s something you want to find there’s probably a UNIX command or, in the spirit of UNIX, a combination of commands that will help you.

Making changes

Once you know which files you want to modify you can break out tools like sed and augeas to actually make your changes. The sed command is a “stream editor”: it applies one or more updates to a stream (either a file or the input supplied to the command). Updates to more structured files like configuration and settings files can be tricky with sed so special purpose tools like augtool, which knows about many configuration file formats, can be more reliable.

  1. To replace every instance of /old/path/ in a file with /new/path/ you can run sed --in-place -Ee 's#/old/path/#/new/path# afile.txt.

  2. To set the START variable in a configuration file to yes you can use augtool set /files/etc/default/puppet/START yes.

  3. To convert a PNG image into a JPEG you can use ImageMagick commands like convert example.jpg example.png.

Again, there are scores of commands that you can use to update, modify and otherwise act on files of all sorts. No matter what change you’re trying to make, there’s almost certainly a tool that can do it for you.

Finding things and making changes

Gluing these two sets of commands together enables you to change large numbers of files in one fell swoop. There are a few different approaches and the one you choose will depend on the task at hand.

  1. Some “change” commands can process multiple files at once while others will only handle one file at a time.

  2. Some commands need arguments to tell them what to do and others are one trick ponies.

  3. As is often the case when working on the command line, many commands will need quoting or escaping to handle file names with spaces and other special characters.

There are a few ways to join your “finding” to your “processing”. Some “finding” commands (like find) can call your “processing” command for you while others leave that for you to figure out. Thankfully there’s another standard UNIX utility that can help: [xargs]. Its sole purpose in life is to read inputs (typically filenames) and pass them on to other commands. While you’ll need to keep an eye out for quoting and escaping (spaces in file names can be tricky!), xargs in one of the workhorses of UNIX.

Let’s consider an example. Suppose you want to replace all the As in a folder full of files with Bs. The first step is find the files that need to be processed:

grep -rl A .

will find only the files containing As. Making a change like this is trivial with sed:

sed -i~ -Ee 's/A/B/' example.txt

Gluing them together straightforward with xargs:

grep -rl A . | xargs -I % sed -i~ -Ee "s/A/B/" "%"

Here xargs will read each line of input from grep and will build and run a command from by replacing every % in sed -i~ -Ee "s/A/B/" "%" with the whole line. So the command will be run on every file that contains A and the command will edit the file, replacing every A with B. Done!

If you’ve looked at the man pages for each command linked above you’ll already know that most of them have quite a lot of options. If you’re at all interested in learning more about these tools, do read the manuals.

Posted by Thomas Sutton
Thomas is an atheist, web developer, semi-lapsed Haskeller, and complainer extraordinaire. He also has a fetish for milk arrowroot biscuits. You can see him make a fool of himself on Twitter as @thsutton.

Leave a reply

The content of this field is kept private and will not be shown publicly.