Along with the glamour and excitement of designing and developing good looking and functional sites, the web industry also has it’s share of work we’d rather not do: the tedium of changing every reference of Widgets to The Widget Company — filling the future’s Christmas stockings™ and the dangerous work of performing surgery on live sites. These tasks are perfect candidates for automation and scripting: have too large an impact when they fail, are too big, too boring, or too error prone to do manually, but just right for our computerised slaves.
While making simple changes to the files and databases that comprise most web-sites is easy making the same change, however simple, to 1000 files is mind numbingly boring. Luckily there are plenty of tools available that can make this sort of work the matter of minutes. If you work on Mac OS X, Linux or another UNIX-like operating system these tools should already be installed. Windows users should probably switch to something better.
find make it trivial to find files that need updating:
grep allows you to scan for files that contain a certain pattern and
find allows you to find files that match certain conditions (things like permissions, update time, and name). Using them you can quickly find the files you need to modify. Here are two examples:
To find all the files that contain the string
/old/path/(perhaps you’re moving the a section of your site?) you can simply run
grep -lr '/old/path/' *.
To find all the files called something like
*.views.incyou can use
find . -type f -iname '*.views.inc'.
There are a wealth of other tools that can be used to locate other details on a system. The
apropos command searches the manual built-in to most UNIX-like operating systems for the specified words,
which locates the file that will be run when you execute a given command, and
locate scans through a list of every file on your system for those with a matching name.
If there’s something you want to find there’s probably a UNIX command or, in the spirit of UNIX, a combination of commands that will help you.
Once you know which files you want to modify you can break out tools like
augeas to actually make your changes. The
sed command is a “stream editor”: it applies one or more updates to a stream (either a file or the input supplied to the command). Updates to more structured files like configuration and settings files can be tricky with
sed so special purpose tools like
augtool, which knows about many configuration file formats, can be more reliable.
To replace every instance of
/old/path/in a file with
/new/path/you can run
sed --in-place -Ee 's#/old/path/#/new/path# afile.txt.
To set the
STARTvariable in a configuration file to
yesyou can use
augtool set /files/etc/default/puppet/START yes.
To convert a PNG image into a JPEG you can use ImageMagick commands like
convert example.jpg example.png.
Again, there are scores of commands that you can use to update, modify and otherwise act on files of all sorts. No matter what change you’re trying to make, there’s almost certainly a tool that can do it for you.
Finding things and making changes
Gluing these two sets of commands together enables you to change large numbers of files in one fell swoop. There are a few different approaches and the one you choose will depend on the task at hand.
Some “change” commands can process multiple files at once while others will only handle one file at a time.
Some commands need arguments to tell them what to do and others are one trick ponies.
As is often the case when working on the command line, many commands will need quoting or escaping to handle file names with spaces and other special characters.
There are a few ways to join your “finding” to your “processing”. Some “finding” commands (like
find) can call your “processing” command for you while others leave that for you to figure out. Thankfully there’s another standard UNIX utility that can help: [
xargs]. Its sole purpose in life is to read inputs (typically filenames) and pass them on to other commands. While you’ll need to keep an eye out for quoting and escaping (spaces in file names can be tricky!),
xargs in one of the workhorses of UNIX.
Let’s consider an example. Suppose you want to replace all the
As in a folder full of files with
Bs. The first step is find the files that need to be processed:
grep -rl A .
will find only the files containing
As. Making a change like this is trivial with
sed -i~ -Ee 's/A/B/' example.txt
Gluing them together straightforward with
grep -rl A . | xargs -I % sed -i~ -Ee "s/A/B/" "%"
xargs will read each line of input from
grep and will build and run a command from by replacing every
sed -i~ -Ee "s/A/B/" "%" with the whole line. So the command will be run on every file that contains
A and the command will edit the file, replacing every
If you’ve looked at the
man pages for each command linked above you’ll already know that most of them have quite a lot of options. If you’re at all interested in learning more about these tools, do read the manuals.