Digital workflow on the road

This post describes my workflow to handle digital images and how I publish them on my web site while on the road. My camera has shot about 35,000 images, nearly all of which have been taken on my trip which started in August of 2007 (it is now May 2008).

Equipment and software needed

Digital camera, flash memory, spare batteries
External USB disk, 2.5", and USB cable
Portable 2.5" disk drive that can copy cards directly
Good card reader with USB cable
USB flash stick with portable software (see below) and notes
Software: allwaysync, setnametotime, fastimageresizer, winscp, putty, Open Office
A website that lets you upload pictures. It is best to have your own website. I use gallery2 software integrated with drupal.
A laptop would be nice but I use public Internet cafes.

Shooting guidelines

Carry a lot of flash memory. The more the better. It's lightweight and reliable. Use 8 GB or larger cards to avoid having too many cards to worry about. I got 56 GB at about USD$10/GB last year, which is enough for about 20 days continuous shooting (varies depending on subjects). I number each memory card (1-8). When one card fills up I go to the card with the next higher number, rotating back to 1 when the last card fills up. This way I always am several cards away from the oldest pictures on flash memory, which are the old pictures that are typically en route on DVD (see below) to home. By going sequentially I avoid accidentally overwriting pictures that have not been backed up. I also double check the card just before overwriting to make sure it has old pictures. Shoot maximum resolution in raw + small jpeg. Use the JPEGs to upload to review images and upload to your website.

Backing up

The goal here is to avoid losing any images and to have images stored back home. The basic approach is:

1. capture to CF cards
2. duplicate CF cards to portable disk
3. duplicate to second portable disk
4. burn DVDs
5. mail DVDs home
6. copy DVDs at home to large disk drive
7. store DVDs at separate location from main disk drive
8. clear CF cards and duplicate copies on portable disks as space is needed

This way, there are always at least 2 copies of the data. If there is a problem, e.g., DVDs lost in mail, I can always burn another DVD. Having two disks means I can maintain a second copy even if I have had to recycle CF cards before step 7 (above) is confirmed. As a further precaution, I avoid having all of my copies plugged into a computer at the same time--just in case there's a malicious virus or a power surge.

Backup to a portable disk. Before plugging any of your equipment into a public computer make sure they have a decent antivirus program, e.g., nod32, Kaspersky, Symantec, etc. A lot of places use AVG freeware, which is not very good, but it is better than nothing. You will probably still get viruses on your media and have to clean them up, so keep backups of everything. As an insurance policy, backup your CF cards to a portable disk before plugging them into a public computer. I have been using a HyperDrive Space disk enclosure as a portable drive, though it is not well constructed. While it still works I would not purchase another one. Maybe the Nexto portable drive would work better, though it is more expensive. Purchase the disk drive separately from the enclosure to save some money and get the disk you want. I use a Seagate 120 GB 2.5" IDE disk. I also have an external USB enclosure with an 80 GB 2.5" disk, both of which I got in China, though a higher capacity disk would be preferrable.

After copying the cards to the portable drive I use Internet cafe computers to copy CF cards to the 80 GB USB disk as follows:

Split the files on the CF cards into DVD size folders. For an 8gb disk this means two folders. I put half the files in one folder and the other half in the 2nd folder. Folders are named "yyyymmdd-yyyymmdd" (start/end) dates. be sure to carry a high quality card reader and your own USB cable since the ones in Internet cafes are nearly universally crap (slow slow slow).

2. Use allwaysync to copy the files from the CF card to the external drive. Alwaysync is a free file syncing program. You will probably need to plug it directly into the USB sockets in the back of the computer, even if the computer has front sockets, as the front sockets usually can't handle a disk drive.

Burn each folder to a DVD, working from the external disk drive. Burn the files from the external USB disk drive, which is better than a CF card that can hiccup and then you have a coaster instead of a DVD. I name the dvds "yy-mmdd-mmdd". Each DVD is also numbered, I'm up to "083" now. On the DVD I write the country and main places or cities in the pictures. I also have a file listing all of my DVDs along with basic contents and status (see below). This is a plain text file I can edit anywhere. I update it as i burn DVDs and then email a copy to myself as a backup as well as keep a copy of the file on a portable flash drive.

It is cheapest to buy your own DVDs. Try to get brand-name disks, like Sony, Ricoh, TDK, Maxell, etc. You can still use no-name disks and they usually work. Get DVD-R or DVD+R 16x disks in 20 disk spindles if possible as these are cheapest. Pick up a packet of nylon sleeves, which are more compact and lightweight than slim jewel cases. It can take some searching to find DVDs for a decent price and a lot of haggling in Asian countries and India, but in the end a blank DVD should cost USD$0.50 and not more than USD$1.00 (unless the USD continues to plunge). Some places will charge to use a DVD burner, again haggle like mad to get the price down to say 50 Rupees in India or go to a shop that doesn't charge, which is common in AU/NZ.

An example file with list of DVDs:

Disk drives:
1: 120GB Seagate in HyperDrive SPACE enclosure
2: 80GB Hitachi in DataStorage enclosure
Column headings (1=done, 0=not done):
sel: selected pictures for uploading
svd: copied nefs of uploaded pictures
cf: cf card number
hd: disk numbers
dvd: dvd number
pst: posted dvd home
rcv: received dvd at home
cpy: copied dvd at home to disk drive
dates: date range of images (yyyymmdd-yyyymmdd, utc dates)
images: image numbers
contents: brief description of contents

sel	svd	cf	hd	dvd	pst	rcv	cpy	dates	images	contents
1	1	1	1	1	1	1	1	20070810-20070824	3507-3953	us, japan:tokyo

Selecting files

Go through the images picking out the ones you like for your website. I do this by loading the first picture in MS Picture Viewer, which is on every Windows PC and can be used by right-clicking the first JPEG in a folder. I put the picture numbers in a spreadsheet that looks like this:

folder	image	country	location	description	copy
20070810-20070824	3511	japan	tokyo	street scene in tokyo	copy j:\photos\20070810-20070824\_DSC3511*.* j:\photos\upload-20070910

The copy column is a Windows copy command which is generated automatically from the previous columns (minimal knowledge of a spreadsheet program is needed for this). I use the Open Office Calc software as the spreadsheet. You need to create directories as appropriate for the places visited, e.g., japan/tokyo, etc.

Once I have the pictures to upload I copy the "copy" commands into a text file, name it "upload-20070910.bat" (or something similar) and run the script, making sure that all the commands work, since Windows batch files keep on going even if there's an error copying. The script copies the full-size JPEG and raw files. The raw files of the selected images are kept with me, but, to free up disk space, I delete the raw files in the original folders after the DVDs have been received at home.

Uploading images

After the files are copied I run "setnametotime" to rename the selected files as "yyyymmmdd-hhhmmss[.nn]z-.jpg". This ensures that each file has a unique name, even when my camera rolls around its counter. The name is a timestamp using UTC time since I can't be bothered to reset my camera's clock, hence the z. The original name is included to be able to find the original file in the original folder.

The images are resized for the web by droping the upload folder on "fastimageresizer" to get images rotated as per camera EXIF data and resized to 75 medium quality 1024x768 jpegs. You need to configure the options in fastimageresizer to dump images in a separate folder and use exif data etc. Fastimageresizer crashes if you have more than 2 folders deep in recursive resizing, but that's not hard to avoid. The resized folder is then zipped using any convenient zip program--either built into Windows or 7 Zip. The folder is uploaded to my website using winscp. I log on using putty to handle file munging on the server. Using the gallery software on my website I load the images into the gallery.