The first step in organizing a new repository is getting the files all in one place. The goal is to clean off all of the various drives on which they are found. In this step, duplicate files and directories will be created.
Do not worry. Eventually, duplicates will be removed. Until then, err on the side of caution so, in your zeal to clean and purge, nothing important is lost.
Assemble the Ingredients
Before starting, collect the following items to be used in the process.
- All field drives (e.g. mobile hard drives, USB drives and CF/SD cards)
- Any cables (USB3, Thunderbolt, FW800, FW400), power supplies, docks or card readers required to connect your field drives to a workstation.
- All Aperture/iPhoto libraries stored on all of your workstations and laptops.
- Three large cardboard boxes.
- A permanent marker.
- A workstation with all of the ports (Thunderbolt, Firewire, USB3) required to access your various field drives.
- Two, clean, clear DAM Drives.
- Power drill and metal drill bit.
Dedicate A Workstation
If you have an older, seldom used computer, consider making it your dedicated consolidation machine. As previously noted, this process can be time intensive; using your primary workstation increases the chances that the process will be interrupted. Ideally, this system supports Thunderbolt and/or USB3, which make the process go faster.
If you must use your primary workstation, execute the Data Consolidation step at the end of the workday and let it run overnight. This takes longer, but reduces the chances the process is interrupted by trying to multi-task during the workday.
Build or Buy Two New DAM Drives
I built two 12TB Drobo 5Ds that, with Drobos built-in data protection, gave me 8TB of storage for each unit. Pick up the biggest file stores you can afford. One can never have too much drive space.
If you have existing RAIDs, you can get away with a folder on those systems if you have the drive space. However, a new, dedicated DAM drive reduces confusion. If your current archive is small enough, you may be able to get away with something like Seagates 5TB Expansion Drives. If your system supports Thunderbolt or USB3, make sure your new DAM drives support these technologies as well.
The Physical Workstation
Remember those three large cardboard boxes? With a permanent marker, mark the boxes as STRONG, WIPE and CLEAR. Place SOURCE to the left of your work station. Place WIPE immediately to the right of the workstation. Place CLEAR immediately to the right of WIPE.
Create Your Repositories
In the root directory of each DAM drive, create the following base repositories (folders):
- 00_LIGHTROOM (Lightroom Catalogs)
- 01_PROJECTS (Working Media Files)
- 99_SOURCE_ARCHIVE (Source Media Files)
The two number code at the beginning of the repository name serves two purposes. First, it causes the repositories to line up in the file system in order, with the most frequently used (00_LIGHTROOM) at the top of the directory. Second, this code structure will be used to help group similar projects in subfolders.
When rebuilding our DAM, the 99_SOURCE_ARCHIVE repository is used as catchall for media currently spread across our workstations and field drives. When finished, it will act as our deep archive and 01_PROJECTS will house the final, working repository.
With base repositories in place, create subfolders that suit your post workflow. When finished, your source and DAM drives will look like this:
My repositories center on specific projects grouped into broad categories like 01_FAMILY and 02_CLIENTS. Notice that media is not sorted by type (e.g. photos, video, audio) at this level. That is done at the project level as shown here:
There are many ways to arrange your folders. Find one that makes sense for your workflow.
Assemble Source Drives & Libraries
If your existing Aperture/iPhoto libraries are already well organized, move them using Rich Harringtons “The Essential Guide to Moving an Aperture Library to Adobe Lightroom“.
If your Aperture and iPhoto libraries are a mess, move them to a dedicated folder on your workstation. If these libraries are Managed, it is as simple as copying the library into a new folder. If the library is Referenced, consider converting it to Managed first. This puts everything in one place and ensures nothing gets lost in the move.
If your consolidation workstation is not your primary workstation, copy these Aperture/iPhoto libraries to one of your source drives.
Place all of the source drives collected in Step #1 in the SOURCE box.
As you fill your SOURCE box, don’t rule anything out. The goal is to get it all in one place.
Connect a single source drive to your workstation. On the source drive, create a new folder called 00_MOVED.
Resist the temptation to connect multiple source drives and start multiple copy processes. First, pushing to/from multiple drives only slows the copies down as it fills up the workstations data pipe. Second, by taxing the system, you increase the risk of data corruption. Third, multiple source drives make it easier for you to lose track of what has been copied and what has not, which increases the chances you will create duplicates you will need to clean out later.
One source folder at a time, start copying data from your source drive to the appropriate folder on BOTH of your new DAM drives, which creates mirrored volumes for safety. To do this, I used Red Giants Offload. I love this tool for two reasons. First, it automates backups to two separate DAM drives. Second, as it makes the copy, it does a checksum on each file. This ensures a clean, reliable copy. It takes a bit longer, but the peace of mind is worth it.
When the DAM drive copy is complete, move the copied folder into the 00_MOVED folder on the source drive. This helps you keep track of moved folders as you process the drive.
Repeat this process with all of the folders on the source drive until everything is on both DAM drives and in the 00_MOVED folder on the source drive. When finished, your source and DAM drives will now look like this:
When the entire source drive has been processed, disconnect it and place it in the WIPE box. DO NOT erase the drives yet. You still need them as backups, albeit messy ones, for a little bit longer.
Rinse & Repeat
Repeat this process with each drive or card in the SOURCE box until it is empty.
NOTE: As you process your source drives, you may find what appear to be duplicate folders & files. When this occurs, do not copy over or merge the folders on your new DAM drives. Doing so makes it very likely that you will corrupt or erase data.
Instead, append a numeric sequence to the duplicate folder (e.g. 2012_BAHAMAS, 2012_BAHAMAS_02, 2012_BAHAMAS_03) then copy the apparent duplicate as well.
This is the one time we will knowingly create duplicates, but we are doing this for safety purposes. Well cleanly purge duplicates in a later step.
When you are done, you’ll have an empty SOURCE box, a full WIPE box (acting as a temporary archive) and roughly organized backups of everything on the DAM drives. It will take a while to complete, but you are now ready to start cleaning things up using the CONFORM and CULL steps.