Get in touch


FCSvr to P5 Archive

Migrate media and metadata from Final Cut Server to Archiware P5 Archive

If you are still running Final Cut Server, you might want to migrate the media and metadata it manages to another solution.

If you still wonder why you should migrate your data out of Final Cut Server at all, think about this: Final Cut Server has been discontinued in June 2011, which means you've been running an unsupported software for almost five years, now, and this piece of software manages some or all of your most valuable media assets... If something breaks finding somebody who still supports an old system like that is getting harder every day.

Existing migration options

There are multiple systems available to which you can easily migrate your existing Final Cut Server data:

Archiware P5 and the P5 Archive App

While there are lots of migration options available already, this article is about taking a different route again by migrating your Final Cut Server data to Archiware's P5 Archive.

P5 Archive is an archive solution which moves your online data away to tape or disk. It allows you to use single-tape drives as well as large enterprise libraries and makes storing large amounts of data real cheap.

You can migrate your Final Cut Server data into P5 Archive and then, whenever you need a file from tape, look it up in P5's web interface and restore the file to your online storage.

One of the great things about a MAM, though, is that it allows you to see both your online and offline data side by side in the same interface. Archiware came up with the P5 Archive App to unify the display of online and archived data. Their approach is unique as the interface for all your data is the OS X Finder. And compared with the solutions above, the basic MAM features of the P5 Archive App already includes a comprehensive archive solution, while the solutions above need to be extended with an archive solution once you manage more than just a handful of media. The latter increases both price and complexity.

You can archive one or more files or folders by right-clicking on them and choosing "Archive to P5":

ArchiveToP5

The P5 Archive App job monitor informs you about the archive progress:

02-Monitoring

and once the archive process has completed, the P5 Archive App creates a very small "stub" file at the same location like your original file, which allows you to double-click on the file to see a preview and related metadata:

Once you want to restore an item to your online disk, either right-click on one or more files or folders and choose "Restore from P5", or choose "Restore" in the preview window of an item.

This means you never need to leave the Finder to manage your archived media!

On a sidenote: the P5 Archive App understands Final Cut Pro X libraries and can archive and restore complete Final Cut Pro X libraries and projects including external media. It can even clean up your projects before archiving.

Now let's see how we can migrate your existing Final Cut Server to P5 Archive and the P5 Archive App.

Data extraction from Final Cut Server

On the Final Cut Server machine, run our free Final Cut Server export tool fcsvr_export.command. It will create a folder named "fcsvr_export", which you will need to copy to a Mac.

Preparing P5 import XML files

On the Mac, create a new file named get_md_fields.py and copy the following content into the file:

#!/usr/bin/env python3 # -*- coding: utf-8 -*- """ This script fetches a list of metadata being used in your Final Cut Server assets and creates all necessary fields in your index. """ import os import xml.etree.ElementTree as ET from xml.sax.saxutils import escape import shutil # adapt the following settings to your environment asset_xml_folder = os.path.join("/", "Users", "andre", "Documents", "work", "freelance", "clients", "MSM", "tmp", "ASSET_XML") target_folder = os.path.join("/", "Users", "andre", "Documents", "work", "freelance", "clients", "MSM", "IMPORT_XMLs") # Here starts the processing metadata_root = os.path.join(target_folder, ".com.archiware", "presstore", "metadata") try: shutil.rmtree(metadata_root) except FileNotFoundError: pass if not os.path.exists(metadata_root): os.makedirs(metadata_root) all_metadata = {} all_metadata_file = os.path.join(target_folder, "all_metadata.txt") my_asset_xmls = [ f for f in os.listdir(asset_xml_folder) if "_members.xml" not in f] item_no = len(my_asset_xmls) i = 0 duplicates = set() duplicates_file = os.path.join(target_folder, "duplicates.txt") for source_xml_name in my_asset_xmls: i += 1 print("Reading file %s of %s." % (i, item_no)) source_xml_path = os.path.join(asset_xml_folder, source_xml_name) # print("XML file path is '%s'." % source_xml_path) metadata = {} # read in xml for this device. xml = ET.parse(source_xml_path).getroot() filename = False # Get all metadata and write them as key value pair into dictionary. for field in xml.findall("./values/value"): field_key = field.attrib["id"] field_value = field[0].text if field_key == "PA_MD_CUST_FILENAME": filename = field_value if field_key not in all_metadata and field_value is not None: all_metadata[field_key] = set() all_metadata[field_key].add(field_value) elif field_key in all_metadata and field_value is not None: all_metadata[field_key].add(field_value) if field_value is not None: metadata[field_key] = field_value if filename and metadata != {}: try: os.makedirs(os.path.join(metadata_root, filename)) except FileExistsError: print("==================> Folder '%s' already exists. Skipping it." % filename) duplicates.add(filename) continue target_xml_path = os.path.join(metadata_root, filename, "metadata.plist") xml_content = """\n""" xml_content += """\n""" xml_content += """\n""" xml_content += """\n""" for key in metadata: xml_content += """ user_%s\n""" % escape(key.lower()) xml_content += """ %s\n""" % escape(metadata[key]) xml_content += """\n""" xml_content += """""" with open(target_xml_path, "w") as f: f.write(xml_content) # write a list containing all metadata fields into a text file. try: os.remove(all_metadata_file) except FileNotFoundError: pass with open(all_metadata_file, "a") as f: for field in all_metadata: if len(all_metadata[field]) > 50: f.write("%s; => free text field in P5\n" % field) else: f.write("%s;%s\n" % (field, all_metadata[field])) # write a list containing all duplicate file names into a text file. try: os.remove(duplicates_file) except FileNotFoundError: pass with open(duplicates_file, "a") as f: f.write("The following %s items are duplicates.\n" % len(duplicates)) for filename in duplicates: f.write("%s\n" % filename)

You might have noticed that this script requires Python 3. If you haven't installed it, yet, please get your copy at https://www.python.org.

In lines 16 to 21 of the script, you need to adapt these lines to your environment:

asset_xml_folder = os.path.join("/", "Users", "andre", "Documents", "work", "freelance", "clients", "MSM", "tmp", "ASSET_XML") target_folder = os.path.join("/", "Users", "andre", "Documents", "work", "freelance", "clients", "MSM", "IMPORT_XMLs")

The variable "asset_xml_folder" needs to point to the ASSET_XML folder inside the fcsvr_export folder we created above.

"target_folder" points to the folder where your media reside. This assumes that all your Final Cut Server media reside in a flat hierarchy inside a single folder. If you used multiple Final Cut Server storages or subfolders on your storage, please get in touch so that we can modify the script for you.

Now make the get_md_fields.py file executable:

chmod +x get_md_fields.py

and run the script

./get_md_fields.py

You will find an invisible folder inside your media folder named .archiware, which holds the metadata for all your assets.

In addition, there will be two new files in your media folder:

Setting up P5 and P5 Archive App

Now set up a new archive plan in Archiware P5 Archiware. Make sure it doesn't delete files and folders after archiving, and make it incremental to avoid duplicate files on tape. You can either use the default settings for preview generation, or you can set up your own preview generation like described here.

Please note: You can also reuse your existing Final Cut Server proxies instead. Please get in touch if you want to reuse them.

Now create the file /usr/local/aw/md_import.sh with the following content:

#!/bin/sh exit 0

and make it executable:

chmod +x /usr/local/aw/md_import.sh

Tell your archive plan to use this script for metadata import:

Now start the P5 Archive App and set it up:

P5ArchiveAppSettings

You don't need to activate "Add metadata during archive process". This feature would open a metadata window each time you archive a file, so that you can manually add metadata, but this is not needed for automatic metadata extraction from our .archiware folder.

Make sure you can archive and restore files other than your Final Cut Server data using the P5 Archive App.

Once that works, we need to add the former Final Cut Server fields to our P5 Archive index.

In the P5 web UI choose Archive -> Manage Indexes, then select the index you are using in your archive plan and click on Fields:

You will then see a list of metadata fields used in your index:

If you open the file "all_metadata.txt" mentioned above, you will see a list of fields we need to create here. Take the name on the left from the semicolon, and use its lowercase version as the internal name in P5.

E.g. PA_MD_CUST_MOTIV_1 in Final Cut Server needs to be entered as pa_md_cust_motiv_1 and will be translated into user_pa_md_cust_motiv_1.

Make all field types "TEXT". Wherever a Drop-down field makes sense, add all values listed in the all_metadata.txt file to the possible values of your fields.

Data migration

Once you have completed this, archive a single file using the P5 Archive App. If everything works well, the .p5a file will show up in the Finder (like above), and a double-click on the file will show you the migrated metadata as well as a link to the preview file.

If the migration process works fine for a single item, select more or all of your items and archive them using the P5 Archive App. It might make sense not to select more than 10.000 items at once.

Liability, service and consulting

Though we did our best to describe the migration workflow, we do not take any liability if your system is impaired or damaged while you are testing what we describe in this article. It is highly recommended to try this on a test system first and always have backups in place before you run the tools above.

Having said that: if you need help with the migration, please get in touch with us and we will be more than happy to support you!


Click here to read about updates, new features and future plans.


Imprint
1