Friday, March 20, 2015

CinemaDNG and Premiere's Difficulty in Importing It

While visiting a friend I noticed an empty Premiere project open with an importing dialog box that didn't seem to be moving. When I asked him if his machine was frozen he revealed that no, it wasn't, he had simply dropped a folder containing a feature film's worth of Cinema DNG footage into Premiere and it was taking a while to import. A while in this case meant having run for about 5 hours and having a progress bar that was only barely visible.

I tried to explain his machine or Premiere would probably crash before it could finish, and that even if the import did finish he would not like or want the results. Another friend who works in video/tech was present and surprisingly agreed that although it may take a while, dragging and dropping the whole shebang into the project pane should be just fine. I did my best to resit the urge to facepalm.

He's a shooter rather than editor by trade, and the problem here is a weird one, so I don't fault him for not knowing any better.  We've known each other since film school though, and he should know better than to ignore my advice on such matters!

Some 30 hours later I got a text message:
My system has run out of application memory!!!
"quit any applications you are not using"
I'm not using any others!

THE CONTEXT

The need is to import a large amount of Blackmagic Pocket Cinema camera footage (in raw CinemaDNG format) into Premiere. There is a master folder containing a subfolder for each scene, which in turn contain the clips for that scene. It is desirable to translate this scene folder structure into bins in Premiere, as it is the only manner of organization that exists for the footage.

A quick primer on CinemaDNG if you haven't yet had the pleasure. It's much like DPX in that it stores each frame as a separate image file. So you end up with a folder for each clip that contains the image sequence taht makes up the video frames alongside other assets like in camera audio.

A clip from the BMPCC. The Clip's name is that of the folder, TV_1_2015-06-22_2130_C0000. You can see we have an in camera audio file as well.
So what was I aware of that my friends weren't?

THE PROBLEM

Batch import of Cinema DNG is broken in Premiere. Adobe claims it's fully supported and I have to wonder if they mean that in the sense that duct tape can support your car's bumper to the frame.. If you go into the clip folder and have the eyeball set to the proper format you're fine, but trying to import multiple clips (i.e. folders, as we saw above) at once makes it take forever and then still isn't quite right.

Here we are looking at the contents of  "TV_1_2016-06-22_2130_C000000" just as we were in the finder. If your eyeball is set properly Premiere  interprets everything properly and you can drag that clip to a bin no problem
All is not well though. If we look at the parent folder bmpc raw we can no longer set our eyeball and Premiere can't see the folders for what they really are.

As I just illustrated Premiere is capable of seeing these folders as clips if you are only looking at one at a time, but try and look at a folder containing multiple clips and it plays dumb. I suppose this is what people mean when they say "can't see the forest for the trees". Trying to import one or any number of these folders will give you a major headache as my friend learned the hard way. Here's a video from a Creative Cow Post illustrating how you can import clips one at a time, but not in bulk. I can only assume the uploader had to mute the audio due to many numerous expletives uttered.


What that video doesn't show is what happens if you let that import finish. I thought I'd try using the clip I've been showing in my screenshots so far. This is the result.

Do not want.
First the good news: all of those clips play back fine with sync sound and are exactly the same. The bad news: you also get a containing bin with the clip name and the standalone .wav audio (not pictured). It seems each frame is interpreted as an entire clip you're left with a redundant bin and .wav file plus a number of duplicate clips equal to the number of total frames in your clip. For this 13 second clip at 24 fps that's 312 in.

Which is probably why it takes so long to import. If you watch activity monitor you can see the import process is only single threaded and it has to read every DNG in sequence 312 times over! If you'll recall the clip should only take 1.6 seconds to import properly. Using this method though, on a New Mac Pro with 32 GB of RAM, dual D700 GPU's, connected via Fibre to our SAN storage capable of 1000 MB/s read/write it took 1 min and 45 seconds... to import our 13 second clip which is 661.6 MB on disk. Which is actually faster than the math comes out to but we'll chalk that up to read caching on the volume. Frankly with the amount of footage from the feature I'm surprised my friend's machine held out for 30 hours, after which it was "slightly more than halfway" done.

To make matters worse, once if the import finishes Premiere still sees these as unique clips and will instantly start generating media cache info such as peak files, thumbnails, and the like for each. Not only is this a waste of storage space, it all but ensures you'll be unable to hit save before your computer curses you with it's last dying beep.

This workflow is broken by design in Premiere. Seemingly the only option is to import clips one at a time, as shown in the above video. Which is fine if you've got a handful of clips, but we're talking about a feature length film here with about 1/3 of the shot footage in Cinema DNG format. How do we avoid being a washing machine here?

THE SOLUTION

To be honest, although I was aware of the problem I didn't know of a better way than what I mentioned above. In the few cases where this came up, the amount of footage was small enough that importing each clip individually wasn't the end of the world. After I my "I told you so" out of the way, I set about finding a better solution to help my friend.

Generally my first idea whenever something impedes my workflow is to check if Blackmagic's software can... *ahem* Resolve the issue. (Can we please make Resolve a verb like Photoshop? "Did you Resolve those proxies yet?")

Resolve is a fantastically designed piece of software that is constantly bettering itself. It does so much more than color grading and that there is a free version with a large amount of the full feature set makes it a viable tool for everyone. Maybe I'll do a toolbox post on it one day as I use it for numerous things.

I know for a fact Resolve handles CinemaDNG's flawlessly as I've graded many programs in the format. Also the footage is coming from a camera by the same company so I wondered if it could pull out any tricks that might help.

A view of the same bmpc raw folder in Resolve. This is more like it. Those subfolder to bin options? Be still my beating heart.
So getting things into resolve with bins is a no brainer. But how to get that info to Premiere?  And let's assume we don't want to transcode everything to prores, because well, my friend didn't. Let's also assume we can't convince my friend to just edit the film in Resolve because it's really none of my business what NLE he uses.

Creating a timeline string out of a bin in Resolve is a trivial process and we could then export an xml for each, importing into matched bins in Premiere. If we had enough metadata to go by we could simply make a timeline of all the footage in Resolve and export an xml, and then sort things out to bins using the search functions in Premiere. In this case, there was no shot/take/reel metadata recorded and so the first option would be required. It's not what I'd call a great solution, but it has the benefit of requiring no expense and is certainly a step up from Premiere's option.

I did a quick test and noticed when I brought in the xml to Premiere it couldn't link up to the files. After comparing an xml of a CinemaDNG clip from Premiere to one from Resolve I quickly spotted why.

Here's how Premiere sees a clip.

<pathurl>file://localhost/Users/edit1/Desktop/bmpc%20raw/TV_1_2015-06-22_2131_C0000/TV_1_2015-06-22_2131_C0000_000000.dng</pathurl>

And how Resolve sees the same clip.

<pathurl>file:///localhost/Users/edit1/Desktop/bmpc%20raw/TV_1_2015-06-22_2131_C0000/TV_1_2015-06-22_2131_C0000_%5B000000-000401%5D.dng</pathurl>

Premiere only identifies the first frame in the image sequence whereas Resolve uses notation to precisely indicate the range of files that makeup the clip. It wouldn't be difficult to use whatever utility you are comfortable to do a find and replace operation on the Resolve XML to match it to Premiere's desired format. Trying exactly that I can report it works as expected. This workflow was already a bit more involved than I'd like and now you're talking about running through a text-editing utility that may cost money or require specialized knowledge to operate. It could work, but it still feels a bit too wishy washy to me.

I then found this excellent post on Digital Bolex with a really clever workaround that harnesses the power of Spotlight on the Mac. Most relevant bit below:

Navigate to the date folder where all of your DNG files are stored. (If you have multiple dates, you will want to search within the DCIM folder.)
In the Search Bar at the top of the dialog, search for "0001.dng"—this will find the first frame from each shot of your footage, and it's importing the first frame which helps Premiere recognize that you are trying to load an image sequence. Using the search bar to find the first frames is the preferable and recommended way right now to important CinemaDNG footage into Premiere.
I tested for myself and have to say it's quite a brilliant solution considering how simple it is. Depending on the folder you search you can either bring in all your clips at once, or in groups by their folder structure for placement directly into corresponding bins.

P.S. Windows users I can't help you but this thread might. I have not tested it's claims in any way but it seems to operate on the same principle as the Spotlight trick.

FURTHER THOUGHTS

I considered the matter dealt with until it struck me that the whole experience had a bit of similarity to my recent woes with reconnecting R3D footage in Premiere.

Sure enough, trying this trick to reconnect the prores files to the r3d worked.

Note: File Name and File Extension must be unchecked otherwise the .mov - .r3d mismatch will through things off, in it's place I turned on Media Start and Tape Name to give it something to match with. Also make sure Use Media Browser to locate files is unchecked or you won't get the ability to use spotlight. 

Here is what the Finder import window looked like using the Spotlight trick

Exposing the first piece of each spanned R3D clip in the RDM directory.
In the above screenshots we're looking to reconnect to the original for A006_C001_0112DV.mov proxy. So after we selected that first R3D file in the list, everything else in the RDM directy came online perfectly. So much win.

Except, I had to cheat a bit to get you the above picture. I copied some of the R3D footage to a local drive and ran it from there because our editors work off an Xsan on which we've disabled Spotlight because of numerous reasons and errors. It was causing massive instability and affecting the volume performance. So this amazing trick doesn't work.

That's ok though because we can use the same basic principle, displaying the first file of each clip together in one "virtual" folder to Premiere. We can do this by creating a folder with symbolic links.

To do that we can right a simple bash script like this:

!#/bin/bash
mkdir ./links
for clip in */*001.R3D
do
ln -s "$PWD"/"$clip" ./links
done

The above example would be run from the RDM folder containing the individual RDC folders, but with a little tweaking you can make it process down as many subdirectories as you'd like and make a link folder for each one.

Here's the result of running our footagelinker script. The little arrows on the icons indicate the file is a link, pointing to an original elsewhere. You can see it's displaying the original media in the preview pane, even though the file itself is essentially empty.

To reconnect in Premiere we would then start off the same way as we would have with the Spotlight trick, checking/unchecking the appropriate boxes in the relink dialogue. The difference is that we then could reconnect normally using the media browser to point at the appropriate link for the first offline file, and the rest will reconnect accordingly.

This same process could be used for the original Cinema DNG subject matter of this post instead of the Spotlight method. You would just need to change the line

for clip in */*001.R3D

to

for clip in */*000000.dng

Or in the original case of my friend, with one master folder containing subfolders for each scene's worth of clips you can tweak the processing just a bit.

!#/bin/bash
for clip in /*000000.dng
do
folder=$(dirname "$clip" | rev | cut -d / -f 2 | rev)
test "$folder"/links -e || mkdir "$folder"/links
ln -s "$PWD"/"$clip" "$folder"/links
done
This creates a links folder inside of each scene folder with all the clips from that scene. Either way you can simply drag and drop the links folder(s) into Premiere and it will properly and quickly import the original footage.

If we ever need to edit Cinema DNG in Premiere off our Xsan this is definitely how we will do it.

No comments:

Post a Comment