Workflow for saving video data - sd card transfer and other


#1

My wife and I do some studio video interview shoots for a channel involved in novel startup ventures and such. We use 3 cameras and 3 or 4 audio tracks with a basic host / subject seated format. These may run over an hour of studio and video time out of which we might want to get 30 minutes of edited video. There may be one or more clips per camera. The cameras will shoot dual cards for backup but we’er currently just shooting one card (soon to change).

Audio is recorded with a Tascam DR70d recorder mixer to a SD single card in poly wav format. 3 or 4 tracks come from XLR wired lav mics x 2 or 3 plus one condenser shotgun mic. We usually roll one audio clip on all channels for the entire time.

All video plus audio amounts to around 120GB total data for one session.

One bottleneck in the process is transfer of data from the SD cards to an SSD. This SSD is used exclusively for the one vid session as an external MacOS HDD and also hold project files and other needed A/V assets (added to the SSD) for graphics, logos, music, intro, outro, etc. This is then edited in Adobe Premiere.

Are you aware of any device or technique for higher speed transfer of data from the SD cards (video x 3, audio x 1) to the SSD? We are using class 128GB 10 SD cards.

Currently it takes about 1.5 hrs to transfer data using an integrated SD reader on a Macbook with the external SSD.

Now I’m just realizing that we probably have USB 2. Apparently USB 3 is 10x faster but I’d have to figure out if the card would then be the limitation.

But I guess my question remains. I’m trying to speed up the process of transferring the data from the cards. I may go to Microcenter to see if I can find a USB 3 reader and give it a try.

If you care to comment on any of the following, please do - there are a couple of additional topics here . . .

  • Fast method to transfer data from multiple SD cards to external USB drive
  • Please comment if you are interested in some paid work editing such work with Adobe Premiere. There may be an issue making this data available to you considering its size.

Thank you


#2

If you’re really lucky your Mac will support USB 3.1 gen2.

USB 3.0 and USB 3.1 Gen 1 can transfer data up to 5 Gbps, and USB 3.1 Gen 2 can transfer data up to 10 Gbps.

Check out this article by Apple to make sure your specific Mac can support the USB version you desire. I’ve been down this USB road before and if the Mac won’t support it, buying the card/reader won’t help.


#3

John - thanks for the reference. Yes, it’s a bit confusing . . . USB 3.1, USB C, Thunderbolt. When we all have USB C then things will be less confusing, I hope.

So, I grabbed a few items at MicroCenter and here are some observations.

First of all, BlackMagicDesign’s free Disk Speed Test is really good and will show how a drive might function with various video protocols / codecs.

Here is what I tested.
SD card - SanDisk UHS-I 128GB
SSD - ? brand with USB-3 adapter and case
USB 3.0 hub 4-port
USB 3.0 ‘pocket’ adapter (cheap)
SD card slot on the MacBook
2015 Macbook Pro with USB 3.0

I transferred a 6.4 GB video file to test.

I/O speed to external high speed SSD

Fake 3.0 128GB thumb drive bought at MicroCenter last night - $7 - it is actually a USB 2.0 but it’s even slow for a 2.0 drive. It is labelled and sold as 3.0. I’m taking it back later today.

Sales person said “They are great. Nobody has every complained about them.”

Right. Nobody has ever tested them. These are the ones at the checkout counter. They have an entire rack of these bogus memory sticks of all sizes.

Class UHS-I SD card in the Macbook slot - got the exact readings using the more expensive USB 3.0 external reader.

The Cheap 3.0 SD reader - a bit slower than above but it turns out not to be limiting when using this to transfer files.

Just for grins, my 4TB WD Passport 5400 RPM HDD from Costco

I’m have more info, some actual speed tests with the test file and the implications for transferring data from SD to external media . . . coming up in a bit when I have another moment to post . . .


#4

Standard USB 3.0 should have more than enough bandwidth to tap out the read speed of most cards – USB 3.0 has a bus speed of 5Gbps (625 MB/s) full-duplex (in/out).

It sounds like your real bottleneck is the read performance of the recording media, as you’ve got plenty of bandwidth on the SSD to write (also supported by the data you posted above.)

You might consider grabbing some UHS-II cards, as they provide much higher read speeds for those times when you need to dump data quickly, as well as backward comparability with previous generations so it will work with your existing gear. In effect, write the data using your current gear, and read it out using a faster UHS-II reader to SSD and by the numbers, you should see a 3-4x improvement over those mid 50MB/s reads.

Something like this USH-II compatible reader from SanDisk (plus, it’s USB 3.0 – so your existing Mac should work great with it).

However, note – you won’t see a performance increase on media that isn’t UHS-II compatible, so your existing media may need to be evaluated to take advantage of the improvements.


#5

Thank you, DM . . .

That was a passing thought but thanks for rekindling that idea! I’ll look at prices . . . UHS I vs UHS II. I’m already invested in a good number of the UHS I but I’ll think it through.

Yes, right you are about the read speed. I have not posted it yet but a summary of my observations on just going from a USB 2.0 (old macbook) to USB 3.0 (newer macbook) was a 6x speed increase - roughly 1 minute vs 6 minutes for a 6 GB file . . . SD card to fast ext drive through the respective old vs newer macbook. In addition, I have tried transferring simultaneously through two card readers to the ext drives. There was no speed reduction per card as the bottleneck was the read speed. I have not tried three cards (we have three cameras) but I bet that will, too, improve the overall throughput without slowing down on the transfer.

This, alone, may be enough of a resolution to this bottleneck. Our current status is, until I can take over, that we have a paid production assistant and we are cautious about our expenses and paying to wait for media transfer to hand to our PA, also the editor, is not an effective use of our limited capital.

We have learned the hard way not to let any data out of our hands to outsourced editors without backup on the premises. And the size of the data plus our internet speeds preclude cloud transfer as a solution.

Thus, overall, by

  • going to USB 3.0 hardware (6x improvement)
  • doing concurrent transfer (3 cards, 3 readers, 3x improvement)

may give us perhaps 18x improvement in transfer speed

Which, would be going from 120 minutes to around 7 minutes. That’s tolerable.

I had also thought about pulling SD cards and swapping from the cameras early and reading them during the last part of the photo shoot, then reading the rest. But that’s asking for trouble, messing with the cameras in the middle of things. Something’s gonna get bumped out of frame or out of focus . . .


#6

I have UHS-II SD cards from Lexar and a matching reader. The performance is very good. When downloading photos I’m getting their advertised speed (150MB/s read). They have faster cards available now because I purchased these in 2016. The stuff they sell now advertises 300MB/s reads.


That’s the same card reader I have but that’s a different SD card than what I have (they don’t sell mine anymore). The card and the card reader are both important. Integrated card readers won’t be anywhere near as fast even if the card is capable of it.

Correction, they still sell the card I have.


#7

A picture is worth a thousand words. This is why the card and card reader are both important. There are additional physical contacts for UHS-II. The left is my Lexar card. The right is a typical SD card (came with a 3D printer).


#8

Luke - Great info and I might not have thought about this.

Obviously, going from UHS I to UHS II involves an upgraded SD interface for those extra pins on the SD card.

So, as far as cameras go, they, too would have to be wired for the UHS II cards if they were going to take advantage of the UHS II transfer rate, I suppose.


#9

Just a thought . . . UHS-II on a micro SD card (if they exist - OK, they do, I just looked). I wonder if UHS-II involves different pinout on the micro card and thus would need the concomitant SD carrier with both internal (for the micro card) and external pins to support USH-II.

Not that I need micro SD, but when shopping for SD cards, I’ll take whatever I can find - plain SD or micro SD fitted into the SD adapter.

If I do ever go UHS-II and end up with a micro plus adapter then - note to self - don’t mix up the adapters with the UHS-I cards.