Old 03-30-2015, 02:35 AM   #1
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default 0 byte recorded files

Hey folks,

after years and hundreds of hours of flawless live recording with reaper, I recently experienced a massive issue

After ~3 hours of recording time (see specs below) I hit stop. Beach ball turned for a few seconds, as usual, then I ended up with a „Error while trying to save project“ message. Reaper created wav-Files for all of the tracks, but obviously no data was being saved: all files were 0 byte in size.
I tried to reimport them in another reaper session, without success. I checked the drive itself and it seemed to work properly. Then I unmounted the drive and did not reconnect it to this day, so nothing gets overwritten in case the data was there.

recording specs:

~3h of recording time, no errors whatsoever during recording, waveforms updated regularly
~90 mono tracks, 48kHz
external sync to LTC, no errors

system specs:

macbook pro 13“ retina 10,2, 2012, 2,5GHz i5, 8GB RAM
OS X 10.10.1
Reaper 4.76/64
interface: RME HDSPe MADI FX via Sonnett Echo Express
recording onto Toshiba 2TB USB3 drive, ~70% free space

Did anybody experience anything similar?

Any hints on how to proceed?

Thank you very much!

Cheers
ribbon is offline   Reply With Quote
Old 03-30-2015, 10:13 AM   #2
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 6,491
Default

Just a data point for you:

I can tell you that Reaper writes an EOF mark with each bufferful of data to the drive. So, normally if the power were to be cut for example, the recording would be complete right up to that point with a valid readable file complete with EOF mark.


Since you have 0 byte files, I'll speculate that the drive is failing and the index took a hit.
__________________
Mac Pro 8x3.33GHz i7, 16GB, 256GB SSD(OS, apps), 3x2TB 7200 HD(data); MacBook Pro 2.8GHz, 6GB, 128GB SSD HD(OS, apps), 750GB 7200 HD(data); 2xTrue Precision 8; Apogee AD-16; 2xMOTU 828mk3, Evolution UC-33e; Faderport; WiRanger, iPad & the analog mixer has retired
serr is offline   Reply With Quote
Old 03-30-2015, 11:35 AM   #3
cyrano
Human being with feelings
 
cyrano's Avatar
 
Join Date: Jun 2011
Location: Belgium
Posts: 2,561
Default

Before you do any more experiments, download the demo for Data Rescue:

http://www.prosofteng.com/datarescue4/

Let it scan the drive and see if it finds your files. Usually, DR 4 will recover everything.

Trying to "repair" the disk with a lot of other tools (like DiskWarrior, for instance) will ruin your data. Once your data is recovered, format and test the drive.

Any chance your external drive was FAT or NTFS formatted?

Passing the 2 GB file size limit on FAT disks might just lead to the kind of data loss you're experiencing.
cyrano is online now   Reply With Quote
Old 03-31-2015, 06:58 AM   #4
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Thank you so far for your precious hints!

I have experienced corrupted files once after a power loss. QT did not open the files, but I could import them seamlessly into reaper with just the last 3 or so seconds missing. But at that time, I could see the actual file size in finder.

I am going to try DR4 and will get get back with results.

The drive is formatted HFS+ Journaled. Reaper was set to split the files at 1 or 2GB anyway.
ribbon is offline   Reply With Quote
Old 03-31-2015, 03:31 PM   #5
cyrano
Human being with feelings
 
cyrano's Avatar
 
Join Date: Jun 2011
Location: Belgium
Posts: 2,561
Default

OK, DR 4 will read the journal and that will help a lot! Success!
cyrano is online now   Reply With Quote
Old 04-02-2015, 04:55 PM   #6
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Alright, I performed a deleted files scan plus a deep scan with DR4 today and got the result: no sign of the missing data.

It seems like reaper did not write anything to the disk from second 1. Weird: I am quite sure I did see waveform overviews when I hit stop.

When connected, the drive itself appears and seems to behave in a normal way.

Do you by now see any malfunction on hard drive side?
Can we surely leave aside any suspicion on reaper or OS?

Is there any kind of test for hard drives to avoid this scenario in the future?

I usually record onto a Raid 1, but for some reason I am not quite convinced that it had helped in this specific case.
ribbon is offline   Reply With Quote
Old 04-10-2015, 07:55 PM   #7
Hypex
Human being with feelings
 
Join Date: Mar 2015
Location: Australia
Posts: 101
Default

If we put this into perspective. Three hours worth of 90 tracks at 48Khz in 16-bit? That's 3 x 60 x 60 x 48000 x 2 for each track or ~988MB. Almost a gig for one track. If my math is correct and ~88997MB or ~86GB needed for data storage. So you certainly should have enough space.

Do you usually record 90 tracks at once? That sounds very taxing even over USB3. Each second 96000 bytes would need to be written into 90 separate files on the physical disk. Depending on the buffer write size. And also if separate files are used. A huge 90 track multitrack WAV if that is possible would allow all data to be written at once on each pass and would be a better option. Less stress on the hardware.

Now you said DR failed to find any data but what method does it use to locate it? It it looks for a WAV header then that may not even exist. Given the file is dynamically recorded to it's possible the WAV header would be written on completion and success. And since an error was reported the header may not have been written.

That being said, given the samples are written on the fly I still think it's possible that the data is still there and intact. It's just a matter of finding it. I don't know if OSX has any disk editors but if you could examine a section of drive, spot a pattern of sound samples, then be able to save it out as a separate file of a specific linear length you could recreate the files.

If there was software that let you "listen" to a HD then you might spot it easier. Of course now I'm getting into advanced topics. Just trying to help. :-D
Hypex is offline   Reply With Quote
Old 04-11-2015, 03:33 AM   #8
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Thanks for sharing your thoughts!

Indeed I've recorded many many hours of 100+ tracks onto this kind of external drives in various sizes, even on this specific device. Interesting thing is that recording has never been a problem, while playback of more than a hundred tracks results in stutter. My thought has been that reaper might write the files in a linear way, but the read head has to jump through a hundred files when playing back. Maybe someone of the development team does read along and can explain the effect?

As said, I have never had a single issue or dropout even with hour long recordings on hot summer open air days. To be honest, reaper is the first piece of software I have put so much trust in. Even Pro Tools (among others) and hardware recorders have failed at recording multiple hours of 128 tracks.

Does anybody have an approach to Hypex' idea of spotting the raw data on the drive?

I am still trying to find out reasons for this problem and make sure not to run into trouble again.
In my tests, the system seems to run smoothly again now...
ribbon is offline   Reply With Quote
Old 04-11-2015, 10:07 AM   #9
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 6,491
Default

When recording with Reaper, the blank files are written as step 1. Then as audio is recorded, the I/O buffer of data is written to the open file(s) with an EOF mark every time.

The WAV header portion of the file is updated with the correct file size when the recording stops. However, the files are still complete valid files that report their size correctly to the Finder (OSX).

Normally the fallout from this would be a valid file but with 'corrupt' header info. Some audio apps would just look at the header and stop there. Reaper actually will ignore invalid headers when there is data there.


That's what is supposed to happen.

You had some hardware crash with the drive that resulted in it NOT reporting an error to OSX and basically acting like it was running but it was just being a black hole.

I saw something like this happen a dozen or so years ago. I was probably running Digital Performer v3. A power supply in an external drive started to fail and basically the drive power cycled during use. This somehow was able to happen without OSX getting an unexpected device removal error. Possibly something like the 12v supply failing but the 5v supply for the logic still live the whole time. So the system kept running and the data I/O was just going through the motions. No data was saved.

I'm sorry. This is shitty news for you! But that's what it looks like just happened to you. I'd replace the power supply in that external drive box.

The Reaper crash would have been it trying to access those files (it was told were still there and being written to by the drive the whole time) to update the header info at the end of the recording. Drive didn't respond as expected so - beach ball. Files weren't there so - Reaper error message.
__________________
Mac Pro 8x3.33GHz i7, 16GB, 256GB SSD(OS, apps), 3x2TB 7200 HD(data); MacBook Pro 2.8GHz, 6GB, 128GB SSD HD(OS, apps), 750GB 7200 HD(data); 2xTrue Precision 8; Apogee AD-16; 2xMOTU 828mk3, Evolution UC-33e; Faderport; WiRanger, iPad & the analog mixer has retired
serr is offline   Reply With Quote
Old 04-11-2015, 10:22 AM   #10
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Thanks serr!

Your story tells me the problem is rather unlikely to happen again.

Is there any test routine one could perform on a hard drive prior to recordings?
ribbon is offline   Reply With Quote
Old 04-11-2015, 11:19 AM   #11
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 6,491
Default

Quote:
Originally Posted by ribbon View Post
Thanks serr!

Your story tells me the problem is rather unlikely to happen again.

Is there any test routine one could perform on a hard drive prior to recordings?
Not really. It works and tests OK until it doesn't...

Raid 1 is a solution for this. (Two mirrored drives. Doubles your hard drive cost for that insurance.)

If you're recording this level of audio (3 hour chunks of 90+ tracks) on a regular basis, I'd recommend staying away from the budget hard drives (USB-only externals in general and Seagate specifically).

Streaming that many files...
This would be a good job for a SSD. Remove all that seeking from the equation. That and remove the need for the CPU to manage a USB connection and watch your performance increase by magnitudes.


There's also a preference setting in Reaper to split files every xGB when recording. If you were to set this to 1GB or 2GB, in the above scenario you would have got the error and crash when the first file split happened.
__________________
Mac Pro 8x3.33GHz i7, 16GB, 256GB SSD(OS, apps), 3x2TB 7200 HD(data); MacBook Pro 2.8GHz, 6GB, 128GB SSD HD(OS, apps), 750GB 7200 HD(data); 2xTrue Precision 8; Apogee AD-16; 2xMOTU 828mk3, Evolution UC-33e; Faderport; WiRanger, iPad & the analog mixer has retired
serr is offline   Reply With Quote
Old 04-13-2015, 02:55 AM   #12
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Ironically I usually record onto a Raid 1 which I did not use in this case because we had several power losses during the rehearsal period. So I switched to bus powered drives. I obviously better should not have done that...

As said, my experience with USB3 has been absolutely flawless so far. CPU load is negligible with reaper.

SSDs in the required size are still costly, let's see what the next years bring.

Another thought on the 'black hole': If I get i correctly, reaper does not get any kind of feedback from the drive, saying "the last package has been written". Thus reaper builds the waveform displays out of an own cache?
A workaround for checking the drive connection on a regular basis might then be to simply save the session every few minutes? (however, why was there no feedback from autosave then?)

Excuse my persistent questions, I am just curious on this.
ribbon is offline   Reply With Quote
Old 04-13-2015, 09:31 AM   #13
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 6,491
Default

Normally if a drive were to fail, you would in fact get an error. Either a 'file could not be read or written to' error or a 'improper device removal' error.

It sounds like in this case the hardware drive itself had a failure. But it's logic was still running, connected to the computer, and responding as though it was accepting the data stream and writing it to the open files. Except it was actually crashing and burning and not actually functioning.

In this scenario, Reaper and even OSX could do nothing for you because they were being 'told' the hardware was functioning. This is the one scenario where the only thing that could save you would be a raid 1 setup.
__________________
Mac Pro 8x3.33GHz i7, 16GB, 256GB SSD(OS, apps), 3x2TB 7200 HD(data); MacBook Pro 2.8GHz, 6GB, 128GB SSD HD(OS, apps), 750GB 7200 HD(data); 2xTrue Precision 8; Apogee AD-16; 2xMOTU 828mk3, Evolution UC-33e; Faderport; WiRanger, iPad & the analog mixer has retired
serr is offline   Reply With Quote
Old 04-13-2015, 11:32 AM   #14
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

Quote:
Originally Posted by serr View Post
In this scenario, Reaper and even OSX could do nothing for you because they were being 'told' the hardware was functioning. This is the one scenario where the only thing that could save you would be a raid 1 setup.
Well, if the raid controller fails and tells the system everything was fine we end up at the same point.

Nothing like a backup recorder.
ribbon is offline   Reply With Quote
Old 04-23-2015, 03:46 AM   #15
ribbon
Human being with feelings
 
Join Date: Mar 2015
Posts: 11
Default

In case anybody is interested in this:

I found the potential source for all the trouble. A sensor on the computer's logic board was faulty and caused system components to reboot randomly.

I hope to kill the beast by replacing the system.

Thanks for your help!
ribbon is offline   Reply With Quote
Old 04-23-2015, 06:18 AM   #16
Ollie
Super Moderator (no feelings)
 
Ollie's Avatar
 
Join Date: Dec 2007
Location: On or near a dike
Posts: 9,669
Default

Crossing my fingers for you. Issues like this (very intermittent, very likely hardware-based, no or at best hard-to-find clues) are really bad and a troubleshooting nightmare.
Ollie is offline   Reply With Quote
Old 04-24-2015, 03:56 PM   #17
Hypex
Human being with feelings
 
Join Date: Mar 2015
Location: Australia
Posts: 101
Default

Quote:
Originally Posted by ribbon View Post
I found the potential source for all the trouble. A sensor on the computer's logic board was faulty and caused system components to reboot randomly.
That's a hard thing to track down. So in this case it would affect USB. Still, I would have expected some kind of error, being that a data packet would be sent off the bus and should have got an "okay" packet back. But, if OSX is like Linux and it can act similar, then all errors will go to some useless error log and you will will get no error on screen. Whoever thought that was a smart idea hasn't lost data I think!
Hypex is offline   Reply With Quote
Old 04-24-2015, 05:17 PM   #18
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 6,491
Default

Quote:
Originally Posted by ribbon View Post
In case anybody is interested in this:

I found the potential source for all the trouble. A sensor on the computer's logic board was faulty and caused system components to reboot randomly.

I hope to kill the beast by replacing the system.

Thanks for your help!
That's an odd scenario too.

Are you saying you ran the Apple service diagnostic (ASD 3S150 for that machine) and got a fail on a sensor? Which one?

I've seen stranger things than that but my first reaction would be if a failing chip on the logic board is causing other chips to reset... that computer isn't really going to be running at all.
Normally if just about any of the various sensors go out on a Mac the fans will kick on full blast. In my experience anyway.

You had a bus powered external USB drive right?
I bet that thing had a power issue and the drive controller is what reset (or flaked out in some way).
__________________
Mac Pro 8x3.33GHz i7, 16GB, 256GB SSD(OS, apps), 3x2TB 7200 HD(data); MacBook Pro 2.8GHz, 6GB, 128GB SSD HD(OS, apps), 750GB 7200 HD(data); 2xTrue Precision 8; Apogee AD-16; 2xMOTU 828mk3, Evolution UC-33e; Faderport; WiRanger, iPad & the analog mixer has retired
serr is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 03:40 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.