Scan all databases action rescans existing files

[Travesty]

Hi,

Recently I made a startup script which scans all databases.

Now each time I open reaper it starts scanning files which have not changed in ages. I have databases of gamedata sounds which are years old, but reaper starts scanning through sounds and dialogue which have definitely not changed.

Even if I close and reopen reaper it will start scanning things.


Repro steps

Create a database from a folder with a large amount of samples in it. It needs to be a big library to show the issue, hundreds of thousands of files. Tested on a library with around 180k files.
When database has finished scanning, rescan the database

The scan will seemingly find new files each time. The number of files it "discovers" seems to increase with time. If you immediately scan again, it will be fairly quick, but the next day, it will find more files and take longer.

As I have a startup action which rescans all databases, I see activity daily from purchased libraries which have not changed.
Two of collegues have also noticed this issue. It's obvious when you have a lot of databases and you scan them all, it can take a while to get into reaper, and I often have to cancel the scan to get to work.

[Travesty]

Edited and added some repro steps. It's a longwinded repro unfortunately as it requires large databases to see the issue

[Justin]

It may be that just scanning the file system for new files takes a while? (if the file system paths are not in the OS cache, etc)

[Travesty]

I don't think so, the databases which are being scanned are years old in some cases and have not changed. Plus we only noticed this in the past couple of months.

I just opened Reaper and it's scannning a foley archive which hasn't been added to in at least 2 years.

[schwa]

I think we've got this figured out, if you are able to test 6.24rc2 that would be appreciated!

https://forum.cockos.com/forumdisplay.php?f=37

[Travesty]

Great, I'll check when it's finished scanning databases

[Travesty]

It's still happening unfortunately.

I'm noticing the same files flashing up each time though. I'll get a video and see if it is actually the same files.

[Travesty]

As far as I can tell it is the same files each time. I probably thought it was different files before, because I kept deleting old databases to speed up the scanning. I've got no more I can do without now.

I wonder if these databases have become corrupted in some way.
I can't see any similarities in the sounds themselves. Some are from last year, some from 5 years ago.
Some have bwf description metadata, some don't.

[schwa]

Could you send us one of the problem files, please? support at cockos.

[Travesty]

I've sent one over, cheers

[Travesty]

It's not just wave files either, there are rpp files, rpp-bak, zip, reapeaks...

[schwa]

Well, it doesn't seem to have anything to do with the particular file -- if I create a database with only that file, it scans once only. This is a super pain to debug because of the apparent dependency on the database containing tens of thousands of files.

Do you get the same behavior in 6.24rc2 whether you run "scan database for new files" or "scan all databases for new files" from the shortcut list context menu?

[Travesty]

Yeah exactly, it's a pain

I'll give it a go. I was just opening reaper and letting the startup action run "scan all databases for new files"

[Travesty]

I just tried scan for new files on the oldest database, the one containing the file I sent, and I get the same issue.

Seven of the other databases are fine, and one shows a few files being scanned, but it is over quite quickly. This is the largest database, and it's local to this machine.

The one which is taking all the time is the oldest one I mentioned above. This is on a network drive accessed via vpn, though I don't think that would be an issue?

I could try deleting the database and recreating it, which might work, but then we wouldn't know what caused it in the first place.

There were other databases which had the issue, but I deleted those to try and speed up the scans.

[Travesty]

I deleted the worst offending database and then recreated it. The issue still exists, so we can rule out corruption.

I found some of the files which keep popping up in the scan and made a db from just that folder. That reproduced the issue with much fewer files, but these were propriatory, so I can't share them. I narrowed this further to a freely available library of astronomical audio files.

https://www.sonidosdealma.org/en/

The website is broken at the moment, but I should be able to send the sounds over, they are open source. It's 3000 files but only 14MB size.

Creating a db from these reproduced the issue reliably, but only over the vpn on the network drive.

So I think either the issue is related to creating databases over the network somehow, or the latency is causing the issue to be visible, where it would be too quick to spot on a local drive.

[Travesty]

Hi, this is still an issue, despite having FIXED in the summary.

[Travesty]

I recently made another db from a network drive and some of these files pop up now each time I open reaper, so maybe that's the key to reproducing the issue. I don't have this issue at home, where I don't have a network.

[Tim Rideout]

Did you ever get your script to work (scan all databases action)? I am looking for such a thing to trigger suring startup, but can't seem to find (or code one with ChatGPT either..) Thanks!