Old 02-14-2018, 08:02 AM   #1
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 7,679
Default ReaScript EnumerateFiles function Cache files list ?

Hi,


It seeùs that the EnumerateFiles cache file list for each subfolder.

This means that if you os.rename so file, you will not see the file list change, even if you recall EnumerateFiles .

Here is a sample code:

Quote:
function EnumerateFiles( folder )
local files = {}
local i = 0
repeat
local retval = reaper.EnumerateFiles( folder, i )
table.insert(files, retval)
i = i + 1
until not retval
return files
end

path = "E:\\Bureau\\Test\"

files_a = EnumerateFiles( path )
retval, message = os.rename(path .. "1.txt", path .. "4.txt")
files__b = EnumerateFiles( path )
And here is the result:



But note that the files do change, it is just the EnumerateFiles which stay the same.

Here is test code from cfillion;



I would have expect EnumerateFiles to NOT cache the result, and let the coder cache it by himself and use virtual folder structure if he want (surely faster), at least as an optional argument, as it is a bit unexpected (and undocumented) behavior.

Thanks !
X-Raym is offline   Reply With Quote
Old 02-14-2018, 06:12 PM   #2
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 7,679
Default

According to cfillion test, the cache last 17 ticks (approx 500ms).
X-Raym is offline   Reply With Quote
Old 01-05-2021, 01:55 AM   #3
Buy One
Human being with feelings
 
Join Date: Sep 2019
Posts: 531
Default

This is true for reaper.EnumerateSubdirectories as well

When checking if a directory has been created immediately afterwards it's not registered and not getting listed in the output.

Can be cured with
Code:
local t_point = reaper.time_precise()
repeat until reaper.time_precise() - t_point > 0.5

Last edited by Buy One; 01-05-2021 at 02:03 AM.
Buy One is offline   Reply With Quote
Old 01-05-2021, 07:08 AM   #4
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: Apr 2020
Location: Leipzig
Posts: 2,716
Default

Yeah. The reason, why my GetProjectStateChunk-function fails often.

The update latency can be much longer than 500ms.
Sometimes waiting 20 seconds isn't enough.
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper
Bugreports&Docs notes please do here:https://github.com/Ultraschall/ultra...-reaper/issues - Donate, if you wish
Meo-Ada Mespotine is online now   Reply With Quote
Old 01-06-2021, 05:05 PM   #5
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 13,697
Default

To clear the cache, you can scan a different path. Unfortunately there's a bug in the caching logic where the other path has to be valid (oops). So, we'll be fixing the caching logic, and also adding an additional thing where passing idx=-1 will invalidate the cache.
Justin is offline   Reply With Quote
Old 01-06-2021, 05:15 PM   #6
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 13,697
Default

Quote:
Originally Posted by Meo-Ada Mespotine View Post
Yeah. The reason, why my GetProjectStateChunk-function fails often.

The update latency can be much longer than 500ms.
Sometimes waiting 20 seconds isn't enough.
Hmm not sure I understand this -- can you explain?
Justin is offline   Reply With Quote
Old 01-21-2021, 08:46 AM   #7
Buy One
Human being with feelings
 
Join Date: Sep 2019
Posts: 531
Default

Should be fixed in build 6.20

from the change log

Quote:
+ ReaScript: EnumerateFiles()/EnumerateSubDirectories() allow manual cache invalidation [t=203235]
+ ReaScript: fix EnumerateFiles()/EnumerateSubDirectories() cache bug [t=203235]
Now cache clearing besides passing -1 also works with invalid directory

Last edited by Buy One; 04-25-2021 at 01:42 PM.
Buy One is offline   Reply With Quote
Old 01-21-2021, 10:17 AM   #8
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: Apr 2020
Location: Leipzig
Posts: 2,716
Default

It is and works like a charm.

One needs to include an additional functioncall of it, where the index=-1 to make it work.
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper
Bugreports&Docs notes please do here:https://github.com/Ultraschall/ultra...-reaper/issues - Donate, if you wish
Meo-Ada Mespotine is online now   Reply With Quote
Old 01-21-2021, 11:33 AM   #9
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 7,679
Default

Quote:
Originally Posted by Meo-Ada Mespotine View Post
One needs to include an additional functioncall of it, where the index=-1 to make it work.



Whatfor ?


--

Thanks for the fix !!
X-Raym is offline   Reply With Quote
Old 01-21-2021, 02:58 PM   #10
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: Apr 2020
Location: Leipzig
Posts: 2,716
Default

To clear the cache before reenumerating the folder/files. Otherwise it will have the old behavior, as far as I understood.
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper
Bugreports&Docs notes please do here:https://github.com/Ultraschall/ultra...-reaper/issues - Donate, if you wish
Meo-Ada Mespotine is online now   Reply With Quote
Old 01-21-2021, 03:43 PM   #11
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 7,679
Default

@meo-mespotine
I dont understand, the new functions precisely does that, invalidate the cache so the directory can be rescanned.
X-Raym is offline   Reply With Quote
Old 01-21-2021, 04:22 PM   #12
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: Apr 2020
Location: Leipzig
Posts: 2,716
Default

Only, if you set file/dirindex to -1.

It's described in the docs that way.

And they are actually quite old. They just got enhanced.

Edit:

For a get all resource path-files function, you would add one more line, the one that refreshes the cache(index -1).
After that, you can be sure, that you always get the most recent files in the folder available.
Same goes for EnumerateSubdirectories().

Code:
function GetAllResourcePathFiles()
  reaper.EnumerateFiles(reaper.GetResourcePath(), -1) -- refresh the cache,
                                                      -- after this, the filelist will be updated
                                                      -- to the most recent one
  
  -- get all current files in resourcepath
  local index=0
  local filelist={}
  while reaper.EnumerateFiles(reaper.GetResourcePath(), index)~=nil do
    filelist[index+1]=reaper.EnumerateFiles(reaper.GetResourcePath(), index)
    index=index+1
  end
  
  return index, filelist
end

A,B=GetAllResourcePathFiles()
According to Justin, older versions of Reaper allow the same, but need a different approach, by enumerating a file in a different subdirectory than the last one used.
So the next should refresh the cache on older Reaper versions than 6.20:

Code:
function GetAllFilenamesInPath(path)
  -- let's refresh the folder by enumerating a file in a different folder
  reaper.EnumerateFiles(reaper.GetResourcePath(), 1) -- read first file of resource-path; 
  -- to avoid the fact, that the last directory could be the resource-path used by the scripter outside this function,
  -- let's enumerate a different file in a different folder as well to refresh the cache definitely
  reaper.EnumerateFiles(reaper.GetResourcePath().."/ColorThemes", 1) -- read first file of resource-path; 

  -- now, the cache is using the ColorThemes-folder. 
  -- So if we change it again to scriptpath in our example, Reaper will refresh the cache again to the latest one
  local index=0
  local filelist={}
  while reaper.EnumerateFiles(path, index)~=nil do
    filelist[index+1]=reaper.EnumerateFiles(path, index)
    index=index+1
  end
  
  return index, filelist
end

A,B=GetAllFilenamesInPath(reaper.GetResourcePath().."/Scripts")
This is the way to do it on Reaper 6.19 and earlier.
It's important, that the paths exist, when you do the two lines to refresh the cache.
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper
Bugreports&Docs notes please do here:https://github.com/Ultraschall/ultra...-reaper/issues - Donate, if you wish

Last edited by Meo-Ada Mespotine; 01-21-2021 at 04:42 PM.
Meo-Ada Mespotine is online now   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 11:30 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.