ReaScript EnumerateFiles function Cache files list ?

[X-Raym]

Hi,


It seeùs that the EnumerateFiles cache file list for each subfolder.

This means that if you os.rename so file, you will not see the file list change, even if you recall EnumerateFiles .

Here is a sample code:

function EnumerateFiles( folder )
local files = {}
local i = 0
repeat
local retval = reaper.EnumerateFiles( folder, i )
table.insert(files, retval)
i = i + 1
until not retval
return files
end

path = "E:\\Bureau\\Test\"

files_a = EnumerateFiles( path )
retval, message = os.rename(path .. "1.txt", path .. "4.txt")
files__b = EnumerateFiles( path )

And here is the result:



But note that the files do change, it is just the EnumerateFiles which stay the same.

Here is test code from cfillion;



I would have expect EnumerateFiles to NOT cache the result, and let the coder cache it by himself and use virtual folder structure if he want (surely faster), at least as an optional argument, as it is a bit unexpected (and undocumented) behavior.

Thanks !

[X-Raym]

According to cfillion test, the cache last 17 ticks (approx 500ms).

[Buy One]

This is true for reaper.EnumerateSubdirectories as well

When checking if a directory has been created immediately afterwards it's not registered and not getting listed in the output.

Can be cured with

local t_point = reaper.time_precise()
repeat until reaper.time_precise() - t_point > 0.5

[Meo-Ada Mespotine]

Yeah. The reason, why my GetProjectStateChunk-function fails often.

The update latency can be much longer than 500ms.
Sometimes waiting 20 seconds isn't enough.

[Justin]

To clear the cache, you can scan a different path. Unfortunately there's a bug in the caching logic where the other path has to be valid (oops). So, we'll be fixing the caching logic, and also adding an additional thing where passing idx=-1 will invalidate the cache.

[Justin]

Originally Posted by Meo-Ada Mespotine

Yeah. The reason, why my GetProjectStateChunk-function fails often.

The update latency can be much longer than 500ms.
Sometimes waiting 20 seconds isn't enough.

Hmm not sure I understand this -- can you explain?

[Buy One]

Should be fixed in build 6.20

from the change log

+ ReaScript: EnumerateFiles()/EnumerateSubDirectories() allow manual cache invalidation [t=203235]
+ ReaScript: fix EnumerateFiles()/EnumerateSubDirectories() cache bug [t=203235]

Now cache clearing besides passing -1 also works with invalid directory

[Meo-Ada Mespotine]

It is and works like a charm.

One needs to include an additional functioncall of it, where the index=-1 to make it work.

[X-Raym]

Originally Posted by Meo-Ada Mespotine

One needs to include an additional functioncall of it, where the index=-1 to make it work.

Whatfor ?


--

Thanks for the fix !!

[Meo-Ada Mespotine]

To clear the cache before reenumerating the folder/files. Otherwise it will have the old behavior, as far as I understood.

[X-Raym]

@meo-mespotine
I dont understand, the new functions precisely does that, invalidate the cache so the directory can be rescanned.

[Meo-Ada Mespotine]

Only, if you set file/dirindex to -1.

It's described in the docs that way.

And they are actually quite old. They just got enhanced.

Edit:

For a get all resource path-files function, you would add one more line, the one that refreshes the cache(index -1).
After that, you can be sure, that you always get the most recent files in the folder available.
Same goes for EnumerateSubdirectories().

function GetAllResourcePathFiles()
  reaper.EnumerateFiles(reaper.GetResourcePath(), -1) -- refresh the cache,
                                                      -- after this, the filelist will be updated
                                                      -- to the most recent one
  
  -- get all current files in resourcepath
  local index=0
  local filelist={}
  while reaper.EnumerateFiles(reaper.GetResourcePath(), index)~=nil do
    filelist[index+1]=reaper.EnumerateFiles(reaper.GetResourcePath(), index)
    index=index+1
  end
  
  return index, filelist
end

A,B=GetAllResourcePathFiles()

According to Justin, older versions of Reaper allow the same, but need a different approach, by enumerating a file in a different subdirectory than the last one used.
So the next should refresh the cache on older Reaper versions than 6.20:

function GetAllFilenamesInPath(path)
  -- let's refresh the folder by enumerating a file in a different folder
  reaper.EnumerateFiles(reaper.GetResourcePath(), 1) -- read first file of resource-path; 
  -- to avoid the fact, that the last directory could be the resource-path used by the scripter outside this function,
  -- let's enumerate a different file in a different folder as well to refresh the cache definitely
  reaper.EnumerateFiles(reaper.GetResourcePath().."/ColorThemes", 1) -- read first file of resource-path; 

  -- now, the cache is using the ColorThemes-folder. 
  -- So if we change it again to scriptpath in our example, Reaper will refresh the cache again to the latest one
  local index=0
  local filelist={}
  while reaper.EnumerateFiles(path, index)~=nil do
    filelist[index+1]=reaper.EnumerateFiles(path, index)
    index=index+1
  end
  
  return index, filelist
end

A,B=GetAllFilenamesInPath(reaper.GetResourcePath().."/Scripts")

This is the way to do it on Reaper 6.19 and earlier.
It's important, that the paths exist, when you do the two lines to refresh the cache.