Script: Copy Files to Project Folder (WIN ONLY). And a question about it [SOLVED]

[dsyrock]

I made a request for an action to copy files to project folder before, https://forum.cockos.com/showthread.php?t=223241 .

Recently I finally creat this script. It works well, but comparing to the build in way which is doing it in Media Bay, it is much slower.

I've try my best to optimize the algorithm in this script, but it make nearly no effort on speed.

What make it so slow? And why the hell does Media Bay copy files so fast?


UPDATE

After @snooks releasing his incredible extension "fileops", https://forum.cockos.com/showthread.php?t=225701
I'm able to speed up my script. And thanks to MusoBob's advise, now the script doesn't force to rebuild peaks (in some specific conditions).

Now you need to copy the fileops.dll file to your Userplugins folder, then run the script.

If you CHECK the option

Put new peak files in peaks/ subfolder relative to media (Preferences--Media)

and UNCHECK

Store all peak caches (.reapeaks)in alternate path: (Preferences--General--Paths)

the script will copy each peak file to your project folder and doesn't need to rebuild them. Otherwise it will force to rebuild peak files, cause when you store them all in a folder, they will get random names.

[Meo-Ada Mespotine]

You use ExecProcess, right?
This uses the command-line and this makes execution much much slower, as the commandline must be started time and again and this slows down the whole stuff.

I don't know the options of the copy-command, but maybe you could create a .bat-file, which hold all copy-commands in it and you only start the ExecProcess-function once.
(If I got the code right, I just threw a glance at it)

[dsyrock]

Originally Posted by mespotine

You use ExecProcess, right?
This uses the command-line and this makes execution much much slower, as the commandline must be started time and again and this slows down the whole stuff.

I don't know the options of the copy-command, but maybe you could create a .bat-file, which hold all copy-commands in it and you only start the ExecProcess-function once.
(If I got the code right, I just threw a glance at it)

Yes, I use ExecProcess, but not only in copy command, but also in getting files' md5 value using "certutil -hashfile filename MD5", to tell if they are duplicate or not. Cause ExecProcess can get messages that the command return. But in MediaBay, it seems doesn't need any time to judge and compare any files and then start to copy as soon as I click

[snooks]

It might be much faster using Lua IO calls instead of using OS calls. I've changed your main function to open the files and read the contents to directly compare and write when copying. It's completely untested (*see edit), but should/might work just now:

function copy_to_project() -- main function
    local num = reaper.CountSelectedMediaItems(0)
    if num == 0 then return end

    reaper.Main_OnCommand(40440, 0) -- offline
    local files = getProjectMediaFilenames()
    local files_copy = {}

    local file_ori = "" -- a string to concat all file path
    if #files > 0 then
        table.sort(files, byname_only) -- sort by name
        for k, v in pairs(files) do
            file_ori = file_ori .. v .. "|" -- seperate each file path with a "|", such as "a.wav|b.wav|c.wav|..."
        end
    end

    for i = 0, num - 1 do -- iterate and store all selected items' file path
        local it = reaper.GetSelectedMediaItem(0, i)
        local tk = reaper.GetActiveTake(it)
        if tk then
            local sr = reaper.GetMediaItemTake_Source(tk)
            local type_name = reaper.GetMediaSourceType(sr, "")
            if type_name ~= "MIDI" then
                local sr, sr_file = get_take_file(tk)
                local path_only, name = DSY_path(sr_file)
                files_copy[#files_copy + 1] = {}
                files_copy[#files_copy].tk = tk
                files_copy[#files_copy].path = sr_file
                files_copy[#files_copy].name = name
            end
        end
    end

    if #files_copy > 0 then
        table.sort(files_copy, byname) -- resort by name
        local path_sr_last, path_des_last
        for k, v in pairs(files_copy) do -- iterate all the files which are going to copy to project folder
            local path_only = DSY_path(v.path) -- extract path from the whole path from "E:\media\a.wav" to "E:\media\"
            local copy_mode = "normal" -- normal means no duplicate file in project folder, will copy it directly
            if path_only ~= path_local then -- file is isn't in project folder, means it need to copy
                if v.path == path_sr_last then -- this file has already been copying to folder, just connect it to the file
                    reaper.BR_SetTakeSourceFromFile(v.tk, path_des_last, true)
                else
                    path_sr_last = v.path -- save it as the last path

-- changed          -- get contents of file to be potentially copied
                    local fh_copy = io.open(v.path, "rb")
                    local copy_contents = fh_copy:read("*a")
                    fh_copy:close()
                    
                    if #files > 0 then
                        if file_ori:find(v.name, 1, true) then -- search the filename in the big string "file_ori"
                            local path_ori = path_local .. v.name -- found a file with same name, concat the filename with project media path
                            
-- changed                  -- get contents of file with same name...
                            local fh_path_ori = io.open(path_ori, "rb")
                            local path_ori_contents = fh_path_ori:read("*a")
                            fh_path_ori:close()
                            
                            -- ... and compare
                            if copy_contents == path_ori_contents then
                                reaper.BR_SetTakeSourceFromFile(v.tk, path_ori, true) -- connect it to the old file
                                copy_mode = "no" -- "no" means doesn't need to copy, cause there is a duplicate file
                                path_des_last = path_ori -- save it as the last path
                            else -- if they have different contents
                                copy_mode = "new" -- "new" means they are not duplicate files
                                local name_new, path_new = copy_new(v.name, v.path, path, v.tk) -- copy it to project folder and rename it
                                path_des_last = path_new -- save it as the last path
                            end

                            file_ori = file_ori:gsub(v.name .. "|(.+)", "%1") -- for example, "a.wav|b.wav|c.wav|d.wav|", if b.wav is a duplicate file, the string before b.wav is useless. Cut it to save time.
                        end
                    end

                    if copy_mode == "normal" then -- normal means the file is going to be copied to project folder directly
                        -- cmd command: copy copy_source_filepath copy_destination_path
                        local copy_source = v.path
                        if not copy_source:find("\\") then
                            copy_source = copy_source:gsub("/([^/]+)$", "\\%1") -- if the last seperator is "/", the copy command will fail
                        end

                        -- write destination file with contents of file to be copied
-- changed              local copy_dest = path .. "\\" ..  v.name
                        local fh_copy_dest = io.open(copy_dest, "wb")
                        fh_copy_dest:write(copy_contents)
                        fh_copy_dest:close()
                        
                        reaper.BR_SetTakeSourceFromFile(v.tk, copy_dest, true)
                        path_des_last = copy_des .. v.name -- save it as the last path
                    end
                end
            end
        end
    end
    reaper.UpdateArrange()
    reaper.Main_OnCommand(40441, 0) -- rebuild peaks
    reaper.Main_OnCommand(41858, 0) -- set take name from file
end

You should probably change the "copy_new" function too so that you are passing it the contents that have been read (copy_contents) and it is using the same IO methods. Let me know if you need any help!

Btw, one tiny piece of advice. If you see the code has multiple levels of indentation it's probably time to create some sensibly named functions so that it's nicer to read the code and it documents itself.

edit: I tested copying a few files with Lua and the "b" (for binary) flag needed to be set for reading and writing.

[Meo-Ada Mespotine]

One thing I would add in general to comparing files.
I think, Reaper doesn't actually compare the files, but rather the modification-date and time.

So if the modificationtime changes, Reaper knows that the sourcefile has been changed somehow and readds the stuff again in the MediaBay.
So you should rather check, whether the last modification changes since last time checking the files.
Maybe by putting the last timestamp into a ProjExtState.
And if you find a later modification-date in the files than the one your stored into the ProjExtState, you can do the copying.

This speeds things up by magnitudes and Reaper does this in many occasions.
Comparing files, especially if there are many/big ones is always very very slow.

[MusoBob]

dsyrock script was working for me but not snooks the script saved but did not do anything.
I tried this on Win but are these going to work on Mac ?

[Meo-Ada Mespotine]

I think it doesn't, as it's made for Windows specifically.

Though I think, it should be alterable for Mac and Linux-compatibility.

As far as I could see, one would need to alter the folder-separator for that to work.

[snooks]

The bit I posted was just one function so doesn't do anything by itself. If it's put where it should be in the original script it will (should) work on Windows only still. As well as what mespotine says, there's another function that uses a Windows shell command.

[dsyrock]

Originally Posted by snooks

It might be much faster using Lua IO calls instead of using OS calls.

Wow that's a clever way to copy files, I didn't know it before. Thanks

[dsyrock]

Originally Posted by mespotine

One thing I would add in general to comparing files.
I think, Reaper doesn't actually compare the files, but rather the modification-date and time.

So if the modificationtime changes, Reaper knows that the sourcefile has been changed somehow and readds the stuff again in the MediaBay.
So you should rather check, whether the last modification changes since last time checking the files.
Maybe by putting the last timestamp into a ProjExtState.
And if you find a later modification-date in the files than the one your stored into the ProjExtState, you can do the copying.

This speeds things up by magnitudes and Reaper does this in many occasions.
Comparing files, especially if there are many/big ones is always very very slow.

Yeah I think comparing modification-date is much faster then comparing file content. But can Lua get a file's modification-date directly?

[dsyrock]

Originally Posted by mespotine

I think it doesn't, as it's made for Windows specifically.

Though I think, it should be alterable for Mac and Linux-compatibility.

As far as I could see, one would need to alter the folder-separator for that to work.

And the copy command I used that only work in Windows. Maybe using snoobs'way could avoid this problem

[Meo-Ada Mespotine]

Originally Posted by dsyrock

Yeah I think comparing modification-date is much faster then comparing file content. But can Lua get a file's modification-date directly?

Unfortunately not. I requested a function for SWS that would do that but with no luck til now.

So my best guess would be using dir or ls recursively and parse it's output with pattern matching.

[MusoBob]

I will see how you guys go with the script here if not I will have a look at them and try and the put the reaper.GetOS() for the backslash/forward slash and shell/terminal.

[Lokasenna]

All three OSes should be fine with / as a separator; it's the commands and how they take arguments that will be different.

[dsyrock]

Originally Posted by snooks

using Lua IO calls instead of using OS calls

I did a quick test

file1="k:\\test1.mp4"

time=os.clock()
reaper.ExecProcess("cmd /c copy k:\\test1.mp4 k:\\test3.mp4", 0)

msg("using copy command: "..os.clock()-time)

time=os.clock()
local file=io.open(file1, "rb")
local content = file:read("*a")
file:close()
local file = io.open("K:\\test4.mp4", "wb")
file:write(content)
file:close()
msg("using file write: "..os.clock()-time)

And the result is

using copy command: 0.067999999999302
using file write: 0.37300000002142

Seems like using copy command is faster than usintg Lua IO calls.

[MusoBob]

The original script you made was fast enough for me.
It does have to re-create the reapeaks, could you copy them also would that speed it up ?
If you copy the reapeaks to the folder first it won't need to create them when the source wav is changed.

[dsyrock]

Originally Posted by MusoBob

The original script you made was fast enough for me.
It does have to re-create the reapeaks, could you copy them also would that speed it up ?
If you copy the reapeaks to the folder first it won't need to create them when the source wav is changed.

I'm not sure if it's possible to copy only the specialized peaks files.

[MusoBob]

I haven't really look into it but could you just run a function first to copy sr_file..".reapeaks" then the next function for just copy sr_file
if the sr_file contains mp3 then copy sr_file..".reapindex" also (if it exists as mp3 needs to be over a certain length to create .reapindex also).
I will look at it more in the morning.

[snooks]

Originally Posted by dsyrock

I did a quick test

file1="k:\\test1.mp4"

time=os.clock()
reaper.ExecProcess("cmd /c copy k:\\test1.mp4 k:\\test3.mp4", 0)

msg("using copy command: "..os.clock()-time)

time=os.clock()
local file=io.open(file1, "rb")
local content = file:read("*a")
file:close()
local file = io.open("K:\\test4.mp4", "wb")
file:write(content)
file:close()
msg("using file write: "..os.clock()-time)

And the result is

using copy command: 0.067999999999302
using file write: 0.37300000002142

Seems like using copy command is faster than usintg Lua IO calls.

It depends on the size of the file, for small files it's much faster.

I've attached a dll that wraps some Windows file commands with Lua. Here's code for the benchmark I did that shows how to use the dll too:

local function print(...)
  local args = { n = select("#", ...), ... }
  for i = 1, args.n do
    reaper.ShowConsoleMsg(tostring(args[i]) .. "\t   :   ")
  end
  reaper.ShowConsoleMsg("\n")
end

local path = "C:/Users/paul-/AppData/Roaming/REAPER/UserPlugins/fileops.dll"
local copyFile = package.loadlib(path, "copyFile")
assert(copyFile, "\nError: failed to load function from dll")
-- also compareFileTimesFromPaths and getLastModified functions in there

local function sleep(a) 
  local sec = tonumber(os.clock() + a); 
  while (os.clock() < sec) do 
  end 
end

local function copy_file_lua_io(path_a, path_b, msg)
  local start_time = os.clock()
  local src = io.open(path_a, "rb")
  local dst = io.open(path_b, "wb")
  if not src then print("Error: could not open file.") return end
  local contents = src:read("*a")
  dst:write(contents)
  src:close()
  dst:close()
  local end_time = os.clock()
  print("Lua API " .. msg .. " Time: ", end_time - start_time)
end

local function copy_file_win_api(path_a, path_b, msg)
  local start_time = os.clock()
  local ok, res = copyFile(path_a, path_b)
  local end_time = os.clock()
  print("Win API " .. msg .. " Time: ", end_time - start_time)
end

local function copy_file_execprocess(path_a, path_b, msg)
  local start_time = os.clock()
  reaper.ExecProcess("cmd /c copy " .. path_a .. " " .. path_b, 0)
  local end_time = os.clock()
  print("ExecProcess " .. msg .. " Time: ", end_time - start_time)
end

do
  local src_path = "c:/dev/lua/wav-file.wav"
  local dst_path = "c:/dev/lua/wav-file-copy.wav"
  local msg = "0.6MB"
  copy_file_lua_io(src_path, dst_path, msg)
  copy_file_win_api(src_path, dst_path, msg)
  copy_file_execprocess(src_path, dst_path, msg) 
end

do
  local src_path = "c:/dev/lua/video.mp4"
  local dst_path = "c:/dev/lua/video-copy.mp4"
  local msg = "76MB"
  copy_file_lua_io(src_path, dst_path, msg)
  copy_file_win_api(src_path, dst_path, msg)
  copy_file_execprocess(src_path, dst_path, msg)
end

Here's my results for a couple of different sized files:

Lua API 0.6MB Time: 	   :   0.012000000000626	  
Win API 0.6MB Time: 	   :   0.0020000000004075	    
ExecProcess 0.6MB Time:   :   0.096000000000458	    
Lua API 76MB Time: 	   :   0.29300000000057	  
Win API 76MB Time: 	   :   0.0649999999996	  
ExecProcess 76MB Time:    :   0.078000000000429

[Edgemeal]

Originally Posted by snooks

I've attached a dll that wraps some Windows file commands with Lua.

Nice idea! How do I use getlastmodified? I tried like this but only returned the year and a zero in the format, "2019.0".

local path = "M:/Reaper Portable/UserPlugins/fileops.dll"
local LastModified = package.loadlib(path, "getLastModified")
assert(LastModified, "\nError: failed to load function from dll")

local function GetLastModified(file)
  local ok, res = LastModified(file)
  return res 
end

fn = 'M:/Metal Sux 2.rpp'
date = GetLastModified(fn)
msg(date) -- returns '2019.0' ???

Win10 (1903), REAPER v5.983/x64

[snooks]

It's a bit stupid at the moment, it returns 8 arguments:

local ok, year_or_error_msg, month, day, hour, minute, second, millisecond =
  getLastModified()

I should really keep the raw numbers from...

https://docs.microsoft.com/en-us/win...pi-getfiletime

... and convert for easy comparison.

The current numbers follow this format:

https://docs.microsoft.com/en-us/win...ase-systemtime

[Edgemeal]

Originally Posted by snooks

It's a bit stupid at the moment, it returns 8 arguments:

OK cool, I don't know how to compare dates in Lua, but I see you have a compare function, but I must be using that wrong too , it just returns 0.0.

function CompareModified(fileA, fileB)
  local ok, res = compareLastModified(fileA, fileB )
  msg(ok)  -- returns 'true'
  msg(res) -- returns '0.0'  ???
end

[snooks]

That one is from here...

https://docs.microsoft.com/en-us/win...omparefiletime

... so the Return Value section of that page tells you what the -1, 0 or 1 return value mean. It's good for a quick check.


edit

Readers from the both the near and far future, there is a thread that documents this now:

https://forum.cockos.com/showthread.php?t=225701

[Edgemeal]

Originally Posted by snooks

That one is from here...

https://docs.microsoft.com/en-us/win...omparefiletime

... so the Return Value section of that page tells you what the -1, 0 or 1 return value mean. It's good for a quick check.

Oh OK it works, but oddly if I copy files to a new folder and compare them it returns zero. I just assumed the function was comparing the files using LastWriteTime.

[snooks]

Yeah, so did I.

I'll get a handier function in there l8trs. I created a thread to document the dll since I hadn't here, I'll update there when done.

[dsyrock]

Originally Posted by snooks

I've attached a dll that wraps some Windows file commands with Lua.

Wow that's an incredible things! Is that dll thing you make it? Are there any more dll files like this?

[dsyrock]

@snooks
A question about fileops.dll. When a path or filename contains non-English words, copyFile(pathA, pathB) doesn't work. It returns a error message: Error: first arg path does not exist. It seems that it doesn't support UTF-8 encoding?

Or is there any document about it?

[snooks]

Originally Posted by dsyrock

Wow that's an incredible things! Is that dll thing you make it? Are there any more dll files like this?

Yes, I made it and it's the first I've seen. It's another good tool to have in the box!

It seems that it doesn't support UTF-8 encoding?

Ooh, I'll look into that. Thanks for letting me know!

一会见

[MusoBob]

These are only short files 32 bars.
I tied it without copying the reapeaks first

Full Screen


then with copying the reapeaks first

Full Screen

[MusoBob]

Here's 5 x 128 bars

Full Screen


then with copying the reapeaks first

Full Screen

[dsyrock]

Originally Posted by Lokasenna

All three OSes should be fine with / as a separator; it's the commands and how they take arguments that will be different.

But in Windows, the last separator of a LOCAL FILE PATH should not be a "/"

copy C:\Project\Media\Test.wav C:\Project\Media\Test1.wav

This work.

copy C:\Project\Media/Test.wav C:\Project\Media/Test1.wav

This will give you an error message.

[Lokasenna]

Huh.

I had read ages ago that forward slashes were fine too, tested out a bunch of commands and they all worked, but missed that one. Just did some reading and it sounds like the Windows devs don't know how to be consistent. :/

Thanks!

[dsyrock]

Originally Posted by MusoBob

then with copying the reapeaks first

Yeah it is a good way. But in my case, I store all the peak files in a specific folder, instead if the project media folder. So I think copy the peak file is not a common solution that fits everyone.

[MusoBob]

If it's in the REAPER.ini

altpeaks=1

then copy to

altpeakspath=C:\Users\User\Documents\REAPER Media\REAPEAKS
altpeaksopathlist=

else

altpeaks=0

copy to project folder

[dsyrock]

If I don't rebuild peaks, it will look like this

[MusoBob]

I'm not getting that:

[dsyrock]

Originally Posted by MusoBob

I'm not getting that:

Well it's weird...

[MusoBob]

If the peaks are already there it adds -import on the copied wav
so to get it to work when I tried it I just changed

local name_new=name:sub(1, edge-1)..""..tale --.."-import"..tale

to get it to keep the original file name.

[dsyrock]

Already updated a new version at 1#

[MusoBob]

Thanks !