|
|
|
09-28-2019, 10:00 AM
|
#1
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Script: Copy Files to Project Folder (WIN ONLY). And a question about it [SOLVED]
I made a request for an action to copy files to project folder before, https://forum.cockos.com/showthread.php?t=223241 .
Recently I finally creat this script. It works well, but comparing to the build in way which is doing it in Media Bay, it is much slower.
I've try my best to optimize the algorithm in this script, but it make nearly no effort on speed.
What make it so slow? And why the hell does Media Bay copy files so fast?
UPDATE
After @snooks releasing his incredible extension "fileops", https://forum.cockos.com/showthread.php?t=225701
I'm able to speed up my script. And thanks to MusoBob's advise, now the script doesn't force to rebuild peaks (in some specific conditions).
Now you need to copy the fileops.dll file to your Userplugins folder, then run the script.
If you CHECK the option
Put new peak files in peaks/ subfolder relative to media (Preferences--Media)
and UNCHECK
Store all peak caches (.reapeaks)in alternate path: (Preferences--General--Paths)
the script will copy each peak file to your project folder and doesn't need to rebuild them. Otherwise it will force to rebuild peak files, cause when you store them all in a folder, they will get random names.
Last edited by dsyrock; 10-07-2019 at 10:03 AM.
Reason: Update the script
|
|
|
09-29-2019, 06:20 AM
|
#2
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
You use ExecProcess, right?
This uses the command-line and this makes execution much much slower, as the commandline must be started time and again and this slows down the whole stuff.
I don't know the options of the copy-command, but maybe you could create a .bat-file, which hold all copy-commands in it and you only start the ExecProcess-function once.
(If I got the code right, I just threw a glance at it)
|
|
|
09-29-2019, 06:49 PM
|
#3
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by mespotine
You use ExecProcess, right?
This uses the command-line and this makes execution much much slower, as the commandline must be started time and again and this slows down the whole stuff.
I don't know the options of the copy-command, but maybe you could create a .bat-file, which hold all copy-commands in it and you only start the ExecProcess-function once.
(If I got the code right, I just threw a glance at it)
|
Yes, I use ExecProcess, but not only in copy command, but also in getting files' md5 value using "certutil -hashfile filename MD5", to tell if they are duplicate or not. Cause ExecProcess can get messages that the command return. But in MediaBay, it seems doesn't need any time to judge and compare any files and then start to copy as soon as I click
|
|
|
09-30-2019, 02:49 PM
|
#4
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
It might be much faster using Lua IO calls instead of using OS calls. I've changed your main function to open the files and read the contents to directly compare and write when copying. It's completely untested (*see edit), but should/might work just now:
Code:
function copy_to_project() -- main function
local num = reaper.CountSelectedMediaItems(0)
if num == 0 then return end
reaper.Main_OnCommand(40440, 0) -- offline
local files = getProjectMediaFilenames()
local files_copy = {}
local file_ori = "" -- a string to concat all file path
if #files > 0 then
table.sort(files, byname_only) -- sort by name
for k, v in pairs(files) do
file_ori = file_ori .. v .. "|" -- seperate each file path with a "|", such as "a.wav|b.wav|c.wav|..."
end
end
for i = 0, num - 1 do -- iterate and store all selected items' file path
local it = reaper.GetSelectedMediaItem(0, i)
local tk = reaper.GetActiveTake(it)
if tk then
local sr = reaper.GetMediaItemTake_Source(tk)
local type_name = reaper.GetMediaSourceType(sr, "")
if type_name ~= "MIDI" then
local sr, sr_file = get_take_file(tk)
local path_only, name = DSY_path(sr_file)
files_copy[#files_copy + 1] = {}
files_copy[#files_copy].tk = tk
files_copy[#files_copy].path = sr_file
files_copy[#files_copy].name = name
end
end
end
if #files_copy > 0 then
table.sort(files_copy, byname) -- resort by name
local path_sr_last, path_des_last
for k, v in pairs(files_copy) do -- iterate all the files which are going to copy to project folder
local path_only = DSY_path(v.path) -- extract path from the whole path from "E:\media\a.wav" to "E:\media\"
local copy_mode = "normal" -- normal means no duplicate file in project folder, will copy it directly
if path_only ~= path_local then -- file is isn't in project folder, means it need to copy
if v.path == path_sr_last then -- this file has already been copying to folder, just connect it to the file
reaper.BR_SetTakeSourceFromFile(v.tk, path_des_last, true)
else
path_sr_last = v.path -- save it as the last path
-- changed -- get contents of file to be potentially copied
local fh_copy = io.open(v.path, "rb")
local copy_contents = fh_copy:read("*a")
fh_copy:close()
if #files > 0 then
if file_ori:find(v.name, 1, true) then -- search the filename in the big string "file_ori"
local path_ori = path_local .. v.name -- found a file with same name, concat the filename with project media path
-- changed -- get contents of file with same name...
local fh_path_ori = io.open(path_ori, "rb")
local path_ori_contents = fh_path_ori:read("*a")
fh_path_ori:close()
-- ... and compare
if copy_contents == path_ori_contents then
reaper.BR_SetTakeSourceFromFile(v.tk, path_ori, true) -- connect it to the old file
copy_mode = "no" -- "no" means doesn't need to copy, cause there is a duplicate file
path_des_last = path_ori -- save it as the last path
else -- if they have different contents
copy_mode = "new" -- "new" means they are not duplicate files
local name_new, path_new = copy_new(v.name, v.path, path, v.tk) -- copy it to project folder and rename it
path_des_last = path_new -- save it as the last path
end
file_ori = file_ori:gsub(v.name .. "|(.+)", "%1") -- for example, "a.wav|b.wav|c.wav|d.wav|", if b.wav is a duplicate file, the string before b.wav is useless. Cut it to save time.
end
end
if copy_mode == "normal" then -- normal means the file is going to be copied to project folder directly
-- cmd command: copy copy_source_filepath copy_destination_path
local copy_source = v.path
if not copy_source:find("\\") then
copy_source = copy_source:gsub("/([^/]+)$", "\\%1") -- if the last seperator is "/", the copy command will fail
end
-- write destination file with contents of file to be copied
-- changed local copy_dest = path .. "\\" .. v.name
local fh_copy_dest = io.open(copy_dest, "wb")
fh_copy_dest:write(copy_contents)
fh_copy_dest:close()
reaper.BR_SetTakeSourceFromFile(v.tk, copy_dest, true)
path_des_last = copy_des .. v.name -- save it as the last path
end
end
end
end
end
reaper.UpdateArrange()
reaper.Main_OnCommand(40441, 0) -- rebuild peaks
reaper.Main_OnCommand(41858, 0) -- set take name from file
end
You should probably change the "copy_new" function too so that you are passing it the contents that have been read (copy_contents) and it is using the same IO methods. Let me know if you need any help!
Btw, one tiny piece of advice. If you see the code has multiple levels of indentation it's probably time to create some sensibly named functions so that it's nicer to read the code and it documents itself.
edit: I tested copying a few files with Lua and the "b" (for binary) flag needed to be set for reading and writing.
Last edited by snooks; 10-01-2019 at 04:28 AM.
Reason: reads files in binary mode, previous didn't work on a bunch of files
|
|
|
10-02-2019, 06:47 AM
|
#5
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
One thing I would add in general to comparing files.
I think, Reaper doesn't actually compare the files, but rather the modification-date and time.
So if the modificationtime changes, Reaper knows that the sourcefile has been changed somehow and readds the stuff again in the MediaBay.
So you should rather check, whether the last modification changes since last time checking the files.
Maybe by putting the last timestamp into a ProjExtState.
And if you find a later modification-date in the files than the one your stored into the ProjExtState, you can do the copying.
This speeds things up by magnitudes and Reaper does this in many occasions.
Comparing files, especially if there are many/big ones is always very very slow.
|
|
|
10-02-2019, 07:20 AM
|
#6
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
dsyrock script was working for me but not snooks the script saved but did not do anything.
I tried this on Win but are these going to work on Mac ?
Last edited by MusoBob; 10-02-2019 at 07:37 AM.
|
|
|
10-02-2019, 07:44 AM
|
#7
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
I think it doesn't, as it's made for Windows specifically.
Though I think, it should be alterable for Mac and Linux-compatibility.
As far as I could see, one would need to alter the folder-separator for that to work.
|
|
|
10-02-2019, 08:33 AM
|
#8
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
The bit I posted was just one function so doesn't do anything by itself. If it's put where it should be in the original script it will (should) work on Windows only still. As well as what mespotine says, there's another function that uses a Windows shell command.
|
|
|
10-02-2019, 08:57 AM
|
#9
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by snooks
It might be much faster using Lua IO calls instead of using OS calls.
|
Wow that's a clever way to copy files, I didn't know it before. Thanks
Last edited by dsyrock; 10-02-2019 at 09:08 AM.
|
|
|
10-02-2019, 09:06 AM
|
#10
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by mespotine
One thing I would add in general to comparing files.
I think, Reaper doesn't actually compare the files, but rather the modification-date and time.
So if the modificationtime changes, Reaper knows that the sourcefile has been changed somehow and readds the stuff again in the MediaBay.
So you should rather check, whether the last modification changes since last time checking the files.
Maybe by putting the last timestamp into a ProjExtState.
And if you find a later modification-date in the files than the one your stored into the ProjExtState, you can do the copying.
This speeds things up by magnitudes and Reaper does this in many occasions.
Comparing files, especially if there are many/big ones is always very very slow.
|
Yeah I think comparing modification-date is much faster then comparing file content. But can Lua get a file's modification-date directly?
|
|
|
10-02-2019, 09:10 AM
|
#11
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by mespotine
I think it doesn't, as it's made for Windows specifically.
Though I think, it should be alterable for Mac and Linux-compatibility.
As far as I could see, one would need to alter the folder-separator for that to work.
|
And the copy command I used that only work in Windows. Maybe using snoobs'way could avoid this problem
|
|
|
10-02-2019, 09:14 AM
|
#12
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
Quote:
Originally Posted by dsyrock
Yeah I think comparing modification-date is much faster then comparing file content. But can Lua get a file's modification-date directly?
|
Unfortunately not. I requested a function for SWS that would do that but with no luck til now.
So my best guess would be using dir or ls recursively and parse it's output with pattern matching.
|
|
|
10-02-2019, 01:57 PM
|
#13
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
I will see how you guys go with the script here if not I will have a look at them and try and the put the reaper.GetOS() for the backslash/forward slash and shell/terminal.
|
|
|
10-02-2019, 02:02 PM
|
#14
|
Human being with feelings
Join Date: Sep 2008
Location: Calgary, AB, Canada
Posts: 6,551
|
All three OSes should be fine with / as a separator; it's the commands and how they take arguments that will be different.
|
|
|
10-02-2019, 11:32 PM
|
#15
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by snooks
using Lua IO calls instead of using OS calls
|
I did a quick test
Code:
file1="k:\\test1.mp4"
time=os.clock()
reaper.ExecProcess("cmd /c copy k:\\test1.mp4 k:\\test3.mp4", 0)
msg("using copy command: "..os.clock()-time)
time=os.clock()
local file=io.open(file1, "rb")
local content = file:read("*a")
file:close()
local file = io.open("K:\\test4.mp4", "wb")
file:write(content)
file:close()
msg("using file write: "..os.clock()-time)
And the result is
Code:
using copy command: 0.067999999999302
using file write: 0.37300000002142
Seems like using copy command is faster than usintg Lua IO calls.
|
|
|
10-03-2019, 01:58 AM
|
#16
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
The original script you made was fast enough for me.
It does have to re-create the reapeaks, could you copy them also would that speed it up ?
If you copy the reapeaks to the folder first it won't need to create them when the source wav is changed.
Last edited by MusoBob; 10-03-2019 at 02:04 AM.
|
|
|
10-03-2019, 02:31 AM
|
#17
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by MusoBob
The original script you made was fast enough for me.
It does have to re-create the reapeaks, could you copy them also would that speed it up ?
If you copy the reapeaks to the folder first it won't need to create them when the source wav is changed.
|
I'm not sure if it's possible to copy only the specialized peaks files.
|
|
|
10-03-2019, 03:15 AM
|
#18
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
I haven't really look into it but could you just run a function first to copy sr_file..".reapeaks" then the next function for just copy sr_file
if the sr_file contains mp3 then copy sr_file..".reapindex" also (if it exists as mp3 needs to be over a certain length to create .reapindex also).
I will look at it more in the morning.
|
|
|
10-03-2019, 06:23 AM
|
#19
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
Quote:
Originally Posted by dsyrock
I did a quick test
Code:
file1="k:\\test1.mp4"
time=os.clock()
reaper.ExecProcess("cmd /c copy k:\\test1.mp4 k:\\test3.mp4", 0)
msg("using copy command: "..os.clock()-time)
time=os.clock()
local file=io.open(file1, "rb")
local content = file:read("*a")
file:close()
local file = io.open("K:\\test4.mp4", "wb")
file:write(content)
file:close()
msg("using file write: "..os.clock()-time)
And the result is
Code:
using copy command: 0.067999999999302
using file write: 0.37300000002142
Seems like using copy command is faster than usintg Lua IO calls.
|
It depends on the size of the file, for small files it's much faster.
I've attached a dll that wraps some Windows file commands with Lua. Here's code for the benchmark I did that shows how to use the dll too:
Code:
local function print(...)
local args = { n = select("#", ...), ... }
for i = 1, args.n do
reaper.ShowConsoleMsg(tostring(args[i]) .. "\t : ")
end
reaper.ShowConsoleMsg("\n")
end
local path = "C:/Users/paul-/AppData/Roaming/REAPER/UserPlugins/fileops.dll"
local copyFile = package.loadlib(path, "copyFile")
assert(copyFile, "\nError: failed to load function from dll")
-- also compareFileTimesFromPaths and getLastModified functions in there
local function sleep(a)
local sec = tonumber(os.clock() + a);
while (os.clock() < sec) do
end
end
local function copy_file_lua_io(path_a, path_b, msg)
local start_time = os.clock()
local src = io.open(path_a, "rb")
local dst = io.open(path_b, "wb")
if not src then print("Error: could not open file.") return end
local contents = src:read("*a")
dst:write(contents)
src:close()
dst:close()
local end_time = os.clock()
print("Lua API " .. msg .. " Time: ", end_time - start_time)
end
local function copy_file_win_api(path_a, path_b, msg)
local start_time = os.clock()
local ok, res = copyFile(path_a, path_b)
local end_time = os.clock()
print("Win API " .. msg .. " Time: ", end_time - start_time)
end
local function copy_file_execprocess(path_a, path_b, msg)
local start_time = os.clock()
reaper.ExecProcess("cmd /c copy " .. path_a .. " " .. path_b, 0)
local end_time = os.clock()
print("ExecProcess " .. msg .. " Time: ", end_time - start_time)
end
do
local src_path = "c:/dev/lua/wav-file.wav"
local dst_path = "c:/dev/lua/wav-file-copy.wav"
local msg = "0.6MB"
copy_file_lua_io(src_path, dst_path, msg)
copy_file_win_api(src_path, dst_path, msg)
copy_file_execprocess(src_path, dst_path, msg)
end
do
local src_path = "c:/dev/lua/video.mp4"
local dst_path = "c:/dev/lua/video-copy.mp4"
local msg = "76MB"
copy_file_lua_io(src_path, dst_path, msg)
copy_file_win_api(src_path, dst_path, msg)
copy_file_execprocess(src_path, dst_path, msg)
end
Here's my results for a couple of different sized files:
Code:
Lua API 0.6MB Time: : 0.012000000000626
Win API 0.6MB Time: : 0.0020000000004075
ExecProcess 0.6MB Time: : 0.096000000000458
Lua API 76MB Time: : 0.29300000000057
Win API 76MB Time: : 0.0649999999996
ExecProcess 76MB Time: : 0.078000000000429
|
|
|
10-03-2019, 07:31 AM
|
#20
|
Human being with feelings
Join Date: Apr 2016
Location: ASU`ogacihC
Posts: 3,921
|
Quote:
Originally Posted by snooks
I've attached a dll that wraps some Windows file commands with Lua.
|
Nice idea! How do I use getlastmodified? I tried like this but only returned the year and a zero in the format, "2019.0".
Code:
local path = "M:/Reaper Portable/UserPlugins/fileops.dll"
local LastModified = package.loadlib(path, "getLastModified")
assert(LastModified, "\nError: failed to load function from dll")
local function GetLastModified(file)
local ok, res = LastModified(file)
return res
end
fn = 'M:/Metal Sux 2.rpp'
date = GetLastModified(fn)
msg(date) -- returns '2019.0' ???
Win10 (1903), REAPER v5.983/x64
|
|
|
10-03-2019, 08:39 AM
|
#21
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
It's a bit stupid at the moment, it returns 8 arguments:
Code:
local ok, year_or_error_msg, month, day, hour, minute, second, millisecond =
getLastModified()
I should really keep the raw numbers from...
https://docs.microsoft.com/en-us/win...pi-getfiletime
... and convert for easy comparison.
The current numbers follow this format:
https://docs.microsoft.com/en-us/win...ase-systemtime
|
|
|
10-03-2019, 09:30 AM
|
#22
|
Human being with feelings
Join Date: Apr 2016
Location: ASU`ogacihC
Posts: 3,921
|
Quote:
Originally Posted by snooks
It's a bit stupid at the moment, it returns 8 arguments:
|
OK cool, I don't know how to compare dates in Lua, but I see you have a compare function, but I must be using that wrong too , it just returns 0.0.
Code:
function CompareModified(fileA, fileB)
local ok, res = compareLastModified(fileA, fileB )
msg(ok) -- returns 'true'
msg(res) -- returns '0.0' ???
end
|
|
|
10-03-2019, 10:14 AM
|
#24
|
Human being with feelings
Join Date: Apr 2016
Location: ASU`ogacihC
Posts: 3,921
|
Quote:
Originally Posted by snooks
|
Oh OK it works, but oddly if I copy files to a new folder and compare them it returns zero. I just assumed the function was comparing the files using LastWriteTime.
|
|
|
10-03-2019, 10:21 AM
|
#25
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
Yeah, so did I.
I'll get a handier function in there l8trs. I created a thread to document the dll since I hadn't here, I'll update there when done.
|
|
|
10-03-2019, 11:06 AM
|
#26
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by snooks
I've attached a dll that wraps some Windows file commands with Lua.
|
Wow that's an incredible things! Is that dll thing you make it? Are there any more dll files like this?
|
|
|
10-03-2019, 12:00 PM
|
#27
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
@snooks
A question about fileops.dll. When a path or filename contains non-English words, copyFile(pathA, pathB) doesn't work. It returns a error message: Error: first arg path does not exist. It seems that it doesn't support UTF-8 encoding?
Or is there any document about it?
|
|
|
10-03-2019, 12:01 PM
|
#28
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
Quote:
Originally Posted by dsyrock
Wow that's an incredible things! Is that dll thing you make it? Are there any more dll files like this?
|
Yes, I made it and it's the first I've seen. It's another good tool to have in the box!
Quote:
It seems that it doesn't support UTF-8 encoding?
|
Ooh, I'll look into that. Thanks for letting me know!
一会见
|
|
|
10-03-2019, 01:23 PM
|
#29
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
These are only short files 32 bars.
I tied it without copying the reapeaks first
Full Screen
then with copying the reapeaks first
Full Screen
|
|
|
10-03-2019, 03:16 PM
|
#30
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
Here's 5 x 128 bars
Full Screen
then with copying the reapeaks first
Full Screen
|
|
|
10-03-2019, 07:11 PM
|
#31
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by Lokasenna
All three OSes should be fine with / as a separator; it's the commands and how they take arguments that will be different.
|
But in Windows, the last separator of a LOCAL FILE PATH should not be a "/"
Code:
copy C:\Project\Media\Test.wav C:\Project\Media\Test1.wav
This work.
Code:
copy C:\Project\Media/Test.wav C:\Project\Media/Test1.wav
This will give you an error message.
Last edited by dsyrock; 10-03-2019 at 08:47 PM.
|
|
|
10-03-2019, 08:19 PM
|
#32
|
Human being with feelings
Join Date: Sep 2008
Location: Calgary, AB, Canada
Posts: 6,551
|
Huh.
I had read ages ago that forward slashes were fine too, tested out a bunch of commands and they all worked, but missed that one. Just did some reading and it sounds like the Windows devs don't know how to be consistent. :/
Thanks!
|
|
|
10-03-2019, 08:53 PM
|
#33
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by MusoBob
then with copying the reapeaks first
|
Yeah it is a good way. But in my case, I store all the peak files in a specific folder, instead if the project media folder. So I think copy the peak file is not a common solution that fits everyone.
|
|
|
10-03-2019, 10:29 PM
|
#34
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
If it's in the REAPER.ini
then copy to
Quote:
altpeakspath=C:\Users\User\Documents\REAPER Media\REAPEAKS
altpeaksopathlist=
|
else
copy to project folder
|
|
|
10-04-2019, 02:24 AM
|
#35
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
If I don't rebuild peaks, it will look like this
|
|
|
10-04-2019, 03:27 AM
|
#36
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
I'm not getting that:
|
|
|
10-04-2019, 05:17 AM
|
#37
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Quote:
Originally Posted by MusoBob
I'm not getting that:
|
Well it's weird...
|
|
|
10-04-2019, 12:50 PM
|
#38
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
If the peaks are already there it adds -import on the copied wav
so to get it to work when I tried it I just changed
Code:
local name_new=name:sub(1, edge-1)..""..tale --.."-import"..tale
to get it to keep the original file name.
|
|
|
10-07-2019, 10:04 AM
|
#39
|
Human being with feelings
Join Date: Sep 2018
Location: China
Posts: 565
|
Already updated a new version at 1#
|
|
|
10-08-2019, 03:35 AM
|
#40
|
Human being with feelings
Join Date: Sep 2014
Posts: 2,643
|
Thanks !
|
|
|
Thread Tools |
|
Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -7. The time now is 12:36 AM.
|