ReaPack: Package Manager for REAPER (developer thread)

[X-Raym]

what make my script header easy to parse is the * at the start of the line : it is a nice way to not have error by parsing variables or functions names.
Putting this script header in all my scripts was how I prepare the future script downloader during this last months
(And it is alsways useful to have meta infos, even for users in order to see of tbeh have the required reaper version, or to see the screenshot, and the fprum thread, the author link etc)

[cfillion]

Status Update!

The extension now downloads the latest version of every packages it knows about to REAPER's resource folder:



TODO:

- Generate the package database from the git repository using the metadata parser
- Download and use the real database instead of the dummy one I used for testing
- Keep track of installed versions
- Download only the updated scripts
- A nice hideable dialog to show the download progress?

[X-Raym]

Very promising cfillion, good work !

[mpl]

Very nice!
Hope I'll find some time to add/update headers this weekend and continue edit headers later to make potential database works with versions, release dates, Reaper/sws requirements, etc

[semiquaver]

very cool indeed ! - thanks in advance to all involved

[cfillion]

Originally Posted by mpl

Hope I'll find some time to add/update headers this weekend and continue edit headers later to make potential database works with versions, release dates, Reaper/sws requirements, etc

I wrote a metadata parser and checker the other day (post 39) to help with this. The checker lists all the scripts with missing or bad metadata and provide a description of what is wrong so it's easier to fix the headers.

Example 1 (broken script): https://github.com/cfillion/test_rea...etadata/pull/2 ("Details" -> the script's error report is at the bottom)
Example 2 (fixed script): https://github.com/cfillion/test_rea...etadata/pull/1

May I set this setup in a pull request? To make this work, travis-ci.org will need access to the commit statuses on the repository so it can report back the result of the test script.

[mpl]

Sure you can, for example into /Test folder (create it). I give members "Write access", don't know is that means they can commit directly to master without approval.

Question to you: is it possible to collect my repo, X-Raym repo and common repo by downloaded/updater, so we (and maybe whoever in future) could develop our repos independently without commits/cloning to "common" repo (but with same header of course to make it compatible). That would be excellent solution at least for me.

[cfillion]

Originally Posted by mpl

Sure you can, for example into /Test folder (create it). I give members "Write access", don't know is that means they can commit directly to master without approval.

Authorization request for Travis sent, working on the code now.
EDIT: Pushed on the "metadata" branch. Will make the pull request after setting up travis (it should report that every script has invalid version metadata ).

Originally Posted by mpl

Question to you: is it possible to collect my repo, X-Raym repo and common repo by downloaded/updater, so we (and maybe whoever in future) could develop our repos independently without commits/cloning to "common" repo (but with same header of course to make it compatible). That would be excellent solution at least for me.

I plan to let the user add repositories to the extension so the independent developers can maintain their own and the user can "subscribe" to them by adding an URL to a list or something like that.

The manager already put the scripts it installs in a dedicated directory with the repository name (currently "ReaScripts" is hard-coded) so, in the future when this feature is done, if a script is duplicated in different repositories it won't conflict (also it's nicer).

[X-Raym]

I plan to let the user add repositories to the extension so the independent developers can maintain their own

That would be really nice !!
This project is evolving nicely.

[mpl]

OK, no problem! I'll leave my scripts only in my repo to not duplicate them. Thanks!


Just you to know I dont speak English well, so to be clear, should I merge metadata to master or you have done already everything by yourself? (Didn't get completely how member permissions work)

[cfillion]

Originally Posted by mpl

Just you to know I dont speak English well, so to be clear, should I merge metadata to master or you have done already everything by yourself? (Didn't get completely how member permissions work)

English is not my main language either (I speak French).

I'm using that branch as a work-in-progress for tuning the metadata tag syntax and adapting the script's headers. It will probably take a while before it's ready to be merged.

Currently the syntax I'm using is like this (similar to greasemonkey and doxygen):
https://github.com/cfillion/metaheader#readme

.lua file:

-- @version 1.0pre3

reaper.DoSomething();

.eel file:

/*
 * @version 1.0pre3
 */

DoSomething();

.eel file (alternative):

// @version 1.0pre3

DoSomething();

It's quite easy to parse and tell apart from normal comments not containing data, but I realise now that it's quite different from what everyone is used to...

[mpl]

Hmm, so X-Raym style header is ok for parsing (looks like first eel example)?

--[[ 
* value1_description: value
* value2_description: value
* value3_description: value
* value4_description: value
--]]

We can point a strongly required condition for this if needs for adding new scripts, I don`t see any problem here.

[mpl]

OK guys, did this:

to prevent duplicating scripts and for to make common repo place for scripts, which scattered all over the stash, forums etc. -

1) removed my stuff from repo

to make repos compatible however we will decide to download / sort scripts -

2) named folders in common ReaTeam repo like in X-Raym repo (except MIDI Editor - there are also not only MIDI Editor related scripts),
3) named folders in my repo like in X-Raym repo, add Track Editing folder (X-Raym, just let you know about name if you`ll decide sort your track related scripts also)

4) moved my test folder to another test repo, so now my pack have only working stuff and not for testing
5) add links in readme to my repo and X-Raym repo as big parts, I think extension should check for now 3 places - X-Raym repo, my repo and common repo (and maybe any other repo in future if it will be compatible by header and folders structure)
6) delete 1 Heda script and links to HeDa website. HeDa, can you update and sort you github stuff like X-Raym repo named folders and add header (at least empty) so we will can use your stuff also? If you`ll do so, I`ll add link to your github, so you`ll can keep scripts updated without pushing commits to common repo. OR can I upload your scripts to the repo (I`ll add header, will sort as needed etc so you will not need to do something)?

[X-Raym]

@mpl
Nice thank you MIDI Editor scripts are the scripts that needs to be loaded in the MIDI editor section of the Acton list. That's why it is not simply MIDI (other MIDI related scripts can works in Arrange View).

Also, consider using Plural in folder names.


Some other GitHub repo:

• nofishonfriday/ReaScripts
• przemoc/REAPER-ReaScripts
• Lazzle/ReaMIDI

[mpl]

In my repo and in common:

Renamed Track(s) Properties, sorted take scripts, add Takes Envelope (I guess it will grow up, so it needs to be separated from Takes Properties), Takes Properties, Take Editing, separate MIDI/MIDI Editor (I know russian guy who did some scripts to edit MIDI outside MIDI Editor).

Thinking about small repos... It will be hard to ask every guy to do different things (header and proper folder names, for example). So here I think it will be better ask guys to duplicate their scripts to common repo and let them member privileges to edit their stuff (so they shouldn`t worry about following different rules and we can manually fix something).

Edit
About /Functions and other imported stuff. Should it be in pack or distributed differently? What I got thinking about that:
- all scripts with imported functions should be merged OR placed in separated folders with imported functions;
- all scripts with imported graphics should be placed in separated folders;
- when installing/updating should be check which says "Install developer (/Function and /Templates) scripts and include into Action List", so there will be option for newcoming developers, who wants not only to use but to learn scripting also, this check should be unchecked by default to prevent user questions like "What spk77_class.lua do?"

[nofish]

First, let me say, mpl great initiative, bravo and thank you.

Originally Posted by mpl

Thinking about small repos... It will be hard to ask every guy to do different things (header and proper folder names, for example). So here I think it will be better ask guys to duplicate their scripts to common repo and let them member privileges to edit their stuff (so they shouldn`t worry about following different rules and we can manually fix something).

Since I'm mentioned in X-Raym's post above and 'small repos' also applies to me somehow, I'd be happy to contribute my scripts to this central repo if the 'ReaTeam' thinks they would fit.

And I'm totally fine with the above suggestion.

I'd need a 'step by step' though how to duplicate to the central repo, as I'm not too familiar with GitHub yet, I've just started using it.

[spk77]

Thank you mpl and cfillion

I think I'll start to move my scripts to the github (spk1977/ReaScripts).

A "step by step instructions for GitHub" are certainly needed.

[planetnine]

Originally Posted by spk77

...

A "step by step instructions for GitHub" are certainly needed.

Yes please!

Plus any links to GitHub intro/familiarity/jargon articles/videos etc please.



>

[mpl]

noname, thanks.
I invite you to ReaTeam and added your scripts, except:

- nofish_Write tracknames to textfile - I would add this if there was native function to ask user for file destination (Save as window), showing in console is enough I guess since it can be easy copied from console and manually pasted to any text editor.
- nofish_Realtime clock - bug with folder name (first run I got // in the end of destination folder)
- nofish_Turn toolbar btn on when take pooled - 'BR_GetMidiTakePoolGUID' argument 1: expected MediaItem_Take* on first run, doesn`t work

I named nofish_Dynamic mixer (for use with toolbar button) so because user can be frustrated and don`t know how to stop this behaviour.

Looks I should add simple tutorial for using GitHub and its app, but I didn`t made any tutorial in my life))
I also don`t know too much about GitHub, here how I edit repo (I can be wrong but I guess it is only because I`m owner and members only can commit and wait for review and merging with master repo):


Dows anybody know how members privileges works? For now I set up write access for member. Does that mean they can edit master themself without my review?

spk77, yes, you have much scripts, but note, we will should add header to every script to make parsing metadata possible (can be useful for filter, for example to not install scripts, which unsupported by current REAPER/SWS version).

[spk77]

Originally Posted by mpl

Dows anybody know how members privileges works? For now I set up write access for member. Does that mean they can edit master themself without my review?

It seems to work like that - I edited the "ReaScripts/Various/spk77_Clock.eel"

Originally Posted by mpl

spk77, yes, you have much scripts, but note, we will should add header to every script to make parsing metadata possible (can be useful for filter, for example to not install scripts, which unsupported by current REAPER/SWS version).

I can wait until the header format is decided

[mpl]

I think if eel with * is OK for cfillion then same lua format is ok also.

One more impotant thing I should say. Because of it is time we build something big, it needs to setup rules somehow for members/outside collaborators. Currenly I suggest at least this rules:

• ScripterName_script name.lua format for names
• self explainable name/title (except concept things, Quantize Tool or Dynamic Mixer for example)
• Folders names (check now 3 repos - mine, X-Raym and common ReaTeam repo)
• Header - check X-Raym repo > Templates
• scripts with imported functions - merge it (for example /Various/spk77_Track IO.lua). updater will be hard to know which script is master and which is slave (OR maybe some parseable info file in directory with main and imported scripts should say to updater about that?)
• same with imported graphics
• may be some file which contain all info about scripter (donation, links) and this should be used by updater/manager extension(?)
• defered scripts should be able to stop somehow (by dedicated action, for example) to prevent user frustration
• scripts should have as less bugs as possible of course and tested before adding/updating in repo
• add customizable scripts to /Templates(?)

[X-Raym]

Here it for Lua

--[[
 * ReaScript Name: Go to start of previous region
 * Description: Move edit or play cursor to previous region. Move view and seek play.
 * Instructions: Place edit cursor inside a region. Run.
 * Screenshot: http://
 * Author: X-Raym
 * Author URl: http://extremraym.com
 * Repository: GitHub > X-Raym > EEL Scripts for Cockos REAPER
 * Repository URl: https://github.com/X-Raym/REAPER-EEL-Scripts
 * File URl: https://github.com/X-Raym/REAPER-EEL-Scripts/scriptName.eel
 * Licence: GPL v3
 * Forum Thread: Scripts: Transport (various)
 * Forum Thread URl: http://forum.cockos.com/showthread.php?p=1601342
 * REAPER: 5.0
 * Extensions: None
--]]
 
--[[
 * Changelog:
 * v1.0 (2015-11-27)
    + Initial Release
--]]

for EEL, just replace --[[ by /**

Screenshot field is facultative, just like File direct URL. I guess we don't need it anymore, and it is a bit too painful to update everytime ^^

We could also imagine other faculative field like a Donation field.

[X-Raym]

@mpl

scripts with imported functions - merge it

I think it is better to have the scripts (main and external classes) contain in one sub subdfolder, like a little plugin.
for Track IO subfolder into Tracks Properties

Some huge scripts are easier to maintain in several files than a several thousand lines big one.
Also, best place to have all external media required (image etc).

[mpl]

Then it should be also file which says to extension "This folder has script with main and external classes. Please add to action list only Main.lua as main script and don`t add external classes" something like this

class_def.txt :

master=main.lua
external=class.lua,sliderclass.lua,menuclass.lua
graphics=slider.png

[X-Raym]

@mpl
or we can prefix.suffix classes function by a certain character or a certain word. These would be excluded from importing without even opening it.

What about all scripts that ends with _class ?

exemple

author_My super finctions library_class.lua
spk77_Slider functions_class.lua

Should be enough

[mpl]

You`re right, it`s better! But than means spk77 and you will should rewrite some code for this (I don`t know, who else used importing?).

[heda]

How to include external files that are also needed for the script to work? In my regions from items script for example, to install it you need also to put a custom cursor file inside the REAPER's cursors folder. Which is not in the same folder where the lua script would go.
I've been quite busy and not able to do scripting lately as you may have noticed :/ I hope to find time to put all in order in github once everything is defined. I'm glad to see the X-Raym headers finally becoming a standard

[planetnine]

Originally Posted by X-Raym

@mpl

I think it is better to have the scripts (main and external classes) contain in one sub subdfolder, like a little plugin.
for Track IO subfolder into Tracks Properties

Some huge scripts are easier to maintain in several files than a several thousand lines big one.
Also, best place to have all external media required (image etc).

This makes sense to me.

I did try to create a subfolder in the "Stretch Markers" folder in ReaTeam so I could add my stretch marker scripts/actions, but I couldn't find a way to do it. I oresumed I didn't have the right permissions.




>

[mpl]

planetnine,



heda, I think here could be two ways:
- just putting all needed files in folder and add file with instruction why all these files placed here
- add "/Additional Stuff/Cursors" folder to keep there clearly described all scripts related stuff (like HeDa_Regions&Markers from items - pointinghand.cur maybe)

But I found first way more logical and cleaner.

[mpl]

Very qiuck and rougly recorded tutorial how to use GitHub app:

[X-Raym]

One note about git:

Be careful, don't move files on your computer, you will lose the history linked to these files, as it will be considered as deleting the source file, and creating a new file (in the destination folder), rather than "moving".

You can use the Online repo to do such thing, or the console,
I can't tell you how, I have to check every time I want to do it ^^

[mpl]

^^Agree.
I sorted everything in my and common repos hardly yesterday since I didn`t care about history. But yes, to keep history now we need to do moving online or something like

mkdir lib
git mv hello.html lib
git status

in console, if I`m not wrong (maybe "move" for Windows instead OSX-like "mv")

[spk77]

Maybe we should make "test" and "examples" -folders. We could share our code snippets there and also ask for help.
Or how should this be done?

[mpl]

As I said above:

- when installing/updating should be check which says "Install developer (/Functions and /Templates) scripts and include into Action List", so there will be option for newcoming developers, who wants not only to use but to learn scripting also, this check should be unchecked by default to prevent user questions like "What spk77_class.lua do?"

we could add these folders, but somehow exclude it from regular install. And for testing we could add another dedicated repo (not folder, like ReaTeam/ReaScripts-Test/spk77/your temporary test scripts), where scripts sorted by username. So we will have one repo for sharing (for user download) and repo for developing. Today I did this for myself: https://github.com/MichaelPilyavskiy?tab=repositories

[spk77]

Originally Posted by mpl

As I said above:
we could add these folders, but somehow exclude it from regular install. And for testing we could add another dedicated repo (not folder, like ReaTeam/ReaScripts-Test/spk77/your temporary test scripts), where scripts sorted by username.

Yes, a testing repo would be good.

[mpl]

Done https://github.com/ReaTeam/ReaScript-Test
So here you or another member can share whatever he needs inside user folders (here sorting by category at high level not needed, I think)

[spk77]

Thanks
Added "spk77/EEL" folder.

This might be a good repo for practicing how to actually use the GitHub

[nofish]

Originally Posted by mpl

I invite you to ReaTeam and added your scripts, except:

- nofish_Write tracknames to textfile - I would add this if there was native function to ask user for file destination (Save as window), showing in console is enough I guess since it can be easy copied from console and manually pasted to any text editor.
- nofish_Realtime clock - bug with folder name (first run I got // in the end of destination folder)
- nofish_Turn toolbar btn on when take pooled - 'BR_GetMidiTakePoolGUID' argument 1: expected MediaItem_Take* on first run, doesn`t work

I named nofish_Dynamic mixer (for use with toolbar button) so because user can be frustrated and don`t know how to stop this behaviour.

Thanks, I'll try to fix theese issues.

[Sexan]

.........

[cfillion]

I got the database generator tool working yesterday. It remembers the last commit it checked so it doesn't rescan the same files over and over (thanks to git).
For now it only supports @author, @version and @changelog metadata (I'll start adding them to the scripts in the metadata branch). More can be added later when the extension gains new features.

With this the full distribution process works like this:

git repository -> metadata header at the top of the scripts -> database generator -> index.xml -> C++ extension on REAPER -> reascripts installed and updated on the user's computer

------

Originally Posted by mpl

I think if eel with * is OK for cfillion then same lua format is ok also.

Yes it's ok. Also unsupported metadata does not need to be changed (until it's supported) or removed. This is a valid header for a script (from X-Raym's example):

--[[
 * @author X-Raym
 * @version 1.0
 * @changelog (2015-11-27)
 *   + Initial Release
 *
 * ReaScript Name: Go to start of previous region
 * Description: Move edit or play cursor to previous region. Move view and seek play.
 * Instructions: Place edit cursor inside a region. Run.
 * Author URl: http://extremraym.com
 * Repository: GitHub > X-Raym > EEL Scripts for Cockos REAPER
 * Repository URl: https://github.com/X-Raym/REAPER-EEL-Scripts
 * File URl: https://github.com/X-Raym/REAPER-EEL-Scripts/scriptName.eel
 * Licence: GPL v3
 * Forum Thread: Scripts: Transport (various)
 * Forum Thread URl: http://forum.cockos.com/showthread.php?p=1601342
 * REAPER: 5.0
 * Extensions: None
--]]

In fact it will parse as long there is a @key (and value, if needed) somewhere in the line. Also, parsing stops at the first blank line found:

-- (leading text is ignored) @key value <-- key and value recognized as metadata

-- @key value <-- ignored because it's after the first blank line

------

Originally Posted by mpl

scripts with imported functions - merge it (for example /Various/spk77_Track IO.lua). updater will be hard to know which script is master and which is slave (OR maybe some parseable info file in directory with main and imported scripts should say to updater about that?)

As long as the manager installs every script in a given repository then this is not an issue right? If it's decided to let the user choose which scripts are installed then a dependency system can solve this:

@depends Library/Console Messages.lua

Originally Posted by mpl

[*]same with imported graphics

Mmh... Non-script files (not .lua or .eel) are currently ignored. I can quickly add support for more extensions (png, jpeg etc) but I don't think this is the right way: what if a specific script requires a special type of file to work?

What about something like this:

--[[
 * @depends Resources/Folder With The Script Name/backgroung.jpg
 * @depends Resources/Folder With The Script Name/fader.png
 * @depends Resources/Folder With The Script Name/names.txt
--]]

These extra files would be included in the distribution as long as at least one script @depends on them.

------

mpl, can you accept the authorization request for Travis-CI on ReaTeam I sent the other day? I think it would be great to have the pull requests tested early before they are merged.