Old 09-19-2016, 04:39 PM   #1
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default Q: Parse CSV file with JSFX

Hi,

I am a bit confused that string format and function in JSFX.

imagine a file like
Code:
45-50
40-51
68-98
The seperator - can be replace with any other separator.


Here is the code for sampling a file (from MIDI Pattern/Scale Variation Generator [IXix])
Code:
@slider
seqFile != slider8 ?
(
  seqFile = slider8;
  seqSize = 0;
  fileHandle = file_open(slider8);
  fileHandle > 0 && file_text(fileHandle) ?
  (
    while
    (
      file_var(fileHandle,seq[seqSize]);
      file_avail(fileHandle) ? seqSize += 1;
    );
    file_close(fileHandle);
  );
);
You can then access a line with seq[x] where x is the number of the line.

But how will you return two variables from that line ? How would you parse the "45-50" into two int variable ?
I only succeed to get int when there is only one number per line, but not two with a sperator.

Thanks for your help !

Last edited by X-Raym; 09-19-2016 at 07:30 PM.
X-Raym is offline   Reply With Quote
Old 09-19-2016, 07:01 PM   #2
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

The internal parsing that IXIX's code is doing is tied specifically to sliders and ignores non numbers, and con only read a single variable.

however, you could do

Code:
45.50
40.51
68.98
and manipulate it mathematically to parse the variables.

so you'd have to use a text editor to convert " - " into "."


You can use file_string(handle,str) to get the text, however, this only reads the first line - it should be able to do newlines, might be some encoding issue, might be a bug. will AskJF.

*asked justin and the great oracle answered see post #11

Last edited by James HE; 09-20-2016 at 05:22 PM.
James HE is offline   Reply With Quote
Old 09-19-2016, 07:44 PM   #3
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

@JamesHE
thank you james for the precision, it helps,
I missed the file_string function, is the second parameter supposed to be the line number in "read" mod ?

But yes indeed, a solution without match/string function can work for my particular case.

The number tricks is a good idea, not very difficult to implement (just math floor for the first value)

I will change the - to . inside the script which generates this CSV.

Stay tuned for the script, should came out tomorrow :P

Last edited by X-Raym; 09-20-2016 at 04:03 AM.
X-Raym is offline   Reply With Quote
Old 09-19-2016, 10:40 PM   #4
planetnine
Human being with feelings
 
planetnine's Avatar
 
Join Date: Oct 2007
Location: Lincoln, UK
Posts: 7,585
Default

What about a comma?

Isn't that what the "C" is in CSV? (comma separated values)




>
planetnine is offline   Reply With Quote
Old 09-20-2016, 01:30 AM   #5
DarkStar
Human being with feelings
 
DarkStar's Avatar
 
Join Date: May 2006
Location: Surrey, UK
Posts: 18,090
Default

Wouldn't the match() function be the one to use?

I would post the answer but have never really understood how to define the matching pattern
__________________
DarkStar ... interesting, if true. . . . Inspired by ...
DarkStar is offline   Reply With Quote
Old 09-20-2016, 03:01 AM   #6
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

@planetnine
Actuallly, it is for "Character", it is very often a space or a a tab :P

But the problem is more about variable value than line formatting, I can change it without problem.

@DarkStar
spk77 and gofer already made a example of match function used in eel,
ReaScripts-Templates/gofer-spk77-Functions - User inputs fields.eel at master · ReaTeam/ReaScripts-Templates
but for some reason it didn't work with the way the file is parsed using the code I show in post 1.
JamesHE explained why: this parsing method doesn't support strings

@all
If anyone find a workable solution involving more columns (without math tricks), I'm sure it could be very useful.
Thanks for your suggestions !
X-Raym is offline   Reply With Quote
Old 09-20-2016, 04:37 AM   #7
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Hm the math trick is a bit harde rthan expected,
hard to differentiate between
vel = 1
vel = 10
vel = 100
if it is store as a decimal 0.1, 0.10, 0.100... it is all the same in math functions. :/

Can't we return string from file based on a line number ?

EDIT: oh wait, no problem in fact, as I can change output format from my MIDI to CSV script, I can make it encode velocity as 0.xxx number instead of just 0.vel :P

Last edited by X-Raym; 09-20-2016 at 04:57 AM.
X-Raym is offline   Reply With Quote
Old 09-20-2016, 09:42 AM   #8
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

Actually, file_string() can read multiple lines, you just need to call it multiple times.
James HE is offline   Reply With Quote
Old 09-20-2016, 09:46 AM   #9
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

@JamesHE
Cool !
Do you have any working code snippet about how we can loop in all lines of a file ?
X-Raym is offline   Reply With Quote
Old 09-20-2016, 04:02 PM   #10
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

Quote:
Originally Posted by X-Raym View Post
@JamesHE
Cool !
Do you have any working code snippet about how we can loop in all lines of a file ?
writing the functions now

back in a bit.
James HE is offline   Reply With Quote
Old 09-20-2016, 05:08 PM   #11
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

wow, what a hard challenge it can be to translate what the jsfx help is telling you ha ha ha. It's funny (sometimes beautiful) how simple it is once you get it.


Code:
function file_to_numbered_strings(file,offset)local(handle)
//offset= starting string number
//returns number of lines read
(
  handle=file_open(file);
  file_string(handle,offset);  
  while( strlen(offset) && offset < 1023 )
    ( offset+=1;
      file_string(handle,offset);      
    );
  file_close(file);
  offset;
);



so this breaks up each line in a txt file into numbered strings. From there, matching into variables should be fine.

Offset will generally be 0, but you never know - you might want to to chain these together for different files - just pass the returned offset to the next function call.

Note that there are 1024 slots for numbered strings, so I made this stop reading at that limit (cause I'm not sure what happens then). If you'd need to read war and Peace with JS, you could start stuffing the strings into memory - or to even serialize them, but for now, lets just keep it simple and assume 1024 lines is enough.

A different function could also parse the lines into a #NamedBigString - possibly adding newlines or some other character (or not) - and then you just get your variables from it. I would do that if I needed to serialize the string - or even if I needed to make sure some other function doesn't overwrite the numbered strings and i needed to preserve the slots (or pass it to another function later in the code - cause things get weird if you don't)


If you need anything else, let me know. I have EEL string functions that can chop up complex strings (i.e a track chunk) like a sou chef chopping an onion. (peek into my ReaLComps script files (the library that comes with it) if you want - most are in there)

Last edited by James HE; 09-20-2016 at 05:26 PM.
James HE is offline   Reply With Quote
Old 09-20-2016, 11:17 PM   #12
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

@JamesHE
Very cool james !
I put this snippet on ReaTeam template repo:
ReaScripts-Templates/JamesHE_Parse file lines into string.jsfx at master · ReaTeam/ReaScripts-Templates
I'm sure other script may find it useful.

Quote:
what a hard challenge it can be to translate what the jsfx help is telling you
Indeed, it is usually where I am stuck haha :P
Quote:
If you need anything else, let me know.
Thanks for you offer
if you are ok and this is not too much to ask (you already made a lot with your function), making a complete functional demo script it could be very helpful

Like having a the script looking for files from a slider in the a Data subfolder (or in project folder, if this is possible - I didn't find how), and a way to call/debug one particular line, and optionally to make a pattern search like if each lines is a two column CSV.
For the absolute beginner, all details can matter, like where to declare the function within the script, where to call it, how to name the file in the call (does it require to put the .txt extension or not)...

Are you in ? :P
X-Raym is offline   Reply With Quote
Old 09-21-2016, 06:49 PM   #13
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

Quote:
Originally Posted by X-Raym View Post
@JamesHE


Like having a the script looking for files from a slider in the a Data subfolder (or in project folder, if this is possible - I didn't find how),

I think to use text files with a slider, they either need to be in the exact same folder as the JS plugin (I think REAPER actually looks there first) or in the DATA directory.

I just always use the DATA directory cause I know where to find it

Here is a working JS plug example that just displays the context of the text files. The user will have to provide the files of course

Code:
desc:.txt Files examples >  by James HE

slider1:/text files:none:Text File 

@init

//FUNCTIONS for reading .txt files
//offset = starting string number
//returns number of lines read

function file_to_numbered_strings(file,offset)local(handle)
//this version can be used when you define the file via ~ filename:0,sometext.txt at the beginning of your code
//use slider version if you want to pick from text files in a folder
(
  handle=file_open(file);
  file_string(handle,offset);  
  while( strlen(offset) && offset < 1023 )
    ( offset+=1;
      file_string(handle,offset);      
    );
  file_close(file);
  offset;
);  


function slider_file_to_numbered_strings(slidernumber,offset)local(handle,lastvalue,lines)
//USE ONLY THE NUMBER OF THE SLIDER FOR "slidernumber"  - do not use "sliderx"
(

  slider(slidernumber) != lastvalue ? reload=0;
  
  !reload ? (
    handle=file_open(slider(slidernumber));
    file_string(handle,offset);  
    while( strlen(offset) && offset < 1023 )
      ( offset+=1;
        file_string(handle,offset);      
      );
    file_close(file);
    lines=offset;
    lastvalue=slider(slidernumber);
    reload=1;
  );
  lines;
); 
  

  
@slider
lines=slider_file_to_numbered_strings(1,offset);



@gfx 650 450


//some guidance
gfx_setfont(1, Ariel,14);
gfx_x=40; gfx_y=10;
gfx_r=gfx_g=gfx_b=1;
gfx_drawstr("Create a folder in your REAPER Data directory named \"text files\", then create or put some .txt files in the folder.
The text within those files will be displayed below when selected by the slider");


//display file
gfx_setfont(2, Ariel,18);
gfx_r=1;gfx_g=gfx_b=0;
xx=30;
yy=55;
gfx_x=xx;gfx_y=yy;
i=0;
loop ( lines,
  
  gfx_drawstr(i);
  gfx_x=xx;
  gfx_y+=gfx_texth+3;
  i+=1;
);

*When working with .txt files like this and something seems to disappear, the file might not have been closed properly. OFFLINE the plugin and bring it back if something goes awry.
James HE is offline   Reply With Quote
Old 09-22-2016, 01:36 AM   #14
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Excellent !
I updated the code snippet on GitHub Thanks !

As I can see, to display the line of text, we only use gfx_drawstr(i);.

So it seems that the strings are loaded in some kind of buffer where we only need to pass the line number as argument of the draw function to display them in GFX.

But how could we access a particular line without drawing it in GFX (to store it in another variable for exemle). ?
X-Raym is offline   Reply With Quote
Old 09-22-2016, 04:27 AM   #15
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

Quote:
Originally Posted by X-Raym View Post
Excellent !
I updated the code snippet on GitHub Thanks !

As I can see, to display the line of text, we only use gfx_drawstr(i);.

So it seems that the strings are loaded in some kind of buffer where we only need to pass the line number as argument of the draw function to display them in GFX.

But how could we access a particular line without drawing it in GFX (to store it in another variable for exemle). ?

each line is stored in the numbered string slots. To get a variable from it, you use match() to get what you are looking from out of that numbered string.
James HE is offline   Reply With Quote
Old 09-22-2016, 05:11 AM   #16
DarkStar
Human being with feelings
 
DarkStar's Avatar
 
Join Date: May 2006
Location: Surrey, UK
Posts: 18,090
Default

^^^
That's what I hinted at

Code:
str_11 = "45-50-53";  match("%d%d%d",     str_11, v11a, v11b, v11c); 
str_12 = "40 51 55";  match("%d %d %d",   str_12, v12a, v12b, v12c);
str_13 = "68,98,108";  match("%d,%d,%d",  str_13, v13a, v13b, v13c); 

gfx_x =20; gfx_y =20;
gfx_r=1.00; gfx_g=0.80; gfx_b =0.00; gfx_a = 0.80;
gfx_x =20; gfx_y +=15; gfx_printf("%3d  %3d  %3d", v11a, v11b, v11c);
gfx_x =20; gfx_y +=15; gfx_printf("%3d  %3d  %3d", v12a, v12b, v12c); 
gfx_x =20; gfx_y +=15; gfx_printf("%3d  %3d  %3d", v13a, v13b, v13c);
Note that the '-' character is parsed as negating the following digits, not as a separator.
__________________
DarkStar ... interesting, if true. . . . Inspired by ...
DarkStar is offline   Reply With Quote
Old 09-22-2016, 05:42 AM   #17
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

thanks both of you, I get it :P

Code:
//display file
gfx_setfont(2, Ariel,18);
gfx_r=1;gfx_g=gfx_b=0;
xx=30;
yy=55;
gfx_x=xx;gfx_y=yy;
i=0;
loop ( lines,
  match("%d-%d-%d", i, v11a, v11b, v11c); // Parse line i
  sprintf(str, "%d", v11a); // Convert to string for display in GFX
  gfx_drawstr(str); // Draw line str
  gfx_x=xx;
  gfx_y+=gfx_texth+3;
  i+=1;
  
);
each line get parsed into several variable,
and as extra feature, one of these is converted to string and displayed in GFX.
X-Raym is offline   Reply With Quote
Old 06-25-2019, 04:07 PM   #18
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Hi,


Is it possible to parse CSV file like


Code:
36 Kick
37 Snare
etc...


so that the first number become index of an array containing the string as value ?


I tried a lot of things including,



Code:
map = 100;

//Load map
mapFile != slider30 | 0 || reload == 1 ?
(
  reload = 0;
  mapSize = 0;
  mapFile = slider30;
  fileHandle = file_open(slider30);
  fileHandle > 0 && file_text(fileHandle) ?
  (
    while
    (
      file_string(fileHandle, str);
      match("%d %s", str, d, #s);
      map[d] = #s;
      file_avail(fileHandle) ? mapSize += 1;
    );
    file_close(fileHandle);
  );

);

Without success :S
X-Raym is offline   Reply With Quote
Old 06-26-2019, 01:50 AM   #19
Tale
Human being with feelings
 
Tale's Avatar
 
Join Date: Jul 2008
Location: The Netherlands
Posts: 3,007
Default

Something like this should probably work:

Code:
undef = 0;
strcpy(undef, "[Undef]");
memset(map, undef, 128);

handle = file_open(slider30);
handle >= 0 ? (
  file_text(handle) ? (
    str = undef + 1;
    while(
      file_string(handle, #line);
      file_avail(handle) ? (
        // You could probably deal with line-endings more efficiently...
        (match("%d %s\r\n", #line, note, str) ||
        match("%d %s\n", #line, note, str)) &&
        note >= 0 && note < 128 ? (
          map[note] = str;
          str += 1;
        );
        1; // while
      );
    );
  );
  file_close(handle);
);
Tale is offline   Reply With Quote
Old 06-26-2019, 04:08 AM   #20
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

@Tale
Thx for your help !


Unfortunatly this face the same problem has I had when testing: either only the first or last line works.


Here is a full exemple:


CSV:


Code:
37 banjo.png
36 cowbell.png

JSFX:
Code:
desc:Parse MIDI Note Name File

slider2:36<0,127,1>Base Note

slider30:/track_icons:none:Text File 

////////////////////////////////////////////////////////////////////////////////
@init

////////////////////////////////////////////////////////////////////////////////
@slider
undef = 0;
strcpy(undef, "[Undef]");
memset(map, undef, 128);

handle = file_open(slider30);
handle >= 0 ? (
  file_text(handle) ? (
    str = undef + 1;
    while(
      file_string(handle, #line);
      file_avail(handle) ? (
        // You could probably deal with line-endings more efficiently...
        (match("%d %s\r\n", #line, note, str) ||
        match("%d %s\n", #line, note, str)) &&
        note >= 0 && note < 128 ? (
          map[note] = str;
          str += 1;
        );
        1; // while
      );
    );
  );
  file_close(handle);
);

@gfx 100 32

gfx_setfont(1, "Arial", 20);
gfx_r=1; gfx_b=1; gfx_g=1; gfx_a=1;
gfx_x=gfx_y=5;

gfx_drawstr(map[slider2]);

In this case, 37 do return "banjo.Png" as expected, but 36 return "[Undef]"...



Any idea ?
X-Raym is offline   Reply With Quote
Old 06-26-2019, 04:10 AM   #21
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Oh I know what is wrong,


The last line isn't return for some reason, I tested with more entries. I wonder how to fix it.
X-Raym is offline   Reply With Quote
Old 06-26-2019, 04:23 AM   #22
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

This seems to work;


Code:
desc:Parse MIDI Note Name File

slider2:36<0,127,1>Base Note

slider30:/track_icons:none:Text File 

////////////////////////////////////////////////////////////////////////////////
@init

////////////////////////////////////////////////////////////////////////////////
@slider
undef = 0;
strcpy(undef, "[Undef]");
memset(map, undef, 128);

handle = file_open(slider30);
handle >= 0 ? (
  file_text(handle) ? (
    str = undef + 1;
    while(
      file_string(handle, #line);

        // You could probably deal with line-endings more efficiently...
        match("%d %s", #line, note, str) &&
        note >= 0 && note < 128 ? (
          map[note] = str;
          str += 1;
        
      );
      file_avail(handle) ;
    );
  );
  file_close(handle);
);

@gfx 100 32

gfx_setfont(1, "Arial", 20);
gfx_r=1; gfx_b=1; gfx_g=1; gfx_a=1;
gfx_x=gfx_y=5;

gfx_drawstr(map[slider2]);

Differences:
  • remove new line character from patterns
  • while loop condition is not file_avail
Do you see anything wrong with this method ?
X-Raym is offline   Reply With Quote
Old 06-26-2019, 04:46 AM   #23
Tale
Human being with feelings
 
Tale's Avatar
 
Join Date: Jul 2008
Location: The Netherlands
Posts: 3,007
Default

Quote:
Originally Posted by X-Raym View Post
Do you see anything wrong with this method ?
Well, if you don't remove the newline chars now, then you will likely still have to deal with them at some point (especially because there could de different versions, depending on source file, OS, etc.).

And about file_avail() and text files, this is what the JSFX documentation says:

Quote:
Note that file_avail() should be called to check for EOF after each read, and if it returns 0, the last file_var() should be ignored.
It doesn't mention file_string(), so maybe it works different from file_var() in this regard? Or maybe it depends on whether the last line has newline chars or not?
Tale is offline   Reply With Quote
Old 06-26-2019, 05:13 AM   #24
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Quote:
Well, if you don't remove the newline chars now, then you will likely still have to deal with them at some point (especially because there could de different versions, depending on source file, OS, etc.).
As I'm not sure file_string actually return this character, it seems to check file line after line (so stripping new line character).
Anyway, it works on windows like that, so if it works on mac too, it will be a valid solution


Have you tried it on Mac ?



Quote:
It doesn't mention file_string(), so maybe it works different from file_var() in this regard? Or maybe it depends on whether the last line has newline chars or not?
Not sure what happen there indeed, but from the end user point of view, having to know that he have to leave an empty line at the end of the file is a necessary information (it will be better to not have to remember such details).


Thx for the assistance
X-Raym is offline   Reply With Quote
Old 06-26-2019, 06:59 AM   #25
Tale
Human being with feelings
 
Tale's Avatar
 
Join Date: Jul 2008
Location: The Netherlands
Posts: 3,007
Default

Quote:
Originally Posted by X-Raym View Post
As I'm not sure file_string actually return this character, it seems to check file line after line (so stripping new line character).
Anyway, it works on windows like that, so if it works on mac too, it will be a valid solution
Are you sure? Because here "\r\n" or "\n" are not stripped. Note that simply doing gfx_drawstr() will mask the issue, but if you replace it with this then you will probably see what I mean:

Code:
gfx_drawchar('<');
gfx_drawstr(map[slider2]);
gfx_drawchar('>');
Quote:
Originally Posted by X-Raym View Post
Have you tried it on Mac ?
I just did, and it seems to work the same as on Windows, i.e. it also doesn't strip (or convert) "\r\n" or "\n". But at least it does work consistent across Windows and macOS (and probably also Linux).

--

I think you are right about file_string() and file_avail(), so maybe this would make more sense then:

Code:
while(file_avail(handle)) (
  file_string(handle, #line);
  ...
);
Tale is offline   Reply With Quote
Old 06-26-2019, 07:14 AM   #26
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Ok for stripping, I can still handle after parsing


Cheers !
X-Raym is offline   Reply With Quote
Old 06-26-2019, 07:21 AM   #27
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,114
Default

Quote:
match("%d %s*", #line, note, str)
This seems to do the trick !
X-Raym is offline   Reply With Quote
Old 06-26-2019, 07:22 AM   #28
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 8,194
Default

Indeed file_string() works fine for reading lines. See ReaPack->MIDI CCTable.
-Michael
mschnell is online now   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 01:42 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.