Old 09-14-2020, 07:44 AM   #1
scipherT
Human being with feelings
 
Join Date: Oct 2019
Posts: 5
Default EEL2: Reading/writing all 64 bits in virtual memory location?

Hi folks,

I have been using EEL2 for a few months, and find it to be a nice, minimalistic language. One thing I would like to be able to do, but have failed to (or at least I think so), is to use the virtual memory space to store arbitrary bits and bytes tightly packed in each memory location. And I suspect this is due to the fact that the memory locations are only accessible as binary64 values (due to their C back-end implementation, please correct me if I'm wrong here!). The bit shift operators only operate on 32 bit integers, which makes it possible to do bit manipulation on half of the 64-bit word, and I have successfully done that. But to set/get bits or bytes on more than half of the memory word seems more or less impossible. My goal would be to store 8 bytes of data in a (64-bit) memory word and be able to read and write each byte individually, without performing overly complex arithmetic.

Is there anyone here that has managed to use more than 32 bits of a 64-bit memory word to store arbitrary bits or bytes of data? Can it be done?

I'm very grateful for any comments or insights!
scipherT is offline   Reply With Quote
Old 09-14-2020, 08:39 AM   #2
DarkStar
Human being with feelings
 
DarkStar's Avatar
 
Join Date: May 2006
Location: Surrey, UK
Posts: 18,565
Default

A background question: do you really need to cram all those 8-bit bytes into the 64-bit words? Have you got billions of bytes to sotre?

First thoughts (thinking as I type):
-- write a function that that places the 8 bits into the specified 8-bit "slot" of the 64-bit word

pack(this_byte, this_word, slotN);
// 1 & (AND) this_word to get the right-most part, after the specified slot,
// 2 >> (shift right) this_word with 1 or 2 shift operations to get the part before the slot
// 3 << shift that left by 8
// 4 + (add in) this_byte
// 5 << (shift left) that to the top-most bits
// 6 + (add) that to the right-most part (from step 1).

You'll need some temporary (local) variables too
__________________
DarkStar ... interesting, if true. . . . Inspired by ...
DarkStar is offline   Reply With Quote
Old 09-14-2020, 01:34 PM   #3
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

As there are invalid "NAN" bit combinations for real values, you can't use all 64 bits.

You can store 5 bytes in the mantissa by appropriate operations (I have code for this). Trying to sqeeze more in a real supposedly is rather complicated.

-Michael
mschnell is offline   Reply With Quote
Old 09-15-2020, 03:07 PM   #4
scipherT
Human being with feelings
 
Join Date: Oct 2019
Posts: 5
Default

Quote:
Originally Posted by DarkStar View Post
A background question: do you really need to cram all those 8-bit bytes into the 64-bit words? Have you got billions of bytes to sotre?

First thoughts (thinking as I type):
-- write a function that that places the 8 bits into the specified 8-bit "slot" of the 64-bit word

pack(this_byte, this_word, slotN);
// 1 & (AND) this_word to get the right-most part, after the specified slot,
// 2 >> (shift right) this_word with 1 or 2 shift operations to get the part before the slot
// 3 << shift that left by 8
// 4 + (add in) this_byte
// 5 << (shift left) that to the top-most bits
// 6 + (add) that to the right-most part (from step 1).

You'll need some temporary (local) variables too
Thanks DarkStar for the suggestion, I might try it.

To answer your question if I need billions of bytes (or as I interpret it; more than is available in EEL2); no, not really. At least not yet. I am designing a state machine implemented as a quite large table that could potentially reach a size of a few million words. However, I like to keep things tidy, and consume less resources, even if it is not needed. And it helps me understand the limits of the language. I have previously managed to pack bytes into what I believe is the lower 32 bits of the 64 bit float that virtual memory is made out of. I also must admit I don't fully comprehend your algorithm. Do you think it can overcome the 32 bit limitation (considering that the shift operator can only shift within the boundaries of 32 bits)?

Quote:
Originally Posted by mschnell View Post
As there are invalid "NAN" bit combinations for real values, you can't use all 64 bits.

You can store 5 bytes in the mantissa by appropriate operations (I have code for this). Trying to sqeeze more in a real supposedly is rather complicated.

-Michael
Thanks Michael for the information.

Do you know the exact implementation of the double precision floating point variables used by EEL2? Is it IEEE 754 binary64? In that case the fraction part is 52 bits and that would in theory be able to accommodate up to 6 bytes (and 4 extra bits). To actually get the bits in the right place is another matter of course.

Five bytes in the mantissa is at least one more byte than I have managed. If you care to share any hint to how you did it I would be very interested. One idea I have myself is to use multiplication instead of shifting, to overcome the 32 bit limitation of the shift operator. But multiplication might perhaps be more computational intense than bit shift. Or not. It depends on the implementation I guess. And I realize; if computational speed is important, refraining from packing bits and bytes at all to begin with is the easiest option.


Anyway, thanks guys for answering my original post which was the first one from me on this forum. I will experiment further and your input has been helpful.

-s

Last edited by scipherT; 09-15-2020 at 04:51 PM.
scipherT is offline   Reply With Quote
Old 09-15-2020, 10:30 PM   #5
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

EEL 2 uses 64 bit integer as supplied by the hardware. Hence it will be IEEE.


I once used this code to store a string in a variable zo allow for passing strings via gmem:

Code:
function encode_string(s) local(c_s, i) (
  i = 0;
  c_s = 0;
  while(
    c_s = c_s << 8;
    c_s = c_s + str_getchar(s, 4-i);  
    i = i+1;
    i < 5;
  );
  c_s;
);

function encode_5(r*, s) (
  r._1 = encode_string(strcpy_substr(#, s, 0, 4));
  r._2 = encode_string(strcpy_substr(#, s, 4, 4));
  r._3 = encode_string(strcpy_substr(#, s, 8, 4));
  r._4 = encode_string(strcpy_substr(#, s, 12, 4));
  r._5 = encode_string(strcpy_substr(#, s, 16, 4));
);

function decode_string(c_s) local (i, c_ss, _) (
  i = 0;
  strcpy (_, "");
  while(
    c_ss = c_s & 0xFF;
    c_s = c_s >> 8;
    c_ss >= 32 ? (
      str_setchar(_, -0.25, c_ss);
        i += 1;
    );  
    (i < 4) & (c_ss >= 32);
  );
  _;
);

function decode_5(r*) (  
  #__ =  decode_string(r._1);
  #__ += decode_string(r._2);
  #__ += decode_string(r._3);
  #__ += decode_string(r._4);
  #__ += decode_string(r._5);
);
Maybe this is really only 4 bytes per real variable (but 5 variables per string), as the bit-logical operations in EEL are done in 32 bit.

-Michael

Last edited by mschnell; 09-15-2020 at 10:37 PM.
mschnell is offline   Reply With Quote
Old 09-15-2020, 11:40 PM   #6
Tale
Human being with feelings
 
Tale's Avatar
 
Join Date: Jul 2008
Location: The Netherlands
Posts: 3,134
Default

You could probably use multiplication/division instead of bit shifting, and addition instead of bitwise or i.e.:

Code:
x = byte1 + byte2*2^8 + byte3*2^16 + byte4*2^24 + byte5*2^32 + byte6*2^40;

byte1 = x & 0xFF;
byte2 = (x / 2^8) & 0xFF;
byte3 = (x / 2^16) & 0xFF;
byte4 = (x / 2^24) & 0xFF;
byte5 = (x / 2^32) & 0xFF;
byte6 = (x / 2^40) & 0xFF;
Tale is offline   Reply With Quote
Old 09-16-2020, 02:33 AM   #7
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

Here functions that encode and decode seven bytes to an EEL (float 64) variable:

Code:
desc: 64
@init
  f._1 = 2^8;
  f._2 = 2^16;
  f._3 = 2^24;
  f._4 = 2^32;
  f._5 = 2^40;
  f._6 = 2^48;
  log2_52 = log(2)*52;

@block

function encode64(b*) local (r,s) (
  r = b._6 & 0x0F;
  s = b._0 + f._1*b._1 + f._2*b._2 + f._3*b._3 + f._4*b._4 + f._5*b._5 + f._6*r;
  r = ((b._6 >> 4) & 0x0F) * 52;
  s == 0 ? -(2^r) :  s * 2^r;
);

function decode64(b*, x) (
  x < 0 ? (
    r = log(-x)/log2_52;
    r &= 0x0F;
    b._0 = 0;
    b._1 = 0;
    b._2 = 0;
    b._3 = 0;
    b._4 = 0;
    b._5 = 0;
    b._6 = r<<4;
   ) : ( 
    r = log(x)/log2_52;
    r &= 0x0F;
    x /= 2 ^ (r * 52);
    b._0 = x & 0xFF;
    b._1 = (x / f._1) & 0xFF;
    b._2 = (x / f._2) & 0xFF;
    b._3 = (x / f._3) & 0xFF;
    b._4 = (x / f._4) & 0xFF;
    b._5 = (x / f._5) & 0xFF;
    b._6 =((x / f._6) & 0x0F) | (r<<4);
  );  
);

  x._0 =  0;
  x._1 =  0;
  x._2 =  0;
  x._3 =  0;
  x._4 =  0;
  x._5 =  0;
  x._6 =  0;
  z1= encode64(x);
  decode64(y1, z1);
  
  x._6 += 0xE0;  
  z2= encode64(x);
  decode64(y2, z2);
-Michael

Last edited by mschnell; 09-16-2020 at 03:02 AM.
mschnell is offline   Reply With Quote
Old 09-16-2020, 10:20 AM   #8
DarkStar
Human being with feelings
 
DarkStar's Avatar
 
Join Date: May 2006
Location: Surrey, UK
Posts: 18,565
Default

It could be interesting to compare the additional CPU usage for packing / unpacking with the additional memory usage of the "simple" version.
__________________
DarkStar ... interesting, if true. . . . Inspired by ...
DarkStar is offline   Reply With Quote
Old 09-16-2020, 01:49 PM   #9
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

Tale's version handles 6 bytes instead of 7, so they can't be compared.

-Michael
mschnell is offline   Reply With Quote
Old 09-16-2020, 05:46 PM   #10
scipherT
Human being with feelings
 
Join Date: Oct 2019
Posts: 5
Default

@Tale, @Michael; thanks for your effort here! I'm really happy about how quickly you people are replying and commenting.

@Tale: I do understand the 6-byte approach, it looks straightforward. Definitely something I will try.

@Michael: The 7-byte approach looks quite advanced. I can't say I fully grasp it, yet. I'm kind of guessing it uses the exponent to store user bits as well. I will try to wrap my head around it

I might be diverging a bit when I say that there also are some EEL2 concepts in that code that I haven't grasped yet. I think I have read all the official JSFX and EEL2 references. If you feel you have the patience to answer, that would be awesome:

- "b*" as formal parameter; what does it mean? Anything to do with reference?
- "b._1", "b._2", etc; what does it mean? Namespaces? Dereferencing? I have the feeling it is related to "b*" and thus not a namespace thing. But if not, what does "._1" etc mean?

Sorry for asking a lot of (possibly stupid) general EEL2 language stuff, I'm just a bit confused. And eager to learn

-s
scipherT is offline   Reply With Quote
Old 09-17-2020, 04:19 AM   #11
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

The 7 byte implementation uses the 52 bit mantissa for six and a half byte and four bits in the exponent for the remaining half byte, (11 Bits in the exponent: one unusable (sign), 6 used for making the mantissa an integer, 4 remaining for us.)

If the mantissa is zero, obviously the exponent bits are inaccessible, hence in that case a negative value with a dummy mantissa is used.

The dot notation (object.whatever, "whatever" being the actual variable for one value) and the star notation for formal parameters is the EEL2 way to implement "objects" that contain multiple values and can be easily managed and passed to functions. (I seem to remember that in the docs it is called "namespace", pointing to some even more sophisticated usage.) I feel this is very similar to waht is done in Python,

Have fun !
-Michael

Last edited by mschnell; 09-23-2020 at 11:51 AM.
mschnell is offline   Reply With Quote
Old 09-17-2020, 06:08 AM   #12
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

I suppose with some more effort, wi can sqeeze 15 Byte in two variables,

I'll try tomorrow...
-Michael
mschnell is offline   Reply With Quote
Old 09-17-2020, 12:02 PM   #13
scipherT
Human being with feelings
 
Join Date: Oct 2019
Posts: 5
Default

@Michael: Thanks for clarifying the star notation, I didn't find it in the language reference. Really cool stuff doing that fraction/exponent Tetris thing. I find it intrinsically fascinating.

Last edited by scipherT; 09-17-2020 at 01:42 PM.
scipherT is offline   Reply With Quote
Old 09-18-2020, 12:43 AM   #14
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

Unfortunately i found that the 7 Byte version does not work for certain sets of bytes (e.g. all 0xFF).
Trying to fix...
-Michael
mschnell is offline   Reply With Quote
Old 09-18-2020, 02:33 AM   #15
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

This version seems to work

Now the sign bit is used instead of the highest mantissa bit, avoidig too high mantissa values, that seem to create a rounding problem.

To avoid mantissa = 0, an offset of 1 is added to it.

TODO: use unique names for the constants.

Code:
desc: 64
@init

  mbits    = 52;
  offset   = 0;

  shiftl   = 56-mbits;
  shiftr   = 8-shiftl;
  masklow  = 0xff >> shiftr;
  f._0 = 2^(0-offset);
  f._1 = 2^(8-offset);
  f._2 = 2^(16-offset);
  f._3 = 2^(24-offset);
  f._4 = 2^(32-offset);
  f._5 = 2^(40-offset);
  f._6 = 2^(48-offset);
  
  log2_mbits = log(2)*mbits;

@block



function encode64(b*) local (r,s) (
  b._6 & 0x08 ? (
    r = b._6 & 0x07& masklow;
    s =  f._0*b._0 + f._1*b._1 + f._2*b._2 + f._3*b._3 + f._4*b._4 + f._5*b._5 + f._6*r + 1;
    r = ((b._6 >> shiftr) & masklow) * mbits;
    -s * 2^r;
   ) : ( 
    r = b._6 & masklow;
    s =  f._0*b._0 + f._1*b._1 + f._2*b._2 + f._3*b._3 + f._4*b._4 + f._5*b._5 + f._6*r + 1;
    r = ((b._6 >> shiftr) & masklow) * mbits;
    s * 2^r;
  );  
);

function decode64(b*, x) (
  x < 0 ? (
    x = -x;
    r = log(x)/log2_mbits;
    ___r = r;
    r &= masklow;
    ___r1 = r;
    x /= 2 ^ (r * mbits);
    x -= 1;
    b._0 = (x / f._0) & 0xFF;
    b._1 = (x / f._1) & 0xFF;
    b._2 = (x / f._2) & 0xFF;
    b._3 = (x / f._3) & 0xFF;
    b._4 = (x / f._4) & 0xFF;
    b._5 =((x / f._5) & 0xFF) ;
    b._6 =((x / f._6) & masklow) | 0x08 | (r<<shiftl);
   ) : ( 
    r = log(x)/log2_mbits;
    ___r = r;
    r &= masklow;
    ___r1 = r;
    x /= 2 ^ (r * mbits);
    x -= 1;
    b._0 = (x / f._0) & 0xFF;
    b._1 = (x / f._1) & 0xFF;
    b._2 = (x / f._2) & 0xFF;
    b._3 = (x / f._3) & 0xFF;
    b._4 = (x / f._4) & 0xFF;
    b._5 = (x / f._5) & 0xFF;
    b._6 =((x / f._6) & masklow) | (r<<shiftl);
  );  
);

  x._0 =  0;
  x._1 =  0;
  x._2 =  0;
  x._3 =  0;
  x._4 =  0;
  x._5 =  0;
  x._6 =  0x10;
  z0= encode64(x);
  decode64(y0, z0);


  x._0 =  11;
  x._1 =  22;
  x._2 =  33;
  x._3 =  44;
  x._4 =  55;
  x._5 =  66;
  x._6 =  77;
  z1= encode64(x);
  decode64(y1, z1);
  
/*
  x._0 =  0x49;
  x._1 =  0xFF;
  x._2 =  0xFF;
  x._3 =  0xFF;
  x._4 =  0xFF;
  x._5 =  0xFF;
  x._6 =  0xFF;
  z2= encode64(x);
  decode64(y2, z2);
*/  
  
  x._0 =  0xFF;
  x._1 =  0xFF;
  x._2 =  0xFF;
  x._3 =  0xFF;
  x._4 =  0xFF;
  x._5 =  0xFF;
  x._6 =  0xFF;
  z3= encode64(x);
  decode64(y3, z3);
  
/*
  x._0 =  0;
  x._1 =  0;
  x._2 =  0;
  x._3 =  0;
  x._4 =  0;
  x._5 =  0;
  x._6 =  0x1FF;
  z4= encode64(x);
  decode64(y4, z4);
/*
-Michael

Last edited by mschnell; 09-18-2020 at 02:42 AM.
mschnell is offline   Reply With Quote
Old 09-18-2020, 06:47 AM   #16
scipherT
Human being with feelings
 
Join Date: Oct 2019
Posts: 5
Default

This is getting better and better, thanks! I think I might try both the straight-froward mantissa-only approach that Tale suggested and the more advanced mantissa+exponent+sign approach by Michael, and then see if both are fast enough for my application. It might take a while to try this with a reasonable load, but eventually I hope to get there.

You have all been very helpful. One of the reasons I started using Reaper is because I anticipated a helpful community. My expectations have been surpassed, you guys are the best!

-s
scipherT is offline   Reply With Quote
Old 09-20-2020, 04:31 AM   #17
mschnell
Human being with feelings
 
mschnell's Avatar
 
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 9,532
Default

Due to automatic normalization, 6 bits in the exponent are not accessible.

Hence there only is one additional bit that might be used to store something (the sign of the exponent). So storing 15 bytes in two variables is not possible.

Here a (final) version with more secure variable names, that might be slightly faster than the previous one..

We could do a thingy that stores an eight character ASCII string in a variable, though...

Have fun !

Code:
desc: 64
@init

  e64_mbits    = 52;
  e64_offset   = 0;

  e64_shiftl   = 56-e64_mbits;
  e64_shiftr   = 8-e64_shiftl;
  e64_masklow  = 0xff >> e64_shiftr;
//  e64_f._0 = 2^(0-e64_offset);
  e64_f._1 = 2^(8-e64_offset);
  e64_f._2 = 2^(16-e64_offset);
  e64_f._3 = 2^(24-e64_offset);
  e64_f._4 = 2^(32-e64_offset);
  e64_f._5 = 2^(40-e64_offset);
  e64_f._6 = 2^(48-e64_offset);
  
  e64_ = log(2)*e64_mbits;

@block



function encode64(b*) local (r,s) (
  b._6 & 0x08 ? (
    r = b._6 & 0x07& e64_masklow;
//    s =  e64_f._0*b._0 + e64_f._1*b._1 + e64_f._2*b._2 + e64_f._3*b._3 + e64_f._4*b._4 + e64_f._5*b._5 + e64_f._6*r + 1;
    s =  b._0 + e64_f._1*b._1 + e64_f._2*b._2 + e64_f._3*b._3 + e64_f._4*b._4 + e64_f._5*b._5 + e64_f._6*r + 1;
    r = ((b._6 >> e64_shiftr) & e64_masklow) * e64_mbits;
    -s * 2^r;
   ) : ( 
    r = b._6 & e64_masklow;
//    s =  b._0 + e64_f._1*b._1 + e64_f._2*b._2 + e64_f._3*b._3 + e64_f._4*b._4 + e64_f._5*b._5 + e64_f._6*r + 1;
    s =  b._0 + e64_f._1*b._1 + e64_f._2*b._2 + e64_f._3*b._3 + e64_f._4*b._4 + e64_f._5*b._5 + e64_f._6*r + 1;
    r = ((b._6 >> e64_shiftr) & e64_masklow) * e64_mbits;
    s * 2^r;
  );  
);

function decode64(b*, x) local(r)(
  x < 0 ? (
    x = -x;
    r = log(x)/e64_;
    r &= e64_masklow;
    x /= 2 ^ (r * e64_mbits);
    x -= 1;
//    b._0 = (x / e64_f._0) & 0xFF;
    b._0 =  x         & 0xFF;
    b._1 = (x / e64_f._1) & 0xFF;
    b._2 = (x / e64_f._2) & 0xFF;
    b._3 = (x / e64_f._3) & 0xFF;
    b._4 = (x / e64_f._4) & 0xFF;
    b._5 =((x / e64_f._5) & 0xFF) ;
    b._6 =((x / e64_f._6) & e64_masklow) | 0x08 | (r<<e64_shiftl);
   ) : ( 
    r = log(x)/e64_;
    r &= e64_masklow;
    x /= 2 ^ (r * e64_mbits);
    x -= 1;
//    b._0 = (x / e64_f._0) & 0xFF;
    b._0 =  x         & 0xFF;
    b._1 = (x / e64_f._1) & 0xFF;
    b._2 = (x / e64_f._2) & 0xFF;
    b._3 = (x / e64_f._3) & 0xFF;
    b._4 = (x / e64_f._4) & 0xFF;
    b._5 = (x / e64_f._5) & 0xFF;
    b._6 =((x / e64_f._6) & e64_masklow) | (r<<e64_shiftl);
  );  
);

  x._0 =  0;
  x._1 =  0;
  x._2 =  0;
  x._3 =  0;
  x._4 =  0;
  x._5 =  0;
  x._6 =  0x10;
  z0= encode64(x);
  decode64(y0, z0);


  x._0 =  11;
  x._1 =  22;
  x._2 =  33;
  x._3 =  44;
  x._4 =  55;
  x._5 =  66;
  x._6 =  77;
  z1= encode64(x);
  decode64(y1, z1);
  
/*
  x._0 =  0x49;
  x._1 =  0xFF;
  x._2 =  0xFF;
  x._3 =  0xFF;
  x._4 =  0xFF;
  x._5 =  0xFF;
  x._6 =  0xFF;
  z2= encode64(x);
  decode64(y2, z2);
*/  
  
  x._0 =  0xFF;
  x._1 =  0xFF;
  x._2 =  0xFF;
  x._3 =  0xFF;
  x._4 =  0xFF;
  x._5 =  0xFF;
  x._6 =  0xFF;
  z3= encode64(x);
  decode64(y3, z3);
  
/*
  x._0 =  0;
  x._1 =  0;
  x._2 =  0;
  x._3 =  0;
  x._4 =  0;
  x._5 =  0;
  x._6 =  0x1FF;
  z4= encode64(x);
  decode64(y4, z4);
/*

Last edited by mschnell; 09-20-2020 at 10:42 PM.
mschnell is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:10 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.