Audio input via TCP/IP

[mschnell]

Hi experts,

We would like to use Reaper to do a multi-channel (in fact quite a lot of channels) recording of multiple (very) remote signal sources, that come in as digital streams, for each of which we will provide a small Linux box to receive the Stream.

Now the Linux boxes will be connected to a "Server" via Ethernet.

We can do the software and drivers on that Linux box, and so we can support virtually any TCP/IP stream the "Server" might want to be fed with.

How can Reaper receive audio Streams via TCP/IP (or UDP) ?

- Is there some kind of TCP/IP to ASIO "Driver" that can be installed ?
- Is the Reaper <-> Reaper "Remote" TCP/IP protocol suitable and decently documented to be usable ?
- "Network Voice Protocol (NVP)" ?
- any other software-only options ?

Thanks
-Michael

[Banned]

Have you checked ReaStream yet?

[mschnell]

Originally Posted by Banned

Have you checked ReaStream yet?

I did not find that one yet.

Thanks a lot for the pointer !

-Michael

[mschnell]

Originally Posted by Banned

Have you checked ReaStream yet?

Great. But I do not find a documentation how to use it (the general "ReaEffectsGuide" just states:

"19 ReaStream: ReaStream is not an audio effect and therefore strictly speaking outside the scope of this document."

But does not provide a link to the actual description.

What is the format of the IP Data Stream it accepts as an Audio input ?

-Michael

[Banned]

Well, it seems easy enough to use - but your question is about reverse engineering it, I guess? Not the sort of thing user manuals explain, is it?

[mschnell]

Originally Posted by Banned

Well, it seems easy enough to use

In fact I did not even find any hint/documentation for what purpose it is meant to be used.

Originally Posted by Banned

- but your question is about reverse engineering it, I guess? Not the sort of thing user manuals explain, is it?

For me, with networking tools, the description of the protocol running via the network is the user manual.


In fact in the " Reastream Audio format" thread in this forum I did see that Grag38 already did trace the network traffic ReaStream sends out (and I suppose it might be able to receive the same data format). So this might be a starter.

A problem for us might be that ReaStream seemingly can't be configured to use a certain IP port number for receiving a Stream. As we will have many (some 30) streams coming in, of course each would need it's own dedicated port.

-Michael

[msmucr]

Here is one mention about ReaStream protocol by Justin..

http://www.askjf.com/index.php?q=2440s
http://www.1014.org/shiz/reastream.txt

Good luck,

Michal

[mschnell]

Great. Thanks a lot.

Problem: if really only a single Receive IP Port is supported, we can't use the VST on multiple tracks and have multiple sender boxes connect to each of them

-Michael

[msmucr]

Originally Posted by mschnell

Great. Thanks a lot.

Problem: if really only a single Receive IP Port is supported, we can't use the VST on multiple tracks and have multiple sender boxes connect to each of them

-Michael

Hi Michael,

frankly I haven't tried it, but it should be technically possible using multicast/broadcast packets.
In case of standard unicasts is maybe socket opened via first instance of plugin in DAW and then shared, i really don't know, just guessing, but on two machines and using broadcasts, it is working, so one is able to insert several sending plugins at one daw and several receiving plugins in other one. And everything is working as expected using identifier keywords.
I have only two Windows computers by hand, third should be in virtual machine, where I haven't Reaper installed, so I can't try it now.

Just for case, I recall one alternative plugin for that called Wormhole.
https://code.google.com/p/wormhole2/
AFAIK it works by using among multiple machines and also using broadcast UDP packets.

This type of project really suits Ravenna, Dante (in future both under AES67 umbrella) audio protocols, which both provides virtual ASIO drivers for Windows (Dante also for Mac) and solves also few other related tasks like clock synchronization etc.
But Linux sender requirement is probably show-stopper here, neither ACL-X or Audinate is going to create virtual ALSA drivers for Linux. There are some proprietary boards with MCUs and ASIC chips for that, but it is more like reference implementations of standard and quite pricey.

Michal

[mschnell]

Thanks a lot for the additional pointers.

-Michael

[mschnell]

Another issue is that I did not yet find out if/how it is possible to have Reaper save the output of the effect chain (i.e. the VST) in a wave file (i.e. do the recording) in a highly automated session (i.e. we need to have Reaper listen to a remote control via OSC commands to start and stop the recording)

Thanks again,
-Michael

[msmucr]

Originally Posted by mschnell

Another issue is that I did not yet find out if/how it is possible to have Reaper save the output of the effect chain (i.e. the VST) in a wave file (i.e. do the recording) in a highly automated session (i.e. we need to have Reaper listen to a remote control via OSC commands to start and stop the recording)

- recording of VSTs is possible, just right click on red arm button and from the context menu select record output (either stereo, mono, multich.), this will actually record exactly, what you hear.
- record trigger via MIDI/OSC is possible, just find appropriate action in Action list and map shortcut (of course you'll need to setup OSC device first at Preferences/Control surfaces and allow binding to actions)

Michal

[mschnell]

Originally Posted by msmucr

- recording of VSTs is possible, just right click on red arm button and from the context menu select record output (either stereo, mono, multich.), this will actually record exactly, what you hear.

GREAT ! Thanks.

Originally Posted by msmucr

- record trigger via MIDI/OSC is possible, just find appropriate action in Action list and map shortcut (of course you'll need to setup OSC device first at Preferences/Control surfaces and allow binding to actions)

I did know this.

Just as I have your attention (though it is off-topic in this thread) We would additionally need to send out and/or query the recording on/paused state via OSC.

Do you know if this is possible ?

-Michael

[msmucr]

Originally Posted by mschnell

Just as I have your attention (though it is off-topic in this thread) We would additionally need to send out and/or query the recording on/paused state via OSC.

Do you know if this is possible ?

-Michael

I believe, that osc default messages for play/stop/records are triggers (check commented osc/Default.ReaperOSC file at your Reaper's resource directory), you can't "query" its status. If you send respective osc message from your control device, it will fire the trigger (change state) and send same message as feedback - it won't work. Similarly, if you change state of record switch from Reaper itself, it will send feedback to the device.
I'm not really sure but, maybe it will be worth of checking, if there aren't some OSC messages automatically send from the Reaper after establishing of OSC connection.

Anyway, check this forum and possibly create new thread regarding this..
http://forum.cockos.com/forumdisplay.php?f=47

Another way to do real query functionality is to write custom JS EEL script, which will be running at background, communicate via TCP and send whatever custom messages. There are both functions for socket communication and also functions for querying state of Reaper's playback/record. But of course, it won't be really OSC then.

Michal

[mschnell]

Originally Posted by msmucr

Another way to do real query functionality is to write custom JS EEL script, which will be running at background, communicate via TCP and send whatever custom messages. There are both functions for socket communication and also functions for querying state of Reaper's playback/record. But of course, it won't be really OSC then.

Reaper is just great

Thanks a lot !
-Michael

[mschnell]

Rerurnintg to the original topic:

Maybe in Reaper a custom EEL script is even able to handle an Audio Stream and feed it into a Reaper tracks for having it recorded ?!?!?!

In fact for the friend, I am doing this research for, a Python script would be even more attractive, as he is a Python Pro, anyway.

But I did not take a look at the Reaper Python integration, yet.

-Michael

[msmucr]

Originally Posted by mschnell

Rerurnintg to the original topic:

Maybe in Reaper a custom EEL script is even able to handle an Audio Stream and feed it into a Reaper tracks for having it recorded ?!?!?!

In fact for the friend, I am doing this research for, a Python script would be even more attractive, as he is a Python Pro, anyway.

But I did not take a look at the Reaper Python integration, yet.

-Michael

I don't think it would be good option for that, VST plugins are directly in audio path of Reaper and will be much more efficient for streaming audio tasks, asynchronous communication etc. It is big difference between some status query or transport control message and audio streaming from multiple source endpoints.
And I'm not really sure about required REAScript (regardless it is EEL or Python flavor) functionality to create new PCM data in project (there are sink function at API, but I've never tried it).
REAScript is IMO really meant primarily for controlling of Reaper built-in functionality (eg. naming, tracks, effects, automation, mangling with already present items eg.).

Also with regards to Python.. normally script blocks Reaper's UI thread until it is finished, EEL is possible to run at background (check some of spk77's eel scripts at corresponding forum). For background running of Python check thread about contributed Python library "beyond.Reaper". http://forums.cockos.com/showthread.php?p=1290595

Michal

[mschnell]

Thanks a lot for all the interesting insight you provided !

-Michael

[mschnell]

After doing some tests and starting to do the software for a "BeagleBone" that (some 20 Boxes) will be used as USB->Ethernet converter.

- ReaStream does not work in such a setup. It obviously does not have a "PLL" that allows for appropriate re-sampling of Sample streams with different clocks. Quite obviously for ReaStraem you need to provide a common master-clock to the audio inputs of all connected boxes.

- Wormhole does work on that behalf.

So we analyzed the protocol Wormhole uses by means of WireShark and looking at the source code:

It uses UDP broadcast in both directions
- The senders notify the receiver that they exist on a regular schedule
- The sender gets a message from the receiver when it wants to get data
- The sender sends a stream without getting any acknowlagements.

Hence the job seems to be doable, as the receiver (Wormhole VST in Reaper) needs to do the resampling and jitter compensation while the sender (Beaglebone with new software) just needs to send data blocks when getting enough samples from the ALSA subsystem.)

Thanks for the pointers !

-Michael

[msmucr]

Hi Michael,

thanks for an update!

Just one note regarding chosen BeagleBone.
By coincidence, I've spent some time with this board and Sitara SoC early last year.
You've mentioned, you'd like to use it with USB (presumably audio interface). There is quite nasty hardware bug at SoC, which basically fires USB controller DMA earlier than buffer is really processed by downstream hardware. It affects mostly isochronous transfers, which is used for instance by audio interfaces.
This is quite lethal, so there are basically two choices for handling of this. One is PIO mode, which is usually default option for BB kernel builds. Of course PIO means very high CPU drain compared to working DMA transfers. Other choice is implement some workaround for checking of real buffer status. This also takes some cpu cycles, which was unfortunately unacceptable for audio playback with other tasks running at the box.
I haven't checked it later (project was moved to iMX6). But there were new patches by Daniel Mack
revert of original workaround:
https://git.kernel.org/cgit/linux/ke...43074844af83c0
Part of new workaround with explanation:
https://git.kernel.org/cgit/linux/ke...da866f7fd836e4

Just to point you to issue.. Maybe you've already tested it and newer kernels works for you or you're fine with PIO mode. But be sure to check it before you'll order 20+ BeagleBones.

Good luck!

Michal

[mschnell]

Hi Michal.

Thanks a lot for these hints !

We'll let you know what we find out.

-Michael

[mschnell]

In fact we already did do some software for the BeagleBone. Right now we are implementing the WormHole "Sender" protocol.

We use the Beaglebone with standard Debian and attach an Audio A/D device via USB using the standard Debian USB-ALSA driver.

There is a standard command-line software that allows to send an ALSA audio stream to its STDOUT, which hence can be used as any Linux Pipe and e.g. directed into a file via a normal bash command line.

We did not notice any problem doing this.

I suppose the standard mode used for the isochronous USB transfers is PIO, which of course is a lot less effective than DMA.

But we need to do just a single audio stream mono 16 Bit, 16 K samples/second.

Supposedly this is handled easily in PIO mode.

OTOH we will have lots of these BeagleBone based devices in a Network, all sending to a WormHole VST in Reaper that is supposed to separate the input in mono tracks and save the streams in files.

Thanks a lot !
-Michael

[msmucr]

I also still like BeagleBone Black except of this issue.

Your use case is bit different.
Our main planned usage was audio streamer (incl. high resolution files) to a class compliant USB soundcard. Just playback in PIO mode was so-so, although kernel handler for USB interrupts almost eated CPU. But when it was combined with network activity over SMB (eg. recursive reading of shared directories with audio files and fetching its headers), audio x-runs were all over place.

But I'm glad, if it is working for you. I just recall our issue and want to mention it.

Michal

[mschnell]

We got it working just nicely:

- Audio device generating 16 K Samples / sec 16 Bit Integer
-> USB 2.0 (class compliant)
-> Beagle Bone USB port
-> Standard Debian Audio USB driver
-> ALSA Subsystem
-> Standard command-line program reading´the stream from ALSA and copying it to STDOUT
-> Homebrew application reading STDIN (blocking read Buffers with 64 Mono Samples)
-> Converting each buffer to Wormhome format (64 Stereo Samples 32 Bit float)
-> Sending Buffer to the Reaper PC via UDP
-> Wormhole receiving Audio Stream
-> Reaper (set to 16 K Samples / sec) fetches the Stream from the VST
-> Reaper saves the track to files.

As we also use class compliant Audio on USB, we will test of a standard stereo 44.1 K Sample / sec 24 Bit device might be working as well.

-Michael

[airon]

This looks very interesting indeed.

I wanted to help out some friends who run LAN tournaments, and this plugin looks like the perfect way to monitor a VNC-remote-controlled mix done in Reaper, while monitoring on a Reaper instance here, two continents away.

[mschnell]

Right there are some problems:

We need to use very high buffer settings to prevent dropouts. This results in a rather big latency (no problem for us, if the latency is similar for all devices we attach. We still need to check this out. If you want to use the Internet for transport you will need a latency of 10...30 seconds, as the net delay is huge.

Right now out devices only can send. We are planning to add receive support, but receiving is a lot more complicated than sending, as the receiver needs to handle the jitter compensation.

Right now we only tested a single channel 16 Bits 16 K Samples / sec.

-Michael