Reaper doesn't import SysEx command of the inserted MIDI file (FIXED)

[John1900]

I have a problem with importing a midi file in Reaper. It's a midi file, generated by the Hauptwerk organ simulator. This file contains a lot of SysEx commands for register settings, etc.
It starts with a SysEx message, which is a cancel for Hauptwerk.
After that several CC messages are in the file, then a new set of SysEx commands and then notes, that are played.

All SysEx messages are not imported in Reaper (I cannot find them in the Eventlists editor) and because (??) the SysEx message at the start of the file, all CC messages are suffled and some are gone (not imported in Reaper)

I haven't found an option for filtering SysEx or something like that, so in my opinion also SysEx commands must be shown in Reaper.
But they aren't.

I think that ther is a bug in Reaper for this (import a midi file with the command: Insert --> Media file): SysEx messages are not imported and a result is the the cc messages after the SysEx message are not correct imported.

I have several screen dumps of eventlists in Reaper, Anvil Studio and Music Studio Producer to show the difference.
The pictures "<prg> (Start)" shows the begin of the midi file
The pictures "<prg> (play)" shows the moment of playing and the missing SysEx messages (because limit of 5 attachments, the 6th picture (Anvil Studio) cannot be uploaded but it looks the same as Music Studio Producer).


The Hauptwerk file is available if you need this.

[John1900]

The 6th picture and the midi file (zipped)

John1900

[DarkStar]

Hello John1900 and welcome to these forums. You'll find many helpful people here and a lot of useful information.

?? looks OK to me, I can see 68 SYSEX messages here in the MIDI clip:

... Big pic: https://i.imgur.com/pzOFeXY.png

And here's is the start of the clip showing the CC messages:
... Big pic: https://i.imgur.com/GO01enD.png

[John1900]

Hello DarkStar

thanks for looking to my problem.

Can you tell me how it is possible that you can see the SysEx messages?
Are there special settings in Reaper? I have take a look in the manual, the forum and the Options --> Preferences configuration windows and cannot find a setting for a filter of something like that.
I use the latest version of Reaper in Windows XP. Aftre starting Reaper I do Insert --> Media file and choose the midi file. Because the 4 tracks in this file I expand this to 4 separate tracks in Reaper

[DarkStar]

Nothing special; I changed the Channel selector to 6 so that the CC messages were not displayed.

Or you can display them in a Controller lane by selecting 'SYSEX' in the drop-down list:

... Big pic: https://i.imgur.com/DCJdiob.png

[John1900]

Hello DarkStar

This morning I have done some tests.
I install a new portable Reaper (Evaluation version) on XP; nothing is changed in preferences. With this I redo the test on both a XP and a Windows 7 PC with the same midi file; it gives the same results: no SysEx messages.

Looking to the difference between your latest picture on the forum and my piano roll editor, your picture contains much more notes (events). After some experiences I discover that you insert my 4 channel midi file with the option "Multichannel item on a single track" while I do the other option "Single channel items on multiple tracks".
In your case you have SysEx commands in the Reaper editor, in my case not.

I have chosen the "Single channel items on multiple tracks" option because the Hauptwerk organ simulator has more channels; each channel is for a organ manual or pedal. You can play on a manual and pedal and a solo voice on an other manual. In my case there are 4 manuals.
In case with each manual on his own channel in Reaper, it's easier for me to see and to edit notes on the different organ manuals comparing with the score.

I continue the test with inserting the midi file with the "Multichannel item on a single track".
I have exported this "single track" midi file (see attachment) and test this with my organ. It doesn't work correctly. I discover that Reaper shuffles the Non-Reg Parm LSB, NRP MSB and the Data Entry MSB.
See picture https://dl.dropboxusercontent.com/u/...%20file%29.jpg. The start is different comparing the pictures, attached in my first topic and near event 42/43 the MSB (99) parameter is not in the exported midi file.

Questions:
1) is my way of handling the midi file wrong?
2) is there a bug in Reaper with the cc messages after a SysEx message (it shuffle items) as shown in the start of my midi file?
3) is there a solution to do it in my way using "Single channel items on multiple tracks" without the SysEx problem?
4) or is it a bug in Reaper that in case of a multi channel midi file and using the "Single channel items on multiple tracks" option kills all SysEx events?

[EvilDragon]

Can you try importing your MIDI as one multichannel item and see if that retains the sysex for you? It is not a problem to explode the item onto one track for each channel after you load it this way.

[John1900]

Yes, in that case the SysEx messages are correct but the CC messages Non-Reg Parm LSB, NRP MSB and the Data Entry MSB are shuffled. See picture https://dl.dropboxusercontent.com/u/...%20file%29.jpg. The start is different comparing the Anvil Studio picture and the MSP picture, attached in my first topic and near event 42/43 the MSB (99) parameter is not in the exported midi file.
The exported midi file is in the attachment of my topic above your topic.
This file is not usable for me because this shuffling.

Question (I'm a newbee): how do I explode the item onto one track for each channel after I load the file in "Multichannel item on a single track" mode. I cannot find an option in the main window or editor window.

[EvilDragon]

Well - the problem is that all the NRPN data has the same position in the MIDI file. That's pretty non-deterministic. Try to move each set of NRPN LSB, MSB and CC6 by one tick and you should be just fine.

Search Reaper's Action list for the word "explode", you'll find it.

[DarkStar]

Originally Posted by John1900

Looking to the difference between your latest picture on the forum and my piano roll editor, your picture contains much more notes (events). After some experiences I discover that you insert my 4 channel midi file with the option "Multichannel item on a single track" while I do the other option "Single channel items on multiple tracks".
In your case you have SysEx commands in the Reaper editor, in my case not.

You're right - that looks like a Bug in the Import - perhaps, Reaper should put those MIDI messages not on a channel into an additional track? (The same goes for the "Explode multichannel item ...") I'll raise it in the Issue Tracker.

Originally Posted by John1900

I have chosen the "Single channel items on multiple tracks" option because the Hauptwerk organ simulator has more channels; each channel is for a organ manual or pedal. You can play on a manual and pedal and a solo voice on an other manual. In my case there are 4 manuals.

In case with each manual on his own channel in Reaper, it's easier for me to see and to edit notes on the different organ manuals comparing with the score.

Understood. This might help: load the MIDI as a Multichannel clip on a single track, open it in the MIDI Editor and select the Colour mode to be 'Channel' and use the Channel selector to select the channel to display. As in this screenshot (Using a modified JanneHybrid theme and my own toolbar for Channel Management, available here: http://forum.cockos.com/showthread.php?t=128632 ):

... Big pic: https://i.imgur.com/zA3MsIg.png

Originally Posted by John1900

I continue the test with inserting the midi file with the "Multichannel item on a single track".
I have exported this "single track" midi file (see attachment) and test this with my organ. It doesn't work correctly. I discover that Reaper shuffles the Non-Reg Parm LSB, NRP MSB and the Data Entry MSB.
See picture https://dl.dropboxusercontent.com/u/...%20file%29.jpg. The start is different comparing the pictures, attached in my first topic and near event 42/43 the MSB (99) parameter is not in the exported midi file.

I don't know where those CC messages are going -they are present in the MIDI file here. Here is your screenshot alongside mine:

... Big pic: https://i.imgur.com/NFqsJZ0.png

I do notice that your position precision is different (don't know how that happens.

Originally Posted by John1900

Questions:
1) is my way of handling the midi file wrong?

No, it should work.

Originally Posted by John1900

2) is there a bug in Reaper with the cc messages after a SysEx message (it shuffle items) as shown in the start of my midi file?

Cannot reproduce that here.

Originally Posted by John1900

3) is there a solution to do it in my way using "Single channel items on multiple tracks" without the SysEx problem?

See my idea above.

Originally Posted by John1900

4) or is it a bug in Reaper that in case of a multi channel midi file and using the "Single channel items on multiple tracks" option kills all SysEx events?

Yes, in my opinion.

[DarkStar]

Now in the Issue Tracker:

http://forum.cockos.com/project.php?issueid=4940

As suggested above: perhaps, Reaper should put those MIDI messages that are not on a channel into an additional track. (And the same goes for the "Explode multichannel item ...".)

[John1900]

Originally Posted by EvilDragon

Well - the problem is that all the NRPN data has the same position in the MIDI file. That's pretty non-deterministic. Try to move each set of NRPN LSB, MSB and CC6 by one tick and you should be just fine.

Hello EvilDragon

I know that all NRPN are at the same time, but I cannot change this because it's generated by my organ simulator. I can edit it, but need an good midi editor (Reaper).
But the strange thing is that when importing the same (!!) 4 channel midi file in a "Single channel items on multiple tracks" mode all NRPN are shown but all SysEx commands are gone (not imported). In my opinion and also DarkStar's it's a Reaper import midi file bug.

John1900

[John1900]

Hello DarkStar

Thanks for the extended reply.

I will try your "toolbar for Channel Management" (it look nice) but need some extra info (I'm a newbee, so sorry for that)
In which subdirectory must the "ME_Channels_DS.ReaperMenu" file?
How do I activate it?

Originally Posted by DarkStar

I don't know where those CC messages are going -they are present in the MIDI file here. Here is your screenshot alongside mine:

... Big pic: https://i.imgur.com/NFqsJZ0.png

For your information: all the NRPN pairs (LSB, MSB) are unique and are codes for the stops of my organ. The date (0 or 127) are signals for activate and deactivate that stop.

That means that 68, 17, 127 (first NRPN pair in the exported midi file - left part of your screenshot alongside mine) is not a vaild combination for a stop. The same for others. All NRP data is shuffled and also the list is shorter that the list you have.
So, when a midi file is exported after editting, the NRP data is shuffled and is not a complete list regarding the contents of the eventlist in Reaper midi editor.
So, in my opinion, also the midi export module contains a bug.
That is what I mean in my question 2).

Maybe you can take a look at it and reproduce it.
If so, I cannot use your toolbar because the export contains a bug. But I will try your toolbar so I can use it if the bugs are solved.

John1900

[DarkStar]

Installing a toolbar:
Mine are in a sub-folder in the AppData folder (click [Options] >> 'Show Reaper resource path ...' to get to that folder.
C:\Users\DarkStar\AppData\Roaming\REAPER\MenuSets

• click [Options] > 'Customize menus/toolbars'
• for safety, click [Export] >> ' Export all menus/toolbar ...'
• browse to your MenuSets folder, type in a name and click [Save],
• in the top drop-down list, select 'Floating MIDI toolbar 1' (I presume that you have not used this for anything so far)
• click [Import]
• browse to your MenuSets folder and select the toolbar,
• back in the Customize menus/toolbars window, click [Retilte] and name that toolbar,
• click [Save]

To display it, right-click on an empty part of the MIDI Editor (or Main) toolbar and select 'Open toolbar' then the toolbar you want.

MIDI File Export:
The file that you uploaded is the file you Exported from the Hauptwerk organ simulator, isn't it?. All the triplets of CC messages in the uploaded MIDI file are present here and match your MSP screenshot.

Can you Export the MIDI from Reaper and upload that?

I am using Reaper v4.33. What version are you using?

[John1900]

Originally Posted by EvilDragon

Search Reaper's Action list for the word "explode", you'll find it.

Thanks for the keyword, so I can search for it.
Result: explode is not possible in Midi situation; see the picture
https://dl.dropboxusercontent.com/u/...plode-midi.jpg

[EvilDragon]

Wrong. It's totally possible:




You just can't do it on an EMPTY MIDI item. You have to have channels in it, you know. And you're also looking in a wrong place. This is not a take-related action. It's an item related action, so go in "Item processing" submenu there.

[DarkStar]

^^^^^
Exploding has the same problem: the non-channel MIDI messages are not exploded.

You could Explode the clip, then unMute the original and remove all the MIDI notes and CC messages from it, leaving only the non-channel MIDI messages.

[John1900]

hello DarkStar,

Thanks for the toolbar descriptions; I will give it a try and let you know.

Originally Posted by DarkStar

MIDI File Export:
The file that you uploaded is the file you Exported from the Hauptwerk organ simulator, isn't it?. All the triplets of CC messages in the uploaded MIDI file are present here and match your MSP screenshot.

Can you Export the MIDI from Reaper and upload that?

I am using Reaper v4.33. What version are you using?

The file in the 2nd topic above (Reaper - Doesburg Surr.zip) is the original 4 channel midi file from my Hauptwerk organ simulator.
The "midi_export (single).zip", posted above is the export of Reaper after an insert of a midi file in "Multichannel item on a single track" mode. In this case the Sysex messages are correct but the NRPN events are shuffled and incomplete.
I have attached on this topic "midi_export (multi track).zip"; this file is an export of Reaper after inserting the midi file in "Single channel items on multiple tracks" mode. In this case all SysEx messages are gone, but the NRPN events are complete as far I can see (I cannot test it with my organ simulator because the missing SysEx message.

I'm using the latest version in evaluation mode and portable. Last 2 weeks I have tested a lot of other (free and trial) midi editors, and Reaper is the best one for editing. Now bugsolving ... If it works I will buy Reaper.

[DarkStar]

Ok, I looked at "Reaper - Doesburg Surr.mid" and "Reaper - Doesburg Surr - midi_export (single).mid"

And, yes, the second one only has the alternate messages form the first.

So, I Exported the first one to a new MIDI file and loaded it back in again - it has all the CC messages. I'm stumped.

----------------------------
Did you try the "way of working" that I suggested in reply#10 ?

[John1900]

Are ALL CC messages in the file?
As written above I have found that the CC messages are incomplete and shuffled. Can you take a look at it of do you mean that you have a situation after import ans export that the file is complete (incl. all SysEx en CC messages)

About your question: Did you try the "way of working" that I suggested in reply#10 ? load the MIDI as a Multichannel clip on a single track, open it in the MIDI Editor and select the Colour mode to be 'Channel' and use the Channel selector to select the channel to display.
Yes I have done but not exported when it was coloured (I thought that it only was a way of presentation). Is there a difference for the export?


I have expoded the single track midi file as suggested by you and EvilDragon. All notes and CC events are deleted in track 1.
I have just created a new export file and have to start my organ simulator for a new test (about 20 minutes start time).

[DarkStar]

This is what I see In Reaper / MIDI Editor Event List:

• Reaper - Doesburg Surr.mid - 3735 MIDI messages
• Reaper - Doesburg Surr - midi_export (single).mid - 3696 messages

  (dropped some CC messages)

• Reaper - Doesburg Surr [chan 1] - midi_export (multi track).mid - 3670 messages

  (no Sysex messages)

• original_exported_and_reimported.mid - 3735 messages

I suggested that method in reply #10 as you may find that a reasonable way of working. You can work on any of the channels. Yes, the Colour: channel and Channel: number only affect the display.

[John1900]

Originally Posted by DarkStar

[*]Reaper - Doesburg Surr - midi_export (single).mid - 3696 messages

(dropped some CC messages)

Not only dropped some CC messages but also shuffles the contents of the messages (de LSB/MSB/Data unique combinations are not the same of the original file)

My conclusion is that there are 2 bugs in the Midi part of Reaper in case of a multi channel midi file:
1)
insert a midi file in "Single channel items on multiple tracks" mode, the SysEx messages are dropped
2)
export a midi file, which is imported in "Multichannel item on a single track" mode, not all CC messages are in the exported midi file and also the content of the CC messages is shuffled

Originally Posted by DarkStar

original_exported_and_reimported.mid - 3735 messages

What do you mean by original exported and reimported? By importing of the midi file SysEx messages are dropped or the CC messages are incomplete and shuffled; both depending of methode of import.
How is it possible that you have the 3735 MIDI messages in this case, the same as in the original file?

Originally Posted by DarkStar

I suggested that method in reply #10 as you may find that a reasonable way of working.

I have found that the following method works.
1) insert a midi file in "Single channel items on multiple tracks" mode
2) explode this file in multi track
3) unmute track 0
4) remove in track 0 alle notes and all CC messages (remains only the SysEx messages)
5) after editting export to a midi file with options "Multitrack midi file" and "all Midi items"

[DarkStar]

Originally Posted by John1900

Not only dropped some CC messages but also shuffles the contents of the messages (de LSB/MSB/Data unique combinations are not the same of the original file)

I do not think they are shuffled. In this screenshot, the marked message is the left-hand file are those preset in the right-hand file. The others are missing.
... Big pic: https://i.imgur.com/nxp88uP.png

Originally Posted by John1900

What do you mean by original exported and reimported? By importing of the midi file SysEx messages are dropped or the CC messages are incomplete and shuffled; both depending of method of import.

How is it possible that you have the 3735 MIDI messages in this case, the same as in the original file?

I loaded "Reaper - Doesburg Surr.mid" as a multichannel item on a single track (so, nothing lost). I Exported it, giving the exported file the name "original_exported_and_reimported.mid". I loaded that file as a multichannel item on a single track . It also has 3735 messages - nothing at all lost.

[John1900]

About shuffling: your opinion is correct.

About "original exported and reimported": understood.

In my opinion this case can be closed. For the time being I have a workaround. Will you add the second bug to the tracker?

I thank you for all the time, spend in my problem.
I have learned a lot new things.

John1900

[schwa]

Regarding what to do with non-channel MIDI messages when importing by channel, or exploding by channel: creating a new track seems reasonable, but it could cause some unexpected results. For example, in many cases it would create a new track that contains only the MIDI meta-message for the track name.

I'm not clear on exactly what the "2nd bug" above is. Could I get a concise explanation please?

[John1900]

I have imported a 4 channel midi file in "Multichannel item on a single track" mode in Reaper.

After the export of this file not all CC messages are in the exported midi file. DarkStar has made an overview. In the right part of the picture https://dl.dropboxusercontent.com/u/...20messages.png (the exported midifile) you will see the pairs of CC messages. In the left picture (the original midi file) you see the according cc messages.

So, CC not all messages are exported in this case (import a 4 channel midi file in "Multichannel item on a single track" mode)

[EvilDragon]

Originally Posted by schwa

Regarding what to do with non-channel MIDI messages when importing by channel, or exploding by channel: creating a new track seems reasonable, but it could cause some unexpected results. For example, in many cases it would create a new track that contains only the MIDI meta-message for the track name.

How about an additional dialog:

"x meta events found in the MIDI file. Explode onto an additional track?"

[DarkStar]

Originally Posted by schwa

Regarding what to do with non-channel MIDI messages when importing by channel, or exploding by channel: creating a new track seems reasonable, but it could cause some unexpected results. For example, in many cases it would create a new track that contains only the MIDI meta-message for the track name.

I'd go with that - no need for an additional dialogue, imo.

The second bug is the loss of alternate (NRPN?) CC messages, but I cannot reproduce that here. But I'm using v4.33, so it may be in a more recent version.

[schwa]

Originally Posted by John1900

So, CC not all messages are exported in this case (import a 4 channel midi file in "Multichannel item on a single track" mode)

The problem is actually on reimporting the exported file I think, the exported .mid file does appear to contain the complete NRPN dump.

It also appears that recent versions of REAPER did reimport the file correctly, so hopefully this won't be too complicated to figure out.

[John1900]

Originally Posted by DarkStar

The second bug is the loss of alternate (NRPN?) CC messages, but I cannot reproduce that here. But I'm using v4.33, so it may be in a more recent version.

I have tested this with version 4.52

[John1900]

Originally Posted by schwa

The problem is actually on reimporting the exported file I think, the exported .mid file does appear to contain the complete NRPN dump.

It also appears that recent versions of REAPER did reimport the file correctly, so hopefully this won't be too complicated to figure out.

I always have imported the same original file in the test, which is attached in topic #2 above.
So, it is no reimported file

[schwa]

I believe the exported file does contain all of the NRPN data. When the exported file is then reimported into REAPER, not all of the data appears. In other words, the right hand side of this image does not show all of the events that are actually in the file:
https://dl.dropboxusercontent.com/u/...20messages.png

If I look at the raw MIDI data in the exported .mid file, not using REAPER, I do see all of the NRPN events.

[John1900]

Hello schwa

Sorry, I don't agree with your opinion.
Therefor I have imported the "midi_export (single).mid" file in Anvil Studio (another midi editor but less powerfull as Reaper)
This screen looks the same as the Reaper screen shot. See
https://dl.dropboxusercontent.com/u/...%20file%29.jpg

I have also tested the "midi_export (single).mid" file in my organ simulator program Hauptwerk. It doesn't work.

For your information: all the NRPN pairs (LSB, MSB) are unique and are codes for the stops of my organ. The date (0 or 127) are signals for activate and deactivate that stop.

That means that 68, 17, 127 (first NRPN pair in the exported midi file) is not a vaild combination for a stop. The same for others. Some NRP data is lost during the export in Reaper. That is the reason that the exported file doesn't work in the Hauptwerk program.

[schwa]

Sorry, I wasn't understanding this properly. The bug is fixed for the next build, thanks for the report.

[John1900]

Hello schwa

Thanks. Solving it should be fine for me

[schwa]

This should be fixed in the latest prerelease build:
http://forum.cockos.com/showthread.php?t=129003