Windows Compiler Benchmarking Test: NOW in the 2018 Edition!

[Justin]

I've updated the benchmarking page here:

http://landoleet.org/bench/

There are new builds of 5.11 5.71pre12a with VS2013 to compare against the older (VC + ICC) builds.

These versions should work on Windows XP SP2 and later, but require SSE2 (which I can't imagine anyone lacks anymore).

If you would like to help give us valuable feedback, the following information will be most helpful with the results:

• CPU type (AMD or Intel, approximate kind e.g. "Sandy Bridge")
• OS version and bit-ness (e.g. Windows 7 64-bit)
• REAPER architecture used (win32 or win64) -- ideally if you can, please test both versions of each architecture and give us results
• Audio device block size and samplerate
• How many tracks were in the project tested, how many plug-ins (approximately), how many non-Cockos plug-ins.
• Anything else interesting or unusual about your project (routing, folders, compressed media, etc)

Try loading your project, playing it a bit, and letting everything settle before taking a measurement. When you do take a measurement, try to make it over a reasonably long time (such as 30 seconds), and try to make sure that your sample is taken from the same part of the project in each test.

Note also: there's no need for multiple users to test the same project. The purpose of these tests is as much about reporting results on different projects as it is about different CPUs -- diversity in both is important.


Your feedback would be much appreciated, thank you in advance!

[Justin]

I'll go first!

iMac C2D @ 2.93ghz, Win7 64-bit
WASAPI exclusive, 256 sample buffer, 48khz
Anticipative FX on @ 500ms (oops, left that up from an earlier test), Live FX off, old 4.x worker thread left on (oops again)
angry mob 29.1 to 41.1 (about 40sec) -- synthetic edit with extra FX
48khz, all 44k mono media, mix of 32-bit FP and 24 bit integer .wav "Good (192 pt sinc)" resample mode
9 2-channel tracks, 1 folder, bunch of reaxcomp, reacomp, reafir, and reaeq (synthetic) and a couple of JS, a reaverb (LL/ZL), a readelay.
No third party plug-ins.

3 separate samplings for each version

REAPER 64-bit: ICC 22.8%,21.67%,22.6%, VS13 21.1%, 24.1%, 20.6%.

I'd call these about the same (the 24% in the VS13 was probably another application doing something)

REAPER 32-bit: similar results, though each averages closer to 23.5% CPU, so slightly higher than 64-bit

Conclusion: VS13 performs about equivalent to ICC for this project on this system

Another test:

Macbook Pro i7-3720QM @ 2.60
Windows 10 64-bit
WASAPI exclusive 256 samples

Duplicated the tracks twice (4x track count, 224 FX total), the difference between compilers is really negligible on this system as well (there is as much variation in tests with the same version as is between different versions).

[xpander]

Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x64
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.


VS2005 & ICC x64



VS Express 2013 x64


Slightly smaller RT CPU and RAM use with VS Express, certainly nothing noticeable in project play.
Later maybe the 32-bit versions...

[evolution]

Intel Xeon X5690 (Westmere EP, 6 cores/12 threads) @ 4.32GHz
Windows 10 64bit
Lynx ASIO, 512 samples buffer, 44.1kHz

512 stereo tracks (44.1kHz 24bit stereo WAV)
1024 plugins (ReaEQ & ReaComp on each track)

No folders or special routings.

Average CPU use over the first 3 minutes:

ICC 64bit: 51.6%
VS13 64bit: 50.7%
ICC 32bit: 53.0%
VS13 32bit: 51.2%

The VS13 builds appear to perform slightly better than the ICC builds, and the 64bit builds slightly better than the 32bit builds.

[xpander]

The 32-bit Reaper versions, system and project the same as with earlier x64 tests.

Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x32 versions
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.

VC6 + ICC 10.0.027 x32


Visual Studio Express 2013 x32


Again VS Express 2013 seems to use just slightly less CPU/RT CPU. No project performance difference noticed.

[Justin]

The RT CPU indicator might not be the best indicator, since a ton of processing gets offloaded to worker threads... maybe try it with anticipative FX disabled, too?

Looking forward to seeing some AMD tests. I'm going to go test on my Atom netbook soon as well.

[xpander]

Ok, here are the test results with everything else being the same but with no anticipative FX. Due to big performance difference I had to up the sample buffer from 32 to 1024 to keep clean playback though. Lower buffers produced progressively worse dropouts and glitches.

---
Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x64 versions
ASIO Hammerfall DSP, 1024 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.

VS2005 & ICC x64 no anticipative FX


VS Express 2013 x64 no anticipative FX



Later maybe 32-bit versions...but the performance differences seem about as small as previously?

[Tranquil]

Originally Posted by Justin

Looking forward to seeing some AMD tests. I'm going to go test on my Atom netbook soon as well.

You asked for some AMD tests so it was the perfect opportunity to flirt with my old machine.

AMD Phenom II X4 @ 3 GHz
Windows 7 64-bit
RAM 10 GB (Unganged)
WASAPI 256 samples, 48kHz (Exclusive)
(testing: funny days from Lonely_7_11)

win32-icc: avg: 6.60% (range: 2-32.5%)
win32-vs13: avg: 9.60% (range: 2-24.2%)

win64-icc: avg 7.90% (range: 1.6-21%)
win64-vs13: avg 11.78% (range: 3.1-22.2%)

Justin, I guess those builds don't include the bool replacement I asked as favor in a private email last month or so. If such a test is meaningless I fully understand by the way.

[d. gauss]

Originally Posted by Justin

Another test:

Macbook Pro i7-3720QM @ 2.60
Windows 10 64-bit
WASAPI exclusive 256 samples

i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.

[innuendo]

Originally Posted by d. gauss

i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.

Check out the "bootcamp assistant" software. If you have questions, I'd suggest you to create a new thread.

[Justin]

Originally Posted by d. gauss

i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.

Yeah I have my macbook pro (mid 2012) dualbooting Win10/OSX. Works pretty nicely, though I don't use anything thunderbolt.

[Justin]

Testing on my ASUS T100 (Atom quad core), Win 10 32-bit.

20 tracks, bunch of reaxcomp, reaeq, reafir, readelay, reacomp, resampling from 44khz to 48khz (medium 64pt sinc).

VS13 version: 41% avg
ICC version: 35% avg

ICC is a pretty good win on this (low-end) netbook/tablet.

[timothys_monster]

Intel Core i3 4030U (Haswell ULT) @1.9GHz
Windows 7, 64 Bit
OnBoard Soundcard with ASIO4ALL @ 512 samples 44.100 Hz
30 tracks
112 plugins all REAPER plugins (no JS though)

No sends or folders, 12 MIDI tracks and 18 audio tracks, stretchmarkers used

REAPER x32 release:
79.1% 88.6% 76.7%

REAPER x32 VS2013:
85.2% 82.0% 76.5%

REAPER x64 release:
54.1% 61.9% 69.8%

REAPER x64 VS 2013:
63.1% 64.8% 53.7%

[srdmusic]

Originally Posted by Justin

I've updated the benchmarking page here:

http://landoleet.org/bench/

There are new builds of 5.11 with VS2013 to compare against the older (VC + ICC) builds.

Hi Justin,

Thank you very much for pushing the envelope and finding an even more efficient way of processing audio.

I've done some fairly extensive testing between the versions and I have detailed findings for you.

To sum up briefly, I'm seeing a 10-15% decrease in system resources in the 5.11 VS2013 compared to the latest pre-release 5.33 on larger sessions(850 tracks). Congrats on that as it's a huge improvement. This is btw, about a 50% decrease in processing compared to Steinberg's Cubase with the same template. I'm completely blow away by how much lower I can have my buffer settings in Reaper. THANK YOU!

There is also a very significant improvement in video processing. There is however, some visual freezing glitches on the VS2013 version, that I believe have something to do with my auto save preferences. This 5.11 version essentially solved my video processing issues, but if we can narrow down the freezing visual glitches that would help out a lot. Is there a way to recompile the latest pre-release in VS2013 so that I can test apples to apples? Just want to make sure it's not a newer preference causing the difference on the video side.

In addition, I'm wondering if it is possible to speak over the phone or via a Teamviewer session. It would be much easier to show you my findings, specs and processing preferences, then to try and typing them all out. I totally understand you'd rather continue through the forum but if that is possible I can direct message or email me for my contact info.

[Justin]

For this to be useful, can you test against the 5.11 release version as well?

Also what CPU are you using?

[srdmusic]

Will do.


Here's my Computer Specs:

Windows 10 Enterprise (latest Build)
2 x Intel Xeon E52687 v4 3.0Ghz 12 Core (totalling 24 Cores or 48 with hyper threading turned on)
128 GB of RAM
AMD W7100 FirePro Graphics Card
1 x 500GB SSD for the OS
7TB Hardware Raided SSDs for Samples
3 x 500GB Software Raid SSDs for the Project and Picture Drive

[Justin]

Let's test this again, now with 5.71pre12a!

https://landoleet.org/bench/ updated.

[Edgemeal]

FWIW, these two files are also posted in "https://www.landoleet.org/",
reaper571pre12a-vs13-install.exe 2018-01-15 00:10
reaper571pre12a-vs13_x64-install.exe 2018-01-15 00:10

Not sure if they belong there?

[BlackBart]

System:
Dual Xeon i5 2687W-v4
Supermicro MB
128G RAM
Win10 Pro/Reaper 64/Buffer 32

Lots of 3rd party plugins:
Vienna Ensemble
Kontakt
Tone 2 Synths, Diva, Blue 2
Melda, Hofa


Essentially no difference. I played the same passage several times on both versions, at no time were the more than 1% apart at the same locations.

[Justin]

Originally Posted by Edgemeal

FWIW, these two files are also posted in "https://www.landoleet.org/",
reaper571pre12a-vs13-install.exe 2018-01-15 00:10
reaper571pre12a-vs13_x64-install.exe 2018-01-15 00:10

Not sure if they belong there?

Yeah they're fine to be there too.

[Janne83]

Ive tested on one of my normal work sessions on my laptop:

Session about 120 tracks
native vst
VST plugins (Eventide,TDR,Melodyne,...)
vsti (Kontakt,u-he Hive,..)

HP Elitebook
i7 3630QM
12gb RAM
Win 10 1709 64 bit

results:
ICC:



VS2013:




Regards J

[lachrimae]

• i7-5820k Haswell
• Win10 Pro 1709 x64
• Tested Reaper x64 only (for now)
• RME UFX: 128 Samples 48k
• 5 Tracks, 19 FX (0 Cockos): Bazille, Battery 4, Absynth, Izotope, -
• Arturia, Hornet, Stillwell, Melda, Waves
• All MIDI only

VS05


VS13


Got this error when launching VS13 and had to restart Reaper:


I can make this project larger, or with Cockos FX if it would be useful.

[xpander]

Intel i7 4770k @ 3.5GHz
Windows 10 Home (1607) 64-bit
ASIO Hammerfall DSP, 1024 samples, 44.1kHz
Tracks: 80 2-ch. audio tracks, all files 96kHz stereo WAV so they were resampled in project.
Plugins: 240 instances (ReaEQ, ReaComp and ReaFir on every track), no anticipative FX processing.
Other notes: All tracks straight to master. All pics taken after 1 minute of playback. Playback was just about on threshold of clean with both Win32 versions, Win64 versions were cleaner.



edit:better test run

[Tale]

From this post, 17 tracks (1x white noise, 8x ReaComp + ReaEQ, 8x ReaFIR):

Intel Core i7-7700 3.60 GHz, Win10 Pro x64, Arturia AudioFuse (44.1 kHz, 128 samples)

reaper571pre12a-install.exe: 1.76% avg
reaper571pre12a-vs13-install.exe: 2.06% avg
reaper571pre12a_x64-install.exe: 1.95% avg
reaper571pre12a-vs13_x64-install.exe 2.25% avg

Intel Core i5-5257U 2.70 GHz, macOS 10.13.2, built-in audio (44.1 kHz, 512 samples)

reaper571pre12a_i386.dmg: 9.81% avg
reaper571pre12a-clang_i386.dmg: 11.34% avg
reaper571pre12a_x86_64.dmg: 11.00%
reaper571pre12a-clang_x86_64.dmg: 10.76%

[xpander]

Fwiw, new test runs, this time with a test project from the DAWBench site.
http://www.dawbench.com/benchmarks.htm

I used the Dawbench-DSP-R5-RXC-EXT project, had my RME HDSP 9632 at 44.1 kHz and the smallest sample block 32. Then I activated FX instances up to a point where dropouts started to happen at playback, which was 200 instances (and 120 bypassed). Like earlier, no anticipative FX processing.

Intel i7 4770k @ 3.5GHz
Windows 10 Home (1607) 64-bit
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 5 audio tracks, all files 44.1 kHz WAV, 40 FX tracks with 320 instances of ReaComp (200 active, 120 bypassed). No anticipative FX processing.
Other notes: All pics taken after 1 minute of playback.




---
Just for fun, the same test project run in v5.71pre13 with the same settings, except the anticipative FX processing set ON. I could've gone even higher, but stopped at 1800 active FX instances.

[HighVoltage]

Originally Posted by xpander

Fwiw, new test runs, this time with a test project from the DAWBench site.
http://www.dawbench.com/benchmarks.htm

Intel i7 4770k @ 3.5GHz
Windows 10 Home (1607) 64-bit
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 5 audio tracks, all files 44.1 kHz WAV, 40 FX tracks with 320 instances of ReaComp (200 active, 120 bypassed). No anticipative FX processing.

Dude, seriously WTF!!!!

I have the exact same configuration, except my CPU is 4790k, which should be an upgraded 4770k. I'm running at overclocked ~4.2Ghz.

At 32 samples, i can get 56 instances of ReaXcomp before stuttering.
Even at 512 samples, i get only 132 instances with anticipative OFF.
Maximum instances wint anticipative ON, is around 176, and then all 4 cores are at 100%, and my mouse is barely moving (no audio droputs, huge kudos to Reaper team)

Something is very off with your test, or i'm gonna just throw my pc out of the window.
Are you sure you have loaded the Dawbench-DSP-R5-RXC-EXT file?
(Well even with the non EXT version, i get 318 fx @256)

I'm curious though, what Motherboard you use?

[osxmidi]

What about the Meltdown and Spectre patches, do they have any performance effect.

[HighVoltage]

Originally Posted by osxmidi

What about the Meltdown and Spectre patches, do they have any performance effect.

http://www.scanproaudio.info/2018/01...-workstations/

[fixerdave]

hi, might be a daft question but do these different builds produce different code but produce the same functionality from a common source code?? so does one produce tighter more efficient code? dave

[EvilDragon]

Yes, it's all about compiler optimizations.

[xpander]

Originally Posted by HighVoltage

Something is very off with your test, or i'm gonna just throw my pc out of the window.
Are you sure you have loaded the Dawbench-DSP-R5-RXC-EXT file?
(Well even with the non EXT version, i get 318 fx @256)

I'm curious though, what Motherboard you use?

Please just don't throw away your PC yet, it's fine.

Yes, I did use that benchmark project, but I was obviously not too clear about the FX I actually used. It was the regular ReaComp, not the heavier standalone ReaXcomp like the original project had. Reason for that was that I changed it already earlier for some of my personal tests to make it more like real life use I have here. Since the tests in this topic are not about benchmarking the computer hw and referring the results to others, I didn't bother to change it all back to what the original benchmark project had. Not my intention to confuse with the specs, sorry about that.

But for your peace of mind I did run the same project with standalone ReaXcomp, though only using the current Reaper release. With that, I can have 36 instances without dropouts and noise, about 40 with occasional glitch. Strength of Reaper, I can activate up 120 instances and while the noise is certainly not acceptable at all, Reaper still responds to everything just fine without freezing...RT CPU at 96% at that point.

FYI, my motherboard is AsRock Z87 Extreme4. There have been some PC hardware benchmark topics around, maybe we can continue in one of those or make a new one for that if needed?

[HighVoltage]

Originally Posted by xpander

Please just don't throw away your PC yet, it's fine.

Yes, I did use that benchmark project, but I was obviously not too clear about the FX I actually used. It was the regular ReaComp, not the heavier standalone ReaXcomp like the original project had. Reason for that was that I changed it already earlier for some of my personal tests to make it more like real life use I have here. Since the tests in this topic are not about benchmarking the computer hw and referring the results to others, I didn't bother to change it all back to what the original benchmark project had. Not my intention to confuse with the specs, sorry about that.

But for your peace of mind I did run the same project with standalone ReaXcomp, though only using the current Reaper release. With that, I can have 36 instances without dropouts and noise, about 40 with occasional glitch. Strength of Reaper, I can activate up 120 instances and while the noise is certainly not acceptable at all, Reaper still responds to everything just fine without freezing...RT CPU at 96% at that point.

FYI, my motherboard is AsRock Z87 Extreme4. There have been some PC hardware benchmark topics around, maybe we can continue in one of those or make a new one for that if needed?

Thanks for the clarification!

To be somewhat useful to this thread, i did not notice any difference in performance between the versions. I made some very heavy stress tests, and it both performed the same.

4790k
Win10, Reaper x64
Buffer sizes from 32-1024

[vitalker]

Originally Posted by HighVoltage

Win10, Reaper x64
Buffer sizes from 32-1024

Ok, but did you try it with your REAL projects, not synthetic
ones?

[HighVoltage]

Originally Posted by vitalker

Ok, but did you try it with your REAL projects, not synthetic
ones?

Will do.
Edit:

Just tested my current project, pretty same, if anything, the vs13 version seems to be a tad slower. Around ~0.3%

[vitalker]

@Justin, do you still need testers?