Old 01-04-2016, 05:26 PM   #1
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default Windows Compiler Benchmarking Test: 2016 Edition!

I've updated the benchmarking page here:

http://landoleet.org/bench/

There are new builds of 5.11 with VS2013 to compare against the older (VC + ICC) builds.

These versions should work on Windows XP and later, but require SSE2 (which I can't imagine anyone lacks anymore).

Please test these with moderate-to-heavy projects, ideally ones that use a significant number of Cockos plug-ins. If you would like to help give us valuable feedback, the following information will be most helpful with the results:
  • CPU type (AMD or Intel, approximate kind e.g. "Sandy Bridge")
  • OS version and bit-ness (e.g. Windows 7 64-bit)
  • REAPER architecture used (win32 or win64) -- ideally if you can, please test both versions of each architecture and give us results
  • Audio device block size and samplerate
  • How many tracks were in the project tested, how many plug-ins (approximately), how many non-Cockos plug-ins.
  • Anything else interesting or unusual about your project (routing, folders, compressed media, etc)

Try loading your project, playing it a bit, and letting everything settle before taking a measurement. When you do take a measurement, try to make it over a reasonably long time (such as 30 seconds), and try to make sure that your sample is taken from the same part of the project in each test.

Note also: there's no need for multiple users to test the same project. The purpose of these tests is as much about reporting results on different projects as it is about different CPUs -- diversity in both is important.


Your feedback would be much appreciated, thank you in advance!
Justin is offline   Reply With Quote
Old 01-04-2016, 07:30 PM   #2
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default

I'll go first!

iMac C2D @ 2.93ghz, Win7 64-bit
WASAPI exclusive, 256 sample buffer, 48khz
Anticipative FX on @ 500ms (oops, left that up from an earlier test), Live FX off, old 4.x worker thread left on (oops again)
angry mob 29.1 to 41.1 (about 40sec) -- synthetic edit with extra FX
48khz, all 44k mono media, mix of 32-bit FP and 24 bit integer .wav "Good (192 pt sinc)" resample mode
9 2-channel tracks, 1 folder, bunch of reaxcomp, reacomp, reafir, and reaeq (synthetic) and a couple of JS, a reaverb (LL/ZL), a readelay.
No third party plug-ins.

3 separate samplings for each version

REAPER 64-bit: ICC 22.8%,21.67%,22.6%, VS13 21.1%, 24.1%, 20.6%.

I'd call these about the same (the 24% in the VS13 was probably another application doing something)

REAPER 32-bit: similar results, though each averages closer to 23.5% CPU, so slightly higher than 64-bit

Conclusion: VS13 performs about equivalent to ICC for this project on this system

Another test:

Macbook Pro i7-3720QM @ 2.60
Windows 10 64-bit
WASAPI exclusive 256 samples

Duplicated the tracks twice (4x track count, 224 FX total), the difference between compilers is really negligible on this system as well (there is as much variation in tests with the same version as is between different versions).

Last edited by Justin; 01-04-2016 at 08:02 PM.
Justin is offline   Reply With Quote
Old 01-04-2016, 10:00 PM   #3
xpander
Human being with feelings
 
xpander's Avatar
 
Join Date: Jun 2007
Location: Terra incognita
Posts: 3,669
Default

Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x64
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.


VS2005 & ICC x64




VS Express 2013 x64



Slightly smaller RT CPU and RAM use with VS Express, certainly nothing noticeable in project play.
Later maybe the 32-bit versions...

Last edited by xpander; 01-04-2016 at 10:08 PM.
xpander is offline   Reply With Quote
Old 01-05-2016, 08:04 AM   #4
evolution
Human being with feelings
 
Join Date: May 2009
Posts: 18
Default

Intel Xeon X5690 (Westmere EP, 6 cores/12 threads) @ 4.32GHz
Windows 10 64bit
Lynx ASIO, 512 samples buffer, 44.1kHz

512 stereo tracks (44.1kHz 24bit stereo WAV)
1024 plugins (ReaEQ & ReaComp on each track)

No folders or special routings.

Average CPU use over the first 3 minutes:

ICC 64bit: 51.6%
VS13 64bit: 50.7%
ICC 32bit: 53.0%
VS13 32bit: 51.2%

The VS13 builds appear to perform slightly better than the ICC builds, and the 64bit builds slightly better than the 32bit builds.
evolution is offline   Reply With Quote
Old 01-05-2016, 08:41 AM   #5
xpander
Human being with feelings
 
xpander's Avatar
 
Join Date: Jun 2007
Location: Terra incognita
Posts: 3,669
Default

The 32-bit Reaper versions, system and project the same as with earlier x64 tests.

Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x32 versions
ASIO Hammerfall DSP, 32 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.

VC6 + ICC 10.0.027 x32


Visual Studio Express 2013 x32


Again VS Express 2013 seems to use just slightly less CPU/RT CPU. No project performance difference noticed.
xpander is offline   Reply With Quote
Old 01-05-2016, 10:55 AM   #6
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default

The RT CPU indicator might not be the best indicator, since a ton of processing gets offloaded to worker threads... maybe try it with anticipative FX disabled, too?

Looking forward to seeing some AMD tests. I'm going to go test on my Atom netbook soon as well.
Justin is offline   Reply With Quote
Old 01-05-2016, 12:03 PM   #7
xpander
Human being with feelings
 
xpander's Avatar
 
Join Date: Jun 2007
Location: Terra incognita
Posts: 3,669
Default

Ok, here are the test results with everything else being the same but with no anticipative FX. Due to big performance difference I had to up the sample buffer from 32 to 1024 to keep clean playback though. Lower buffers produced progressively worse dropouts and glitches.

---
Intel i7 4770k @ 3.5GHz
Windows 10 64-bit
Reaper x64 versions
ASIO Hammerfall DSP, 1024 samples, 44.1kHz
Tracks: 84 2-ch. audio tracks, all files 44.1kHz stereo WAV
Plugins: 252 instances (ReaEQ, ReaComp and ReaFir on every track)
No 3rd party plugins
Other notes: All tracks straight to master, no folders, no sub-busses.
All measurements taken 40 seconds into project.

VS2005 & ICC x64 no anticipative FX


VS Express 2013 x64 no anticipative FX



Later maybe 32-bit versions...but the performance differences seem about as small as previously?
xpander is offline   Reply With Quote
Old 01-06-2016, 01:04 PM   #8
Tranquil
Human being with feelings
 
Tranquil's Avatar
 
Join Date: Feb 2005
Posts: 88
Default

Quote:
Originally Posted by Justin View Post
Looking forward to seeing some AMD tests. I'm going to go test on my Atom netbook soon as well.
You asked for some AMD tests so it was the perfect opportunity to flirt with my old machine.

AMD Phenom II X4 @ 3 GHz
Windows 7 64-bit
RAM 10 GB (Unganged)
WASAPI 256 samples, 48kHz (Exclusive)
(testing: funny days from Lonely_7_11)

win32-icc: avg: 6.60% (range: 2-32.5%)
win32-vs13: avg: 9.60% (range: 2-24.2%)

win64-icc: avg 7.90% (range: 1.6-21%)
win64-vs13: avg 11.78% (range: 3.1-22.2%)

Justin, I guess those builds don't include the bool replacement I asked as favor in a private email last month or so. If such a test is meaningless I fully understand by the way.
Tranquil is online now   Reply With Quote
Old 01-06-2016, 10:40 PM   #9
d. gauss
Human being with feelings
 
Join Date: May 2006
Posts: 1,347
Default

Quote:
Originally Posted by Justin View Post

Another test:

Macbook Pro i7-3720QM @ 2.60
Windows 10 64-bit
WASAPI exclusive 256 samples
i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.
d. gauss is offline   Reply With Quote
Old 01-07-2016, 08:33 AM   #10
innuendo
Human being with feelings
 
Join Date: Nov 2013
Location: Jerusalem, Israel
Posts: 659
Default

Quote:
Originally Posted by d. gauss View Post
i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.
Check out the "bootcamp assistant" software. If you have questions, I'd suggest you to create a new thread.
innuendo is offline   Reply With Quote
Old 01-07-2016, 12:17 PM   #11
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default

Quote:
Originally Posted by d. gauss View Post
i don't want to hijack (so feel free to delete move this), but i don't know how else this might be seen. justin, are you running reaper on a macbook on windows? vs. mac os? if so, how so? i only ask because i just bought a macbook pro i7 after 100 years of windows because my interface requires thunderbolt and i am trying to transition. thanks in advance.
Yeah I have my macbook pro (mid 2012) dualbooting Win10/OSX. Works pretty nicely, though I don't use anything thunderbolt.
Justin is offline   Reply With Quote
Old 01-07-2016, 12:18 PM   #12
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default

Testing on my ASUS T100 (Atom quad core), Win 10 32-bit.

20 tracks, bunch of reaxcomp, reaeq, reafir, readelay, reacomp, resampling from 44khz to 48khz (medium 64pt sinc).

VS13 version: 41% avg
ICC version: 35% avg

ICC is a pretty good win on this (low-end) netbook/tablet.
Justin is offline   Reply With Quote
Old 01-07-2016, 02:46 PM   #13
timothys_monster
Human being with feelings
 
timothys_monster's Avatar
 
Join Date: Jan 2012
Location: Europe
Posts: 644
Default

Intel Core i3 4030U (Haswell ULT) @1.9GHz
Windows 7, 64 Bit
OnBoard Soundcard with ASIO4ALL @ 512 samples 44.100 Hz
30 tracks
112 plugins all REAPER plugins (no JS though)

No sends or folders, 12 MIDI tracks and 18 audio tracks, stretchmarkers used

REAPER x32 release:
79.1% 88.6% 76.7%

REAPER x32 VS2013:
85.2% 82.0% 76.5%

REAPER x64 release:
54.1% 61.9% 69.8%

REAPER x64 VS 2013:
63.1% 64.8% 53.7%

Last edited by timothys_monster; 01-07-2016 at 04:01 PM.
timothys_monster is offline   Reply With Quote
Old 01-26-2017, 12:03 PM   #14
srdmusic
Human being with feelings
 
Join Date: Dec 2016
Posts: 131
Default

Quote:
Originally Posted by Justin View Post
I've updated the benchmarking page here:

http://landoleet.org/bench/

There are new builds of 5.11 with VS2013 to compare against the older (VC + ICC) builds.
Hi Justin,

Thank you very much for pushing the envelope and finding an even more efficient way of processing audio.

I've done some fairly extensive testing between the versions and I have detailed findings for you.

To sum up briefly, I'm seeing a 10-15% decrease in system resources in the 5.11 VS2013 compared to the latest pre-release 5.33 on larger sessions(850 tracks). Congrats on that as it's a huge improvement. This is btw, about a 50% decrease in processing compared to Steinberg's Cubase with the same template. I'm completely blow away by how much lower I can have my buffer settings in Reaper. THANK YOU!

There is also a very significant improvement in video processing. There is however, some visual freezing glitches on the VS2013 version, that I believe have something to do with my auto save preferences. This 5.11 version essentially solved my video processing issues, but if we can narrow down the freezing visual glitches that would help out a lot. Is there a way to recompile the latest pre-release in VS2013 so that I can test apples to apples? Just want to make sure it's not a newer preference causing the difference on the video side.

In addition, I'm wondering if it is possible to speak over the phone or via a Teamviewer session. It would be much easier to show you my findings, specs and processing preferences, then to try and typing them all out. I totally understand you'd rather continue through the forum but if that is possible I can direct message or email me for my contact info.
srdmusic is offline   Reply With Quote
Old 01-26-2017, 12:49 PM   #15
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 10,047
Default

For this to be useful, can you test against the 5.11 release version as well?

Also what CPU are you using?
Justin is offline   Reply With Quote
Old 01-26-2017, 02:03 PM   #16
srdmusic
Human being with feelings
 
Join Date: Dec 2016
Posts: 131
Default Thanks Justin

Will do.


Here's my Computer Specs:

Windows 10 Enterprise (latest Build)
2 x Intel Xeon E52687 v4 3.0Ghz 12 Core (totalling 24 Cores or 48 with hyper threading turned on)
128 GB of RAM
AMD W7100 FirePro Graphics Card
1 x 500GB SSD for the OS
7TB Hardware Raided SSDs for Samples
3 x 500GB Software Raid SSDs for the Project and Picture Drive
srdmusic is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 03:05 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.