Go Back   Cockos Incorporated Forums > REAPER Forums > Recording Technologies and Techniques

Reply
 
Thread Tools Display Modes
Old 04-06-2014, 07:34 PM   #1
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default New blind-testing audio ABX software

Update: 2.39 released... See end of thread.

Hey Reaps -- I was tired of not being able to recommend any solid, current cross-platform ABX software, so i just wrote my own. I'd appreciate any feedback or ideas on it.

http://lacinato.com/cm/software/othersoft/abx

It's in Java, but you don't have to install java to run it, because you can download packaged files from that link that contain everything it needs (like a Reaper portable install.) Makes the files bigger, but no biggie. (Or just download the .jar, if you're savvy). Just download, unpack the archive, and run the program.

Runs on linux 32/64, windows 32/64, and mac (64bit only).

I'm going to be adding the ability to play certain regions of files at some point.

What other features might you want to see? E.g. playing A/B/C/D/E/F... as X? That'd be useful for shootouts, etc.

I haven't been able to test it on a mac yet, so if you have a 64-bit system please give it a shot. I have a feeling there is a smarter way to package it for the mac. It's currently as a .tgz -- will that unwrap correctly? Does the included executable work as expected? Etc.

Thanks! Enjoy.

[update -- lastest image here:]


Last edited by clepsydrae; 03-16-2018 at 01:32 PM.
clepsydrae is offline   Reply With Quote
Old 04-07-2014, 09:24 AM   #2
JHughes
Banned
 
Join Date: Aug 2007
Location: Too close to Charlotte, NC
Posts: 3,554
Default

Wow, this looks great. But I can't try it for a couple of days.
Thanks clepsydrae!
JHughes is offline   Reply With Quote
Old 04-20-2014, 10:45 AM   #3
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default New version!

Hey! Just released version 2.0!

http://lacinato.com/cm/software/othersoft/abx

Lots of major improvements:

- ABCDEF....X testing... i.e. more than just 2 files that it plays from
- blind shootouts! it shuffles them invisibly and you pick your favorite
- play a specfic region of the files
- adjust time offset of the files (to make sure they are aligned)
- adjust gain of the files to match them
- loop when playing
- lots of UI improvements

clepsydrae is offline   Reply With Quote
Old 04-20-2014, 10:02 PM   #4
JHughes
Banned
 
Join Date: Aug 2007
Location: Too close to Charlotte, NC
Posts: 3,554
Default

Thank you for the update!
JHughes is offline   Reply With Quote
Old 04-21-2014, 03:03 AM   #5
zeekat
Human being with feelings
 
zeekat's Avatar
 
Join Date: May 2009
Location: Polandia
Posts: 3,578
Default

Quote:
Originally Posted by clepsydrae View Post
What other features might you want to see? E.g. playing A/B/C/D/E/F... as X? That'd be useful for shootouts, etc.
Continuing playback from the same position when clicking next X, IMO it's quite important while testing for subtle differences.
__________________
AM bient, rund funk and heavy meteo
my bandcamp+youtubings
zeekat is offline   Reply With Quote
Old 04-21-2014, 11:49 AM   #6
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 12,557
Default

Do I understand this correctly that your app has the ability to determine if 2 audio files contain identical parts? Not just identical files (which could be determined by checksum) but different files that contain identical bits?

An example might be taking an album and altering the index points slightly. All the audio would be identical but the (individual song) files would absolutely have different checksums vs. the originals.

And this can be done without the need to load the audio into a DAW and mess around and listen/compare things manually?

This would be a really useful tool every now and then.
serr is offline   Reply With Quote
Old 04-21-2014, 11:51 AM   #7
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
Originally Posted by serr View Post
Do I understand this correctly that your app has the ability to determine if 2 audio files contain identical parts? Not just identical files (which could be determined by checksum) but different files that contain identical bits?

An example might be taking an album and altering the index points slightly. All the audio would be identical but the (individual song) files would absolutely have different checksums vs. the originals.

And this can be done without the need to load the audio into a DAW and mess around and listen/compare things manually?

This would be a really useful tool every now and then.
LOL, it proves whether you can hear those differences or not without having someone else help perform the test and prevents personal bias from clouding the results. IOW, if you really can hear the difference between 48k and 96k it will prove it, if you think you can hear it but really can't, it will prove that too.
__________________
Music is what feelings sound like.
karbomusic is offline   Reply With Quote
Old 04-21-2014, 12:05 PM   #8
serr
Human being with feelings
 
Join Date: Sep 2010
Posts: 12,557
Default

Quote:
Originally Posted by karbomusic View Post
LOL, it proves whether you can hear those differences or not without having someone else help perform the test and prevents personal bias from clouding the results. IOW, if you really can hear the difference between 48k and 96k it will prove it, if you think you can hear it but really can't, it will prove that too.
Oh, OK. You're randomizing/anonymizing the files in question for blind-testing. Very different. Sorry.

The point I would be interested in is exact 1:1 matches for bits of audio (for exact same sample rate and bit depth files). And WITHOUT having to rely on my perception (or spending time lining files up and reversing polarity on one of them)!

I can carefully A/B two plugin settings with the stupid thing bypassed (thinking I'm hearing something) as well as the next guy.
serr is offline   Reply With Quote
Old 04-21-2014, 04:09 PM   #9
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by zeekat View Post
Continuing playback from the same position when clicking next X, IMO it's quite important while testing for subtle differences.
Ah -- I see where you're coming from. I had considered that cheating, in a sense -- i.e. any use of the "play" buttons during an ABX test, but I guess the test isn't always "can you identify the file" and sometimes is "can you discriminate against the other files", so, yeah, great point.

And anyway, when doing a shootout, that's totally a needed feature.

Ok, bam!, implemented. Version 2.1 now available for download (well, uploading now -- might take another couple minutes). Lemme know if I messed anything up. :-) There's now a "Hold pos." button (default on).

(that was actually pretty hard to get working... sheesh... there may be a glitch when switching from file to file -- nothing to be done about that...)

Quote:
Originally Posted by karbomusic View Post
LOL, it proves whether you can hear those differences or not without having someone else help perform the test and prevents personal bias from clouding the results. IOW, if you really can hear the difference between 48k and 96k it will prove it, if you think you can hear it but really can't, it will prove that too.
Yes, and you can also use it to do shootouts: "i think i like this last mix i did better than the others, but maybe that's just cause it's the last one i did..." Load up the mixes into the program, start a shootout, pick your favorite, repeat a bunch of times, and it'll tell you which you actually like the best. (Or the least, depending on the question you want to answer.)

Version 2.1 (which also tidies up a couple more UI bugs):

clepsydrae is offline   Reply With Quote
Old 04-21-2014, 05:35 PM   #10
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
Originally Posted by clepsydrae View Post
Ah Yes, and you can also use it to do shootouts: "i think i like this last mix i did better than the others, but maybe that's just cause it's the last one i did..." Load up the mixes into the program, start a shootout, pick your favorite, repeat a bunch of times, and it'll tell you which you actually like the best. (Or the least, depending on the question you want to answer.)
There should be a result that says something like "You have concluded that every single mix sucks. This app will not execute again until you round out your chops".
__________________
Music is what feelings sound like.
karbomusic is offline   Reply With Quote
Old 04-21-2014, 05:37 PM   #11
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by karbomusic View Post
There should be a result that says something like "You have concluded that every single mix sucks. This app will not execute again until you round out your chops".
Maybe i should just add a "fabricate test results" button so people can create the answer they were hoping for. :-)
clepsydrae is offline   Reply With Quote
Old 04-21-2014, 05:38 PM   #12
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
Originally Posted by clepsydrae View Post
Maybe i should just add a "fabricate test results" button so people can create the answer they were hoping for. :-)
It could go beside the "I got, nuthin' what do you think" button. JK, nice work.
__________________
Music is what feelings sound like.
karbomusic is offline   Reply With Quote
Old 04-21-2014, 05:44 PM   #13
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by karbomusic View Post
It could go beside the "I got, nuthin' what do you think" button.
You know, in a way that button exists; my philosophy was to let users cheat if they wanted to, e.g. by showing test results as you were testing, showing which X was playing, etc.

I was torn between that kind of flexibility vs. the program being stricter but a little more believable if people posted test result screenshots. Since anyone could fake anything anyway, it seemed like covering all the possible use cases was the better way to go.

E.g. before if you adjusted the gain or offset of a file or anything like that, the test would immediately reset, because the consistency would be violated. As it stands now, users will just have to come to their own conclusions about how much corruption of the test is acceptable.
clepsydrae is offline   Reply With Quote
Old 04-21-2014, 05:52 PM   #14
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
Originally Posted by clepsydrae View Post
You know, in a way that button exists; my philosophy was to let users cheat if they wanted to, e.g. by showing test results as you were testing, showing which X was playing, etc.

I was torn between that kind of flexibility vs. the program being stricter but a little more believable if people posted test result screenshots. Since anyone could fake anything anyway, it seemed like covering all the possible use cases was the better way to go.

E.g. before if you adjusted the gain or offset of a file or anything like that, the test would immediately reset, because the consistency would be violated. As it stands now, users will just have to come to their own conclusions about how much corruption of the test is acceptable.
I say we dream up some TLS cert based checks for the clepsydrae official authentic seal of approval. That would at least allow you to want to lie really badly to fool everyone else. Again, just having fun here but I agree, the person has to want to know first.
__________________
Music is what feelings sound like.
karbomusic is offline   Reply With Quote
Old 04-21-2014, 05:56 PM   #15
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by karbomusic View Post
I say we dream up some TLS cert based checks for the official authentic seal of approval. That would at least allow you to want to lie really badly to fool everyone else. Again, just having fun here but I agree, the person has to want to know first.
I gave some thought, and intend to give some serious thought, to some way of cryptographically ensuring the results of such tests. it's a really interesting idea.

Speaking of which: the media player library i'm using to play audio is rather limited and frustrating in a lot of ways, but it does appear to support playing URLs over the network...

Would anyone appreciate that functionality? You could post a few files and say "which mix is better" and people could do a blind shootout without having to download them. I'm not sure there's really enough demand, but who knows.
clepsydrae is offline   Reply With Quote
Old 04-22-2014, 10:34 PM   #16
zeekat
Human being with feelings
 
zeekat's Avatar
 
Join Date: May 2009
Location: Polandia
Posts: 3,578
Default

Quote:
Originally Posted by clepsydrae View Post
And anyway, when doing a shootout, that's totally a needed feature.

Ok, bam!, implemented. Version 2.1 now available for download (well, uploading now -- might take another couple minutes). Lemme know if I messed anything up. :-) There's now a "Hold pos." button (default on).

(that was actually pretty hard to get working... sheesh... there may be a glitch when switching from file to file -- nothing to be done about that...)
Nice! Now that's a fast FR implementation . Works good - FooBar's ABX plugin has a short gap when switching too, so I guess it's normal.
__________________
AM bient, rund funk and heavy meteo
my bandcamp+youtubings
zeekat is offline   Reply With Quote
Old 04-22-2014, 10:52 PM   #17
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by zeekat View Post
FooBar's ABX plugin has a short gap when switching too, so I guess it's normal.
Yeah, short of implementing a mini-DAW it's kind of hard to avoid (Speaking of which, where are the Reaper ABX and shootout plugins?? :-) )

I was actually pretty surprised that the glitch was as minimal as it is... on my system, if I load up four identical files and jump around between them, i usually can't hear anything. The MediaPlayer object I'm using is not generally so smooth, so that was nice to see...
clepsydrae is offline   Reply With Quote
Old 04-22-2014, 11:52 PM   #18
G-Sun
Human being with feelings
 
G-Sun's Avatar
 
Join Date: May 2010
Location: Norway
Posts: 7,318
Default

Quote:
Originally Posted by clepsydrae View Post
Update: new version! See below.

Hey Reaps -- I was tired of not being able to recommend any solid, current cross-platform ABX software, so i just wrote my own. I'd appreciate any feedback or ideas on it.

http://lacinato.com/cm/software/othersoft/abx
This seems very useful.
Thanks a lot!
__________________
Reaper x64, win 11
Composer, text-writer, producer
Bandcamp
G-Sun is offline   Reply With Quote
Old 04-24-2014, 12:43 PM   #19
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Version 2.2 now out... adds drag-and-drop support: drag a file (or group of files) to the program. If you drag a single file to a particular row, it will replace that file. Otherwise it will insert them.

Makes it a lot easier to set up your tests.
clepsydrae is offline   Reply With Quote
Old 07-04-2014, 10:28 PM   #20
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Quote:
Originally Posted by clepsydrae View Post
Version 2.2 now out... adds drag-and-drop support: drag a file (or group of files) to the program. If you drag a single file to a particular row, it will replace that file. Otherwise it will insert them.

Makes it a lot easier to set up your tests.
Just made my first pass at 2.2, thank you so much for your project. My principal use of the program is to test discrimination between parts of the recording chain, and the ability to adjust the starting point helps in that. Currently I can set the start time, but I don't know where I am in the file in order to choose a value. A play clock would be very helpful, or a "move the start point to this moment" button.

I managed to lock up the "stop" button while listening repeatedly to the sighted files. "Reset" got things going again though.

I plan to do a blog post about your software if that's OK with you. I'll fool with it for a few more days so I don't make a fool of myself when I write it up.

Thanks again,
Fran
Fran Guidry is offline   Reply With Quote
Old 07-06-2014, 04:39 PM   #21
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Hi Fran -- thanks a lot for the feedback.

Quote:
Originally Posted by Fran Guidry View Post
Just made my first pass at 2.2, thank you so much for your project. My principal use of the program is to test discrimination between parts of the recording chain, and the ability to adjust the starting point helps in that. Currently I can set the start time, but I don't know where I am in the file in order to choose a value. A play clock would be very helpful, or a "move the start point to this moment" button.
Great idea -- thanks. It will depend on the (rather limited) sound API that I'm constrained by, but I have a feeling that should be easily possible. I'll definitely try to make that happen. Ideas like that are so obvious in retrospect that I always wonder why they didn't occur to me before. :-)

I like the idea of a "set start to now" and "set end to now" button... Would a text field allowing precise entry of start/end times provide significant value to you beyond those buttons? I'd prefer to avoid the UI clutter of the text field, but i could add it if there was some reason setting values more precisely than the sliders was important to someone.

Quote:
I managed to lock up the "stop" button while listening repeatedly to the sighted files. "Reset" got things going again though.
Oh jeese -- how exactly do you mean "lock up"? Do you mean that it was disabled when the audio was actually playing? If you ever find a way to repeat the issue, do let me know. I work really hard to make sure all the silly UI bugs are ironed out, but there are always more lurking... underlying bugs in the sound API make it ridiculously hard to make that stop button toggle correctly. I should probably just not disable it, but it was handy/efficient to have it double as an indicator that something was playing.

Quote:
I plan to do a blog post about your software if that's OK with you.
Absolutely!

Quote:
I'll fool with it for a few more days so I don't make a fool of myself when I write it up.
Feel free to ask any questions. Glad it's useful,

-c
clepsydrae is offline   Reply With Quote
Old 07-07-2014, 02:45 PM   #22
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Quote:
Originally Posted by clepsydrae View Post
...
Great idea -- thanks. It will depend on the (rather limited) sound API that I'm constrained by, but I have a feeling that should be easily possible. I'll definitely try to make that happen. Ideas like that are so obvious in retrospect that I always wonder why they didn't occur to me before. :-)

I like the idea of a "set start to now" and "set end to now" button... Would a text field allowing precise entry of start/end times provide significant value to you beyond those buttons? I'd prefer to avoid the UI clutter of the text field, but i could add it if there was some reason setting values more precisely than the sliders was important to someone.

...
Set Start Now and (less importantly) Set End Now would be just fine. No need for a text entry. Even having a timer running would be enough, in order to know where to set your existing sliders.

Fran
Fran Guidry is offline   Reply With Quote
Old 07-30-2014, 01:39 PM   #23
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Thanks to Casey for this nifty tool. I did a blog post at Homebrewedmusic.com and shot a couple of demo videos of the ABX-er and the Shootout-er, they're included in the blog post: http://www.homebrewedmusic.com/2014/...-new-abx-tool/

It can be quite enlightening to put a couple of same source level matched clips from different microphones into the ABX-er - mics that might seem wildly different, like a Rode NT2a and a Schoeps CMC64, seem a lot closer than one might expect if one can't see the labels.

Fran
Fran Guidry is offline   Reply With Quote
Old 07-30-2014, 02:14 PM   #24
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
It can be quite enlightening to put a couple of same source level matched clips from different microphones into the ABX-er - mics that might seem wildly different, like a Rode NT2a and a Schoeps CMC64, seem a lot closer than one might expect if one can't see the labels.
What happens when you take the mics out of their sweet spots spec wise? Don't know, just asking because that's the only thing that ever mattered to me when A/Bing gear because it is always the extremes or edges of performance boundaries that matters when comparing gear, not price or middle of the specs (because in the real world, I deal more with extremes than perfect conditions). For me, to place two mics precisely where they do their best, I would certainly hope there not be much difference. Just doesn't seem like the right place to be looking for differences; or at least the ones that really matter.

I do however applaud what you are calling out in general though.
__________________
Music is what feelings sound like.

Last edited by karbomusic; 07-30-2014 at 08:34 PM.
karbomusic is offline   Reply With Quote
Old 07-31-2014, 08:50 AM   #25
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Quote:
Originally Posted by karbomusic View Post
What happens when you take the mics out of their sweet spots spec wise? Don't know, just asking because that's the only thing that ever mattered to me when A/Bing gear because it is always the extremes or edges of performance boundaries that matters when comparing gear, not price or middle of the specs (because in the real world, I deal more with extremes than perfect conditions). For me, to place two mics precisely where they do their best, I would certainly hope there not be much difference. Just doesn't seem like the right place to be looking for differences; or at least the ones that really matter.

I do however applaud what you are calling out in general though.
Perhaps you could explain what you mean by "sweet spot" and "extremes or edges of performance boundaries" because I don't think of mics in those terms. Are you referring to sounds at the upper limit of the mic's dynamic range? Or near the mic's noise floor? Why would one not "place two mics precisely where they do their best" in a controlled studio situation?

Since my recordings are nearly exclusively solo acoustic guitar instrumentals that is the source I use for my comparisons. I encourage folks who deal with other sources to try the techniques I'm using on their own sources. Have you experimented with level matched same source comparisons?

As far as expecting two mics to sound the same, you must be reading a different internet than I am. From my internet research it's absolutely clear that LDCs are vocal mics and sound warm and sexy, SDCs are clinical and fast, mics that cost more sound better, German mics are better than Asian (or Australian) ones and will make compelling recordings more easily.

With all this carefully gathered insight imagine how surprised I was when I could barely hear a difference between a cheap Australian LDC and an expensive German SDC!!

Fran
Fran Guidry is offline   Reply With Quote
Old 07-31-2014, 10:50 AM   #26
karbomusic
Human being with feelings
 
karbomusic's Avatar
 
Join Date: May 2009
Posts: 29,260
Default

Quote:
Originally Posted by Fran Guidry View Post
Why would one not "place two mics precisely where they do their best" in a controlled studio situation?
Because we typically need to (or be prepared to) fit the mic to the conditions not the other way around so IMHO that is not really where price makes the difference "when" price makes a difference.

Quote:
Since my recordings are nearly exclusively solo acoustic guitar instrumentals that is the source I use for my comparisons. I encourage folks who deal with other sources to try the techniques I'm using on their own sources. Have you experimented with level matched same source comparisons?
As stated I applaud what you are calling out but I stand behind the idea that comparing two mics in this fashion doesn't test the parts that matter the most. It only covers audiophile arguments which is great but there is more to value than that. I'm not disagreeing with what you are doing here, I promise! I'm adding that there is more to consider for those wanting to properly evaluate value that's all.

Quote:
From my internet research it's absolutely clear that LDCs are vocal mics and sound warm and sexy, SDCs are clinical and fast, mics that cost more sound better, German mics are better than Asian (or Australian) ones and will make compelling recordings more easily.
I suppose I simply never fell for those coming up; I was probably too broke and figured out what mattered by the time I actually had the cash. I've always justified my placement of value on many factors and I absolutely think it is dangerous for anyone to ONLY consider the sound they hear in perfect conditions. Doing so simply doesn't match my real world experience. My experience is dealing with where a microphone (insert any gear hear) falls down in less than perfect conditions and most of the times I've paid extra it was to solve those type issues. It was never because I thought something was going to sound more 3d and sexy lol.

Quote:
With all this carefully gathered insight imagine how surprised I was when I could barely hear a difference between a cheap Australian LDC and an expensive German SDC!!
I never expected a high dollar mic to sound 1000 times better because I already knew why they cost more, sometimes that price is justified, sometimes it is not justified. The goal is to be able to know which is which.

The following statement is not about you, just a general observation... I truly despise this dumbing down on the net as of late that the immediate sound of something is the only place value lives. It isn't. My Preamp (the one I have that costs more) costs more because it can supply 300 volts of rail power when it happens to need it. That's a real need that costs more but no one would ever hear/consider that when testing in a "sweet spot".

To be clear I have far, far more cheap mics and pres than expensive ones. I'm not defending any side here, I don't believe in sides, I believe in being able to tell where the value is or isn't.
__________________
Music is what feelings sound like.

Last edited by karbomusic; 07-31-2014 at 11:27 AM.
karbomusic is offline   Reply With Quote
Old 08-03-2014, 04:13 PM   #27
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Hey Fran -- thanks a bunch for the write-up and videos! -- it's exciting to know people actually use the stuff you make sometimes. :-)

One comment: I sorta consider it cheating to listen to A or B after the ABX has started, but that's admittedly kinda arbitrary, and I did leave the option open so that people could do it if they felt it was helpful. It just seems like it enables the same bias you're usually trying to screen out.
clepsydrae is offline   Reply With Quote
Old 08-03-2014, 11:31 PM   #28
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Quote:
Originally Posted by clepsydrae View Post
Hey Fran -- thanks a bunch for the write-up and videos! -- it's exciting to know people actually use the stuff you make sometimes. :-)

One comment: I sorta consider it cheating to listen to A or B after the ABX has started, but that's admittedly kinda arbitrary, and I did leave the option open so that people could do it if they felt it was helpful. It just seems like it enables the same bias you're usually trying to screen out.
I wouldn't care to try to compare sounds without being able to refresh the memory repeatedly. Any single user ABX is all about user control, with the guiding rule being to attempt to achieve certainty before making a choice. The bias is screened out by the anonymity of X.

Remember, ABX is _not_ about preference, it's about discrimination. Preference is measured with other tools, like your shootouter for instance, but there's no value in those tools if in fact the samples cannot be distinguished.

Fran
Fran Guidry is offline   Reply With Quote
Old 08-03-2014, 11:57 PM   #29
mete0r
Human being with feelings
 
Join Date: Jul 2013
Posts: 121
Default

Thank you! This software is pretty cool. Maybe add flac so people can insulting each other? :P :P :P

Seriously though. I *might* be able to distinguish between 256k mp3 and 128k mp3... but your software is evil! HE was trying to trick me! I know it



Will repeat this test and do some others... and I am also interested in trying some things with friends as guinea pigs
mete0r is offline   Reply With Quote
Old 08-04-2014, 12:31 AM   #30
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by Fran Guidry View Post
I wouldn't care to try to compare sounds without being able to refresh the memory repeatedly. Any single user ABX is all about user control, with the guiding rule being to attempt to achieve certainty before making a choice. The bias is screened out by the anonymity of X.

Remember, ABX is _not_ about preference, it's about discrimination. Preference is measured with other tools, like your shootouter for instance, but there's no value in those tools if in fact the samples cannot be distinguished.

Fran
Agreed that ABX is about discrimination. But I think it depends on the specific type of "discrimination" you mean. If I'm trying to discriminate between two mics, it seems more relevant to me to discriminate based on the overall sound, and only on a listen or two, rather than say the exact timbre of a transient T sound over a .5-second clip that I play repeatedly over and over in comparison to the same section of A and B, unless of course your goal is to prove a lack of discrimination ability.

In other words, to prove that things sound effectively identical, it's certainly the stronger result if you can say "I let myself play A and B and X repeatedly and still couldn't tell the difference". On the other hand, if you're trying to prove that things exhibit differences, it doesn't mean as much if you had to play A and B 15 times against X to be able to hear it. And if you just want to answer a more general, un-biased "are these two things different enough to be able to tell them apart", it seems more logical to play them in a more "natural" way that doesn't involve microscopic repeated comparisons.

When I said "bias" I meant the bias that creeps in when you can discriminate through means that aren't quite as "natural". It creeps in with shootouts where you are comparing two mics to see which you prefer: if you know one mic has slightly brighter transient response, and a certain T sound pops out, you can start identifying which is which based on that, rather than the unbiased preference you were trying to tease out, and it kinda ruins the objectivity. I feel a similar thing in ABX: you can hone in on a small difference and it lets you discriminate which is which, which decisively answers the question "is it possible by any means to discriminate", but doesn't really help you answer the question "are they significantly different?", if in fact that was the question you were trying to answer (and it's more often the one I'm interested in.)
clepsydrae is offline   Reply With Quote
Old 08-04-2014, 12:35 AM   #31
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Quote:
Originally Posted by mete0r View Post
Seriously though. I *might* be able to distinguish between 256k mp3 and 128k mp3... but your software is evil! HE was trying to trick me! I know it
Nice work!

256 vs 320 is a much tougher one, I'd say. :-)

And 320 vs. anything better is the killer: that Trust Me I'm a Scientist challenge still stands for anyone that thinks they can hear 320 vs. any higher quality audio standard.

Quote:
Will repeat this test and do some others... and I am also interested in trying some things with friends as guinea pigs
Go easy on them. It can be traumatic. :-)

We'd love to hear about the results.
clepsydrae is offline   Reply With Quote
Old 08-04-2014, 11:23 AM   #32
Fran Guidry
Human being with feelings
 
Fran Guidry's Avatar
 
Join Date: Dec 2007
Location: Walnut Creek, CA
Posts: 805
Default

Quote:
Originally Posted by clepsydrae View Post
Agreed that ABX is about discrimination. But I think it depends on the specific type of "discrimination" you mean. If I'm trying to discriminate between two mics, it seems more relevant to me to discriminate based on the overall sound, and only on a listen or two, rather than say the exact timbre of a transient T sound over a .5-second clip that I play repeatedly over and over in comparison to the same section of A and B, unless of course your goal is to prove a lack of discrimination ability.

In other words, to prove that things sound effectively identical, it's certainly the stronger result if you can say "I let myself play A and B and X repeatedly and still couldn't tell the difference". On the other hand, if you're trying to prove that things exhibit differences, it doesn't mean as much if you had to play A and B 15 times against X to be able to hear it. And if you just want to answer a more general, un-biased "are these two things different enough to be able to tell them apart", it seems more logical to play them in a more "natural" way that doesn't involve microscopic repeated comparisons.

When I said "bias" I meant the bias that creeps in when you can discriminate through means that aren't quite as "natural". It creeps in with shootouts where you are comparing two mics to see which you prefer: if you know one mic has slightly brighter transient response, and a certain T sound pops out, you can start identifying which is which based on that, rather than the unbiased preference you were trying to tease out, and it kinda ruins the objectivity. I feel a similar thing in ABX: you can hone in on a small difference and it lets you discriminate which is which, which decisively answers the question "is it possible by any means to discriminate", but doesn't really help you answer the question "are they significantly different?", if in fact that was the question you were trying to answer (and it's more often the one I'm interested in.)
If you want to add a feature that makes it possible to judge _how different_ two samples are, you might try to conceive of a timer that records how long it takes to make a choice. If the differences are significant the choice should happen more quickly, right? I'm not aware of a comparator with this feature, but I've been part of discussions where the selection time was estimated and offered as evidence of the degree of difference.

But at its heart ABX _is_ about fine grained "is it possible to tell the difference" kinds of comparisons, at least in my understanding and use of the process.

Fran
Fran Guidry is offline   Reply With Quote
Old 09-19-2014, 01:13 AM   #33
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Version 2.3 just released!

- now you can save/load sets of files/gains/offsets (with absolute or relative paths, so you can save a fileset along with the audio files in the same directory and share your tests with others)
- set the play position with its own slider, optionally set the start/end slider positions with buttons
- a bunch of UI improvements (timer showing file position, copy text from the results pane, etc.)

Let me know if you find any bugs. I think it's pretty much "feature-complete" at this point, so I don't foresee much more work happening on it unless folks think of something clever.

(Unfortunately there is some jankyness in the media player: sometimes the loop breaks if you're moving the sliders around a lot while it plays, etc. Nothing to be done about that.)

Thanks!
-c

clepsydrae is offline   Reply With Quote
Old 09-29-2014, 09:09 PM   #34
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

2.32 adds support for drag/drop of *.abx files and opening *.abx files directly (once they are associated in the OS to "open with" the program.)
clepsydrae is offline   Reply With Quote
Old 10-10-2014, 01:33 PM   #35
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Lots of little UI improvements for 2.34, up now... (well, in 5 or 10m when upload is done.)

http://lacinato.com/cm/software/othersoft/abx
clepsydrae is offline   Reply With Quote
Old 07-20-2015, 10:26 PM   #36
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Version 2.35 fixes a lockup/freeze that some Mac OS X users were seeing.
clepsydrae is offline   Reply With Quote
Old 07-21-2015, 02:07 AM   #37
ivansc
Human being with feelings
 
Join Date: Aug 2007
Location: Near Cambridge UK and Near Questembert, France
Posts: 22,754
Default

O.K. As soon as this surfaces on GearSlutz, I am buying shares in popcorn companies!
ivansc is offline   Reply With Quote
Old 12-25-2015, 02:25 AM   #38
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default

Version 2.36 released --

* in shootouts you can now optionally score each file 1-10 as you go
* auto-detect changes to audio files and prompt to refresh/ignore (except on Windows due to Java bug)
* keyboard shortcuts
* prompts to prevent accidental loss of test results
* lots of little UI improvements

See the website.

And that's not all -- I've just released a web-based (javascript) version of my ABX/Shootout tool. You can check it out here.

It's a free-for-non-commercial use tool that you can put on your web page so that visitors to your site can do ABX tests or shootouts. It has some other fancy features, too.

Let me know if you find any bugs. :-)

Thanks!
clepsydrae is offline   Reply With Quote
Old 03-16-2018, 01:34 PM   #39
clepsydrae
Human being with feelings
 
clepsydrae's Avatar
 
Join Date: Nov 2011
Posts: 3,409
Default FLAC support!

Version 2.39 released. New features:

2.39 fixes a few seeking/looping bugs, enables check for new version on startup (optionally), adds FLAC support via conversion to WAV, as well as MP3 support for linux distros that lack the needed system libraries, and adds a config.txt to set configuration variables

2.38 accessibility release: adds a few keyboard shortcuts, the ability to save results to a text file, a simpler alternate .abx syntax, a bell/buzzer for ABX guess success/failure, and fixes a few very minor bugs

2.37 fixes MP3 playback in some recent operating system versions

Download at the website.
clepsydrae is offline   Reply With Quote
Old 03-17-2018, 01:14 AM   #40
drumphil
Human being with feelings
 
drumphil's Avatar
 
Join Date: Jun 2006
Location: Australia
Posts: 3,738
Default

Does it change the sample rate of the audio interface to match the sample rate of the files you are testing on the fly, so you aren't testing the quality of windows resampling without realizing it?

For example, if the first file is 48K, and the second is 96K, does it switch to 96K output for the second file, or just leave it to windows to resample the 96K audio down to 48K?
drumphil is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 06:13 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.