Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER Pre-Release Discussion

Reply
 
Thread Tools Display Modes
Old 05-11-2022, 03:39 PM   #1
msmucr
Human being with feelings
 
Join Date: Jun 2009
Location: Praha, Czech republic
Posts: 595
Default WMF support in Reaper 6.57+ / glitches

Hello to all,

it's a quite some time when I last visited the forum.. I'm pleased with new features added to Reaper including support for Windows Media Foundation in the recent release.

For example decoding and playback of common long-GOP AVC in MP4 container is definitely much more responsive and fluent than any other method available on Windows (ffmpeg, VLC, DS) according to my tests. Big thumbs up for such improvement.

However I few things in both 6.57 release and current 6.57+dev0511, which would be nice to address.

The first is related directly to AVC (H.264) decoding. There is some vertical shift in decoded color planes. Related artifacts are visible at attached screenshot.

I've tried that at different computers with the same results, plus test file decodes correctly using WMF in Windows built-in "Movies & TV" app.

The generated 1 sec. test file was done with ffmpeg and following command:

ffmpeg -f lavfi -i smptehdbars=size=hd1080:duration=1 -pix_fmt yuv420p -c:v libx264 .\test_smpte_420_avc.mp4

To cross check that I've also tried to generate similar files for other WMF supported formats - VP9 in WEBM container and WMV, which were both decoded fine.

Next thing is, WMF decoder in Reaper isn't used for HEVC (H.265) files in MP4 container. HEVC won't work out of box in Windows 10, but it's $1 MS official extension from Windows Store https://www.microsoft.com/store/productId/9NMZLZ57R3T7

Other additional formats for WMF from the Store like VP9 or AV1 works well in Reaper. So HEVC would be potentially also nice (assuming, there might be similar performance gain like with AVC).


Finally I tested WMF render and found some glitches.

MP4 file with both video and audio (H.264, AAC respectively) has incorrect suffix. It should be mp4 instead of m4v. Exported file is correct and contains both, just output filename is wrong.

WMV render has incorrect codec choices. There should be WMV and WMA instead of H.264 and AAC. Such render produces zero length file.


Thank you very much for great ongoing support,

Michal
Attached Images
File Type: png reaper_657_wmf_decoded_frame.png (18.2 KB, 185 views)
File Type: png wmf export to mpeg4.PNG (33.8 KB, 191 views)
File Type: png wmf export to windows media video.PNG (33.3 KB, 186 views)
__________________
FRs: Better FX bypass
msmucr is offline   Reply With Quote
Old 05-11-2022, 04:57 PM   #2
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Quote:
Originally Posted by msmucr View Post
MP4 file with both video and audio (H.264, AAC respectively) has incorrect suffix. It should be mp4 instead of m4v. Exported file is correct and contains both, just output filename is wrong.
We'll look at these other issues, but fyi the file extension can be set by the user. The default is .m4v but you can type .mp4 (or anything else) in the render dialog, output section, file name box.
schwa is offline   Reply With Quote
Old 05-11-2022, 06:19 PM   #3
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Quote:
Originally Posted by msmucr View Post
WMV render has incorrect codec choices. There should be WMV and WMA instead of H.264 and AAC. Such render produces zero length file.
WMV/WMA with the default H264/AAC encoders does work as expected in the 6.57 release version, as far as I can tell.

This render does not work in 6.57+dev0511 because the fix to handle multichannel audio is incompatible with WMV/WMA. We'll figure out a workaround for that.

You may already know this, but it is confusing: WMV and WMA are used to describe both a file container type and a codec. It's valid to encode H264/AAC to the WMV/WMA file container. The Windows documentation for the WMV/WMA codecs is not as complete as the documentation for the H264/AAC codecs, in particular there is very little information about how to parameterize the encoders, and there didn't seem to be any reason to support them so we didn't. Decoding should work as expected though.
schwa is offline   Reply With Quote
Old 05-12-2022, 04:38 AM   #4
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Quote:
Originally Posted by msmucr View Post
There is some vertical shift in decoded color planes. Related artifacts are visible at attached screenshot.
Thank you for catching this! It should be fixed in the +dev0512 build that we will post later today.
schwa is offline   Reply With Quote
Old 05-12-2022, 06:01 AM   #5
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Adding this, because I can't find any satisfactory documentation for this and maybe somebody with some experience will have some insight.

The WMF decoder will sometimes return YV12 frames that are padded with extra rows. This is not unreasonable but what is odd is that when there are extra rows, the row layout of the plane data is no longer contiguous.

Here is a sample project that demonstrates this offset by displaying the first frame of a video. The unexpected adjustment is noted with a comment. If those offsets are ignored and the raw frame is displayed instead of the adjusted frame, the decoding is clearly not correct.

There is also commented-out code that will display the not-offset decoded frame in the same window, above the offset decoded frame, but for that to be useful you need a smaller video or a bigger monitor than what I'm using at the moment.

https://github.com/schwaaa/wmf_yv12
schwa is offline   Reply With Quote
Old 05-12-2022, 01:46 PM   #6
msmucr
Human being with feelings
 
Join Date: Jun 2009
Location: Praha, Czech republic
Posts: 595
Default

Quote:
Originally Posted by schwa View Post
Thank you for catching this! It should be fixed in the +dev0512 build that we will post later today.
Hello Schwa, thank you very much for such quick fix. I've just tested the new build and it seems, the issue is completely resolved.
MP4 with AVC is still most common format so far for guide/preview videos, it's much smoother with WMF decoder in Reaper.
Happy camper here

Michal
__________________
FRs: Better FX bypass
msmucr is offline   Reply With Quote
Old 05-12-2022, 04:28 PM   #7
msmucr
Human being with feelings
 
Join Date: Jun 2009
Location: Praha, Czech republic
Posts: 595
Default

Quote:
Originally Posted by schwa View Post
We'll look at these other issues, but fyi the file extension can be set by the user. The default is .m4v but you can type .mp4 (or anything else) in the render dialog, output section, file name box.
I didn't know that. I recall some tests with templates, naming and suffixes in the render dialog couple years ago, where I concluded it isn't possible. But apparently, I just tried it with output format, which doesn't support that manually entered extension.
For example it doesn't work in WAV or WavPack, but works for video files or say AIFF. And it makes sense, because WAV doesn't have ambiguous extensions. Nice

Quote:
Originally Posted by schwa View Post
WMV/WMA with the default H264/AAC encoders does work as expected in the 6.57 release version, as far as I can tell.

This render does not work in 6.57+dev0511 because the fix to handle multichannel audio is incompatible with WMV/WMA. We'll figure out a workaround for that.
You're true, I've checked that again, that issue was just in the pre-release version. I should be more clear about that.

Quote:
You may already know this, but it is confusing: WMV and WMA are used to describe both a file container type and a codec. It's valid to encode H264/AAC to the WMV/WMA file container. The Windows documentation for the WMV/WMA codecs is not as complete as the documentation for the H264/AAC codecs, in particular there is very little information about how to parameterize the encoders, and there didn't seem to be any reason to support them so we didn't. Decoding should work as expected though.

I'm aware, ASF (generic name of the container) is designed to carry pretty much any audio and video stream format, you'll mux into it.. be it audio or video. It will work, if there is matching decoder for particular GUID format identifier (insane Rosetta stone with identifiers, FOURCCs over there https://gix.github.io/media-types/ ). It was also designed in flexible way to support anything from streaming, digital distribution of audio or video content with DRM, archiving, recording etc.
Together with their codecs, it was cornerstone of MS endeavor to succeed in those markets from late 90s. As you've touched, this common container has different filename suffix according to codec and content (.WMA, .WMV, but also .DVR-MS) and different MIME types, when served via HTTP.
Their own codecs was also very serious (and likely expensive) effort, their latest iteration of WMV video codec was accepted as a standard by SMPTE VC-1. It can have very good picture quality for HD content (you can even author high-bitrate Blu-Rays with it).

Anyway despite all those efforts all of that lost its traction. Latest traces of related development was 10-15y ago. Almost in every market it was superseded by other solutions. There's just handful of Blu-Rays encoded with VC-1, Zune store which sold WMAs is long gone, all of MS proprietary video streaming stuff was replaced with solutions via W3C media source extensions in browsers and HLS or DASH streaming. Google pushed market significantly with its own open codecs and formats with no strings attached. Plus you have ubiquitous and well supported standards under MPEG umbrella.

So today you have almost no real reason to use ASF container for H.264 and AAC streams, even if you technically could do that. Standard MPEG-4 container is simply more compatible with any device, platform. From distribution platforms/video CDNs, mobile devices, web browsers, different operating systems etc.
Also TBH I've never seen such file in practice, except my own tests. So I think, it's pretty redundant in Reaper.

The only remaining (and rare TBH) reason to use ASF is in variant with own Microsoft codecs.. eg. WMV and WMA files. It's still the only format, which can be immediately played with almost every Windows version and edition from XP onwards. No additional codec or player is necessary there. Plus maybe some very old devices based with embeded Windows (for example some vintage digital signage stuff or so) plays such clips.
If someone has such requirement, it might be sill useful.
And as latest Reaper can directly utilize MS libraries via WMF, there is inherent advantage, as ffmpeg (or more precisely libavformat) can encode only very old and subpar variants of WMV and has no support for encoding of WMA Pro.
So if some ASF suport will be worthwhile there, it's this one IMO, IME of course.

I'm sorry for such long waffle I was a system engineer for TV stations and postproduction studios, so I can have stories about dealing with codecs and containers.

Michal

P.S. If you want, I can further expand a bit about useful parameters for WMV for encoding.
__________________
FRs: Better FX bypass
msmucr is offline   Reply With Quote
Old 05-12-2022, 05:16 PM   #8
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Thank you, that is an interesting historical perspective. To avoid confusion I think we will remove the options to render to .wma and .wmv files, since as you say there is no real benefit to writing to those file formats when using modern codecs, and there's also not much reason to support encoding with the windows media audio/video codecs.

So the WMF render options will be mpeg-4 video or audio, with H.264 video codec and AAC or FLAC audio codec.

Decoding (playing back files) will still work for .wma and .wmv, of course.
schwa is offline   Reply With Quote
Old 05-13-2022, 02:09 PM   #9
Fabian
Human being with feelings
 
Fabian's Avatar
 
Join Date: Sep 2008
Location: Sweden
Posts: 7,417
Default

things are severely messed up with my videos in v6.57. Is this due to "Using Windows Media Foundation v2 decoder for video"? Why is that taking priority? Previously things worked just fine, but now, everything just looks like... well a gif says more than a thousand words:


Source properties are these:


And this is Reaper v6.57/x64, W7x64

EDIT: OK, removing wmf and setting vlc as decoder and things are back to normal! *phew*
__________________
// MVHMF
I never always did the right thing, but all I did wasn't wrong...
Fabian is online now   Reply With Quote
Old 05-13-2022, 02:22 PM   #10
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

I think this decoding problem with certain videos is likely already fixed in the 6.57+dev builds, but we'd like to make sure, could you please share that video?
schwa is offline   Reply With Quote
Old 05-15-2022, 03:03 PM   #11
msmucr
Human being with feelings
 
Join Date: Jun 2009
Location: Praha, Czech republic
Posts: 595
Default

Quote:
Originally Posted by schwa View Post
Thank you, that is an interesting historical perspective. To avoid confusion I think we will remove the options to render to .wma and .wmv files, since as you say there is no real benefit to writing to those file formats when using modern codecs, and there's also not much reason to support encoding with the windows media audio/video codecs.

So the WMF render options will be mpeg-4 video or audio, with H.264 video codec and AAC or FLAC audio codec.

Decoding (playing back files) will still work for .wma and .wmv, of course.
You're welcome and thank you, I think the removal of ASF render with H.264/AAC streams is for the better.

I thought a bit about options in the render dialog. Currently (the latest dev version) there is main format as "MPEG-4/Windows Media", with another sub selection for "MPEG-4 video" and "MPEG-4 audio" formats and finally you can select codec for each respective stream, including NONE option.
It seems to be a bit confusing.

If you decided to don't add WMA/WMV codecs for render, then main format could be just "MPEG-4/Windows Media Foundation".
Subformat choice between audio/video also seems redundant now, because essentially there is just one type of container - MPEG-4 and codecs there can be used for audio/video distinction.
The codec selection could be also used for filename suffix.
Default should be mp4. If audio codec is NONE, then suffix is m4v. If video codec is NONE, then suffix is m4a.
This seems like most logical and correct behavior to me.

I've also noticed, there are new text boxes with some presets for video ffmpeg/libav codec options like for CRF value, GOP length, quality factor for VBR audio etc.
How is that supposed to be used?
Can I use just presets from pull down menu or can I freely enter some space separated keyword values, like "g=50 crf=23"?
I assumed, everything after semicolon is likely treated as a comment.
Also specification of a CRF or q factor likely take a precedence over entered bitrate, right? Because in most cases/codecs both together doesn't make much sense, each is for different rate control mode.

Thank you,

Michal
__________________
FRs: Better FX bypass
msmucr is offline   Reply With Quote
Old 05-15-2022, 03:30 PM   #12
msmucr
Human being with feelings
 
Join Date: Jun 2009
Location: Praha, Czech republic
Posts: 595
Default

Speaking of video decoding via WMF I haven't noticed any other issues (besides that with shifted planes, which is already fixed).
I've tested bunch of different H.264 files in various containers (MP4, MOV, MKV), from various software and hardware encoders and never seen the error, which Fabian has reported before.
I'm curious, what that would be. Maybe it's related to Windows 7, he's using, all of my tests are on Windows 10.
There were definitely updates in built-in decoding WMF libraries between Windows versions.

I recently found only one WMF bug/oddity with AC3 decoding. Not that is most common format for opening in Reaper, if one would be using that, it's most likely some recorded/remuxed TV show, DVD or Blu-Ray rip etc.
However it works when decoded via VLC and ffmpeg, but decodes incorrectly via WMF.

Decoded ac3 stream has 1/3 of its nominal rate. So everything sounds pitched down.
If it is together with video stream, it is unsynchronized and only first 1/3 of audio stream will be decoded.

It should be easily reproduced with test files.

ffmpeg -f lavfi -i "sine=frequency=1000:duration=5" -c:a ac3 -ac 2 test_ac3_matroska.mkv
or
ffmpeg -f lavfi -i "sine=frequency=1000:duration=5" -c:a ac3 -ac 2 test_ac3_mpeg-4.mp4
(MPEG-4 container can also technically use AC3 stream, although it's not recommended nor standard practice, here is just for testing, because the issues is common to both container types)

whereas following AAC stream is decoded fine.
ffmpeg -f lavfi -i "sine=frequency=1000:duration=5" -c:a aac -ac 2 test_aac_matroska.mkv

For completeness, other WMF apps decodes all examples fine.

Michal
__________________
FRs: Better FX bypass

Last edited by msmucr; 05-15-2022 at 03:56 PM.
msmucr is offline   Reply With Quote
Old 05-15-2022, 04:44 PM   #13
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Thanks again for the testing, we'll check out that AC3 issue.

The decoding problem reported a few posts up was Windows 7 specific and should be fixed in the current +dev builds.
schwa is offline   Reply With Quote
Old 05-16-2022, 12:02 PM   #14
Fabian
Human being with feelings
 
Fabian's Avatar
 
Join Date: Sep 2008
Location: Sweden
Posts: 7,417
Default

Quote:
Originally Posted by schwa View Post
The decoding problem reported a few posts up was Windows 7 specific and should be fixed in the current +dev builds.
Yep! Works in v6.58rc1 here. Thanks for fixing.
__________________
// MVHMF
I never always did the right thing, but all I did wasn't wrong...
Fabian is online now   Reply With Quote
Old 06-13-2022, 08:36 AM   #15
deseipel
Human being with feelings
 
Join Date: Jul 2020
Posts: 55
Default

I'm still seeing issues with wmf in Reaper 6.60. Videos downloaded from Twitch have audio playing back at 1/4 speed and is unsynchronized. When I change the decoder prioroty to "ffmpeg wmf vlc..." the video plays back fine after restarting reaper.

Here is what ffmpeg sees

Audio 48000Hz 1ch 32bps

Using ffmpeg/lbav DLL decoder v58.134.100 for video and audio.

Video: h264(High)(avc1/0x31637661).yuv420p(tv). 1920x1080, 4576kb/s
Audio: aac (HE-AACv2)(mp4 / 0x6134706D). 48000 Hz, stereo, fttp, 33kb/s

When using wmf, it sees this on the same track

Audio: 24000Hz 1ch 16bps
bit rate: 33128
deseipel is offline   Reply With Quote
Old 06-13-2022, 09:09 AM   #16
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,750
Default

Would you mind checking against the current development builds as per the suggestion in this thread?

https://forum.cockos.com/showthread.php?t=267725
schwa is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:44 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.