Fascinating Reading: Proprietary Audio Plugins for Linux, but the right way

[audiojunkie]

I just discovered and read this fascinating thread by well known Linux developers from Bitwig, yabridge, Ardour, Surge, Overtone, etc. For anyone wanting to better understand the state of proprietary audio plugins currently on Linux, as well as better understanding the difficulties these developers go through to provide these plugins (versus Windows or Mac), this thread won't disappoint!! There were so many takeaways from this conversation! It was really, REALLY eye opening! :-)

* It makes so much more sense why the developers seem to only put out proprietary packages for just Ubuntu (and sometimes Debian).

* It shows how much work goes into creating and supporting Linux binaries.

* It gives me a huge insight on how small the market share is currently for Linux, and yet also how it is growing daily.

* It shows the collaborative nature of these awesome Linux developers.

* And especially, I understand much better the strategies and reasonings for plugin library management, reducing the number of dependencies, etc. I don't think I will ever complain about ugly, sparse Linux GUIs ever again! :-)

* I understand why Chris (of AirWindows fame) uses no GUIs.

I understand why Reaper is so minimalist and why it works as well as it does.

There are these and many more takeaways in this fascinating thread:

https://gist.github.com/abique/4c1b9...591d2a7c760db4

[audiojunkie]

In addition, to supplement the Gist thread above, I found the following article a very useful and informative addition:

https://en.wikipedia.org/wiki/Dependency_hell

[audiojunkie]

One thing that I want to research more is Arch/Debian pros/cons: Arch User Repository vs Ubuntu native distro supported proprietary plugins. Both of these distro families each have a unique strength, and I would LOVE to find a way to have both, in the most compatible way possible. For example, the number of unique source packages in the AUR far outweighs any other "common" distro family. And yet, Ubuntu seems to be the binary of choice for proprietary plugin developers. I guess my future recommendations for choice of audio distro will be to ask the querying user where they plan to get the majority of their audio software? ie Open Source or Proprietary purchases.

If the majority of the plugins and software that a user will be utilizing will come from open source, then the Arch family would be the best recommendation, because of the AUR.

However, if the majority of the plugins and software that a user will but utilizing will come from proprietary software (ie freeware or for purchase), my future recommendation may point to the Debian family, and in particular, Ubuntu.

Personally, I wish this would change, because there are so many reasons that I don't want to use Ubuntu, and currently prefer Arch. I'll need to assess things again. As it stands, of the two families, my personal preferences would be Arch or Debian over Ubuntu, because of the freedom and choice I get from independent over corporate controlled distributions.

So, as it stands, as long as the KXStudio repos remain well maintained and up to date, it looks like Ubuntu may have the edge because developers prioritize Ubuntu for proprietary audio plugin development and because the KXStudio repos provide most of what the AUR would provide as far as audio software goes. However, if the KXStudio Repos are not well maintained, reliable and kept up to date, the leading edge goes to Arch for their Arch User Repository support.

What would be nice would be if developers support at least Ubuntu and Arch for proprietary binaries, even of they don't support other distros. Fedora, OpenSuse and Gentoo all have audio packages too, but because of the incompleteness of their repositories, they would be less than optimal targets for proprietary binary support, because more people will likely use either AUR supported distros or Ubuntu for their DAWs.

....which brings us all back around to the developer's discussions of the best way of supporting Proprietary Audio Plugins for multiple Linux distro families.

[flechtwerk]

Originally Posted by audiojunkie

I just discovered and read this fascinating thread by well known Linux developers from Bitwig, Ardour, Ardour, Surge, Overtone, etc. For anyone wanting to better understand the state of proprietary audio plugins currently on Linux, as well as better understanding the difficulties these developers go through to provide these plugins (versus Windows or Mac), this thread won't disappoint!! There were so many takeaways from this conversation! It was really, REALLY eye opening! :-)

Oh wow, thanks for the link! I'll dive in straight away.

[audiojunkie]

By the way, @robbert-vdh I accidently wrote Ardour twice when I meant to add your name for yabridge! No slight was intended, and the original post has been corrected. Sorry!

[4duhwinnn]

Thanks for sharing the results of your ongoing research!
I think one of the troubles that insures a small linux userbase, is that so many distro maintainers create or use half-baked system installation routines, followed by inflexible login managers, followed by sundry new-and-improved package management systems, making it needlessly difficult for new linux users to test and to get up and running with the latest distros, which ideally would be easily populated with current software releases, and in a system gui chosen from a login manager that allows the user to choose among the installed possibilities at every startup cycle.

On the bright side, I doubt Linux 11 will be released any time soon...
Cheers

[shosty]

The kxstudio repos are not well maintained at the moment for anything other than the kxstudio apps. I think plugins have taken a backseat as falktx appears to be working hard on other things rather than spending time packaging.

It makes me keen to move to arch. However, I did install manjaro on a laptop and installed loads of stuff with AUR which was great but some plugins were not recognised as the same by Reaper and Renoise. Perhaps there is a simple fix for that by editing the project files but it's an issue which probably will keep me on KDE Neon for a while.

[audiojunkie]

Originally Posted by shosty

The kxstudio repos are not well maintained at the moment for anything other than the kxstudio apps. I think plugins have taken a backseat as falktx appears to be working hard on other things rather than spending time packaging.

It makes me keen to move to arch. However, I did install manjaro on a laptop and installed loads of stuff with AUR which was great but some plugins were not recognised as the same by Reaper and Renoise. Perhaps there is a simple fix for that by editing the project files but it's an issue which probably will keep me on KDE Neon for a while.

Very good to know. This has been an on-going concern for me. Without KXStudio and commercial binaries, the Debian family (as much as I personally love Debian), is no better than Fedora or OpenSuse (or any other non-AUR using distro). If only we could get the developers to provide binaries for commercial apps for the Arch family (instead of everything just being Ubuntu), the Pro Audio distro question would be resolved. With the Arch family, there is something for everyone: Manjaro is as easy as Ubuntu. Arch is for those who want to do things their own way. And EndeavourOS is for Intermediate users. Combined with the AUR, it's a very powerful distro family.

The KXStudio repositories are a tremendous resource, but it is kept and maintained by only one superhuman. The Arch User Repository, however, is kept and maintained by hundreds of experienced packagers. There are better odds of the AUR not going away than there are of the KXStudios no going away. In fact, IIRC, the KXStudio Repos did go away at one time for some issue. It was only a couple of months, but there's still the issue of the KXStudio repos being kept up and maintained well. I just personally think the Arch family would be better than Ubuntu for primary commercial support. Thank goodness Linux developers are working hard to try to maintain as broad of support as possible (including the Arch family).

[shosty]

Is there much of a problem with commercial not working on Arch except for it coming in .deb form?

[audiojunkie]

Originally Posted by shosty

Is there much of a problem with commercial not working on Arch except for it coming in .deb form?

Most of the developers will say something in the requirements like:

"Works with Ubuntu 18.04 or higher."

Some will say:

"Works with Ubuntu 18.04 or similar."

The second may be more flexible, as it may indicate that a "similar" distro may also work, but it's always best to check with the developer first.

The key thing to remember though, is that when a developer compiles a project in a VM with a particular distro, and then tests it for that particular distro, he is only truly supporting that distro that he compiled and tested on. It's just like how Reaper is labeled, "Experimental" as a way of limiting the business liability.

When it comes right down to it, these developers are actively trying to support the broadest number of distros as possible. And, more than likely, the plugins will work on different distros (because the developers purposely try to program the code to be as compatible as possible. However, there are no promises. They've stated that they have built and tested and support Ubuntu Ubuntu 18.04 and up. It gives them legal room and it lets the customer know ahead of time what they are buying.

Like I said, it will likely work. But, because of the differences between different versions, there are risks of it not working. See:

https://en.wikipedia.org/wiki/Dependency_hell

Often what developers will do, is provide a demo version, which you are expected to try on your machine to make sure the app will work, before purchasing. This way, you can know if it will likely work or not.

[shosty]

I really know next to nothing about developing but I think it's good that they use that as a base to be honest. The flip side is that if they develop for Arch then it's quite possible that only people on bleeding edge will be able to use them and I think that number is less than those using more conservative distros. Most linux savvy devs I've seen (VCV Rack being a good example) use an old toolchain to build against because if it works on old versions of packages/libraries/compilers then most of the time it will work with new versions too. The reverse is much more of a gamble and for binaries it can result in them not running at all.

The problems with running more bleeding edge distros aren't confined to closed source stuff though, I don't know if it's still an issue but there used to be VST versions of the gxplugins which wouldn't show their UI in hosts when using Arch because of a fundamental library version difference (I don't remember which it was now).

[audiojunkie]

Originally Posted by shosty

I really know next to nothing about developing but I think it's good that they use that as a base to be honest. The flip side is that if they develop for Arch then it's quite possible that only people on bleeding edge will be able to use them and I think that number is less than those using more conservative distros. Most linux savvy devs I've seen (VCV Rack being a good example) use an old toolchain to build against because if it works on old versions of packages/libraries/compilers then most of the time it will work with new versions too. The reverse is much more of a gamble and for binaries it can result in them not running at all.

The problems with running more bleeding edge distros aren't confined to closed source stuff though, I don't know if it's still an issue but there used to be VST versions of the gxplugins which wouldn't show their UI in hosts when using Arch because of a fundamental library version difference (I don't remember which it was now).

You make a good point. It's not an easy answer. And, if really smart people like the developers struggle to find the right answer, I can't expect much from myself. But I do admit that I see why it is so problematic for Linux.

However, the thought occurs to me that if developers compile just twice for linux, they could get the Ubuntu crowd as well as the Arch crowd. But then, their goal is to not have to spend so much time compiling. In the thread I referenced above, a developer gave a example of what goes into compiling for a binary release. It was something like this (I'm paraphrasing, but you'll get the gist):

Windows x64 VST3
Windows x32 VST3
Mac ARM M1 VST3
Mac Intel VST3
Mac ARM M1 AU
Mac Intel AU
Linux i686
Linux AArch64
Linux ARM v7
Linux X64 (Ubuntu)

The above would feasibly be ten separate binaries if intel and the ARM architectures alone are supported. But it's not that simple. Each distro has different library versions depending on how conservative or bleeding edge the distro is. And, each distro has multiple potential architectures. Let's say that Developers choose to not support 32-bit architectures, and only support the Mac M1 ARM architecture, they would still have a lot of binaries to create:

Windows x64 VST3
Mac ARM M1 VST3
Mac Intel VST3
Mac ARM M1 AU
Mac Intel AU
Linux X64 VST3 (Ubuntu)


If the developers can't find a way to reduce the number of binaries they have to create, and if they support just the main distro families with binaries, it could feasibly balloon to this:

Windows x64 VST3
Mac ARM M1 VST3
Mac Intel VST3
Mac ARM M1 AU
Mac Intel AU
Linux X64 VST3 (Debian Family)
Linux X64 VST3 (Fedora Family)
Linux X64 VST3 (OpenSuse Family)
Linux X64 VST3 (Arch Family)
etc.
etc.

The problem is, that's still not sufficient to guarantee 100% compatibility, because distros within the same family can have different library versions. If one were looking at it clinically, one could justifiably argue that each separate distro should have its own binary. For example, a binary for Debian, a binary for Ubuntu, a binary for Mint, etc.

As you can see, binaries are an exponentially growing headache for developers. Let's say that it takes 30 minutes to compile a binary. Let's say that the developer has multiple computers, each running a VM for a clean binary. It could still possibly take days to babysit everything. It's just not feasible.

The Proprietary Audio Plugins for Linux thread discusses some of these problems. One of their current strategies is to use the least number of dependencies as possible. Most of these dependencies come just from the GUI. App compatibility is likely the chief reason Chris from Airwindows doesn't use GUIs. But Chris' plugins are with a small number of parameters and will work without a GUI. But some plugins use hundreds of parameters, and it's just not possible to efficiently use a VST with no GUI and 600+ parameters. It becomes a real problem.

It may not be so bad, if Linux only used a single desktop environment, but there are 20+ desktop environments and windows managers and such. Available libraries that could potentially cause dependency hell are numerous.

And so far, I've only mentioned VST3. I've left out LV2, AAX and other plugin formats.

So, As I understand it, the thread above tries to find best practices to achieve the most compatibility with a reasonable number of binary builds. Even then, there are no guarantees. It's very complicated and up to bigger brains than my own.

[flechtwerk]

This is a very good blog post on the subject (though not about music software specifically). It's highly critical of solutions like Flatpak etc. where each package ships with its own libraries and opts for general backward compatibility of libraries instead:
https://ludocode.com/blog/flatpak-is-not-the-future

[audiojunkie]

Originally Posted by flechtwerk

This is a very good blog post on the subject (though not about music software specifically). It's highly critical of solutions like Flatpak etc. where each package ships with its own libraries and opts for general backward compatibility of libraries instead:
https://ludocode.com/blog/flatpak-is-not-the-future

Cool! I'll happily read it! The more I understand this stuff, the better!

[flechtwerk]

The solution laid out in that blog post would be that once a binary runs stable in a more conservative distro like Ubuntu, it should also run in a bleeding edge distro since the dependency libraries are backwards compatible. (Which used to be not the case in Linux but is being taken care of more and more - a notable exception being, according to the blog post, the GTK toolkit.)

Of course there remains the question of how to obtain the binaries, and I understand that companies like Bitwig might be worried about the "official" method of installing their commercial product in a distro like Arch is via the AUR - so packaging is taken care of by the user base, with all the security dangers this entails. But as one of the Arch users I would argue that this is not a problem - there is trust in the community and also trust in one's own judgement. The packaging process is transparent after all and one can quickly glance over the installation scripts to detect whether they are doing what they are supposed to do. (Many Arch user wouldn't trust commercial packaging systems anyway that would hide what they are doing.) In the end the installation and updating process is very convenient, I'd prefer it over the Windows way of constantly downloading and installing new versions of Bitwig or Reaper etc. individually any day.

One the question of distros from I user perspective I'd say that Arch (or possibly its derivates though I have no real experience with them) is by far the best distro for audio production if one is a bit tech savvy and willing to put in the time. And otherwise there is something like Ubuntu Studio which works quite perfectly out of the box, in my experience. If companies could develop for these more conservative distros and then the community of the rolling release distros would take care of reasonable backwards compatibility, things could maybe get quite smooth for the developers.

[wastee]

Nice share!

[audiojunkie]

Originally Posted by flechtwerk

The solution laid out in that blog post would be that once a binary runs stable in a more conservative distro like Ubuntu, it should also run in a bleeding edge distro since the dependency libraries are backwards compatible. (Which used to be not the case in Linux but is being taken care of more and more - a notable exception being, according to the blog post, the GTK toolkit.)

Of course there remains the question of how to obtain the binaries, and I understand that companies like Bitwig might be worried about the "official" method of installing their commercial product in a distro like Arch is via the AUR - so packaging is taken care of by the user base, with all the security dangers this entails. But as one of the Arch users I would argue that this is not a problem - there is trust in the community and also trust in one's own judgement. The packaging process is transparent after all and one can quickly glance over the installation scripts to detect whether they are doing what they are supposed to do. (Many Arch user wouldn't trust commercial packaging systems anyway that would hide what they are doing.) In the end the installation and updating process is very convenient, I'd prefer it over the Windows way of constantly downloading and installing new versions of Bitwig or Reaper etc. individually any day.

One the question of distros from I user perspective I'd say that Arch (or possibly its derivates though I have no real experience with them) is by far the best distro for audio production if one is a bit tech savvy and willing to put in the time. And otherwise there is something like Ubuntu Studio which works quite perfectly out of the box, in my experience. If companies could develop for these more conservative distros and then the community of the rolling release distros would take care of reasonable backwards compatibility, things could maybe get quite smooth for the developers.

Very interesting insights! And the blog post was quite informative as well. I would go so far as to suggest that the reason developers are choosing Ubuntu as their main proprietary distribution distro is because it is conservative. I hadn’t thought about that prior to the blog post and your posts but it makes sense because arch is so bleeding edge.

[MiddleC]

https://www.youtube.com/watch?v=Pzl1B7nB9Kc

Linus Torvalds on this topic...

[audiojunkie]

Originally Posted by MiddleC

https://www.youtube.com/watch?v=Pzl1B7nB9Kc

Linus Torvalds on this topic...

Well said! Linus really gets to the point. :-)

[audiojunkie]

So, as far as dealing with Proprietary/commercial/closed-source binaries across distros on Linux, current thought argues for/against the following strategies:

A. Build a separate binary for every OS, chip architecture, or distro release. This is the Original problem and currently the only current guaranteed way to make sure everything will work. It is also so counter productive that NO ONE will do it.

B. Develop something like the OBS (Open Build System) https://build.opensuse.org/ But cover every OS, chip architecture, or distro release. The OBS seems to be really cool and has a lot of potential, but doesn't support everything that is needed, and has its own set of problems.

C. Listen to Linus when he says: Don't "Break Userspace!" In other words Library developers need to maintain 100% backwards compatibility No Matter What, and distro developers need to stick to this core set of compatible libraries.

D. Containerization such as Flatpak, SNAP, etc.

E. While waiting for better solutions to come along, use the recipe for hosts and plug-ins to play nicely together:

1. Keep dependencies minimal (for both host and plugins) and don't expose unnecessary symbols in either (-fvisibility=hidden etc, and don't assume that just 'stripping' a binary does anything of the kind)
2. Dynamically link to system libs in preference to requiring custom library versions or using LD_LIBRARY_PATH hacks
3. Build to the earliest version of system libs you want to be compatible with (most APIs will maintain compatibility in future versions, the reverse is seldom true)
4. Statically link, only if you absolutely have to, and if its compatible with licenses etc.
5. Hosts should load plug-ins with dlopen(RTLD_LOCAL | RTLD_LAZY) as they already do, and that should be enough to keep plug-ins namespaces segregated from one another. There are still some edge cases, like host application vs plugin symbol collisions, but most of these turn out to be rare if the other suggestions are adhered to.

"Then you have something which works at least as well as it does on other platforms. (Intentionally or otherwise, this seems to be what happens with for example Reaper on Linux, the host has minimal dependencies and seems to dynamically link to only a few system infrastructure libs, and by far the vast majority of Linux plug-ins appear to work without problems)."

In addition to this, the Linux binary needs to be compiled for a conservative distro using the older libraries, with the expectation that it should mostly work on a newer distro, like Arch, with bleeding edge binaries.

I would add to this that in all fairness, commercial developers need to provide a way for users to test the software on their own distros by providing a way to demo the software on their personal distro. Refund requests are probably the reason why developers are afraid to label the their software anything other than "experimental" or provide more than limited Linux support. But that doesn't encourage the customer to buy software, even if it limits their liability. There needs to be a way for the consumer to feel more safe that their purchase will work on their distro. Labeling the build as supporting only Ubuntu cuts into their sales.

Honestly, I don't know any of the answers. I do understand the problem much better now though, and I see that there is no easy solution as it stands. Hopefully a solution that far outweighs the likely included cons will be presented for proprietary binary distribution for Linux. Right now, there is not a perfect solution, and it benefits us as consumers to understand the point of view of the developers.

[4duhwinnn]

Ages ago, Puppy linux developed .sfs files, which contain the app(s) and needed libs, utilities etc, and the .sfs files can be loaded on the fly. There was one that contained zynaddsubfx, hydrogen, jackd and qjackctl, among others, so users of a small nimble non-media-centric distribution, could load a working audio production environment whenever-wherever, as Puppy is very physically portable.

Long ago, Alexander Bique nearly single-handedly took the great U-he instruments and effects library, and made them each installable by a script, and Rui Capella sussed out the issue of multiple plugins being provided in one installer...

Per those examples, the issue is really one of human nature. The incumbent stubborness and selfishness of so-called 'free' software developers over-riding common sense, and what benevolence survives 1st world indoctrinations, all in the name of freedumb (aka I'll do it my way, so the 'lesser' distros and apps and libs can all go pound sand).

As for me, I use one distro for audio editing, another for sessions using strictly commercial audio tools, another for mixed-license sessions, another for printing/office duties, and another for travel. (I'll soon add a 6th, for Fedora Pipewire, and see how the dust settles)

I have and use a lot of delightful plugins, so juggling a mere 5 distros is trivial, and major issues popping up are almost nonexistant. For those who will stick to one distro on one computer, sojourns into dependency hell can be minimized by an hour or four in a package manger gui, making a carefully stripped-down distro sans the typical linux shovelware. Only keep what you actually use.
Cheers

[4duhwinnn]

As a counterpoint to the above, I'm pretty sure the Surge coding team
members never shared the same crib when growing up, and are modeling cooperation, efficiency, common sense, and communication, in the arena of genious, extremely well. An example distro creators/maintainers might do well to ponder.

[audiojunkie]

Originally Posted by 4duhwinnn

As a counterpoint to the above, I'm pretty sure the Surge coding team
members never shared the same crib when growing up, and are modeling cooperation, efficiency, common sense, and communication, in the arena of genious, extremely well. An example distro creators/maintainers might do well to ponder.

For about 15 years, we had the LSB (Linux Standard Base), but it died in 2015. It would sure be nice if everyone would play well with each other. Unfortunately, I don't think it's something that I can bank on. In the meantime, I think developing a set of strategies on how to best deal with the situation as it currently stands is our best bet. Using comparison tools like ldd is very useful. I wonder what else is out there.

[audiojunkie]

I discovered this document that relates to the whole conversation at hand:

https://itvision.altervista.org/why....p.current.html

Major Linux Problems
on the Desktop, 2021 edition

It's regularly updated and contains a lot of what we've been discussing.

[audiojunkie]

I've been going back and reading the comments to the blog post, "Flatpak Is Not The Future", and they have been interesting. There are some good arguments FOR containerization. Some of the comments:

Quoted Post:

I’m sorry but I’m going to have to disagree strongly with you, for the same reason that “sysvinit/OpenRC/etc. is all you need” anti-systemd people are wrong. RPM and DEB are only good enough if your use-cases are circumscribed by the needs they were originally developed for.

To some extent, the containerization side of Flatpak is EXACTLY like systemd, in that it’s a “we’ve given you twenty years to fix this. We’re tired of waiting” response to people who preach about “the right way to do it” and then fail to get sufficient traction out of initiatives like the LSB.

For example:

1. I want to run an LTS distro, but I also need one or two newer apps. My choices are a PPA, which is the wild west as far as trust goes, or Flatpak, which provides a centralized system of sandboxing, allows me to easily customize the sandboxing manifest, and, if it’s Flathub, approval isn’t *fully* automatic like for PPAs.

2. Ubuntu has decided to stop packaging Firefox with APT in favour of official Mozilla snaps. Mozilla also officially maintains their Flatpak and binary tarballs with automatic updates… which one has sandboxing?

3. I want to sandbox as many desktop applications as possible for improved security. Which is less likely to cause me strife: Retrofitting other people’s packages using Firejail or just tweaking the Flatpak installs using Flatseal?

Beyond that, I’m reminded of articles like Drew DeVault’s “Developers: Let distros do their job” which say “Ship your software as a simple tarball. Don’t ship pre-built binaries” and “One thing you shouldn’t do is go around asking distros to add your program to their repos. Once you ship your tarballs, your job is done. It’s the users who will go to their distro and ask for a new package.” and brush under the rug the effect it has: “We developers should be elite gatekeepers over everyone who doesn’t know how to compile software from source”… imagine if audiophiles got to have veto power over every product targeted at the market segment of people who don’t know how to hook up a multi-component hi-fi system!

"This is apparently working as intended. They want runtimes to be a free-for-all, filling your hard drive with gigabytes of custom junk for every app."

No, this is “Red Hat/Debian/whoever didn’t make an RPM/Deb/etc. for the app I wanted” all over again… literally. Flatpak is just less willing to foist “Run alien and pick up the pieces” off on the end users in the 21st century.

"Today, on my machine, the KCalc Snap takes a full seven seconds to start up. Not just the first time after boot; every time, without fail. Seven seconds to start a calculator."

No argument. Snap’s architecture is garbage, even by containerized application standards.

"Flatpak allows apps to declare that they need full access to your filesystem or your home folder, yet graphical software stores still claim such apps are sandboxed."

Do I really need to start enumerating things GNOME gets wrong in other spheres?

The Flatpak CLI is the first-party UI and it gets announcing required permissions right. The KDE Discover plugin doesn’t make a half effort like that and just presents Flatpak packages with the same UI as native packages. Don’t be that infamous anti-Flatpak FUD site I won’t give free publicity to.

In short, this is another case of “WORKSFORME/WONTFIX: Your needs are invalid”, just like the last two decades that inspired Flatpak and snaps and AppImage and the like.

"This is the most complicated and brittle way to implement this"

The passage you quoted says “meaning that applications don’t need to do any additional work”. “Don’t need”, not “shouldn’t”. Emphasis mine.

It also omits the next sentence: “Applications that aren’t using a toolkit with support for portals can refer to the xdg-desktop-portal API documentation for information on how to use them.”

That sounds like a pretty standard “Don’t reinvent platform APIs if you can work within them” that would also be appropriate for something like ensuring reliability on the Linux Standard Base or SDL or any other means of ensuring forward compatibility with the proposed paradigm.

This argument sounds like a disingenuous case of “Any solution but mine must run up against a heavy incumbent advantage that favours my solution”. Retrofitting the platform APIs is a very elegant way to get most of the way there with little to no demand on each individual upstream developer’s time and the main reason it works as poorly as it does in my experience is that GtkFileChooserNative is a young API.

"How can Fedora justify publishing apps while masquerading as the upstream developers?"

How is this any different from the mess with Debian vs. Fedora RPM vs. OpenSuSE RPM vs. etc. when it comes to package naming? A good tool can’t magically fix a bad maintainer.

Don’t play the nirvana fallacy card when APT/RPM, Linux itself, and UNIX before it, are perfect examples of “good enough” solutions running rings around attempts to attain perfection.

The post even mentions “Worse is Better” in the next section, so I get a strong sense that this is a disingenuous article, not arguing in good faith. As I said before, people who think like you have had 20 years to come up with a solution that meets the needs Flatpak has set its sights on.

"Proponents of containerization believe it is the solution to every problem, the hammer for every nail."

More an acknowledgement that Linux can’t afford to ignore the “Windows won because it was backwards compatible with DOS. Later Windows won because all 32-bit versions of Windows are still compatible with 16-bit Windows apps. etc. etc. etc.” network effects.

Flatpak is an acknowledgement that you only argue for a clean-slate approach to sandboxing if you *want* it to fail.

Not pictured: Steam and the game running directly on the user’s native environment, like they do on every other OS.

"Not pictured: Steam spyware-ing the user’s DNS cache and doing other shady things in the name of “anti-cheat”."

"A major goal of most of these technologies is to support an “app store” experience […] Flathub only says they don’t process payments at present. It’s coming."

This is willfully blind at best.

Canonical already tried this using traditional APT-and-Deb packaging, no snaps needed. Containerized packaging is orthogonal to attempts to monetize.

All you need is a repository-based package manager (APT) and suitable metadata (AppStream)… both of which are requirements completely independent of these new packaging formats.

"Just in the past week, I’ve used several Linux apps distributed as plain binary tarballs that run on my native environment."

Oh, like my Humble Bundle games where, to this day, I have to root around through old Ubuntu/Debian package archives for libraries with the right `.so` version or google up answers for why they crash mysteriously on startup?

"GOG.com’s strategy for Linux support is just about the opposite of Steam’s, and works just like traditional installers for Windows."

…and every time I upgrade my Ubuntu, I’ll have at least one or two games where I have to go in and delete one of the bundled libraries to resolve a “crashes on startup” issue.

Not to mention, vintage games don’t have the “security updates are being released that need to make it to the end users promptly” issue.

As a retro-computing enthusiast, this reminds me of the Good Old Days™ of Windows 9x when malware was in its infancy and gamers hadn’t yet started to trickle away to the convenience of consoles before Steam made things more convenient.

"Imagine if millions of office workers used Excel on Ubuntu. How loudly do you think businesses would complain if a distribution upgrade broke it?"

An “Easier said than done.” argument. Where do you draw the line on which libraries are safe to assume will be present in perpetuity? How do you get distros to agree on it? How do you ensure they keep offering access to the same ancient versions forever? What if upstream don’t properly follow SONAME versioning for ABI compatibility? (Automatic analysis has shown humans suck at that)

How do propose that we *now* get all the distros agreeing on dependency metadata for packages they consider beneath their notice when they weren’t before.

Again, the phrase that defines Flatpak’s raison d’etre… “things the open-source ecosystem is failing to deliver on”… just like why systemd replaced sysvinit.

"The GTK problem"

I have no argument with this. As a KDE user, I agree that the “and pre-installed on the vast majority of Linux desktop” makes his argument accurate.

…I won’t use GTK 3.x now that it’s grown too many GNOME-isms and still lacks stuff Qt provides by default like minimal toolbar customization, but I agree with the idea to arm-twist GNOME into being better stewards of it.

"Is Flatpak Fixable?"

Let me know when you manage to get sufficient buy-in on that “declare library dependencies that don’t care about Debian vs. Fedora vs. OpenSuSE vs. Arch vs. … naming conventions”, “Convince all app developers to retrofit their apps for a new API”, and “Deprecate install-time permissions so aggressively as to kill off any momentum already gained”.

Final Verdict: Talk is cheap. Prove you can do differently than the last 20 years of APT and RPM that inspired these packaging systems and I’ll listen.

[audiojunkie]

I suspect that while Flatpak may not be the complete solution, and maybe not the solution at all for plugins, it will be a solution for some commercial software. (I personally don't like SNAP or Canonical's planned store monopoly, so I'm not even considering it.) Which brings us back around to my opinion that the developers have a big struggle on their hands and a problem with no easy answer. I can understand why proprietary (commercial) binaries haven't taken off on Linux. I still hope to see things get better, and I can see for myself that the developers are currently doing the best they can, given the circumstances, to make grow Pro Audio on Linux. I have nothing but absolute respect for Linux audio software developers!!

Which brings me to another thought. In the Linux realm, as it stands now with the problems developers have with supporting commercial software across large numbers of distros, it makes me consider WINE again. WINE, aside from the occasional regression, is continuously getting better, and continuously supporting more and more commercial applications. I don't expect this trend to change. I've (so far) avoided everything with WINE and focused on native applications. I've always felt that this is the most stable way, and that supporting native apps will continue to improve the situation as developers see that Linux consumers want commercial products on Linux. However, in light of what I am learning, I am also suspecting that while the number of proprietary applications will continue to grow (and the growth, I suspect will become progressively faster as well), I think it won't grow as fast as I'm wanting, and maybe I need to consider, for the time being, being willing to use WINE.

After all, to quote WINE's main page, WINE is:

"a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, macOS, & BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop."

There is almost no loss in speed, and compatibility improves daily.

I'm now of the opinion that we should freely use whatever is available for us, whether that be native apps from developers who code to best practice strategies for binary compatibility, containerization such as Flatpak, or compatibility layers such as WINE. As of now, there is not really a truly best solution.

Another thing that I have realized is that where the musician plans to obtain the bulk of his music-making tools (commercial vs open source) matters a great deal in what distro I would recommend to a new user. Right now, I am firmly coming to the belief that there are really only two distros worthy of consideration Ubuntu and Arch (and by way of Arch, I am including OSes binary compatible with Arch, such as EndeavourOS or Manjaro. If a user just wants to make music and doesn't want to deal with library compatibility problems, these are the best two choices. Ubuntu, for those who want to purchase proprietary plugins and software, and Arch for those who want the highest number of open source packages available on a distro for music creation. Granted, advanced users can pretty much use any Linux distro and pull the binaries and manually make the libraries work, but most music producers just want to make music and not worry about the details of the distro.

Over all, so very interesting articles and developer comments on the state of proprietary software in the Linux world!

[4duhwinnn]

Another way to look at such problems, is that there are actually so many solutions from disparate sources, that are good enough in actual use, that a massive concerted unified centralized effort to close the gap on dominating 'the desktop' is only going to make things marginally better, because the technology margin in the arena of finishing tasks, is not all that big.

One devil in the details is that kids all use mac and win systems at school, and apart from android icon plucking on phones, and a few tablets, see little need to learn a new (linux) system. They are immersed, with no need to dry off.

A second devil, is the myth of software without a $price tag, somehow being 'free'. A developer with a day job, who is also coding a side-job linux project is charging (time-is-money) their family and friends and community, the high price of lost family time, and beneficial personal involvements, and quite possibly their own long term physical and mental health will suffer. The passion for coding, like any other after-work activity, needs to be carefully managed. Even un-married singles fresh out of uni, need healthy friendships, and those take time to develope and maintain.

Conversely, can you imagine if the top 500 linux coders had been paid market rate wages by a pair of corporate giants, for 50 hours per week for the last 5 years, how great the linux systems they would produce might be?

(Or maybe we would just have Systems E, F, G, H, I, and J...following in the
footsteps of System D?)

[audiojunkie]

Originally Posted by 4duhwinnn

Another way to look at such problems, is that there are actually so many solutions from disparate sources, that are good enough in actual use, that a massive concerted unified centralized effort to close the gap on dominating 'the desktop' is only going to make things marginally better, because the technology margin in the arena of finishing tasks, is not all that big.

One devil in the details is that kids all use mac and win systems at school, and apart from android icon plucking on phones, and a few tablets, see little need to learn a new (linux) system. They are immersed, with no need to dry off.

A second devil, is the myth of software without a $price tag, somehow being 'free'. A developer with a day job, who is also coding a side-job linux project is charging (time-is-money) their family and friends and community, the high price of lost family time, and beneficial personal involvements, and quite possibly their own long term physical and mental health will suffer. The passion for coding, like any other after-work activity, needs to be carefully managed. Even un-married singles fresh out of uni, need healthy friendships, and those take time to develope and maintain.

Conversely, can you imagine if the top 500 linux coders had been paid market rate wages by a pair of corporate giants, for 50 hours per week for the last 5 years, how great the linux systems they would produce might be?

(Or maybe we would just have Systems E, F, G, H, I, and J...following in the
footsteps of System D?)

Very good points. I think it comes down to this: development for closed source commercial apps on Linux is difficult. We need to be really appreciative of those developers who actually develop commercial Software for linux. I’ve studied flatpak recently and it’s getting better. Those sites that argue about the bad things in flatpak have some nuggets of truth in them. However, I have found that it is a continuous work in progress and the project has much to do before they are ready. Apps are being released now, but the project is not complete and many of the problems that they argue about being bad are little by little being fixed. I am hopeful on that regard. Also those who have been arguing about libraries and the need to keep everything backwards compatible are correct, and this would be the correct way to do things. But 20 years have gone by and the problem there is not solved either so something has to be done. I think in the end linux is really open source friendly and not well prepared for close source. I think developers are aware and there are many fronts that are trying to solve the problem. In the meantime, we need to take whatever we can get to be thankful. That includes linux commercial developers who produce apps, and also WINE and yabridge. It’s getting better for Linux users year by year, but my eyes are now open to the realities of closed source commercial linux development.

[MiddleC]

I don't want to start a flame war, so ask and I shall delete.

I don't do linux desktop. I just don't. Tried many times over many years. I always - AWAYS - got tripped up on trivial things that drove me mad.

I've lived my professional life in *nix-land. From Solaris on SPARC through RHEL, through virtualization and replatforming, through porting of mission critical systems in many languages as well as maintenance and development of new code... For 30 years it has been my technical life. I have loved it, and wouldn't want it any other way.

But no. No desktop. I just don't have it in me any more to spend time recompiling anything I didn't personally write, and researching broken dependencies.

[Glennbo]

Originally Posted by MiddleC

But no. No desktop. I just don't have it in me any more to spend time recompiling anything I didn't personally write, and researching broken dependencies.

When do you have to start compiling things?

I've used Linux with the XFCE desktop for about three and a half years now, and I've never had to compile anything to make music with REAPER. Of the 40 tracks on my music page, about the first 30 were recorded in Linux with an XFCE desktop.

[shosty]

Originally Posted by MiddleC

But no. No desktop. I just don't have it in me any more to spend time recompiling anything I didn't personally write, and researching broken dependencies.

That's fair enough. But yeah, not sure why you would need to compile anything. If you want stable then there are options for music, AVLinux for example. I haven't had dependency problems for years to be honest, if you stick to ubuntu LTS then most stuff just works and there are PPAs if you need something newer.

[MiddleC]

Originally Posted by Glennbo

When do you have to start compiling things? I've used Linux with the XFCE desktop for about three and a half years now, and I've never had to compile anything to make music with REAPER.

I wasn't specifically talking about making music. Music is one slice of my personal computer needs.

But even if it wasn't about compiling, the last few times I tried to use linux for music, the fighting I had to do with ALSA and Jack was infuriating. Granted that was more than a couple of years ago.

I came the long way. My first linux edition was slackware back in the mid-90s. I fought the x-windows nightmare. I earned my stripes! I would try new desktops every 18 months or so, eventually abandoning them for some reason or another. I finally gave up some time in 2018.

Linux is not banished from my space. I still love it as an O/S. My NAS runs headless linux containers for the things I need.

[Glennbo]

Originally Posted by MiddleC

I wasn't specifically talking about making music. Music is one slice of my personal computer needs.

But even if it wasn't about compiling, the last few times I tried to use linux for music, the fighting I had to do with ALSA and Jack was infuriating. Granted that was more than a couple of years ago.

I came the long way. My first linux edition was slackware back in the mid-90s. I fought the x-windows nightmare. I earned my stripes! I would try new desktops every 18 months or so, eventually abandoning them for some reason or another. I finally gave up some time in 2018.

Linux is not banished from my space. I still love it as an O/S. My NAS runs headless linux containers for the things I need.

I first setup a Linux MythTV server in 2012. It's still running today, and has been upgraded every major release. When Windows 7 die date was six months in the future, I switched each of 6 other machines on my network over to Linux.

The best one was my wife's machine. I put a new SSD in with Linux, copied all her browser configuration and links, all her email and history, all her pictures, videos, and music, along with some Windows games. I still had the un-modified Windows 7 drive if something couldn't be made to work the same, but as it turned out, I never heard one complaint or problem. The only thing that came up was how to scan with an HP printer in Linux, and that was just knowing what app to use.

I used to be a programmer in a language made for Windows, and I've coded some stuff with it, and run a stock ticker I wrote in a Windows language on Linux in WINE. Out of six machines in my house all running Linux desktops, zero of them have been any more problematic than the Windows that ran on the same hardware before.

[MiddleC]

Glad to hear it. And I also know others with positive experiences similar to yours. If it "just works", it gets my seal of approval.

This industrial systems programmer and all round technical-type is writing this reply on an iPad.

[audiojunkie]

Originally Posted by MiddleC

I wasn't specifically talking about making music. Music is one slice of my personal computer needs.

But even if it wasn't about compiling, the last few times I tried to use linux for music, the fighting I had to do with ALSA and Jack was infuriating. Granted that was more than a couple of years ago.

I came the long way. My first linux edition was slackware back in the mid-90s. I fought the x-windows nightmare. I earned my stripes! I would try new desktops every 18 months or so, eventually abandoning them for some reason or another. I finally gave up some time in 2018.

Linux is not banished from my space. I still love it as an O/S. My NAS runs headless linux containers for the things I need.

I too have been using Linux for years and years. It's all I use at home. :-) However, in all the years I've used Linux, I have seen nothing but steady improvement, year after year. It's OK if you have no desire anymore to use Linux, but surely you agree that things have been getting better throughout the years. :-)

[audiojunkie]

Back to the topic...

I found this interesting two web pages:

http://flatkill.org/

https://theevilskeleton.gitlab.io/20...tkill-org.html

The first is an infamous anti-Flatpak page, and the second is a response from one of the Flatpak developers.

TL;DR -- Yes, Flatpak has issues, but that's because they are not done with creating the program. It's in a "usable" state. It works, but it does have security issues. However, with each and every update, things are getting better. Is Flatpak the best ideal solution? Probably not. The ideal solution would be for the Linux world to work on library compatibility and stop Dependency Hell and inter-distro incompatibilities. But the Flatpak team are realists--the Linux world has had 20 years to solve the problem, but they are as incompatible now as they were then. Flatpak's solution is to accept that all of the distros will continue to remain incompatible with each other's libraries, and then containerize applications along with their libraries. This allows Flatpak to work with every distro with a single package. It solves the problem of dealing with incompatible linux distros, and allows software developers to only have to create one package that works for everyone. Are there drawbacks to the technology? Of course! Flatpak is not the ideal solution--the ideal solution doesn't look like it's coming any time soon--if ever. Therefore, something has to be done, and Flatpak is their answer. The biggest drawback that I see is the space wasted from packaged libraries. But they are working on that as well. The security issues are there, but they are being fixed as things progress. The other issue is a little added CPU overhead.

In my opinion, I see Flatpak as a "developing technology" that is a work in progress and not complete. Is it usable? Yes, but it's not complete. Accepting it for what it is and what it is not is the key. I'm personally grateful for Flatpak, as I am grateful for WINE+ yabridge. I am also grateful for plugin developers who do the best they can to create closed source packages that support Linux natively and have as broad as possible distro support.

I think this little journey has been very valuable in helping me to understand realistically what the current state of commercialized (closed source) apps are for Linux, and the direction the industry is working towards. I don't know if plugins can even be used with Flatpak yet, so Flatpak may not even be an option for developers of plugins at this time. What I do know, is that I'm more grateful than ever to Linux developers.

I think because of the unlikelihood of distro developers becoming compatible with each other, containerization (and probably containerization leader, Flatpak), a second best solution, is probably, eventually going to become the best possible solution, considering the alternatives.

Appimage is good too, but I suspect it is too limited in its abilities to meet all commercial needs. Snaps, are totally under the control of Canonical, which I don't like. Canonical also has a monopoly on their Snap store, which I don't like. Snaps are really not an option for me.

So, in the end, I think we as Linux users need to accept Linux for what it currently is. Linux currently is great for open source development, but solutions are being worked on to improve the situation for closed-source (commercialized) development.

In the past, I have not embraced the WINE+yabridge solution, because I preferred native only apps. I still do prefer native only apps, and just bought several plugins for Black Friday. :-) But as a realist, I recognize that commercial development is very hard for Linux developers, and that is going to hold back commercial development in today's Linux world. I am absolutly grateful for those developers who continue to make native commercial apps! At the same time, I will likely embrace the WINE+yabridge solution as well.

In fact, there is a good argument for continuing with WINE+yabridge: WINE is continually getting better--year after year. WINDOWS apps in WINE will never cause any dependency problems. More are more apps are working with it. In fact, I'd bet that some Windows XP era plugins may now run better on Linux + WINE than on Windows 11. I have CDs of collected VST favorites that I may pull out and try. I still prefer native plugins, but this, in my opinion, is going to be a valid source of plugins for at least the next few years.

[Glennbo]

When I was still working in IT and as a Windows software developer, creating installers that checked the system being installed to and having the installer install components not found or upgrade ones that were found, but older versions was a tangled mess of endless possibilities to code the installer for.

I would sometimes farm out modules to a genius C++ coder I worked with, who would create for me Windows executable files that had no dependencies or support files needed, so I could just drop the single .exe into place on customer's machines and it worked.

I guess what we devised for some of the more tech oriented stuff (like Windows Image Acquisition and image displaying) was something similar to how AppImage files work in Linux. Not having to deal with any dependencies by using a single .exe file kept us from having to troubleshoot our customer's machines.

[audiojunkie]

Originally Posted by MiddleC

I wasn't specifically talking about making music. Music is one slice of my personal computer needs.

But even if it wasn't about compiling, the last few times I tried to use linux for music, the fighting I had to do with ALSA and Jack was infuriating. Granted that was more than a couple of years ago.

I came the long way. My first linux edition was slackware back in the mid-90s. I fought the x-windows nightmare. I earned my stripes! I would try new desktops every 18 months or so, eventually abandoning them for some reason or another. I finally gave up some time in 2018.

Linux is not banished from my space. I still love it as an O/S. My NAS runs headless linux containers for the things I need.

While JACK isn't going away (it's being integrated via PipeWire), there is a growing consensus that JACK is the old way of making music on Linux. Back before there was VST for Linux and LV2, there was JACK. Nowadays, we have plugins and hosts like Windows and Mac. JACK is a viable tool that is there for those who what to use it, but it's not the only way, and really isn't necessary for most users. Music making on Linux is much easier when you use Alsa and a host that supports plugins--like Reaper. :-)

My advice to you, if you ever want to try Linux in its current state, is to use Reaper, and some plugins, and run stratight through ALSA. It will simplify your life a great deal, and make music making enjoyable again. :-)

[audiojunkie]

Originally Posted by Glennbo

When I was still working in IT and as a Windows software developer, creating installers that checked the system being installed to and having the installer install components not found or upgrade ones that were found, but older versions was a tangled mess of endless possibilities to code the installer for.

I would sometimes farm out modules to a genius C++ coder I worked with, who would create for me Windows executable files that had no dependencies or support files needed, so I could just drop the single .exe into place on customer's machines and it worked.

I guess what we devised for some of the more tech oriented stuff (like Windows Image Acquisition and image displaying) was something similar to how AppImage files work in Linux. Not having to deal with any dependencies by using a single .exe file kept us from having to troubleshoot our customer's machines.

I like App images too, but I don't think they will cover the extensive scope that is planned for Flatpak. Flatpak is a work in progress and is not feature complete. It will continue to get better. I don't know if Appimage has further plans to extend its abilities or not. I'll just have to watch with interest to see how things go. I won't do Snaps, so that's out of the picture for me. :-) However, I'm convinced now, understanding the library incompatability issue that keeps various distros incompatible, that containerization, although not the ideal solution, may be the only solution left for closed source and commercial applications across all distros if one wants to have true complete compatibility.

[Glennbo]

Originally Posted by audiojunkie

I like App images too, but I don't think they will cover the extensive scope that is planned for Flatpak. Flatpak is a work in progress and is not feature complete. It will continue to get better. I don't know if Appimage has further plans to extend its abilities or not. I'll just have to watch with interest to see how things go. I won't do Snaps, so that's out of the picture for me. :-) However, I'm convinced now, understanding the library incompatability issue that keeps various distros incompatible, that containerization, although not the ideal solution, may be the only solution left for closed source and commercial applications across all distros if one wants to have true complete compatibility.

I have to wonder what big issues people are having with Linux as it is. Almost every application I used in Windows also has a native Linux version, and for the two or three apps that didn't have native Linux versions, I found equivalent native Linux replacements.

I mean, I switched my wife's machine out from Windows 7 to Xubuntu Linux while she was at work, without letting her know in advance, and I'm still here to tell the tale. That ought to say a lot about Linux right there. She has an art studio and uses the computer to generate tags, scan stuff, convert to pdf, print flyers, and other stuff like that, and it all works so similar to Windows that I never heard one complaint.

Every Linux plugin that I've purchased continue to work, even when I switched from Xubuntu to Manjaro. All my bridged Windows plugins also work fine before and after the switch.