> A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal...
That's not encouraging.
---
Another approach: Can we make the kernel vulns less important? Has anyone had luck moving more things to run under gvisor or firecracker or such?
ordu 1 hours ago [-]
> Like, running emerge -u @world on a regular basis
You can run emerge -u sys-kernel/whatever-kernel-u-use, maybe followed by `cd /usr/src/linux; make bzImage modules install modules_install...` well, probably you'd use genkernel or something like that, instead of hand-crafted scripts.
The point is: `emerge -u @world` can run into issues esp. if you customized a lot, it can't be automated fully, but I've never run into any issues with updating the kernel, and it can be automated.
It is not so hard to upgrade kernel, the issue is with the reboot you need do automate. Or with live patching, which doesn't seem encouraging, as you say.
himata4113 3 hours ago [-]
Expanding on gentoo's recommendations:
I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already have that at a lower level with normal application updates.
Canonical has thus far proved that it can be safe, but they're also a massive organization that is locking this feature for $200/yr for any commercial use.
It would be neat if such patches could retroactively replace tagged functions that have identical sematics so that means it would automatically get backported without extra effort from the maintainers.
nine_k 2 hours ago [-]
Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro.
$200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce.
I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.
graemep 2 hours ago [-]
Public goods are not generally funded by large corporations.
throwa356262 38 minutes ago [-]
I dont belive in live patching unless you are AWS.
But I absolutely belive we should have a method for changing kernel configuration (e.g. kernel module blacklists) and syscall firewalls and alike.
c7b 2 hours ago [-]
After the npm supply chain attacks people suggested automating delays before installing updates, now we're talking about automating update delivery... I'm afraid there won't be any easy or quick fix after decades of treating security as an afterthought.
ordu 1 hours ago [-]
Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least.
Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed.
Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible.
Whenever you read about an incredibly unlucky criminal, there's a chance that the unlucky event is a parallel construction to the classified real reason why they were caught. Not sure how exactly that would have worked in this case.
TacticalCoder 2 hours ago [-]
> I wonder if we should just universally accept that live patching should become part of the linux kernel?
I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching.
One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use.
Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules.
We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory.
In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.
yjftsjthsd-h 2 hours ago [-]
> we're now into the "users installing our distro are getting hacked left and right" territory
Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild?
(Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
bombcar 2 hours ago [-]
It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module.
Perhaps we should tend toward the first.
yjftsjthsd-h 2 hours ago [-]
It seems like a reasonable middle ground for most distros is to put things in kernel modules, but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver`; if you do need it, just install the package and it just works without needing to compile anything and you get automatic updates and everything.
For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.
romaniv 44 minutes ago [-]
Clearly, the future is LLM-generated patches that get instantly vibecoded and installed on all machines without any human review. In fact, this is such a good idea that it should be illegal and impossible to run your computer without being connected to such a system. There are no other alternatives. /sarcasm
clircle 3 hours ago [-]
Is Gentoo an outlier or do all Linux distributions deal with this problem?
ordu 55 minutes ago [-]
All of them need to do it. There maybe differences, like different number of versions of kernel supported, so less of backporting, but still distros have to provide fixed kernels.
With Gentoo I believe it is more fun, because of all the options gentoo provides out of a box. More kernels, more work to do.
Not all these directories are different kernel packages, but anything with -kernel or -sources at the end is.
c0balt 3 hours ago [-]
Many distros deal with the problem of learning about these issues the same time as the public. Some have fast track processes to ensure patches can get into their stable/rolling releases but it is still a lot of work (especially as kernel updates usually mean that automatic updates won't fully shipped you (without alsp automatically rebooting after an update)).
Like, running emerge -u @world on a regular basis, or ...
/me searches
Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says,
> A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal...
That's not encouraging.
---
Another approach: Can we make the kernel vulns less important? Has anyone had luck moving more things to run under gvisor or firecracker or such?
You can run emerge -u sys-kernel/whatever-kernel-u-use, maybe followed by `cd /usr/src/linux; make bzImage modules install modules_install...` well, probably you'd use genkernel or something like that, instead of hand-crafted scripts.
The point is: `emerge -u @world` can run into issues esp. if you customized a lot, it can't be automated fully, but I've never run into any issues with updating the kernel, and it can be automated.
It is not so hard to upgrade kernel, the issue is with the reboot you need do automate. Or with live patching, which doesn't seem encouraging, as you say.
I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already have that at a lower level with normal application updates.
Canonical has thus far proved that it can be safe, but they're also a massive organization that is locking this feature for $200/yr for any commercial use.
It would be neat if such patches could retroactively replace tagged functions that have identical sematics so that means it would automatically get backported without extra effort from the maintainers.
$200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce.
I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.
But I absolutely belive we should have a method for changing kernel configuration (e.g. kernel module blacklists) and syscall firewalls and alike.
Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed.
Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible.
[1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor
I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching.
One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use.
Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules.
We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory.
In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.
Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild?
(Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
Perhaps we should tend toward the first.
For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.
With Gentoo I believe it is more fun, because of all the options gentoo provides out of a box. More kernels, more work to do.
Not all these directories are different kernel packages, but anything with -kernel or -sources at the end is.