How to avoid high latency near OOM situation?2019 Community Moderator ElectionHow to disable one CPUOOM killer doesn't work properly, leads to a frozen OSRestart system if it runs out of memory?How to set OOM killer adjustments for daemons permanently?Configure the OOM to avoid having the system grind to a haltceph fs apply latency too high resulting in high load in VMsLinux using whole swap, becoming unresponsive while there is plenty of free RAMvarnishd killed by OOM, How can I configure the varnishd to avoid this problem?Debugging high (near total) CPU/memory usage of “Web Content” application on Linux MintHow to make OOM killer log into /var/log/messages when it kills any process?How can the OOM killer kill ulimit(ed) process?How to avoid a situation where the system is too out-of-memory to even show or kill processes?How to reduce OOM score for X11 with systemd
How do you justify more code being written by following clean code practices?
Does convergence of polynomials imply that of its coefficients?
Why didn't Héctor fade away after this character died in the movie Coco?
How do researchers send unsolicited emails asking for feedback on their works?
PTIJ: Which Dr. Seuss books should one obtain?
Is VPN a layer 3 concept?
How to test the sharpness of a knife?
TDE Master Key Rotation
Do I need to convey a moral for each of my blog post?
Do I need an EFI partition for each 18.04 ubuntu I have on my HD?
Homology of the fiber
What kind of footwear is suitable for walking in micro gravity environment?
Is there any common country to visit for uk and schengen visa?
Norwegian Refugee travel document
Extraneous elements in "Europe countries" list
Pre-Employment Background Check With Consent For Future Checks
If I cast the Enlarge/Reduce spell on an arrow, what weapon could it count as?
Would mining huge amounts of resources on the Moon change its orbit?
Print a physical multiplication table
Error in master's thesis, I do not know what to do
How can an organ that provides biological immortality be unable to regenerate?
Why doesn't the fusion process of the sun speed up?
Is "inadequate referencing" a euphemism for plagiarism?
What are rules for concealing thieves tools (or items in general)?
How to avoid high latency near OOM situation?
2019 Community Moderator ElectionHow to disable one CPUOOM killer doesn't work properly, leads to a frozen OSRestart system if it runs out of memory?How to set OOM killer adjustments for daemons permanently?Configure the OOM to avoid having the system grind to a haltceph fs apply latency too high resulting in high load in VMsLinux using whole swap, becoming unresponsive while there is plenty of free RAMvarnishd killed by OOM, How can I configure the varnishd to avoid this problem?Debugging high (near total) CPU/memory usage of “Web Content” application on Linux MintHow to make OOM killer log into /var/log/messages when it kills any process?How can the OOM killer kill ulimit(ed) process?How to avoid a situation where the system is too out-of-memory to even show or kill processes?How to reduce OOM score for X11 with systemd
Minimal test case when Linux system does not have swap (or run sudo swapoff -a before testing). Run following bash one-liner as normal user:
while true; do date; nice -20 stress --vm-bytes $(awk '/MemAvailable/printf "%dn", $2 + 4000;' < /proc/meminfo)k --vm-keep -m 1 --timeout 10s; sleep 5s; done
and run following bash one-liner with high priority root shell (e.g. sudo nice -n -19 bash):
while true; do NS=$(date '+%N' | sed 's/^0*//'); let "S=998000000 - $NS"; S=$(( S > 0 ? S : 0)); LC_ALL=C sleep "0.$S"; date --iso=ns; done
The high priority process is supposed to run date every second as accurately as possible. However, even if this process is running with priority -19, the background process running on priority 20 is able to cause major delays. It seems that there's no limit for the latency induced by the low priority background process because higher delays can be activated by increasing the stress --timeout value.
Is there a way to limit maximum latency and automatically kill the stress if needed to accomplish that? Increasing /proc/sys/vm/user_reserve_kbytes or /proc/sys/vm/admin_reserve_kbytes or /proc/sys/vm/min_free_kbytes does not seem to help.
linux out-of-memory
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
add a comment |
Minimal test case when Linux system does not have swap (or run sudo swapoff -a before testing). Run following bash one-liner as normal user:
while true; do date; nice -20 stress --vm-bytes $(awk '/MemAvailable/printf "%dn", $2 + 4000;' < /proc/meminfo)k --vm-keep -m 1 --timeout 10s; sleep 5s; done
and run following bash one-liner with high priority root shell (e.g. sudo nice -n -19 bash):
while true; do NS=$(date '+%N' | sed 's/^0*//'); let "S=998000000 - $NS"; S=$(( S > 0 ? S : 0)); LC_ALL=C sleep "0.$S"; date --iso=ns; done
The high priority process is supposed to run date every second as accurately as possible. However, even if this process is running with priority -19, the background process running on priority 20 is able to cause major delays. It seems that there's no limit for the latency induced by the low priority background process because higher delays can be activated by increasing the stress --timeout value.
Is there a way to limit maximum latency and automatically kill the stress if needed to accomplish that? Increasing /proc/sys/vm/user_reserve_kbytes or /proc/sys/vm/admin_reserve_kbytes or /proc/sys/vm/min_free_kbytes does not seem to help.
linux out-of-memory
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such asdate. As far as I can see it, the problem is memory starvation, not CPU starvation.
– Mikko Rantalainen
Feb 10 '18 at 19:15
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19
add a comment |
Minimal test case when Linux system does not have swap (or run sudo swapoff -a before testing). Run following bash one-liner as normal user:
while true; do date; nice -20 stress --vm-bytes $(awk '/MemAvailable/printf "%dn", $2 + 4000;' < /proc/meminfo)k --vm-keep -m 1 --timeout 10s; sleep 5s; done
and run following bash one-liner with high priority root shell (e.g. sudo nice -n -19 bash):
while true; do NS=$(date '+%N' | sed 's/^0*//'); let "S=998000000 - $NS"; S=$(( S > 0 ? S : 0)); LC_ALL=C sleep "0.$S"; date --iso=ns; done
The high priority process is supposed to run date every second as accurately as possible. However, even if this process is running with priority -19, the background process running on priority 20 is able to cause major delays. It seems that there's no limit for the latency induced by the low priority background process because higher delays can be activated by increasing the stress --timeout value.
Is there a way to limit maximum latency and automatically kill the stress if needed to accomplish that? Increasing /proc/sys/vm/user_reserve_kbytes or /proc/sys/vm/admin_reserve_kbytes or /proc/sys/vm/min_free_kbytes does not seem to help.
linux out-of-memory
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
Minimal test case when Linux system does not have swap (or run sudo swapoff -a before testing). Run following bash one-liner as normal user:
while true; do date; nice -20 stress --vm-bytes $(awk '/MemAvailable/printf "%dn", $2 + 4000;' < /proc/meminfo)k --vm-keep -m 1 --timeout 10s; sleep 5s; done
and run following bash one-liner with high priority root shell (e.g. sudo nice -n -19 bash):
while true; do NS=$(date '+%N' | sed 's/^0*//'); let "S=998000000 - $NS"; S=$(( S > 0 ? S : 0)); LC_ALL=C sleep "0.$S"; date --iso=ns; done
The high priority process is supposed to run date every second as accurately as possible. However, even if this process is running with priority -19, the background process running on priority 20 is able to cause major delays. It seems that there's no limit for the latency induced by the low priority background process because higher delays can be activated by increasing the stress --timeout value.
Is there a way to limit maximum latency and automatically kill the stress if needed to accomplish that? Increasing /proc/sys/vm/user_reserve_kbytes or /proc/sys/vm/admin_reserve_kbytes or /proc/sys/vm/min_free_kbytes does not seem to help.
linux out-of-memory
linux out-of-memory
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
asked Feb 10 '18 at 17:59
Mikko RantalainenMikko Rantalainen
1,542916
1,542916
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such asdate. As far as I can see it, the problem is memory starvation, not CPU starvation.
– Mikko Rantalainen
Feb 10 '18 at 19:15
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19
add a comment |
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such asdate. As far as I can see it, the problem is memory starvation, not CPU starvation.
– Mikko Rantalainen
Feb 10 '18 at 19:15
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such as
date. As far as I can see it, the problem is memory starvation, not CPU starvation.– Mikko Rantalainen
Feb 10 '18 at 19:15
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such as
date. As far as I can see it, the problem is memory starvation, not CPU starvation.– Mikko Rantalainen
Feb 10 '18 at 19:15
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19
add a comment |
2 Answers
2
active
oldest
votes
Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).
* or even suggesting a better way?
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoidingkswapdbut real time benchmarking could be different.
– Mikko Rantalainen
Aug 30 '18 at 9:52
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
@MikkoRantalainen it is the sameMemory allocation stall watchdogpatch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!
– Marcus Linsner
Aug 31 '18 at 11:03
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
|
show 11 more comments
There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:
earlyoom, probably good enough for desktop/laptop computers
nohang, a more configurable solution- Facebook's solution oomd for their own servers.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f423261%2fhow-to-avoid-high-latency-near-oom-situation%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).
* or even suggesting a better way?
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoidingkswapdbut real time benchmarking could be different.
– Mikko Rantalainen
Aug 30 '18 at 9:52
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
@MikkoRantalainen it is the sameMemory allocation stall watchdogpatch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!
– Marcus Linsner
Aug 31 '18 at 11:03
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
|
show 11 more comments
Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).
* or even suggesting a better way?
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoidingkswapdbut real time benchmarking could be different.
– Mikko Rantalainen
Aug 30 '18 at 9:52
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
@MikkoRantalainen it is the sameMemory allocation stall watchdogpatch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!
– Marcus Linsner
Aug 31 '18 at 11:03
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
|
show 11 more comments
Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).
* or even suggesting a better way?
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).
* or even suggesting a better way?
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
answered Aug 29 '18 at 11:20
Marcus LinsnerMarcus Linsner
31617
31617
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoidingkswapdbut real time benchmarking could be different.
– Mikko Rantalainen
Aug 30 '18 at 9:52
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
@MikkoRantalainen it is the sameMemory allocation stall watchdogpatch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!
– Marcus Linsner
Aug 31 '18 at 11:03
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
|
show 11 more comments
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoidingkswapdbut real time benchmarking could be different.
– Mikko Rantalainen
Aug 30 '18 at 9:52
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
@MikkoRantalainen it is the sameMemory allocation stall watchdogpatch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!
– Marcus Linsner
Aug 31 '18 at 11:03
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
1
1
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoiding
kswapd but real time benchmarking could be different.– Mikko Rantalainen
Aug 30 '18 at 9:52
Interesting patch. I think it's a bit heavy handed but triggering OOM Killer sooner is definitely the correct behavior. I guess triggering on increased mm allocation latency is better than avoiding
kswapd but real time benchmarking could be different.– Mikko Rantalainen
Aug 30 '18 at 9:52
1
1
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
Were you able to get any dmesg output by using the patch that you mentioned in a comment to your question ( lkml.org/lkml/2017/5/20/186 ) which I adapted to 4.18.5 here: github.com/constantoverride/qubes-linux-kernel/blob/… I ask because I got no output from it (unless I missed it? or something) and I think I should get some output according to your previous comment: if mm allocation latency is at play.
– Marcus Linsner
Aug 30 '18 at 13:03
1
1
@MikkoRantalainen it is the same
Memory allocation stall watchdog patch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!– Marcus Linsner
Aug 31 '18 at 11:03
@MikkoRantalainen it is the same
Memory allocation stall watchdog patch that you mentioned in a comment in OP and that's where I originally found it from. But the one you just linked is older (15 March 2017 now, vs 20 May 2017 in OP comment). It is a good patch and I'm keeping both(it and mine) currently applied. Cheers!– Marcus Linsner
Aug 31 '18 at 11:03
1
1
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
The kernel does not have enough history per page to do really clever stuff. As far as I know, it basically knows if a page has been loaded back from swap sometimes in history but it has no idea how long ago that happened. And I'm not sure if it even remembers that it had to re-read the file from the disk (in case of executable file).
– Mikko Rantalainen
Sep 2 '18 at 15:36
1
1
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
you're right that if one sets up a hard malloc timeout trigger for OOM Killer, the system may end up killing a process even with half the memory still free. It should not happen in normal case but if you're not running PREEMPT or RT kernel, I guess it could happen because of locking between different kernel threads if multiple user processes use lots of CPU. However, if you're looking for guaranteed latency, killing processes even with 50% free may be exactly what you want!
– Mikko Rantalainen
Sep 2 '18 at 15:46
|
show 11 more comments
There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:
earlyoom, probably good enough for desktop/laptop computers
nohang, a more configurable solution- Facebook's solution oomd for their own servers.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:
earlyoom, probably good enough for desktop/laptop computers
nohang, a more configurable solution- Facebook's solution oomd for their own servers.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:
earlyoom, probably good enough for desktop/laptop computers
nohang, a more configurable solution- Facebook's solution oomd for their own servers.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:
earlyoom, probably good enough for desktop/laptop computers
nohang, a more configurable solution- Facebook's solution oomd for their own servers.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 11 hours ago
nat choufnat chouf
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 11 hours ago
nat choufnat chouf
101
answered 11 hours ago
nat choufnat chouf
101
101
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
nat chouf is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f423261%2fhow-to-avoid-high-latency-near-oom-situation%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
CPU pinning to at least the most priority process might somewhat mitigate it. used it with some success in the past for similar real-world situations. have a look at unix.stackexchange.com/questions/417672/how-to-disable-cpu/…
– Rui F Ribeiro
Feb 10 '18 at 18:01
I believe that the latency is caused by near OOM situation and the high priority process still needs to launch small new processes. Pinning to another CPU does not help if there is not enough RAM to start even a small new process such as
date. As far as I can see it, the problem is memory starvation, not CPU starvation.– Mikko Rantalainen
Feb 10 '18 at 19:15
When you have one, you usually end up having another. Granted, there are situations where it would not help. Depending on the situation, having a controlled reboot under a watchdog might be preferable of starting killing things remotely. unix.stackexchange.com/questions/366973/…
– Rui F Ribeiro
Feb 10 '18 at 19:17
I think I'm hitting some kernel bug. lkml.org/lkml/2017/5/20/186
– Mikko Rantalainen
Feb 11 '18 at 18:19