Ygtjki

Multi tool use

Return the Closest Prime Number

Would a high gravity rocky planet be guaranteed to have an atmosphere?

Is expanding the research of a group into machine learning as a PhD student risky?

How can I get through very long and very dry, but also very useful technical documents when learning a new tool?

I'm in charge of equipment buying but no one's ever happy with what I choose. How to fix this?

Gears on left are inverse to gears on right?

Are student evaluations of teaching assistants read by others in the faculty?

Failed to fetch jessie backports repository

Opposite of a diet

Do sorcerers' Subtle Spells require a skill check to be unseen?

Is the destination of a commercial flight important for the pilot?

Go Pregnant or Go Home

How to write papers efficiently when English isn't my first language?

Do the temporary hit points from the Battlerager barbarian's Reckless Abandon stack if I make multiple attacks on my turn?

Two monoidal structures and copowering

Why not increase contact surface when reentering the atmosphere?

Crossing the line between justified force and brutality

Is there a korbon needed for conversion?

Trouble understanding the speech of overseas colleagues

Unreliable Magic - Is it worth it?

Applicability of Single Responsibility Principle

What is the intuitive meaning of having a linear relationship between the logs of two variables?

How to run a prison with the smallest amount of guards?

What is the difference between "behavior" and "behaviour"?

gnu parallel exit process with timeout

Using GNU Parallel With SplitWhen GNU parallel exit，my program also failConcurrency problems with GNU parallelgnu parallel with no argument scriptBASH: parallel runGNU Parallel to process multiple files in parallel in Unix with bashKeeping dirs in order with GNU ParallelGNU parallel with for loop?Copying & Renaming Files with GNU Parallelgnu parallel with bash array

1

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing:

while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

 /usr/bin/time -f "$timefmt" 
 printf '%sn' "$ids[@]" | parallel --jobs 0 recon-all -s . -all - 
 qcache -parallel -openmp 8

 n=$(( n + 1 ))
 ids=( "$all_ids[@]:n*4:4" ) # pick out the next eight IDs
done

and some patients in recon-all process inside parallel couldn't be completed for some reasons (could run several days, which abnormal).
Could I limit the runtime inside parallel for 9 hours, so the command will run another group in the cycle?

bash gnu-parallel

share|improve this question

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Not really sure why you want to run in batches of 8. Why not just use -j8 and run all ids with 8 running constantly?
  
  – Ole Tange
  13 hours ago
  
  

  
  
  
  

add a comment | 

1

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing:

while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

 /usr/bin/time -f "$timefmt" 
 printf '%sn' "$ids[@]" | parallel --jobs 0 recon-all -s . -all - 
 qcache -parallel -openmp 8

 n=$(( n + 1 ))
 ids=( "$all_ids[@]:n*4:4" ) # pick out the next eight IDs
done

and some patients in recon-all process inside parallel couldn't be completed for some reasons (could run several days, which abnormal).
Could I limit the runtime inside parallel for 9 hours, so the command will run another group in the cycle?

bash gnu-parallel

share|improve this question

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Not really sure why you want to run in batches of 8. Why not just use -j8 and run all ids with 8 running constantly?
  
  – Ole Tange
  13 hours ago
  
  

  
  
  
  

add a comment | 

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing:

while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

 /usr/bin/time -f "$timefmt" 
 printf '%sn' "$ids[@]" | parallel --jobs 0 recon-all -s . -all - 
 qcache -parallel -openmp 8

 n=$(( n + 1 ))
 ids=( "$all_ids[@]:n*4:4" ) # pick out the next eight IDs
done

and some patients in recon-all process inside parallel couldn't be completed for some reasons (could run several days, which abnormal).
Could I limit the runtime inside parallel for 9 hours, so the command will run another group in the cycle?

bash gnu-parallel

share|improve this question

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

 /usr/bin/time -f "$timefmt" 
 printf '%sn' "$ids[@]" | parallel --jobs 0 recon-all -s . -all - 
 qcache -parallel -openmp 8

 n=$(( n + 1 ))
 ids=( "$all_ids[@]:n*4:4" ) # pick out the next eight IDs
done

share|improve this question

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

share|improve this question

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

edited yesterday

Jeff Schaller♦

44k1161142

edited yesterday

Jeff Schaller♦

44k1161142

asked yesterday

RelyativistRelyativist

296

asked yesterday

RelyativistRelyativist

296

1 Answer
1

active

oldest

votes

0

You are looking for --timeout.

You can do --timeout 9h or you can do --timeout 1000%. The last will measure how long the median time is for a job to succeed, and given the median it will compute a timeout that is 1000% of the median run time.

The neat thing about using a percentage is that if the compute program gets faster or slower for the normal case, you will not need to change the timeout.

See it in action:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3
# Compute program gets 10 times faster
parallel --timeout 300% 'sleep =$_ /= 10 =; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3

The median (not average) runtime is measured as the median of the succesfully completed jobs (though minimum 3). So if you have 8 jobs with job 5 being infinite, it will get killed when the runtime hits the percentage of the median timeout:

parallel --timeout 300% 'sleep ; echo ' ::: 1 2 1 2 100 2 1 2

This also works if the first job is the one that is stuck:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 1 2 1 2 1 2

The only situation it does not work is if all jobslots are stuck on their first job:

parallel -j4 --timeout 300% 'sleep ; echo ' ::: 100 100 100 100 1 2 1 2

share|improve this answer

edited 13 hours ago

answered yesterday

Ole TangeOle Tange

12.9k1457107

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The problem is that issues can occur on first pack of patients, so 7 of 8 done, but e.g 5th have infinite runtime (because it stuck at "CORRECTING DEFECT 5 for example), so the parallel can't measure average processing time to --timeout with %
  
  – Relyativist
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  In most cases it can. See edit.
  
  – Ole Tange
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  gnu parallel doesn't work with 9h right? you should write it in seconds? 6000?
  
  – Relyativist
  11 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Of course it works with 9h.
  
  – Ole Tange
  7 hours ago
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f508693%2fgnu-parallel-exit-process-with-timeout%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

You are looking for --timeout.

You can do --timeout 9h or you can do --timeout 1000%. The last will measure how long the median time is for a job to succeed, and given the median it will compute a timeout that is 1000% of the median run time.

The neat thing about using a percentage is that if the compute program gets faster or slower for the normal case, you will not need to change the timeout.

See it in action:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3
# Compute program gets 10 times faster
parallel --timeout 300% 'sleep =$_ /= 10 =; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3

The median (not average) runtime is measured as the median of the succesfully completed jobs (though minimum 3). So if you have 8 jobs with job 5 being infinite, it will get killed when the runtime hits the percentage of the median timeout:

parallel --timeout 300% 'sleep ; echo ' ::: 1 2 1 2 100 2 1 2

This also works if the first job is the one that is stuck:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 1 2 1 2 1 2

The only situation it does not work is if all jobslots are stuck on their first job:

parallel -j4 --timeout 300% 'sleep ; echo ' ::: 100 100 100 100 1 2 1 2

share|improve this answer

edited 13 hours ago

answered yesterday

Ole TangeOle Tange

12.9k1457107

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The problem is that issues can occur on first pack of patients, so 7 of 8 done, but e.g 5th have infinite runtime (because it stuck at "CORRECTING DEFECT 5 for example), so the parallel can't measure average processing time to --timeout with %
  
  – Relyativist
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  In most cases it can. See edit.
  
  – Ole Tange
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  gnu parallel doesn't work with 9h right? you should write it in seconds? 6000?
  
  – Relyativist
  11 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Of course it works with 9h.
  
  – Ole Tange
  7 hours ago
  
  

  
  
  
  

add a comment | 

0

You are looking for --timeout.

You can do --timeout 9h or you can do --timeout 1000%. The last will measure how long the median time is for a job to succeed, and given the median it will compute a timeout that is 1000% of the median run time.

The neat thing about using a percentage is that if the compute program gets faster or slower for the normal case, you will not need to change the timeout.

See it in action:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3
# Compute program gets 10 times faster
parallel --timeout 300% 'sleep =$_ /= 10 =; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3

The median (not average) runtime is measured as the median of the succesfully completed jobs (though minimum 3). So if you have 8 jobs with job 5 being infinite, it will get killed when the runtime hits the percentage of the median timeout:

parallel --timeout 300% 'sleep ; echo ' ::: 1 2 1 2 100 2 1 2

This also works if the first job is the one that is stuck:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 1 2 1 2 1 2

The only situation it does not work is if all jobslots are stuck on their first job:

parallel -j4 --timeout 300% 'sleep ; echo ' ::: 100 100 100 100 1 2 1 2

share|improve this answer

edited 13 hours ago

answered yesterday

Ole TangeOle Tange

12.9k1457107

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The problem is that issues can occur on first pack of patients, so 7 of 8 done, but e.g 5th have infinite runtime (because it stuck at "CORRECTING DEFECT 5 for example), so the parallel can't measure average processing time to --timeout with %
  
  – Relyativist
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  In most cases it can. See edit.
  
  – Ole Tange
  13 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  gnu parallel doesn't work with 9h right? you should write it in seconds? 6000?
  
  – Relyativist
  11 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Of course it works with 9h.
  
  – Ole Tange
  7 hours ago
  
  

  
  
  
  

add a comment | 

You are looking for --timeout.

You can do --timeout 9h or you can do --timeout 1000%. The last will measure how long the median time is for a job to succeed, and given the median it will compute a timeout that is 1000% of the median run time.

The neat thing about using a percentage is that if the compute program gets faster or slower for the normal case, you will not need to change the timeout.

See it in action:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3
# Compute program gets 10 times faster
parallel --timeout 300% 'sleep =$_ /= 10 =; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3

The median (not average) runtime is measured as the median of the succesfully completed jobs (though minimum 3). So if you have 8 jobs with job 5 being infinite, it will get killed when the runtime hits the percentage of the median timeout:

parallel --timeout 300% 'sleep ; echo ' ::: 1 2 1 2 100 2 1 2

This also works if the first job is the one that is stuck:

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 1 2 1 2 1 2

The only situation it does not work is if all jobslots are stuck on their first job:

parallel -j4 --timeout 300% 'sleep ; echo ' ::: 100 100 100 100 1 2 1 2

share|improve this answer

edited 13 hours ago

answered yesterday

Ole TangeOle Tange

12.9k1457107

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3
# Compute program gets 10 times faster
parallel --timeout 300% 'sleep =$_ /= 10 =; echo ' ::: 100 2 3 1 50 2 3 1 2 1 3 2 1 4 2 1 2 3

parallel --timeout 300% 'sleep ; echo ' ::: 1 2 1 2 100 2 1 2

parallel --timeout 300% 'sleep ; echo ' ::: 100 2 1 2 1 2 1 2

parallel -j4 --timeout 300% 'sleep ; echo ' ::: 100 100 100 100 1 2 1 2

share|improve this answer

edited 13 hours ago

answered yesterday

Ole TangeOle Tange

12.9k1457107

answered yesterday

Ole TangeOle Tange

12.9k1457107

answered yesterday

Ole TangeOle Tange

12.9k1457107