Ygtjki

Lay out the Carpet

Trouble understanding the speech of overseas colleagues

Tiptoe or tiphoof? Adjusting words to better fit fantasy races

Customer Requests (Sometimes) Drive Me Bonkers!

How to pronounce the slash sign

Why Were Madagascar and New Zealand Discovered So Late?

Integer addition + constant, is it a group?

Pre-amplifier input protection

Is a stroke of luck acceptable after a series of unfavorable events?

How easy is it to start Magic from scratch?

How do I extract a value from a time formatted value in excel?

Sequence of Tenses: Translating the subjunctive

Why escape if the_content isnt?

How to Reset Passwords on Multiple Websites Easily?

Would this custom Sorcerer variant that can only learn any verbal-component-only spell be unbalanced?

Implement the Thanos sorting algorithm

Why not increase contact surface when reentering the atmosphere?

Why, precisely, is argon used in neutrino experiments?

Go Pregnant or Go Home

Escape a backup date in a file name

Avoiding estate tax by giving multiple gifts

Why didn't Theresa May consult with Parliament before negotiating a deal with the EU?

How to write papers efficiently when English isn't my first language?

Is oxalic acid dihydrate considered a primary acid standard in analytical chemistry?

dd command oflag direct and sync flags

Why does a gunzip to dd pipeline slow down at the end?System lags when doing large R/W operations on external disksWhy is `sync + drop_caches` not dropping caches?Block device cache v.s. a filesystemUnpack files and directories from an image created by dd?Backing up a partition: doesn't reflect the status properly'seek' argument in command dddd command lost partitionWhat is the difference between 'bs', 'count' and 'seek' in dd command?dd, seek with one block size, write with another block sizechange tape on the fly scriptWhy does ddrescue not utilize full bandwidth?dd command in script not running when executed from udev ruleUsing 'sync' when wiping a device with /dev/urandom

0

I have the below shell script and I wonder whether oflag's direct does the sync automatically or it is explicitly required

dd bs=10M oflag=direct,sync of=ofile.bin

Also what is the difference in saying oflag=sync and conv=sync or conv=fsync?

What impact does it bring if I change the code as below?

dd bs=10M conv=fsync oflag=direct of=ofile.bin

dd

share|improve this question

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  direct uses direct i/o, without buffer cache (check your blocksize though), oflag=sync uses synchronous data/metadata processing. conv=fsync uses sync() call after processing, conv=sync pads input blocks with zeroes to the blocksize.
  
  – stoney
  yesterday
  

  
  
  
  

add a comment | 

0

I have the below shell script and I wonder whether oflag's direct does the sync automatically or it is explicitly required

dd bs=10M oflag=direct,sync of=ofile.bin

Also what is the difference in saying oflag=sync and conv=sync or conv=fsync?

What impact does it bring if I change the code as below?

dd bs=10M conv=fsync oflag=direct of=ofile.bin

dd

share|improve this question

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  direct uses direct i/o, without buffer cache (check your blocksize though), oflag=sync uses synchronous data/metadata processing. conv=fsync uses sync() call after processing, conv=sync pads input blocks with zeroes to the blocksize.
  
  – stoney
  yesterday
  

  
  
  
  

add a comment | 

I have the below shell script and I wonder whether oflag's direct does the sync automatically or it is explicitly required

dd bs=10M oflag=direct,sync of=ofile.bin

Also what is the difference in saying oflag=sync and conv=sync or conv=fsync?

What impact does it bring if I change the code as below?

dd bs=10M conv=fsync oflag=direct of=ofile.bin

dd

share|improve this question

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

dd bs=10M oflag=direct,sync of=ofile.bin

dd bs=10M conv=fsync oflag=direct of=ofile.bin

share|improve this question

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

share|improve this question

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

asked yesterday

Ibrahim QuraishIbrahim Quraish

14112

1 Answer
1

active

oldest

votes

0

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs

---

oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.




2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.




3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].

---

[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When writing directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). I really do not recommend relying on this. However it explained something for me. Sometimes I see people who do not explicitly sync when writing to a block device, and it seems to work OK for them :-). See: Block device cache v.s. a filesystem

share|improve this answer

edited yesterday

answered yesterday

sourcejedisourcejedi

25.5k445110

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f508701%2fdd-command-oflag-direct-and-sync-flags%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs

---

oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.




2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.




3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].

---

[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When writing directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). I really do not recommend relying on this. However it explained something for me. Sometimes I see people who do not explicitly sync when writing to a block device, and it seems to work OK for them :-). See: Block device cache v.s. a filesystem

share|improve this answer

edited yesterday

answered yesterday

sourcejedisourcejedi

25.5k445110

add a comment | 

0

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs

---

oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.




2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.




3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].

---

[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When writing directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). I really do not recommend relying on this. However it explained something for me. Sometimes I see people who do not explicitly sync when writing to a block device, and it seems to work OK for them :-). See: Block device cache v.s. a filesystem

share|improve this answer

edited yesterday

answered yesterday

sourcejedisourcejedi

25.5k445110

add a comment | 

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with
block or unblock, pad with spaces rather than NULs

---

oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.




2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.




3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].

---

[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When writing directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). I really do not recommend relying on this. However it explained something for me. Sometimes I see people who do not explicitly sync when writing to a block device, and it seems to work OK for them :-). See: Block device cache v.s. a filesystem

share|improve this answer

edited yesterday

answered yesterday

sourcejedisourcejedi

25.5k445110

share|improve this answer

edited yesterday

answered yesterday

sourcejedisourcejedi

25.5k445110

answered yesterday

sourcejedisourcejedi

25.5k445110

answered yesterday

sourcejedisourcejedi

25.5k445110