Ygtjki

Consider this script:

tmpfile=$(mktemp)

cat <<EOS > "$tmpfile"
line 1
line 2
line 3
EOS

cat <(tail -1 "$tmpfile") "$tmpfile"

This works and outputs:

line 3
line 1
line 2
line 3

Let's say that our input source, rather than being an actual file, was instead stdin:

cat <<EOS | # what goes here now?
line 1
line 2
line 3
EOS

How do we modify the command:

cat <(tail -1 "$tmpfile") "$tmpfile"

So that it still produces the same output, in this different context?

NOTE: The specific Heredoc I'm catting, as well as the use of a Heredoc itself, is merely illustrative. Any acceptable answer should assume that it is receiving arbitrary data via stdin.

Try:

awk 'x=x $0 ORS; ENDprintf "%s", $0 ORS x'

Example

Define a variable with our input:

$ input="line 1
> line 2
> line 3"

Run our command:

$ echo "$input" | awk 'x=x $0 ORS; ENDprintf "%s", $0 ORS x'
line 3
line 1
line 2
line 3

Alternatively, of course, we could use a here-doc:

$ cat <<EOS | awk 'x=x $0 ORS; ENDprintf "%s", $0 ORS x'
line 1
line 2
line 3
EOS
line 3
line 1
line 2
line 3

How it works

• 
  

  x=x $0 ORS

  
  
  

  This appends each line of input to the variable x.

  
  
  

  In awk, ORS is the output record separator. By default, it is a newline character.

  
  


• 
  

  ENDprintf "%s", $0 ORS x

  
  
  

  After the we have read in the whole file, this prints the last line, $0, followed by the contents of the whole file, x.

  
  

Since this reads the whole input into memory, it would not be appropriate for large (e.g. gigabyte) inputs.

If stdin points to a seekable file (like in the case of bash's (but not all other shell's) here documents which are implemented with temp files), you can get the tail and then seek back before reading the full contents:

seek operators are available in the zsh or ksh93 shells, or scripting languages like tcl/perl/python, but not in bash. But you can always call those more advanced interpreters from bash if you have to use bash.

ksh93 -c 'tail -n1; cat <#((0))' <<...

Or

zsh -c 'zmodload zsh/system; tail -n1; sysseek 0; cat' <<...

Now, that won't work when stdin points to a non-seekable files like a pipe or socket. Then, the only option is to read and store (in memory or in a temporary file...) the whole input.

Some solutions for storing in memory have already been given.

With a tempfile, with zsh, you could do it with:

seq 10 | zsh -c ' cat =(sed $w/dev/fd/3); 3>&1'

If on Linux, with bash or zsh or any shell that uses temp files for here-documents, you could actually use the temp file created by a here-document to store the output:

seq 10 | cat > /dev/fd/3; tail -n1 /dev/fd/3; cat <&3; 3<<EOF
EOF

cat <<EOS | sed -ne '1h;d;' -e 'H;$G;p;'
line 1
line 2
line 3
EOS

The issue with translating this to something that uses tail is that tail needs to read the whole file to find the end of it. To use that in your pipeline, you need to

1. Provide the full contents of the document to tail.


2. Provide it again to cat.


3. In that order.

The tricky bit is not to duplicate the document's content (tee does that) but to get the output of tail to happen before the rest of the document is outputted, without using an intermediate temporary file.

Using sed (or awk, as John1024 does) gets rid of the double parsing of the data and the ordering issue by storing the data in memory.

The sed solution that I propose is to

1. 
   1h;d;, store the first line in the hold space, as-is, and skip to the next line.


2. 
   H, append each other line to the hold space with an embedded newline.


3. 
   $G;p;, append the hold space to the last line with an embedded newline and print the resulting data.

This is quite a literal translation of John1024's solution into sed, with the caveat that the POSIX standard only guarantees that the hold space is at lest 8192 bytes (8 KiB; but it recommends that this buffer is dynamically allocated and expanded as needed, which both GNU sed and BSD sed is doing).

If you allow yourself to use a named pipe:

mkfifo mypipe
cat <<EOS | tee mypipe | cat <( tail -n 1 mypipe ) -
line 1
line 2
line 3
EOS
rm -f mypipe

This uses tee to send the data down mypipe and at the same time to cat. The cat utility will first read the output from tail (which reads from mypipe, which tee is writing to), and then append the copy of the document coming directly from tee.

There's a serious flaw in this though, in that if the document is too large (larger than the pipe's buffer size), tee's writing to mypipe and cat would block while waiting for the (unnamed) pipe to empty. It would not be emptied until cat read from it. cat would not read from it until tail had finished. And tail would not finish until tee had finished. This is a classic deadlock situation.

The variation

tee >( tail -n 1 >mypipe ) | cat mypipe -

has the same issue.

There is a tool named pee in a collection of command-line utilities usually packaged with the name "moreutils” (or otherwise retrievable from its home website).

If you can have it on your system then the equivalent for your example would be like:

cat <<EOS | pee 'tail -1' cat 
line 1
line 2
line 3
EOS

Ordering of the commands run through pee is important because they get executed in the sequence provided.

Try:

cat <<EOS # | what goes here now? Nothing!
line 3
line 1
line 2
line 3
EOS

Since the whole thing is literal data (a "here-is document"), and the difference between it and the desired output is trivial, just massage that literal data right there to match the output.

Now suppose line 3 comes from somewhere and is stored in a variable called lastline:

cat <<EOS # | what goes here now? Nothing!
$lastline
line 1
line 2
$lastline
EOS

In a here doc, we can generate text by substituting variables. Not only that but we can calculate text using command substitution:

cat <<EOS
this is template text
here we have a hex conversion: $(printf "%x" 42)
EOS

We can interpolate multiple lines:

cat <<EOS
multi line
preamble
$(for x in 3 1 2 3; do echo line $x ; done)
epilog
EOS

In general, avoid text processing the here doc template; try to generate it using interpolated code.