Bash One-Liners Explained, Part III: All about redirections

Posted: 23 Aug 2012 05:40 AM PDT

This is the third part of the Bash One-Liners Explained article series. In this part I'll teach you all about input/output redirection. I'll use only the best bash practices, various bash idioms and tricks. I want to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

See the first part of the series for introduction. After I'm done with the series I'll release an ebook (similar to my ebooks on awk, sed, and perl), and also bash1line.txt (similar to my perl1line.txt).

Also see my other articles about working fast in bash from 2007 and 2008:

Let's start.

Part III: Redirections

1. Redirect the standard output of a command to a file

  $ command >file

Operator > is the output redirection operator. Bash first tries to open the file for writing and if it succeeds it sends the stdout of command to the newly opened file. If it fails opening the file, the whole command fails.

Writing command >file is the same as writing command 1>file. The number 1 stands for stdin, which is the standard output.

In general you can write command n>file, which will redirect the file descriptor n to file.

For example,

  $ ls > file_list

Redirects the output of the ls command to the file_list file.

2. Redirect the standard error of a command to a file

  $ command 2> file

Here bash redirects the stderr to file. The number 2 stands for stderr.

3. Redirect both standard output and standard error to a file

  $ command &>file

This one-liner uses the &> operator to redirect both output streams - stdout and stderr - from command to file. This is bash shortcut for quickly redirecting both streams.

It's exactly the same as writing:

  $ command >file 2>&1

This is a much more common way to redirect both streams to a file. Here the 2>&1 operator duplicates file descriptors. In this case file descriptor 2 (stderr) is made to be a copy of file descriptor 1 (stdout). So when the command writes to stderr, the output actually ends in stdout.

Be careful here! Writing:

  command >file 2>&1

Is not the same as writing:

  $ command 2>&1 >file

The order of redirects in bash matters! This command redirects only the standard output to the file, because the standard error was duplicated as standard output before the standard output was redirected to the file.

Think of it this way - before running the command both stdout and stderr were redirected to console. Now bash sees the first redirect 2>&1 and duplicates stderr as stdout. Then bash proceeds and sees the second redirect >file, so it now redirects stdout to file. Stderr, however, still points to stdout as it wasn't changed. So what happens is stderr gets printed to console, and stdout gets sent to file!

Also note that in bash, writing this:

  $ command &>file

Is exactly the same as:

  $ command >&file

The first form is preferred however.

4. Discard the standard output of a command

  $ command > /dev/null

The special file /dev/null discards all data written to it. So what we're doing here is redirecting stdout to this special file and it gets discarded.

Similarly, by combining the previous one-liners, we can discard both stdout and stderr by doing:

  $ command >/dev/null 2>&1

Or just simply:

  $ command &>/dev/null

5. Redirect the contents of a file to the stdin of a command

  $ command <file

Here bash tries to open the file for reading before running any commands. If opening the file fails, bash quits with error and doesn't run the command. If opening the file succeeds, bash uses the file descriptor of the opened file as the stdin file descriptor for the command.

Here is an example. Suppose you want to read the first line of the file in a variable. You can simply do this:

  $ read -r line < file

Bash's built-in read command reads a single line from standard input. By using the input redirection operator < we set it up to read the line from the file.

6. Redirect a bunch of text to the stdin of a command

  $ command <<EOL  your  multi-line  text  goes  here  EOL

Here we use the here-document redirection operator <<TEXT. This operator instructs bash to read the input from stdin until a line containing only TEXT is found, and then passes the all the read input to the stdin of the command.

Here is a common example. Suppose you've copied a bunch of URLs to the clipboard and you want to remove http:// part of them. A quick one-liner would be to do this:

  $ sed 's|http://||' <<EOL  http://url1.com  http://url2.com  http://url3.com  EOL

Here the input of a list of URLs is redirected to the sed command that strips http:// from the input.

This example produces this output:

  url1.com  url2.com  url3.com

7. Redirect a single line of text to the stdin of a command

  $ command <<< "foo bar baz"

For example, let's say you quickly want to pass the text in your clipboard as the stdin to a command. Instead of doing something like:

  $ echo "clipboard contents" | command

You can now just write:

  $ command <<< "clipboard contents"

This trick changed my life when I learned it!

8. Redirect stderr of all commands to a file forever

  $ exec 2>file  $ command1  $ command2  $ ...

This one-liner uses the built-in exec bash command. If you specify redirects after it, then they will last forever, meaning until you change them or exit the script/shell.

In this case the 2>file redirect is setup that redirects the stderr of the current shell to the file. Running commands after setting up this redirect will have the stderr of all of them redirected to file. It's really useful in situations when you want to have a complete log of all errors that happened in the script, but you don't want to specify 2>file after every single command!

In general exec can take an optional argument of a command. If it's specified, bash replaces itself with the command. So what you get is only that command running, and there is no more shell.

9. Open a file for reading using a custom file descriptor

  $ exec 3<file

Here we use the exec command again and specify the 3<file redirect to it. What this does is opens the file for reading and assigns the opened file-descriptor to the shell's file descriptor number 3.

Now you can read from the file descriptor 3, like this:

  $ read -u 3 line

This reads a line from the file that we just opened as fd 3.

Or you can use regular shell commands such as grep and operate on file descriptor 3:

  $ grep "foo" <&3

What happens here is file descriptor 3 gets duplicated to file descriptor 1 - stdin of grep. Just remember that once you read the file descriptor it's been exhausted and you need to close it and open it again to use it. (You can't rewind an fd in bash.)

After you're done using fd 3, you can close it this way:

  $ exec 3>&-

Here the file descriptor 3 is duped to -, which is bash's special way to say "close this fd".

10. Open a file for writing using a custom file descriptor

  $ exec 4>file

Here we simply tell bash to open file for writing and assign it number 4.

Then we can simply write to the file descriptor 4:

  $ echo "foo" >&4

And we can close the file descriptor 4:

  $ exec 4>&-

It's so simple now once we learned how to work with custom file descriptors!

11. Open a file both for writing and reading

  $ exec 3<>file

Here we use bash's diamond operator <>. The diamond operator opens a file descriptor for both reading and writing.

So for example, if you do this:

  $ echo "foo bar" > file   # write string "foo bar" to file "file".  $ exec 5<> file           # open "file" for rw and assign it fd 5.  $ read -n 3 var <&5       # read the first 3 characters from fd 5.  $ echo $var

This will output foo as we just read first 3 chars from the file.

Now we can write some stuff to the file:

  $ echo -n + >&5           # write "+" at 4th position.  $ exec 5>&-               # close fd 5.  $ cat file

This will output foo+bar as we wrote the + char at 4th position in the file.

12. Send the output from multiple commands to a file

  $ (command1; command2) >file

This one-liner uses the (list) construct that runs commands in the list in a sub-shell. A sub-shell is a child process launched by the current shell.

So what happens here is the commands command1 and command2 get executed in the sub-shell, and bash redirects their output to file.

Enjoy!

Enjoy the article and let me know in the comments what you think about it! If you think that I forgot some interesting bash one-liners related to redirections, let me know in the comments below!