Bash Redirection & Pipes#

Standard streams#

Redirection operators#

# Redirect stdout to file (overwrite)
command > file.txt

# Redirect stdout (append)
command >> file.txt

# Redirect stderr
command 2> error.log

# Redirect both stdout and stderr to same file
command &> all.log
command > all.log 2>&1   # older form, POSIX portable

# Discard output
command > /dev/null 2>&1

# Redirect stdin from file
command < input.txt

# Here-doc (multiline stdin)
cat <<EOF
line one
line two
EOF

# Here-string (single line stdin)
base64 <<< "hello world"

Output (cat <<EOF … EOF):

line one
line two

Output (base64 <<< "hello world"):

aGVsbG8gd29ybGQK

Pipes#

# Basic pipe
ps aux | grep nginx

# Pipe stderr through pipe (bash 4+)
command 2>&1 | grep ERROR

# Pipe with tee (write to file AND stdout)
make 2>&1 | tee build.log

# Process substitution (pipe without subshell for IDs)
diff <(sort file1.txt) <(sort file2.txt)

# Named pipe (FIFO)
mkfifo /tmp/mypipe
tail -f /var/log/syslog > /tmp/mypipe &
grep "ERROR" < /tmp/mypipe

Output (ps aux | grep nginx):

www-data  1234  0.0  0.1  55680  2048 ?  S  09:01  0:00 nginx: worker process
www-data  1235  0.0  0.1  55680  2048 ?  S  09:01  0:00 nginx: worker process

Output (diff <(sort file1.txt) <(sort file2.txt)):

3c3
< banana
---
> blueberry
5a6
> mango

Output (make 2>&1 | tee build.log):

gcc -o main main.c utils.c
Linking...
Build complete.

Output is written to the terminal in real time AND saved to build.log simultaneously.

Output (tail -f /var/log/auth.log | grep --color=always "Failed"):

Apr 26 10:33:01 server sshd[4821]: Failed password for invalid user admin from 203.0.113.42 port 51234 ssh2
Apr 26 10:33:08 server sshd[4822]: Failed password for root from 198.51.100.7 port 60412 ssh2

Useful patterns#

Capture stderr into a variable (discard stdout):

err=$(command 2>&1 >/dev/null)

Run command, capture all output, and check exit code:

if ! output=$(some-command 2>&1); then
  echo "Failed: $output" >&2
  exit 1
fi

Swap stdout and stderr:

command 3>&1 1>&2 2>&3 3>&-

Tail a live log with colour preserved through the pipe:

tail -f /var/log/auth.log | grep --color=always "Failed"

File descriptors in depth#

Every Unix process inherits three open file descriptors from its parent: 0 (stdin), 1 (stdout), 2 (stderr). The kernel doesn’t treat them specially — they’re just the first three slots in the process’s file-descriptor table, and you can open more (3, 4, 5, …) whenever you need to. Redirection operators in bash are syntactic sugar over the dup2(2) system call: they wire one descriptor to point at the same file as another descriptor or to a file on disk.

# Show this shell's open descriptors
ls -l /proc/$$/fd

# Open a new descriptor 3 for reading from a file
exec 3< /etc/hostname
read -r hostname <&3
exec 3<&-             # close fd 3
echo "$hostname"

# Open fd 4 for writing to a log
exec 4> /tmp/script.log
echo "starting" >&4
date >&4
exec 4>&-             # close

Output (ls -l /proc/$$/fd):

lrwx------ 1 alice alice 64 May 24 10:00 0 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 1 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 2 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 255 -> /dev/pts/3

All three standard descriptors point at the same pty (/dev/pts/3) — that’s why typing in the terminal shows the same place where output appears.

[!TIP] Higher descriptors (3+) survive sub-shell creation and process-substitution boundaries. They’re the right tool for shell scripts that need a persistent log channel without colliding with subcommands that write to stdout/stderr.

Closing a descriptor#

>&- and <&- close an open descriptor. Closing inherited fds is occasionally important — when you spawn a long-running child you don’t want it holding open a pipe to your script.

# Close stdin in the child
some-daemon <&-

# Close fd 3 after using it
exec 3>&-

>, >>, and >| — overwrite, append, noclobber#

> opens the target file with O_WRONLY | O_CREAT | O_TRUNC — it creates the file if it doesn’t exist and truncates it to zero length if it does. >> uses O_APPEND instead, so writes go to the end. >| is the same as > but bypasses the noclobber shell option (set -o noclobber), which otherwise forbids overwriting an existing file.

# Default: overwrite
echo "hello" > /tmp/greeting.txt

# Append
echo "world" >> /tmp/greeting.txt

# Enable noclobber to prevent accidental overwrites
set -o noclobber
echo "oops" > /tmp/greeting.txt          # error: cannot overwrite
echo "force" >| /tmp/greeting.txt        # explicit override
set +o noclobber

Output (with noclobber):

bash: /tmp/greeting.txt: cannot overwrite existing file

[!TIP] set -o noclobber is a cheap script-safety win: it turns an accidental > output.txt into an error instead of silently losing previous data. Pair with >| for the rare case where overwriting is intentional.

2>&1 — order matters#

2>&1 makes file descriptor 2 (stderr) point at whatever fd 1 (stdout) currently points at. Redirections are processed left-to-right, so the order you write them in changes the result. This is the single most surprising thing about bash redirection.

# Send BOTH stdout and stderr to all.log
command > all.log 2>&1
#         ↑ first: fd 1 → all.log
#                  ↑ second: fd 2 → wherever fd 1 points = all.log
# Result: both go to all.log. ✓

# Same operators, different order — does NOT do the same thing
command 2>&1 > all.log
#       ↑ first: fd 2 → wherever fd 1 points (still the terminal)
#            ↑ second: fd 1 → all.log
# Result: stdout goes to all.log, stderr still goes to the terminal. ✗

The mental model: think of > and 2>&1 not as “merge streams” but as “set this fd to point at the same place as that one right now”.

&> and |& — the bash shortcuts#

bash 4 adds &> as a one-token shorthand for > ... 2>&1 and |& as shorthand for 2>&1 |. They’re not POSIX, so don’t use them in /bin/sh scripts; they’re fine in any #!/usr/bin/env bash script.

# Both stdout and stderr to a file
command &> all.log

# Both stdout and stderr through a pipe
command |& grep ERROR

# Append both to a file (bash 4+)
command &>> all.log

# POSIX equivalents
command > all.log 2>&1
command 2>&1 | grep ERROR
command >> all.log 2>&1

Process substitution#

<(...) and >(...) make a command’s stdout (or stdin) look like a regular file path that other commands can open. bash implements them with /dev/fd/<n> (or named FIFOs on systems without /dev/fd). This lets you feed multiple command outputs to a tool that takes file arguments — diff, comm, paste, and anything else that won’t read from stdin.

# Diff two sorted streams without temp files
diff <(sort file1.txt) <(sort file2.txt)

# Compare command outputs across hosts
diff <(ssh host1 'systemctl list-units --no-pager') \
     <(ssh host2 'systemctl list-units --no-pager')

# Three-way input: paste matched lines from three streams
paste <(cut -d, -f1 names.csv) <(cut -d, -f2 names.csv) <(cut -d, -f3 names.csv)

# Output substitution: send a command's input to multiple consumers via tee
ls -la | tee >(grep '.txt$' > text.list) >(grep '.log$' > log.list) >/dev/null

# Inspect what bash actually creates
echo <(true)
# /dev/fd/63

Output (echo <(true)):

/dev/fd/63

That /dev/fd/63 is the descriptor bash opened on the pipe between the shell and the substituted command. Tools that fopen() it get exactly the same byte stream the substituted command writes.

[!WARN] Process substitution does not propagate exit status. diff <(failing-cmd) <(other-cmd) returns the exit code of diff, not of failing-cmd. If you need the inner command’s status, capture it via a temp file or coproc instead.

Here-docs#

A here-document feeds a literal block of text into a command’s stdin, ending at a sentinel line. The most common use is templating config files or scripts; the variants control quoting, indentation, and where the body comes from.

# Standard here-doc — variables ARE expanded
cat <<EOF
Hostname: $(hostname)
User: $USER
Date: $(date)
EOF

# Quoted delimiter — variables and command substitution are NOT expanded
# (use this for embedded shell snippets, JSON templates, awk programs, etc.)
cat <<'EOF'
Literal $USER and $(date) — no expansion.
This script doesn't try to evaluate anything inside.
EOF

# Leading-tab stripping with <<- so the heredoc can be indented in source
if true; then
    cat <<-EOF
	    one
	    two
	    three
	EOF
fi

# Redirect the heredoc output to a file
cat > /tmp/config.toml <<EOF
[server]
host = "$(hostname)"
port = 8080
EOF

Output (first heredoc, cat <<EOF):

Hostname: myhost
User: alicedev
Date: Sat May 24 10:00:00 EDT 2026

[!TIP] Use <<'EOF' (quoted) whenever the body contains shell metacharacters you do not want bash to interpret — for example, when embedding a Python or awk script. Reserve unquoted <<EOF for templates that genuinely need variable expansion.

<<< here-strings#

A here-string feeds a single line of text on stdin without spawning echo. It’s slightly faster than echo "x" | cmd and avoids the subshell that | creates, which lets you use read to capture the result into the current shell.

# base64-encode a literal string
base64 <<< "hello world"

# Feed a value into read without losing it in a subshell
read -r year month day <<< "2026 05 24"
echo "$year-$month-$day"

# Quick JSON pretty-print of a string
jq . <<< "$json_payload"

Output (base64 <<< "hello world"):

aGVsbG8gd29ybGQK

Named pipes (FIFOs)#

mkfifo creates a special file that one process writes to and another reads from — like an anonymous pipe but persistent on disk, so the two processes don’t need a common parent. Useful for connecting backgrounded jobs, building tee-like fan-out across machines, and giving one process the ability to send commands to another.

# Create a FIFO
mkfifo /tmp/mypipe

# Producer in the background
tail -f /var/log/auth.log > /tmp/mypipe &

# Consumer in the foreground
grep "Failed" < /tmp/mypipe

# Many-writer, one-reader pattern
mkfifo /tmp/cmd
while read -r line < /tmp/cmd; do
    echo "received: $line"
done &
echo "hello" > /tmp/cmd
echo "world" > /tmp/cmd

# Clean up
rm /tmp/mypipe /tmp/cmd

Output (consumer):

received: hello
received: world

A FIFO blocks until both ends are open: opening for reading blocks until someone opens for writing, and vice versa. For producer-consumer patterns where you don’t want that, use O_NONBLOCK with exec 3<> /tmp/pipe (open for read+write yourself).

[!WARN] FIFOs persist after the processes exit. Always rm them when you’re done — leftover FIFOs in shared directories can hang other scripts that happen to find them.

tee and friends#

tee reads stdin, writes to stdout, and also writes to one or more files — the T-pipe of Unix. It’s how you save a command’s output while also watching it scroll past in real time.

# Save output AND see it live
make 2>&1 | tee build.log

# Append instead of overwrite
make 2>&1 | tee -a build.log

# Tee to multiple files at once
some-cmd | tee out1.log out2.log

# Run a downstream command on the SAME stream that's being saved
some-cmd | tee build.log | grep -i error

# Write to a file that requires root, without making the whole pipeline root
echo "127.0.0.1 myhost.local" | sudo tee -a /etc/hosts

# Discard tee's stdout (useful for "fan out to N files only")
some-cmd | tee out1.log out2.log > /dev/null

Output (make 2>&1 | tee build.log):

gcc -o main main.c utils.c
Linking...
Build complete.

echo X | sudo tee -a /etc/file is the canonical idiom for appending to a root-owned file from a non-root shell — it avoids bash: /etc/file: Permission denied because the redirection runs in the shell process before sudo takes effect.

exec — global redirection inside a script#

exec followed by redirections (and no command) modifies the current shell’s open descriptors permanently — every subsequent command in the script inherits them. This is how you say “send everything from here on to /tmp/script.log”.

#!/usr/bin/env bash
# /home/alice/bin/job.sh — redirect ALL output from this point forward

# Redirect stdout to a log file (everything after this goes to the log)
exec >> /home/alice/log/job.log

# Redirect stderr to the same place
exec 2>&1

echo "[$(date '+%F %T')] starting"
do-work
echo "[$(date '+%F %T')] done"

# Variant: ALSO see output on the terminal via tee + process substitution
exec > >(tee -a /home/alice/log/job.log)
exec 2>&1
echo "this appears in both the terminal and the log"

# Open a dedicated logging fd that's separate from stdout/stderr
exec 4>> /home/alice/log/script-events.log
echo "starting" >&4
some-cmd                       # stdout/stderr unaffected
echo "finished" >&4

The first form (single redirect) is the cron-friendly idiom — set up logging at the top of the script and forget about it. The tee form gives you the live terminal output as well, useful for interactive runs.

coproc — bidirectional pipes#

A coprocess is a background process whose stdin and stdout are connected to two file descriptors in the parent shell. Reach for coproc when you need to interact with a long-running child (send a query, read the response, send another query) — a one-way pipe can’t do that.

# Launch bc as a coprocess
coproc CALC { bc -l; }

# Send a query and read the answer
echo "scale=4; sqrt(2)" >&"${CALC[1]}"
read -r answer <&"${CALC[0]}"
echo "sqrt(2) = $answer"

# Send another query — same process, no restart cost
echo "2^10" >&"${CALC[1]}"
read -r answer <&"${CALC[0]}"
echo "2^10 = $answer"

# Close down
exec {CALC[1]}>&-
wait "$CALC_PID"

Output:

sqrt(2) = 1.4142
2^10 = 1024

coproc is overkill for one-shot command output (use var=$(cmd)), but it’s the right tool for “spawn a REPL once, ask many questions”. Common targets: bc -l, sqlite3, an SSH master, a long-lived python3 -u.

set -o pipefail and the exit-status trap#

By default, a pipeline’s exit status is the status of the last command only. That means failing-cmd | tee log exits 0 even when failing-cmd failed, because tee succeeded. set -o pipefail changes the rule: the pipeline exits with the status of the rightmost non-zero command (or 0 if all succeeded). Combine with set -e and set -u for the standard “strict mode” script preamble.

#!/usr/bin/env bash
set -euo pipefail        # exit on error, error on unset var, pipefail

# Without pipefail, this would exit 0 even though `false` returned 1:
false | tee /tmp/out

# With pipefail set, the pipeline exits 1 and `set -e` kills the script.

Inspect ${PIPESTATUS[@]} for the status of every command in the most recent pipeline — useful when you want fine-grained reporting beyond pass/fail.

false | true | true
echo "${PIPESTATUS[@]}"
# 1 0 0

[!TIP] set -euo pipefail is the single best preamble for shell scripts. It turns silent failures into loud ones and makes debugging dramatically easier. Combine with trap 'echo "error on line $LINENO" >&2' ERR for line-number reporting.

Common pitfalls#

1. cmd > file 2>&1 vs cmd 2>&1 > file — the first works; the second does not. Order is left-to-right.
2. echo "$x" | read y — the right-hand side runs in a subshell, so $y is unset back in the parent. Use read y <<< "$x" (here-string) or process substitution instead.
3. tee losing exit status — without pipefail, failing | tee log reports success. Always set -o pipefail in production scripts, or check ${PIPESTATUS[0]}.
4. Unquoted heredoc expanding $variables unintentionally — embedding a Python or awk program in <<EOF will mangle $1/$NF/$VAR. Use <<'EOF' for verbatim bodies.
5. <<-EOF indentation requires tabs — leading spaces are NOT stripped; only tab characters are. Editors that auto-convert tabs to spaces will silently break the heredoc body.
6. Process substitution exit codes ignored — diff <(may-fail) <(other) returns diff’s status; the inner command’s failure is invisible. Capture via a temp file when you need it.
7. /dev/null 2>&1 order — > /dev/null 2>&1 discards both; 2>&1 > /dev/null discards stdout only and leaves stderr on the terminal.
8. Forgetting to close descriptors — exec 3> log without a later exec 3>&- means descriptor 3 leaks into every child process you spawn. Usually harmless but occasionally surprising.
9. FIFOs blocking unexpectedly — opening a FIFO for reading blocks until something opens it for writing (and vice versa). tail -f log > fifo & is the right pattern; cat fifo alone hangs forever.
10. >> file vs tee -a file — when running under sudo, only the latter works for a root-owned file. The >> redirection happens in the unprivileged shell process.

Real-world recipes#

Log a script’s full output to file + terminal#

The standard “run interactively but also save the transcript” pattern. Everything (stdout and stderr) is captured to the log while still visible on the terminal.

#!/usr/bin/env bash
set -euo pipefail

LOG=/home/alice/log/install.log
exec > >(tee -a "$LOG") 2>&1

echo "[$(date '+%F %T')] install starting on $(hostname)"
sudo apt-get update
sudo apt-get install -y nginx postgresql
echo "[$(date '+%F %T')] install finished"

The tee -a keeps the log when the script is re-run; the process substitution lets exec redirect the parent’s stdout to tee’s stdin transparently.

Capture stdout and stderr to separate files#

For build tools that produce useful warnings on stderr but mountains of progress on stdout, you may want them in separate files for triage.

build-tool > build.out 2> build.err

# With pipefail-aware exit reporting
{ build-tool 2>&3 | tee build.out >/dev/null; } 3>&1 1>&2 | tee build.err >/dev/null

The second form is the canonical “tee both streams without merging” idiom. Read it right-to-left: fd 3 is opened to point at the original stdout, fd 2 of build-tool is redirected to 3 (so stderr now goes where the outer pipe is), and the outer pipeline tees the visible stderr while the inner tees stdout.

Capture stderr into a variable, discard stdout#

For checking error messages from a command whose stdout output is irrelevant. The swap pattern (3>&1 1>&2 2>&3 3>&-) is the magic spell.

err=$(some-cmd 2>&1 >/dev/null)

# OR — keep stdout in $out and stderr in $err
{
    IFS= read -rd '' out
    IFS= read -rd '' err
} < <({ some-cmd 2> >(printf '%s\0' "$(cat)" >&2) ; printf '\0'; } 2>&1)

The simpler err=$(cmd 2>&1 >/dev/null) works for most cases; the second form is when you need both streams independently in the parent shell.

Diff two command outputs#

The classic process-substitution use case. Especially handy when you can’t (or don’t want to) write the outputs to disk first.

# Compare package lists between two hosts
diff <(ssh host1 dpkg -l | awk '{print $2}' | sort) \
     <(ssh host2 dpkg -l | awk '{print $2}' | sort)

# Spot what changed in a sysctl after a tuning patch
diff <(sysctl -a 2>/dev/null | sort) /tmp/sysctl.before

Fan-out a stream to multiple consumers#

When one upstream produces a stream that several downstream tools need to consume in parallel.

# Save raw + run two analyses, all from one pcap capture
tcpdump -i eth0 -U -w - 2>/dev/null \
  | tee >(tcpdump -r - 'tcp port 80' > http.pcap) \
        >(tcpdump -r - 'udp port 53' > dns.pcap) \
        > /tmp/full.pcap

tee writes the stream to each >(...) consumer; bash sets each one up as a sub-process reading on a /dev/fd/<n>.

Atomic file write with a temp file + rename#

Don’t redirect directly to the destination — write to a temporary, then mv into place. This way a partial write never appears as the target file.

tmp=$(mktemp /tmp/config.XXXXXX)
trap 'rm -f "$tmp"' EXIT
generate-config > "$tmp"
mv "$tmp" /etc/myapp/config.toml

Append to a root-owned file from a normal shell#

sudo echo >> /etc/file does NOT do what you want — >> runs as your user before sudo is involved. The fix is sudo tee -a.

echo "127.0.0.1 myhost.local" | sudo tee -a /etc/hosts > /dev/null

The trailing > /dev/null keeps tee from echoing the line back to your terminal.

Build a one-liner that returns the right exit code#

When piping through tee (e.g. for a CI build), make sure a failing build actually fails the pipeline.

set -o pipefail
make 2>&1 | tee build.log
# Pipeline now exits with make's status, not tee's.

[!TIP] When in doubt about a redirection, prepend set -x (or run bash -x script.sh) — bash prints every expanded command and its redirections to stderr, which makes order-of-evaluation bugs immediately obvious. Pair with shellcheck for static analysis before running.

[!WARN] > file is destructive and instantaneous. There is no undo. Pair set -o noclobber with >| for explicit overwrites in interactive shells, and write to temp files plus mv in scripts.