How I Stream Files Over SSH Without a Temporary Archive

In this article

The trick is standard input and standard output

OpenSSH can run a command on the remote machine instead of opening an interactive shell. Without a pseudo-terminal, the session is transparent enough to carry binary data. GNU tar can write an archive to standard output when its archive name is -. Put those two behaviors together and the archive bytes never need a temporary file on the source machine.

Save the following as a Bash script and run it with Bash, rather than pasting individual steps into an interactive shell. Replace backup-host with your SSH host alias and set SOURCE_DIR or change the default source path. On the remote host, first provision ~/incoming as a directory owned by your backup account and inaccessible to untrusted users. Choose a new project-transfer-001 name for each run, changing every occurrence together. The script refuses to reuse that directory; it does not overwrite a previous transfer.

#!/usr/bin/env bash
set -euo pipefail
readonly source_dir="${SOURCE_DIR:-/srv/project}"

# The remote incoming directory must already exist and be privately owned.
# Choose a new transfer directory for each run; mkdir refuses an existing one.
ssh -T backup-host \
  'umask 077; mkdir "$HOME/incoming/project-transfer-001"'

if tar -C "$source_dir" -czf - . |
    ssh -T backup-host \
      'umask 077; cat > "$HOME/incoming/project-transfer-001/archive.tar.gz.part"'; then
  ssh -T backup-host \
    'tar -tzf "$HOME/incoming/project-transfer-001/archive.tar.gz.part" >/dev/null &&
     mv "$HOME/incoming/project-transfer-001/archive.tar.gz.part" \
        "$HOME/incoming/project-transfer-001/archive.tar.gz"'
else
  status=$?
  printf 'Transfer failed (status %s); leave the .part file unpromoted.\n' "$status" >&2
  exit "$status"
fi

-C sets tar’s source directory. The final dot means its contents; -czf - creates a gzip-compressed archive on standard output. ssh -T avoids a pseudo-terminal. The remote umask restricts the new directory and file permissions. Check available space and source read permissions before starting; neither compression nor SSH creates a consistent snapshot of changing files.

pipefail has to control the next step

In Bash, a pipeline normally returns the last command’s status. A source-side tar failure can be hidden by a successful SSH receiver. With pipefail, failure at either end takes the else path and leaves the temporary file unpromoted.

The condition matters as much as the option. If archive validation and renaming are pasted as a separate command, they can run after a failed transfer. An archive can list successfully even though tar reported a source read problem or failed to include everything requested. The example requires both a successful transfer pipeline and successful remote validation before renaming within the dedicated transfer directory.

A failed run leaves its directory for inspection. Do not rename its .part file merely because tar -t accepts it. Choose a new directory for a retry. If the SSH connection drops during the final step, inspect the destination before retrying: the remote rename may already have completed even though the client reports an error.

Listing an archive checks its structure, not whether it captured the intended application state or can restore your service. Databases and other live state need their own snapshot or export process. The final rename does not make the whole transfer transactional or guarantee storage durability after a power failure.

Why I usually reach for rsync instead

A tar stream starts over after an interrupted connection. It also turns a directory tree into one destination object, which is useful only when I actually want an archive.

For a normal directory copy, this is the less clever answer:

rsync -a --partial --info=progress2 \
  /srv/project/ backup-host:~/backups/project/

The rsync project documents remote-shell transfers with the familiar host:path form and uses SSH by default. Its delta-transfer algorithm can avoid sending unchanged data, and --partial keeps transferred data that may help a later run continue. Rsync must be installed at both ends.

The trailing slash after /srv/project/ matters: it means copy the directory’s contents into the destination. Without that slash, rsync creates another project level under the destination. I use -a when archive-mode metadata fits the two systems, but I do not assume owners, groups, ACLs, or extended attributes will map cleanly between every source and destination.

I also avoid adding --delete to a casual copy command. Deletion can be correct for a mirror, but it turns a transfer into a destination-cleanup operation. That deserves a dry run and a clearly owned target.

Streaming a disc image is the same shape with a sharper tool

The command I originally saved read a CD device with dd and piped the bytes into SSH. This Bash script uses a fresh private destination directory, refuses to reuse it, and leaves the image unpromoted. Provision ~/incoming as described above, replace the host alias, verify the optical-device path, and choose a new disc-transfer-001 name for every attempt:

#!/usr/bin/env bash
set -euo pipefail

# Provision a private ~/incoming directory on backup-host first.
# Choose a new disc-transfer-001 name for every attempt.
ssh -T backup-host \
  'umask 077; mkdir "$HOME/incoming/disc-transfer-001"'

if sudo dd if=/dev/sr0 bs=4M status=progress |
    ssh -T backup-host \
      'umask 077; set -C; cat > "$HOME/incoming/disc-transfer-001/disc.iso.part"'; then
  printf 'Transfer completed; keep the .part name until checksum verification succeeds.\n'
else
  status=$?
  printf 'Transfer failed (status %s); do not promote the .part file.\n' "$status" >&2
  exit "$status"
fi

GNU dd reads from the file named by if= and writes to standard output when no of= is supplied. status=progress periodically reports transfer statistics on standard error, so those status lines do not enter the image stream.

I verify the device name with a read-only inventory tool such as lsblk before running this. dd is indifferent to whether a path is the disc I intended, another block device, or a regular file. The example reads an optical device and writes a remote regular file; it does not write to a local disk. I still treat a mistaken if= as a serious error because it can copy the wrong data and expose it to the destination.

Verify more than “SSH returned zero”

For a tar archive, I list it. For an image, I compare a cryptographic checksum computed at the source and destination. When the source is removable media, reading the entire device a second time for a source checksum costs another full pass, but that is the comparison that proves the stored file matches a fresh read.

sudo sha256sum /dev/sr0
ssh backup-host 'sha256sum ~/incoming/disc-transfer-001/disc.iso.part'

Keep the disc unchanged between reads. Require successful exit statuses from the image transfer and both checksum commands before comparing the digests; partial output from a failed checksum command is not verification. If the two digests match after those checks, I can rename the remote file. If they do not, I keep the failed object under its temporary name long enough to investigate and do not present it as the finished image.

Matching SHA-256 digests provide strong evidence that the stored image matches that successful source read. They do not prove the filesystem is mountable or that its application data is consistent. If the media is damaged, a recovery-oriented tool such as GNU ddrescue is a better choice than repeatedly forcing plain dd through read errors.

Do not put secrets in the remote command

The SSH command and its arguments can be visible to local process inspection, shell history, logging, and the remote shell. I use a normal SSH host alias and key or interactive authentication. I do not put a password, token, or private key body inside the pipeline.

Host-key checking matters too. OpenSSH records host identities and warns when an existing host key changes. I do not disable that check just to make an unattended copy proceed. A data stream encrypted to the wrong machine is still a failed transfer.

Choose the transfer by its failure mode

What I changed from the 2008 version

The old post was seven short paragraphs and two commands. It wrote directly to the final destination name, did not enable pipefail, did not validate the resulting archive or image, and offered no restart advice. It also assumed that a successful-looking shell prompt meant the copy was good.

I kept the part worth keeping: Unix tools can pass useful binary streams through standard input and output, and SSH can carry them without a local staging file. I replaced the part that aged badly: a clever one-liner is not a backup procedure.

For estimating whether the upload window is practical, I use the backup upload-time calculator. The one-terabyte planning guide covers sustained throughput and restore boundaries. The Ansible home-lab article shows how I separate repeatable configuration, observed snapshots, and actual recovery.

What the example checks

The command behavior is grounded in the official OpenSSH client manual, GNU tar manual, GNU dd documentation, and rsync manual.

I reviewed the manuals and the original archived post for this replacement. Isolated shell tests use an empty fixture file and a local stand-in for SSH: successful transfer, refusal to reuse an existing destination, source failure after emitting a valid archive, and receiver failure. The source-failure case leaves a listable archive under its temporary name and returns the failure status. These are shell-control-flow tests, not a network or restore test. I did not image physical media, send private data to another host, simulate a broken network, mount the resulting ISO, or perform a restore for this article. The commands use placeholder paths and hostnames and should be adapted only after checking the source and destination.