How I Stream Files Over SSH Without a Temporary Archive

Stream a file or disk image through SSH when avoiding a temporary local archive matters and restartability does not. Add pipefail, verify the destination, and prefer rsync when interrupted transfers are likely.

The trick is standard input and standard output

OpenSSH can run a command on the remote machine instead of opening an interactive shell. Without a pseudo-terminal, the session is transparent enough to carry binary data. GNU tar can write an archive to standard output when its archive name is -. Put those two behaviors together and the archive bytes never need a temporary file on the source machine.

set -o pipefail
tar -C /srv/project -czf - . |
  ssh -T backup-host 'cat > ~/incoming/project.tar.gz.part'

-C /srv/project changes tar’s working directory before it reads anything. The final dot means “archive the contents here.” -c creates the archive, -z compresses it with gzip, -f - sends the archive to standard output, and ssh -T avoids allocating a pseudo-terminal.

The remote shell opens project.tar.gz.part for writing. That redirection truncates an existing file with the same name, so I choose the destination deliberately and do not paste the command before checking it. The .part suffix is not magic; it simply stops an interrupted stream from looking finished.

pipefail is the line my old snippet was missing

In Bash, a pipeline normally reports the status of its last command. If tar fails to read part of the source but SSH and the remote cat exit successfully, the pipeline can look successful even though the archive is incomplete. set -o pipefail makes the pipeline fail when an earlier stage fails.

That still does not make the transfer transactional. The destination file exists while bytes are arriving. A network break can leave a partial archive. The safe sequence is: write the temporary name, require a successful pipeline, test the archive remotely, and only then rename it.

ssh backup-host \
  'tar -tzf ~/incoming/project.tar.gz.part >/dev/null &&
   mv ~/incoming/project.tar.gz.part ~/incoming/project.tar.gz'

tar -t reads the archive directory without extracting it. This catches a truncated gzip stream and many archive-format failures. The source may still be inconsistent, and the resulting archive may still be useless as a backup. Databases and other live state need their own snapshot or export process.

Why I usually reach for rsync instead

A tar stream starts over after an interrupted connection. It also turns a directory tree into one destination object, which is useful only when I actually want an archive.

For a normal directory copy, this is the less clever answer:

rsync -a --partial --info=progress2 \
  /srv/project/ backup-host:~/backups/project/

The rsync project documents remote-shell transfers with the familiar host:path form and uses SSH by default. Its delta-transfer algorithm can avoid sending unchanged data, and --partial keeps transferred data that may help a later run continue. Rsync must be installed at both ends.

The trailing slash after /srv/project/ matters: it means copy the directory’s contents into the destination. Without that slash, rsync creates another project level under the destination. I use -a when archive-mode metadata fits the two systems, but I do not assume owners, groups, ACLs, or extended attributes will map cleanly between every source and destination.

I also avoid adding --delete to a casual copy command. Deletion can be correct for a mirror, but it turns a transfer into a destination-cleanup operation. That deserves a dry run and a clearly owned target.

Streaming a disc image is the same shape with a sharper tool

The command I originally saved read a CD device with dd and piped the bytes into SSH. A modern GNU version can show progress:

set -o pipefail
sudo dd if=/dev/sr0 bs=4M status=progress |
  ssh -T backup-host 'cat > ~/incoming/disc.iso.part'

GNU dd reads from the file named by if= and writes to standard output when no of= is supplied. status=progress periodically reports transfer statistics on standard error, so those status lines do not enter the image stream.

I verify the device name with a read-only inventory tool such as lsblk before running this. dd is indifferent to whether a path is the disc I intended, another block device, or a regular file. The example reads an optical device and writes a remote regular file; it does not write to a local disk. I still treat a mistaken if= as a serious error because it can copy the wrong data and expose it to the destination.

Verify more than “SSH returned zero”

For a tar archive, I list it. For an image, I compare a cryptographic checksum computed at the source and destination. When the source is removable media, reading the entire device a second time for a source checksum costs another full pass, but that is the comparison that proves the stored file matches a fresh read.

sudo sha256sum /dev/sr0
ssh backup-host 'sha256sum ~/incoming/disc.iso.part'

If the two digests match, I can rename the remote file. If they do not, I keep the failed object under its temporary name long enough to investigate and do not present it as the finished image.

A matching hash proves byte equality for those two reads. The disc can still contain read errors, an unmountable filesystem, or inconsistent application data. If the media is damaged, a recovery-oriented tool such as GNU ddrescue is a better choice than repeatedly forcing plain dd through read errors.

Do not put secrets in the remote command

The SSH command and its arguments can be visible to local process inspection, shell history, logging, and the remote shell. I use a normal SSH host alias and key or interactive authentication. I do not put a password, token, or private key body inside the pipeline.

Host-key checking matters too. OpenSSH records host identities and warns when an existing host key changes. I do not disable that check just to make an unattended copy proceed. A data stream encrypted to the wrong machine is still a failed transfer.

Choose the transfer by its failure mode

What I changed from the 2008 version

The old post was seven short paragraphs and two commands. It wrote directly to the final destination name, did not enable pipefail, did not validate the resulting archive or image, and offered no restart advice. It also assumed that a successful-looking shell prompt meant the copy was good.

I kept the part worth keeping: Unix tools can pass useful binary streams through standard input and output, and SSH can carry them without a local staging file. I replaced the part that aged badly: a clever one-liner is not a backup procedure.

For estimating whether the upload window is practical, I use the backup upload-time calculator. The one-terabyte planning guide covers sustained throughput and restore boundaries. The Ansible home-lab article shows how I separate repeatable configuration, observed snapshots, and actual recovery.

Where this command was tested

The command behavior is grounded in the official OpenSSH client manual, GNU tar manual, GNU dd documentation, and rsync manual.

I reviewed the current manuals and the original archived post for this replacement. I did not image physical media, send private data to another host, simulate a broken network, mount the resulting ISO, or perform a restore for this article. The commands use placeholder paths and hostnames and should be adapted only after checking the source and destination.