August 6, 2017 · Don't Forget

Downloading large files from a server running only SSH/SCP (or how to sync files through ssh shell)

Another one to add to the eternal bucket of "great simple commands to keep in mind". rsync's speciality lies in its ability to analyse files and only copy the changes made to files rather than all files. This can lead to enormous improvements when copying a directory tree a second time.

$ rsync --compress --partial --progress --rsh=ssh --archive user@host:/PATH/TO/SOURCE /PATH/TO/DESTINATION

It will even show a nice progress bar ;)

Another variation of this command:

rsync -e ssh -avz ./FOLDER ssh.server.address:NEW_FOLDER_LOCATION/

-a Archive mode, most likely you should always keep this on. Preserves file permissions and does not follow symlinks.
-v Verbose, lists files being copied
-z Enable compression, this will compress each file as it gets sent over the pipe. 
-e ssh Uses ssh as the transport, this should always be specified.

Additionally, assuming you have a bunch of folders you want to transfer, but they each have thousands of little files. . .

$ find . -type d -maxdepth 1 -mindepth 1 -exec tar cvf {}.tar {}  \;
$ find ./*.tar -exec gzip -S .gz -9 {}  \;

Then you can adjust your rsync to download all .gz files