Transferring Files from a Command Line
Command line interface data-transfer utilities (clients) are available on macOS and Linux systems, and through applications such as puTTY and Cygwin on Windows. The common utilities, sftp, rsync and scp, are described on this page.
SFTP: Secure File Transfer (Command-line-based)
SFTP is a command-line-based transfer tool that is the equivalent of a GUI-based sftp client. Like SSH, SFTP requires an initial login and authentication, and your session will remain open until you exit or are disconnected. You will remain connected to HPC and able to upload(put) and download(get) files without further login or authentication.
From your laptop, login to hpc-transfer.usc.edu and authenticate via DUO.
If it is your first time logging into hpc-transfer you will be asked Are you sure you want to continue connecting (yes/no)?. Answer “yes”. You will see the following once you are connected:
Connected to hpc-transfer.usc.edu. sftp>
The help command will display the available sftp commands.
sftp> help Available commands:
Use commands like pwd, ls, and cd, and their local equivalents lpwd, lls, and lcd, to navigate to the source and destination directories for file transfer.
# Navigate locally sftp> lpwd Local working directory: /Users/tommy sftp> lcd myimages sftp> lls myplot1.jpg myplot2.jpg # Navigate remotely sftp> pwd Remote working directory: /auto/rcf-40/ttrojan sftp> cd /home/rcf-proj/tt/ttrojan/images # Upload file from local to remote computer sftp> put myplot1.jpg myplot.jpg Uploading myplot1.jpg to /auto/rcf-proj/ttroj/myplot.jpg myplot1.jpg 100% 10KB 2.4MB/s 00:00
RSync: Remote Synchronization (Command-line-based)
Rsync is a fast and versatile transfer utility for synchronizing files and directories. It is a good choice for manually transferring large amounts of data on or to HPC because only new or updated files are copied.
The command synchronizes the directory on the local computer, /source/path, with the directory on the remote computer, /destination/path. If /destination/path doesn’t exist, it will be created.
rsync </source/path> <hostname:/destination/path>
Note: Unlike SFTP, login and authentication are requested for each use of the command.
The following example copies a file from a local computer to a project directory on HPC. The command was typed into the terminal window on a local computer.
rsync -av --progress myplot1.jpg email@example.com:/home/rcf-proj/tt/ttrojan/ [login and authenticate] building file list ... 1 file to consider myplot1.jpg 10079 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1) sent 10208 bytes received 42 bytes 1205.88 bytes/sec total size is 10079 speedup is 0.98
The a option specifies that it will be an archive (and so preserves permissions and modification times, and recursively transfers directories), the v option turns on verbose mode, and the progress option shows progress. Rsync provides many options—see the manual page (“man rsync”) for details.
SCP: Secure Copy Protocol (Command-line-based)
scp is another convenient way to transfer a single file or directory. The following description assumes that scp is being used to transfer files between your local computer and HPC. Due to security risks, please be mindful of the type of information being transferred. Where possible, omit all information that may be considered confidential. For examples of confidential information that requires additional consideration visit, itservices.usc.edu/security/sensitive-info.
Note: Unlike SFTP, login and authentication are requested for each use of the command. If you plan to do multiple transfers it is recommended that you use sftp — either the command line version or a GUI-based client, which will keep your session open so that multiple authentications are not required.
You can copy a file(s) from a local directory (/source/path) to a remote directory (/destination/path), and vice-versa.
scp </source/path> <hostname:/destination/path> scp <hostname:/source/path> </destination/path>
scp myplot1.jpg firstname.lastname@example.org:/home/rcf-proj/tt/ttrojan scp myplot1.jpg myplot2.jpg email@example.com:/home/rcf-proj/tt/ttrojan scp *.jpg firstname.lastname@example.org:/home/rcf-proj/tt/ttrojan
Similar to the copy (cp) command, you can copy whole directories using the -r option. The source directory will be copied (created) too.
scp -r ./myplots email@example.com:/home/rcf-proj/tt/ttrojan
scp -r firstname.lastname@example.org:/home/rcf-proj/tt/ttrojan/myplots .
The a option specifies that it will be an archive (and so preserves permissions and modification times, and recursively transfers directories), the v option turns on verbose mode, and the progress option shows progress. Rsync provides many options—see the manual page ("man rsync") for details.