Uploading Data to LabCAS Server

Large datasets should be transferred to the LabCAS server via SFTP or WebDAV.

Very large datasets should be transferred to the LabCAS server via some means other than a web browser. At this time, LabCAS offers two ways to upload data directly: SFTP and WebDAV. Both protocols are supported on a variety of platforms including Linux, Mac OSX, and Windows. Before uploading data, please make sure to:

  • Organize your data in a directory structure corresponding to one collection and one or more datasets, as described above
  • For each dataset, provide as much metadata as possible in a file colocated with the dataset, and named <dataset>.cfg

Then proceed through one of the two methods below.

SFTP

In this section we describe how to upload data using a Unix-like system (macOS, Linux, FreeBSD, Solaris, etc.) and on Windows.

Unix

On Unix-like systems, do the following:

  • Generate a public/private RSA key pair with the following command, or use an existing pair if you have one:
    • ssh -keygen -t rsa
  • Send your public key (id_rsa.pub) to the LabCAS team so they can enter it in the list of allowed user keys
  • Simply use the SFTP client of your choice to connect to the LabCAS server (with no username or password, since authentication is provided by the key), navigate to your assigned root directory, create new directories for the Collection and Dataset to upload, then transfer the data (all files in the current local directory, including the metadata file <dataset>.cfg):

Windows

On Windows systems, you first generate a public/private key pair:

  1. Download and install both PuTTY and WinSCP. Both are free, open-source software packages.
  2. Installing PuTTY makes a new program available, PuTTYgen. Start this.
  3. At the bottom, select SSH-2 RSA and enter 2048 bits. (This should be the default.)
  4. Press "Generate".
  5. Move the mouse around in the blank area to generate random numbers until the progress bar is filled.
  6. Click "Save public key" and save the key to a file. Mail that file to JPL and we'll install it in the right place. Alternatively, you can click in the Public key box, select ALL the text, and paste that in an email. This key should be a single long line of text that starts with "ssh-rsa AAAA…"
  7. Enter a passphrase (and confirm it) then press "Save private key". Choose a filename (say "LabCAS") and save it to your Documents folder.
  8. Close PuTTYgen.

At this point, you'll need to await confirmation from JPL that your public key has been installed. Once you hear back, you can start uploading data with the following:

  1. Open WinSCP.
  2. Press the "Advanced" button.
  3. On the left side, under "SSH", click "Authentication".
  4. Under "Authentication parameters", click the ellipsis … button under "Private key file".
  5. Select the private key you saved in the Documents folder in step 7 (above). Click "OK".
  6. Under "Session", select SFTP. For Host name, enter ec2-13-52-172-197.us-west-1.compute.amazonaws.com; for user name, enter "sftpxfer".
  7. You will be warned the first time you're connecting to an unknown server. Press the "Copy Key" button. Then press "Yes".
  8. Enter the passphrase for the private key you created in step 7 (above). Check the box by "Remember password for this session". Then press "OK".
  9. You now have a window on the left with your local files. To send a file, select it on the left and click "Upload". Or simply drag and drop. You'll be prompted the first time to set up transfer settings. The defaults are fine, so check "Do not show this dialog box again" and press "OK".

 

Note that at this time uploading data through SFTP or WebDAV will NOT automatically trigger data publishing (although it will in the near future). Please contact the LabCAS team to let them know new data is available, and to coordinate the publishing phase.