Uploading Data to LabCAS Server

Large datasets should be transferred to the LabCAS server via SFTP or WebDAV.

Very large datasets should be transferred to the LabCAS server using SFTP. Before uploading data, please make sure to:

  • Organize your data in a directory structure corresponding to one collection and one or more datasets, as described above
  • For your data provide as much metadata as possible using the metadata worksheet. Colocate this with the data when you upload.

SFTP

Unix

On Unix-like systems, do the following:

  • Generate a public/private RSA key pair with the following command, or use an existing pair if you have one:
    • ssh -keygen -t rsa
  • Send your public key (id_rsa.pub) to the LabCAS team so they can enter it in the list of allowed user keys
  • Simply use the SFTP client of your choice to connect to the LabCAS server (with no username or password, since authentication is provided by the key), navigate to your assigned root directory, create new directories for the Collection and Dataset to upload, then transfer the data (all files in the current local directory, including the metadata file <dataset>.cfg):

Windows

On Windows systems, you first generate a public/private key pair:

  1. Download and install both PuTTY and WinSCP. Both are free, open-source software packages.
  2. Installing PuTTY makes a new program available, PuTTYgen. Start this.
  3. At the bottom, select SSH-2 RSA and enter 2048 bits. (This should be the default.)
  4. Press "Generate".
  5. Move the mouse around in the blank area to generate random numbers until the progress bar is filled.
  6. Click "Save public key" and save the key to a file. 
    • Send your public key (id_rsa.pub) to the LabCAS team so they can enter it in the list of allowed user keys
    Alternatively, you can click in the Public key box, select ALL the text, and paste that in an email. This key should be a single long line of text that starts with "ssh-rsa AAAA…"
  7. Enter a passphrase (and confirm it) then press "Save private key". Choose a filename (say "LabCAS") and save it to your Documents folder.
  8. Close PuTTYgen.

Once you hear back, you can start uploading data with the following:

  1. Open WinSCP.
  2. Press the "Advanced" button.
  3. On the left side, under "SSH", click "Authentication".
  4. Under "Authentication parameters", click the ellipsis … button under "Private key file".
  5. Select the private key you saved in the Documents folder in step 7 (above). Click "OK".
  6. Under "Session", select SFTP. For Host name, enter ec2-13-52-172-197.us-west-1.compute.amazonaws.com; for user name, enter "sftpxfer".
  7. You will be warned the first time you're connecting to an unknown server. Press the "Copy Key" button. Then press "Yes".
  8. Enter the passphrase for the private key you created in step 7 (above). Check the box by "Remember password for this session". Then press "OK".
  9. You now have a window on the left with your local files. To send a file, select it on the left and click "Upload". Or simply drag and drop. You'll be prompted the first time to set up transfer settings. The defaults are fine, so check "Do not show this dialog box again" and press "OK".

 

Note that at this time uploading data through SFTP will NOT automatically trigger data publishing (although it will in the near future). Please contact the LabCAS team to let them know new data is available, and to coordinate the publishing phase.