NMR Data Transfer

Data can be transferred to the Spectrometer by any of the following means -

1. First loading files to the SFTP server and then downloading them to the spectrometer. This is normally available automatically by a sychronization process between the server and the spectrometers' computers. The files can be uploaded only into the directory: '/nysbc_nmr/home/username'. The other folders under 'nysbc_nmr' are not writable by the user.
2. If one is logged on to a Spectrometer, users are allowed to connect to their home institutions via ssh from the spectrometer to download files.
3. Both of these operations can also be carried out using the remote desktop connection.

Using the NYSBC SFTP Server

Introduction

1. Data generated by the spectrometers is available for download via SFTP. The data on the SFTP server is refreshed about every 10 minutes. The test for copying the file is that its modification date is in the period since the last mirroring. Some files will not then be mirrored -- e.g. tar restores with original creation times. Users may need to 'touch' these files to update them. Files deleted on the spectrometer are not deleted on the mirror. However, there is no version control, and overwriting a previous file will result in loss of the original data on the mirror.
2. You should be able to read data associated with your own user id, or the group id's you made readable on the original spectrometer computer. Your username and password will be the same as the username / password used to log on to the spectrometers.
3. The system will be unavailable for maintenance every Monday between 10 am and 2pm. The system may appear to be on-line during this period, but attempted transfers may fail.
4. We will make every effort to ensure that data is available on the mirror for 30 days. There is presently no provision for any regular restoration of data beyond 30 days.
5. For the spectrometer computers, we appreciate the need to preserve the small setup files, and the procedure will be to move 'large' (yet to be defined) files to ,[filename] at later than 60 days from original creation, and for these to be deleted 14 days later.
6. Your login directories will now be in the Bruker data space in the spectrometer computers. Any old data in /home/[loginname] will be moved.
7. The use of the old institutional logins is deprecated, The institutional logins will have the same user id number as the principal institutional user. That user can then remove, rename, etc. these files. The principal institutional user will be the NMR committee representative, unless we are otherwise informed. We recommend that this user make the files group-readable, if there are multiple users associated with the old account.
8. The recommended method for transferring files between spectrometers is to do so using the mirrored files on the sftp server. For security reasons, direct sftp between spectrometer computers will no longer be available. The use of the mirror should be entirely effective, with the caveat that data may be up to 10 minutes out of date.
9. There is a known issue while transferring large files that are still being written to - for example ser files. As the data on the SFTP server is refreshed every 10 minutes, a large file may be overwritten in the middle of a SFTP download, and the transfer will terminate with errors. Transfer of large files (>100mb) that are still being written to will fail. To download a large file, see the notes at the bottom here for special procedures.

Who to call

1. Please address technical issues about the transfers operations to:
   infotech@nysbc.org, x 420#
2. Please address issues about NMR files on the spectrometers to Dr. Bhattacharya at:
   sbhattacharya@nysbc.org, x 201#
3. Please address comments about the overall operation or suggestions for additional features to Dr. Goger.

Client Software

1. SSH Secure Shell for Workstations - Compatible client with both Windows and Unix is available at -
   http://www.ssh.com/support/downloads/secureshellwks/non-commercial.html. Unix versions may require complilation.
2. GFTP - GFTP is a compatible client for Linux systems available on Linux 7.2 and higher. It can be downloaded from
   http://gftp.org. To access the NYSBC SFTP Server Version 2.0.9 or higher is needed.

Connecting To The SFTP Server

• Using Windows GUI with SSH shell Type in Host Name and your User Name. Leave all other settings the same, as shown in figure 1

Figure 1

• Enter Password at password screen

Figure 2

• Change to the /nysbc_nmr directory to view a list of spectrometers.

Figure 3

• Change to your appropriate directory and transfer files using drag and drop or select, right click, and download.
• Using SFTP2

1. 1. Change directory to /nysbc_nmr.
   2. Spectrometers are listed in the /nysbc_nmr directory.

SFTP2 allows use of the following commands -

Transferring Large Files

• There is a known issue while transferring large files that are still being written to - for example ser files. As the data on the SFTP server is refreshed every 10 minutes, a large file may be overwritten in the middle of a SFTP download, and the transfer will terminate with errors. Transfer of large files (>100mb) that are still being written to will fail. To download a large file, one must either wait for the file to be written fully, or use rsync to transfer files.

Using rsync to Transfer Large Files

• Large ser files that are still being written to, can be transferred by using the following command (Linux/Unix only)-

rsync -avz --progress -e ssh username@sftp.nysbc.net:/nysbc_nmr/spect_name/topspin/data/username/nmr/filename /local_directory 

Note: Omitting the --progress switch causes some shells to timeout due to inactivity. Using the progress indicator keeps the shell alive.

Termination / Disaffiliation

• Set ALLOWTOPICVIEW =

-- DavidCowburn - 15 Mar 2006