Data management

This page presents the way to move data between the different locations used for either processing or storing data. The three main locations are hereafter referred to as: laptop, cluster, and NAS.

• Laptop: the personnal computer you use for your daily work (usually a laptop)
• Cluster: one of the clusters managed by the DCSR (wally or axiom)
• NAS: the Network Attached Storage from UNIL

Laptop <-> Cluster

Note: If you work from home, you first need to connect to Unil's VPN using Pulse Secure (see VPN instructions).

To copy data between your laptop and a cluster you will have to use scp (secure copy).

Linux/Mac

Linux and Mac users simply have to open a terminal.

From laptop to cluster: $ scp /path/to/fileToCopy <username>@<server>:/path/to/dest

From cluster to laptop: $ scp <username>@<server>:/path/to/fileToCopy /path/to/dest

Where:

<username>	your UNIL username
<cluster>	wally or axiom
<server>	<cluster>-front1.unil.ch

Examples

To copy file "file.txt" to your home directory:

scp file.txt ulambda@wally-front1.unil.ch:~

To copy the entire directory "dirA" to your home directory:

scp -r dirA ulambda@wally-front1.unil.ch:~

Windows

Windows users can use WinSCP.

Step 1: create a new session

Step 2: enter the configuration elements (SCP file protocol, hostname, and credentials). Finally, save the session details

 

Step 3: enter an explicit session name

Step 4: login

Step 5: enter your password

Step 6: navigate through folders (1 is for local files, 2 is for cluster files) and copy the files in one way or the other

 

Laptop <-> NAS

Note: If you work from home, you first need to connect to Unil's VPN using Pulse Secure (see VPN instructions).

Samba server

Windows

Step 1: right click on Network in File explorer and choose "Map network drive…"

Step 2: enter the address of the Samba server and click to finish

Step 3: enter your Unil credentials and clikc "OK"

• username: ad\unil_username  (for example ad\ulambda)
• password

Step 4: click on "This PC" and then on the new network drive

 

Linux

It is possible to use a GUI but there are as much solutions as desktop managers. So here is a way to mount the NAS using command line interface. Don't forget to adapt the username= field in the sudo mount command!

# First install the packages
# Debian/Ubuntu version
$ sudo apt install samba cifs-utils
# Fedora/RedHat version
$ sudo dnf install cifs-utils

# In the following, ~/nas is chosen as a mountpoint, but it could be elsewhere
$ mkdir ~/nasdcsr
$ sudo mount -t cifs -o username=unil_username,domain=ad,rw,iocharset=utf8,dir_mode=0700,file_mode=0700,uid=$(id -u) //nasdcsr.unil.ch/RECHERCHE ~/nasdcsr

# Perform the reauired operations
$ cd ~/nasdcsr
…

# When finished, don't forget to umount it
$ sudo umount ~/nasdcsr

 

Mac

Step 1: open the Finder, in Go menu, select "Connect to Server"

Step 2: enter the NAS url (smb://nasdcsr.unil.ch/RECHERCHE), you also may add this server as a favourite by clicking on + symbol

Step 3: enter your credentials

Step 4: navigate through the directories to find your working directory and push/fetch files as you see fit

Cluster <-> NAS

Note: If you work from home, you first need to connect to Unil's VPN using Pulse Secure (see VPN instructions).

From the login nodes the NAS is available under /nas. The folder hierarchy is:

/nas/FAC/<your_faculty>/<your_department>/<your_PI>/<your_project>

Cluster -> NAS

To copy a file to the new NAS:

cp /path/to/file /nas/FAC/<your_faculty>/<your_department>/<your_PI>/<your_project>

To copy a folder to the new NAS:

cp -r /path/to/folder /nas/FAC/<your_faculty>/<your_department>/<your_PI>/<your_project>

For more complex operations, consider using rsync. For the documentation see the man page:

man rsync

or check out this link.

NAS -> cluster

As above, just swapping the source and destination:

cp /nas/FAC/<your_faculty>/<your_department>/<your_PI>/<your_project>/file /path/to/dest

cp -r /nas/FAC/<your_faculty>/<your_department>/<your_PI>/<your_project>/folder /path/to/dest

Accessing DCSR NAS from CHUV network

Windows and Mac users can install the graphical desktop client from DataExpedition as well if they wish so:

Download Expedat from here, installation instruction from here, and product documentation from there.

Here is an example of ExpeDat session:

On that screenshot you can see: