Storage and costs
Non-sensitive data (Curnagl)
Summary of storage file systems
The costs page is available here (UNIL login is required).
| Storage | Use it for | Copies | Safety | Quota cost (CHF/TB/year) | Accessible from |
| /users (home) | Configuration files | 1 | Safe | Free, quota is strict (50GB) | Curnagl login node |
|
/scratch
|
Temporary files of running jobs that need large amounts of data | 1 | No safety, may be deleted at any time | Free, no quota |
Curnagl login and compute nodes |
| /work | Data used in current computations | 1 | Moderate | With quota, see "U1 cost" for "GPFS (HPC - normal)" in the costs page | Curnagl login and compute nodes |
| /nas D2c | Important results of ongoing projects | 2 | Very Safe | With quota, add "U1 cost" for "Stockage copie primaire (Isilon)" and "Stockage 2ème copie (Isilon)" in the costs page | Curnagl login node and local mount (see wiki) |
| /nas D1c | Reloadable datasets of ongoing projects | 1 | Safe | With quota, see "U1 cost" for "Stockage copie primaire (Isilon)" in the costs page | Curnagl login node and local mount (see wiki) |
| LTS (tape) | Data of closed projects, archive | 2 | Very Safe | Free, no quota |
Once data is on tape: upon request (see wiki) |
| S3 | Share data with external collaborators | 1 | Safe | With quota, see "U1 cost" for "Stockage Objet" in the costs page. |
Curnagl login node and local mount (see wiki) |
Each PI has 1 TB of free /nas space, shared across all their open projects: 500 GB in D2c or 1 TB in D1c.
Where to store your non-sensitive data during a project
This diagram shows where to keep your data at each stage of a project.
Read it from left to right: each column is a project stage, each row a storage system, and the arrows show where data moves. Amber storage has a single copy or is temporary, so move important data out of it.
Letters refer to storage systems, which wiki pages are given below:
Each number links to a wiki page with detailed instructions:
- Transfert S3 DCSR to other support
- Transfer files to/from Curnagl
- Access NAS DCSR from the cluster
- How to run a job on Curnagl
- How much space am I using?
- Share files from a bucket with presign keys
- Give access to DCSR NAS files to an external collaborator
- Filetransfer from the cluster
- SWITCHfilesender from the cluster
- How to make an LTS request
Sensitive Data (Urblauna)
Summary of storage file systems
| Storage | Use it for | Copies | Safety | Quota cost (CHF/TB/year) | Accessible from |
|
/users (home) |
Configuration files | 1 | Safe | Free, quota is strict (50GB) | Urblauna login node |
|
/scratch
|
Temporary files of running jobs that need large amounts of data | 1 | No safety, may be deleted at any time | Free |
Urblauna login and compute nodes |
| /data | Data used in current computations | 1 | Moderate | See "U1 cost" for "GPFS (HPC - normal)" in the costs page | Urblauna login and compute nodes |
| /work (read-only from Curnagl) |
Read-only from Urblauna Software installed from Curnagl (with internet connection) |
1 | Moderate | See "U1 cost" for "GPFS (HPC - sensible)" in the costs page | Urblauna login and compute nodes in read-only |
| /archive (tape) | Data of closed projects or not accessed regularly | 2 | Very Safe | See "U1 cost" for "Archivage (sensible)" in the costs page |
Urblauna login node, quickly if data still in cache, very slowly otherwise (see wiki) |
Where to store your sensitive data during a project
This diagram shows where to keep your data at each stage of a project.
Read it from left to right: each column is a project stage, each row a storage system, and the arrows show where data moves. Amber storage has a single copy or is temporary, so move important data out of it.
Letters refer to storage systems, which wiki pages are given below:
Each number links to a wiki page with detailed instructions:
- Urblauna access and data transfer
- Installing your own software
- How much space am I using?
- Can I collaborate with external collaborators?
Good practices
- Move your results out of /scratch and /work (or /data for Urblauna) as soon as your computations are finished.
- Do not copy large read-only datasets to /scratch: /work (or /data for Urblauna) performance is more than sufficient for reading them.
- Using the
touchcommand to prevent files from being purged from /scratch is strictly forbidden. - Move data of closed projects:
- to LTS to free up /nas space for non-sensitive data,
- to /archive for sensitive data.
Quota increase
To increase a /nas or a /work quota, email helpdesk@unil.ch with "DCSR quota increase" in the subject line and the exact amount required. As this may increase quarterly invoices, we need the PI's approval before applying the change.