Accessing Old Data

It is highly recommended that you access and backup all your data as soon as possible after your visit to Diamond. However, there are occasions when this is not possible and Diamond do continue to make your raw data available to you. With the large volumes of data collected at Diamond each day, data is only guaranteed to be stored on disk for 30 days after your experiment. Older data are stored on tape.

While still on disk, large sets (20GB or more) are more easily transferred using the globus service.

Data no longer accessible directly from the data directories on the beamline can be retrieved from the Diamond archive, Topcat (https://icat.diamond.ac.uk). The login are your Fed ID and password.

In additon to accessing data in the initial 30 days after your visit, you can also download it from ICAT, or restage it on the Diamond system. Note that when the data is restaged, the files will be found at a different link For I03, I04, I04-1, I23 & I24 this is:

• Restaged data - /dls/staging/dls/{beamline}/data/{year}/{visit}
• Original location, prior to archiving - /dls/{beamline}/data/{year}/{visit}

And for VMXi & VMXm these are the locations:

• Restaged data - /dls/staging/dls/mx/data/{proposal}/{visit}
• Original location, prior to archiving - /dls/mx/data/{proposal}/{visit}

If you were not an investigator on the original session or the proposal when the data were collected, the Principal Investigator or Alternate Contact can add you in UAS to the session, as a "Remote Investigator" for you to be able to access the data.

Note that the raw data and the processed information are stored in different locations:

• Raw data is stored at - dls/{beamline}/data/{year}/{visit}/{sample name or directory}...
• Processed results/logs are stored at - dls/{beamline}/data/{year}/{visit}/processed/{sample name or directory}

1. Once you have logged in to ICAT, search for the data that you want to find. You can also browse through all data that you have access to.

2. Your search can be facilitated by looking for data folders in ISPyB.

3. Navigate into the visit and select the directories you want. Include the processing one to get previous processing work from when the data was live, including that from the Autoprocessing Pipelines.

4. Once you have selected some data, the orange bar will appear and so click on "Go To Selection".

5. This will show you how much data you will need to download - which may be relevant when downloading data to your institution. 

6. When you are happy with your selection, click the "Download Selection" button - whether you want to restage or download it.

7. You will be presented with this window. To download data, you can choose between HTTPS and Globus. Globus is good for datasets over 20GB.

The options are:

• HTTPS
• Globus
• Restore to Diamond

Please heed the Data Restore policy - in particular new processing on the restaged data will not be kept, so you must download that within the 15 days. 

8. Retrieving data from tape takes time, so adding your email address here will trigger a notification when it is ready for you. Note this comes from a no-reply@diamond.ac.uk address.

9. Click "Download" to start the process.

Finally, please ensure you recieve the "Successfully submitted download request" before leaving ICAT.