Data Storage and Transfer

Owned by UKY CCS RCD

Aug 09, 2024

1 min read

Research data storage is a complex and difficult topic. There are many options in terms of location, media type, performance, cost and each has its own pros and cons.

Though each use case is unique, there is a basic concept of 3-2-1 backup rule that researchers should strive to follow for important data:

3 copies of data
2 different media types
1 offsite copy

The chart below is an overview of various storage types currently available to some/most of UK researchers. For more information, please read the corresponding sub-pages and contact us.

For further information regarding One-drive, Teams and Sharepoint, please see the ITS website:
https://its.uky.edu/news/how-choose-and-manage-document-storage-systems-uk?j=743169&sfmc_sub=251378705&l=21_HTML&u=30047096&mid=10966798&jb=6008

<style></style><div class="cells-worksheet" data-sheet-number="0" data-sheet-name="Sheet1"><table border="0" cellpadding="0" cellspacing="0" style="border-collapse:collapse;table-layout:fixed;width:1683pt"> <colgroup> <col class="x16" style="mso-width-source:userset;background:none;width:247px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:140px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:145px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:140px"></col> <col class="x16" span="2" style="mso-width-source:userset;background:none;width:145px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:146px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:138px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:145px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:142px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:141px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:140px"></col> <col class="x16" span="2" style="mso-width-source:userset;background:none;width:144px"></col> <col class="x16" style="mso-width-source:userset;background:none;width:143px"></col> </colgroup> <tbody> <tr style="mso-height-source:userset;height:40pt" id="r0"> <td class="x17" style="height:35.5pt;width:185.25pt">Storage</td> <td class="x17" style="width:105pt">LCC/MCC Scratch Space Storage</td> <td class="x17" style="width:108.75pt">Network Attached Storage (Gemini) paid Condo Storage</td> <td class="x17" style="width:105pt">Ceph Object Storage</td> <td class="x17" style="width:108.75pt">UKY Tape Archive -Paid Service</td> <td class="x17" style="width:108.75pt">OURRstore Tape Archival</td> <td class="x17" style="width:109.5pt">OSN (Open Storage Network)</td> <td class="x17" style="width:103.5pt">ACCESS Resource Provider Storage Space Allocation</td> <td class="x17" style="width:108.75pt">Sharepoint and One Drive</td> <td class="x17" style="width:106.5pt">Google Drive</td> <td class="x17" style="width:105.75pt">Dropbox</td> <td class="x17" style="width:105pt">Box (the other DropBox)</td> <td class="x17" style="width:108pt">Azure Blob Archival Storage (Paid Service via UKY)</td> <td class="x17" style="width:108pt">AWS Glacier (Paid Service via UKY)</td> <td class="x17" style="width:107.25pt">Lab Archives</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r1"> <td class="x18" style="height:11pt">Location</td> <td class="x19">On-site@UKY</td> <td class="x19">On-site@UKY</td> <td class="x19">On-site@UKY</td> <td class="x19">On-site@UKY</td> <td class="x19">Remote@Oklahoma</td> <td class="x19">Remote(Distributed)</td> <td class="x19">Remote(Distributed)</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> <td class="x19">Cloud</td> </tr> <tr style="mso-height-source:userset;height:76pt" id="r2"> <td class="x18" style="height:71.5pt">Purpose</td> <td class="x19">Mounted to HPC cluster. Short-term scratch.</td> <td class="x19">Stand-alone on-site disk storage. POSIX</td> <td class="x19">Stand-alone object store for lots of data</td> <td class="x19">Write once; read only if you lose all other copies</td> <td class="x19">Write once; read only if you lose all other copies; OU keeps one set of tapes and sends back another set to PI</td> <td class="x19">Stand-alone object store for lots of data</td> <td class="x19">Dpends on allocated HPC resource</td> <td class="x19">General</td> <td class="x19">General</td> <td class="x19">General</td> <td class="x19">General</td> <td class="x19">Object storage for lots of data</td> <td class="x19">Object storage for lots of data</td> <td class="x19">Electronic Research Notebook for collaboration, notes and data management</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r3"> <td class="x18" style="height:11pt">Storage Data "temperature"</td> <td class="x19">Hot</td> <td class="x19">Warm</td> <td class="x19">Cool</td> <td class="x19">Cold</td> <td class="x19">Cold</td> <td class="x19">Warm</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Various</td> <td class="x19">Hot</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r4"> <td class="x18" style="height:11pt">Non-research data</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">No</td> </tr> <tr style="mso-height-source:userset;height:38.5pt" id="r5"> <td class="x18" style="height:34pt">Can store HIPAA or FERPA data</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Depends on the allocated HPC resource</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">HIPAA compliant, seeking approval</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r6"> <td class="x18" style="height:11pt">Accessible by non-UK researchers</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> </tr> <tr style="mso-height-source:userset;height:51pt" id="r7"> <td class="x18" style="height:46.5pt">Quota (individual)</td> <td class="x19">25T</td> <td class="x19">Minimum purchase quota is 100TB</td> <td class="x19">No Individual Quota; only group quotas</td> <td class="x19">Based on quota purchased</td> <td class="x19">Based on TAPES purchased</td> <td class="x19">10TB</td> <td class="x19">Depends on the allocated HPC resource</td> <td class="x19">5T to 25TB</td> <td class="x19">15G</td> <td class="x19">2G; unlimited paid</td> <td class="x19">10G; unlimited paid</td> <td class="x19">Based on Quota purchased</td> <td class="x19">Based on Quota purchased</td> <td class="x19">Unlimited number of potential Notebooks and attached files under 4GB in size</td> </tr> <tr style="mso-height-source:userset;height:63.5pt" id="r8"> <td class="x18" style="height:59pt">Quota (group)</td> <td class="x19">50T</td> <td class="x19">Purchased Amount</td> <td class="x19">30T</td> <td class="x19">Based on quota purchased</td> <td class="x19">Based on TAPES purchased</td> <td class="x19"></td> <td class="x19">Depends on the allocated HPC resource</td> <td class="x19">5TB-25TB upon request(Will be reduced to 5TB shortly)</td> <td class="x19">None</td> <td class="x19"></td> <td class="x19"></td> <td class="x19">Based on Quota purchased</td> <td class="x19">BAsed on Quota purchased</td> <td class="x19">Can establish Guest or Group privaleges for sharing and access for each Notebook</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r9"> <td class="x18" style="height:11pt">Frequent Read</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Maybe</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">depends</td> <td class="x19">No (cost adds up)</td> <td class="x19">Yes</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r10"> <td class="x18" style="height:11pt">Frequent Write</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Maybe</td> <td class="x19">Yes</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">depends</td> <td class="x19">depends</td> <td class="x19">Yes</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r11"> <td class="x18" style="height:11pt">Frequent Modification</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">depends</td> <td class="x19">No</td> <td class="x19">Yes with Versioning</td> </tr> <tr style="mso-height-source:userset;height:15.5pt" id="r12"> <td class="x18" style="height:11pt">Web GUI for read/write</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">No</td> <td class="x19">Globus interface</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">Yes</td> <td class="x19">?</td> <td class="x19">Yes</td> </tr> <tr style="mso-height-source:userset;height:26pt" id="r13"> <td class="x18" style="height:21.5pt">Tools for read/write</td> <td class="x19">DTN node; Globus; Rclone etc.</td> <td class="x19">login node; Globus; Rclone etc.</td> <td class="x19">Rclone</td> <td class="x19"></td> <td class="x19">DTN node; Globus; Rclone etc.</td> <td class="x19">S3 (rclone etc.)</td> <td class="x19">Globus interface; rclone; etc</td> <td class="x19">Rclone, webGUI, Globus</td> <td class="x19">Rclone, webGUI, Globus</td> <td class="x19">Rclone, webGUI</td> <td class="x19">Rclone, webGUI</td> <td class="x19">WebGUI</td> <td class="x19">WebGUI</td> <td class="x19">Yes via Folder Monitor</td> </tr> <tr style="mso-height-source:userset;height:26pt" id="r14"> <td class="x18" style="height:21.5pt">Bandwidth (Maximum)</td> <td class="x19">LCC = 40Gb, MCC= 10Gb</td> <td class="x19">10Gb Node, 12Gb SAS</td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19">Internet Connection speed</td> </tr> <tr style="mso-height-source:userset;height:26pt" id="r15"> <td class="x18" style="height:21.5pt">Bandwidth (Average)</td> <td class="x19">5Gb/s</td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19">30~150MB/s using Globus</td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19"></td> <td class="x19">Internet Connection speed</td> </tr> <tr style="mso-height-source:userset;height:26pt" id="r16"> <td class="x18" style="height:21.5pt">Cost (TB)</td> <td class="x19">Free (see quotas)</td> <td class="x19">Condo/PAID</td> <td class="x19">Free (see quotas)</td> <td class="x19">Paid service</td> <td class="x19">ca. $12/TB for 2 copies</td> <td class="x19">Free</td> <td class="x19">Free</td> <td class="x19">Free (see quotas)</td> <td class="x19">Free (see quotas)</td> <td class="x19">Free (see quotas)</td> <td class="x19">Free (see quotas)</td> <td class="x19">Paid</td> <td class="x19">Paid</td> <td class="x19">Free for UKy Researchers</td> </tr> <tr style="mso-height-source:userset;height:126pt" id="r17"> <td class="x18" style="height:121.5pt">Good</td> <td class="x19">Convinient for HPC users</td> <td class="x19">Inexpensive. Long-term. POSIX, Globus endpoint</td> <td class="x19">Inexpensive storage. Easy sharing of data within the group.</td> <td class="x19">Less expensive than some other optinons.</td> <td class="x19">Cheap option for backing up massive quantity of data; Globus endpoint</td> <td class="x19">Easy to get access, but need to write a short proposal</td> <td class="x19">Easy to get access, but need to write a short proposal</td> <td class="x19">Almost unlimited storage space through UK</td> <td class="x19">Smooth intuitive Web GUI;</td> <td class="x19">Pricing for individual/team plans are comparable to Google Drive. Able to purchase as much space as needed (>5TB) without being on an enterprise plan.</td> <td class="x19">Claim better security protection than Dropbox; can upload large files; file transfer claimed to be faster than Dropbox.</td> <td class="x19">Appears to be cheaper than AWS Glacier (as of spring 2023) though this could also depend on use case</td> <td class="x19">Relatively inexpensive storage for large amount of data.</td> <td class="x19">Great for General Research Projects that are NOT Data Intensive, free, secure, backed up, collaborative. Can establish Guest or Group privaleges for sharing and access for each Notebook</td> </tr> <tr style="mso-height-source:userset;height:126pt" id="r18"> <td class="x18" style="height:121.5pt">Bad</td> <td class="x19">Short-term only</td> <td class="x19">Resiliency; reliability; Lack of file manager GUI other than Globus; partitions can be awkward</td> <td class="x19">Not intuitive. Steep linitial earning curve.</td> <td class="x19">Limited Use case. Minimal support. Expensive for tape storage,</td> <td class="x19">Data preparation is not trivial. File size must be 1GB~1TB. Initial set-up (tape purchase, tutorial session etc.) can be confusing.</td> <td class="x19">Storage is available only until the active proposal duration and not long term</td> <td class="x19">Storage is available only until the active proposal duration and not long term</td> <td class="x19">slow data transfer speeds; clumsy Web GUI; very confusing sharing options</td> <td class="x19">15G is nothing</td> <td class="x19">Expensive storage option for cloud providers.</td> <td class="x19">Not as good free storage space as Google Drive</td> <td class="x19">Users need to purchase this on their own.</td> <td class="x19">Users need to purchase this on their own. Egress charges are high</td> <td class="x19">Intended for notebook attachments rather than a place to store data in general. Max File size for upload is 16GB, NOT searchable. Access for large files via Links only.</td> </tr> <tr style="display:none"> <td style="width:185.25pt"></td> <td style="width:105pt"></td> <td style="width:108.75pt"></td> <td style="width:105pt"></td> <td style="width:108.75pt"></td> <td style="width:108.75pt"></td> <td style="width:109.5pt"></td> <td style="width:103.5pt"></td> <td style="width:108.75pt"></td> <td style="width:106.5pt"></td> <td style="width:105.75pt"></td> <td style="width:105pt"></td> <td style="width:108pt"></td> <td style="width:108pt"></td> <td style="width:107.25pt"></td> </tr> </tbody></table></div>

Center for Computational Sciences