When transferring files from LCC/MCC to other storage options like OneDrive/Sharepoint, you may need to modify your files based on any limitations the new storage system has. While some transformation might have simple solutions, like renaming files, some storage systems may have limitations that required more complicated transformations. This document details such transformations and tools useful to do them.

Combine smaller files into one: tar

tar (tape archive) is a standard Linux command-line tool used to combine multiple files into one archive file. This is useful if the data you plan to archive consists of a large amount of small files. This tool also allows you to compress the resulting archive file to reduce total file size and more easily transfer data.

To generate a uncompressed archive file from a set of files, run the following command.

Add the -z option generate a compressed archive file.

If a specific compression method is needed (such as pigz to compress files in parallel), use the --use-compress-program option instead.

To later extract the files out of an archive file, run the following command. Add the -z option for compressed archive files.

Suppose I have a 500 GB collection of files in a directory  research-data/ where each file is less than 1 GB. I can create one 500 GB archive file research-data.tar by running the following command.

If the files were instead spread across multiple directories (research-data-1/ and research-data-2/), I could archive both of them together by running the following command.

If I want to instead generate a compressed archive file research-data.tgz, I would run the the following command.

Alternatively, if there were many files in research-data/ and I wanted to use the pigz compression method to compress files in parallel and speed up the process, I would run the following command.

Splitting a large file into smaller ones: split

split is another standard Linux command-line tool used to split a large file into multiple smaller files. This is useful if the data you plan to archive consists of a large amount of small files.

To split a file into multiple files with a upper limit on the size , run split with the -b option.

To instead split a file into a specific number of equally-sized files, run split with the -n option.

Suppose I had a 2 TB file data.csv. If I want to split it up into 200GB files, I would run the following command.

The command would result in 10 new files; data.csv.aa through data.csv.aj.

Alternative, I could have done the same thing by specifying the number of files instead.

If you want, you can change the suffix of generated files from letter to numbers with the -d option. If I ran the command with that flag for the previous example, the generated files would instead be data.csv.00 through data.csv.10.

To combine the files back into a single file, use the cat command on the split files to combine all of their contents into one source, then redirect the output of cat to a file.

For more information, visit the links below:

• tar man page: https://man7.org/linux/man-pages/man1/tar.1.html

• split man page: https://man7.org/linux/man-pages/man1/split.1.html