Moving To and Between Cloud Storage Services
Need to move some data, files, buckets, containers, blobs or objects between clouds?
There are also tools for moving objects and blobs among different services. Some tools in my toolbox include AWS CLI (on my EC2 instances), S3 Browser, s3fs (maps S3 buckets to my Linux servers as a mount point), Cloudberry, S3motion (moves buckets, containers, objects, blobs between services or storage systems), curl and REX-Ray, among others. REX-Ray, for those not familiar, enables persistent container storage for Docker among other platforms including accessing EFS (when running in AWS) among other local solutions. Using S3fs, I can do to standard Linux type activities involving files, folders, sub-directors with S3 buckets as though accessing any standard NAS or file server.
The catch is that many of those rely on pulling data out of one cloud and uploading into another. One way to get around being in the middle is run an instance or container at one cloud send and receive at the other.
Sure there is still a process which may have a cost (compute time) in the middle; however, this approach saves on plugging up your network. Also, watch out for network charges when accessing and downloading data.
Another storage migration option is “seeding” by shipping a single HDD, SSD or tape to large scale appliances capable of processing petabytes through services such as AWS Snowball.
Cloud Storage Applications
Besides using cloud storage as a target from on-premise or within clouds, another related topic is storage applications running within cloud services. For example, NetApp ONTAP Cloud software can run on AWS and Azure compute instances. ONTAP is the software that traditionally makes up the FAS storage system appliance. Cloud ONTAP on an AWS (or Azure) instance functions like a traditional storage appliance, except it is running in the cloud using AWS storage such as EBS (block) and S3 (bulk object).
To other NetApp systems, ONTAP Cloud is simply another node accessible from within or outside the cloud service. By being in the cloud using cloud storage while communicating with other nodes, the need for a cloud gateway is also eliminated, not to mention that it provides high availability (HA), business continuity (BC) and disaster recovery (DR) capabilities.
NetApp is not alone with implementing software-defined storage cloud instances of their solutions. Another variation is that VMware has announced support for vSphere and vSAN along with NSX and other tools on AWS. However, VMware tools will not be running on the normal EC2 virtual machines. Instead, they will be on dedicated private servers (DPS). You too can have the equivalent of a DPS with AWS via their Lightsail offerings, and other cloud vendors have similar offerings.
Note that cloud instances such as those at AWS, Azure, Softlayer, Google and many others also support onboard, on-instance direct-attached SSD storage. However, those also tend to be ephemeral, meaning they are not persistent across instance start/stop. Look into your particular preferred service provider as to what they support. Also note that performance will vary between onboard cloud instance SSD storage and general cloud block, file and object resources.
Cloud Storage Options
Cloud storage of some type is in your future, and it might not be just bulk object or file sync and share. There are many different types of clouds and cloud storage. While object and bulk are the most commonly discussed by the industry, there are many other options. The reason for these other options is to simply address the various needs and requirements of those using clouds, as well as what their applications need today.
Greg Schulz is Founder of Server StorageIO, author of several books including Intel Recommended Reading List titles, and a new one available in early 2017 called “Software-Defined Data Infrastructure Essentials.” He is also a Microsoft MVP as well as a VMware vSAN vExpert. Learn more at www.storageio.com and www.storageioblog.com. Follow on Twitter @StorageIO.
Photo courtesy of Shutterstock.