Effective Cloud Strategies
ChronoSync 4.7.0 introduced the capability to directly connect to a cloud service for offsite backups. While opening a brand new vehicle for backing up data, cloud storage services vary widely with their capabilities and compatibility with macOS file system objects. This guide attempts to explain some of the drawbacks along with explaining the best techniques to minimize the deficiencies in order to get the most out of cloud-based backup.
This guide also breaks down some of the options presented to you during cloud service setup, giving detailed explanations. If you’re looking for a more step-by-step guide on an effective approach to setting up a cloud backup strategy, our video tutorial, Effective Cloud Strategies, shows you how to use an assistant to create and backup to a dmg and then backup that dmg to the Backblaze B2 cloud service.
EXPECTATIONS
Cloud-based Mac backup using ChronoSync offers cost-effective storage available from virtually any location with an internet connection. This ubiquitous availability makes it an attractive solution to backups, especially on-the-go. Being internet-based, security is an important feature of any cloud-based backup solution. ChronoSync defaults to industry-standard, secure protocols when communicating with cloud services. In addition, some services include the capability to further encrypt data stored with the service and ChronoSync supports these features where available. When evaluating any cloud-based backup solution, security needs to be the top priority.
This sounds great; Let’s get started! Except there is always a “but.” The overriding issue with cloud-based storage it that it can be slow. In simple terms, backing up to a cloud-based file system can be at least 100 times slower than a local file system (Internal or external hard drive or SSD).
For those interested in the details, the first topic is Internet Connections. In the USA, most homes have ‘Fast’ internet provided by telephone or cable television services. These wired connections advertise ‘maximum’ speeds that may or may not be achievable from a specific connection point. In addition, these numbers are usually expressed separately in Upload and Download speeds.
Take, for example, a local internet service that actually provides 7.5Mbps Download and 1.4Mbps Upload speeds. (Learn about ‘Mbps’ at: https://en.wikipedia.org/wiki/Data_rate_units#Megabit_per_second) This discrepancy in upload versus download speed is called an Asymmetric Connection. Downloads, files and data flowing to the connected computer, are faster than data that flows from the computer to some destination or site on the Internet. So files being backed up to a cloud service over this connection are being sent substantially slower than files being retrieved from the service.
There are may connections that are faster and some slower. Symmetric connections, where the upload and download speeds are the same, are usually available for an additional fee, but Asymmetric connections seem to be the norm in the USA.
Beyond the raw performance possibilities, a ChronoSync backup to a cloud service will incur substantial overhead beyond just copying data. ChronoSync updates metadata such as dates and attributes so that they match the source files on the local Mac. ChronoSync also employs a Safe Copy algorithm that copies the files to a temporary filename at the cloud service, confirms success, and only then replaces any existing file of the same name with the updated file. All of these operations add overhead to the connection and affect performance.
This discussion has yet to touch on the actual service used in the cloud-based backup. While these services can vary in performance and capabilities, ChronoSync takes an agnostic approach to communicating with them and backing up so as to minimize any differences seen from the synchronizer task.
Cloud service performance can be affected by the specific tier of service purchased (they offer varying levels of responsiveness for a fee) and they deal with thousands of operations every second–Except that last part is false. These cloud services are dealing with millions of operations per second, so it is not unusual to wait a few seconds, or many seconds, for the service to respond to a message sent from ChronoSync. On top of that, messages get lost sometimes over the internet and have to be resent. Let patience be the guide when working with cloud-based backups.
With that said, ChronoSync and cloud-based backup can be an effective layer in a multi-tiered strategy to keep your data safe. Econ Technologies recommends layering backups so that in the event of one failure, there are other layers that can cover the loss. Econ suggests a First Tier strategy that utilizes local, external storage, such as keeping a Bootable Backup next to the Mac.
The Second Tier of backups should utilize quickly-accessible remote storage such as a local Server, NAS, or remote drive connected to another Mac running ChronoAgent.
The Final Tier of backups should employ your cloud-based services. The cloud should be a backup of your backups because it can take time, both to get backed up, and to retrieve data in the event of catastrophe
CONFIGURATION
ChronoSync currently supports connections to Amazon S3, Google Cloud and Backblaze B2. Setup is pretty straightforward. Create an account with the desired service, setup a storage location commonly called a ‘Bucket’ and then configure a new Connection Profile in ChronoSync (ChronoSync->Preferences->Connections).
For detailed step-by-step instructions explaining account setup, storage creation, and creating a ChronoSync Connection Profile for Amazon S3, please refer to the guide ‘Configure and Backup to Amazon S3 Storage’. The same information for Google Cloud is available in the guide ‘Configure and Backup to Google Cloud Storage’. There is also a guide for setting up Backblaze B2: ‘Configure and Backup to Backblaze B2 Storage’
Each cloud service provider includes the capability to access files stored at the cloud service either through a web-page or app. Files and folders created by ChronoSync on these services are fully viewable, accessible, and available via these ‘Dashboard’ interfaces. Nothing created by ChronoSync is packaged in a proprietary bundle of data that is rendered unaccessible without ChronoSync. This allows easy access to confirm data is stored on the cloud, retrieve files, and view data from other devices.
While the focus here is on cloud-based storage it should be pointed out that ChronoSync also supports direct connections to SFTP servers, usually available from web hosting providers, NAS devices, and Network Servers. SFTP setup is covered in the guide ‘Configure a Mac as an SFTP Server and have ChronoSync Backup to it’.
OPTIMIZED CONNECTION SETTINGS
Beyond the initial setup, there are several settings available on the ‘Advanced’ panel of each ChronoSync Connection Profile. While these are specific to the connection type, cloud-based services share similar settings which are discussed below.
Caching
ChronoSync only caches attribute data, not any file data. Enabling caching can significantly reduce the number of requests made to the cloud service. Thus, Connection Profiles for Google, Amazon S3 and Backblaze B2 enable full caching by default. This is generally the preferred and best-performing setting, but won’t work well when backing up to a common destination folder.
This means any chosen folder on the cloud service where others also access files via ‘Dashboard’, other app, or even other tasks, concurrently. Unless shared access to files is absolutely necessary, (which is workgroup sharing, another use for cloud that we’ll cover in another guide), try to isolate backups by creating separate buckets or folders within the buckets such that only one user/sync task is ever accessing that bucket/folder. This keeps the “full” cache setting a safe option.
Other settings include ‘None’ and ‘Transient. ‘None’ will perform the slowest, as every time ChronoSync needs to evaluate file or folder attributes, a request to the cloud service is made. The ‘Transient’ setting limits caching to the life of the discrete action such as collecting the contents of a folder. Once that operation completes, the cache is purged. This can be useful if there is the possibility of simultaneous access to the files and folders from another app or sync task.
Metadata
Connection Profiles for Google, Amazon S3 and Backblaze B2 are configured to preserve all file metadata by default. This is an expensive operation since it increases the number of transactions per file as dates and other file characteristics are updated to match exactly the settings of the source file/folder on the local Mac. One should review this setting and determine if preserving all file metadata is necessary. Certainly if this particular backup is a last line of defense to cover against catastrophic data loss, perhaps preserving metadata isn’t that important. If this is the case, the ‘Metadata’ popup should be set to “none” to improve overall backup performance.
PACKAGE HANDLING
Package files are a common construct in the macOS file system. This is a single file such as an App or Photos Library file that is actually a container for a whole hierarchy of files and folders hidden within. An easy way to determine if a file is indeed a package, is to Control-Click (or Right-Click) the item in question and see if Finder displays the ‘Show package contents’ item in the contextual menu.
ChronoSync handles package files in a special manner. ChronoSync scans the entire contents of a package file to determine if any item inside the package has been modified since the last backup. This is a process referred to as ‘Package Aggregation’ and is used to determine if a package file has been changed and needs backup/updating by the sync task. The package is handled based upon the newest attributes found after reviewing all the items inside the package.
Aggregating packages is an expensive operation when performed on files stored on the local file system. Aggregating packages stored on a cloud service becomes extremely slow. ChronoSync provides capability to specify just how package files are handled as part of the sync or backup. On the ‘Options Panel’, in the ‘Special File/Folder Handling’ section is the ‘Package Handling’ popup menu.
The best performing ‘Package Handling’ setting when connecting to a cloud service for backup is ‘Dissect’. The ‘Standard’ setting will perform the package aggregation and incur the highest performance penalties. The ‘Merge’ setting behaves like ‘Standard’ when connecting to a cloud service.
LIMITING THE SCOPE OF DATA
With the knowledge that backing up to cloud services is likely to be substantially slower than backing up to local storage, it is important to evaluate just what needs to be stored in the cloud. While one could theoretically just choose the home folder and back it up to the cloud, that will result in a LOT of files getting backed up, most of which have little or no value. The better approach is to Instead focus on specific sets of data and perhaps create multiple backups for Documents, Pictures, Music, Movies, etc.
Know that items that change more frequently, such as active projects in the Documents folder may need backup more frequently than maybe Movies that are changed less frequently. Also, evaluate the necessity of what you backup to the cloud. Maybe personal photos of the family that are irreplaceable need an extra backup to the cloud whereas music, movies or apps purchased and readily available to re-download from the app store don’t really need backup to the cloud since they can be recovered elsewhere.
There are three general approaches to controlling the scope of items backed up. First, create separate tasks to backup specific folders or sub-folders rather than targeting the entire home folder. A second technique is to visit the Analyze Panel of the task and selectively ‘Exclude’ files and folder to prevent them from being synced. Finally, rules can be created that limit the scope of items backed up. These three techniques and specific steps to employ them are available in our guide ‘How to Backup Specific Files and Folders’.
BACKUP TO DISK IMAGES (DMGS)
Disk Images (DMG’s) are basically files that can be mounted and then accessed as a you would a physical drive. One type of Disk Image, the Sparse Bundle, can be created so that it dynamically expands to accommodate the size and number of files written to it. The Sparse Bundle Disk Image is especially useful in the context of cloud backup because of the structure of the DMG file. The Sparse Bundle is a package file comprised of a series of individual files called ‘Bands.’ Each band is a separate file and stores a portion of the data.
The idea is to create a disk image and then create a ChronoSync task that backs up to the DMG. The simplest way to do this is by using the ‘Backup to a Disk Image’ assistant. This assistant can create the disk image for you or be instructed to use an existing one. Whichever you decide, follow the step-by-step instructions from the assistant to set up your synchronization task to back up the desired data to the DMG.
Once you complete the steps of the assistant, ChronoSync can automatically mount a DMG to sync, even encrypted DMGs. This is considered a first-tier backup of important data to local, readily accessible storage. Learn the specifics of backing up to a Disk Image in the guide ‘Synchronizing with a Disk Image in ChronoSync v4.7’.
The easy next step is to create a task that backs up the DMG file to the cloud. This creates a two-tiered level of backups but also creates a more efficient backup to the cloud. ChronoSync can backup these band files to the cloud much more efficiently than it could the individual files that are stored to the DMG. Backing up several bands consisting of many megabytes of data each is much more efficient when communicating with a cloud service than backing up hundreds of smaller, individual files. This substantially reduces the overhead explained earlier as well as allows ChronoSync to employ bulk data upload operations.
The final step in this strategy is to take the two tasks and add them to a Task Container. Then schedule the Container to backup periodically. This runs the local backup to the DMG, then backs up the changes to the DMG to the cloud service as a single operation.
ARCHIVING REPLACED AND/OR DELETED ITEMS
ChronoSync includes the option to ‘Archive’ replaced and/or deleted items as part of the sync/backup. Archiving creates a special folder where older versions of files and folders are retained per the ‘Archive Handling’ settings on the ‘Options Panel’ of the sync task. The overhead of moving items to the archive and managing the archive in the cloud can be a slow operation and certainly does not take advantage of any archival features of the supported cloud services. Since the cloud-based backup is functioning as a backup of backups, archiving is not as important and will adversely affect performance. Disabling Archiving is recommended when syncing to cloud services.
WHEN THINGS GO WRONG
We live in an imperfect world and sometimes things just don’t work out as expected. If you encounter any problems when connecting and backing up to cloud services while using ChronoSync, here are some resources that may help you get back on track:
ChronoSync Help
From the “Help” menu in ChronoSync, you can try some of the following search terms (minus the quotes):
- “Amazon S3”
- “Google Cloud”
- “Backblaze B2”
- “Connections”
- “SFTP”
Guides
We have several guides that may expand upon the various topics covered in this guide:
Technical Support
Contact our technical support team and just ask! We don’t mind — we’re here to help!