File Providers
The File Provider mechanism was introduced by Apple in macOS 11 and refined over subsequent OS releases. In macOS 13 it has become the defacto standard mechanism for implementing integrated cloud storage services on macOS. Major services such as Box, DropBox, Google Drive and Microsoft OneDrive are implemented as file providers. Apple has made it clear that this is the only approved mechanism for offering this capability, so eventually all cloud storage vendors will be implemented as file provider extensions.
Accessing cloud storage via the File Provider mechanism is transparent to most applications. Apps can create, delete, read and write document files to cloud storage just as they would to local storage. In fact, most applications are completely unaware of the fact that they may be accessing files on cloud storage versus local storage.
So “what’s the big deal” you ask? Shouldn’t sync & backup programs like ChronoSync just work with File Provider implemented cloud storage just like any other storage device? The short answer is “yes”, they should and do work seamlessly with file providers, and ChronoSync is no different. However, as with many things, the devil is in the details.
HOW FILE PROVIDERS WORK
A significant, often overlooked detail, is the fact that the File Provider mechanism is piggy-backed on top of your Mac’s local storage. When an app accesses a file that is stored “in the cloud”, it is really interacting with a copy of that file that is locally stored. All files read or written to cloud storage using a File Provider must pass through local storage and consume local storage space, albeit temporarily. So for example, when you use a service such as Google Drive and create a document in “My Drive”, it is first created in a hidden folder on your local storage device. It is then uploaded to the cloud in the background. Only after it has been completely uploaded, and the Google Drive software determines you are finished using it, will it be purged from local storage to free up space. This process is known as eviction.
Conversely, when you open a document stored on “My Drive”, it is first downloaded from Google’s servers to a hidden folder on local storage. Only when the download is complete will it open in your application. When you are finished with it, the Google Drive software may evict it from local storage or it may let it stick around for a while, consuming local storage space. When your storage space gets low, macOS will ask File Providers to evict files to clear up space, keeping your computer usable.
The mechanism of piggy-backing on local storage works well in most cases but it poses problems for backup and synchronization software such as ChronoSync. Imagine a situation where you have an external drive used to store photographs. You would like to place a backup copy of these photos up in the cloud using your Microsoft OneDrive account. This process would begin by copying the files to your OneDrive folder, which would actually be placing them on local storage until they can be uploaded to OneDrive. If you had a large amount of photos, say 500 GB worth, and your Mac’s internal storage was only 250 GB, you would quickly fill up all your local storage. You may find your Mac becoming sluggish and unusable until all the photo files could pass through local storage, be uploaded and eventually evicted.
The opposite scenario is equally bad. Suppose your primary photos were stored in the cloud using Box, perhaps to share them with other photographers. If you wanted to create a local backup on external storage, each one of those photos would be downloaded to your local storage before they could be copied to the external device. If the amount of data exceeds the size of local storage, you would quickly fill up local storage as this is happening. Again, your Mac may become sluggish and unusable until the software has a chance to catch up and evict all the files that have passed through local storage.
THE CHRONOSYNC DIFFERENCE
Where ChronoSync differs from other backup and synchronization software is that it takes into account the underlaying mechanism that File Providers use to store files into the cloud. ChronoSync realizes that when copying a file from cloud storage to another destination, you don’t want a copy of that file to linger around consuming local storage space. Thus ChronoSync can explicitly evict such files after a local copy is no longer needed.
Conversely, when copying files to cloud storage, ChronoSync can be instructed to evict these files when they are no longer needed. It will even wait for the files to completely upload to your cloud service before proceeding, which is the only way to guarantee that copied files have actually made it up into the cloud.
GETTING STARTED
Configuring a ChronoSync task to sync or backup cloud-stored files via a File Provider is not very different from any other sync or backup task. The first step is to instruct ChronoSync that a particular target is managed by a File Provider. To do this, choose “File Provider Extensions” from the “Connect to” pop-up menu for the desired target.
Note: If “File Provider Extensions” is dimmed, then ChronoSync has not detected any installed on your system.
Once “File Provider Extensions” is selected, click the “Choose” button to find the specific folder on your cloud service that will serve as this sync task’s target.
All installed File provider Extensions will appear in the left column of the target selector sheet window. They are analogous to “Volumes” that would appear in the standard file selector window. Simply choose the File Provider you are interested in and then navigate through the file hierarchy to find the desired folder that will serve as your target. Once found, click “Select”.
Once chosen, you will see that ChronoSync identifies the target folder as being managed by a specific File Provider. In this example, the target folder is “Disk Masters Backup” managed by the “Box” File Provider Extension.
Note: If you happen to navigate to a File Provider managed folder in Finder, and then drag it to the desired target, or if you happen to hit “Choose” without first selecting “File provider Extension”, you will see a long, fairly cryptic folder path displayed in your target.
This is actually the path to the hidden folder where the File provider maps its locally stored files. Selecting a target like this is not good because ChronoSync will not treat the target as being managed by a File Provider, and thus will not employ its special logic to download, upload and evict files.
As a convenience, ChronoSync will detect that you’ve incorrectly specified the target and will present a Readiness Error that prevents you from continuing.
To help you out of this predicament, click the “wrench” icon and choose “Fix” from the pop-up window that is displayed. This will automatically change the target to be File Provider aware, and will display your target folder properly.
Always make sure you have properly specified the target when dealing with cloud storage that is managed by a File Provider Extension. This is critical in order for ChronoSync to function properly. If you absolutely don’t want ChronoSync to be File Provider aware, click the “X” icon next to the Readiness Error and ignore the error. No extra logic will be utilized when running your sync task.
CUSTOM BEHAVIORS
Once a File Provider target has been specified, you can click the Target Options button to present a list of custom behaviors.
The Target Options sheet window appears as follows:
The “Strict volume identification” option has the same meaning as it always has for all types of targets. Refer to the online help if you need more information.
The following settings are where the real power lies and are what makes ChronoSync’s File Provider support unique among all macOS backup and synchronization applications.
“Evict downloaded files” indicates that any file that ChronoSync needs to download from cloud storage to your local storage will be purged from local storage when ChronoSync is finished with it. Thus if you are backing up many photos in the cloud to an external storage device, ChronoSync will need to download each one before it can copy them to external storage. With this option ON, ChronoSync will purge (or evict) the local copy when it is finished copying it. The file still exists in the cloud, it just won’t unnecessarily consume local storage space.
“Evict uploaded files” is the opposite setting. With this setting ON, any file that ChronoSync copies to cloud storage will be marked for eviction so that the file provider software will purge the local copy once it has finished uploading. This prevents an extraneous copy of the file laying around and consuming local storage space. Thus if you are backing up a lot of photos on an external device to cloud storage, you won’t have to worry about completely filling up local storage while the uploads are taking place.
“Wait for uploads to complete” is only available if “Evict uploaded files” is ON. This setting has significant implications. Normally, when copying files to cloud storage, they will reside in a folder managed by the cloud storage file provider until they are uploaded. This is why you generally want “Evict uploaded files” turned ON so that these files can be purged when upload is complete. However, what many people overlook is that the upload to cloud storage is not instantaneous. The time required to upload is directly proportional to the size of the files and the speed of your internet connection. In many cases, it can take quite a while to actually complete!
Copying files to a folder on cloud storage can give the appearance of near instantaneous completion but it may actually take several seconds… minutes… or even longer to complete. If you don’t wait for the uploads to actually finish, you might shutdown your computer or disconnect from the local network before they have been uploaded. This gives the appearance that you have backed up to the cloud but, in reality, the backup has not finished. Imagine doing this before leaving for a flight, losing your laptop on that flight, and finding out your precious backup did not complete before you left for the airport! The “Wait for uploads to complete” setting prevents this from happening. It will slow your backups/syncs down noticeably, but it is the only way to be sure your data is fully backed up to the cloud.
EFFECTIVE STRATEGIES
Using ChronoSync with File Provider Extensions works best when you are performing one of two classes of operations: 1) Backing up a collection of files to your cloud service, and/or 2) Backing up cloud-stored files to offline storage. What doesn’t work so well is attempting to backup your entire device (e.g. Home Folder or Data Volume) to the cloud via a File Provider. The sheer volume of files involved in such a backup makes the process extremely inefficient and time consuming. Likewise, attempting bidirectional synchronization in an effort to collaborate and share files with a workgroup tends not to work well. In such instances, it is better to use the collaboration tools provided by your cloud service provider.
BACKUP YOUR FILES
ChronoSync is perfectly suited to backup a collection of working documents that are stored locally to a cloud service, via the File Provider mechanism. In such instances, simply choose the folder that contains your working documents as your left target. Select the operation as “Backup left-to-right.” Then choose a folder on your desired cloud service as your destination.
Here we have configured a sync task to backup our “Illustrations” folder to an identically named folder managed by DropBox.
All the default settings work very well in this scenario. Files that don’t exist on the destination will get copied when the sync task is run. Afterwards, only files that are modified on the source target will get copied to the destination. Without making any other changes, ChronoSync will wait for each and every file to get uploaded to DropBox before moving on to the next file. It will also mark each of these files for eviction so they do not consume extra local storage space.
While the default configuration works perfectly fine, you can further customize the operation with ChronoSync-specific features such as rule and special file/folder handling. You can even enable archiving to maintain a history of modified and deleted files. Just be aware that archiving introduces a lot of File Provider overhead that will slow down your sync task significantly. You may want to schedule such tasks to execute at odd hours so as to not interfere with your use of the computer.
BACKUP YOUR CLOUD
Another perfect application of ChronoSync’s File Provider support is the concept of backing up your cloud. Cloud Storage is often viewed as redundant storage and a form of backup in its own right. In today’s computing landscape, however, more and more people are using cloud storage as their primary storage. This means working documents are created and modified within folders managed by a File Provider. The thinking of most users is that all changes are immediately getting synchronized to the cloud, and that provides an intrinsic form of backup.
The problem with this thinking is that it really doesn’t offer any form of redundancy. Keeping your working documents “live”, in the cloud, makes it even more likely that you’ll lose data. Most data loss comes from human actions, not from hardware or software failure. Thus if you accidentally delete files that only exist in the cloud, or make errant modifications to them, you may not be able to get back old copies. Sure, many cloud services offer the ability to “undelete” files, but that usually has a limited time window e.g. 30 days. If you don’t discover your lost data within that window, good luck getting it back! Not to mention accessing such files is completely dependent on an internet connection. If your internet goes down for whatever reason, accessing critical files that have been evicted from local storage will be impossible.
With this in mind, setting up a ChronoSync task to backup cloud-stored files to a local attached storage device is a very prudent strategy. Configuring such a task is the opposite of backing up to the cloud. Your source target would be the cloud managed folder that houses your critical data. The destination would be a folder on local storage, preferably an external storage device. The operation would be a simple backup.
Here we have configured a simple backup task that will backup our Google Drive “Documents” folder to a folder on an external SSD connected via Thunderbolt.
Just like when backing up your files to the cloud, the default settings work perfectly in this scenario. Any file that doesn’t exist in the destination will get copied when this task is run. Afterwards, only files that have been modified in “Documents” will get copied. Any file that needs to be downloaded from Google Drive to be copied to the external device will be evicted from local storage after the copy completes.
As with any ChronoSync task, you can customize this task to your heart’s content. Enabling archiving will keep historical copies of your data. You can even place the archive on another volume if you have an extensive history of file changes. And of course, the task can be scheduled to take place during odd hours so as to not interfere with your use of the computer.
INTERACTING WITH FILE PROVIDER MANAGED FILES
Once a sync task is configured, interacting with files in your sync tree is no different than any other local or remote sync task that you can configure in ChronoSync. There are a few enhancements that you should be aware of, however.
Evicted files can be identified in the Analyze Panel by inspecting the File/Folder Information area after selecting a file. If a file is presently evicted from the local system, a small cloud icon will appear where the file size is listed.
When a file that is being inspected is being managed by a File provider, you can also click the file’s icon to display a list of File Provider-specific options.
Specifically, “Download Local Copy” will be presented as an option if the file is presently evicted, as shown in the above screen shot. Selecting it, as the name implies, will force the file to be downloaded.
Conversely, if the file presently resides on local storage, “Evict Local Copy” will be an option. Selecting it, as the name implies, will evict the file.
It is handy to know that selecting “Download Local Copy” or “Evict Local Copy” on a folder will recursively perform that operation on all entries within that folder. This is a convenient way to download or evict an entire folder branch in one quick action.
FILE PROVIDER CAVEATS
It wouldn’t be modern computing without caveats! The File Provider mechanism implemented in macOS is very robust and powerful. Unfortunately, it is still a relatively new technology and hasn’t quite matured to the point that it is bulletproof. Part of this is due to Apple’s implementation at the OS level, but a significant part is the cloud service vendor’s File Provider Extension implementation. It will simply take time for all the kinks to get worked out.
With this in mind, we’ve prepared a tech note that summarizes some of the quirks and shortcomings of various popular cloud storage service and their File Provider Extensions. You can check it out here.
WHEN THINGS GO WRONG
For most people, syncing or backing up to cloud storage via File Providers should work without issues. However, we live in an imperfect world and sometimes things just don’t work out as expected. If you encounter any problems following this tutorial, contact our technical support team and ask! We don’t mind — we’re here to help!