TNCS-0051 – File Provider Quirks and Nuances
The File Provider mechanism built into macOS provides a convenient way for services such as Google Drive and DropBox to extend the local file system into the cloud. Unfortunately, this technology is relatively new and none of the mainstream cloud storage services offer a user experience that is yet to work flawlessly. Some of the shortcomings are due to problems with the File Provider mechanism itself, but most problems arise from lack of maturity in each of the major vendors’ implementations. This tech note describes many of the quirks and shortcomings of several File Provider implementations. Knowing the limitations of a File Provider implementation will help you properly setup your ChronoSync Sync Tasks.
Note: This tech note presents various quirks and shortcomings in several of the mainstream File Providers as of October 26th, 2023. It is our intention to periodically update this tech note to account for the fact that the various File Provider implementations should improve over time.
QUICK OVERVIEW
Before diving in to quirks and shortcomings, it’s important to know how the File Provider mechanism works. In essence, a folder on your local storage device is mapped to a specific File Provider. Applications can read and write from/to files stored in this mapped folder. When a file is written to storage, the associated File Provider will take care of the details of uploading the file to the cloud. Once all data has been uploaded, the File Provider may be asked by the OS to ‘evict’ the local copy to free up storage space. If an application attempts to access such an evicted file, the File Provider will step in and download (or ‘materialize’) the file from the cloud before the application can begin reading from it.
THE CHRONOSYNC CONNECTION
Most applications can access files managed by a File Provider without any knowledge that the file is being managed by a provider. ChronoSync, however, benefits from knowing that files are externally managed. Before reading from a file to copy it, for example, it will check to see if the file needs to be downloaded. When finished with the file, ChronoSync can ask the File Provider to evict the copy that was downloaded. Conversely, when ChronoSync is writing to a folder managed by a File Provider, it can wait for the file to be fully uploaded before moving on. It can also evict the local copy so as not to consume excessive space on local storage. All of this behavior can be controlled by clicking the Target “Options” button for a target that is managed by a File Provider.
REQUESTS & SUGGESTIONS
It is important to realize that applications such as ChronoSync do not explicitly control the process of download, upload or eviction of files. That is the sole responsibility of the File Provider extension for a specific cloud storage service. All ChronoSync can do is request a file be download, uploaded or evicted from storage. Whether or not the File Provider takes action is entirely up to the provider. For example, ChronoSync can request a copied file be uploaded but the File Provider may be busy performing other tasks at the time, and may not be able to honor that request in a timely fashion. Similarly, ChronoSync can suggest file eviction but a provider may decide it’s not efficient to evict at this time, thus the suggestion will go unanswered. ChronoSync has extensive logic to deal with a provider that is not honoring its requests but, ultimately, a File Provider may have a mind of its own and there’s nothing ChronoSync can do about that.
BEHAVIORS COMMON TO ALL FILE PROVIDERS
Below are some behaviors that occur regardless of what vendor you are using as a File Provider. Some of these behaviors are clearly bugs and/or limitations introduced by the File Provider mechanism that is implemented by the operating system.
- Files need to be downloaded to safely apply attributes. Whether you are updating a timestamp or applying a Finder tag, attribute-only changes can only be safely applied to a file that is locally stored. Thus if a file is presently evicted, it must be downloaded first before applying attributes. It can then be evicted to free up disk space. Failure to do this can result in the cloud-stored version of the file disappearing. This apparently is a bug at the OS level. Because of this, you may want to think twice about configuring a sync task to trigger on attribute changes.
- Files must be downloaded to move them (including to trash). If a file is being moved from one folder to another, it must first be fully downloaded before the move can take place. Failure to do this can result in the cloud-stored version of the file disappearing. This apparently is a bug at the OS level. Keep this in mind if you are synchronizing deletions in a sync task and/or are maintaining an archive.
- Cloud storage services can act unpredictably when you run out of storage. If you have paid for a certain amount of storage space, or are restricted by a free plan, you may encounter strange behavior when you have used up your allotted space. A File Provider may just simply stop working or it may appear that certain files just disappear from your system. This behavior is File Provider-specific. Some will behave better than others. If you start to get really strange behavior on a sync task (usually in the form of an indeterminate hang), check to see if you have run out of storage space.
- All files will have the ‘tracked’ BSD flag added to them. This isn’t really a problem, per se, but it will make it appear that BSD flags differ between source and destination files. This can cause problems if setting up a bidirectional sync or mirror operation. You may want to consider disabling the BSD flags sync trigger.
- A file that is moved, evicted and then followed by replacement of the original, moved item will result in data loss. This again is apparently a bug at the OS level. There is logic in ChronoSync to identify when this may be an issue. Pay attention to any Readiness Warnings/Errors that get presented and use the “fix” function to clear them, if it is available.
- The frequent download and eviction of files updates the timestamp of the folder containing them. This will result in a lot of attribute-only syncs for folder modification dates. Harmless, but something you should be aware of.
- Some file providers don’t treat invisible files nicely. You should consider enabling “Ignore Invisible” in the file handling section of Options just to be sure.
SPECIFIC FILE PROVIDERS
Below are behaviors specific to the major cloud storage vendors. Not all Fie Provider implementations are the same. You should take the time to evaluate a specific implementation and how it works with ChronoSync. If you’re noticing an odd behavior, feel free to contact support@econtechnolies.com and we’ll help determine if this is a File Provider quirk or something in ChronoSync that can be improved.
One Drive
- Doesn’t move to trash. Attempts to do so will delete the files immediately.
- Randomly downloads files post eviction. This means a file can be copied to a One Drive managed folder, uploaded, evicted but then mysteriously re-downloaded again on its own
- Files remain in a hybrid, uploaded-but-not-synchronized state for a while after being copied. After copying a file to a One Drive managed folder, the file can be uploaded fairly quickly but then it remains in a not-synchronized state for an indeterminate amount of time. In this state, it cannot be evicted.
- Package files are spastic but will eventually evict. When copying the contents of package files, the state of the package itself will alternate from uploading, to synchronizing, to eventually evicting. This happens even after ChronoSync is completely finished with the package file and has moved on to other files.
Google Drive
- Cannot be used for Bidirectional or Mirror Synchronization since it doesn’t preserve modification dates of copied files. This is a particularly worrisome behavior since the timestamp of files placed on Google Drive will not be preserved. If you pay close attention, they ARE preserved immediately after copy, but the Google Drive File Provider will then step in and update the modification date to the time of the copy. Mysteriously, attribute-only syncs WILL preserve modification date time stamps on files that already exist on Google Drive.
- Adds extended attributes to files to keep track of their state. This is another worrisome behavior because Google Drive is corrupting the integrity of extended attributes. Files copied from Google Drive may contain these state-tracking EAs, which could cause problems if the same files are copied back to Google Drive. At the very least, it will appear that the fidelity of a file’s metadata is not being maintained.
- Package files are spastic and don’t reliably evict. When copying the contents of package files, the state of the package itself will alternate from uploading, to synchronizing, to (possibly) evicting. This happens even after ChronoSync is completely finished with the package file and has moved on to other files.
Box
- Generates errors when files are moved to the trash. You should always choose “Delete immediately” instead.
- Package files are spastic and don’t reliably evict. When copying the contents of package files, the state of the package itself will alternate from uploading, to synchronizing, to (possibly) evicting. This happens even after ChronoSync is completely finished with the package file and has moved on to other files.
DropBox
- Adds extended attributes to files to keep track of their state. This is worrisome behavior because DropBox is corrupting the integrity of extended attributes. Files copied from DropBox may contain these state-tracking EAs, which could cause problems if the same files are copied back to DropBox. At the very least, it will appear that the fidelity of a file’s metadata is not being maintained.
- Doesn’t move to trash. Attempts to do so will delete the files immediately.
- Package files are spastic and don’t reliably evict. When copying the contents of package files, the state of the package itself will alternate from uploading, to synchronizing, to (possibly) evicting. This happens even after ChronoSync is completely finished with the package file and has moved on to other files.
iCloud Drive
There are two flavors of iCloud Drive. Which one you’re using depends on what OS version you are running. We’ve dubbed the original implementation as “iCloud Classic”, and it applies to all operating systems BEFORE macOS Sonoma. This implementation is not exactly via a File Provider Extension but it behaves similarly. On macOS Sonoma, iCloud Drive is implemented as a true File Provider. We refer to it as “iCloud Sonoma”.
iCloud Sonoma
- Finder Tag EAs are corrupted. They are still displayed properly in Finder but they have an internal corruption that prevents ChronoSync from displaying them properly. This is clearly an OS bug.
- Evictions are only possible if “Optimize Mac Storage” is enabled in the iCloud System Settings which are accessible when managing your Apple ID.
iCloud Classic
- No real problems have been observed.
- Evictions are only possible if “Optimize Mac Storage” is enabled in the iCloud System Settings which are accessible when managing your Apple ID.
CONCLUSION
The File Provider mechanism in macOS is a powerful way to to extend the file system beyond local storage. Unfortunately, at this time, it is not fully mature and free of flaws. This will certainly improve as macOS, and the vendor implementations, are progressively updated. For the time being, if we had to recommend a specific service to use with ChronoSync, we’d have to say that iCloud (especially iCloud Classic) is the clear winner.
REVISION HISTORY
Nov-02-2023 – Created.