TNCS-0031 – Understanding ChronoSync’s Progress Feedback
Created: November 3, 2015
ChronoSync v4.6.3 introduced enhancements to the progress feedback presented when running manual synchronizations and backups. In addition to the standard progress feedback bar and file counter, elapsed time and an effective throughput metric were added. In ChronoSync v4.6.4, we refined the throughput metric to more accurately reflect the actual file processing that was occurring. Many of our users welcomed this change but for some this has caused nothing but confusion. Given these recent changes and the fact that progress feedback has long been a hot topic, the need for this tech note became obvious. In it we will describe exactly what goes on behind the progress indicator.
TALE OF TWO SCANNERS: SINGLE-PASS AND TWO-PASS
When you initiate the standard “Synchronize” command, ChronoSync performs a single-pass scan of the file system. It reads the root folder of your targets and descends the folder hierarchy, processing items that need to be synchronized as it encounters them.
When you initiate the “Trial Sync” command, however, ChronoSync performs a two-pass scan of the file system. It descends the entire hierarchy on the first pass, making note of all items that need to be processed. It then presents the results to the user in the Trial Sync Selector sheet window, after which — once you choose to “Synchronize” — it makes a second pass and actually processes the files.
The single-pass scanner offers better performance but at the cost of not having any idea how many files will need to be processed or how big those files will be. It has no way of knowing until it encounters them and, since it processes such files when they are encountered, it cannot give any meaningful time estimate of how long the operation will take.
In fact, on its first-ever run, the one-pass synchronizer has no idea how many files are going to be scanned. Thus it displays an “indeterminate” progress feedback bar to show it is at least doing something. On subsequent runs, it takes an educated guess how many files it will scan based on the previous run. Thus a progress feedback bar that closely approximates the entire process is displayed.
The two-pass synchronizer has the benefit of knowing just how many files it is going to process and how big each of these files are. Thus after it completes the first pass (which can only display an indeterminate progress feedback bar until it completes the pass), a meaningful time estimate of how long the sync operation will take can be displayed. The drawback is that it may take some time to complete the first pass, during which you have no idea how long the complete process will take — not to mention a slower overall sync operation compared to the one-pass synchronizer.
PROGRESS FEEDBACK BAR AND FILE COUNTER
In both cases, the progress bar will indicate progress in terms of scanning files, not how long the overall sync operation will take. Thus the progress bar can — and usually does — advance at varying speeds. If you invoked the two-pass synchronizer (via Trial Sync), a fairly accurate estimate of remaining time is presented beneath the progress bar.
Another component of the progress feedback sheet is the file counter that appears at the lower left. This presents two numbers. The first is the number of files/folders scanned and the second is the number of files/folders discovered.
It works like this: suppose the root folder is loaded by ChronoSync and is determined to contain 1000 files. The ‘files discovered’ number (the second one displayed) is now 1000. ChronoSync then compares each of those files to each other and the ‘files scanned’ (the first number) increases by 1 for each one. Thus the file counter will appear like “1/1000”, “2/1000”, “3/1000” onward until “999/1000” and then “1000/1000”, at which time the sync operation is complete.
Things get more complicated as folders are encountered. Suppose in this example that the 900th entry in the root folder is another folder that contains 1000 more files. Suddenly the file counter will jump to “900/2000.” It will then count upward until “2000/2000” is reached. If more folders are encountered along the way, the ‘files discovered’ number will likewise increase.
On subsequent syncs, ChronoSync remembers the ‘files discovered’ value from the previous sync and it uses this as an educated guess for how many files will need to be scanned. Thus the ‘files discovered’ value begins with a large value and stays that way unless/until ChronoSync actually encounters more files than it did last time. This, in conjunction with the progress bar, provides a good indication of how quickly ChronoSync is scanning files and how many are left to scan.
THROUGHPUT METRICS
ChronoSync v4.6.3 introduced some new values in the progress feedback sheet. Both the elapsed time and effective data throughput of the sync are now displayed. The elapsed time is pretty easy to understand: how much time has transpired since the sync operation was initiated. This is updated in near-real time and provides a pretty good indication of how much time is left IF you know — generally — how long your sync operation typically takes to run. For example, if you know it normally takes 10 minutes to complete, you can assume it will take close to that amount of time on this run and thus can calculate approximately when it will finish.
The effective throughput figure, however, has caused much confusion. Initially it displayed the effective throughput of the sync operation as a whole. For example, if a sync operation takes 1 minute to scan 100,000 files but copied only 1 file that was 300K in size, the effective throughput was calculated as 300K/60 seconds yielding a pathetic throughput of 5K/sec.
We started to hear things like “How can this be? I’ve got a 100 MB/sec interface — why is it only copying at 1/20,000th that speed?” The answer is that when it copied that 300K file, it DID copy at the maximum speed that it could. It just spent the remaining 59.999 seconds scanning those 99,000 remaining files in the file system.
ChronoSync v4.6.4 sought to eliminate such confusion and now the throughput is calculated by
The explanation for the new figures boils down to the term ‘processing files’ that is used above. The throughput metrics are not measuring interface speed — they are measuring how quickly ChronoSync can ‘process’ a file. This typically involves opening the source file, reading data from it, creating the destination file, writing data to it, and closing both files. However, we’re not done!
It also involves deleting the replaced file (or moving it to an archive, possibly compressing it at the same time), renaming the destination file, reading back attributes from it, updating internal data structures and, if your sync document is so configured, reading back and verifying all the copied data. All this ‘processing’ adds up and thus will yield a value smaller that the theoretical maximum data throughput of the underlying hardware.
Hopefully this tech-note clears up the confusion.
REVISION HISTORY
Nov-03-2015 – Created from Internal Support Notes.