Initial Sync after migrating Persistent Stores

[unniverzal]

Hello Paul.

When trying to migrate data from a NSSQLiteStoreType to SMStore type persistent store, the data is migrated to the new store but not loaded to Cloud. If data is added ,after migration in SMStore persistence store, or the previous data is edited then they are loaded to Cloud.

What I am doing is:

   do {
                SMStore.registerStoreClass()
                SMStore.syncAutomatically = true
                try persistentStoreCoordinator.migratePersistentStore(oldStore, to: cloudStoreURL, options: CoreDataStack.cloudKitPersistentStoreOptions, withType: SMStore.type)
                setupCloudKit()
            } catch {
                fatalError("Failed to migrate store: \(error)")
            }

 func setupCloudKit() {
        guard let seamStore = self.persistentStoreCoordinator.persistentStores.first as? SMStore else { return }
        self.seamStore = seamStore
        
        self.seamStore.recordConflictResolutionBlock = {
                (_ serverRecord:CKRecord,_ clientRecord:CKRecord, _ ancestorRecord:CKRecord) -> CKRecord in
            return serverRecord
        }
        
        validateCloudKitAndSync {
            print("finished")
        }
        print(self.seamStore)
    }

And the options are:

static var cloudKitPersistentStoreOptions: [String: Any] {
    return [
        NSMigratePersistentStoresAutomaticallyOption: true,
        NSInferMappingModelAutomaticallyOption: true,
        SMStore.SMStoreContainerOption: "iCloud.NameOfCustomContainer", SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)
    ]
}

[Perjan]

@paulw11 I am having the same issue. Can you please take a look into this?
Thanks in advance!

[tifroz]

There could be other issues, but one thing that jumped at me looking at your code, is that you are enabling syncAutomatically by default before the migration. It is a lot cleaner and safer to create some separation between the migration process and the initial sync:

1. Disable syncAutomatically
2. Perform the migration
3. Manually perform triggerSync
4. Upon completion of the initial sync, enable syncAutomatically so subsequent changes will be picked up.

Important note: I am not using the published version of Seam3 - I have a pending Pull Request #57 (PR is waiting for @paulw11 to merge)that does a few things to facilitate the migration, in particular:

• Adds a completion handler to the triggerSync method that makes toggling syncAutomatically after completion of the initial sync a lot easier.
• There are a some bug fixes that relate to mass changes (it is typical when performing a migration to dump of ton of data into the Seam3 store, which introduces some challenges compared to adding one record at a time)
• See the PR for more details

[unniverzal]

Thank you @tifroz for taking the time to answer.
I took your advice and tried to implement it, but with no success still. There could indeed be other issues, but I find it interesting that my previous data gets migrated to the new SMStore, but it just doesn't get synced. This is the full code that I'm running now to get a bigger picture.
@DJ-Glock seemed to have the same issue did you solve it completely?

func migrateToCloudStore() {
    guard let documentDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).last, let localStoreURL = self.storeURL() else { return }
    let cloudStoreURL = documentDirectory.appendingPathComponent("NewStore.sqlite")
    var oldStore: NSPersistentStore?
    if let _ = self.managedObjectModel {
        
        if let store = persistentStoreCoordinator.persistentStore(for: localStoreURL) {
            oldStore = store
        } else {
            oldStore = try? persistentStoreCoordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: localStoreURL, options: CoreDataStack.defaultPersistentStoreOptions)
        }
        
        guard let oldStore = oldStore else {
            fatalError("Failed to reference old store")
        }
        
        do {
            SMStore.registerStoreClass()
            SMStore.syncAutomatically = false
            self.seamStore = try persistentStoreCoordinator.migratePersistentStore(oldStore, to: cloudStoreURL, options: CoreDataStack.cloudKitPersistentStoreOptions, withType: SMStore.type) as? SMStore
            self.setupCloudKit()
        } catch {
            fatalError("Failed to migrate store: \(error)")
        }
    }
    print("**** Persistence store coordinator count: \(persistentStoreCoordinator.persistentStores.count)" )
}


func setupCloudKit() {
    
    self.seamStore.recordConflictResolutionBlock = {
            (_ serverRecord:CKRecord,_ clientRecord:CKRecord, _ ancestorRecord:CKRecord) -> CKRecord in
        return serverRecord
    }
    
    validateCloudKitAndSync {
        print("finished")
    }
    print(self.seamStore)
}

func validateCloudKitAndSync(_ completion:@escaping (() -> Void)) {
    self.seamStore?.verifyCloudKitConnectionAndUser() { (status, user, error) in
        guard status == .available, error == nil else {
            NSLog("Unable to verify CloudKit Connection \(error!)")
            return
        }
        guard let currentUser = user else {
            NSLog("No current CloudKit user")
            return
        }
        if let previousUser = UserDefaults.standard.string(forKey: "CloudKitUser") {
            if previousUser != currentUser {
                do {
                    print("New user")
                    try self.seamStore?.resetBackingStore()
                } catch {
                    NSLog("Error resetting backing store - \(error.localizedDescription)")
                    return
                }
            }
        }
        UserDefaults.standard.set(currentUser, forKey:"CloudKitUser")
        self.seamStore?.triggerSync(complete: true, fetchCompletionHandler: { (error) in
            guard error == nil else {
                print(error)
                return
            }
            self.seamStore?.syncAutomatically = true
        })
        completion()
    }
}

Store options are:

static var cloudKitPersistentStoreOptions: [String: Any] {
    return [
        NSMigratePersistentStoresAutomaticallyOption: true,
        NSInferMappingModelAutomaticallyOption: true,
        SMStore.SMStoreContainerOption: "iCloud.NameOfCustomContainer", SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)
    ]
}

Thank you once again.

[DJ-Glock]

@unniverzal Hey. Yes, I did. My code:

//
//  AppDelegate.swift
//  CafeManager
//
//  Created by Denis Kurashko on 03.05.17.
//  Copyright © 2017 Denis Kurashko. All rights reserved.
//

import UIKit
import CoreData
import CloudKit
import Seam3
import UserNotifications

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate, UNUserNotificationCenterDelegate {
    var window: UIWindow?
    var smStore: SMStore?
    
    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
        
        /// UIAppearance configuration
        ChangeGUITheme.configureThemeForApplication()
        
        // Seam3 configuring
        if #available(iOS 10.0, *) {
            let storeDescriptionType = AppDelegate.persistentContainer.persistentStoreCoordinator.persistentStores.first?.type
            if storeDescriptionType == SMStore.type {
                print("Store is SMStore")
                print()
                self.smStore = AppDelegate.persistentContainer.persistentStoreCoordinator.persistentStores.first as? SMStore
            }
        } else {
            let storeDescriptionType = AppDelegate.managedObjectContext.persistentStoreCoordinator?.persistentStores.first?.type
            if storeDescriptionType == SMStore.type {
                print("Store is SMStore")
                print()
                self.smStore = AppDelegate.managedObjectContext.persistentStoreCoordinator?.persistentStores.first as? SMStore
            }
        }
        
        // Closure for ClientTellWhichRecordWins conflict resolution way
        self.smStore?.recordConflictResolutionBlock =  ({(serverRecord, clientRecord, ancestorRecord) -> CKRecord in
            if let serverInt = serverRecord["intAttribute"] as? NSNumber,
                let clientInt = clientRecord["intAttribute"] as? NSNumber {
                if clientInt.intValue > serverInt.intValue {
                    serverRecord["intAttribute"] = clientInt
                }
            }
            return serverRecord
        } )
        
        // Save sync status after sync is finished
        NotificationCenter.default.addObserver(forName: Notification.Name(rawValue: SMStoreNotification.SyncDidFinish), object: nil, queue: nil) { notification in
            if notification.userInfo != nil {
                GenericStuff.syncStatus = "BAD"
                let appDelegate = UIApplication.shared.delegate as! AppDelegate
                appDelegate.smStore?.triggerSync(complete: true)
            } else {
                GenericStuff.syncStatus = GenericStuff.convertDateToString(inputDate: Date())
            }
        }
        
        let notificationSettings = UIUserNotificationSettings(types: .badge, categories: nil)
        application.registerUserNotificationSettings(notificationSettings)
        application.registerForRemoteNotifications()
        return true
    }
    
    func application(_ application: UIApplication, didReceiveRemoteNotification userInfo: [AnyHashable : Any], fetchCompletionHandler completionHandler: @escaping (UIBackgroundFetchResult) -> Void) {
        print("Received push")
        self.smStore?.handlePush(userInfo: userInfo)
        completionHandler(.newData)
    }
    
    // Function to start sync
    func validateCloudKitAndSync(_ completion:@escaping (() -> Void)) {
        self.smStore?.verifyCloudKitConnectionAndUser() { (status, user, error) in
            guard status == .available, error == nil else {
                NSLog("Unable to verify CloudKit Connection \(error!)")
                return
            }
            guard let currentUser = user else {
                NSLog("No current CloudKit user")
                return
            }
            let previousUser = UserDefaults.standard.string(forKey: "CloudKitUser")
            if  previousUser != currentUser && previousUser != nil {
                do {
                    print("New user")
                    try self.smStore?.resetBackingStore()
                } catch {
                    NSLog("Error resetting backing store - \(error.localizedDescription)")
                    return
                }
            }
            UserDefaults.standard.set(currentUser, forKey:"CloudKitUser")
            self.smStore?.triggerSync(complete: true)
            completion()
        }
    }
    
    // Lifecycle functions
    func applicationWillResignActive(_ application: UIApplication) {
        // Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
        // Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game.
    }
    
    func applicationDidEnterBackground(_ application: UIApplication) {
        // Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
        // If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
    }
    
    func applicationWillEnterForeground(_ application: UIApplication) {
        // Called as part of the transition from the background to the active state; here you can undo many of the changes made on entering the background.
    }
    
    func applicationDidBecomeActive(_ application: UIApplication) {
        // Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
    }
    
    func applicationWillTerminate(_ application: UIApplication) {
        // Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:.
        // Saves changes in the application's managed object context before the application terminates.
        self.saveContext()
    }
    
    // MARK: - Core Data stack for iOS 10+
    @available(iOS 10.0, *)
    static var persistentContainer: NSPersistentContainer = {
        let container = NSPersistentContainer(name: "CafeManager")
        let persistentStoreCoordinator = container.persistentStoreCoordinator
        
        // Preparing URL
        let applicationDocumentsDirectory: URL = {
            let urls = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)
            return urls[urls.count-1]
        }()
        
        // Initializing Seam3
        SMStore.registerStoreClass()
        SMStore.syncAutomatically = false
        
        let newURL = applicationDocumentsDirectory.appendingPathComponent("CafeManagerSeam3.sqlite")
        
        // Check if SQLite store has been already migrated by checking if CafeManagerSeam3.sqlite exists.
        let seamStoreExists = FileManager.default.fileExists(atPath: newURL.path)
        
        if seamStoreExists {
            // If exists, then use it because it has been already migrated to Seam3 storage
            print("Already migrated, using \(newURL)")
            
            let storeDescription = NSPersistentStoreDescription(url: newURL)
            storeDescription.type = SMStore.type
            storeDescription.setOption("iCloud.iGlock.CafeManager.com" as NSString, forKey: SMStore.SMStoreContainerOption)
            storeDescription.setOption(NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue), forKey:SMStore.SMStoreSyncConflictResolutionPolicyOption)
            container.persistentStoreDescriptions=[storeDescription]
            
            container.loadPersistentStores(completionHandler: { (storeDescription, error) in
                if let error = error as NSError? {
                    fatalError("Unresolved error \(error), \(error.userInfo)")
                }
            })
            print("Current persistent store descriptions are: \(container.persistentStoreDescriptions)")
            return container
            
        } else {
            // If does not exist, then migrate old storage to Seam3.
            print("Not yet migrated, migrating to \(newURL)")
            
            // Loadig default store
            container.loadPersistentStores(completionHandler: { (storeDescription, error) in
                if let error = error as NSError? {
                    fatalError("Failed to load default store \(error), \(error.userInfo)")
                }
            })
            let defaultPersistentStore = container.persistentStoreCoordinator.persistentStores.last
            print("Default store is located here: \(defaultPersistentStore!.url!)")
            
            // Migrating default store to new Seam store
            let options: [String : Any] = [SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com"]
            do {
                try persistentStoreCoordinator.migratePersistentStore(defaultPersistentStore!, to: newURL, options: nil, withType:SMStore.type)
            }
            catch {
                fatalError("Failed to migrate to Seam store: \(error)")
            }
            return container
        }
    }()
    
    // MARK: - Core Data stack for iOS 9 (8+)
    static var managedObjectContext: NSManagedObjectContext = {
        var managedObjectModel: NSManagedObjectModel = {
            let modelURL = Bundle.main.url(forResource: "CafeManager", withExtension: "momd")!
            return NSManagedObjectModel(contentsOf: modelURL)!
        }()
        
        // Preparing URL
        var applicationDocumentsDirectory: URL = {
            let urls = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)
            return urls[urls.count-1]
        }()
        
        // Initializing Seam3
        SMStore.registerStoreClass()
        SMStore.syncAutomatically = false
        
        let oldURL = applicationDocumentsDirectory.appendingPathComponent("SingleViewCoreData.sqlite")
        let newURL = applicationDocumentsDirectory.appendingPathComponent("CafeManagerSeam3.sqlite")
        
        // Check if SQLite store has been already migrated by checking if CafeManagerSeam3.sqlite exists.
        let seamStoreExists = FileManager.default.fileExists(atPath: newURL.path)
        
        if seamStoreExists {
            // If exists, then use it because it has been already migrated to Seam3 storage
            print("Already migrated, using \(newURL)")
            
            var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
                let coordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
                do
                {
                    let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                   NSInferMappingModelAutomaticallyOption: true,
                                   SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com",
                                   SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)]
                    try coordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: newURL, options: options)
                } catch {
                    NSLog("Error initializing smStore for iOS 8+ - \(error.localizedDescription)")
                }
                return coordinator
            }()
            
            var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
            managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
            return managedObjectContext
        } else {
            // If does not exist, then migrate old storage to Seam3.
            print("Not yet migrated, migrating to \(newURL)")

            var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
                let coordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
                
                // Initializing old store
                do {
                    let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                                   NSInferMappingModelAutomaticallyOption: true]
                    let oldStore = try coordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: oldURL, options: options)
                    
                    // Migrate old store to Seam3
                    do
                    {
                        let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                                       NSInferMappingModelAutomaticallyOption: true,
                                                       SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com",
                                                       SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)]
                        try coordinator.migratePersistentStore(oldStore, to: newURL, options: options, withType: SMStore.type)
                    } catch {
                        NSLog("Error initializing smStore for iOS 9 - \(error.localizedDescription)")
                    }
                } catch {
                    NSLog("Error initializing default NSSQLiteStore for iOS 9 - \(error.localizedDescription)")
                }
                
                return coordinator
            }()
            
            var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
            managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
            return managedObjectContext
        }
    }()
    
    // MARK: View Context for all iOS versions
    static var viewContext: NSManagedObjectContext {
        if #available(iOS 10.0, *) {
            return persistentContainer.viewContext
        } else {
            return AppDelegate.managedObjectContext
        }
    }
    
    // MARK: Core Data Saving context for all iOS versions
    func saveContext () {
        let context = AppDelegate.viewContext
        if context.hasChanges {
            do {
                try context.save()
            } catch {
                let nserror = error as NSError
                fatalError("Unresolved error \(nserror), \(nserror.userInfo)")
            }
        }
    }
}

[paulw11]

I have now merged @tifroz PR into the master branch. I will update the POD shortly

[DJ-Glock]

@unniverzal
Actually my issue was caused by SQLite file that I have taken from real device and put into simulator.
I have tested many times with normally created file in simulator, the first sync after migration works fine.
Sorry if confused you.

[unniverzal]

@DJ-Glock Hey I went through your code when are you calling validateCloudKitAndSync function? And on your managedObjectContext when seamStoreExists you add a persistence store with NSSQLiteStoreType rather than SMStore is there a reason for that?

[DJ-Glock]

@unniverzal Hey. Nowhere. I have decided to not use auto sync and it is run manually by pressing the button. Not an ideal solution because of issue #61 :(
So you can follow advice from tifroz.

[DJ-Glock]

@unniverzal Hey!
I have just found veeery bad mistake in my code that was deployed to production :(
I forgot to change type of storage here:
try coordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: newURL, options: options)
So correct way will be:

if seamStoreExists {
            // If exists, then use it because it has been already migrated to Seam3 storage
            print("Already migrated, using \(newURL)")
            
            var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
                let coordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
                do
                {
                    let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                   NSInferMappingModelAutomaticallyOption: true,
                                   SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com",
                                   SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)]
                    try coordinator.addPersistentStore(ofType: SMStore.type, configurationName: nil, at: newURL, options: options)
                } catch {
                    NSLog("Error initializing smStore for iOS 8+ - \(error.localizedDescription)")
                }
                return coordinator
            }()

[bcye]

@DJ-Glock Hey I am trying to figure this out myself.

What exactly are you doing here?

if seamStoreExists {
            // If exists, then use it because it has been already migrated to Seam3 storage
            print("Already migrated, using \(newURL)")
            
            var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
                let coordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
                do
                {
                    let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                   NSInferMappingModelAutomaticallyOption: true,
                                   SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com",
                                   SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)]
                    try coordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: newURL, options: options)
                } catch {
                    NSLog("Error initializing smStore for iOS 8+ - \(error.localizedDescription)")
                }
                return coordinator
            }()
            
            var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
            managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
            return managedObjectContext
        } else {
            // If does not exist, then migrate old storage to Seam3.
            print("Not yet migrated, migrating to \(newURL)")

            var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
                let coordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
                
                // Initializing old store
                do {
                    let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                                   NSInferMappingModelAutomaticallyOption: true]
                    let oldStore = try coordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: oldURL, options: options)
                    
                    // Migrate old store to Seam3
                    do
                    {
                        let options: [String : Any] = [NSMigratePersistentStoresAutomaticallyOption: true,
                                                       NSInferMappingModelAutomaticallyOption: true,
                                                       SMStore.SMStoreContainerOption: "iCloud.iGlock.CafeManager.com",
                                                       SMStore.SMStoreSyncConflictResolutionPolicyOption: NSNumber(value:SMSyncConflictResolutionPolicy.clientTellsWhichWins.rawValue)]
                        try coordinator.migratePersistentStore(oldStore, to: newURL, options: options, withType: SMStore.type)
                    } catch {
                        NSLog("Error initializing smStore for iOS 9 - \(error.localizedDescription)")
                    }
                } catch {
                    NSLog("Error initializing default NSSQLiteStore for iOS 9 - \(error.localizedDescription)")
                }
                
                return coordinator
            }()
            
            var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
            managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
            return managedObjectContext
        }

Sepcifically, why are you creating a coordinator if the Seam store exists?