SyncServerII Framework

SyncServerII is a framework and server with these main goals:

Contents:

Development Status

Use Cases

1) Private Social Media

Current social media: Data is stored on corporate servers; users can lose ownership of data both “physically” and in terms of intellectual property.

If mobile app developer makes use of SyncServer:

  • Data content (e.g., images) is stored in app users cloud (e.g., Google Drive)
  • Data can also be safely shared with friends through various sign-in methods (e.g., Facebook)
  • Users retain intellectual property rights
  • The data is not searchable on search engines, and has that privacy.

2) Data Longevity

Apps sometimes hide their data: For example, you enter your favorite restaurant meals and preferences, and images of those events into an app

  • App stores data internally
  • If app stops being supported, users lose access to data
  • If you lose your device or your backup fails, you lose access to data

If mobile app developer makes use of SyncServer:

  • Data is stored in normal files (not in special database or otherwise obscured formats)
  • Data is stored in normal places in users cloud storage.
    • E.g., in Dropbox, /Apps/AppName/YourFiles
  • Also enables backup and recovery in the same manner

3) Developers Save $$$

  • Why should developers or small companies pay for storing users data?
  • SyncServer stores content data in users own cloud storage
  • Meta-data (e.g., file identifiers, version numbers), & some user credentials are stored in SyncServerII
  • Estimate from SharedImages: SyncServerII data is about 1/100th size of user owned content data

Concepts and Terminology

Client Interface Documentation

The API for the iOS client interface is documented here.

Current Limitations

Standing up a Server

Each separate mobile app needs its own server. That is, for iOS, each app with a separate bundle id needs its own server.

I have gone through a few incarnations of methods for installing the server, each of them on AWS. Currently, I am using a Docker image, and also use AWS Elastic Beanstalk. Previously, I was building the server directly on an AWS EC2 instance.

The two basic parts of my current deployment method are described below.

Docker

Docker is now central to my testing and deployment process for the server. For testing, I use the same Docker image that I use for building.

On my system, the following command launches that Docker image:

docker run --rm -i -t -v /Users/chris/Desktop/Apps/:/root/Apps crspybits/swift-ubuntu:4.0.0

And, of course, the path `/Users/chris/Desktop/Apps/` is specific to my system.

And running under that Docker image, this builds the server:

swift build -Xswiftc -DDEBUG -Xswiftc -DSERVER

For runtime, I use this Docker base-image and I run the script to create a Docker image for a specific release of the SyncServerII server code. Those release builds of the server Docker images live at hub.docker.

AWS Elastic Beanstalk

I use AWS's Elastic Beanstalk for reasons including:

  1. ease of deploying new versions of my server
  2. its ability to run servers from Docker images
  3. the relative ease of spinning up (and down) of new server environments-- so I can create and use a staging or testing environment for just a few hours
  4. my (potential) eventual need for scaling the server to deal with a larger load from client apps.

My deployment process to the AWS Elastic Beanstalk takes several steps, some of them manual.

Server configuration

When the server starts, it reads configuration information from a JSON file. The following details the format of that configuration file. 

            
  {
    "mySQL.host": "URL-for-mySQL-server",
    "mySQL.user": "usernameForMySQL",
    "mySQL.password": "passwordForMySQL",
    "mySQL.database": "mySQL-database-name",
    "port": "8080",
    "allowedSignInTypes.Facebook": "true",
    "allowedSignInTypes.Google": "true",
    "allowedSignInTypes.Dropbox": "true",
    "ssl.selfSigning": "false",
    "GoogleServerClientId": "Get this when you create your Google app",
    "GoogleServerSecret": "as above",
    "FacebookClientId" : "Get this when you create your Facebook app",
    "FacebookClientSecret": "as above",
    "owningUserAccountCreation.initialFileName": "IMPORTANT_README.txt",
    "owningUserAccountCreation.initialFileContents": "IMPORTANT: The files placed in this directory should not be changed in any way. They were created by the EXAMPLE app. If they are changed, that can cause the EXAMPLE app to crash or behave unpredictably. Of course, if you want to stop using EXAMPLE, these are your files, and you are fully able to use them as you wish."
  }

When a new owning user account is created (i.e., one with cloud storage), a single non-shareable file is first created. That file has a name given by owningUserAccountCreation.initialFileName and contents given by owningUserAccountCreation.initialFileContents. The intent is that this file serves as a warning to users to not modify the files used by the app.

Installation in a new iOS app

The best instructions for setting up a new iOS app are likely found by looking over the SharedImages app, but some details are also given below.

The example app in the iOS client repo can also provide information on setting up your iOS app.

  1. Setup your Podfile.
    2. The SyncServer iOS client API is a Cocoapod. Google Drive and Dropbox are the current options for cloud storage. Facebook can be used to allow shared sign-in's, and is optional. 
      # Some pods used by SyncServer haven't yet been released to cocoapods.org, so the following is needed.
  source 'https://github.com/crspybits/Specs.git'
  source 'https://github.com/CocoaPods/Specs.git'

  use_frameworks!

  target 'YourApp' do
      # The iOS client.
      pod 'SyncServer'
      
      # Only needed if you are allowing Facebook users to share.
      pod 'SyncServer/Facebook'

      # Only needed if you are using Dropbox for cloud storage
      pod 'SyncServer/Dropbox'
      
      # Only needed if you are using Google Drive for cloud storage
      pod 'SyncServer/Google'
      
      target 'YourAppTests' do
          inherit! :search_paths
      end
  end
  2. Additions to your AppDelegate
    1. Add the following import
      2. import SyncServer
    2. Setup your sign-in's in the didFinishLaunchingWithOptions method
      3. If you are allowing use of Google Drive, you will need: 
        let googleSignIn = GoogleSyncServerSignIn(serverClientId: serverClientId, appClientId: appClientId)
  SignInManager.session.addSignIn(googleSignIn, launchOptions: options)
      If you are also using Dropbox, you'll need: 
        let dropboxSignIn = DropboxSyncServerSignIn(appKey: dropboxAppKey)
  SignInManager.session.addSignIn(dropboxSignIn, launchOptions: options)
      If you are also using Facebook sign-in, you'll need: 
        let facebookSignIn = FacebookSyncServerSignIn()
  SignInManager.session.addSignIn(facebookSignIn, launchOptions: options)
      Where: serverClientId and appClientId come from setting up your app with Google. See below.
    3. Also in the didFinishLaunchingWithOptions method, you need to call
      4. SyncServer.session.appLaunchSetup(withServerURL: serverURL, cloudFolderName:cloudFolderName)
      Where:
      • serverURL is the URL of your SyncServer server.
      • cloudFolderName is the folder name (not a path, just a folder) within Google Drive in which your files will be stored (if you are using Google Drive). Dropbox, if you are using that, doesn't need this.
        • E.g., I'm using "SharedImages.Folder" for the SharedImages app.
    4. Add the following into the app:open:options: method
      5.   func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool {
      return SignInManager.session.application(app, open: url, options: options) ||
      SharingInvitation.session.application(app, open: url, options: options)
  }
    5. To enable background downloading/uploading add this into your AppDelegate
      6.   func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
    SyncServer.session.application(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
  }
  3. Setup the SharingInvitationDelegate to deal with sharing invitations received.
    4. For an example of how this is done, search for SharingInvitationDelegate in the SharedImages app.
  4. Setup the UI for sign-in's in your app.
      5. You will need to do this for at least Google. For an example of how this is done, look in the SharedImages app.
    1. Call the `setupSignInButton` and setup delegates for your selected sign-ins. For example:
      2.   // For the GoogleSyncServerSignIn, the delegate in this params argument must a view controller abiding by GoogleSignInUIProtocol
  let googleSignInButton = googleSignIn.setupSignInButton(params: ["delegate": self])
  googleSignIn.delegate = self // abiding by GenericSignInDelegate

  // If you are using Facebook sign-in:
  let facebookSignInButton = facebookSignIn.setupSignInButton(params:nil)
  // Just to balance out the lengths of the sign-in buttons-- the Google button is shorter.
  facebookSignInButton.frameWidth = googleSignInButton.frameWidth
  facebookSignIn.delegate = self

  // If you are using Dropbox sign-in:
  let dropboxSignInButton = dropboxSignIn.setupSignInButton(params: ["viewController": self])
  dropboxSignInButton.frameSize = CGSize(width: googleSignInButton.frameWidth, height: googleSignInButton.frameHeight * 0.75)
  dropboxSignIn.delegate = self
    2. For convenience if you are using multiple sign-in's (say, Google, Facebook, and Dropbox, the SyncServer SignIn widget can enable you to display the sign-in's automatically:
      3.   let signIn:SignIn = SignIn.createFromXib()!
  // and add this SignIn view into an appropriate view controller in your UI.
      The SignIn view uses the SyncServer SignInManager to determine which sign-in buttons to display. The SharedImages app uses the SignIn view, so see that for additional code details.

Building and Running SharedImages

Downloading SharedImages and building it in Xcode will not give a clean build. You will get something like this:

These errors occur because two files, Server.plist and GoogleService-Info.plist are present only as symbolic links-- you need to populate them for your own server and for your own Google Sign In setup.

Your own server

SharedImages relies on having a SyncServerII instance running. The Server.plist file references this server, and other setup-- for Google and Dropbox sign-in's. Thus, you will need to stand up a SyncServerII server. Once you have done that, the format of the Server.plist file is:

The references to Staging in the above .plist can be ignored if you are not using a staging server.

Using Google Drive and Google Sign-In (optional)

  1. Create Google App/Developer Credentials
    2. To enable access to user Google Drive accounts, you must create Google Developer credentials for your iOS app and SyncServer server. These credentials need to be installed in your app making use of the SyncServer client Framework. Go to https://developers.google.com/identity/sign-in/ios/start.

    You need to generate a configuration file-- this will typically be named: `GoogleService-Info.plist`, and add that file to your Xcode project. Amongst other information, this .plist file contains the Google `CLIENT_ID` for your iOS app.

  2. You also need to make sure you enable the Google Drive API for your Google project.
    3. You can do this by going to https://console.developers.google.com, looking for `ENABLE API`, and then `Drive API`.
  3. Obtain your `OAuth 2.0 client IDs`.
    4. Within https://console.developers.google.com, you also need to obtain the `OAuth 2.0 client IDs` for your `Web client` (see under "Credentials). In the SyncServer configuration file this is called the GoogleServerClientId. You will need both the `CLIENT_ID` (for your iOS app) and the GoogleServerClientId (for the SyncServer server) in order for users to sign in to Google Drive from your iOS app.
  4. Add URL scheme for Google Sign in to your app.
    5. You do this in XCode under the "Info" tab. Look for "URL Types" and paste the CLIENT_ID into the "URL Schemes" field of a new URL Type (press the "+" button).

Using Facebook Sign-In (optional)

  1. Create a new Facebook app
    2. To do this, go to https://developers.facebook.com. You will need to sign in with a Facebook account. If you don't already have one, you'll have to make one. After signing in, in the upper right, under "My Apps", click on "Add a New App". Follow the instructions. They're pretty good.
  2. For the Server: ClientId and Secret.
    3. You will need the FacebookClientId (App ID) and FacebookClientSecret (App Secret) for the SyncServer server Server.json file from your new app on https://developers.facebook.com.

Using Dropbox Sign-In and Cloud Storage (optional)

  1. Register your application with Dropbox
    2. To do this, use this Dropbox link. The more general Dropbox docs are here.
  2. Configure your project
    3. You will need to then follow the steps under "Configure your project" in the more general Dropbox docs. You do *not* need to explicitly include the Dropbox Cocoapod in your Podfile for your Xcode project if you have followed the above Podfile instructions.

Using AWS SNS for Push Notifications (optional)

While the particular hosting service you use for running SyncServerII is your choice (I've used Docker containers, running under AWS and Elastic Beanstalk), the current implemention of push notifications in SyncServerII is specifically dependent on AWS SNS.
  1. Create development and production certificates with Apple Currently only Apple Push Notifications are supported (APNS). For example, see.
  2. Add code in your iOS app to incorporate push notifications
    3. For example, see.
  3. Register your SNS application with AWS SNS
    4. You also need to create a platform application arn for both development and production. For example, see
  4. You also need an access key and secret key for AWS
  5. Add the following keys into the .json configuration files for your development and production servers:
    6. "awssns.accessKeyId" "awssns.secretKey" "awssns.region" "awssns.platformApplicationArn" And, of course, give them their appropriate values.
  6. Modify your iOS app with messages.
    7. Push notifications can be included when you use the SyncServer.session.sync call. The intent is that you are able to send out a push notification when file(s) are uploaded or upload deleted. These messages are included with the iOS app itself because the contents of push notification messages tends to be application specific.

Further Reading

  1. Talk given at Denver Swift Developers Group (10/25/18)
    2. Keynote slides PDF
  2. Talk given at roster technologies brown bag lunch series (2/6/18)
    3. PowerPoint slides YouTube video.
  3. Talk given to the Boulder iOS Meetup (11/21/17)
    4. PowerPoint slides YouTube video (teaser-- first 10 minutes! :) ).
  4. Talk given to the Denver Open Source Users Group (8/1/17)
    5. PowerPoint slides
  5. Blog articles on SyncServer development
    1. Load Testing Adventures With SyncServerII: mySQL Locking and Deadlocks (6/15/19)
    2. The Google OAuth Review Process (3/20/19)
    3. Privacy + Self-Ownership in Social Media: SyncServerII (8/18/18)
    4. The Client-Side SyncServer Sign-In System: Making Sign-in Sticky (10/23/17)
    5. Making Downloads More Flexible in the SyncServer (9/15/17)
    6. To Lock or Not to Lock in the SMSyncServer (9/3/16)
    7. Conflict Management in the SMSyncServer (5/11/16)
    8. Re-Architecting the SMSyncServer File System (5/9/16)
    9. The Many Senses of Recovery in SMSyncServer (4/26/16)
    10. Design Issue: Changing Cloud Storage Accounts With The SMSyncServer (4/2/16)
    11. Blitz to get SMSyncServer Ready for Open-Source (1/21/16)
    12. The SyncServer: Permanent Access to Your App Data (12/29/15)