{"action":"created","issue":{"url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/24","repository_url":"https://api.github.com/repos/dmurph/indexed-db-observers","labels_url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/24/comments","events_url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/24/events","html_url":"https://github.com/dmurph/indexed-db-observers/issues/24","id":154594871,"number":24,"title":"Matching Observer creation, like IntersectionObserver","user":{"login":"dmurph","id":294026,"avatar_url":"https://avatars.githubusercontent.com/u/294026?v=3","gravatar_id":"","url":"https://api.github.com/users/dmurph","html_url":"https://github.com/dmurph","followers_url":"https://api.github.com/users/dmurph/followers","following_url":"https://api.github.com/users/dmurph/following{/other_user}","gists_url":"https://api.github.com/users/dmurph/gists{/gist_id}","starred_url":"https://api.github.com/users/dmurph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmurph/subscriptions","organizations_url":"https://api.github.com/users/dmurph/orgs","repos_url":"https://api.github.com/users/dmurph/repos","events_url":"https://api.github.com/users/dmurph/events{/privacy}","received_events_url":"https://api.github.com/users/dmurph/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-05-12T22:22:15Z","updated_at":"2016-05-19T02:36:29Z","closed_at":null,"body":"Observer creation should more closely resemble current observers in the web. We want to match something like [IntersectionObserver](https://github.com/WICG/IntersectionObserver).\r\n\r\nThis proposal only go over changes to the creation, we assume that the changes callback and behavior there is the same as in the [Explainer](/dmurph/indexed-db-observers/blob/gh-pages/EXPLAINER.md).\r\n\r\n# Objectives\r\n\r\nExample use cases:\r\n\r\n1. A UI element that's syncing it's state to the database.\r\n2. JS working synchronizing DB changes to a webserver.\r\n3. Using an observer to maintain an in-memory cache.\r\n4. Observing when object store changes to do custom refresh logic.\r\n\r\nWe need to make sure we can always do the following:\r\n\r\n1. Know exactly when our observing starts in the transaction chain. This means we should be able to read in data and start an observer, and not lose any changes in between the two that the observer doesn't see. This is important in sync services as outlined below.\r\n2. Listen to multiple object stores at the same time, so that a transaction with both object stores will generate just one change event.\r\n\r\nWants:\r\n\r\n1. Minimize possible web developer error cases and bugs.\r\n2. Minimize code amount & complexity.\r\n3. Defaults that are more performant. Examples: require the developer to always specify object stores (tables) that they are observering. Only send the key of the change by default to reduce memory used.\r\n\r\n# Proposal\r\nI'll show examples first.\r\n\r\n## UI Element\r\nLet's say we're a polymer or angular webapp, and we have databinding working. **We rely on the guarantee that the observer will start immediately after the transaction it was created with**.\r\n \r\n```javascript\r\n// uiComponent contains logic for updating it's own data.\r\nvar uiComponent = this;\r\n// This function updates our UI component when the database is changed.\r\nvar updateUICallback = function(changes) {\r\n  changes.records.get('users').forEach(function(record) {\r\n    switch(record.type) {\r\n      case 'clear':\r\n        uiComponent.clear();\r\n        break;\r\n      case 'add':\r\n        uiComponent.addUser(change.key, change.value);\r\n        break;\r\n      case 'put':\r\n        uiComponent.updateUser(change.key, change.value);\r\n        break;\r\n      case 'delete':\r\n        uiComponent.removeUser(change.key);\r\n        break;\r\n    }\r\n  });\r\n}\r\n// Observer creation. We want to include the values,\r\n// as we'll always use them to populate the UI.\r\nvar observer = new IndexedDBObserver(updateUICallback, { values: true });\r\n// Create or transaction for both reading the table and attaching the observer.\r\nvar txn = db.transaction('users', 'readonly');\r\n// We'll start seeing changes after 'txn' is complete.\r\nobserver.observe(db, txn);\r\n// Now we read in our initial state for the component.\r\nvar usersTable = txn.objectStore('users');\r\nvar request = usersTable.getAll();\r\nrequest.onsuccess = function() {\r\n  request.result.forEach(function(user) {\r\n    uiComponent.addUser(user.id, user);\r\n  });\r\n}\r\ntxn.oncomplete = function() {\r\n  console.log('component initialized and observer started');\r\n}\r\n```\r\n## Server sync worker\r\nWe want to synchronize changes to a webserver. Let's say we're storing descrete transformations, so it's only add operations. **We rely on the guarantee that the observer will start immediately after the transaction it was created with**.\r\n\r\n```javascript\r\n// We are bad developers and don't batch our network calls.\r\n// We just send them whenever there's a change.\r\nvar sendChangesToWebserver = function(changes) {\r\n  var changesToSend = [];\r\n  changes.records.get('transformations').forEach(function(record) {\r\n    if (record.type != 'add') return;\r\n    changesToSend.push(record.value);\r\n  });\r\n  sendChangesToNetwork(changesToSend);\r\n}\r\nvar observer = new IndexedDBObserver(sendChangesToWebserver,\r\n    { onlyExternal: true, values: true });\r\n// Make our network call to populate the transformations.\r\ngetTranformsFromNetwork(function(transforms) {\r\n  var txn = db.transact('transformations', 'readwrite');\r\n  observer.observe(db, txn);\r\n  var os = txn.objectStore('transformations');\r\n  transforms.forEach(function(transform) {\r\n    os.put(transform);\r\n  });\r\n  txn.oncomplete = function() {\r\n    console.log('Database is initialized and we are syncing changes');\r\n  }\r\n});\r\n// Listen for changes from network.\r\n// Note: our observer won't see these changes, as it has onlyExternal: true.\r\nthis.onNetworkChanges = function(changesFromNetwork) {\r\n  var txn = db.transact('transformations', 'readwrite');\r\n  var os = txn.objectStore('transformations');\r\n  transforms.forEach(function(transform) {\r\n    os.put(transform);\r\n  });\r\n}\r\n```\r\n\r\n## Maintaining an in-memory data cache\r\nIDB can be slow due to disk, so it's a good idea to have an in-memory cache.  Note that we don't include values here, as we want to optimize our memory usage by reading in the cache in a larger batch, and at an opportune time. **We rely on the guarantee that the observer will start immediately after the transaction it was created with**.\r\n\r\n```javascript\r\n// let's assume our cache batches together reads once we get enough changes\r\n// or a timeout occurs. So we want to give it changes, and let it optionally read\r\n// in a bunch of data.\r\nvar usersCache = this.cache;\r\nvar updateUsersCache = function(changes) {\r\n  usersCache.addChanges(changes.records.get('users'));\r\n  usersCache.maybeResolveChanges(changes.transaction);\r\n}\r\nvar observer = new IndexedDBObserver(updateUsersCache, { transaction: true });\r\nvar txn = db.transaction('users', 'readonly');\r\n// Attach our observer.\r\nvar optionsMap = new Map();\r\noptionsMap.put('users', { ranges: IDBKeyRange.bound(0, 1000)});\r\nobserver.observe(db, txn, optionsMap);\r\n// Read initial contents of the cache.\r\nvar os = txn.objectStore('users');\r\nvar readRequest = os.getAll(IDBKeyRange.bound(0, 1000), 50);\r\nreadRequest.onsuccess = function() {\r\n  var users = readRequest.result;\r\n  usersCache.addUsers(users);\r\n}\r\ntxn.oncomplete = function() {\r\n  console.log(\"Cache initialized and listening\");\r\n}\r\n\r\n```\r\n\r\n## Custom refresh logic\r\nIf we just want to know when an object store has changed.\r\n```javascript\r\n// We disable records, so we just get the callback without any data.\r\nvar observer = new IndexedDBObserver(refreshUserDataCallback, { records: false });\r\nobserver.observe(db, db.transact('users', 'readonly'));\r\n```\r\n\r\n## Summary\r\nOther than the two step construction, we've split up the options. All options except ranges have moved to a flat options object passed into the observer creation. We could even just make the Map values for `observer.observe(db, txn, map)` be the ranges array itself, instead of an object that contains the 'ranges' attribute.\r\n\r\nI'm also having the **database and transaction required arguments**. In almost every use case, the person needs the guarentee that the observation starts after the read or write to the database. Otherwise changes can be leaked, where the observer didn't see them yet or the observer sees a change that we thought it wouldn't.\r\n\r\n# Possible changes / thoughts\r\n\r\n1. It's a little cumbersome to have a transaction. It would be nice to remove, but we don't want people to be able to just listen to the whole database by default. We could allow the second argument to be an array of object store names, but we don't want overloaded functions.\r\n2. We could remove the `db` argument and just use the connection from the transaction. This changes the current observer pattern in the web platform though, which ties the lifetime of the observer to the first argument. Since we're tieing to the database connection, then this makes sense to leave in there.\r\n3. The map of objectstore options is a little weird... maybe somehow we could decompose these options into separate observe calls? This seems way more confusing though. I think moving all options but the ranges option to the observer construction simplifies this, and then we can just have it be a map<string, sequence<IDBKeyRange>> instead of the object as the value."},"comment":{"url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/comments/220213094","html_url":"https://github.com/dmurph/indexed-db-observers/issues/24#issuecomment-220213094","issue_url":"https://api.github.com/repos/dmurph/indexed-db-observers/issues/24","id":220213094,"user":{"login":"bevis-tseng","id":7510903,"avatar_url":"https://avatars.githubusercontent.com/u/7510903?v=3","gravatar_id":"","url":"https://api.github.com/users/bevis-tseng","html_url":"https://github.com/bevis-tseng","followers_url":"https://api.github.com/users/bevis-tseng/followers","following_url":"https://api.github.com/users/bevis-tseng/following{/other_user}","gists_url":"https://api.github.com/users/bevis-tseng/gists{/gist_id}","starred_url":"https://api.github.com/users/bevis-tseng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bevis-tseng/subscriptions","organizations_url":"https://api.github.com/users/bevis-tseng/orgs","repos_url":"https://api.github.com/users/bevis-tseng/repos","events_url":"https://api.github.com/users/bevis-tseng/events{/privacy}","received_events_url":"https://api.github.com/users/bevis-tseng/received_events","type":"User","site_admin":false},"created_at":"2016-05-19T02:36:29Z","updated_at":"2016-05-19T02:36:29Z","body":"My bad. You have mentioned that clearly in the comment:\r\n// Note: our observer won't see these changes, as it has onlyExternal: true."}}