{"action":"created","issue":{"url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/1004","repository_url":"https://api.github.com/repos/TeamNewPipe/NewPipe","labels_url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/1004/labels{/name}","comments_url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/1004/comments","events_url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/1004/events","html_url":"https://github.com/TeamNewPipe/NewPipe/pull/1004","id":289449422,"number":1004,"title":"[WIP] Local Playlist and Database Normalization","user":{"login":"karyogamy","id":5570482,"avatar_url":"https://avatars2.githubusercontent.com/u/5570482?v=4","gravatar_id":"","url":"https://api.github.com/users/karyogamy","html_url":"https://github.com/karyogamy","followers_url":"https://api.github.com/users/karyogamy/followers","following_url":"https://api.github.com/users/karyogamy/following{/other_user}","gists_url":"https://api.github.com/users/karyogamy/gists{/gist_id}","starred_url":"https://api.github.com/users/karyogamy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karyogamy/subscriptions","organizations_url":"https://api.github.com/users/karyogamy/orgs","repos_url":"https://api.github.com/users/karyogamy/repos","events_url":"https://api.github.com/users/karyogamy/events{/privacy}","received_events_url":"https://api.github.com/users/karyogamy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-01-17T23:19:54Z","updated_at":"2018-02-06T06:53:01Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/TeamNewPipe/NewPipe/pulls/1004","html_url":"https://github.com/TeamNewPipe/NewPipe/pull/1004","diff_url":"https://github.com/TeamNewPipe/NewPipe/pull/1004.diff","patch_url":"https://github.com/TeamNewPipe/NewPipe/pull/1004.patch"},"body":"IMPORTANT: Please test this on an emulator since this PR includes a database migration script (that normalizes the watch history table) which is currently irreversible.\r\n\r\n - This PR aims at implementing #870 and is a work in progress.\r\n - Along with local playlists, this PR also tries to normalize the database schema so stream metadata can be reused. \r\n - This allows us to create smart playlists, such as latest unique watch history (#998 is also implemented as part of this) and most watched streams. In the future, we can use this data to experiment with video recommendation using related videos and tags.\r\n\r\nHere are some screenshots, which is probably better at explaining the new features. These UI are not final, please feel free provide ideas:\r\n\r\n<img src=\"https://user-images.githubusercontent.com/5570482/35072056-e0028200-fb97-11e7-9f53-ed3c41a3987c.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072057-e017e8a2-fb97-11e7-894f-66921e394809.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072050-df8ea3bc-fb97-11e7-870b-90cc1d590129.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072051-dfad7aee-fb97-11e7-8f61-069c023e6ed5.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072052-dfc20018-fb97-11e7-9257-6a7d2af22a6c.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072053-dfd8e9d6-fb97-11e7-87e2-65ebfc368c67.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072055-dfec97c4-fb97-11e7-96c5-74fee01720fc.png\" width=\"200\"><img src=\"https://user-images.githubusercontent.com/5570482/35072362-1d961aa4-fb99-11e7-98b6-e6cf304be6f5.png\" width=\"200\">\r\n\r\nHere's an overview of the new database schemas:\r\n\r\n- Streams: contains extracted stream metadata\r\n- Stream History: [FK on Streams] contains timestamps of every replay\r\n- Stream State: [FK on Streams] contains user states of a stream, such as saved progress of streams\r\n- Playlist: contains name of user created playlist\r\n- Playlist Stream Join: [FK on Streams and Playlist] contains what streams are in each playlist\r\n\r\nTodos (not in order):\r\n- [x] Bulk add playlist from queues (saving ~2000 videos takes about just under a second on emulator)\r\n- [x] Integrate normalized database to display on history activity\r\n- [x] Simpify history activity to use dialog instead of gesture for deleting entries\r\n- [x] Allow modifiable playlist (i.e. reordering and deleting streams, rename playlist and change thumbnail)\r\n- [x] Add query for wiping stream entries which have no joins with other tables\r\n- [x] Refactor/Improve info InfoItem holders to show additional items\r\n- [x] UI improvements\r\n- [x] Add timestamp recording when streams changes midway (#537), no resuming in this PR \r\n- [ ] Code cleanup\r\n\r\nFuture PR:\r\n - Searchable playlist dialog.\r\n - Add UI for resuming past videos\r\n - Add query for list by services, currently throws unsupported exceptions\r\n - Video recommendations using history statistics and related video\r\n\r\nPlease feel free to give recommendations and report bugs, though please don't test this on a real device yet."},"comment":{"url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/comments/363328123","html_url":"https://github.com/TeamNewPipe/NewPipe/pull/1004#issuecomment-363328123","issue_url":"https://api.github.com/repos/TeamNewPipe/NewPipe/issues/1004","id":363328123,"user":{"login":"oceanwaves90","id":10182619,"avatar_url":"https://avatars2.githubusercontent.com/u/10182619?v=4","gravatar_id":"","url":"https://api.github.com/users/oceanwaves90","html_url":"https://github.com/oceanwaves90","followers_url":"https://api.github.com/users/oceanwaves90/followers","following_url":"https://api.github.com/users/oceanwaves90/following{/other_user}","gists_url":"https://api.github.com/users/oceanwaves90/gists{/gist_id}","starred_url":"https://api.github.com/users/oceanwaves90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oceanwaves90/subscriptions","organizations_url":"https://api.github.com/users/oceanwaves90/orgs","repos_url":"https://api.github.com/users/oceanwaves90/repos","events_url":"https://api.github.com/users/oceanwaves90/events{/privacy}","received_events_url":"https://api.github.com/users/oceanwaves90/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T06:53:01Z","updated_at":"2018-02-06T06:53:01Z","author_association":"NONE","body":"Hi. Thank you for your work!\r\n\r\nBy the way It throws NPE on the lastest commit(actually on some recent commits), only when I try to open the video [https://www.youtube.com/watch?v=-Vpwatutnko](url). Other videos seem working right except for this video. I hope you fix this issue! thank you!\r\n\r\n`4380/org.schabi.newpipe.debug E/class org.schabi.newpipe.report.ErrorActivity: java.lang.NullPointerException: Attempt to invoke virtual method 'int com.grack.nanojson.JsonArray.size()' on a null object reference\r\n                                                                                                             at org.schabi.newpipe.extractor.services.youtube.YoutubeStreamExtractor.getAvailableSubtitlesInfo(YoutubeStreamExtractor.java:743)\r\n                                                                                                             at org.schabi.newpipe.extractor.services.youtube.YoutubeStreamExtractor.onFetchPage(YoutubeStreamExtractor.java:558)\r\n                                                                                                             at org.schabi.newpipe.extractor.Extractor.fetchPage(Extractor.java:61)\r\n                                                                                                             at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:248)\r\n                                                                                                             at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:240)\r\n                                                                                                             at org.schabi.newpipe.util.ExtractorHelper.lambda$getStreamInfo$3$ExtractorHelper(ExtractorHelper.java:109)\r\n                                                                                                             at org.schabi.newpipe.util.ExtractorHelper$$Lambda$3.call(Unknown Source)\r\n                                                                                                             at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:45)\r\n                                                                                                             at io.reactivex.Single.subscribe(Single.java:2801)\r\n                                                                                                             at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)\r\n                                                                                                             at io.reactivex.Single.subscribe(Single.java:2801)\r\n                                                                                                             at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)\r\n                                                                                                             at io.reactivex.Maybe.subscribe(Maybe.java:3749)\r\n                                                                                                             at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)\r\n                                                                                                             at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)\r\n                                                                                                             at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)\r\n                                                                                                             at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)\r\n                                                                                                             at io.reactivex.Flowable.subscribe(Flowable.java:13068)\r\n                                                                                                             at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)\r\n                                                                                                             at io.reactivex.Maybe.subscribe(Maybe.java:3749)\r\n                                                                                                             at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)\r\n                                                                                                             at io.reactivex.Single.subscribe(Single.java:2801)\r\n                                                                                                             at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)\r\n                                                                                                             at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:452)\r\n                                                                                                             at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)\r\n                                                                                                             at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)\r\n                                                                                                             at java.util.concurrent.FutureTask.run(FutureTask.java:237)\r\n                                                                                                             at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:154)\r\n                                                                                                             at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269)\r\n                                                                                                             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)\r\n                                                                                                             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)\r\n                                                                                                             at java.lang.Thread.run(Thread.java:818)`"}}
{"actor":{"display_login":"oceanwaves90"}}