{"action":"created","issue":{"url":"https://api.github.com/repos/datalust/seq-tickets/issues/777","repository_url":"https://api.github.com/repos/datalust/seq-tickets","labels_url":"https://api.github.com/repos/datalust/seq-tickets/issues/777/labels{/name}","comments_url":"https://api.github.com/repos/datalust/seq-tickets/issues/777/comments","events_url":"https://api.github.com/repos/datalust/seq-tickets/issues/777/events","html_url":"https://github.com/datalust/seq-tickets/issues/777","id":394000463,"node_id":"MDU6SXNzdWUzOTQwMDA0NjM=","number":777,"title":"Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".","user":{"login":"SeriousM","id":462013,"node_id":"MDQ6VXNlcjQ2MjAxMw==","avatar_url":"https://avatars3.githubusercontent.com/u/462013?v=4","gravatar_id":"","url":"https://api.github.com/users/SeriousM","html_url":"https://github.com/SeriousM","followers_url":"https://api.github.com/users/SeriousM/followers","following_url":"https://api.github.com/users/SeriousM/following{/other_user}","gists_url":"https://api.github.com/users/SeriousM/gists{/gist_id}","starred_url":"https://api.github.com/users/SeriousM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeriousM/subscriptions","organizations_url":"https://api.github.com/users/SeriousM/orgs","repos_url":"https://api.github.com/users/SeriousM/repos","events_url":"https://api.github.com/users/SeriousM/events{/privacy}","received_events_url":"https://api.github.com/users/SeriousM/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-25T10:12:14Z","updated_at":"2018-12-25T18:36:40Z","closed_at":null,"author_association":"NONE","body":"We run seq in our kubernetes cluster and experienced sometimes evicted pods. After the last incident we had over 700 evicted pods and after cleaning up this mess seq can't start with the error in the log below.\r\nIs there any way to rescue the logs?\r\n\r\n```log\r\n ────────────────────────────────────────\r\n  Seq ♦ Machine data, for humans.\r\n─────────── © 2018 Datalust Pty Ltd ────\r\nRunning as server; press Ctrl+C to exit.\r\n[09:57:21 INF] Seq \"5.0.2362\" running on OS \"Linux 4.15.0-1030-azure #31~16.04.1-Ubuntu SMP Tue Oct 30 19:40:01 UTC 2018\"\r\n[09:57:22 INF] Seq listening on [\"http://localhost/\", \"http://localhost:5341/\"]\r\n[09:57:22 INF] Seq using canonical URI \"xxxxxxxxxxxxxxxxx\"\r\n[09:57:23 INF] Opening document store \"/data/Documents/documents.lmdb\"\r\n[09:57:24 INF] Closing document store\r\n[09:57:24 ERR] Uncaught exception from worker \"Start background processes\" (\"ba93e9a8-1602-4602-aacb-ff33ae2cf4f5\")\r\nAutofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = TelemetryController (ReflectionActivator), Services = [Seq.Server.Features.Telemetry.TelemetryController], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = SystemSettings (ReflectionActivator), Services = [Seq.Server.Features.Settings.SystemSettings], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = IDocumentStore (DelegateActivator), Services = [Quince.IDocumentStore], Lifetime = Autofac.Core.Lifetime.CurrentScopeLifetime, Sharing = None, Ownership = ExternallyOwned ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".\r\n   caused by: No such file or directory (os error 2) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = SystemSettings (ReflectionActivator), Services = [Seq.Server.Features.Settings.SystemSettings], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = IDocumentStore (DelegateActivator), Services = [Quince.IDocumentStore], Lifetime = Autofac.Core.Lifetime.CurrentScopeLifetime, Sharing = None, Ownership = ExternallyOwned ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".\r\n   caused by: No such file or directory (os error 2) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = IDocumentStore (DelegateActivator), Services = [Quince.IDocumentStore], Lifetime = Autofac.Core.Lifetime.CurrentScopeLifetime, Sharing = None, Ownership = ExternallyOwned ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".\r\n   caused by: No such file or directory (os error 2) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = StorageSubsystem (DelegateActivator), Services = [Seq.Server.Storage.StorageSubsystem], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".\r\n   caused by: No such file or directory (os error 2) (See inner exception for details.) ---> System.Exception: Flare native storage failed (InternalError), failed to open the key-value store at: \"/data/Extents/2018-11-21_2018-11-28/extent.0.metadata\".\r\n   caused by: No such file or directory (os error 2)\r\n   at Flare.Storage.Native.FlareResult.EnsureSuccess()\r\n   at Flare.Storage.Store.Open(String path)\r\n   at Flare.Storage.Native.FlareStorageExtent.OpenStore()\r\n   at Flare.Storage.Native.FlareStorageExtent..ctor(IStorageEngine storageEngine, ExtentRange range, String dataFile)\r\n   at Flare.Storage.Native.FlareStorageEngine.CreateExtent(ExtentRange range, String dataFile, String instanceName)\r\n   at Flare.Storage.Extents.ExtentMap..ctor(IStorageEngine[] storageEngines, String extentsPath)\r\n   at Flare.Events.EventStore..ctor(IStorageEngine[] storageEngines, String extentsPath, WorkerPool workerPool, MemoryThresholds memoryThresholds, BroadcastChannel broadcast, IClock clock)\r\n   at Seq.Server.SeqServerModule.InitStorageSubsystem(IComponentContext c)\r\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\r\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\r\n   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n   --- End of inner exception stack trace ---\r\n   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)\r\n   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)\r\n   at Autofac.Core.Resolving.InstanceLookup.Execute() \r\n```"},"comment":{"url":"https://api.github.com/repos/datalust/seq-tickets/issues/comments/449867355","html_url":"https://github.com/datalust/seq-tickets/issues/777#issuecomment-449867355","issue_url":"https://api.github.com/repos/datalust/seq-tickets/issues/777","id":449867355,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0OTg2NzM1NQ==","user":{"login":"SeriousM","id":462013,"node_id":"MDQ6VXNlcjQ2MjAxMw==","avatar_url":"https://avatars3.githubusercontent.com/u/462013?v=4","gravatar_id":"","url":"https://api.github.com/users/SeriousM","html_url":"https://github.com/SeriousM","followers_url":"https://api.github.com/users/SeriousM/followers","following_url":"https://api.github.com/users/SeriousM/following{/other_user}","gists_url":"https://api.github.com/users/SeriousM/gists{/gist_id}","starred_url":"https://api.github.com/users/SeriousM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeriousM/subscriptions","organizations_url":"https://api.github.com/users/SeriousM/orgs","repos_url":"https://api.github.com/users/SeriousM/repos","events_url":"https://api.github.com/users/SeriousM/events{/privacy}","received_events_url":"https://api.github.com/users/SeriousM/received_events","type":"User","site_admin":false},"created_at":"2018-12-25T18:36:40Z","updated_at":"2018-12-25T18:36:40Z","author_association":"NONE","body":"I just updated to the latest version '5.0.2362' but still get the error. I hope we can fix the problem without reset?"}}
{"actor":{"display_login":"SeriousM"}}