title: OrientDB corruption state in Nexus Repository version 3.2.0-01
body: <p>We are using Nexus Repository version 3.2.0-01 and started getting the following error in nexus.log. OrientDB has got corrupted. Nexus is no longer starting. </p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>2017-03-21 13:00:36,329+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.13
  2017-03-21 13:00:36,348+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.13 is starting up...
  2017-03-21 13:00:36,355+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /c9/setup/apps/nexus/sonatype-work/nexus3/db
  2017-03-21 13:00:36,647+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies - Not enough physical memory available for DISKCACHE: 1,873MB (heap=1,161MB direct=2,048MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB
  2017-03-21 13:00:36,648+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies - OrientDB config DISKCACHE=256MB (heap=1,161MB direct=2,048MB os=1,873MB)
  2017-03-21 13:00:36,735+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Found ORIENTDB_ROOT_PASSWORD variable, using this value as root's password
  2017-03-21 13:00:36,899+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.handler.OJMXPlugin - JMX plugin installed and active: 
  
  profilerManaged=true
  2017-03-21 13:00:36,901+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue 
  
  http://localhost:2480/studio/index.html}
  2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.13.
  2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Activated
  2017-03-21 13:00:36,904+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
  2017-03-21 13:00:37,632+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
  com.orientechnologies.orient.core.exception.OSecurityAccessException: User or password not valid for database: 'config'^M
          DB name="config"
          at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
          at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
          at com.orientechnologies.orient.core.metadata.security.OSecurityExternal.authenticate(OSecurityExternal.java:62) [na:na]
          at com.orientechnologies.orient.core.metadata.security.OSecurityProxy.authenticate(OSecurityProxy.java:107) [na:na]
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:274) [na:na]
          at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:142) [org.sonatype.nexus.orient:3.2.0.01]
          at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56) [na:na]
          at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
          at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44) [org.sonatype.goodies.lifecycle:2.2.3]
          at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:258) [org.sonatype.nexus.orient:3.2.0.01]
          at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:232) [org.sonatype.nexus.orient:3.2.0.01]
          at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:61) [org.sonatype.nexus.base:3.2.0.01]
          at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:1) [org.sonatype.nexus.base:3.2.0.01]
          at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
          at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:133) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
          at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [com.google.inject:4.1.0]
          at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [com.google.inject:4.1.0]
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
           at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81) [org.eclipse.sisu.inject:0.3.2]
          at org.eclipse.sisu.wire.BeanProviders.firstOf(BeanProviders.java:179) [org.eclipse.sisu.inject:0.3.2]
          at org.eclipse.sisu.wire.PlaceholderBeanProvider.lookup(PlaceholderBeanProvider.java:122) [org.eclipse.sisu.inject:0.3.2]
          at org.eclipse.sisu.wire.PlaceholderBeanProvider.get(PlaceholderBeanProvider.java:90) [org.eclipse.sisu.inject:0.3.2]
          at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
          at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
          at org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176) [org.eclipse.sisu.inject:0.3.2]
          at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
          at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
          at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
          at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
          at org.sonatype.nexus.upgrade.internal.ModelVersionStore.doStart(ModelVersionStore.java:82) [org.sonatype.nexus.upgrade:3.2.0.01]
          at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.2.0.01]
          at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.CGLIB$start$5(&lt;generated&gt;) [3.2.0-01:na]
          at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f$$FastClassByGuice$$94ba58c4.invoke(&lt;generated&gt;) [3.2.0-01:na]
          at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [com.google.inject:4.1.0]
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [com.google.inject:4.1.0]
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.2.0.01]
          at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.2.0.01]
          at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.2.0.01]
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [com.google.inject:4.1.0]
          at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [com.google.inject:4.1.0]
          at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.start(&lt;generated&gt;) [3.2.0-01:na]
          at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:80) [org.sonatype.nexus.upgrade:3.2.0.01]
          at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
          at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:154) [org.sonatype.nexus.extender:3.2.0.01]
          at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:94) [org.sonatype.nexus.extender:3.2.0.01]
          at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:185) [org.sonatype.nexus.extender:3.2.0.01]
          at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.4.0.jar:na]
          at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.4.0.jar:na]
          at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]</code></pre>
  </div>
  </div>
  </p>
  
  <p><strong>Root cause</strong>
  Root cause of this was found to be an unclean shutdown of Nexus because of disk space getting full, this resulted in DB corruption. </p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>2017-03-17 07:09:49,480+0000 WARN  [elasticsearch[A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4][local_transport][T#1]] *SYSTEM org.elasticsearch.cluster.action.shard - [A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4] [cab22f445c1892acef64d17ac2da6d0086023e21][0] received shard failed for [cab22f445c1892acef64d17ac2da6d0086023e21][0], node[HBjARpI7RyqqqruVqEJ8Ig], [P], v[11], s[INITIALIZING], a[id=FmxOHS7nQcGNz_93VlTpKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-03-17T07:09:49.101Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [component#36e3dec8de528c9b41c5c6ac3b18e3f7]]; nested: NotSerializableExceptionWrapper[No space left on device]; ]], indexUUID [fLN4hdE8QPKH7Pj5RpLtmw], message [engine failure, reason [index]], failure [NotSerializableExceptionWrapper[No space left on device]]
  org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: No space left on device
  at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [na:1.8.0_121]
  at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) [na:1.8.0_121]
  at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [na:1.8.0_121]
  at sun.nio.ch.IOUtil.write(IOUtil.java:65) [na:1.8.0_121]
  at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) [na:1.8.0_121]
  at org.elasticsearch.common.io.Channels.writeToChannel(Channels.java:211) [na:na]
  at org.elasticsearch.common.bytes.PagedBytesReference.writeTo(PagedBytesReference.java:132) [na:na]
  at org.elasticsearch.index.translog.BufferingTranslogWriter.add(BufferingTranslogWriter.java:57) [na:na]</code></pre>
  </div>
  </div>
  </p>
  
  <p><strong>No backup</strong> 
  We have not backed up Nexus VM, therefore we wont be able to restore it back. What is the possible way out - recreating the database - with least damage. Appreciate any help.</p>