title: Hazelcast: LinkageError "attempted duplicate class definition for name"
body: <p>I saw this error in our hazelcast cluster.</p>
  
  <p>The code is trying to call executeOnKey() on an IMap.  e.g.</p>
  
  <pre><code>    IMap&lt;MyKey, MyCachedClass&gt; myMap = hz.getMap("my-map");
      myMap.executeOnKey(myKey, new EntryProcessor() {
          @Override
          public Object process(Map.Entry entry) {
              return entry.getValue().setItem(item);
          }
  
          @Override
          public EntryBackupProcessor getBackupProcessor() {
              return null;
          }
      });
  </code></pre>
  
  <p>...and am getting this exception:</p>
  
  <pre><code>Exception in thread "MYTHREAD" java.lang.LinkageError: loader (instance of com/hazelcast/internal/usercodedeployment/impl/ClassSource): attempted duplicate class definition for name: "com/mycompany/model/Item$ItemEnum"
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
  at com.hazelcast.internal.usercodedeployment.impl.ClassSource.define(ClassSource.java:50)
  at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:163)
  at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
  at com.hazelcast.internal.usercodedeployment.impl.ClassSource.loadClass(ClassSource.java:65)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
  at com.hazelcast.internal.usercodedeployment.impl.ClassSource.define(ClassSource.java:50)
  at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:161)
  at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
  at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentService.handleClassNotFoundException(UserCodeDeploymentService.java:89)
  at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentClassLoader.loadClass(UserCodeDeploymentClassLoader.java:57)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)
  at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:237)
  at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:646)
  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
  at java.util.ArrayList.readObject(ArrayList.java:797)
  at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
  at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:82)
  at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:75)
  at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
  at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:269)
  at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:574)
  at com.hazelcast.map.impl.operation.EntryOperation.readInternal(EntryOperation.java:263)
  at com.hazelcast.spi.Operation.readData(Operation.java:728)
  at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
  at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
  at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
  at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
  at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
  at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:323)
  at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:398)
  at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:153)
  at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:123)
  at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110)
  at ------ submitted from ------.(Unknown Source)
  at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:127)
  at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79)
  at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:162)
  at com.hazelcast.map.impl.proxy.MapProxySupport.executeOnKeyInternal(MapProxySupport.java:1099)
  at com.hazelcast.map.impl.proxy.MapProxyImpl.executeOnKeyInternal(MapProxyImpl.java:109)
  at com.hazelcast.map.impl.proxy.MapProxyImpl.executeOnKey(MapProxyImpl.java:757)
  at com.mycompany.Processor.java
  </code></pre>
  
  <p>So to me this seemed like:</p>
  
  <ul>
  <li>The 'executeOnKey(item)' command is being sent to the 'other' node in
  the cluster where this data is located.</li>
  <li>The other node is attempting to deserialize the 'item'.</li>
  <li>The other node cannot find the class for the inner enum 'Item$ItemEnum'</li>
  <li>The other node makes a request to UserCodeDeploymentService to fetch class definition.</li>
  <li>Once the class definition is fetched, and it attempts to create the new class, a
  LinkageError occurs.</li>
  </ul>
  
  <p>I just cannot understand why it's getting a duplicate class exception, when it first claimed it couldn't find the class.  And this error is proving difficult to replicate.</p>
  
  <p>We do have 'userCodeDeployment' set to allow loading of remote class definitions.  Our config looks like this:</p>
  
  <pre><code>UserCodeDeploymentConfig ucdConfig = new UserCodeDeploymentConfig();
  ucdConfig.setEnabled(true);
  ucdConfig.setClassCacheMode(UserCodeDeploymentConfig.ClassCacheMode.ETERNAL);
  ucdConfig.setProviderMode(UserCodeDeploymentConfig.ProviderMode.LOCAL_AND_CACHED_CLASSES);
  config.setUserCodeDeploymentConfig(ucdConfig);
  </code></pre>
  
  <p>So it sort of looks like it's doing what I'd expect.  i.e. getting a remote class definition because it can't find it locally.  Is just a puzzle as to why the exception is occurring.</p>