title: Why is ignite failing to deserialize a GridClosureProcessor object?
body: <p>I know a similar  stack has been posted by others. My question is "it looks like Ignite is trying to deserialize a GridClosureProcessor (or a closure from it?) . If so, why is it doing that? I am trying to root cause this issue but none of my code is in the stack except MyCallable mentioned at the top (not actually in the stack).</p>
  
  <p>No cache puts are happening in this code path unless they are internal. I mention this because of a comment on another post that "Unknown pair" can be caused by a cache put of the wrong type.</p>
  
  <p>I am focusing on</p>
  
  <blockquote>
    <p>Failed to deserialize object
    [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]</p>
  </blockquote>
  
  <p>Here's the rest.</p>
  
  <blockquote>
    <p>[2019-04-08 22:20:23,724][ERROR][pub-#63][GridJobWorker] Failed to
    initialize job
    [jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
    ses=GridJobSessionImpl [ses=GridTaskSessionImpl
    [taskName=com.obfucorp.aa.project.core.jobs.MyCallable,
    dep=GridDeployment [ts=1554761996013, depMode=SHARED,
    clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
    clsLdrId=730290ff961-8d93b961-09f2-48c3-bd2f-49db31aae61e, userVer=0,
    loc=true,
    sampleClsName=o.a.i.i.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
    pendingUndeploy=false, undeployed=false, usage=1],
    taskClsName=com.obfucorp.aa.project.core.jobs.MyCallable,
    sesId=700890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
    startTime=1554762023663, endTime=9223372036854775807,
    taskNodeId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
    clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false,
    cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false,
    internal=false, topPred=null,
    subjId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, mapFut=IgniteFuture
    [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
    hash=314803578]], execName=null],
    jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a]] class
    org.apache.ignite.IgniteCheckedException: Failed to deserialize object
    [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
      at
    org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9908)
      at
    org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:438)
      at
    org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1117)
      at
    org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1921)
      at
    org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
      at
    org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
      at
    org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
      at
    org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
      at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748) Caused by: class
    org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
    object
    [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
      at
    org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
      at
    org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
      at
    org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
      at
    org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310)
      at
    org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
      at
    org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
      at
    org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9902)
      ... 10 more Caused by: class
    org.apache.ignite.binary.BinaryInvalidTypeException: Unknown pair
    [platformId=0, typeId=-1409390795]    at
    org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:696)
      at
    org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1755)
      at
    org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
      at
    org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799)
      at
    org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329)
      at
    org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872)
      at
    org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:834)
      ... 16 more Caused by: java.lang.ClassNotFoundException: Unknown pair
    [platformId=0, typeId=-1409390795]    at
    org.apache.ignite.internal.MarshallerContextImpl.getClassName(MarshallerContextImpl.java:385)
      at
    org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:335)
      at
    org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:687)
      ... 22 more</p>
  </blockquote>
  
  <p>update - wanted to note this is happening a fresh deployment. There are no old files or persisted anything around. All classes are either pulled from even or freshly compiled. ty.</p>
  
  <p>Pavel, here is the (Scala) code (redacted)</p>
  
  <pre><code>object MyCallable {
      type FooList = Array[Foo]
  }
  
  class MyCallable(cacheName: String) extends IgniteCallable[FooList] with Serializable with LazyLogging {
      @IgniteInstanceResource
      private var ignite: Ignite = _
  
      override def call(): FooList = {
          logger.debug("callable called.");
          val fooCache = ignite.getOrCreateCache[String, Foo](cacheName)
          val qry = new ScanQuery[String, Foo]()
          qry.setLocal(true)
          val cursor = fooCache.query(qry)
          val ret = cursor.iterator().asScala.map(e =&gt; e.getValue).toArray
          logger.info("load status array: {}", ret.mkString)
          return ret
      }
  
      @IgniteInstanceResource
      def setIgnite(ignite: Ignite): Unit = {
          this.ignite = ignite
      }
  }
  </code></pre>