title: In Apache spark memory MemoryStore tryToPut what does free mean
body: <p>We run out of memory on our standalone spark cluster with multiple jobs.
  While investigating we found these message and began to suspect that too little memory is free</p>
  
  <pre><code>16/09/23 12:30:38 INFO MemoryStore: Block broadcast_50802_piece0 stored as bytes in memory (estimated size 5.1 KB, free 233.5 KB)
  16/09/23 12:30:38 INFO TorrentBroadcast: Reading broadcast variable 50802 took 9 ms
  16/09/23 12:30:38 INFO MemoryStore: Block broadcast_50802 stored as values in memory (estimated size 11.3 KB, free 244.9 KB)
  </code></pre>
  
  <p>In another cluster we typically have free reported as 500MB+ and many log traces on stack overflow show free in GBs.</p>
  
  <p>After analyzing the code it seems this message is misleading. The reported free memory is actually blocksMemoryused</p>
  
  <p><a href="https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/storage/MemoryStore.scala" rel="nofollow">https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/storage/MemoryStore.scala</a></p>
  
  <pre><code>if (enoughMemory) {
          // We acquired enough memory for the block, so go ahead and put it
          val entry = new MemoryEntry(value(), size, deserialized)
          entries.synchronized {
            entries.put(blockId, entry)
          }
          val valuesOrBytes = if (deserialized) "values" else "bytes"
          logInfo("Block %s stored as %s in memory (estimated size %s, free %s)".format(
            blockId, valuesOrBytes, Utils.bytesToString(size), Utils.bytesToString(blocksMemoryUsed)))
        } else {
          // Tell the block manager that we couldn't put it in memory so that it can drop it to
          // disk if the block allows disk storage.
          lazy val data = if (deserialized) {
            Left(value().asInstanceOf[Array[Any]])
          } else {
            Right(value().asInstanceOf[ByteBuffer].duplicate())
          }
          val droppedBlockStatus = blockManager.dropFromMemory(blockId, () =&gt; data)
          droppedBlockStatus.foreach { status =&gt; droppedBlocks += ((blockId, status)) }
        }
  </code></pre>
  
  <p>The documentation states its used memory not free</p>
  
  <pre><code>/**
     * Amount of storage memory, in bytes, used for caching blocks.
     * This does not include memory used for unrolling.
     */
    private def blocksMemoryUsed: Long = memoryManager.synchronized {
      memoryUsed - currentUnrollMemory
    }
  </code></pre>
  
  <p>Question is why is this called free if its actually used memory or am I mis interpreting.</p>