title: Azure Service Bus queue message lock expires at midnight UTC
body: <p>We're seeing some odd behaviour while processing a long-running message on an Azure Service Bus queue. We are renewing the lock as we run over a fairly large dataset and this particular message generally takes around 1 hour to complete. The process works great and runs through to completion, <strong><em>unless</em></strong> it is running over midnight UTC. </p>
  
  <p>Bang on midnight UTC, our process throws an exception: </p>
  
  <p><em>Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.</em></p>
  
  <p>We are able to reproduce this night after night, and it never happens if we don't run the process over midnight.</p>
  
  <p>Could it be that the message lock "ExpiresAtUtc" timestamp calculation doesn't handle crossing over from one day to another very gracefully?</p>
  
  <p>---- UPDATE ----</p>
  
  <p>Some more information that might help:</p>
  
  <p>The long running process keeps the message from becoming visible again on the queue by calling <code>RenewLock</code>. When this processing continues over midnight we notice that the message becomes visible on the queue and processing is restarted. The message is not dropped and does not move to the dead letter queue. It's lock simply expires and the message becomes visible on the queue again so it is picked up by a processor and the process is restarted. As long as the process does not cross the boundary of midnight UTC it completes successfully. </p>
  
  <p>Here are snippets of code we're using to connect / enqueue / dequeue from the queues:</p>
  
  <p>Connect to queue:</p>
  
  <pre><code>private QueueClient GetQueue&lt;TMessage&gt;() =&gt; QueueClient.CreateFromConnectionString(this.configSection.Value.ConnectionString, typeof(TMessage).Name, ReceiveMode.PeekLock);
  </code></pre>
  
  <p>Enqueue message: </p>
  
  <pre><code>using (var brokeredMessage = new BrokeredMessage(message) {ContentType = "application/json"})
  {
      await GetQueue&lt;TMessage&gt;().SendAsync(brokeredMessage).ConfigureAwait(false);
  }
  </code></pre>
  
  <p>Dequeue message:</p>
  
  <pre><code>GetQueue&lt;TMessage&gt;().OnMessageAsync(
      async msg =&gt;
      {
          TMessage body = null;
          try
          {
              body = msg.GetBody&lt;TMessage&gt;();
  
              await handler.HandleMessageAsync(body, msg.RenewLockAsync).ConfigureAwait(false);
              await msg.CompleteAsync().ConfigureAwait(false);
          }
          catch (Exception ex)
          {
              await msg.AbandonAsync().ConfigureAwait(false);
          }
      },
      new OnMessageOptions
      {
          AutoComplete = false
      }
  ); 
  </code></pre>
  
  <p>Below is a screenshot of the Azure Service Bus Metrics - Successful Requests, over midnight UTC (I'm in a UTC+1 timezone so that's 1:00 AM where I am) showing how at midnight <em>something</em> affecting the queue happens:</p>
  
  <p><a href="https://i.stack.imgur.com/synV9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/synV9.png" alt="enter image description here"></a></p>
  
  <p>Below is a screenshot of our internal logging, the processing just stops and then starts up again after just over a minute - when the lock expires and the message becomes visible on the queue again so is picked up by the processor:</p>
  
  <p><a href="https://i.stack.imgur.com/WohfC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WohfC.png" alt="enter image description here"></a></p>