title: Spike in "500 - internal server errors" from web services running on IIS7, Windows Server 2008
body: <p>We have a bunch of websites running on a NLB of 3 web servers. One of the sites is a set of web services which is used to power a news application for an iPad.</p>
  
  <p>On Monday the client called up to say they were seeing some errors on the App. I jumped on the server and noticed a lot of 500s in the IIS logs. I then wrote a script to call the webserverice and managed to reproduce the 500 through fiddler but the response was useless.</p>
  
  <p>The web service gets around 200,000 hits a day. Typically we get less than 10 "500 internal server errors per day". On Sunday however these spiked to 2000 (around 1% of all requests) and have been at that level ever since.</p>
  
  <p>We have had no code deploys since mid June. Our server/network infrastructure has not changed in anyway and I can't explain to the client what the issue is.</p>
  
  <p>I have enabled Troubleshooting Failed Requests Using Tracing in IIS 7 as suggested in this <a href="https://stackoverflow.com/questions/5704744/http-error-500-internal-server-error-on-high-load-asp-net-application">post</a> but the error message is meaningless to me. I have Googled extensively but no dice, trace details below:</p>
  
  <pre><code>114. -MODULE_SET_RESPONSE_ERROR_STATUS 
  ModuleName ManagedPipelineHandler 
  Notification 128 
  HttpStatus 500 
  HttpReason Internal Server Error 
  HttpSubStatus 0 
  ErrorCode 0 
  ConfigExceptionInfo  
  Notification EXECUTE_REQUEST_HANDLER 
  ErrorCode The operation completed successfully. (0x0) 
  </code></pre>
  
  <p>The web config is here (with irrelevant parts removed)</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  </code></pre>
  
  <p></p>
  
  <pre><code>&lt;configSections&gt;
      &lt;sectionGroup name="system.web.extensions" type="System.Web.Configuration.SystemWebExtensionsSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"&gt;
          &lt;sectionGroup name="scripting" type="System.Web.Configuration.ScriptingSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"&gt;
              &lt;section name="scriptResourceHandler" type="System.Web.Configuration.ScriptingScriptResourceHandlerSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /&gt;
              &lt;sectionGroup name="webServices" type="System.Web.Configuration.ScriptingWebServicesSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"&gt;
                  &lt;section name="jsonSerialization" type="System.Web.Configuration.ScriptingJsonSerializationSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="Everywhere" /&gt;
                  &lt;section name="profileService" type="System.Web.Configuration.ScriptingProfileServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /&gt;
                  &lt;section name="authenticationService" type="System.Web.Configuration.ScriptingAuthenticationServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /&gt;
                  &lt;section name="roleService" type="System.Web.Configuration.ScriptingRoleServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /&gt;
              &lt;/sectionGroup&gt;
          &lt;/sectionGroup&gt;
      &lt;/sectionGroup&gt;
  
  &lt;/configSections&gt;
  
  &lt;appSettings /&gt;
  &lt;connectionStrings /&gt;
  
  &lt;system.web&gt;
      &lt;webServices&gt;
          &lt;protocols&gt;
              &lt;add name="HttpGet" /&gt;
              &lt;add name="HttpPost" /&gt;
          &lt;/protocols&gt;
      &lt;/webServices&gt;
      &lt;compilation debug="false"&gt;
          &lt;assemblies&gt;
              &lt;add assembly="System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /&gt;
              &lt;add assembly="System.Data.DataSetExtensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /&gt;
              &lt;add assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
              &lt;add assembly="System.Xml.Linq, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /&gt;
          &lt;/assemblies&gt;
      &lt;/compilation&gt;
      &lt;pages&gt;
          &lt;controls&gt;
              &lt;add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
              &lt;add tagPrefix="asp" namespace="System.Web.UI.WebControls" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
          &lt;/controls&gt;
      &lt;/pages&gt;
      &lt;httpHandlers&gt;
          &lt;remove verb="*" path="*.asmx" /&gt;
          &lt;add verb="*" path="*.asmx" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
          &lt;add verb="*" path="*_AppService.axd" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
          &lt;add verb="GET,HEAD" path="ScriptResource.axd" type="System.Web.Handlers.ScriptResourceHandler, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" validate="false" /&gt;
      &lt;/httpHandlers&gt;
      &lt;httpModules&gt;
          &lt;add name="ScriptModule" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
      &lt;/httpModules&gt;
  &lt;/system.web&gt;
  &lt;system.codedom&gt;
      &lt;compilers&gt;
          &lt;compiler language="c#;cs;csharp" extension=".cs" warningLevel="4" type="Microsoft.CSharp.CSharpCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"&gt;
              &lt;providerOption name="CompilerVersion" value="v3.5" /&gt;
              &lt;providerOption name="WarnAsError" value="false" /&gt;
          &lt;/compiler&gt;
      &lt;/compilers&gt;
  &lt;/system.codedom&gt;
  
  &lt;!-- 
      The system.webServer section is required for running ASP.NET AJAX under Internet
      Information Services 7.0.  It is not necessary for previous version of IIS.
  --&gt;
  &lt;system.webServer&gt;
      &lt;validation validateIntegratedModeConfiguration="false" /&gt;
      &lt;modules&gt;
          &lt;remove name="ScriptModule" /&gt;
          &lt;add name="ScriptModule" preCondition="managedHandler" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
      &lt;/modules&gt;
      &lt;handlers&gt;
          &lt;remove name="WebServiceHandlerFactory-Integrated" /&gt;
          &lt;remove name="ScriptHandlerFactory" /&gt;
          &lt;remove name="ScriptHandlerFactoryAppServices" /&gt;
          &lt;remove name="ScriptResource" /&gt;
          &lt;add name="ScriptHandlerFactory" verb="*" path="*.asmx" preCondition="integratedMode" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
          &lt;add name="ScriptHandlerFactoryAppServices" verb="*" path="*_AppService.axd" preCondition="integratedMode" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
          &lt;add name="ScriptResource" preCondition="integratedMode" verb="GET,HEAD" path="ScriptResource.axd" type="System.Web.Handlers.ScriptResourceHandler, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /&gt;
      &lt;/handlers&gt;
      &lt;tracing&gt;
          &lt;traceFailedRequests&gt;
              &lt;add path="*.asmx"&gt;
                  &lt;traceAreas&gt;
                      &lt;add provider="ASP" verbosity="Verbose" /&gt;
                      &lt;add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" /&gt;
                      &lt;add provider="ISAPI Extension" verbosity="Verbose" /&gt;
                      &lt;add provider="WWW Server" areas="Rewrite,Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module" verbosity="Verbose" /&gt;
                  &lt;/traceAreas&gt;
                  &lt;failureDefinitions timeTaken="00:00:00" statusCodes="500" /&gt;
              &lt;/add&gt;
          &lt;/traceFailedRequests&gt;
      &lt;/tracing&gt;
  &lt;/system.webServer&gt;
  
  &lt;runtime&gt;
      &lt;assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1" appliesTo="v2.0.50727"&gt;
          &lt;dependentAssembly&gt;
              &lt;assemblyIdentity name="System.Web.Extensions" publicKeyToken="31bf3856ad364e35" /&gt;
              &lt;bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0" /&gt;
          &lt;/dependentAssembly&gt;
          &lt;dependentAssembly&gt;
              &lt;assemblyIdentity name="System.Web.Extensions.Design" publicKeyToken="31bf3856ad364e35" /&gt;
              &lt;bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0" /&gt;
          &lt;/dependentAssembly&gt;
      &lt;/assemblyBinding&gt;
  &lt;/runtime&gt;
  </code></pre>
  
  <p>
      
        
          
        
      
    
  </p>