Take the 2-minute tour ×

For about 4-5 months now, I seem to be having this sporadic issue--mainly during our busiest time of the day between 10:30-11:45AM, where all my Windows 2003 web servers in a Microsoft NLB cluster start throwing session state server errors. A sample error is below.

   at System.Web.SessionState.OutOfProcSessionStateStore.MakeRequest(StateProtocolVerb verb, String id, StateProtocolExclusive exclusiveAccess, Int32 extraFlags, Int32 timeout, Int32 lockCookie, Byte[] buf, Int32 cb, Int32 networkTimeout, SessionNDMakeRequestResults& results)
   at System.Web.SessionState.OutOfProcSessionStateStore.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
   at System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Now I'm using ASP.NET State service on a centralized back-end Windows 2003 server that all servers communicate to.

I was originally using SQL Server state for a couple years as well prior to having this issue. The problem with SQL wqas that when the issue occurred, it created a blocking situation which essentially impacted all users across all servers. The product company recommended that I use the standard ASP.NET State service as that was what they technically supported. Why this would make a difference is beyond me -- but I had no choice but to try it!

I have attempted to create multiple application pools, adding additional servers, chaning TCP/IP timeout from 20 to 30 seconds, and even calling Microsoft ASP.NET product support, with very little success. I even recommended that they review whether they are using read-only session state instead of read/write per page request -- as I understand that this basically causes every page to make round-trips to state server even if state isn't being used on the page.

Unfortunately, the application is developed by our product company and they insist that it is something with my environment because other clients do not have these sort of issues. However, I've talked to other clients and they tell me when they've seen issues like they, they've basically had to create another web farm. This issue almost seems like I've simply reached some architectural limit within the application...

Microsoft's position on the issue is that the session state needs to be reduced and the returncode being reported back from the state server indicates buffers are full.

To better understand the scope of issues (rather than wait for customers to call and complain), I installed ELMAH and configured it to send me e-mails when unhandled exceptions occur. I basically get 500-1000 e-mails during the time period of high activity!

If any one has any other ideas I could try or better ways to troubleshoot, I'd appreciate it.

share|improve this question
add comment

3 Answers

We started having issues with .net apps communicating to a remote .Net State Server when MS patch KB967723 was applied. You might look and see if you have that patch applied to the web server and try backing it out.

share|improve this answer
add comment

Measure how much data the application stores in session state for a typical session, and a packet capture between a web server and the state server to measure the total amount of data during peak usage.

share|improve this answer
I'm not sure how to actually measure this. There are virtually 100+ ASP.NET applications installed across 4-5 AppPools. Not all the applications throw state errors, but typically is the ones that communicate with our back-end mainframe to do an account inquery or customer lookup. –  Angry_IT_Guru Dec 18 '09 at 14:07
Understood. I would be interested in this, as out-of-process session state is intended to be used rather sparingly. If an application maintains a lot of state there, or serializes a lot of objects or large objects, the performance will suffer. I reckon you could run NetMon 3.3 on one of the IIS servers, and filter on the PID of the IIS worker process that is hosting your application, and the ip address of the server where the iis state db is located, and the tcp port used by the state service (42424 I think). –  Greg Askew Dec 18 '09 at 21:28
Thanks the the feedback. I will attempt to run a network trace and see if anythign turns up. Since the product company has basically given up on this issue and I've purchased additional servers to scale out with no luck... I think I may have to create a new farm and split my servers between two farms where each farm has its own state server. That is probably going to be the only way I can see if the problem disappears by splitting the load. My issue with all of this is that I'm going to have to probably use a hardware load balancer or a new DNS name; which may lead to other issues. –  Angry_IT_Guru Dec 18 '09 at 22:26
add comment

You may want to increase the number of TCP ports available (the default is 5000). I have seen similar problems (not with Session State) caused by inadequate number of ports.

share|improve this answer
add comment

Your Answer

