Fix Crash under high load

[halter73]

#1327

@Eilon

[Eilon]

This patch bug is approved. Please use the normal code review process w/ a PR and make sure the fix is in the correct branch, then close the bug and mark it as done.

[newbie2006]

I'm doing load test using VS2015 for an application that we're developing. The application is behind IIS/Kestrel. After couple mins, the server stopped responding, however, CPU and memory usage are fairly low. The thread count under the Resource Monitor went up to several thousands. I'm assuming this is caused by the similar issues that were discussed here regarding the thread lock on Kestrel. Does anyone have any recommendations on what I should do to resolve the issues? Thanks.

[Eilon]

@newbie2006 if it's the same issue as what @halter73 fixed in this bug, we'll have the patch out in 2 - 3 weeks.

[cesarblum]

I think @newbie2006 is referring to #1278.

Are you using static files? One solution some people found is to serve static files from a different IIS website without Kestrel.

Are you passing HttpContext.RequestAborted to any WriteAsync() calls in your code?

[halter73]

@CesarBS is likely right. The bug referenced in this issue causes Kestrel's process to die instead of becoming unresponsive. Fortunately, both issues will be fixed in the same patch releases.

[newbie2006]

Do you happen to have a rough estimate when the patch will be release? We are almost ready to release our new app until we hit this problem during load testing. Our backup plan was to take everything that we got and move it to MVC5 but that's the last option.

Just realized that Eilon said it'll be out in 2-3 weeks.

[newbie2006]

@CesarBS, the static files that we have are the *.css, *.js. under wwwroot folder. And we are not passing the HttpContext.RequestAborted to any WriteAsync() calls.

[halter73]

@Eilon gave the rough estimate as 2 - 3 weeks. #1278 will be fixed by the same patch.

The workaround that might be a bit easier than moving to MVC5 in the meantime is to serve static files (e.g. *.css, *.js or whatever else is under the wwwroot folder) using IIS's static file handler instead of using ASP.NET Core's UseStaticFiles/UseDefaultFiles/UseDirectoryBrowser methods until the patched packages are released.

[newbie2006]

@halter73, so does this mean that I would set up another app under IIS that would hold and serve these static files, then just reference these files from the ASP.NET core app and remove the app.UseStaticFiles() from the Startup.cs?

[newbie2006]

I've followed the instructions in this thread https://weblogs.asp.net/imranbaloch/leveraging-iis-static-file-feature-in-aspnetcore. Is that the right way to do it?

[halter73]

@newbie2006 That looks right to me. Make sure you have the extensions for all the static files you're serving configured. By removing app.UseStaticFiles(), it should be pretty obvious if you haven't since you'd likely see 404s.

[newbie2006]

@halter73
I was able to get it to work in VS2015 but when deploying to IIS 8.5, I got the 500 internal server error due to the section below in the web.config file. Am I missing something on the server?

<rewrite>
  <rules>
    <rule name="wwwroot">
      <match url="([\S]+[.](svg|js|css|png|gif|jpg|jpeg))" />
      <action type="Rewrite" url="wwwroot/{R:1}" />
    </rule> 
  </rules>
</rewrite>

[benaadams]

500 probably suggests misconfigured web.config; can you browse the site on localhost (e.g. via rdp) as that would tell you more.

You have the <rewrite> in the right section?

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>

And only a single <system.webServer> and <rewrite> at route? (rather than in a <location> subsection)

[newbie2006]

Yes, the <rewrite> section is under <system.webServer>

[newbie2006]

Installed the URLRewrite extension and that fixed the problem. Thanks for all the help!

Finally had a chance to load test this change. Unfortunately, it didn't seem to resolve the problem with the non-responding kestrel after about 500 user load during testing with Visual Studio 2015. The CPU and memory utilization is fairly low but the web app does not respond until restarting the app pool.