.NET 9.0 Debugging Issue - F10 Step Over executes multiple steps

I recently ran into a debugging issue with my latest VisualStudio 2022 update and Resharper C# installation. It's important to note that I didn't start using this setup until I'd upgraded my projects to .NET 9.0.

I was able to hit breakpoints but found that while stepping over statements (F10), the debugger would sometimes run to the next breakpoint, execute a random number of steps forward to a different line in the call stack, or run to completion. The issue was intermittent too. Ugh. I also use a number of VisualStudio extensions so that didn't make it any easier to find the cause. I ended up trying the following to no avail:

1. disabling Resharper's Async Typing and Debugger Integration. I'd seen some race conditions with the Async Typing and since the debugging issue was intermittent I figured I'd try disabling Async Typing even though it didn't have any apparent connection to the debugger. 
2. disabling all extensions 
3. uninstalling all extensions 
4. repairing the VisualStudio installation

Once I was down to a bare bones VisualStudio installation I figured it was VisualStudio itself. By this time I didn't want to continue debugging by deduction any longer so I didn't want to go down the road of rolling back to previous VisualStudio versions. I went on to the Feedback Forum for VisualStudio to look for an existing bug and open one if I couldn't find one there. I came upon this issue in the Developer Forum: Debugger randomly stepping over user code. It never occurred to me that I was now using .NET 9 and the problem could've been related. 🤯

I ultimately found the following issues opened for the .NET 9.0 runtime which all had the same root cause:

• Debugging Issue VS 2022 .net 9 #110841
• Step becomes a "go" in .NET 9 #109785
• foreach will jump to other code in dotnet9 #109812
• Step Over sometimes executes a lot more than one statement #109885
• And the fix merged here: #110533

The fix was pretty small and there were only two files changed:

The Controller in the CoreClr's Debugger Execution Engine:

And the Thread Suspension class:

Ironically, as I was writing this post I got an email about .NET 9.0.1 being released. I couldn't find any of the issues or the PR for the fix included in the Release Notes. I went back to the Fix a step that becomes a go #110533 PR notes and saw that the Milestone for the fix is 9.0.2 (release date TBD). I was disappointed to find that the fix was not included in the .NET 9.0.1 Release but happy that it is coming in the next release of 9.0.2.

Memory Leaks - Part 2: Debugging Tools to Diagnose a Memory Leak

If you are reading this you may be in the unfortunate position of looking for an elusive and possibly intermittent problem in your production application. Debugging and diagnosing a memory leak or an OutOfMemoryException can be a daunting, intimidating and challenging task. Fortunately, there are a number of tools to help; some of these are "paid license" apps, but there are even more tools that are free. I'll discuss some .NET Debugging Tools available. I've used all of the following tools with the exception of SciTech .NET Memory Profiler. Each has its advantages and disadvantages. Personally, I prefer the free tools for several reasons:

1. Getting approval for licensing and purchase of a 3rd-party tool is an uphill task and you rarely have time to wait for the approval process when it is a production issue.
2. I find the feature set of a combination of the free tools gives you the largest "surface area" of features to help. Some are better at strict data collection, others have graphical views, some are static for post-mortem debugging, and others can do real-time analysis.
3. Even though these are free, they are very robust and provide just as much data as the paid license tools. A good example of this is WinDbg which was developed in 1993 by Microsoft for in-house debugging of the Windows Kernel.

Free Tools

Advantages

• Duh! It's free
• There are lots of tools to choose from
• All of the ones I've seen are from reputable and well-known companies.
• You can often find blog posts (ahem...), articles, reddit threads, etc. that can provide some direction for getting started.

Disadvantages

• Formal documentation can be lacking. Finding a blog post or article is great but comprehensive detail is often missing or at best glossed over. This can make getting started a bit more of a challenge if the tool is new to you.
• Your company may have restrictions on using free tools for fear of malware, liability, or other reasons.

• Windows Task Manager: taskmgr.exe. Make sure to make the following columns visible:
• Memory - Working Set
• Memory - Private Working Set
• Memory - Commit Size
• Threads
• Handles
• USER Objects (if WinForm or Native app)
• GDI Objects (if WinForm or Native app)
• Windows Performance Monitor: perfmon.exe
• Process Explorer (SysInternals)
• VM Map (SysInternals)
• Debugging Tools for Windows (WinDbg, SOS, Kernel Debugger, etc.)
• Microsoft Debug Diagnostic Tool (DebugDiag)
• Psscor2 - WinDbg debugger extension for ASP.NET 2.0, 3.5
• Psscor4 - WinDbg debugger extension for ASP.NET 4.x
• SOSEX from Steve Johnson (.NET 2.0, 3.5)
• SOSEX 4 from Steve Johnson (.NET 4.x)
• IISInfo - WinDbg debugger extension (included with DebugDiag)
• Microsoft Application Verifier
• CLR Profiler 2.0 (good for finding heap fragmentation)
• CLR Profiler 4.0 (good for finding heap fragmentation)
• Windows Performance Toolkit (xperf)

Licensed Tools

• RedGate Ants Memory Profiler
• Micro Focus DevPartner Studio (formerly Compuware)
• SciTech .NET Memory Profiler
• AQ Time Pro

Hope this helps!

How to determine which garbage collector is running

CodeProjectYou can determine which version of GC you're running via 2 methods:

1. calling the System.Runtime.GCSettings.IsServerGC property
2. attaching to the process using WinDbg and checking how many GC threads you have using the command "!sos.threads" without the quotes and (according to the below criteria)...

If you are running a Console app, WinForm app or a Windows Service, you will get the Workstation GC. Just because you are running on a Server OS doesn't mean that you will get the Server version of GC.

• If your app is non-hosted on a multi-proc machine, you will get the Workstation GC - Concurrent by default.
• If your app is hosted on a multi-proc machine, you will get the ServerGC by default.

The following apply to any given .NET Managed Process:

Workstation GC

• Uni-processor machine
• Always suspends threads
• 1 Ephemeral GC Heap (SOH), 1 LOH GC Heap
• Runs on thread that triggered GC
• Thread priority is the same as the thread that triggered GC

Workstation GC - Concurrent

• Only runs concurrent in Gen2/LOH (full collection)
• Mutually exclusive with Server Mode
• Slightly larger working set
• GC Thread expires if not in use after a while
• 1 Ephemeral GC Heap (SOH), 1 LOH GC Heap
• Has a dedicated GC Thread
• Thread priority is Normal

Server GC

• Larger segment sizes
• Faster than Workstation GC
• Always suspends threads
• 1 Ephemeral GC Heap (SOH) for each logical processor (this includes hyperthreaded), 1 LOH GC Heap for each logical processor (this includes hyperthreaded)
• Has dedicated GC Threads
• Thread priority is THREAD_PRIORITY_HIGHEST

There is only 1 Finalizer thread per managed process regardless of GC Mode. Even during a concurrent GC, managed threads are suspended (blocked) twice to do some phases of the GC.

A seldom known fact is that even if you try to set the Server mode of GC, you might not be running in Server GC; the GC ultimately determines which mode will be optimal for your app and WILL override your settings if it determines your ServerGC setting will negatively impact your application. Also, any hosted CLR app will have any manual GC settings overridden.

In CLR 4.0, things change just a little bit

• Concurrent GC is now Background GC
• Background GC only applies to Workstation GC
• Old (Concurrent GC):
• During a Full GC Allowed allocations up to end of ephemeral segment size
• Otherwise, suspends all other threads
• New (Background GC):
• Allows for ephemeral GC’s simultaneously with Background GC if necessary
• Performance is much faster
• Server GC always blocks threads for collection of any generation

In CLR 4.5, things change just a little bit...again

• Background Server GC
• Server GC no longer blocks. Instead, it uses dedicated background GC threads that can run concurrently with user code - see MSDN: Background Server GC

Thus, in .NET 4.5+, all applications now have background GC available to them, regardless of which GC they use.

.NET 4.7.1 GC Improvements

.NET Framework 4.7.1 brings in changes in Garbage Collection (GC) to improve the allocation performance, especially for Large Object Heap (LOH) allocations. This is due to an architectural change to split the heap’s allocation lock into 2, for Small Object Heap (SOH) and LOH. Applications that make a lot of LOH allocations, should see a reduction in allocation lock contention, and see better performance. These improvements allow LOH allocations while Background GC (BGC) is sweeping SOH. Usually the LOH allocator waits for the whole duration of the BGC sweep process before it can satisfy requests to allocate memory. This can hinder performance. You can observe this problem in PerfView’s GCStats where there is an ‘LOH allocation pause (due to background GC) > 200 msec Events’ table. The pause reason is ‘Waiting for BGC to thread free lists’. This feature should help mitigate this problem.

New WinDbg Extension available

I'm happy to announce that a new version of the WinDbg debugging extension PSSCOR2 (now called PSSCOR4) has been released.  It is available for public download here: 

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a06a0fea-a4d4-434e-a527-d6afa2e552dd

Whitepaper on the Debug Diagnostic Tool v1.1 - IIS debugging and debugdiag - Site Home - MSDN Blogs

Whitepaper on the Debug Diagnostic Tool v1.1 - IIS debugging and debugdiag - Site Home - MSDN Blogs

Tools for Assisting with Collection and Analyzing of Data for Problem Diagnosis

I was going through my Google Bookmarks and found a bunch of tools that are extremely helpful in diagnosing  many problems that occur on a web/app server and and interpreting the data collected. I’ve used all of them; they are all free and well known and used within the Community.  By no means is this an exhaustive list of all of the Tools and Add-Ins that I use - I'll get around to posting those someday : )

I’d start with the Data Collector. You can collect everything with Data Collector and then use the Log Parser and Visual Log Parser to help aggregate the data and even run queries against it. You may already know about some of these – but for those you haven’t seen, you should take a look!

Data Collector (supports IIS6 & IIS7, and can pull most other logs from any server):
http://blogs.msdn.com/b/carloc/archive/2010/02/01/idevdatacollector.aspx

Log Parser 2.2:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

Visual Log Parser:
http://visuallogparser.codeplex.com/

Log File Analysis:
http://blogs.msdn.com/b/delay/archive/2007/06/21/powerful-log-file-analysis-for-everyone-releasing-textanalysistool-net.aspx

Event Log Explorer:
http://www.eventlogxp.com/

IIS Log Analyzer (does graphs and reports):
http://www.iis.net/community/default.aspx?tabid=34&g=6&i=1864

IIS Diagnostics Toolkit (a suite of tools diagnosing Authentication, Authorization, SSL, Crash/Hang, etc.):
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9bfa49bc-376b-4a54-95aa-73c9156706e7

and then of course the "big daddy" bunch of them all - Debugging Tools for Windows:
http://www.microsoft.com/whdc/devtools/debugging/default.mspx

I hope you find these useful : )

Want to have "WinDbg and SOS-like" features in VisualStudio?

While I've become adept at using WinDbg and all of it's extensions to do post-mortem crash-dump analysis, I've not yet ventured down the path of live-debugging strictly with WinDbg.  Maybe I'm dense, but it's anything but intuitive to me.  Call me "new-school" but there's just something about having a real UI while debugging that makes me more productive.

So, if you've ever needed to use some functionality of WinDbg or SOS (and any of the other extensions) while debugging in VisualStudio, there is some hope...

I've added 2 feature suggestions on the Microsoft Connect website for two features:

1. View an object's rooted references at runtime - https://connect.microsoft.com/VisualStudio/feedback/details/637376/add-feature-to-debugger-to-view-an-objects-rooted-references
2. View an object's memory footprint/usage - https://connect.microsoft.com/VisualStudio/feedback/details/637373/add-feature-to-debugger-to-view-an-objects-memory-footprint-usage

If either of these are something you'd like to see in an upcoming version of VisualStudio, please click the links above and vote for them!


As a side note, if you can't wait for the next version of VisualStudio, there are 2 different workarounds in the meantime:

1. Load SOS into the Immediate Window as documented here:  http://msdn.microsoft.com/en-us/library/yy6d2sxs.aspx
2. The other option is to stop at a breakpoint in VisualStudio, and then attach WinDbg in "non-invasive" mode.

While these workarounds seem to be mildly satisfactory, it'd be great if these features were already built in to the VS Debugger - so vote!