Application That Can Deadlock

Join the DZone community and get the full member experience.

Code Deadlock in a distributed E-Commerce Application. The customer application in question was a busy e-commerce retail website in the US. The architecture was heavily distributed with several hundred application tiers that included JVMs, LDAP servers, CMS server, message queues, databases and 3rd party web services. A deadlock occurs when the waiting process is still holding on to another resource that the first needs before it can finish. So, an example: Resource A and resource B are used by process X and process Y. X starts to use A. X and Y try to start using B; Y 'wins' and gets B first; now Y needs to use A; A is locked by X, which is waiting for Y. Application deadlock example. Some developers simulate cursors by using two or more connections from DB-Library™. One connection performs a select and the other connection performs updates or deletes on the same tables. This can create application deadlocks. For example: Connection A holds a shared lock on a page. Transaction (Process ID 166) was deadlocked on lock communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Troubleshooting Deadlocks. So we have identified Deadlock happened in the database through our Application Insights. Next logical question is, what caused this deadlock. When the lock monitor detects multiple application threads trapped in a deadlock, it selects one of them to be the deadlock victim. That thread's processing jobs are terminated, and the ongoing transaction is rolled back in the database.

Join For Free

Deadlocks are situations in which two or more actions are waiting for the others to finish, making all actions in a blocked state forever. They can be very hard to detect during development, and they usually require restart of the application in order to recover. To make things worse, deadlocks usually manifest in production under the heaviest load, and are very hard to spot during testing. The reason for this is it’s not practical to test all possible interleavings of a program’s threads. Although some statical analysis libraries exist that can help us detect the possible deadlocks, it is still necessary to be able to detect them during runtime and get some information which can help us fix the issue or alert us so we can restart our application or whatever.

Detect deadlocks programmatically using ThreadMXBean class

Java 5 introduced ThreadMXBean - an interface that provides various monitoring methods for threads. I recommend you to check all of the methods as there are many useful operations for monitoring the performance of your application in case you are not using an external tool. The method of our interest is findMonitorDeadlockedThreads, or, if you are using Java 6,findDeadlockedThreads. The difference is that findDeadlockedThreads can also detect deadlocks caused by owner locks (java.util.concurrent), while findMonitorDeadlockedThreads can only detect monitor locks (i.e. synchronized blocks). Since the old version is kept for compatibility purposes only, I am going to use the second version. The idea is to encapsulate periodical checking for deadlocks into a reusable component so we can just fire and forget about it.

One way to impement scheduling is through executors framework - a set of well abstracted and very easy to use multithreading classes.

Simple as that, we have a runnable called periodically after a certain amount of time determined by period and time unit. Next, we want to make our utility is extensive and allow clients to supply the behaviour that gets triggered after a deadlock is detected. We need a method that receives a list of objects describing threads that are in a deadlock:

Now we have everything we need to implement our deadlock detector class.

Let’s test this in practice. First, we will create a handler to output deadlocked threads information to System.err. We could use this to send email in a real world scenario, for example:

This iterates through all stack traces and prints stack trace for each thread info. This way we can know exactly on which line each thread is waiting, and for which lock. This approach has one downside - it can give false alarms if one of the threads is waiting with a timeout which can actually be seen as a temporary deadlock. Because of that, original thread could no longer exist when we handle our deadlock and findDeadlockedThreads will return null for such threads. To avoid possible NullPointerExceptions, we need to guard for such situations. Finally, lets force a simple deadlock and see our system in action:

Output:

Keep in mind that deadlock detection can be an expensive operation and you should test it with your application to determine if you even need to use it and how frequent you will check. I suggest an interval of at least several minutes as it is not crucial to detect deadlock more frequently than this as we don’t have a recovery plan anyway - we can only debug and fix the error or restart the application and hope it won’t happen again. If you have any suggestions about dealing with deadlocks, or a question about this solution, drop a comment below.

Like This Article? Read More From DZone

Published at DZone with permission of Ivan Korhner , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

A deadlock detection library can be used to track down async/await relateddeadlocks in your code with minimal overhead and effort.

Application That Can Deadlock Start

Overview

Application That Can Deadlock Get

The async/await pattern greatly simplified writing sequential code thatoperates asynchronous. A lot of libraries now use this pattern to allowcode to be more scalable and responsive. Despite of the simplified code,asynchronous code can still be tricky and a potential source of deadlocks.

Most deadlocks are caused by mixing synchronous and asynchronous code andnot dealing with synchronization contexts. The best practice is to use asyncall the way, but this sometimes is not possible. Some examples:

Constructors, destructors, disposal, event handlers require synchronouscode. It's often better to rethink your design and check if you can use analternative implementation, but sometimes you can't.
.NET libraries (or third party libraries) assume your code to be synchronous.Starting a Windows service means overriding the OnStart and OnStop methodsthat are synchronous. If you need to call an asynchronous method insidethese methods, then the easiest option is to use Wait() or Result.

If you need to call Task.Wait()or Task<T>.Resultthen your current thread blocks. If one of the underlying tasks require that itneeds to continue on the same synchronization context, then the applicationwill deadlock.

A good starting point is to install the ConfigureAwait Resharper extension,so you don't forget to explicitly mark each await. If your code doesn't dependon the synchronization context then it's often best to callConfigureAwait(false) to prevent deadlocks. If your method is called by externalcode then be cautious when using ConfigureAwait(true) (or discarding it). Itmight deadlock if the caller uses a synchronization context and uses a blockingoperation.

These deadlocks are hard to debug and I have spent several hours debuggingsuch deadlocks for projects that I work on. To prevent even more work, I thoughtof a simple and lightweight library that I could use to find deadlocks earlierand more easy.

Example

Let's illustrate the functionality with an example. Suppose we have thefollowing program that has a potential deadlock:

If this code is run with a synchronization context (i.e. GUI application or anASP.NET request handler), then it will block. The Test method calls theasynchronous TestAsync method and will block until it completes. However, thedelayed task needs to complete on the thread that is now blocked. We haveencountered a deadlock situation.

These situations can be hard to find and to track down. An application may stopresponding or a service won't respond and it will slowly eat up your systemresources, because it will use more and more threads.

The DeadlockDetection library can help to track down these issues with minimaleffort and overhead. The only thing you need to do is to make sure you injecta custom synchronization context. This synchronization context will interceptall task continuations that have require continuation on the synchronizationcontext.

If the synchronization context is blocked, due to a Wait, Result or otherblocking operation then it will raise a DeadlockException when continuationis requested on the blocked thread.

The only thing that you need to do is to install theDeadlockDetectionSynchronizationContext on your current thread. It's bestto use the Enable.DeadlockDetection method for this purpose. If deadlockdetection is disabled (i.e. for production systems) then it makes sure ithas zero side-effects and no performance penalty.

So this code will now look like this:

In this example I have set the detection mode to the most strict mode, so itwill also raise an exception when a potential deadlock might occur. If you runthis code, then the DeadlockException will be thrown when the TestAsyncmethod starts awaiting.

Suppose that you are calling external code or libraries, then deadlocks willstill be detected. Only the calling code needs to be modified.

Application That Can Deadlock

Caveat

The DeadlockException will be thrown whenever a continuation methods isscheduled while the synchronization context is blocked. It is possible that thesynchronization context is only blocked for a short duration (i.e. Sleep) andit is not actually a deadlock situation.

Although it might be possible that the application will not deadlock, thislibrary will still raise the DeadlockException. It might be considered a bug,but you might also rethink your design. Mixing blocking calls and asynchronouscalls on the same synchronization context is a bit awkward and reduces theasynchronous behavior of your application (and therefore reduce scalability),so you might even consider this a feature that you'll be notified of thesecircumstances :-)

Top free california id card template downloads. Easy ID Card Creator is a handy and reliable software that helps you to easily and quickly create ID cards. ID Flow Photo ID Card Software provides everything you need to design and print ID cards. Nov 05, 2018 Get California Driving License PSD Editable Template and Source file. This is a layered Photoshop blank document which can be used to create a new fake California USA State DL and free to use. Add Name, Address, Signature, License No. Height, Eye Color, Picture etc for verification purpose of your online accounts.

Settings

There are some global settings that can be set to change the behavior of thedeadlock detection library.

DeadlockDetection.GlobalSettings.DefaultDetectionMode

This setting specifies how deadlocks are detected:

Disabled will disable the entire library and the code will run exactlyas if the library wasn't used at all. It won't affect performance orbehavior in any way. It's okay to leave the usings (without setting thedetection mode) in production code this way.
OnlyActualDeadlocks will only raise exceptions when the deadlock detectionis explicitly enabled via the using block. Potential deadlocks will not bedetected.
AlsoPotentialDeadlocks will raise exceptions when both actual or potentialdeadlocks are detected. This should only be enabled during development and/ortesting.

DeadlockDetection.GlobalSettings.GenerateStackTraces

This setting specifies if stack traces are generated during the detection of(potential) deadlocks. Although it is convenient to have the location of theblocking operation, the generation of stack traces might affect performance,so it is best to disable it when not running in development and/or testing.

Under the hood

The deadlock detection library uses existing facilities in theSynchronizationContext class to detect deadlocks. That's why I was able toimplement it without any overhead. It uses the following two techniques:

When a task is continued on the synchronization context, then the CLRwill use the SynchronizationContext.Post method to call the continuationmethod.
Synchronization contexts can opt-in to be notified when it blocks (viathe SynchronizationContext.SetWaitNotificationRequired method). Itcalls the SynchronizationContext.Wait method to block the thread.

The DeadlockDetectionSynchronizationContext opts-in for the wait notificationand sets the _isBlocked flag. If the continuation method is invoked via thePost method of our synchronization context, then it will check if the contextis currently blocked and if so, it throws the DeadlockException.

Integrating in your application

Enabling deadlock detection should be done at the level where thesynchronization context is known and won't change anymore. I will provideseveral examples on how to integrate it the least intrusive way and maximizethe effect.

First you start with installing it via NuGet

Integrating in ASP.NET applications

ASP.NET applications use a synchronization context that is unique for eachrequest. It provides access to the current HTTP context and can be used toflow the principal through the request handler code.

It's important to wrap the proper synchronization context, so to enable Thedeadlock detection it must be enabled in the middleware.

ASP.NET applications using a global.asax file the following lines of codeshould be added to the file:

This will enable deadlock detection for each request. When the request hasfinished, the synchronization context will be reset again.

Integration in WinForms applications

Integrating in a WinForms application is pretty easy and relativenon-intrusive. The typical Main method of a WinForm application looks likethis:

The deadlock detection synchronization context only needs to be installed onthe UI thread. The WinForms library automatically installs theWindowsFormsSynchronizationContext during the creation of the first Windowscontrol. Because the deadlock detection synchronization context needs to wrapthe synchronization context, we need to make sure the synchronization contextis created.

We can simply create a dummy control, which will install the propersynchronization context. We can then wrap it and run the actual application.The Main method will look something like this:

This will enable automatic deadlock detection on the GUI thread.

Integration in WPF applications

I am currently investigating how to inject the deadlock detectionsynchronization context in a non-intrusive way for WPF applications. thedifficulty is that for each message that is processed internally a newDispatcherSynchronizationContext is created. This makes it moredifficult to inject it globally in your application.

You can use the local detection method (described below) if you want todetect deadlocks in a part of your code.

What about other applications?

Normal console applications or Windows services don't have a synchronizationcontext, so they will never generate a deadlock situation. So i is not necessaryto have deadlock detection for these kind of applications.

Local detection

If you don't want to enable the deadlock detection for your entire application,but only for a certain point (i.e. during debugging) you can simply wrap thesynchronization context.

Suppose you know you have a deadlock when clicking a certain button, then youcan use the following code:

Within the using block the deadlocks will be detected.