Photo by Ben Hershey on Unsplash

Recently, my team and I were in the process of “discovering” the endpoints we needed to break out as we progressed through use cases and added more and more features.

One day, a new web service integration was added that was throwing an exception while handling an event published out of one of the “business” endpoints. This was a fairly important event in the system that many other handlers were interested in handling. The exception was causing all the other handlers handling that event to rollback.

Still being fairly new to NServiceBus, I looked up how to work with non-transactional items (like email and web service calls) and found this: Don’t invoke the integration directly from your handler, send a message to do it. Armed with this knowledge, I went searching through the code.

The Problem

To put this post in context (and because I can’t post real code from work), let’s use an example that illustrates the problem I was looking into. Let’s say we wanted to place an Order in a fictitious online store. We send a PlaceOrder command which is handled by PlaceOrderHandler. PlaceOrderHandler needs to:

  1. insert the Order into the database
  2. publish OrderPlaced
public class PlaceOrderHandler : IHandleMessages<PlaceOrder>  
{  
    private readonly IBus bus;

    public PlaceOrderHandler(IBus bus)  
    {  
        this.bus = bus;  
    }

    public void Handle(PlaceOrder message)  
    {  
        using (var context = new Context())  
        {  
            context.Order.Add(new Data.Order { OrderId = message.OrderId, 
                ProductId = message.ProductId, CustomerId = message.CustomerId, 
                Quantity = message.Quantity });  
            context.SaveChanges();  
        }  
        bus.Publish(new OrderPlaced { OrderId = message.OrderId, CustomerId = message.CustomerId });  
    }  
}  

So far, so good. The code is publishing the OrderPlaced event, not invoking a web service. So I tracked down the handler that handled OrderPlaced and made the web service call:

public class InvokeWebServiceHandler : IHandleMessages<OrderPlaced>  
{  
    public void Handle(OrderPlaced message)  
    {  
        //INVOKE THE WEB SERVICE CALL HERE  
    }  
}  

This is also looked correct to me!

  • we were publishing the OrderPlaced event from PlaceOrderHandler and not invoking a web service directly from PlaceOrderHandler.
  • the InvokeWebServiceHandler handled OrderPlaced and invoked the web service.

But there were still rollbacks occurring across multiple handlers that handled OrderPlaced. Looking at the code, I knew we were calling the integration properly, but we were still having problems.

Turns out, the InvokeWebServiceHandler was in the same endpoint as other handlers handling OrderPlaced. Because of how NServiceBus manages transactions, ALL handlers must complete successfully in the same endpoint handling the same event or NONE of them can complete at all.

Here’s an example of another handler handling the OrderPlaced event. This handler is responsible for making a customer preferred:

public class ChangeCustomerPreferredStatusHandler : IHandleMessages<OrderPlaced>  
{  
    public void Handle(OrderPlaced message)  
    {  
        using (var context = new Context())  
        {  
            var customerPreferred = context.CustomerPreferred
                .SingleOrDefault(x => x.CustomerId == message.CustomerId);  
            if (customerPreferred != null)  
                return;

            var customersOrderCount = context.Order
                .Where(x => x.CustomerId == message.CustomerId).Sum(x => x.Quantity);  
            if (customersOrderCount > 100)
            {
                context.CustomerPreferred.Add(new CustomerPreferred 
                { 
                     CustomerId = message.CustomerId, IsPreferred = true 
                });  
            } 
            context.SaveChanges();  
        }  
    }  
}  

When the web service invocation failed in the InvokeWebServiceHandler and caused all handlers handling OrderPlaced to rollback, the customer never had a chance to be marked as preferred. Making a customer preferred is a very important step in our system, and it needs to work correctly. A web service failure should NOT be causing this handler’s transaction to rollback.

The Fix

There were two ways I could solve this problem:

  • move InvokeWebServiceHandler to a separate endpoint and let the endpoint take care of the transactional boundary.
  • leave InvokeWebServiceHandler is its current endpoint, but handling the transactional boundary myself.

Moving InvokeWebServiceHandler to its Own Endpoint

In a perfect world, you could have one endpoint for every handler in your system. So if I created a new endpoint for InvokeWebServiceHandler, that would solve my transactional boundary problem. If the web service invocation failed in the new endpoint, there are no other handlers handling OrderPlaced in the new endpoint that would also rollback. This is one way to go about fixing this issue.

  • Pros: - transaction scope is managed by the endpoint.
  • Cons: - I have to add another endpoint to my system. This means MEM’s, adding another project to the solution, and another service running on our servers.

Keeping InvokeWebServiceHandler in its Current Endpoint

I could keep the InvokeWebServiceHandler in the same endpoint, but instead of invoking the web service from this handler, I could instead have this handler send a command to invoke the web service. This is an example of transforming events into commands. The command being sent starts a new transaction in the handler that will invoke the web service. This is another way to go about fixing this issue.

  • Pros - we don’t have to add a new endpoint to the system
  • Cons - we have to transform OrderPlaced into a command to send that command to a handler to invoke the web service.
  • repetitive code like this could cause a good amount of code bloat in our system if we’re using this approach a good amount. This solution requires me to add another command and a new handler so I can call the web service.

I chose to keep InvokeWebServiceHandler in its current endpoint.

Based on thOrderPlacedHandlerInvokesWebService](GHOST_URL/content/images/2015/07/orderplacedhandlerinvokHandlesOrderPlacedAndSendsACommandToInvokeAWebService](GHOST_URL/content/images/2015/07/handlesorderplacedandsendsacommandtoinvokeawebservice1.png)

Let’s check out some code to clarify.

The Implementation

We’ll start our changes in InvokeWebServiceHandler. It’s still handling OrderPlaced, but we’ve changed the name of the handler to SendCommandToInvokeWebServiceHandler:

public class SendCommandToInvokeWebServiceHandler : IHandleMessages<OrderPlaced>  
{  
    private readonly IBus bus;

    public SendCommandToInvokeWebServiceHandler(IBus bus)  
    {  
        this.bus = bus;  
    }

    public void Handle(OrderPlaced message)  
    {  
        bus.SendLocal(new InvokeWebService { OrderId = message.OrderId, CustomerId = message.CustomerId });  
    }  
}  

Instead of trying to invoke the web service from this handler, we are now sending a command to invoke it instead. This keeps integration operations (such as calling a web service) separate from other handlers that might be doing “business-related stuff” when handling the OrderPlaced event.

This is an example of transforming an event into a command that I mentioned earlier. We’re taking the OrderPlaced event, and sending an InvokeWebService command when we handle the event. Also note the use of bus.SendLocal here. This routes that command right back to the endpoint sending the command, so I don’t need an MEM like if I had chosen to move InvokeWebService handler to its own endpoint.

Finally, here is the new InvokeWebServiceHandler. This is the handler now invoking our web service.

public class InvokeWebServiceHandler : IHandleMessages<InvokeWebService>  
{  
    public void Handle(InvokeWebService message)  
    {  
        //INVOKE WEB SERVICE HERE  
    }  
}  

If this web service invocation fails, it fails within its own transaction created by the command that was sent from the SendCommandToInvokeWebServiceHandler. This means our other business-related handlers that need to do something when OrderPlaced is published can get that work done without being shut down by a “rogue” web service.

In Closing

By sending a command, we allowed OrderPlaced to successfully complete its transaction in the endpoint across all handlers regardless of the outcome of the web service call.

I outlined two ways I could have fixed this issue. Each solution has it’s own pros and cons that you would need to consider and weigh depending on the context of your solution and your business environment.

That being said, now that our solution has been broken down into more granular endpoints, I will be moving the original InvokeWebService handler into its own endpoint and getting rid of the event to command translations, the commands, and the handler for those commands.

All supporting code can be found over at my GitHub account. Before running the companion code to this post:

  • comment IN the “throw new Exception(“the web service call failed”);” line and comment OUT the bus.Send line in SendCommandToInvokeWebServiceHandler. Then try to place an order. Note that CustomerPreferred will NOT be written to the database, because the handler invoking the web service call failed.
  • Now, comment OUT the exception being throw in the ChangeCustomerPreferredStatusHandler and comment IN the bus.Send line. In the InvokeWebServiceHandler, I’m throwing an exception to simulate the web service call failing. When you run the code, even though you still see exceptions being thrown, note that the CustomerPreferred entity is successfully written to the database.