Chaos Engineering - Part 1: Simmy, chaos in code

Chaos Engineering is a hot topic and very cool to dive into. But knowing where and how to start can be a challenge. In this series, I want to demonstrate different sides and technologies you can use. In the first post: Simmy , a package to inject different kinds of chaos in a .NET project.

In this series:

Chaos Engineering talk

I have been talking about Chaos Engineering for multiple groups, for example, the JetBrains .Net Days 2022 , but usually my examples are not easy to understand. I created a simple example project to show some code and dive a bit deeper into the code I used.

Sample project

The sample project for part 1 can be found here. After cloning the repository, run `docker-compose up` to run the project in Docker and go to http://localhost:6000/swagger .

Decorator

This post is not about design patterns , but when you want to wrap different behavior around existing objects, the decorator pattern it a perfect match! I created a decorator for different types of chaos, which I can enable and disable with settings. With a builder (shown below) I create a full set of behaviors. A lot of extra code, but still keeping it clean!

public ExceptionsDecorator(IProductsRepository inner, ExceptionSettings exceptionSettings)
{
    _inner = inner;
    _exceptionPolicy = MonkeyPolicy
        .InjectExceptionAsync((with) =>
            with.Fault(new InvalidDataException("Chaos Monkey says Hi!"))
                .InjectionRate(exceptionSettings.InjectionRate)
                .Enabled());
}

public async Task<List<Product>> All() =>
    await _exceptionPolicy.ExecuteAsync(async () => await _inner.All());

Builder

Another cool design pattern is used to wrap only the needed decorators (or even no decorator at all) around the product’s repository: the builder pattern .

The builder in action in the &lsquo;Program.cs&rsquo; :

var products = productsRepositoryBuilder
                .WithExceptions(chaosSettings.Value.ExceptionSettings.Enabled)
                .WithLatency(chaosSettings.Value.LatencySettings.Enabled)
                .WithResult(chaosSettings.Value.ResultSettings.Enabled)
                .WithBehavior(chaosSettings.Value.BehaviorSettings.Enabled)
                .Build();

Part of the &lsquo;ProductsRepositoryBuilder&rsquo; :

public IProductsRepository Build()
{
    IProductsRepository build = null;
    if (_exceptions)
        build = new ExceptionsDecorator(_productsRepository, _chaosSettings.ExceptionSettings);

    if (_latency)
        build = build == null
            ? new LatencyDecorator(_productsRepository, _chaosSettings.LatencySettings)
            : new LatencyDecorator(build, _chaosSettings.LatencySettings);

    if (_result)
        build = build == null
            ? new ResultDecorator(_productsRepository, _chaosSettings.ResultSettings)
            : new ResultDecorator(build, _chaosSettings.ResultSettings);

    if (_behavior)
        build = build == null
            ? new BehaviorDecorator(_productsRepository, _chaosSettings.BehaviorSettings, _productsDbContext)
            : new BehaviorDecorator(build, _chaosSettings.BehaviorSettings, _productsDbContext);

    return build ?? _productsRepository;
}

Exceptions

This is probably the easiest way to start creating some chaos: injecting exceptions. Simmy will randomly throw exceptions. Based on the injection rate, which can vary between 0% and 100%.

public class ExceptionsDecorator : IProductsRepository
{
    private readonly IProductsRepository _inner;
    private readonly AsyncInjectOutcomePolicy _exceptionPolicy;

    public ExceptionsDecorator(IProductsRepository inner, ExceptionSettings exceptionSettings)
    {
        _inner = inner;
        _exceptionPolicy = MonkeyPolicy
            .InjectExceptionAsync((with) =>
                with.Fault(new InvalidDataException("Chaos Monkey says Hi!"))
                    .InjectionRate(exceptionSettings.InjectionRate)
                    .Enabled());
    }

    public async Task<List<Product>> All() =>
        await _exceptionPolicy.ExecuteAsync(async () => await _inner.All());

    public async Task<Product?> ById(int id) =>
        await _exceptionPolicy.ExecuteAsync(async () => await _inner.ById(id));
}

Latency

When enabled, Simmy can add some latency in the mix. I made the latency a setting, to make it even more random I could have added a lower and upper limit.

public class LatencyDecorator : IProductsRepository
{
    private readonly IProductsRepository _inner;
    private readonly AsyncInjectLatencyPolicy _latencyPolicy;

    public LatencyDecorator(IProductsRepository inner, LatencySettings latencySettings)
    {
        _inner = inner;
        _latencyPolicy = MonkeyPolicy
            .InjectLatencyAsync(with =>
                with.Latency(TimeSpan.FromMilliseconds(latencySettings.MsLatency))
                    .InjectionRate(latencySettings.InjectionRate)
                    .Enabled());
    }

    public async Task<List<Product>> All() =>
        await _latencyPolicy.ExecuteAsync(async () => await _inner.All());

    public async Task<Product?> ById(int id) =>
        await _latencyPolicy.ExecuteAsync(async () => await _inner.ById(id));
}

Results

With the Results option of Simmy it is possible to add some serious chaos to your projects. You can alter results or return unexpected results. In the example below, I return a random number of items when the `All` method is called. When a specific product is requested I even change the name, description and price of a product. I am not sure if this is helpful when doing chaos experiments, returning a product with a different ID might make more sense.

public class ResultDecorator : IProductsRepository
{
    private readonly IProductsRepository _inner;
    private readonly ResultSettings _resultSettings;
    private readonly AsyncInjectOutcomePolicy<HttpResponseMessage> _resultPolicy;
    private readonly Random _random;

    public ResultDecorator(IProductsRepository inner, ResultSettings resultSettings)
    {
        _inner = inner;
        _resultSettings = resultSettings;
        _random = new Random();
    }

    public async Task<List<Product>> All()
    {
        var products = await _inner.All();
        var resultPolicy = MonkeyPolicy.InjectResultAsync<List<Product>>(with =>
            with.Result(products.OrderBy(x => _random.Next()).Take(_random.Next(0, products.Count)).ToList())
                .InjectionRate(_resultSettings.InjectionRate)
                .Enabled());
        return await resultPolicy.ExecuteAsync(() => Task.FromResult(products));
    }

    public async Task<Product?> ById(int id)
    {
        var product = await _inner.ById(id);
        var resultPolicy = MonkeyPolicy.InjectResultAsync<Product>(with =>
            with.Result(new Product()
                {
                    Id = product.Id,
                    Name = $"{product.Name}, the Monkey Version",
                    Description= $"A monkey tampered with your product and changed the description and the price. The original price was ${product.Price}.",
                    Price = product.Price + 0.5m
                })
                .InjectionRate(_resultSettings.InjectionRate)
                .Enabled());
        return await resultPolicy.ExecuteAsync(() => Task.FromResult(product));
    }
}

Behavior

The last type of chaos I am demonstrating here is behavior. And it is some extreme behavior: at line 10 I delete the database before executing the call, resulting in serious chaos. In fact, the application will not work anymore, until someone fixes the issue!

public class BehaviorDecorator : IProductsRepository
{
    private readonly IProductsRepository _inner;
    private readonly AsyncInjectBehaviourPolicy _behaviorPolicy;

    public BehaviorDecorator(IProductsRepository inner, BehaviorSettings behaviorSettings, ProductsDbContext productsDbContext)
    {
        _inner = inner;
        _behaviorPolicy = MonkeyPolicy.InjectBehaviourAsync(with =>
            with.Behaviour(async () => await productsDbContext.Database.EnsureDeletedAsync())
                .InjectionRate(behaviorSettings.InjectionRate)
                .Enabled());
    }

    public async Task<List<Product>> All() =>
        await _behaviorPolicy.ExecuteAsync(async () => await _inner.All());

    public async Task<Product?> ById(int id) =>
        await _behaviorPolicy.ExecuteAsync(async () => await _inner.ById(id));
}

The Configuration object

I’ve kept the configration simple. I can enable every type separately, change the injection rate and for the latency the duration of the latency. In the next post I explain how to update the settings and reload the application without restarting it, for now, change the variables in de appsettings.Development.json when you run the project from your IDE or change the values in the docker-compose.yml file and restart the docker container if you want to test it in Docker.

"ChaosSettings": {
  "ExceptionSettings": {
    "Enabled": false,
    "InjectionRate": 0.25
  },
  "LatencySettings": {
    "Enabled": false,
    "InjectionRate": 0.5,
    "MsLatency": 500
  },
  "ResultSettings": {
    "Enabled": false,
    "InjectionRate": 0.5
  },
  "BehaviorSettings": {
    "Enabled": true,
    "InjectionRate": 0.5
  }
},

Conclusion

Creating chaos with Simmy is easy. It might be a bit harder to come up with some resonable scenario’s, but it can be a good and simple way to start with some chaos experiments. There are some downsides too: as the chaos is injected in code, your projects might actually be more difficult to maintain, unnexessary complexity is added.