Benchmarking with BenchmarkDotNet

In this article, we will explore BenchmarkDotNet which is a powerful tool for benchmarking C# code snippets so that we can track the performance of a method we write. So, let’s see this in action.

Table of Contents

What is Benchmarking?

Benchmarking is the mechanism of measuring the performance of our code written in an application. In other words, Benchmarking is the analysis of code so that we can improve it in a better way if required.

Let’s explore BenchmarkDotNet.

So, to analyse the code performance, we can use BenchmarkDotNet which is a lightweight and powerful .NET library.

BenchmarkDotNet library works with .NET Framework as well as .NET Core applications. In this article, we will explore benchmarking of C# code using the BenchmarkDotNet library.

Install BenchmarkDotNet

To install the BenchmarkDotNet library, we can use the Nuget package manager interface.

Or, we can use the package manager console to install using the below command:

Install-Package BenchmarkDotNet -Version 0.13.12

So, our installation is done, now, let’s create a class and add 2 methods to perform similar tasks.

public string QueryableResult()
{
    int[] numbers = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
    IQueryable<int> queryableNumbers = numbers.AsQueryable();
    var result = queryableNumbers.Where(n => n % 2 == 0);

    foreach (var num in result)
    {
        Console.WriteLine(num);
    }
    return null;
}
public string EnumerableResult()
{
    int[] numbers = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
    IEnumerable<int> numberEnumerable = numbers;
    var result = numberEnumerable.Where(n => n % 2 == 0);
    foreach (int num in result)
    {
        Console.WriteLine(num);
    }
    return null;
}

Here, we create two methods, QueryableResult() and EnumerableResult(). Both methods contain an integer array and then find even numbers from the array. In one of the methods we are using IQueryable and in the second one we use IEnumerable.

So far, we are now ready with our code that performs similar tasks. Now, we will use the BenchmarkDotNet library to track the performance.

Benchmark Attribute

Benchmark is a class that comes under the BenchmarkDotNet library. We can use it as an attribute to a method.

The basic syntax to use Benchmark attribute:

[Benchmark]
public void IQueryableBenchmark()
{

}

Next, we will create a class and add two methods for benchmarking:

[MemoryDiagnoser]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]
[RankColumn]
public class BenchmarkLogicLookup
{
    BenchmarkLogic obj = new BenchmarkLogic();
    [Benchmark]
    public void IQueryableBenchmark()
    {
        obj.QueryableResult();
    }
    [Benchmark]
    public void IEnumerableBenchmark()
    {
        obj.EnumerableResult();
    }
}

In the code snippet, we have a class BenchmarkLogicLookup and it contains two methods which are IQueryableBenchmark() and IEnumerableBenchmark(). We are creating the object of the BenchmarkLogic class to access the methods. Then, we decorated both methods with Benchmark attributes.

BenchmarkRunner

Finally, we will use the BenchmarkRunner class to run the benchmarks of the class BenchmarkLogicLookup.

BenchmarkRunner.Run<BenchmarkLogicLookup>();

Let’s run the application and see the output:

| Method               | Mean     | Error    | StdDev   | Rank | Gen0   | Gen1   | Allocated |
|--------------------- |---------:|---------:|---------:|-----:|-------:|-------:|----------:|
| IEnumerableBenchmark | 582.3 us | 11.31 us | 12.57 us |    1 |      - |      - |     112 B |
| IQueryableBenchmark  | 611.0 us | 12.08 us | 13.92 us |    2 | 4.8828 | 3.9063 |   11061 B |

Here, we see the execution Meantime and allocated memory for a single operation.

Note: This article is not about IQueryable and IEnumerable. The article explores the BenchmarkDotNet library to track the performance of similar tasks done in different ways.

We can explore IQueryable and IEnumerable in this article – IQueryable Vs IEnumerable in C#

Benchmark Output

Let’s look at the output values of Benchmark.

Method: Displays the operation included for Benchmark.
Mean: Mean is the average time of all measures.
Error: The error column in BenchmarkDotNet is a metric that helps you understand the differences between each iteration. In other words, a smaller error means the measurements were more consistent and closer to each other.
StdDev: It is the standard deviation. It describes the actual variation from the mean value.
Rank: It is the relative position.
Allocated: It shows how much memory is allocated for a single operation.

Important points about BenchmarkDotNet

So far, we are now able to understand BenchmarkDotNet and its appropriate uses. Now, we need to note one important point regarding the BenchmarkDotNet library.

The BenchmarkDotNet only works in Release mode. In case, we run this in Debug mode, it will give the below result and in this case, the benchmark will not proceed.

// Validating benchmarks:
//    * Assembly BenchmarkDemo which defines benchmarks is non-optimized
Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.
If you want to debug the benchmarks, please see https://benchmarkdotnet.org/articles/guides/troubleshooting.html#debugging-benchmarks.

Conclusion

So, after reading this article, we are now able to understand the basic concept of the BenchmarkDotNet library. Thus, we can analyze the execution of the same work which is done in different ways.