BaselineCustomAnalyzer raise warning under specific conditions

[filzrev]

I’m trying to write benchmarks that measure performance difference between .NET version.

But when using following settings BaselineCustomAnalyzer raise warnings and Ratio/Ratio column show ?.

1. Set baseline on Job (that running benchmark with .NET 8).
2. Set baseline on benchmark method (It's used when running benchmark with different config)
3. Enable grouping with .AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)

Benchmark results

BaselineCustomAnalyzer
Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline or benchmark could not be found, or the baseline value is too close to zero.

| Method                       | Toolchain | Mean     | Error     | StdDev   | Ratio | RatioSD |
|----------------------------- |---------- |---------:|----------:|---------:|------:|--------:|
| OrderBy_EnumerableDataSource | .NET 8.0  | 153.3 ms |  10.27 ms |  0.56 ms |  1.00 |    0.00 |
| OrderBy_EnumerableDataSource | .NET 9.0  | 149.0 ms | 233.89 ms | 12.82 ms |  0.97 |    0.07 |
| OrderBy_EnumerableDataSource | .NET 10.0 | 140.8 ms | 216.04 ms | 11.84 ms |  0.92 |    0.07 |
|                              |           |          |           |          |       |         |
| OrderBy_ArrayDataSource      | .NET 8.0  | 164.7 ms | 136.52 ms |  7.48 ms |     ? |       ? |
| OrderBy_ArrayDataSource      | .NET 9.0  | 147.1 ms | 223.11 ms | 12.23 ms |     ? |       ? |
| OrderBy_ArrayDataSource      | .NET 10.0 | 148.8 ms |  88.03 ms |  4.83 ms |     ? |       ? |

TestCode

    internal class Program
    {
        static void Main(string[] args)
        {
            var config = DefaultConfig.Instance
                                      .AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)
                                      .WithOrderer(new DefaultOrderer(jobOrderPolicy: JobOrderPolicy.Numeric))
                                      .WithOptions(ConfigOptions.DisableOptimizationsValidator);

            config = config.AddJob(
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp90),
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp10_0));

            var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                                             .Run(args, config)
                                             .ToArray();

            var config2 = summaries[0].BenchmarksCases[0].Config;
        }
    }

    public class Benchmarks
    {
        private readonly Consumer Consumer = new();

        public int N { get; set; } = 1_000_000;

        private readonly IEnumerable<int> EnumerableDataSource;
        private readonly int[] ArrayDataSource;

        public Benchmarks()
        {
            var rand = new Random(0);
            var source = Enumerable.Range(1, N).Select(x => rand.Next());

            EnumerableDataSource = source;
            ArrayDataSource = EnumerableDataSource.ToArray();
        }

        [Benchmark(Baseline = true)]
        public void OrderBy_EnumerableDataSource()
        {
            foreach (var i in EnumerableDataSource.OrderBy(x => x))
            {
                Consumer.Consume(i);
            }
        }

        [Benchmark]
        public void OrderBy_ArrayDataSource()
        {
            foreach (var i in ArrayDataSource.OrderBy(x => x))
            {
                Consumer.Consume(i);
            }
        }
    }

Analysis

It seems be resolved by modifying following logics.

BenchmarkDotNet/src/BenchmarkDotNet/Reports/BaseliningStrategy.cs

Lines 35 to 45 in 94303f4

	public bool IsBaseline(BenchmarkCase benchmark)
	{
	if (!useDescriptors && !useJobs)
	return false;
	bool result = true;
	if (useDescriptors)
	result &= benchmark.Descriptor.Baseline;
	if (useJobs)
	result &= benchmark.Job.Meta.Baseline;
	return result;
	}

By adding following code that enforce to use Job's baseline setting.

if(useDescriptors && useJobs)
    benchmark.Job.Meta.Baseline

I've not created a PR because I'm not sure the side effects of this changes.