Tests are not discovered when using attribute inherited from TheoryData

[NikiforovAll]

The tests are not discovered when I use the inherited attribute - DebugOnlyTheoryAttribute : TheoryAttribute.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class DebugOnlyTheoryAttribute : TheoryAttribute
{
    public DebugOnlyTheoryAttribute(string comment = DebugOnlyConstants.DefaultComment)
    {
        this.SkipIfNotDebugging(comment);
    }
}

[DebugOnlyTheory]
[InlineData("string1")]
[InlineData("string2")]
void  Method1() {}

When I change DebugOnlyTheory to just Theory it works again.

❯ dotnet --version
9.0.100-preview.7.24407.12

[bradwilson]

Is this v2 or v3? What package version(s) are you using?

[NikiforovAll]

xunit 2.9.0, xunit.runner.visualstudio 2.8.2, Microsoft.NET.Test.Sdk 17.11.1

[bradwilson]

I not 100% sure I can reproduce this, in part because I don't know what this.SkipIfNotDebugging(comment); does.

Making a guess as to it throwing some kind of exception, here is my repro:

using System;
using Xunit;

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class DebugOnlyTheoryAttribute : TheoryAttribute
{
    public DebugOnlyTheoryAttribute(string comment = "meh")
    {
#if !DEBUG
        throw new ArgumentException(comment);
#endif
    }
}

public class TestClass
{
    [DebugOnlyTheory]
    [InlineData("string1")]
    [InlineData("string2")]
    public void Method1(string _) { }
}

This is what I see when running in Debug:

$ dotnet test
  Determining projects to restore...
  All projects are up-to-date for restore.
  empty -> C:\Dev\repro\empty.cs\bin\Debug\net8.0\empty.dll
Test run for C:\Dev\repro\empty.cs\bin\Debug\net8.0\empty.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     2, Skipped:     0, Total:     2, Duration: 2 ms - empty.dll (net8.0)

This is what I see when running in Release:

$ dotnet test --configuration Release
  Determining projects to restore...
  All projects are up-to-date for restore.
  empty -> C:\Dev\repro\empty.cs\bin\Release\net8.0\empty.dll
Test run for C:\Dev\repro\empty.cs\bin\Release\net8.0\empty.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.09]     TestClass.Method1 [FAIL]
  Failed TestClass.Method1 [1 ms]
  Error Message:
   System.InvalidOperationException : Exception during discovery:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.ArgumentException: meh
   at DebugOnlyTheoryAttribute..ctor(String comment) in C:\Dev\repro\empty.cs\UnitTest.cs:line 10
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Xunit.Sdk.ReflectionAttributeInfo.Instantiate(CustomAttributeData attributeData) in /_/src/xunit.execution/Sdk/Reflection/ReflectionAttributeInfo.cs:line 145
   at Xunit.Sdk.ReflectionAttributeInfo..ctor(CustomAttributeData attribute) in /_/src/xunit.execution/Sdk/Reflection/ReflectionAttributeInfo.cs:line 26
   at Xunit.Sdk.ReflectionMethodInfo.GetCustomAttributes(MethodInfo method, Type attributeType, AttributeUsageAttribute attributeUsage) in /_/src/xunit.execution/Sdk/Reflection/ReflectionMethodInfo.cs:line 108
   at Xunit.Sdk.ReflectionMethodInfo.GetCustomAttributes(MethodInfo method, String assemblyQualifiedAttributeTypeName) in /_/src/xunit.execution/Sdk/Reflection/ReflectionMethodInfo.cs:line 95
   at Xunit.Sdk.ReflectionMethodInfo.GetCustomAttributes(String assemblyQualifiedAttributeTypeName) in /_/src/xunit.execution/Sdk/Reflection/ReflectionMethodInfo.cs:line 88
   at ReflectionAbstractionExtensions.GetCustomAttributes(IMethodInfo methodInfo, Type attributeType) in /_/src/xunit.execution/Extensions/ReflectionAbstractionExtensions.cs:line 147
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForMethod(ITestMethod testMethod, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 84
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForType(ITestClass testClass, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 135


Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: < 1 ms - empty.dll (net8.0)

(This failure is exactly what I'd expected, given that throwing an exception in the test class constructor should surface like this.)

Can you provide a repro project & steps?