Incorrect behavior when running unit tests for multiple runtimes (.NET and Silverlight)

May 29, 2014 at 1:22 PM
Hello, I submitted the following issue with JetBrains but figured I'd check here to see if there was any possible workaround that could be done on the AgUnit side. Basically, we're trying to run an existing .NET unit test suite on the Silverlight runtime using AgUnit. We'd like to link the existing unit test files to new SL projects but when we run them in VS for one platform, it overwrites the results for the other platform. I was able to repro this with the DummyTests solution from the AgUnit github repo (see http://imgur.com/L0DVTYK,OVJxMIp for screenshots).

Any ideas?

(Original ticket submitted to JetBrains)
We develop a cross-platform (WPF and Silverlight) product and have a suite of unit tests that work great when run under the .NET runtime using the R# unit test runner.

We've recently added StatLight and AgUnit to the mix in order to allow us to run our tests using the Silverlight runtime as there are some important behavioral differences between the platforms.

Instead of duplicating all the unit test files, we're using the standard MS practice of creating a new Silverlight project and linking the source files from our existing .NET unit test assemblies.

When attempting to run the entire suite of unit tests for both platforms, it seems there is a problem when R# encounters two tests with the same namespace/class/method names but inside different assemblies. Specifically, when the tests are run for one platform, they overwrite the results in the results view for /both/ platforms. It's almost like R# is ignoring the assembly or project when making each test "unique" in the results.

Specifically, what we have boils down to two assemblies (one .NET, one Silverlight) that contain the same exact tests (same namespace, same class names, same method names). If I tell R# to run the tests for both assemblies and have the test results view grouped by "project", I can see the tests under the proper projects but the results for one overwrite the other so we potentially have false positives (i.e. something works under .NET but not under Silverlight but the results view shows passed for all of them). If I then run just the one assembly that we know is failing (Silverlight), R# shows the /other/ projects tests are running as well and fails those as well.

We have a workaround with using preprocessor statements to change the namespace of the tests when they are compiled under Silverlight but would rather have better support for this scenario built into R#. I can provide a sample project if the description above isn't clear.