Mocking Crestron

Introduction

Far from what you might think from the title, this article is not about poking fun at the world’s leading automation system. Far from it: Crestron is the only platform that has successfully made the transition from proprietary development environments and niche, specialised programming, to mainstream software development practices.  This article covers one aspect of that transition by describing one approach to “unit testing” – the practice of repeatedly testing small elements of a larger system to ensure they satisfy their design parameters – using “mock” objects.  This gives confidence that your code, when employed in a larger system, will yield the expected behaviour.

Why bother?

Unit testing sits in a broader landscape of modern software development practices such as Test Driven Development (TDD) and Continuous Integration (CI), and it was through our exploration into CI (in this case, not to be confused with our normal understanding of the CI acronym) that we developed a solution to run automated unit tests regularly.  There’s obviously an overhead to this – writing tests is just like writing any other code, and it’s important to write the right tests too, plus to make it worthwhile, it’s useful to automate the whole process so that running the tests doesn’t become a burden.  Once you invest the time in setting up an automated build process, the benefits of having your code automatically tested each time you commit a change become apparent very quickly.  Expand that to working in a team with multiple developers committing changes to the same code base, and there’s really no argument.

Challenges

Before we outline the solution we have built, it’s important to understand there are some challenges to testing for the Crestron environment, and this is where the concept of “mocking” comes in.

Crestron provide a rich set of classes for interacting with the control system hardware.  Running unit tests on actual hardware is doable (as demonstrated by Neil Colvin’s port of the NUnit framework) and this certainly provides the most reliable indication of correct behaviour, but integrating that into an automated build process presents other challenges.  We have chosen to take a different approach where our focus is on testing our own code, rather than the Crestron libraries, and this allows us to run the suite of tests completely within the CI environment using the wealth of standard, and freely available, tools available.

Normally, the tests would fail to run as we can’t run code that uses Crestron’s assemblies (the term used for C# libraries) on a standard Windows machine.  To resolve this, we implement a number of classes to mimic Crestron’s native classes, and compile our code against our “mock” assemblies rather Crestron’s.  Mocking only requires covering the Crestron classes and their dependencies that your code makes use of.  Ideally, this is nothing more than a mechanism to enable the tests to run; the mocked classes are largely empty, or provide extremely simple implementations.  The point being, you want to test your code, not Crestron’s.

Solution

We’ll now outline our solution to unit-testing, continuous integration, and we’ll throw in automated documentation along the way.

First of all, we need to decide how to automate the build process in the first place.

Continuous Integration

There are many ways to approach this; they range from running scheduled, hand-made batch files to sophisticated commercial products.  Fortunately there are also a host of open source solutions in between that provide all the functionality we need.  This is where we, as Crestron developers, can benefit from the more general, mature, software engineering disciplines.

We have chosen a CI product called “Jenkins” which runs on many platforms, but importantly for us, is happy under Windows.  It’s worth mentioning that although the Crestron SIMPL# IDE dates back to 2008, the Windows OS you choose should be reasonably up-to-date.  Some of the tools we will use later on require the more recent OS versions (Windows 7 upwards) and even Crestron Toolbox is no longer supported on XP and Vista.

We don’t intend to provide detailed instructions on how to set up the build platform, as it’s pretty straight forward, so we’ll limit this summary to the key steps.

The image above shows a view of the Ultamation CI instance running Jenkins which is managed via a web browser, so the build machine can quite happily be a headless server or VM.  The view shows 4 jobs that we’ve configured under the framework “view”.

• The first job is a nightly build of the framework release.
  
  This job pulls any changes from source control (SVN in our case, but Jenkins supports Git, Mercurial, CVS etc.) and builds each assembly in the framework sequentially.  Each step simply runs MSBUILD on the same config file that Visual Studio generates when you’re working in the IDE, but you can override build properties along the way to tailor the build if necessary.  If any problems occur along the way, the build dashboard is updated, and Jenkins is configured to email the team to notify them of a problem.
  
  This covers simple build errors where someone has checked in non-compiling code, or forgotten to check in a dependency.
   
• We then have two jobs that generate documentation, and these jobs are triggered not on a schedule, but when the main Framework build completes successfully.  This automates the process of taking the C# XML documentation such as

/// <summary>
/// SIMPL+ accessor to obtain the number of items in
/// the current source list.
/// </summary>

• which can be generated (optionally) during the build.  We then call a tool called Sandcastle which generates MSDN style documentation.
  
  The final step is the resulting HTML documentation is FTP’d to our intranet server as the main reference for the framework code.

• The final job runs the unit tests, which is where the real meat is.  This job runs every 15 minutes (when the repository changes) so that we can be notified as quickly as possible if a committed change fails one of the tests.
  
  This time, we rebuild the framework, unit test and mocked assemblies, under an alternative “Test” configuration, which is explained further in the next section.  Once the build is complete, we then invoke NUnit console test runner to process the tests and generate a report.  Finally, the report is processed so that Jenkins can present the results on the dashboard, and – as before – we’re emailed in the event of any failures.

Unit Testing

Building projects that target the Crestron processor requires no special configuration, other than the usual pre-requisites of Visual Studio 2008 Professional SP1 and The Crestron SIMPL#/Pro project plugin.  You simply install the MSBUILD Jenkins Plugin, and configure it for the appropriate Microsoft.Net framework (v3.5).

When it comes to running code on a Windows machine however, which is required to exercise the testing framework, we hit the issue of linking with the Crestron S# assemblies which won’t run on our build platform.

One technique to work around this is to create “mock” objects which imitate the behaviour of Crestron’s classes.  In many cases, a true “mock”, which should reproduce the same behaviour as the true Crestron object, is unnecessary and a simple object which allows the test to run, but doesn’t provide any functionality itself can be substituted instead.  In this case, the nomenclature is then more correctly a “stub” object.

Fortunately, only the classes that are called in your S# code need to be mocked.

We then create an additional build configuration for each of the assemblies, which we have called ‘Test’, to sit alongside ‘Release’ and ‘Debug’.

To complete the configuration, we then need to instruct MSBUILD to use our mocked assemblies for testing, rather than the Crestron ones.  Unfortunately, Visual Studio doesn’t provide the tools to create conditional build projects directly, but it’s fairly easy to modify the project files by hand to add the necessary conditional mark-up, and this is honoured with Visual Studio when editing your unit tests, so you can see if your mocks are missing anything vital before committing to the repository.

The snippet below shows how the .csproj file can be modified to conditionally select the mock assemblies, instead of the SimplSharpHelperInterface during the test build.

The blue areas are the standard build script, and the red is the modification.

ExampleAssembly.csproj 

Run the unit-test job, and review the results in Jenkins’ dashboard.  All being well, you’ll have a lot of passes, and no fails.

Testing Methods with Internal Scope

Module code will often be assigned the internal access modifier.  Not only does this allow obfuscators to determine which classes and methods can be transformed, but it also hides unnecessary classes from the SIMPL+ API header files, which can be beneficial for a number of reasons.

This presents another problem – how can your unit tests, which should be in a different assembly so that they’re not included in the release code “see” your internal classes and methods?

Fortunately, Microsoft foresaw this scenario and have provided an assembly directive allow “friend” assemblies to see classes and methods that would otherwise be private to the assembly being tested.

Open the SIMPL#(Pro) AssemblyInfo.cs file and add the System.RunTime.CompilerServices namespace and the InternalsVisibileTo directive as shown below:

Ensure that the name of the friend assembly exactly matches that of your unit test assembly, and the unit test assembly isn’t signed, and the SIMPL# assembly internal classes will now be visible to your unit tests.

Next Steps

If you’re looking to implement a CI or unit testing process within your organisation and require assistance, Ultamation can provide consultancy services to take you through the entire process.  Contact Oliver Hall at This email address is being protected from spambots. You need JavaScript enabled to view it. or on +(44) 151 331 5210.