Crestron Home - Diagnostics 101

Crestron Home has a lot going for it - a clean, modern, consistent user interface - a good spread of "in the box" functionality - configured, rather than programmed - but any system that makes things easy in one direction has to make some compromises in others.

One of the biggest deficiencies of Crestron Home is the diagnostic support.  Of course, in the ideal world of "it just works" we shouldn't need any ninja diagnostic skills because... everything is awesome!  But this is real life, not the Lego movie (in case that makes no sense, "Everything is Awesome" is an irritatingly catchy song from the Lego movie).  Things do not "just work".  The reasons for things not working are very hard to pin down and if we just think about device drivers, it could be anything from bad driver code, poor documentation, a use case that isn't supported, driver configuration, device configuration, installation issues, device firmware, faulty equipment, etc.  Hopefully, you see the challenge that Crestron Home and 3rd party driver developers face in trying to make things "just work".

We write drivers, and while we do try to cover the general problems, I admit we tend to focus on the working stuff more than the things that may not work - this is one of the challenges in human nature for software development.  If I look back over the driver code we've written since we started with Crestron Home driver development 5 years ago I can see improvements we've made both in doing things in a more reliable way, and catching common issues to help point out problems should they occur, but - as I pointed to above - it's a bit of a minefield.

If you've come from a background in custom Crestron installation, you'll be familiar with many of the Crestron tools that help with fault diagnostics.  Sadly though, there appears to be a bias towards light touch diagnostics (you might call it de-skilling) in Crestron Home which, in my opinion, doesn't help the product, dealers or their customers.  The pinnacle of Crestron Home "EZ" diagnostics is the truly awful "Logs" page (the article image).  You can't search it.  You can't export it.  It's paginated by the hour, then further paginated into scrolling lists.  Oh - and it's not the traditional error log that has been the standard for Crestron for 25+ years.

I can't really put it any other way - it's utterly useless.  When we ask for logs to be supplied in support tickets, we will sometimes get mobile phone photos of a laptop screen of this page.  Honestly.  I don't blame the dealers here - they've been given a tool which is about as much use as a chocolate teapot, so what else are they to do? (There is a tool calling "The Info Gathering Tool" which is excellent, but also a sledgehammer for many simple issues)

The answer I believe, for many reasons, is to learn how to use the existing tools and resources that Crestron have always provided and seasoned integrators have become comfortable with.  The most versatile being the processor's text console.  Some people may not like the idea of learning console commands and have grown up in a cotton-wool lined iOS or Android cell.  It's time to grow up, learn how to support your customers properly and carry a laptop with you to jobs sites (and the power block).  You don't have to use Toolbox for this, though the Console tool in there is perfectly up to the job.

Now I've gotten that off my chest, here's a quick run down of some incredibly useful commands for diagnosing Crestron Home problems, or at least providing more detail for a support ticket so we can provide a quick resolution.

Fire up a console application (Toolbox Console, Mobaxterm, Putty, etc.) and connect to your processor over SSH.  Commands are shown in MONOCAPS.

  1. The ERR command
    This is by far the best place to start.  The processor's error log (NOT the Crestron Home internal log... why did you make them different Crestron!?) shows a wealth of info, including our licence verification messages.  For every case where the system works for an hour and then stops, it's ALWAYS the licence key.  The info in the error log allows us to spot all of these cases:
    1. The key is fine - the problem is elsewhere (nearly never!)
    2. The key has been entered incorrectly - the CH UI isn't great for typing in keys, and sometimes cut and paste leaves extra characters, or people enter the key with the '['...']' which should be omitted.
    3. The MAC address is incorrect - sometimes, people order a key using the MAC address of the device, rather than the Crestron Home processor.  We can check the order against the error log entry, spot the problem and send you a new key without any further conversation.
  2. Driver Logging
    This is more involved, but if the licence key checks out, and there's still a problem, the data gathered this way could identify a configuration issue or help us improve a driver in situations where we've made an error, or a specific model doesn't conform to the manufacturers standard protocols (it happens way more often than you'd imagine... WYRESTORM).  To enable logging on a specific driver these are the steps.
    1. ENABLEPROGRAMCMD - type this and press enter.  Nothing will happen.  Nothing obvious anyway.  However, this will put Crestron Home into a special diagnostic mode and open up additional, really useful, but not publicly documented commands.
    2. CCDINFO - type this and press enter.  If you've successfully enabled-program-commands earlier, this will spew out a list of all of the drivers that Crestron Home currently has in its head.
      This provides useful info in a number of areas.

      1. It shows when you have "phantom" drivers - the ones that appear on the Crestron Home UI but not in the config app.  Sadly, the only solution here is to roll back your CH system to before the problem occurred - whenever that may have been - or perform open heart surgery on your system.  Some developers have tools to de-risk fat-finger mistakes, but it's still a strategy with some risk (and impossible unless you know where to cut).
      2. It shows the assembly version of the driver - that is, the version of the executing code, rather than the version that the Crestron Home UI reports, which is actually taken from a datafile, and can easily mis-report the version of the driver you think you're running!  The answer here - remove every track of the driver... everywhere... and then perform a GRACEFULREBOOT.  This ensures your configs are saved before rebooting, which is one way to avoid the "phantom" drivers above.
      3. It gives you the driver ID - and this is essential as we dig deeper.  An example if shown below, and I've highlighted the driver ID.
        --- Driver Info for Driver ID 7---
        Driver: Ultamation - Word of the Day
        Type: Miscellaneous
        GUID: 86ef16f2-b822-44a6-9fbd-bd6a4c71b26f
        SDK Version: 3.00.000
        Version: 1.00.006 (11/30/2022 00:00:00)
        Using RADCommon, Version=, Culture=neutral, PublicKeyToken=null
        Using RADCommon, Version=, Culture=neutral, PublicKeyToken=null
        --- Driver Info End ---
    3. CCDLOGGING <id> ON - this turns driver logging on for that specific ID (the one you found in CCDINFO just now) - don't use the '<' '>' by the way, just the number.
      Now you will start to see (in the text console) messages being processed by the driver which can give you (or us) lots of useful info.
      Remember: Always turn logging OFF when you've finished your diagnostics session - too much output can load up the Crestron Home processor.
    4. CCDTXDEBUG <id> ON, CCDRXDEBUG <id> ON - two different commands, separated by a comma here.  These two allow you to see the actual data being sent to and received from the device.  The data is presented in hexadecimal (base 16) which can make reading it a little tedious (there are lots of online tools that will convert it to ASCII, if that's what the device uses) but this is incredibly useful to check if you actually have communication with the end device (do you get RX in response to your TX?) or perhaps the received data (the RX) may say something about why the commands aren't working - e.g. "BAD USER CODE" or "COMMAND NOT SUPPORTED" etc.
    5. CCDSTATES <id> - this is less useful for most users, but can be helpful in understanding the driver's internal state, so we may run this command in trying to understand why a driver isn't connecting to the device or not sending commands when it is connected.

Once a driver licence key has been checked, the next best thing for us to help diagnose any problem is the output from CCDLOGGING, CCDTXDEBUG and CCDRXDEBUG - if you're able to provide that (which can be copied and pasted straight out of the console).  Even large amounts of logging data can be processed fairly quickly, and allow us to spot driver issues (e.g. an unexpected delimiter).

If you can become familiar with these fairly simple commands, we will be able to help you far more effectively and you will spend less time on-site.