Crestron Home - Diagnostics 101

Crestron Home has a lot going for it - a clean, modern, consistent user interface - a good spread of "in the box" functionality - configured, rather than programmed - but any system that makes things easy in one direction has to make some compromises in others.

One of the biggest deficiencies of Crestron Home is the diagnostic support.  Of course, in the ideal world of "it just works" we shouldn't need any ninja diagnostic skills because... everything is awesome!  But this is real life, not the Lego movie (in case that makes no sense, "Everything is Awesome" is an irritatingly catchy song from the Lego movie).  Things do not "just work".  The reasons for things not working are very hard to pin down and if we just think about device drivers, it could be anything from bad driver code, poor documentation, a use case that isn't supported, driver configuration, device configuration, installation issues, device firmware, faulty equipment, etc.  Hopefully, you see the challenge that Crestron Home and 3rd party driver developers face in trying to make things "just work".

We write drivers, and while we do try to cover the general problems, I admit we tend to focus on the working stuff more than the things that may not work - this is one of the challenges in human nature for software development.  If I look back over the driver code we've written since we started with Crestron Home driver development 5 years ago I can see improvements we've made both in doing things in a more reliable way, and catching common issues to help point out problems should they occur, but - as I pointed to above - it's a bit of a minefield.

If you've come from a background in custom Crestron installation, you'll be familiar with many of the Crestron tools that help with fault diagnostics.  Sadly though, there appears to be a bias towards light touch diagnostics (you might call it de-skilling) in Crestron Home which, in my opinion, doesn't help the product, dealers or their customers.  The pinnacle of Crestron Home "EZ" diagnostics is the truly awful "Logs" page (the article image).  You can't search it.  You can't export it.  It's paginated by the hour, then further paginated into scrolling lists.  Oh - and it's not the traditional error log that has been the standard for Crestron for 25+ years.

I can't really put it any other way - it's utterly useless.  When we ask for logs to be supplied in support tickets, we will sometimes get mobile phone photos of a laptop screen of this page.  Honestly.  I don't blame the dealers here - they've been given a tool which is about as much use as a chocolate teapot, so what else are they to do? (There is a tool calling "The Info Gathering Tool" which is excellent, but also a sledgehammer for many simple issues)

The answer I believe, for many reasons, is to learn how to use the existing tools and resources that Crestron have always provided and seasoned integrators have become comfortable with.  The most versatile being the processor's text console.  Some people may not like the idea of learning console commands and have grown up in a cotton-wool lined iOS or Android cell.  It's time to grow up, learn how to support your customers properly and carry a laptop with you to jobs sites (and the power block).  You don't have to use Toolbox for this, though the Console tool in there is perfectly up to the job.

Now I've gotten that off my chest, here's a quick run down of some incredibly useful commands for diagnosing Crestron Home problems, or at least providing more detail for a support ticket so we can provide a quick resolution.

Fire up a console application (Toolbox Console, Mobaxterm, Putty, etc.) and connect to your processor over SSH.  Commands are shown in MONOCAPS.

1. The ERR command
   This is by far the best place to start.  The processor's error log (NOT the Crestron Home internal log... why did you make them different Crestron!?) shows a wealth of info, including our licence verification messages.  For every case where the system works for an hour and then stops, it's ALWAYS the licence key.  The info in the error log allows us to spot all of these cases:
   1. The key is fine - the problem is elsewhere (nearly never!)
   2. The key has been entered incorrectly - the CH UI isn't great for typing in keys, and sometimes cut and paste leaves extra characters, or people enter the key with the '['...']' which should be omitted.
   3. The MAC address is incorrect - sometimes, people order a key using the MAC address of the device, rather than the Crestron Home processor.  We can check the order against the error log entry, spot the problem and send you a new key without any further conversation.
2. Driver Logging
   This is more involved, but if the licence key checks out, and there's still a problem, the data gathered this way could identify a configuration issue or help us improve a driver in situations where we've made an error, or a specific model doesn't conform to the manufacturers standard protocols (it happens way more often than you'd imagine... WYRESTORM).  To enable logging on a specific driver these are the steps.
   1. ENABLEPROGRAMCMD - type this and press enter.  Nothing will happen.  Nothing obvious anyway.  However, this will put Crestron Home into a special diagnostic mode and open up additional, really useful, but not publicly documented commands.
   2. CCDINFO - type this and press enter.  If you've successfully enabled-program-commands earlier, this will spew out a list of all of the drivers that Crestron Home currently has in its head.
      This provides useful info in a number of areas.
      
      
      1. It shows when you have "phantom" drivers - the ones that appear on the Crestron Home UI but not in the config app.  Sadly, the only solution here is to roll back your CH system to before the problem occurred - whenever that may have been - or perform open heart surgery on your system.  Some developers have tools to de-risk fat-finger mistakes, but it's still a strategy with some risk (and impossible unless you know where to cut).
      2. It shows the assembly version of the driver - that is, the version of the executing code, rather than the version that the Crestron Home UI reports, which is actually taken from a datafile, and can easily mis-report the version of the driver you think you're running!  The answer here - remove every track of the driver... everywhere... and then perform a GRACEFULREBOOT.  This ensures your configs are saved before rebooting, which is one way to avoid the "phantom" drivers above.
      3. It gives you the driver ID - and this is essential as we dig deeper.  An example if shown below, and I've highlighted the driver ID.
         --- Driver Info for Driver ID 7---
Driver: Ultamation - Word of the Day
Type: Miscellaneous
GUID: 86ef16f2-b822-44a6-9fbd-bd6a4c71b26f
SDK Version: 3.00.000
Version: 1.00.006 (11/30/2022 00:00:00)
Using RADCommon, Version=16.0.60.0, Culture=neutral, PublicKeyToken=null
Using RADCommon, Version=16.0.60.0, Culture=neutral, PublicKeyToken=null
--- Driver Info End ---
   3. CCDLOGGING <id> ON - this turns driver logging on for that specific ID (the one you found in CCDINFO just now) - don't use the '<' '>' by the way, just the number.
      Now you will start to see (in the text console) messages being processed by the driver which can give you (or us) lots of useful info.
      Remember: Always turn logging OFF when you've finished your diagnostics session - too much output can load up the Crestron Home processor.
   4. CCDTXDEBUG <id> ON, CCDRXDEBUG <id> ON - two different commands, separated by a comma here.  These two allow you to see the actual data being sent to and received from the device.  The data is presented in hexadecimal (base 16) which can make reading it a little tedious (there are lots of online tools that will convert it to ASCII, if that's what the device uses) but this is incredibly useful to check if you actually have communication with the end device (do you get RX in response to your TX?) or perhaps the received data (the RX) may say something about why the commands aren't working - e.g. "BAD USER CODE" or "COMMAND NOT SUPPORTED" etc.
   5. CCDSTATES <id> - this is less useful for most users, but can be helpful in understanding the driver's internal state, so we may run this command in trying to understand why a driver isn't connecting to the device or not sending commands when it is connected.

Once a driver licence key has been checked, the next best thing for us to help diagnose any problem is the output from CCDLOGGING, CCDTXDEBUG and CCDRXDEBUG - if you're able to provide that (which can be copied and pasted straight out of the console).  Even large amounts of logging data can be processed fairly quickly, and allow us to spot driver issues (e.g. an unexpected delimiter).

If you can become familiar with these fairly simple commands, we will be able to help you far more effectively and you will spend less time on-site.

We have a diamond!

Creston Masters 2022 sees Oliver Hall reaches Diamond Crestron Master Programmer status

Last week was a busy one over in Orlando with the return of Crestron Masters in person for the first time in three years! Oliver (our MD) flew out on Friday to help with the set-up and to prepare for the workshop that he was delivering on Test Driven Development alongside Toine C. Leerentveld, Yuri Staal, Christopher Tatton and Andrew Mossberg.  The goal of the session was to present a way of thinking about software development from an angle that many programmers may not have considered before, in order to improve software design and quality.  It included a hands-on C# workshop, applying the principles of “Test Driven Development” to a typical (if a little contrived!) AV challenge that directly applies to the Crestron environment. In addition to focusing on C#, there was also discussion around how the techniques could be applied to SIMPL Windows and SIMPL+. The final part of the workshop covered tools and techniques for automated end-to-end testing through the actual User Interface.  If you missed the session you can find out more information from Crestron’s TrueBlue Support – Answer ID 2015.

After an exciting start, Oliver was then able to join in with other sessions and particularly enjoyed hearing Crestron CEO, Dan Feldstein, giving his thoughts and aspirations for where Crestron is now heading.  The week ended on a high with Oliver, alongside  Jeremy Weatherford, Dean Bayley, Brian Allen, Richard Noullet, and Johnny Wang receiving their CMP Diamond certification – congratulations!

Finally, an enormous thank you to Richard Sassoon and the team of behind-the-scenes helpers who made Crestron Masters 2022 such a huge success!  We’ll see you (a little closer to home) in Madrid in May 2023.

ISE Crestron Home Tip of the Day #1 - Apple TV App Launching

Adding app launching was a huge step forward for our Apple TV IP driver, but you still needed to switch away from the source, open the app launcher tile and then switch back.

With this top tip, those days are over.

You'll need our Apple TV IP driver before you start: Click here for Apple TV

With a few simple steps you can now configure a room display to present commands.  This is normally used for selection of aspect ratios and so on, but you can also add Quick Actions to this list - and they can do ANYTHING.  So while this top tip is about Apple TV app launching, you could trigger a movie lighting scene with our Philips Hue or DMX controller or open the main gate with our Wildcard drivers, all without leaving the source page.

Step 1: Create some Quick Actions

Step 2: In the Display Settings, go to Commands and Enable the Quick Actions

Step 3: Select the Apple TV Source and - hey presto - you have App Launching under the display menu!  You can even re-order them so your favourite is at the top.

Follow us on Facebook, Twitter, Instagram and LinkedIn (@ultamation) for tip #2 and #3 during ISE 2022!  

The Ultamation Help Centre

We're very proud of the reputation Ultamation has for exemplary customer support.  As they say, talk is cheap, and it's not the fact that you say you provide first-rate support, it's whether customers get a quick response to their queries and how far you go to resolve the problem.  We work very hard, often responding to customers out-of-hours and walking them through configurations or diagnostics - usually without charge.

While we firmly hold that the human element to customer support is crucially important, it's important to make information as accessible as possible as this means that potential issues can be avoided without engaging with support at all at times and we can provide thorough responses to common problems without rewriting the same information out for the 10th time, simply by linking relevant, already documented, information.

To that end, we have introduced the Ultamation Help Centre which is a repository of documents that we will continuously update with useful info.  The information isn't public, and you will need to register on the portal to access the documents, but this is a property of the platform we're using and we do not use any personal information gleaned from creating an account.  Sadly, this account is not linked to existing Ultamation shop accounts.

We've also moved over to a new ticketing system for support tickets (generated by emailing This email address is being protected from spambots. You need JavaScript enabled to view it.) and the new system gives registered users a way to keep track of their cases.  If you email This email address is being protected from spambots. You need JavaScript enabled to view it., you will automatically have an account created by the platform to that you can track your case.

To access the Ultamation Help Centre, please follow the link below.

Ultamation Help Centre

As an incentive, we've already started writing up the new Apple TV control FAQ so if you're quick off the mark, you can get access to otherwise unreleased information about the new CompanionLink drivers for Crestron and Crestron Home.

We're also publishing recipes for the Crestron Home wildcard drivers which provide simple "hacks" for giving your clients capabilities that can't by offered by your competition.

Wildcard Recipe: Sending Commands to Display Endpoints

Problem

Your Crestron Home project is using a 3rd party AV switcher but you haven’t been able to use the full capability of sending commands to the displays to turn them on and off.

Solution

The Ultamation Wildcard modules can be used to send these commands to the AV switcher by integrating the commands into hidden quick actions which are triggered when a media zone is powered on or off.

1. Add the appropriate Ultamation Wildcard driver to Crestron Home - this will depend on the capabilities and communications model of your chosen AV switcher.

2. Configure your chosen commands for buttons or discrete commands (under Actions/Sequences) using the information in the table, and examples below. If you work out more commands, please let us know and we’ll add them to the table!

3. You’re in the 1-hour trial period window - give your new integration a test drive!

4. Once you’re happy everything is working, purchase a licence key for your processor for one of the wildcard drivers here… Home

5. Add the licence key to your Crestron Home configuration (under “Installer Settings”)

Command Table

Examples:

The following examples show you how format the the commands in the Installer Settings of the module.

The first element - “Label” - can be whatever you want to have displayed on the button. The other elements must be formatted exactly as shown.

Binary MoIP, using the TCP Wildcard, turn on TV on output 3

Playroom TV On|!CEC3,1\x0A

Binary MoIP, using the TCP Wildcard, turn off TV on output 3

Playroom TV Off|!CEC3,0\x0A

HDANYWHERE, using the HTTP POST Wildcard, turn off TV on output HDBT 'a'

Lounge TV Off|api/command/cecpass/a/1/|{"logicaladdress":"EF","command":"36","arguments":""}

Manufacturers vary greatly in their support for, and implementation of, CEC commands.

Kudos to Isaac Rosario of Voyager Home Systems Voyager Home Systems | Oklahoma City Metro Smart Home Installations for suggesting this really innovative integration solution and providing Binary MoIP data.