Crestron Home - Diagnostics 101

Crestron Home has a lot going for it - a clean, modern, consistent user interface - a good spread of "in the box" functionality - configured, rather than programmed - but any system that makes things easy in one direction has to make some compromises in others.

One of the biggest deficiencies of Crestron Home is the diagnostic support.  Of course, in the ideal world of "it just works" we shouldn't need any ninja diagnostic skills because... everything is awesome!  But this is real life, not the Lego movie (in case that makes no sense, "Everything is Awesome" is an irritatingly catchy song from the Lego movie).  Things do not "just work".  The reasons for things not working are very hard to pin down and if we just think about device drivers, it could be anything from bad driver code, poor documentation, a use case that isn't supported, driver configuration, device configuration, installation issues, device firmware, faulty equipment, etc.  Hopefully, you see the challenge that Crestron Home and 3rd party driver developers face in trying to make things "just work".

We write drivers, and while we do try to cover the general problems, I admit we tend to focus on the working stuff more than the things that may not work - this is one of the challenges in human nature for software development.  If I look back over the driver code we've written since we started with Crestron Home driver development 5 years ago I can see improvements we've made both in doing things in a more reliable way, and catching common issues to help point out problems should they occur, but - as I pointed to above - it's a bit of a minefield.

If you've come from a background in custom Crestron installation, you'll be familiar with many of the Crestron tools that help with fault diagnostics.  Sadly though, there appears to be a bias towards light touch diagnostics (you might call it de-skilling) in Crestron Home which, in my opinion, doesn't help the product, dealers or their customers.  The pinnacle of Crestron Home "EZ" diagnostics is the truly awful "Logs" page (the article image).  You can't search it.  You can't export it.  It's paginated by the hour, then further paginated into scrolling lists.  Oh - and it's not the traditional error log that has been the standard for Crestron for 25+ years.

I can't really put it any other way - it's utterly useless.  When we ask for logs to be supplied in support tickets, we will sometimes get mobile phone photos of a laptop screen of this page.  Honestly.  I don't blame the dealers here - they've been given a tool which is about as much use as a chocolate teapot, so what else are they to do? (There is a tool calling "The Info Gathering Tool" which is excellent, but also a sledgehammer for many simple issues)

The answer I believe, for many reasons, is to learn how to use the existing tools and resources that Crestron have always provided and seasoned integrators have become comfortable with.  The most versatile being the processor's text console.  Some people may not like the idea of learning console commands and have grown up in a cotton-wool lined iOS or Android cell.  It's time to grow up, learn how to support your customers properly and carry a laptop with you to jobs sites (and the power block).  You don't have to use Toolbox for this, though the Console tool in there is perfectly up to the job.

Now I've gotten that off my chest, here's a quick run down of some incredibly useful commands for diagnosing Crestron Home problems, or at least providing more detail for a support ticket so we can provide a quick resolution.

Fire up a console application (Toolbox Console, Mobaxterm, Putty, etc.) and connect to your processor over SSH.  Commands are shown in MONOCAPS.

  1. The ERR command
    This is by far the best place to start.  The processor's error log (NOT the Crestron Home internal log... why did you make them different Crestron!?) shows a wealth of info, including our licence verification messages.  For every case where the system works for an hour and then stops, it's ALWAYS the licence key.  The info in the error log allows us to spot all of these cases:
    1. The key is fine - the problem is elsewhere (nearly never!)
    2. The key has been entered incorrectly - the CH UI isn't great for typing in keys, and sometimes cut and paste leaves extra characters, or people enter the key with the '['...']' which should be omitted.
    3. The MAC address is incorrect - sometimes, people order a key using the MAC address of the device, rather than the Crestron Home processor.  We can check the order against the error log entry, spot the problem and send you a new key without any further conversation.
  2. Driver Logging
    This is more involved, but if the licence key checks out, and there's still a problem, the data gathered this way could identify a configuration issue or help us improve a driver in situations where we've made an error, or a specific model doesn't conform to the manufacturers standard protocols (it happens way more often than you'd imagine... WYRESTORM).  To enable logging on a specific driver these are the steps.
    1. ENABLEPROGRAMCMD - type this and press enter.  Nothing will happen.  Nothing obvious anyway.  However, this will put Crestron Home into a special diagnostic mode and open up additional, really useful, but not publicly documented commands.
    2. CCDINFO - type this and press enter.  If you've successfully enabled-program-commands earlier, this will spew out a list of all of the drivers that Crestron Home currently has in its head.
      This provides useful info in a number of areas.

      1. It shows when you have "phantom" drivers - the ones that appear on the Crestron Home UI but not in the config app.  Sadly, the only solution here is to roll back your CH system to before the problem occurred - whenever that may have been - or perform open heart surgery on your system.  Some developers have tools to de-risk fat-finger mistakes, but it's still a strategy with some risk (and impossible unless you know where to cut).
      2. It shows the assembly version of the driver - that is, the version of the executing code, rather than the version that the Crestron Home UI reports, which is actually taken from a datafile, and can easily mis-report the version of the driver you think you're running!  The answer here - remove every track of the driver... everywhere... and then perform a GRACEFULREBOOT.  This ensures your configs are saved before rebooting, which is one way to avoid the "phantom" drivers above.
      3. It gives you the driver ID - and this is essential as we dig deeper.  An example if shown below, and I've highlighted the driver ID.
        --- Driver Info for Driver ID 7---
        Driver: Ultamation - Word of the Day
        Type: Miscellaneous
        GUID: 86ef16f2-b822-44a6-9fbd-bd6a4c71b26f
        SDK Version: 3.00.000
        Version: 1.00.006 (11/30/2022 00:00:00)
        Using RADCommon, Version=16.0.60.0, Culture=neutral, PublicKeyToken=null
        Using RADCommon, Version=16.0.60.0, Culture=neutral, PublicKeyToken=null
        --- Driver Info End ---
    3. CCDLOGGING <id> ON - this turns driver logging on for that specific ID (the one you found in CCDINFO just now) - don't use the '<' '>' by the way, just the number.
      Now you will start to see (in the text console) messages being processed by the driver which can give you (or us) lots of useful info.
      Remember: Always turn logging OFF when you've finished your diagnostics session - too much output can load up the Crestron Home processor.
    4. CCDTXDEBUG <id> ON, CCDRXDEBUG <id> ON - two different commands, separated by a comma here.  These two allow you to see the actual data being sent to and received from the device.  The data is presented in hexadecimal (base 16) which can make reading it a little tedious (there are lots of online tools that will convert it to ASCII, if that's what the device uses) but this is incredibly useful to check if you actually have communication with the end device (do you get RX in response to your TX?) or perhaps the received data (the RX) may say something about why the commands aren't working - e.g. "BAD USER CODE" or "COMMAND NOT SUPPORTED" etc.
    5. CCDSTATE <id> - this is less useful for most users, but can be helpful in understanding the driver's internal state, so we may run this command in trying to understand why a driver isn't connecting to the device or not sending commands when it is connected.

Once a driver licence key has been checked, the next best thing for us to help diagnose any problem is the output from CCDLOGGING, CCDTXDEBUG and CCDRXDEBUG - if you're able to provide that (which can be copied and pasted straight out of the console).  Even large amounts of logging data can be processed fairly quickly, and allow us to spot driver issues (e.g. an unexpected delimiter).

If you can become familiar with these fairly simple commands, we will be able to help you far more effectively and you will spend less time on-site.

We have a diamond!

Creston Masters 2022 sees Oliver Hall reaches Diamond Crestron Master Programmer status

Last week was a busy one over in Orlando with the return of Crestron Masters in person for the first time in three years! Oliver (our MD) flew out on Friday to help with the set-up and to prepare for the workshop that he was delivering on Test Driven Development alongside Toine C. Leerentveld, Yuri Staal, Christopher Tatton and Andrew Mossberg.  The goal of the session was to present a way of thinking about software development from an angle that many programmers may not have considered before, in order to improve software design and quality.  It included a hands-on C# workshop, applying the principles of “Test Driven Development” to a typical (if a little contrived!) AV challenge that directly applies to the Crestron environment. In addition to focusing on C#, there was also discussion around how the techniques could be applied to SIMPL Windows and SIMPL+. The final part of the workshop covered tools and techniques for automated end-to-end testing through the actual User Interface.  If you missed the session you can find out more information from Crestron’s TrueBlue Support – Answer ID 2015.

After an exciting start, Oliver was then able to join in with other sessions and particularly enjoyed hearing Crestron CEO, Dan Feldstein, giving his thoughts and aspirations for where Crestron is now heading.  The week ended on a high with Oliver, alongside  Jeremy Weatherford, Dean Bayley, Brian Allen, Richard Noullet, and Johnny Wang receiving their CMP Diamond certification – congratulations!

Finally, an enormous thank you to Richard Sassoon and the team of behind-the-scenes helpers who made Crestron Masters 2022 such a huge success!  We’ll see you (a little closer to home) in Madrid in May 2023.

ISE Crestron Home Tip of the Day #1 - Apple TV App Launching

Adding app launching was a huge step forward for our Apple TV IP driver, but you still needed to switch away from the source, open the app launcher tile and then switch back.

With this top tip, those days are over.

You'll need our Apple TV IP driver before you start: Click here for Apple TV

With a few simple steps you can now configure a room display to present commands.  This is normally used for selection of aspect ratios and so on, but you can also add Quick Actions to this list - and they can do ANYTHING.  So while this top tip is about Apple TV app launching, you could trigger a movie lighting scene with our Philips Hue or DMX controller or open the main gate with our Wildcard drivers, all without leaving the source page.

Step 1: Create some Quick Actions

Step 2: In the Display Settings, go to Commands and Enable the Quick Actions

Step 3: Select the Apple TV Source and - hey presto - you have App Launching under the display menu!  You can even re-order them so your favourite is at the top.

Follow us on Facebook, Twitter, Instagram and LinkedIn (@ultamation) for tip #2 and #3 during ISE 2022!  

 

The Ultamation Help Centre

We're very proud of the reputation Ultamation has for exemplary customer support.  As they say, talk is cheap, and it's not the fact that you say you provide first-rate support, it's whether customers get a quick response to their queries and how far you go to resolve the problem.  We work very hard, often responding to customers out-of-hours and walking them through configurations or diagnostics - usually without charge.

While we firmly hold that the human element to customer support is crucially important, it's important to make information as accessible as possible as this means that potential issues can be avoided without engaging with support at all at times and we can provide thorough responses to common problems without rewriting the same information out for the 10th time, simply by linking relevant, already documented, information.

To that end, we have introduced the Ultamation Help Centre which is a repository of documents that we will continuously update with useful info.  The information isn't public, and you will need to register on the portal to access the documents, but this is a property of the platform we're using and we do not use any personal information gleaned from creating an account.  Sadly, this account is not linked to existing Ultamation shop accounts.

We've also moved over to a new ticketing system for support tickets (generated by emailing This email address is being protected from spambots. You need JavaScript enabled to view it.) and the new system gives registered users a way to keep track of their cases.  If you email This email address is being protected from spambots. You need JavaScript enabled to view it., you will automatically have an account created by the platform to that you can track your case.

To access the Ultamation Help Centre, please follow the link below.

Ultamation Help Centre

As an incentive, we've already started writing up the new Apple TV control FAQ so if you're quick off the mark, you can get access to otherwise unreleased information about the new CompanionLink drivers for Crestron and Crestron Home.

We're also publishing recipes for the Crestron Home wildcard drivers which provide simple "hacks" for giving your clients capabilities that can't by offered by your competition.

Wildcard Recipe: Sending Commands to Display Endpoints

Problem

Your Crestron Home project is using a 3rd party AV switcher but you haven’t been able to use the full capability of sending commands to the displays to turn them on and off.

 

Solution

The Ultamation Wildcard modules can be used to send these commands to the AV switcher by integrating the commands into hidden quick actions which are triggered when a media zone is powered on or off.

1. Add the appropriate Ultamation Wildcard driver to Crestron Home - this will depend on the capabilities and communications model of your chosen AV switcher.

2. Configure your chosen commands for buttons or discrete commands (under Actions/Sequences) using the information in the table, and examples below. If you work out more commands, please let us know and we’ll add them to the table!

3. You’re in the 1-hour trial period window - give your new integration a test drive!

4. Once you’re happy everything is working, purchase a licence key for your processor for one of the wildcard drivers here… Home

5. Add the licence key to your Crestron Home configuration (under “Installer Settings”)

 

Command Table

Switcher Manufacturer Model / Range Wildcard Usage Command Data / URL Notes
Binary MoIP TCP port 23 TV On via CEC !CEC={output},1\x0A Replace {output} with the output number for the appropriate display
      TV Off via CEC !CEC={output},0\x0A  
HDANYWHERE All HTTP POST Predefined CEC api/command/cec/{output}/{type}/{id} Replace {output} with the port ID (a,b,c,…), {type} is 0=HDMI or 1=HDBT, and {id} is the uControl CEC library ID
      Freeform CEC api/command/cecpass/{output}/{type}/
POST Data:{ “logicaladdress”: “EF”, “command”: “82”, “arguments”: “10 00” }
Arguments as above. This example will instruct the connected device to switch to input 1. Refer to CEC-O-MATIC for CEC commands.
      B&O Example api/command/cecpass/b/1/
{"logicaladdress": "EF", "command": "82", "arguments": "22 00"}
Bang & Olufsen Soundbar Input HDMI B (logical input 2.2.0.0) The soundbar in reserved on HDMI 2, and the sound bar inputs (A-D) are the second digit of the argument.

Examples:

The following examples show you how format the the commands in the Installer Settings of the module.

The first element - “Label” - can be whatever you want to have displayed on the button. The other elements must be formatted exactly as shown.

Binary MoIP, using the TCP Wildcard, turn on TV on output 3

Playroom TV On|!CEC3,1\x0A

Binary MoIP, using the TCP Wildcard, turn off TV on output 3

Playroom TV Off|!CEC3,0\x0A

HDANYWHERE, using the HTTP POST Wildcard, turn off TV on output HDBT 'a'

Lounge TV Off|api/command/cecpass/a/1/|{"logicaladdress":"EF","command":"36","arguments":""}

Manufacturers vary greatly in their support for, and implementation of, CEC commands.

 

Kudos to Isaac Rosario of Voyager Home Systems Voyager Home Systems | Oklahoma City Metro Smart Home Installations for suggesting this really innovative integration solution and providing Binary MoIP data.

Updating Drivers In Crestron Home

That's NOT my driver!!

Introduction

Crestron Home (Home from hereon) continues to be a great success, bringing a rich Crestron automation experience to homes that may have previously had to compromise for a more “consumer” automation solution.

However, we are still in the relatively early days for the product and many of the current installations are starting to reach the point where updates are desirable or required. The Home operating system is capable of automatic updates but the myriad device drivers, which provide the glue between Home and the 3rd party components around the house, cannot currently be updated automatically. For the sake of clarity, we are discussing updates of the same driver to a new version – as opposed to changing one driver for an entirely different device or model.

In some circumstances this is an inconvenience. In others it can lead to unnecessary support calls. At worst, the driver update process can leave a Home system in a very sorry state requiring roll back to a known working state or – at its most extreme – a fresh install.

This document will first outline the process for safely updating a driver in Home, and then, for the more technically minded, explain why this is necessary.

The Safe Process

Before we start, there is no pretence here – this isn’t a convenient or streamlined process. For various reasons (outlined later) Home cannot, at the present time, simply replace a driver without some user intervention.

So, the first question to ask is, do I need to update the driver at all? Our advice here would be to get confirmation that a driver fixes some known issue or provides required functionality before updating. Simply updating because there is a new, higher, version number may not be necessary. However, if you do need to update your driver, these are the steps that Ultamation recommend:

Step 1: Save your config
I mean, what could go wrong?
You really want to do this before any driver changes – it’s quick and easy.

Step 2: Remove the original driver COMPLETELY.
Using the Home setup app, track down every, last instance of the driver you have configured on the system. For something like a projector driver this may be as simple as a single instance. For something like a lighting driver, you may have tens of instances. They ALL have to go. The downside here is that all routing, sequencing and programming will also be lost.

Step 3: Restart the Home system using the setup app.
You need to restart Home before you can safely pull in a new driver. There is a technical reason for this (see below) but, in a nutshell, this allows Home to tidy up references and ensure the system integrity in maintained.

WARNING: DO NOT reboot the processor by pulling the plug, or typing “Reboot” at the console, as this will restart the processor without any of the vital housekeeping. This is probably one of the best ways to ruin your Home configuration and can lead to all manner of oddness, such as a driver being loaded despite not being visible in the setup app.

Step 4: Confirm that the old driver is really gone!
This is just a verification that the previous steps were successful, but if you don’t do this you run the risk of thinking that you’re running the latest driver, when Home is actually still running the original. (see below for the technical “why”)
Unfortunately, there’s no “easy” way to do this and it involves looking at the Home processor’s file system to check that the driver isn’t shown under “user\data\UsedThirdPartyDrivers” as shown below:

Drivers Under Crestron Home

Step 5: Load the new driver
Either grab the driver from the Crestron driver portal (which will appear in the Home setup app) or sideload the updated driver (covered elsewhere).

Step 6: Reconfigure the driver.
You will now need to recreate all routes and programming ties that were lost during driver removal.

You should now have the updated driver running correctly in Home.

There is clearly a lot of scope for addressing the current challenges of driver updates in Crestron Home and Ultamation have written this guide in response to the increasing number of questions we receive about updating drivers. As Home continues to grow and older systems receive maintenance updates then we expect Home will be enhanced to streamline the driver update process and, hopefully, make this guide redundant.

 

Why All The Pain? - The Technical Reasons

Why isn’t this automatic?
We’re going to brush aside the “but xyz have been doing it for years!” because it’s silly to compare apples and oranges. XYZ probably has a completely different driver architecture to Home so the rules/constraints are naturally very different. But specifically for Home, there are times where a driver could be updated, at least semi-, autonomously. The trouble is, there are numerous ways in which a driver update could break a configuration which would require manual intervention. The list includes:

  • Deprecated features
    What does Home do with that command that just disappeared!?
  • Changes to I/O config
    TV/CBL was just renamed to MEDIA – are they the same? Different?
  • Programmable Operations/Events have been changed
    What does Home do with the actions that were programmed into sequences?

Now – those are some arguments as to why automatic updates are dangerous. But that doesn’t mean the situation couldn’t be better. Home should be able to flag that a driver update is available and perform some consistency checks to see if an update would create any configuration anomalies. If there are no issues detected, then a semi-automatic update process should be possible which would remove most of the inconvenience and fragility issues.

So many drivers for different models – They all seem the same!
That’s because they probably are. This is sadly a numbers game. If you have a driver database and you want to shout about how many drivers you have, the figure you quote is really the number of supported devices you cover, not the actual number of variations of code you have. If every Sony TV uses a protocol that is 98% consistent across all (for arguments sake, 500) models, that’s probably 1 driver, but we advertise it as 500 models – why? – because it means an integrator can verify that a specific model is supported, but also because is makes the driver database appear bigger. It’s not helpful when you’re searching for an elusive feature that doesn’t seem to work correctly on the driver you’re testing, but that’s the way it is. Hopefully there will be something in the future to allow us to see which drivers share a common base so integrators can test once; and move on.

Why do we have to remove the old drivers?
This one is a bit technical and we need to make a distinction between the driver information (the entry in the Home driver list) and the driver code (the running code once a driver has been “loaded” into the memory of the Home processor). This problem lies deep within the operating system (below Home itself). Once the code is loaded into memory, it is there for good. The underlying OS cannot (without architectural changes that Crestron have already considered and dismissed) load updated driver code “over” the original. If you leave a single instance of a driver around then, as soon as Home reboots, it will reload the original driver and you’re back to where you were. They ALL have to go.

Why do we need to reboot Home before loading the new driver?
Read the reason above. The only way to clear the Home processor’s memory of the old driver is to restart the Home process (i.e. a reboot, or program reset).
Additionally, you cannot always rely on drivers being cleanly removed when they are taken out of the setup app config (especially Platform drivers!). This isn’t only true for driver updates, but also when modifying driver properties through user attributes. If in doubt, after you’ve made changes to the Crestron Home configuration, check with a clean restart!

I think I’ve messed up the Home config. What are my options?
The safest bet is just to restore the config you made in step 1. On rare occasions Home may not even boot to the point where you can restore a cloud config though, more often than not, Home will revert to a factory state if it detects serious problems. If you really find yourselves at the edge of the world, call Ultamation and we can try to assist.

But Home says it’s updated and I didn’t do all of this nonsense.
It’s easy to think Home has updated your driver when it hasn’t. For the reasons already discussed, this is what can happen:

That's NOT my driver

Because Home cannot replace code at runtime (i.e. without a restart) the old code is still running even though Home has processed and imported the new driver’s information. In the future, we hope that Home will flag this discrepancy and inform the integrator that the driver update is not complete until the system has been restarted.