Earlier this week Arstechnica broke the story that a security consulting firm found a couple of “backdoors” in an industry standard AV room control system. The fact that the AMX NX-1200 had these hidden accesses with colorful developer names (resembling superheroes) also caused them some concern. It took AMX (now a Harmon company) seven months to respond to the report – finally yesterday explaining that these accesses were never meant to be malicious as they were just leftover items from their legacy code. (Their entire statement is now in the story linked above.)
Now before anyone thinks I’m just piling-on here, let me say that Harmon and AMX are fine companies that make good products. Waiting seven months in the hopes that the news would just go away unfortunately does represent questionable judgement, and probably led to the hysteria that caused the story to be picked-up by CNN, ZDNet and others. It probably also didn’t help that the CNN story used the above picture of an AMX system in a White House meeting room as their lead. But none of this is the really important point that everybody should take away. This wasn’t a meltdown – this was just a huge warning – as big a red flag as the enterprise world will get. This should be considered the Three Mile Island of enterprise AV.
Let me explain why. About a year ago, a large international finance organization hired my firm and I to perform a security audit of their AV systems. Their Information Security team didn’t really understand the nuances of AV and Multimedia, and the firm I work for has specific expertise in Security, IT, Network and AV / Video / Multimedia systems – a rare but growing combination of AV and IT expertise. I led the audit project. The results completely floored me. Ninety percent of the AV systems – the very typical components found in many enterprise conference rooms - failed the most basic requirements from a security perspective. (Findings that I presented about a year ago at InfoComm Connections San Jose. )
I’m an AV guy. I’ve been advocating, designing and installing systems like these for over three decades. I wasn’t happy to discover they are all ticking time bombs. Looking at the history, the problem developed as AV systems wanted and needed to be more IT relevant and take advantage of improved connectivity. Instead of the components communicating via “closures” and then later serial / RS-232 connections, they have all migrated over to communicating via a standard IP network. In some cases this is an isolated network set-up just for the AV gear, but in a rapidly growing majority of installations this gear rides on the standard enterprise network - so it can communicate with the internet and across all enterprise rooms. What I discovered is that “best-practices” AV strategy - using a blend of best-in-breed components in each room – made securing them virtually impossible.
Each room likely uses a different brand of device for display, control, audio processing, video processing, videoconferencing, and a dozen other tasks. Getting these disparate systems to work together in the first place is admittedly such a difficult task that AV companies routinely have to pre-assemble them – a process called “staging” - so they can work out the communication bugs. Custom programming code for each room – the kind run on the AMX system called out above – is then written by third parties to make everything work. Who writes this code? Maybe the AV company; maybe a third party programming firm; maybe an enterprise employee hired for such a purpose; honestly, it can be anybody that has learned the programming skills.
We discovered that each of these components use their own, unique process to write their firmware and communicate to the network. The issues discovered in them were overwhelming. Many components didn’t have any process of authenticating user access at all. Of the ones that did, many only had admin-level access; many sent passwords or transferred files in clear-text; many cannot accept strong passwords; many cannot comply with enterprise required password change timetables; many did not support lock-out after a number of failed password attempts; the ones that used SSL encryption routinely used an outdated version of it with known vulnerabilities; this list can go on and on. Getting any of these AV components updated / patched when a new vulnerability is discovered is nothing short of a nightmare. A very small number of firms put patches and the update process on their support website. Most firms need to be contacted directly to find out if they even intend to patch their systems much less have a process for the end-user doing it. Then, if by some miracle, an enterprise is able to obtain patches for a newly discovered vulnerability for all the components in a room, there is that “staging” issue again. Every newly patched component likely needs to be tested against every other one to ensure they can actually still communicate with each other – just like installing the room all over again but this time with no budget. And, if enterprises simply choose not to go through the bother of keeping these systems patched and safe (if it is even possible to do so) then it only takes one weak link on the network – one hidden or unknown back-door in an AV component that is open to the internet – for a piece of malware to find a safe home on an otherwise secure network and begin infecting other systems. (If you don’t think that is a reality you should really research the Target breach – and remember that all these AV systems are also serviced by an installer’s personal PC.)
Our conclusion was that the way we provide AV in the enterprise – the same “best-practices” we’ve been using since the 1980s – has to change. It doesn’t matter that – without argument - one manufacturer may have the best audio processor and another may make the best control system. The monumental support nightmare that comes from trying to keep these systems safe outweighs the minimal benefits that users derive from the typical 10-20 best-in-breed components mixed together - and supported with custom, third-party programming - in an average AV room installation.
The new “best-practice” clearly has to be putting in rooms with components that all come from one manufacturer – buying one single SKU. This omits the expensive system designs, omits the custom programming, omits the variables that require the “staging”, and moves the onus of keeping the systems patched and safe from the user to the manufacturer. A growing number of manufacturers – including Cisco with the outstanding MX series – now make all-in-one / off-the-shelf AV / videoconferencing room systems. Are they as good as a custom designed integrated system – probably not in every case, but many of them are still very, very good. And using them provides the multiple side benefits of being much less expensive to design, install, and support – and much easier to use (with complete consistency from room to room) – than 1980s style integrated rooms. And when the next Heartbleed or Shellshock or POODLE is discovered it can be immediately addressed across all enterprise AV rooms with a single patch from one manufacturer.
So the AMX BlackWidow and 1MB@tMaN backdoors turn out to only be dangerous if you ignore the warning signs and keep buying and building vulnerable rooms. Then it really will take a superhero to undo the actual and reputational damage to your organization. I and my firm can help you with this journey of improving your security and extending it to modern AV rooms. Like the old commercial says, you can pay now or you can pay later. No matter what you do, don’t say you haven’t been warned.
=========================================================
This article was written by David Danto and contains solely his own, personal opinions. David has had over three decades of delivering successful business outcomes in media and collaboration technology for various firms in the corporate, broadcasting and academic worlds - including AT&T, Bloomberg LP, FNN, Morgan Stanley, NYU, Lehman Brothers and JP Morgan Chase. He now works with Dimension Data as their Business Transformation Consultant for the collaboration, multimedia, video and AV disciplines. He is also the IMCCA’s Director of Emerging Technology. David can be reached at David.Danto@Dimensiondata.com or DDanto@imcca.org and his full bio and other blogs and articles can be seen at Danto.info. David is also the co-founder of Masters Of Communication. Please reach-out to David if you would like to discuss how he can help your organization solve problems, develop a future-proof collaboration strategy for internal use, or if you would like his help developing solid, user-focused go-to-market strategies for your collaboration product or service.