It’s time for a new IT disaster recovery plan
Posted by Matt Weston - 04 January 2018
Last updated 3PM NZDT on 16 January 2018
Two major vulnerabilities have been demonstrated in CPUs which break down the boundaries between Virtual Machines and Applications.
With the code supplied by the research team, it’s possible to craft a malicious application which can read the memory of another application or virtual machine - potentially leaking secret information (like passwords, or database records).
These vulnerabilities are known as “Meltdown” and “Spectre” - or more formally CVE-2017-5754 (Meltdown) and CVE-2017-5753/CVE-2017-5715 (Spectre).
Patches are becoming available, but there have been a couple of false starts, and some vendors have reccomended rolling back patches.
Metdown and Spectre information is still evolving rapidly, and we expect more news to come.
Meltdown affects the “Virtualization Host” layer - allowing virtual machines within the same physical host to read each other’s memory.
At this stage, it is known to affect Intel CPUs - AMD has specifically claimed that they are unaffected by Meltdown. So far this has not been confirmed.
Spectre occurs with the virtual (or physical) machine itself - eg a Server, Laptop, etc. It allows applications to read each other’s memory within a running machine.
At this stage, this has been confirmed to affect Intel CPUs as far back as 2011 as well as AMD CPUs. It’s possible that all CPU (including ARM) may be affected by Spectre.
In both vulnerabilities, the result is that it’s possible for an attacker to read information from memory that developers would have (reasonably!) assumed would be protected by the operating system, or virtualization host.
Patches are becoming available now - but at this stage the biggest risk appears to be for systems in “hosted” environments - either shared hosting, or cloud providers. This is because a number of different customers may share the same physical host layer, which broadens the possibility of an attack.
Patches are being released out of band (i.e. before Microsoft’s normal ‘Patch Tuesday’), and are being made available. We’ll keep this article updated as more patches come to light. This vulnerability was initially embargoed until January 9, however due to media interest and investigation, the information was released early to minimise speculation about the vulnerability. As such, not all patches are available yet.
The patches that have been released so far have triggered unexpected bugs in both software and drivers, and was completely incompatible with some Antivirus software. As such, Microsoft has required that Antivirus vendors specifically flag compatibility with the Spectre patch before it is applied. You can read more about this at Ars Technica.
Intel has released a Microcode update for their Haswell and Broadwell CPUs - but this has also caused unexpected issues, and VMWare has recomended that customers avoid installing this update, and if possible, roll back to a prior version.
At this stage, the best option appears to be to push hard with workstation patches, and approach servers more cautiously. While there has been some incompatiblity and performance loss with workstations, the benefits of having the patches in place definitely outweighs the risk of going unprotected. For servers, at present we've moved to a "patch cautiously" mode. Since it's unusual for servers to run untrusted code anyway, the risk is partially mitigated. We've opted to tighten up the restrictions on what can be run on production environments until performance impact and stability can be fully assessed. For more details on this, see this link.
Given the nature of workloads on SQL Servers, the potential performance impact is a serious concern - especially given that SQL Servers are typically licensed per-core; and that a significant amount of per-core performance can be lost through these patches. Microsoft has released a guide on patching for SQL Server which specifically addresses performance impacts. Two features in particular, Kernel Virtual Address Shadowing (or KVAS) and Indirect Branch Prediction Mitigation Hardware Support (IBC) may not be required, depending on how your SQL Server is configured. This will have less performance impact on your server, but comes with the added risk that effectively some of the patches are effectively turned "off". Through our assessment, we've determined that many (but not all) of our SQL instances fall in to Scenario 2 - and are in the process of evaluating the performance trade-off for enabling these features in our environments.
As at 3PM NZDT on January 16, 2018
|Vendor||Virtualisation Host Layer (Meltdown)|
|Amazon Web Services (AWS)||Patched|
If you utilise other shared hosting services (e.g. local datacentres), you should reach out to them directly to confirm their status.
As at 3PM NZDT on January 16, 2018: