Any organization looking to implement a security solution will face a wide variety of options, including off the shelf products and services, in-house development, or bespoke solutions from specialist providers.
The vast majority of commercial security solutions’ are of a complexity that makes installation and configuration a specialist art form in itself, requiring significant expertise to get right. The problem is that most organizations don’t have access to such experienced staff. That’s why they are purchasing an off the shelf solution in the first place. A lack of in-house resources or access to external expertise is a common constraint for businesses. If the security solution’s configuration isn’t correct, it may well offer little or no security. The danger is that the organization with the poorly configured solution won’t necessarily know that there is a problem and become complacent in the false belief that they are protected. This is potentially worse than knowing that you have no protection but being on guard because of this. Where bespoke solutions are developed in-house or delivered by a specialist supplier, the product has no pedigree to provide assurance that it is effective. There is no large pool of users providing feedback that can identify issues and lead to updates and improvements.
Security solutions come with a wealth of marketing information that promises complete protection with minimum fuss. The reality is often a little different when the product is purchased, and you get to open the shiny box. As anyone who has plugged a new device into a network or installed some new software will know, it’s never as simple as the instructions make out. There’s generally at least one step in the process where things don’t work as they should, and a call to the support helpline or an online search is needed to resolve a problem. The more complex the solution, the greater the number and complexity of the problems will be. It also is not always apparent that everything is working as intended. Just because a software tool is executing without errors doesn’t mean that it’s working. If the software tool is monitoring for a security threat, the only way you can ensure that it’s working as intended is to know that an attack is in progress and that the tool correctly reacts to the attack. Even then, you can only be sure that it worked for that specific attack. You will need a more comprehensive test methodology to provide confidence that it will work for all the possible types of attacks that it’s supposed to and that it reacts correctly.
The most significant problem with misconfigured security solutions regarding potential damage to an organization is open vulnerabilities that an attacker can exploit without detection. There is, however, an additional problem that misconfiguration can lead to excessive false-positive alerts that can easily overwhelm the security team monitoring the security solution, wasting resourcing and potentially masking genuine attacks. False positives are a particular problem for many SIEM, EDR, and NDR solutions, making them at best challenging to manage and at worst ineffective. Even the latest products with next-generation controls do not fully resolve the issue of false-positives. There is a current trend of security solutions that all sound exactly the same, offering artificial intelligence (AI) as a feature, employing machine learning to monitor systems, and, according to the marketing staff, pick the wheat from the chaff to eliminate false positives and capture the real security incidents. However, the technology is in its early evolutionary stages and has a long way to go before delivering the sales claims and if you look under the hood you quickly realize that it certainly doesn’t use any AI by definition. It is far better to stick to a dependable technological solution that is correctly installed and configured to deliver optimum results, assured using a comprehensive validation as part of the commissioning process along with a periodic validation program to ensure that no configuration errors have been introduced and that the security solution is up to date and in step with the latest threats. Suppose you do decide to go down the AI route. In that case, it should be complementary and deployed in conjunction with an existing security solution rather than a direct replacement, at least until the technology has matured to the point where you can rely on the results.
The solution in all these situations is to validate the security solution. To test the solution as a complete entity to ensure that it does what it is supposed to do under all possible conditions. The thoroughness of the testing will depend on the importance and value of the assets that the security solution protects and the available testing budget. Validation can vary in scope from simulating a few attack scenarios to a detailed static analysis of the solution using formal methods.
For example, the simplest test method for assessing the malware detection capability of a solution is to obtain test malware from a trusted source to ensure it is detected and blocked. Test malware is available as executables, as links to test websites, and embedded popular file formats. Testing should cover both the ingress of malware through email, text messages, internet access, and internal detection of infected files. Malware should cover not only data strings found through anti-virus signature detection but also consequential effects such as suspicious network traffic within the network or out of your network using DNS tunneling or file-sharing mechanisms. The greater the number of different test scenarios, the greater the level of assurance you’ll have.
The validation process should be cognizant of your organization’s cyber threat profile, so the effort is focused on the most credible threats for your particular industry sector and geolocation. Attacks can come from a diverse range of sources such as nation-state espionage, industrial espionage, financial crime syndicates, hacktivists, or bedroom script-kiddies. Those that will be of most concern will depend on the nature of your business. An innovative high-tech development company will face different threats than a financial institution.
A by-product of the validation process is that it can identify areas where different security solutions overlap, potentially identifying areas for rationalization by eliminating unnecessary controls. We found the Mitre ATT&CK to be helpful for this type of assessment. However, there needs to be a balance with the requirement for defense in depth. It is never good practice to rely on a single security solution, defense in depth using the onion skin analogy will deliver overall security greater than the sum of the parts. This analogy only holds up if each layer is genuinely independent, so a vulnerability in one layer is unlikely to line up with a vulnerability in a different layer. It’s not hard to see that if two products come from the same vendor and use the same OEM technology or even the same coding team, a bug in one product that causes a vulnerability may very well appear in all the other products they produce—having products that are genuinely independent will eliminate this. Truly independent means different development teams in different companies using different tools, technologies, and methodologies. This may require research and probing questions to the vendors to ascertain. It’s not uncommon for the same core OEM technology to be packaged into different products for sale by different vendors such as Webroot for web filtering being used by so many security vendors or Bitdefender by EDR/AV vendors.
Having an effective validation process can also be invaluable when it comes to acquiring a new or replacement security solution. The process can help identify which product is the most effective if you’re trialing several different options. It can also provide leverage for price negotiation if you can demonstrate that the performance of the solution doesn’t quite make the marketing claims
As a final word, it’s always good to adopt the mindset that the security solution will not prevent all attacks. You can see how we demonstrate this in these two articles (article 1, article 2) while an APT just like our red team is always successful. There are always zero-day exploits that manage to be deployed against high-value targets before the security vendors catch up and deploy countermeasures. Insider attacks are also more common than people realize, and it can be almost impossible to prevent the most determined and capable attacker. The organizational security should include measures to recover from attacks in addition to the detection and prevention controls. Here is where comprehensive incident recovery, business continuity, and disaster recovery processes will come into play. Ransomware often makes the headlines for its disruptive effects, but a good backup policy and recovery plan can solve the problem in a few hours. You can read in this article how one of our clients dealt with a ransomware attack.
