February 24, 2021
I’ve had the privilege of helping hundreds of MSP’s become MDR’s and I’ve helped hundreds of MDR’s break into new market segments. I’ve been fighting the good fight of “technology alone will not win this war,” for almost a decade.
Out of the hundreds of MDR transactions I’ve been part of, one area that is always really entertaining is watching new prospective customers trying to find a MDR partner that’s a good fit for them. It’s more than obvious right out of the gates, who has worked with an MDR before and who hasn’t.
The funny thing is, you don’t need to be a guru to spot these common denominators. The next time you are in the market for an MDR/MSSP, here are five better questions to ask and five questions to avoid.
Don’t Ask: “How many employees are your largest customer?”
I understand why prospects ask this question. The problem is, the answer doesn’t tell you much. What if their largest customer is poorly managed and unhappy? Would their size matter?
Furthermore, Information Security has been one of the fastest growing sectors in tech, and it has attracted large and savvy investors with deep pockets. The market is also still in its infancy; therefore, it is easy for a well funded emerging MDR to answer this question with what might look like an impressive answer. However, in reality you don’t learn much about the MDR by learning their largest customer has 15,000 employees. You have to dig deeper.
The real question a prospect is trying to ask is about complexity - specifically the quantity and quality of that complexity. Therefore, the average MDR Shopper will correlate (no pun intended) the number of employees to complexity. Which might seem logical at a surface level; however, tried and true Information Security professionals know it is deeper than that.
For example, there are several 15,000 - 50,000 employee companies that are not as difficult to manage as a 2,000 employee organization. Why? Factors like industry, company infrastructure, stage of business, leadership, and proprietary information also dictate how often and how aggressive a target is attacked.
Ultimately, there are better questions to ask to learn more about your prospective MDR’s detection methodology.
Do Ask: “What is the most complex environment you manage, and why?”
This is a better way to learn about your provider's detection methodology. No two clients are alike, and understanding how an MDR accounts for that uniqueness will provide deeper insight into the value of their service.
#2 - Customers
Don’t Ask: “How many customers do you have?”
I understand the need to quantify the workload. It helps prospective clients determine if a vendor has too much or too little amount of work to handle. A low amount of customers might signal uncertainty and lack of adoption and a high amount of customers might signal over worked analysts and an overall poor customer experience.
The real question a prospect is trying to ask is “Am I going to get the proper attention?” mixed with, “Do you know what you are doing?” There is a better way of understanding that.
Do Ask: What is your renewal rate?
The amount of customers that stick with an MDR after their contractual agreement is a testament to the customer experience. As the old saying goes, “If it ain’t broke, don’t fix it.” If an MDR is providing value, and a customer sees enough value to renew on a regular basis, it is fair to say customers are getting enough attention, and the MDR knows what they are doing.
No one is perfect; however, understanding the renewal rate, and understanding why certain customers didn’t renew will give you much better insight into the customer experience.
Castra has a 90.24% renewal rate!
Don’t Ask: How many analysts do you have?
There is no point in asking about the workload without asking about the workforce. While I see the logic in this line of questioning, years of experience allows me to see the surface level analysis a novice MDR shopper will do versus a deeper, well thought out analysis a tried and true information security professional will do.
Similar to the previous questions, the answer to this question is almost never accurate, and it still doesn’t tell you much about the inner workings of their Security Operation Center.
Do Ask: What kind of automation are you proud of? Where are your analysts? What is the retention of your analysts?
This will give you a much better understanding of their SOC infrastructure and processes. The automation they’ve built will speak to their technical aptitude as well as their ability to anticipate future needs. Their geographical position will tell you if they are in the good 'ole USA or overseas. And their retention rate will speak to their leadership, growth, and job satisfaction.
Don’t Ask: How many threats have you detected?
It’s a poor question because anything can be defined as a threat. The count of detection is irrelevant because that number is easily mistaken. Many providers have the standard "funnel" or "pyramid" charts showing how many events, alerts, threats, incidents, etc that they've detected.
But those numbers alone don't tell you whether they've leveraged that data properly, vetted it against good Threat Intelligence, effectively tuned down False Positives, etc. They also don't tell you how the provider detected those things and if their responses matched the threat.
Do Ask: What Threat Intelligence do you use to make detections?
You should question an MDR on where they obtain the Intel, how do they apply it to your data, and how does the SOC deal with false positives.
When Castra learns something that works, it's applied where relevant to other clients, and likewise, when something is false positive (intel or otherwise) its proactively removed from other clients and sourcing locations are updated.
#5 Technology and Service?
Don’t Ask: “Why is your tech better?”
MDR’s who have their own homegrown SIEM have an identity crisis. Building an Information Security service is incredibly difficult. Building an Information Security product is also incredibly difficult. Trying to do both at the same time is a waste of resources.
Not only are you trying to boil the ocean, but you lose the ability to obsess about a specific problem. Building a product based company, versus building a service based company requires completely different infrastructure, budgets, personnel, culture and strategy.
Furthermore, an MDR who provides their own tech and their own service want to pigeonhole customers into not leaving. What happens if you don’t like their service? If you drop their service, your data and technology go along with it and vice versa, which is a big problem for any size of company.
Transparency and a separation of powers are the key factors in this equation. It not only displays strength and confidence from an MDR who is transparent, but transparent services also speaks to Security Operations best practices.
Castra teaches their customers how to disconnect from any third party including Castra, on day one. In addition, if an MDR provides their own SIEM, how do you audit what they are doing? How do you audit a "Mystery Box?" Isn’t dropping off a mystery box on your environment a security threat on it’s own? The lack of transparency isn’t comforting, and the lack of options if you decide to change providers is not a good fit as a long term solution.
Do Ask: “How are you different?”
This answer really has to stand out. An MDR’s differentiators should be short and sweet, yet well thought out and with heavy merit. It should tell you that they understand the competitive landscape and the pain customers are going through. But most importantly, their differentiators should tell you how well they can execute on the gaps they see in the industry.
Qualifying an MDR to figure out who is the best partner for your organization is not an easy task. Take it slow, and try to consider all angles. Use the questions in this blog post to get a better understanding of your MDR’s ability to provide robust services in a complex environment. Quantify the quality of their service by their customer and employee renewal rates. Dig deep to understand their threat intelligence and detection methodology, and align yourself with an MDR who isn’t trying to boil the ocean.