Quantcast
Channel: Third Defense Blog » Metrics

Magnificent 7/7: Metrics and Scorecards

0
0

It’s been a fun CSO tools series so let’s close it with a bang. Number 7 is the ever intimidating operations scorecard. I’m also happy to share this post coincides with the release of the Third Defense Metrics Manager application. I’ll leave the marketing to the website but will include some screen shots below as an example how to build and maintain an ops scorecard.

I’ve only seen a handful of actual working scorecards throughout my career and each were different. The lack of standardization is acceptable at this point in the maturity of security measurement and likely beneficial since everyone has different motivations, audience comprehension, and access to information. As mentioned in the balanced scorecard post, the most important step is to just get started and build or advance your culture of transparent and accountable IT services.

The operational scorecard communicates trends and progress across your operational services. It’s a more tactical view that feeds the balanced scorecard. While lower-level, it still only contains business relevant metrics that must pass a few tests. To start, each metric must:

1. have a defined and communicated goal.

2. pass the “so-what” test i.e. a non-technical person can understand what it means to the business.

Of course you need automated, repeatable, etc. criteria. Much has been written on how to define a metric. I think Andrew Jaquith has the best book on the subject but a book (or a blog post) can only take you so far. The key is to take action. I assume you’ve read the books and visited securitymetrics.org so I’ll focus on areas I think are under served. Obviously security metrics are in the infancy stage. We have plenty of resources to tell us what kinds of areas to measure, however we have little showing us how to collect and report the information.

But I don’t have enough data

Yes you do. There’s no chicken-egg dilemma. The key is to start small. You only need two items to call it a scorecard. Heck, Third Defense is just adding our second application and we’re calling it a suite (there’s more on the way)! As you pick your metrics, here’s some additional tips I don’t remember reading anywhere. First, your metrics should tell a story. No one will remember the numbers, they’ll remember how they felt while reading the scorecard. Work backward from the desired impression and pick metrics in the following business-relevant “story lines.” The first two are written about the most:

  • Value to business: areas where your team contributed to revenue. This will be thin but keep refining till you strike a vein e.g. % of priority business initiatives with security involved at design phase (assuming your business prioritizes initiatives). If outsourcing is part of your business, vendor management and assessment metrics can go here. Don’t be tempted to create a metric to highlight a win. Save the specific anecdotes for your quarterly meetings.
  • Reduce impact to business: # business impacting incidents, severity of impacting incidents, MTTR, $ reduction of fraud, % of customer turnover due to security.
  • Efficiency: Take a minute to show off or at least set expectations. Readers need to know you’re valuable and thrifty. No fat cat security teams here e.g. avg days for access certification, hours to provision, % roles with automated profiles, % processes with SLAs, % processes within SLA, even % processes with defined RACIs.
  • Control posture: I disagree with sources that say ops metrics e.g. % devices managed for security, don’t pass the so-what test. One of the emotions you want readers to internalize is “those IT security folks got it covered (or at least they know where they’re going).” This credibility is important as you negotiate controls, assess, and support the business on a sustained basis. If you run a tight shop, or at least a measured shop, people will be less inclined to BS you.

But I still don’t have the data…

Yes you do, try harder. Do you scan your end-points and servers? Do you compare your scan inventory with the spreadsheet or cmdb from IT? Do you participate in incidents? Administer access? Interact with people outside IT? Of course. The key is selecting the relevant metric and communicating it properly. Heck, you could have an early metric simply measuring % of metrics with established baselines and targets! You only need a few to start.

Ok, what’s next?

Now that you’re fired up, I’ll expand on the three basic steps:

  1. Design Metrics: it’s an art form
  2. Data Entry & Management: efficient, scalable as possible
  3. Reporting: make sure you get the bang for your buck

Step One: Design

Isn’t it amazing we don’t have an industry standard scorecard already? I don’t mean books and blog posts, rather every team using an active scorecard they stand behind. Because selecting and organizing metrics is still an art, I haven’t seen a one-size fits all repository of metrics. Below are some of the metrics in our Metrics Manager repository. Feel free to login (to the suite :-) for free to see how we organize and compare notes.

A quick word about organizing metrics. I call these different “collections” e.g. you can shuffle your metrics to line up with your favorite compliance checklist, your service catalog, ecosystem, whatever best tells your story. Below is a list of my favorite metrics. This view doesn’t have all the compliance associations yet. We’ll share the complete version as we pretty it up.

Aside: eventually we’ll get all our material in a bulletin board system with an open creative commons license. For now, let’s go with this: I make no promises or warranties on this list of metrics. It’s simply collected from my experience over the years. I haven’t used all these before so they may or may not be a fit for you. In general, feel free to do as you please with this blog and content. IT security processes and practices should be open source.

rough list of metric examples (we’ll produce a more usable version as soon as we can)

Also, here’s a screen shot of the fields we capture per metric you may find helpful:

Step Two: Data Entry & Management

I’ve done this in a spreadsheet in the past so you have no excuses. There are also tools out there (like ours) so you really have no reason to delay! I need to emphasize a tip I mentioned earlier. Each metric should have a baseline and target. Without them, it’s just a statistic. When you define where you were, are, and want to go, you tell a story. (Kind of funny how much of security is just about telling great stories.) Another benefit from these three datapoints is the ability to create an expected value at any point between them i.e. how fast will you hit the target, or are you already there and just need to optimize. This can get tricky in spreadsheets as the months roll by, but we did it at msft and wamu so I know you can do it too.

Another tip is to let the metric owner drive the baseline and target definitions. It’s empowering to be able to set your own bar. It’s also  motivating to achieve the selected targets. As long as targets are realistic, they make great performance review evidence.

One more note related to metric design. Many of your business relevant metrics will be a combination of tactical metrics. For example, % accuracy of inventory is a calculated field from what you enumerate vs. what IT tracks. You’ll either have to build a business intelligence system, maintain many spreadsheets, or swivel chair the numbers from tactical outputs into your metric tracking tool. I do know of one IT shop investing in a BI platform to integrate source feeds, apply business logic, and present relevant results. I wish we were all there! If you don’t have the resources for BI, check out the middle ground of summarizing tactically, then manually reporting the relevant metrics. We gotta start somewhere. This was the motivation for our Metrics Manager application. Some day the industry will have some magic API that translates raw data into executive presentations. Until then, we make it easier to transform your tactical evidence into compelling visuals that drive decisions.

Also, set aside a place to capture notes for each metric recording period, especially for calculated fields. Dependencies and surprises happen. It doesn’t mean you’re failing, you just need to explain why you’re green one quarter and blazing red the next. No one said transparency is easy, just valuable.

Here’s a shot how Metrics Manager tackles data entry and management. Note the ability to assign a baseline (red dot) and target (green dot). You should also allow for multiple targets to set expectations on the pace of progress. The below example shows linear expected progress. In real life this could be flat for six months then jump up. Or your baseline could already be at your target level and you’re simply tracking progress. The key is to think of a metric target as an “acceptable risk” definition for that point in time. As you reach your targets you can re-evaluate if it’s optimal for the business.

Reporting

All is for not if you don’t have the eye candy to communicate your story. Yes, I feel so strong about this we invested six months in our first applications to make the tasks of organizing and presenting data easier and more effective. You have to draw the future picture. It’s important to define what success looks like and work toward it, otherwise you’ll never get there. Defining what success looks like is also self-fulfilling. If you write down your optimal scorecard but only have two metrics started, you have another nice story to show how additional investment in security (to collect and measure evidence) will translate into better risk decisions for the business.

I have a few goals for metrics reporting:

  • Empower the metric owner to trend their individual metrics.
  • Review the team’s progress as a whole with the ability to drill down where needed.
  • Create an overall roll-up to show program level progress to non-technical stakeholders.

Again, you can do this in spreadsheets it just takes a bit longer. Here are a couple screen shots from our tools to get you started.

Group table summary and individual drill-down. Note we also track if you’re trending toward or away from your expected progress.

The overall metric roll up always sits on top of the table, looking down upon its members. We call ours the “Master Security Index.” This shows the average % distance between actual values and their expected progress values. Thus, you get a high level view if security is trending toward or away from expectations. If the overall index doesn’t represent your story, you can drill down to expose the areas of concern. You can also exclude specific metrics for what-if scenarios.

Wrap up

I hope this post inspired you to start or advance how you collect evidence and communicate your progress. It’s dirty work translating tactical statistics into relevant metrics. However the pay back in credibility and demonstrating the value security brings to the business is well worth it. The Ops folks have their throughput and uptime metrics, show them what security can do. Still don’t believe me? Just give it a try already. Start small with a few experiments and grow from there.

One item I forgot to mention above is the added accountability and pride your team will suffer through. You might even see all five stages of grief here… Every IT security department I’ve seen has areas they know they should be doing better. It really hits home when you broadcast the numbers. Please don’t hide these. Celebrate them! What a great opportunity to take a bath and justify why improving your posture is good for the business. If you still don’t get the resources to raise the bar, no problem. You now have another way to show what acceptable risk is for you business. I’m already sleeping better.

You’ll also be challenged with your team being “too busy” to calculate and enter in their monthly metric data. Don’t be fooled: if you’re too busy to measure what you’re doing, you’re doing the wrong things. Break out the management whip if you have to. In 6 months when you look back and see the trends, you’ll love the fact you have an evidence based story to tell. Count on it :-)



Can metrics save money on PCI compliance?

0
0

I continue to be impressed by the VZ team. Their latest PCI Compliance Report continues their contribution of data sharing with the industry. Here are a couple cherry picked passages from the exec sum:

“Essentially unchanged from last year, only 21 percent of
organizations were fully compliant at the time of their Initial
Report on Compliance (IROC). This is interesting, since most
were validated to be in compliance during their prior assessment.
What causes this erosion over the course of the year?”

….

Organizations struggled most with the following PCI
requirements: 3 (protect stored cardholder data), 10 (track and
monitor access), 11 (regularly test systems and processes), and
12 (maintain security policies).”

Here’s another passage that surprises no one, from pg. 29:

“the secret to maintaining compliance lies largely in treating it as a daily part of conducting business. To achieve this, the correct mind-set must be instilled across the organization, and this type of integration must come from the top down.”

I may be projecting but while I read the report, I kept visualizing the ops and security teams scrambling between the initial and final assessments to hurry up and get compliant. Given the data above, deep down they knew they’ll be doing the same thing next year. This got me wondering: how much time, money, and opportunity cost is spent scrambling to get compliant? How can we reduce this cost?

Here’s a hypothesis: the cost of running a compliant shop is less than the cost to run annual fire drills. Boy I’d love to test this hypothesis. This also fits into my favorite question about metrics: is control effectiveness increased simply by the fact it’s being measured? Mind you I’m talking about measuring, not auditing. This PCI report clearly shows audits don’t improve controls throughout the year. What if we applied metrics to key operating controls under the PCI scope? Would a minor investment in spinning up a metrics program be less than the annual effort to get compliant? Is the cost of maintaining a control less than the cost of scrambling to fix it?

I took the liberty to identify a candidate set of metrics that may help answer these questions. I ran through the requirements and selected 23 metrics. I don’t think we need a metric for every requirement. My goal was to identify metrics that provide visibility to a group of controls. E.g. % of users with completed role verification per some schedule. This metric may knock off multiple requirements e.g. least privilege, segregation of duties, terminations, vendor, and remote access. I also focused on the operational controls vs. design decisions e.g. metrics aren’t a fit to measure your encryption design.

Here are the 23 (feel free to scroll down and read them later if you want more monologue):

Metric Title Category Description
Post-prod. Vulns. Application # Post-production applications vulnerabilities Target: 0 Support PCI requirement 6.5.
Secure Development Application % of in scope application development projects that follow secure development practices. Focus on PCI requirement 6.3 and 6.5.
Change Control Security Change Control Number of changes that reduce or violate security policy. Focus on PCI requirement 6.4.
Production Access Change Control Number of non-operations personnel with access to production systems. Focus on developer access to address PCI 6.4.
PCI Servers Data Number of actual in scope servers holding PCI data compared to known target number. Goal is to set a target and verify at <regular intervals>.
Retention Data Number of in scope records past data retention policy. Frequency: quarterly Target: 0
Device Management Device % of devices managed per PCI requirements. Scanners and/or configuration management agents are good sources to produce this metric. e.g. 1.4 Install personal firewall software on any mobile and/or employee-owned computers 2.2 Develop configuration standards for all system components. 5.1 Antivirus use and management.
Device Vulns Device Number of unaccepted patch & config vulnerabilities beyond predetermined time frames e.g. sev 5 within 30 days. target: 0 source: scanner frequency: as frequent as feasible for your org. prerequisites: policy stating patch SLA in days per vuln severity level, identified asset group by business importance. Focus on PCI requirements 6.1, 6.2, 11.2.
Intrusion Detection Device % of in scope system traffic monitored by IDS/IPS on a <quarterly> basis. target: 100% Focus on PCI requirement 11.4.
Integrity Monitoring Device % of in scope systems with file integrity monitoring on a <quarterly> basis. target: 100% Focus on PCI requirement 11.5.
Firewall Review Firewall % of firewall and router rule sets reviewed <at least every 6 months> target: 100% Supports PCI requirement 1.1 “review firewall and router rule sets at least every six months.”
Default Credentials IAM # servers/infrastructure devices with default credentails target: 0 source: scanner, manual A&P frequency: <depends on org e.g. quarterly> Focus on PCI requirement 8.
Terminations IAM # of Employee terminations outside predetermined time frames Target: 0 Focus on PCI requirement 7.
Role Verification IAM % of users with completed role verification per schedule target: 100% source: manual or automated frequency: semi-annual Focus on PCI requirement 7.
Credential Strength IAM % servers/infrastructure devices with 2-factor or password complexity reqs target: 100% source: config review, manual A&P frequency: semi-annual Focus on PCI requirement 8.
Incident Response Incident Response Number of incidents not handled in accordance to documented incident response procedures. target: 0 Focus on PCI requirement 12.5 and 12.9.
System Monitoring Monitoring % of in scope systems monitored per PCI requirements. target: 100%
Unauthorized WLAN Monitoring Number of unauthorized WLANs identified on a quarterly basis. target: 0 Focus on PCI requirement 11.1.
Visitor Badge Return Physical Security Percentage of visitor badges returned per <time period>. Focus on PCI requirement 9. The goal is to identify an inexpensive metric that provides some attention to physical access. The assumption is measurement in one area will increase the effectiveness of others.
Policy Review Policy % policies reviewed on an <annual basis>. Focus on PCI requirement 12.1.
Closed Risk Dispositions Risk Assessment Number of risks identified during the annual assessment without a risk owner and disposition e.g. accept, mitigate, transfer. target: 0 Focus on PCI requirement 12.2.
Vendor Remediation Vendor # of vendor security findings that have not been addressed within committed time frames. Helps support PCI 2.4 Shared hosting providers must protect each entity’s hosted environment and cardholder data.
Vendor Assessments Vendor Percent of vendors who receive security assessments within policy e.g. vendors with sensitive data/services must be reviewed semi-annually. Helps support PCI 2.4 and 12.8. Shared hosting providers must protect each entity’s hosted environment and cardholder data.
  • An interesting observation is that our MoneySec list of metrics only contains 15. Recall the MoneyBall inspired list of metrics is a candidate list to identify key controls that correlate with reducing incidents. If all our hypotheses were correct, we could show that compliance focuses on too many things that don’t matter (but that’s another post).

I fully understand you’re not going to spin up a metrics program for all 23 without some demonstration of value. So I propose you start with one metric, my favorite, that covers many PCI requirements: number of scanner-sourced vulnerabilities past due. This is a great metric because it requires:

  • you regularly scan
  • have a process to rank vulnerabilities
  • have a pre-negotiated service level in days to fix vulns based on severity level
  • have the ability to age vulnerabilities

This metric knocks off all kinds of PCI reqs e.g. conducting scans, mitigating vulns, default creds, patches, config vulns, everything a scanner catches that’s listed in PCI.

Here’s an example using our Vuln Tracker tool to age vulns from popular scanners. Note you can use other tools or a spreadsheet (as Brian Keefer demonstrated in our MoneySec presentation).

vulnerability tracker

This example shows a couple hundred vulns overdue, some older than 3 months. From a metrics point of view, you could set a target value for Overdue Vulns at say 10 vulns per month. Here’s how that might look using our Metrics Manager (again, any old spreadsheet will suffice).

This shows the history of mitigation performance and how well we performed to our expected target.

What do you think? Does the simple fact of looking at something change its behavior? In my experience, yes. So if you have the annual privilege of meeting your QSA, see if you can save your company money by maintaining compliance vs. getting compliant on an annual basis.

Please do provide feedback on the list of metrics above. It was just me and my pint who produced them and I’m sure they can improve!


Money$ec Evolved Slides

0
0

My first BSides will not be my last. A huge thank you to all the sponsors and volunteers. The BSidesSF folks will publish links to the recording and slides but I’ve had a few requests that couldn’t wait. Another Thank You to Brian Keefer who masterminded the metaphor from Money Ball (long before the movie…).

Let me know if you have any questions, so much is lost without the supporting anecdotes.

Money$ec Evolved


Make a Difference Webinar with Caliber

0
0

Apologies for not linking to my webinar with Tab from Caliber Security. It’s a fun filled 45 minutes with me jawing on about prioritizing and measuring risk. Who doesn’t want more of that…

Event Landing Page

Warning, it’s a webex recorded event so you have to install the app. Here’s the direct link.

 


Source Boston: My Communicate Risk Slides And More

0
0

I had a great time at Source Boston. Many good times, talks, and connections. Source Boston had the best combination of technical and business oriented talks I’ve ever experienced at a conference. Cheers to the organizers for striking a great balance. The videos aren’t up yet but many of the slides are posted – check them out here.

Here’s the slides from my No Victims: How To Measure & Communicate Risk.

In the business track, one of the most popular sessions was Designing a Compelling Security Awareness Program by Bob Rudis. Find the slides here but grab the video when available.

There are a few others I recommend and many more I want to watch. I’ll highlight others when the vids are posted.

Also, for those not familiar with Boston. The city has two bars where you can still enjoy a good cigar. Coming from Seattle, that’s worth the trip alone!


Your Security Executive Dashboard

0
0

Sometimes for fun, I like to concatenate as many buzzwords as possible. How about: our cloud GRC dashboard provides risk intelligence leveraging big data visualization. I bet someone has a copyright on that one! I don’t mind general terms as long as the author gets to the details quickly. So when I say Security Executive Dashboard, I mean:

  • What: two page, printable document to communicate key messages e.g. current state, performance, trends, and progress against plan. It can be viewed on line but I prefer the tactile option if you have high quality color printing.
  • Who: For non-IT Security leadership. The dashboard should convey its points without CISO assistance. This is difficult but very important.
  • Why: IT security is a cost center with a difficult ROI story that usually gets the spotlight when something goes wrong. It’s important to communicate between these times to demonstrate your contribution to the organization, set expectations, and communicate key messages. A successful dashboard is difficult to measure. One measure may be the effectiveness of your budget process. I can’t promise your budget will increase but the outcome should be in context of the business vs. arbitrary allocation when budget leaders are well informed.
  • When: Quarterly updates, revisit graphics and format annually.

I’ve helped build a few of these over my career, one earlier this year. The fascinating part is how different each was. Given the target audience, the dashboard is a powerful communication vehicle and must match the political tone each CISO wants to set. That’s why I titled this “your” dashboard. There’s a whole menu of stories to tell. The trick is identifying the four or five stories that support your organization best.

Before I get into the menu of options, here are a few items dashboards have in common:

  • Visually appealing: more eye candy the better to attract attention and convey a polished message. Yes, sometimes I feel more like a powerpoint jockey than a security pro. I’m sorry but if it’s ugly or an eye chart, the only thing communicated will be your career ceiling. Depending on your budget, sometimes a professional graphics designer is needed to make the story pop. Remember though, this is a living document and requires updating. Is sustained spending for graphics design the best use of resources?
  • Hub-spoke process: odds are the dashboard will stress the team because they’re a pain to update – no matter how much you automate. Please do appoint one person to collect updates.
  • Desire to automate: Depending on the stories you want to tell, your automation options may be limited. Faster-cheaper is always the goal. However I suggest not worrying about automation to start. If you tell the right stories, you’ll have an easier time justifying an investment to automate. Strategic dashboards leverage automated data but crafting the deliverable is usually manual.

Let’s jump into the menu. I prefer a present-past-future approach to my stories. I try to give each story a graphic and a text area to close the story. By “close,” I mean the ability to answer questions, not generate them. This is actually a good QA technique; does this visual raise any questions?

  • State of Security: What’s going on across your team? Red, Yellow, Green icons followed by key sound bites. Highlights and Issues fit here also. I prefer a table design for readers to quickly navigate.
  • Security Events & Incidents: If you have the means to quantify the volume, type, and man-hours to investigate events, this can tell a great story. An even better story is to classify incidents by impact categories. A comment I often hear is, “it doesn’t look like we’re doing anything.” Thus, it’s important to treat incident summaries as an outcome measurement of preventative controls. Communicating year over year performance is even better.
  • Current risk posture: If you have a risk register, some folks like to communicate risk categories on a heatmap. Every category should have a status update for on-going action.
  • Risk trends: If you use a heatmap to communicate risk, arrows indicating risk trends can be effective. Again, an explanation needs to explain why and what you’re doing about it.
  • Risk posture changes over time: If you have new risks or have recently mitigated key risks, an arrow connecting the past to present risk position works great.
  • Initiative performance: Execution is king. A simple gant chart with start/end dates, project status, and % complete helps summarize all the inflight work.
  • Team & money allocation: sometimes it’s important to show where  the time and money is going. This can be a key message for under resourced teams where people ask, if they’re so busy, what do they actually do?
  • Process maturity progress: If you don’t have a mature risk estimation process, communicating actual vs. target maturity can be effective. Many folks also overlay a benchmark maturity value.
  • Top metrics: Actual vs. target performance always tells a story. The trick is to make sure the audience cares. Senior leaders may not care about A/V updates or patches but they might care about % of devices managed for security. Rolling up tactical metrics is a great way to get to 5 or 6 that communicate your posture as a whole.
  • Budget performance: Financially minded teams may want to communicate actual vs. planned and forecast.
  • Compliance: It is what it is. Number of audits, their findings, and mitigation progress is a story to tell.

Recall, each piece of eye candy needs a text area to close the story. I prefer having a two-sided sheet. One side has the fancy pictures, the back page then has tables to answer any questions inspired by the visual. Here’s a basic template. I just pulled five stories to tell as an example. Your dashboard should be different.

 

Back page Template:

An even more basic design template:

The old adage that less is more applies here. Every team I’ve worked with wants to put too much in this communication. I believe it’s far more important to tell a couple stories well then many half-baked. If you tell great stories about risk, metrics, and initiative progress, other leaders will reach out if they need to know about budget performance or other areas. The goal is to demonstrate value, not communicate everything you do.

Finally, if your organization really gets into all this communication overhead, consider expanding the annual update into a booklet that provides more offline reading material. Communications like these really do take a lot of time. Be sure to get exec support before embarking on the journey. Also, be aware how your peers will respond. Will they embrace your executive communication, feel threatened by it, or maybe they’ll even follow suit. Plagiarism is the greatest form of flattery. Construct your dashboard well and you might be the next buzzword!

What say you, is this worth it? If so, what stories do you like to tell?


Vulnerability Management From Scratch

0
0

When it comes to classic processes like identifying, prioritizing, and tracking scanner-based vulnerabilities, I like to dive right into the deeper waters of performance targets and service levels. Who still worries about about finding vulns and deciding which should be fixed? Turns out more than I thought, especially for mid-sized PCI shops in the early stages of an infosec program.

I mentioned this to a friend, Ed Marlow, who’s (currently) an independent consultant. Yeah, larger orgs worked out the basics long ago. However where do you go for reference material if you have to build a vulnerability management (VM) process from scratch? For fun, Ed and I started sketching out the process and noting public examples. My first stop was ye ol’ NIST site. Their Creating a Patch and Vulnerability Management Program guide is on par with other NIST work and covers much of “what” needs building vs. how.  I always learn something from NIST, even if it’s just my relief that I don’t have to do everything they say! Of course there’s a ton of blog and article content  on VM e.g. a pretty good ISACA article popped up. Instead of rehashing content, Ed and I looked to see if certain topics needed more attention. We found a few under served areas: focusing on risk vs. just vulnerability, prioritization examples, and mitigation performance levels.

Here’s a slide I use to place these in the larger process:

It’s All In The Risk

While some reference materials do a pretty good job, I prefer to jump right to the risk conversation. The only person who cares about a raw CVSS score is the PCI auditor. Everyone else needs to prioritize their work and focus on risk – what is the likelihood and severity of an agent exploiting that vuln? Plus, we don’t have time for an exhaustive analysis. Our workflow needs to scale. We need to prioritize 20-30 (or more) vulns in a one hour meeting with security, application, and sys admins who rather be doing something else.  It is important to have an exception path in this process to dig deeper in the risk equation for contentious or costly vulns. Otherwise this process needs to be putting vulns in buckets vs. debating if something is important.

Step 1: Impact

The best way to scale prioritization is through defined tables. The subjective answers should all be in buckets so the prioritization group only has to focus on which bucket to choose. Here’s where I say to Ed that anyone building a VM program already has a data classification policy. Ed just looks at me…

Ok, since we’re focusing on VM I’ll share some of my web searching skills for data class references: ISACA, IIA, even Infosecisland! The key is to make sure you include as many impact examples in your data class definitions as your approval committee can stomach. Each data class should include examples of regulated data, recovery time objectives, and descriptions affecting the financial, brand, and strategic impacts for loss of confidentiality or integrity. If your official policy is too high level, add examples for your VM process so it’s easy to rate the impact in the prioritization meeting. Is the affected asset High Business Impact, Moderate, or Low? Moderate. Everyone agree? Ok. Now for likelihood.

Step 2: Will It Happen To Us?

Given all the data from the vulnerable vendor’s advisory, scanning vendor, and CVSS, the general questions are answered. The important part is to determine how special you are. Does the nature of your network, system builds, or compensating controls raise or lower the chance of successful exploitation. Plus, will damage be amplified e.g. propagated, or minimized/contained? Here’s a simple example to help the group raise or lower priority.

Step 3: Bucket Time

Selecting a mitigation time frame by asset class and vuln priority is straight forward. For example,

In my experience, defining the data ranges is anything but straight forward! This is the perfect time to sneak a peak at a RACI table. Obviously the asset owners are accountable for prioritizing said asset. It’s the control owners (app/db/sys admin) job to commit to a service level given their costs and constraints. It’s security’s job to prioritize the vuln and measure the overall performance of the process. If you’re new to RACI‘s, they can save a lot of consternation. They’re also management overhead so only use when needed. Ed put the following together in 5 minutes so there, I’m not the only person who does this. (remember, there can be only one Accountable)

How’s IT Going (clever?)

I’ve written before how I think the best measurement for VM is % of vulns mitigated per Service Level. I think it’s important to emphasize because patch and config is such a basic IT service. If your company struggles in this area it’s best to make it visible and ensure management wants it that way. Aging your vulns and comparing against a service level can be tricky depending on your tools and volume. However the visual is definitely worth it (snip of our tool below).

The added bonus of measuring VM performance is to see if management wants more measurement and more conversations about what’s acceptable. Before long you just might be managing more than vulnerabilities!

What say you, is VM a boring, old topic or is it worth discussing?

 

 

 


Winning the IT Security Compliance Game

0
0

I’m sure you all follow the New School blog and have read Compliance Lessons from Lance. My take on the post is to find a way to position compliance from a necessary evil to a necessary evil to achieve business success. Since we can’t directly change compliance standards to focus on just the key controls, some folks define winning as spending as little time as possible across all controls. I think the winning strategy is to invest in controls related to business success, then do the bare minimum on the rest.

One great thing about the New School is problems come with solutions. This time around they reference one of my favorite pieces of research in the IT Controls Benchmark. If you’re not familiar with the ITPI’s work, please go now. One of the most powerful results in the research is the “three controls that predict 60% of performance.”

  • Standardized Configuration Strategy
  • Controlled Access To Production Systems
  • Process Discipline

The first two are pretty straight forward. They’re easy to define, develop metrics, and determine how much you need over time. “Process Discipline” is more general and can apply to the whole shop. @RealGeneKim was kind enough to define process discipline in a tweet:

“Culture of controls that are defined, monitored, enforced; especially around chg and configurations”

So what does defined, monitored, and enforced mean? Here’s my brief opinion but check out the ITPI’s work for more (and better) detail.

  • Defined: document what the process does, for who, and why it’s important. Create a swim lane and RACI for key steps and stakeholders. Identify key outcomes and an associated metrics. For each metric, draw a line in the sand for acceptable performance with a target value.
  • Monitored: allocate sufficient resources to collect and report on the outcome-based metrics.
  • Enforced: identify the audience for the metric and schedule appropriate intervals to review actual performance against target.

Obviously it doesn’t make sense to mature every process to the same degree. So which processes should we instill some discipline? Gene focuses on the first two, how about a process to stack rank the rest and a plan to knock them off over time?

Perhaps the below to follow the first two:

  • Incident Response
  • Event Management
  • Secure Development
  • Vulnerability Management
  • Risk and Spend Prioritization
  • Asset Management
  • Vendor Management
  • Continuity Management

I don’t think it matters what your list looks like. The fun is in the fact you have one. The obvious criteria to select which processes to improve are:

  • Relevance to revenue growth e.g. Vendor Management if you’re expanding, Secure Dev if you’re improving production apps, Change Control if outages affected revenue.
  • Relevance to cost reduction: mature areas associated with incidents and where you spend the most time and money e.g. access control, monitoring.

You might have some secondary criteria e.g.

  • Past Incidents
  • Processes your money-makers hate the most e.g. device management, authentication, access control, training, compliance (wait, that’s what we’re doing here!).

Now you have a process to prioritize processes… I’m a big fan of gradual plans to improve so you can do your day job. I like the technique to bucket work items into foundation and investment i.e. run vs. improve the business. Allocate a % of your team to improvement and you’re on the path of winning the Compliance Game.



What matters to you?

0
0

Lots has been written why measuring current control performance contributes to the answer of “How much security do we need?” If you measure what matters, does tactical control performance matter? Maybe it matters to you but does it matter to the business? Associating technical controls to business performance is difficult. I almost always hear, “how does patching this server vuln within 90 days help the business win?” Great question, and if your business doesn’t know the answer, it’s definitely IT Security’s fault.

Ok, off my soapbox. I’ve written about techniques to present metrics, define targets, and visualize performance. Perhaps I obsess over measurement too much but I’m comfortable with my obsessions.  Over the last couple years I’ve noted what folks measure and what they wish they could.

Before I get to the list, I have to include some guidelines:

  • Each metric must have a reporting duration e.g. quarterly, annually
  • Each metric must have a defined target per reporting duration
  • Significant sampling may be used where appropriate
  • Metrics should address coverage or performance e.g. % monitored or # incidents
  • Every metric must be associated to business objectives
  • Executive rollups are % difference from actual vs. target values

I should also define categories mapped to business priorities e.g.

  • Increase Revenue: develop new applications or support business initiatives
  • Protect Brand: minimize PII loss, availability, public breaches, law suits
  • Support Business: protect IP, keep existing revenue streams working
  • Reduce Costs: improve the bottom line
  • Comply Efficiently: optimize regulatory compliance costs

Executive Metrics (count 24)     

Since managers shouldn’t see everything, what stories are important for your business success? The following are cherry picked from the larger Operational Metrics list later in the post.

Security performance supporting availability and IT responsiveness goals        

This one may be too tactical for the CEO but relevant to an audit committee interested in control performance.

Weighted average of % difference of actual vs. target values of the operational metric group:

  • Access
  • Device
  • Monitoring
  • Vendor
  • Change Control

Security performance supporting Business Line growth              

  • % of projects integrating SDL checkpoints
  • # of High impact incidents
  • # of Medium impact incidents
  • # of Low impact incidents
  • % of strategic risks with a treatment decision
  • (If applicable) application development: avg. # hours at risk: from notification of P1 security bugs to production update
  • # of applications with one or more P1 security bugs identified in production

Employee performance               

  • # of High rated incidents related to Employee behavior
  • % of Employees receiving role based awareness activities

Compliance – all from below
Program – all from below

Operational Metrics     

Now the big list (please forgive the cut’n'paste spacing below)

Application Security Team Metrics         

  • % of projects integrating SDL checkpoints and deliverables          Increase Revenue
  • % developers with security training         Increase Revenue
  • % of applications risk ranked       Increase Revenue
  • % of high impact applications with production pen tests                Increase Revenue
  • # of applications with one or more critical vulns in production     Increase Revenue

Application Development Team Metrics             

  • # of static analysis warnings        Increase Revenue
  • # of static analysis false positives              Increase Revenue
  • % of false positives         Increase Revenue
  • # of repeat security bugs              Increase Revenue
  • # of p1 security bugs in code-complete drops to QA        Increase Revenue
  • # of p1 security bugs pre-prod   Increase Revenue

DevOps  Security Metrics            

  • % of teams with automated deployment processes (dev environments = production)    Increase Revenue
  • # of environment differences found in dev from production during deployment               Increase Revenue
  • avg. # hours at risk: from notification of P1 bug to production     Increase Revenue

Access 

  • % employees de-provisioned within policy          Protect Brand
  • % file shares with appropriate access control      Protect Brand
  • % user permissions or roles verified appropropriate        Protect Brand
  • % servers using directory services or centrally managed                Protect Brand
  • % privileged accounts reviewed for appropriateness       Protect Brand

Data      

  • # of senstive data pattern matches via email e.g.  PII, PCI, business defined         Protect Brand
  • # of PII or PCI pattern matches on unapproved file servers          Protect Brand

Employees        

  • % of Employees receiving role based awareness activities            Protect Brand

Change Control               

  • # of unauthorized changes          Support Business
  • # emergency changes    Support Business
  • # emergency changes related to security              Support Business

Corporate Devices         

  • % of servers with owners and classified              Support Business
  • % of servers enrolled in vulnerability management process         Support Business
  • % of high impact servers compliant to minimum baseline standards         Support Business
  • % of vulns fixed within within SLA (may be a dupe of the above)                Support Business
  • % difference of enumerated servers vs. cmdb            Support Business

Workstations/Laptops/Tablets

  • % of workstations enrolled in vulnerability management process              Support Business
  • % of workstations compliant to minimum baseline standards      Support Business

Mobile

  • % mobile devices enrolled in management platform       Support Business

Policy & Standards         

  • % of policies reviewed per schedule            Support Business
  • % of minimum baseline standards reviewed per schedule            Support Business

Monitoring & Response              

  • % production servers monitored              Support Business
  • # of tier 2 investigations                  Reduce Costs
  • # of H,M,L incidents          Reduce Costs
  • # hours, mean time to recover      Reduce Costs
  • # repeat cause incidents                 Reduce Costs

Risk Management          

  • % of strategic risks with a treatment decision      Increase Revenue

Vendor Management   

  • % vendors with risk ranking         Support Business
  • % of vendors assessed <per schedule> Support Business
  • # of overdue vendor issues         Support Business
  • % of new contracts following security process    Support Business
  • % of outsourced hosting (cloud) solutions complying with security policies

Compliance       

  • # of  audits with no significant findings   Comply Efficiently
  • # of overdue findings     Comply Efficiently
  • # of repeat findings        Comply Efficiently
  • # of policy exceptions    Comply Efficiently
  • # of overdue exceptions              Comply Efficiently

Program              

  • % projects completed on time and budget           Reduce Costs
  • % budget Plan to Forecast           Reduce Costs
  • % of processes defined in catalog, RACI, metrics               Reduce Costs
  • plus-minus change to security service satisfaction            Reduce Costs
  • % SLA’s met or exceeded             Reduce Costs

Quite the list! Count breakdown by business relevance:

  • Increase Revenue: 15
  • Protect Brand: 8
  • Support Business: 17
  • Reduce Costs: 6
  • Comply Efficiently: 5
  • Total: 51

Please let me know if you’d like the xlsx of the above. Also, please push back if you think all this measurement is too onerous. If measurement wasn’t built in when the control was implemented, metrics can be really expensive. Deciding if you should even measure performance is a great indicator of your company culture. Deploying a new control without including the expense to measure is also telling…

As always, feel free to contact me if you’d like to discuss measurement or need assistance getting it done. I have to leave you with some metrics eye candy:

metric example


Measuring Security Performance: Governance or Whistleblower?

0
0

I love helping security teams measure control performance (metrics) and improve risk analysis and management programs. Providing visibility into current performance and putting the data in context of business goals helps improve decisions. What happens when the executive team doesn’t want to know, how about they know but refuse to make a decision, or perhaps they make a decision that’s obviously not defensible if an incident ever went public?

Some security teams have formal job descriptions to provide information security governance. To me this means they provide visibility into performance, help define performance expectations, and facilitate decisions when expectations aren’t meant. However I’ve seen these teams pull their hair out because the control owners or decision makers prohibit or resist supporting their mission. Sometimes the resistance is reasonable e.g. too busy, sometimes it’s political e.g. don’t want to look bad. Either way it puts the security team in a pickle. Do they swim up stream, grab the data, and report to the Board without the control owner’s support? Do they perpetuate the situation by going through the motions of spot audits without closing the loop on findings? If there’s no mechanism for you to exercise or establish a governance process, should the security team become a whistle blower or take a complicit role?

I started thinking about analogs outside IT. Most are too controversial so how about the recent Rutgers Controversy? The basketball coach abused his players over a long period of time. I don’t think the NCAA has a compliance check-list of “do not abuse players” so it probably wasn’t a compliance failure. However I believe there are governance processes to protect the welfare of students. So when governance failed it took  a whistle blower to leak video evidence to the public. + 1 for social media. I assume some accountable group e.g. Board of Regents, is seeing how far up the ladder the negligent decision making went.

Now I’m not equating control owners who ignore obvious control deficiencies to abusing players or any other situation that requires a whistle blower. I know many devs and IT ops folks and they love their children too. But the pattern is pretty interesting. How can the security team fulfill their governance function without being a whistle blower? It’s a tough question and I don’t have all the answers. I am happy to share my experience and what I’ve observed to work well.

  • Baby steps: whatever path you choose to improve visibility and encourage treatment decisions, take it slow. The business hasn’t collapsed or may be doing just fine. Highlighting deficiencies when there isn’t evidence of impact might get you a spot in a dark corner.
  • Create the risk story: If you’re measuring something irrelevant to the business, you’re done. Create a well formed story across the potential impacts and their frequencies related to your control data. Much has been written here about framing risk so I’ll move on.
  • Relationships: Obviously support for governance is a top-down culture issue. Get to know the decision makers and control owners. Earn as much goodwill as possible, you’ll be making withdrawals soon enough.
  • Bottoms-up: Consider working directly with a control owner you think will support you before blowing any whistles. Understand their situation and emphasize with their position. If you have experience in ops you’ll have a much easier time. See if you can get them to emphasize with you! There’s a reason why things got the way they are. Include the background when you tell the risk story. See if the control owner wants to cooperate in a solution or prioritizing the solution against their backlog. If these efforts fail, time to…
  • Move up a rung: If the control owner won’t play ball, contact their boss in an informal setting and ask their advice. Every manager knows what you’re doing and will respond in an overt or covert manner. It’s unlikely they’ll just ignore you. Let them know the story will be told and you need their support. If you’re fortunate, you’ll be able to associate your risk story to an objective important to the control owner. If not, time to communicate with someone who has an objective associated with the risk, the…
  • Business Owner: Much has been written about mapping control performance to the biz e.g. my What Matters To You post. If your story falls on deaf ears or you get smacked down, you’re getting closer to the whistle.

Internal politics are snowflakes and the above might be exactly wrong for you. However I hope the above inspires some ideas. The ultimate answer is a governance process that reports to an accountable group outside dev and IT. The trick is to create a process that partners with the control owners. No one likes surprises, especially when they make you look bad.

On a positive note: if you have no choice but to blow the whistle and it falls on deaf ears, there are plenty of businesses looking for your skills and experience. Please share your stories or correct my thinking in the comments. Or contact me directly if you need an empathetic ear.

whistle of last resort






Latest Images