All The Sales President's Men

You ready? Alright. In 1976, Dustin Hoffman and Robert Redford starred in the political thriller "All the President's Men," portraying the reporters Carl Bergen and Bob Woodward as they worked to uncover the events of the 1972 Watergate scandal and all the resulting cover-up that happened as part of the Nixon administration. What was striking to me was the apparent number of people who actually knew about what was going on, but there were very few leaks. And I found a lot of similarities within the sales community. So I named my talk "All the Sales President's Men" because while it's not exactly a conspiracy, there is similar inside knowledge that remains unsubstantial and unconfirmed unless you're part of a community run by

a sales president. And as technologists and hackers, many of us have skills in open source intelligence gathering or social engineering, but we don't necessarily stop to think about how those same skills are used against us to influence our buying decisions. And you might not be the one with purchasing authority, But if you are technical, you're typically asked to participate in a product evaluation, give requirements, make recommendations based on your expertise. You're still relied on in the sales process. And I believe that if you are aware of the tactics that will be used by salespeople, sales engineers against you in each phase of the sales process, then you can inoculate yourself. against being taken advantage of and in some cases even make the

vendors work for you. So before I get started I want everybody to try to think about what was the last purchasing process you were involved in. It could have been something at work, it could have been a personal purchase like buying a car. Think about the vendor interactions that you had and keep that experience in your head as I describe some of the common practices and tactics and hopefully by the end you'll be able to see what was actually used against you. So I'm Patrick McNeil, I ask not to be introduced. Not that you know it since I'm up here, but I'm actually an introvert and I'm an INTJ in the Myers-Briggs scale. My wife

calls me the absent-minded professor, thus the graphic there. I get completely consumed with things, ignore everybody else to a serious point of just being unaware of my surroundings, and I have way too many browser tabs open in my brain. So I'll skip my detailed employment history too, you can look me up on LinkedIn. But I started my career as a programmer, I've got a technical background. I ended up transitioning into network engineering and then later security. And in 2006, I started working in a lab that the sole purpose was to evaluate vendors for inclusion in third party solutions. So I worked with a lot of vendors, it was a rotating door. I slowly started to see some of the same tactics that were used by different

vendors. And then in 2011, I was actually hired by one of those vendors that I evaluated for a sales engineering position coming out of a technical role. And for someone who had been technical with no sales background at all, it was an eye-opening experience. I've been a sales engineer or held sales engineering positions for over six years now. So I've been trained in multiple sales methodologies, and I'll explain what that really means, used by three different companies that I've worked for in that time. I do need to make it clear that I am not speaking on behalf of my employer today. and I will also nod to my imposter syndrome. I am not the most

qualified person on every single topic I'm going to talk about. I'd love to hear stories or feedback from you guys after the talk, whether it's in here or out in the common area. I got the idea for this talk after I saw a sales prospect make what I thought was a really horrible decision. They got sucked in by a tactic that my competitor used, And I really wanted to explain to them exactly what had happened, but at this point it's too late. They've made up their mind. It's not too late for you guys though. So I wanted to give you insight into what happens in each phase of the sales cycle and help you spot

what's being used. Now I'm gonna preface this with For those of you who are not in a sales or marketing type role a lot of what I'm going to talk about is going to be Something you've already suspected and I'm just going to be confirming your suspicions But I'm doing this in a very methodical way laying it out from end to end If you are in a sales or sales engineering role, you're probably going to be nodding your head to a lot of this saying yeah I know and we also do these five other things you didn't mention and Keep in mind, I am just scratching the surface of this space. And I'm sorry, as

much as you might like me to, I'm not going to call out specific vendors in this talk, mostly because I just don't want to get sued, frankly. But this is also not about bashing specific vendors. It's more about inoculating you. So I'm gonna loosely organize the whole discussion into four major sections. The first one, following the theme of "All the President's Men," is the conspiracy, covering the communication that starts at conferences, morphs into some of the marketing you receive, like vendor emails, and then the cold calls that you'll eventually get. the story as it progresses the initial contacts that are made by the sales team the competitive intelligence gathering that's used to kind of build the story for you to explain what the other

vendors can and can't do and the rfi and rfp process when you decide all right i want to start looking at vendors seriously the investigation when you actually bring that vendor into your your company or whatever to do a proof of concept or a bake-off test and hear what they have to say about their product some vendor awards some people refer to and independent testing and then lastly this talk wouldn't be any good without some of the truth so we'll cover how to avoid making a bad purchasing decision alright so your first contact with a vendor will obviously typically be at a conference some kinda marketing you receive like as part of an email campaign or your cold call. So let's look at each of these. Now obviously

conferences are all about generating leads for the vendors that are there presenting and have their booths, right? They need to have prospects to sell to later. So it's all about getting your contact information. So you will typically have booth greeters that are there aggressively trying to scan your badges and yeah, That is the currency of the day in order to get all the free swag, right? Everybody understands that that contact information is going to be collected. What annoys me, of course, is the greeters that are like all the way out in the aisle trying to scan your badge. Can I scan your badge? Can I scan your badge? And you haven't even like engaged them

in a conversation or anything yet. And I've even actually had people walk up to me and scan my badge without asking. Like I'm literally standing there talking to somebody and just... typically after a badge scan you're considered a cold prospect. So what that means is you're just on an email list, I'm sure after, if any of you have gone to Black Hat you know exactly what I'm talking about. The vendor deluge of emails right after Black Hat, right before the next Black Hat, it's all on a schedule. If you had a good conversation in the booth, then you're typically going to be ranked differently. So you will be either a warm or a cold lead at that point, which means you're

definitely more likely to receive some targeted communications. Now if you are serious about your scan privacy, you like that guy, right? If you are serious about your scan privacy, you can do a couple things. Number one, of course, submit fake data when registering. make up a domain name that sounds like a business and this is important and I'll tell you why later register that domain name and just use that domain name for conferences or spam email that you know you're going to receive If you're required to include a phone number, just use a Google Voice number, because you can end up just blocking callers if you need to. And it can actually be pretty interesting

to see, based on the individual email address for that conference, which vendors targeted you. Not that you really want to study it, but it's just interesting to look at. So create a different email address for each one. And if you are on Gmail, you can also set up email aliases on your account. So based on the alias, you go into different buckets. The second thing of course you can protect your badge from being read without your permission So this would typically be somebody who's either not in the market to buy something or maybe you think you can convince the the person scanning for swag to give you the swag without the valid scan a Lot of vendors aren't going to do that, but maybe you can talk

them into it So you can for NFC badges that are in use at places like black hat and obviously this is like my old badge from 2014 you can put the NFC card in an RFID sleeve inside your black hat lanyard or whatever, and then, "Oh, jeez, I don't know what's wrong," and then take it out when you want to be scanned. Or, of course, you can just completely sabotage the card. So I just, having fun after a couple beers one afternoon, I'm like, "Where is this chip anyways?" So high-power flashlight, "Hey, there's the chip right there." Then I marked where the inductive coil wires were thinking, well, I'll just pop those real quick and

then it won't be readable anymore. And I tried killing the badge first by, um, So I marked the wires and then I attempted to murder it with a nail, just like, but I think the nail probably slipped to the side or something, it didn't really work. So then I'm like, well duh, you got a chip here, just kill the chip. So get a little X-Acto knife and one little slit right through where that was, it's almost completely unnoticeable and you can kind of push it down. It's an unreadable card, it's like, hey look, this is my card, you know, whatever, it's completely dead now. QR code badges are used by some conferences. These are typically just read with an Android or iPhone app. And they can also

be rendered unreadable with just a few extra dots from a fine tip Sharpie. So this was my badge and this was the corresponding base 64 read that I got when I just used a barcode scanner. And here it is afterwards in an unreadable state. besides being a crappy photo, the difference is not really easily seen. I mean, this is blown up a bunch of times, right? So at a one inch by one inch form factor scanning from a foot away, it's almost impossible to tell the difference between those two photos, right? So they're not gonna notice it. Okay, so going back to the whole email address thing, if you do wanna go to a vendor

party for free food or booze, know that that throwaway Gmail or Yahoo account you were using, that's not gonna get you in. It may be that you'll be lucky and they'll have enough spaces and you'll squeak in, but when you register for a party with that email address, you're likely to be waitlisted or just outright denied. The whole point, right, is getting business contacts, so they're just gonna filter some of those out In many cases it's automated, other times it's just somebody striking you out manually. Because of the fact that they're trying to get contact information, if you happen to work for a targeted company, and I'll explain how targeting works later, it's not uncommon

for the vendor to assign somebody like me to find you, track you down. You're wearing your Black Hat badge, it has your name right on it. So I'm looking for you. to come in and I see your name, okay, that's the guy I have to go up to and chat up. If you're at the booth and you really are interested in a demo of the vendor's product, you need to get through the greeter gauntlet. And most vendors will employ this because they need to gauge your interest level before getting you into a demo. Just because you idly walk up doesn't mean you're really a potential buyer. They have to protect the small number of knowledgeable

engineers like myself that are there to actually do demos. If you do get a demo, your status is likely going to be bumped up to either warm or even hot, depending on that conversation, and understand, yes, you will get more marketing out of that. Maybe it's actually something you're interested in, so it's not really unwelcome, though. Giveaways. I see lots of contests on the floor, and you can either fill them out or get scanned. They're not as random as you would hope. If you're not a warm or hot lead, you're not very likely to win at all. Some vendors cheated this. They never said that it was completely random. And yes, I've been asked by

the marketing people working the booth before, "Okay, who did you have a really good conversation with?" Because they're the one that's winning the drone. It happens. The whole point is to build some sort of goodwill with a potential prospect. Yes, there are exceptions to that. There are ones that could be completely random, but that's the way it is. Now, I mean, everybody knows that companies use email campaign software, and every email you receive is tracked for engagement. This is just one example of one of the links that's inside of an email. Yes? Google, Google email, is that what they said that they would not track bugs like this from actually... I don't remember the exact specifics, but Google with their Gmail and

G spot, whatever, their corporate Gmail said that they would stop tracker bugs from working. Would that stop things like this from working as well? I'm not familiar with exactly what they're doing, but you know the answer to that? Yeah. It's typical that most of the modern email clients will allow that feature. particular setting yes yeah um they'll typically just disable html graphics in an email and that will take care of all the trackers and and most uh well it'll take care of all the stuff they could detect what happens or that you opened the email the the other embedding like this is going to be if you click the link they know you clicked on the

link and you know, if, if you click on the link then they know to customize the communications and in which way because every link is to a different topic. They're tracking every single topic in the email. It's not just all going to the website, right? So now they know, oh, Jeremy clicked on this link so he's interested in this web application firewall, right? Um, They also know the, from the trackers, or the people who have the email campaign software, they know the account number of the vendor, they know the specific email campaign named, it might be the pre-Black Hat party push vendor email, whatever, so they know which email you opened, the template that was used, and the specific contact you and what you clicked on.

Yes? - I'm just curious, so if I take that code, it's interceptable, Yep. Yes. Yes. And I'm sure most of you have seen some marketing vendor overreach where you're like, that was kind of creepy because I was on this other website shopping for something and now it just showed up in this feed on this other website. It's kind of, you know, it's overreach. And, you know, you see this all the time. When I was looking to switch jobs, The people that I was looking at were showing up in different feeds, including my employer at the time, which was really weird. It was like they were saying, "No, come back." My wife got mad at me because I was browsing

for auto parts really quickly, like running from the garage and look this up and run back out and look something else up. And then it was like, "We're front and center on her Facebook feed." So that wasn't a good day. So anyways, ads can be targeted to you in different ways. So some of the ways this happens would include the pixel based, which was on the previous slide. So you're basically just, site user is using a JavaScript to drop a cookie in your browser. And then when you visit other sites where they have paid for ad time, you see their ad because you have the cookie. List based, where the company actually pays the website

that you're on to show you ads based on the email database. So you log into site A with one email, you sign into a different site with the same email, hey the emails match, show them the content that was paid for. And then account based, which is a little even more targeted, where they're gonna serve you ads based on your IP address range. So what company are you coming from? to visit me so I know all you're coming from you know widget maker a in your hitting the Cisco website widget maker a go get the account team to go give them a call Now, of course, if you go to a vendor's website and just register for white paper, that's probably going to transition you from cold to warm

or even hot. So, of course, if you want to minimize or avoid some of this stuff, you know, common stuff for avoiding viruses and malware and all that other stuff too. Don't open vendor emails. If you do want to go to their site, go to it directly. Make sure your email settings don't include HTML graphics. And that's, like I said, in typical and most clients, you can turn that off now. Block ads and trackers with ad blockers. EFF privacy badger. And I personally love the DNS black holing with PyHole. Just set it up as an alternative DNS and it filters out a lot of the ads. Easy to set up. Browser settings, you know, disable cookies or delete them regularly. Use incognito or private browsing

mode so you can just wipe that every single time. use a fake email address to sign up for things, or bypass registration for downloads. And this one, I don't know, I tweeted something to a bunch of lawyers online to see if somebody could give me a correct answer, but if you look in the source code, there are a number of vendors who are immature and they will have the redirect link right in the source code for you to go just hit the white paper directly and completely bypass registration. Now the flip side of the marketing trackers, right, you're not being tracked anymore, but the ad tracking can enable communications that are actually more relevant to

you. So maybe you don't want to get rid of all that. I used to run a product security incident response team for a vendor, and we actually used an email campaign platform for sending out our security alerts. So we knew for important emails who opened the email and did they try to download the detailed notification with remediation steps or not so we could say hey you know what I'm just picking a name Verizon or AT&T they didn't go download the advisory they're a big customer the account team should probably call them and go hey guess what there's there's an issue you need to go look at marketing can benefit you too. A friend of mine told me that he'd rather see a Hack 5 ad in his

browser than an ad for diapers. So marketers need to be able to track who you are and things like Amazon where they're coming up with book recommendations based on what you ordered previously. Some of them can get really trippy, but some of that tracking is actually beneficial to you. Now what about cold calls? We call them cold because when the term originated, the person who was calling you supposedly had no idea who was on the other side of the phone. They were just randomly dialing numbers. And that's obviously not been true for a number of years in various industries and especially in the security industry. Every sales team has a targeted list of companies assigned to them. and they work on multiple profile factors from Hoovers

and other corporate databases, and they're actually looking for people that work for their targets. So if I'm a sales team and I'm going to go out and try to find somebody in a particular target company, besides the conference contacts that I got or website contacts where they hit my site, it's typical to use things like data.com connect, which used to be Jigsaw. It's a big contact sharing database. Discover org. LinkedIn, which a lot of us have been suspicious of already anyways, and others to find contact details. Some of these services, however, will also scrape from job listings. They'll take input from other salespeople who have worked in the account and basically get, it's like a gamifying data collection. So they will actually go enter in data about somebody they've

sold to or worked with. So Now you as a subscriber have some very specific internal details like the organization structure, perhaps what other systems they've purchased, who they're customers of, and what languages they use, what build servers, orchestration, server technologies, everything. So walking in, I might be able to assume what some of your challenges are and see where I might be able to fit in from an integration perspective. So all of these criteria will be used to sort and filter which sales team targets you. They're all specialized. So you're likely to be categorized as a lead based on who you work for primarily. So what is the annual revenue of your company? What region of the world or country do you live in? That one's probably way more obvious.

Are you an enterprise or carrier? Education or government? Was there a news article about your company being breached lately? Did you hire new employees or are you going through a reorg? All this public information is used to sort out to who exactly is going to target you. And unless you're somebody very senior, like a C-level employee, your first contact cold call is likely to be from some kind of a business development representative or BDR. And their job is to find out whether or not you might be a buyer and literally to get you to agree to a meeting on the calendar. But before they're actually trying to call you, you've been looked up in more than one database, as well as probably several social media platforms. Some

companies even give their BDRs subscriptions, like full subscriptions to multiple services, so they can go sort and filter and figure out who you are. have either supplied sites like this one, IntelTechniques.com, or even custom tools developed in-house for this open source intelligence gathering. BDRs get training in multiple tools. They get trained in a specific methodology to follow. And they learn basically how to approach you to get you to agree to that initial meeting. They learn to bring up specific interests of yours and basically get you to lower your guard to have that initial conversation. So, sounds a little bit like a social engineering pretext, right? Now I do the same stuff as a solutions architect as well. I wanna know who I'm walking in

to talk to, see if I have something that I can use to build rapport. sometimes it's just to get in contact with somebody. And the other day I was trying to find a contact who worked at a specific company but couldn't find his email address. So I pulled up the OSINT framework by Justin Nordine and it's just OSINTframework.com and I punched in the target's name using this email permutator tool and I got a message allowed me to build a list of 35 common email address formats just based on my target's name. Now what I could have done from there is just plug it all into the BCC field of an email and sent it and

who knows if it would make it to him or not. But what I did is I took that and I bounced it against how I've been pwned dot com. One email address came up. so obviously that might have been the right address and sure enough it was so of course if you spend more time on the site there's lots of other tools that I can use to find out information about you all right so I'm gonna move on to the next topic now because I want to get more into what sales teams do on-site however just know I didn't even touch social media engagement at all so that's a whole other topic so now for the story what happens when you start talking to the sales team if you

start If you actually schedule that first meeting, you're likely to be handed off either to an inside sales or outside sales rep who's likely to bring somebody along like myself, the solutions architect, which is fancy terms for senior sales engineer. And on the surface, my job is just to try to discover what your technical requirements are and describe the products and give you a demo and that, but there's a lot more to it. I'm also there as the technical guy to build rapport with the technical person to get you more comfortable to the point where you're by from us. I am also watching out for what we call landmines, sown by competitors, things that maybe

we won't be able to do that now you suddenly think you need, and maybe perhaps plant a few of my own to get you thinking about, "What do I really need here?" And maybe that other guy doesn't have it. I have to keep in mind numerous competitive capabilities and practice a number of communication skills sometimes called a sales methodology. A sales methodology is just a set of techniques and skills that successful salespeople have used over time. Some companies will promote certain sales methodologies internally and train everybody on it because a it's a valuable skill and B It's an effort to make the sales team more effective and use common language Internally just basically to help things work more smoothly inside

the company most methodologies are about trying to understand your pain points and and what you're really looking for, what point in the buying cycle you're really at, what's going to motivate you to buy something. And many salespeople have been through all kinds of training to identify what your personality type is, read your body language, and just build rapport based on what you say or what's on your desk. Also sounds a little bit like social engineering, right? So one of my goals when I started this talk was to figure out based on the questions that were asked, could you figure out what specific sales methodology the salesperson was using so you knew what was coming? I wanted to give you a decoder so

you could kind of figure that out and unfortunately I failed. There was no real specific word or phrase to reliably identify the sales methodology, but I did see a whole bunch of commonalities, especially in the types of discovery questions, those initial things we use to figure out what you're looking for. And the types of questions that you'll be asked are typically leading situation or problem questions to get you to the point where you admit what are called your pain points. Or to help you realize that maybe this is a problem for me. I wasn't really thinking about that. Like, who would be responsible for a breach if it happened today? Oh crap, that's me. I didn't think about that. Yeah, I know that's oversimplified. Open-ended questions to get

you to describe your problem, just to get a lot of general information. control questions to get very specific information for required capabilities how many branches do you have how many routers are in your network what type of firewall do you need us to integrate with the the very specific technical questions I'm not I'm not going along with my bullets here, sorry, impact, need payoff, or implication questions. So to get you to discuss what would happen if you don't really solve your real world problems, what was the estimated loss from that breach you had last year? And then lastly, summary, just to get you to summarize everything that was heard. So sales methodologies can actually help you

figure out what your requirements are if they're used as part of a consultative sale. So don't misunderstand me and think that I'm saying they're evil in any way. You do still need to talk openly to vendors because if you don't, you're not going to get what you really need out of their solution. You need to talk to them so they build a complete picture of what you're trying to accomplish. The word of caution is just talk to several vendors at the same time, because every single one of them is gonna help you uncover a slightly different puzzle piece that helps you put together the full thing. Now, common social engineering tactics are built into sales

methodologies. So, looking at a bunch of these different methodologies, these were some of the tactics that I found commonly employed. pushing through decision paralysis by using your fear of missing out. So you set limited time pricing, oh there's only so many of these left, or exclusivity of the product, well this is only for our platinum customers. Reciprocity, where you basically give something for free, like conference tickets, dinner, free maintenance for two years, and you expect something in return. You're setting up a quid pro quo. Success criteria, where you get an agreement that once your success criteria are demonstrated in a lab test, that's a win. So now the prospect feels a little guilty that they're not buying from you, perhaps it's a quid pro quo situation again. Competitive differentiation,

where you show how you're better or different and highlight your strengths, but also point out the vendor's shortcomings, really get them to doubt what the vendor is capable of. Inoculating a prospect, I'm doing some inoculation today, but this is telling you that your competitor is going to try to use a specific tactic against you or tell you that their box can do X, Y, Z, but it really can't. So when you hear that, you're immediately going, ah, I noticed this, they are slimy, because the other vendor warned me this was going to happen. Promoting instant gratification through quick start, fast arrivals, turnkey service, everybody wants this, right? Left digit manipulation, which I think everybody's familiar with. It's the old $19.99 versus $20

kind of thing. Limited options, if we are presented with too many options, we get confused, we can't make up our mind. So providing flexibility but not too many options. And then lastly, FUD, fear, uncertainty, and doubt. that if you don't buy their product, you're gonna be breached next year. All right, the next topic, competitive intelligence. It's critical to the sales process in the same way that anybody going into battle needs to know something about their enemy. And there are whole teams in most companies that this is their sole function or a major part of what they do. It's basically building competitive intelligence from multiple sources. As you would expect, just about every company has battle cards. And on these

battle cards are strengths and weaknesses of your solution versus the strengths and weaknesses of the competitor. We will have things like trap questions, the specific ask them this to get them thinking about this specific shortcoming of the vendor's solution. While this is interesting information, you can't necessarily hear, or I'm sorry, can't necessarily trust what you hear about a competitor from one particular vendor. Their intelligence could be dated, it could be colored by their particular prejudice or lack of understanding, like they don't like cloud solutions and they're gonna advocate for everything on premise. Or maybe their intelligence department just really isn't very good. They don't have accurate up-to-date information. The job can be difficult sometimes. Open internet sources are primarily used. There's a lot of information out there. But

most people don't think about things like patent applications, poorly redacted presentations where there's a lot of metadata saved. materials in alternate languages like it was published in Japanese and most people couldn't read it outright but all the info is there or Google file type searches looking for an Excel document you know pricing with this particular vendor's name. Other than the internet you know how is open source or how is competitive intelligence gathered? So at conferences booths are typically staked out by product managers and competitive intelligence staff they will look at the booth messaging, what things are they promoting right now. They may even eavesdrop on demos given to prospects. And even sometimes engage in direct conversation. Walk right up to the person at

the booth and go, what's your perspective on the market? Why do you think XYZ product is any good? And sometimes people will just talk because they really want to show you how much better they are than you. It's just human nature. The other typical angle is new employees who actually bring materials with them. Now, it may not be the most ethical thing, but it happens, and there could be a lot of materials that they already have that are just old sales and roadmap materials that might not have even been marked as confidential. So if it's not marked as confidential, kind of a gray area there. Customers are good sources of competitive intel as well. And I've had customers actually share pricing and even

road maps from competitors, which should be under NDA, but it happens. If you're an ethical vendor, you're gonna say, "Nope, I don't need to see that, you're under NDA." it can be an attempt at leverage in the next buying cycle. They want you to drop the price or something like that. But if you've developed a champion in the account, somebody who's really in your corner, they could be just looking out for themselves. Seeing your product replaced you know next year means that they made a poor buying decision in the first year so they're trying to protect that decision and make sure management knows that well I made a good decision see it's work we're continuing to use it some vendors may even

hire a secret shopper and their sole purpose is to basically pretend to be a prospect, ask for demos, ask for pricing, just get as much information as you're willing to give them. So if you're over eager, you could be giving information to your competition. For a vendor's most aggressive competitors, It's not unusual to even acquire competitor gear and test it in a lab to uncover weaknesses or determine what features you need to develop to enhance or increase the strength of your product. And it's actually typical to offer a highly discounted swap out. So we'll come in and say, "If you give us the competitor gear fully licensed and operational, we'll give you this at like bargain basement prices." Maybe we're even taking a loss on it in this one

instance because you have a version that we really want to get our hands on. Or we'll just go through a reseller partner who sells competing products. You know, they sell from multiple vendors in the same space and just tailor the solution to what the customer prefers. So some of you might be required to buy through RFI or RFP mandated by your procurement department. And the experience isn't necessarily bad if you know what your requirements are up front. But in talking to a vendor, there is a chance that you'll find out something that maybe you didn't know about. The first vendor you deal with, your favorite vendor, if they have any clue and they're paying attention, they will want to help you with your

RFP. So what that means is they're going to provide the laundry list of items that every vendor should support and they're going to introduce some trap setting requirements in there to point out deficiencies of the other vendors. Vendors do hate replying to RFPs. We really do because they're very complicated typically they have stuff injected from your purchasing department they require interdepartmental participation so it's just a distractor from all of our other potential customers not every vendor is going to respond to your proposal if it doesn't actually represent in an initial investment or they could see there's a line to future investment that isn't worth the overhead of them actually doing so. So don't count on the

vendor with the absolute best solution to even respond. So after you've gone through the RFP process and you want to evaluate vendors, You'll typically do a bake-off or proof of concept test of some kind, and you have to ask, "How much do you know about the technology you're trying to select?" We've all read up on something quickly and gone, "Oh yeah, I know everything about this space," but we kind of miss the corner use cases. So definitely pull in people that work for you who might know something about this. Do as much reading as you can. When we come up with our test plan, we're typically looking at industry studies and asking a vendor to

or two for their recommendations and similar to RFPs, I'm going to give you my test plan. You have to compare the test plans to possibly see where some of the vendor shortcomings lie. Find those differences and then ask yourself, are those really any of my requirements? Are they relevant to me? Demos and proof of concept testing is usually optimized and sometimes even rigged to the best possible vendor scenario. So you say, "I want to do proof of concept. Here's our test plan. We're going to use this particular program." And then you really have to take a step back and say, "Well, if you're using a test application or tools or generated traffic, who selected it and why?" It may be legitimately hard to test a

real world scenario, but short of a full implementation, you're going to have to decide, you know, do you use their stuff or your own? So try to talk to other customers, check out third party stories and references. Most vendors can provide them for you. Look at case studies and introduce your own traffic and data and test programs where possible. Vendors actually do optimize their code based on specific industry tests or benchmarks. Even if those tests are not realistic or don't represent real-world threats, they will have little hooks in their code that says, "Oh, if you see this, this is in the OWASP benchmark. Bam, go ahead and throw a detection for that." So your production environment might not look like what was in the

lab. So when you're making your product selection, sometimes it's useful to look at vendor magazine awards and stuff like that. The dirty secret about these is they are not always objective, so you have to look at how the research was funded and who did the evaluations. Most evaluations are pay for play and most vendors have the opportunity to go back and say, "You didn't really understand what we do," and somewhat adjust some of those findings. Companies that sell the reports that they produce to consumers tend to be a little bit more reliable. However, if the evaluation was done by somebody who wasn't really an expert in the space, you can run into some issues like these independent test

labs where I have literally gone into a independent test lab with a product that wasn't really a standardized thing, like it's not a switch. And they're like, well, we need a technician to build the solution in the lab. We need one of your technicians to run the test plan. Oh, could you come up with a test plan? So all they're doing is writing up a report and in one instance a lab actually came back to me and said, "Oh, your competitor beat your numbers. Would you like to pay again and we'll retest?" It's like, "Wait a minute. Now I have to send a resource back." So they were literally blackmailing me. Standards compliance. It's not atypical for a vendor to brag about their standards compliance and tout certifications

like common criteria, but you have to look at what the scope of the testing was One vendor that I used to work for achieved common criteria, but it was only on the administrative interface and not on any of the services interface passing actual traffic. Another vendor bragged about their certification, but the target of evaluation was way off and I managed to get competitive gear and just put their boxes into a complete reboot loop. So, and you have to realize that these certifications are typically for one specific release of software, hardware in this environment. Oh, we're targeting it for government, not necessarily for civilian enterprise. So after this long cautionary tale, what do you do to not get taken? Try

to understand your requirements before talking to a vendor. Talk to your end users, see what they really need and understand what your real risks are and You can listen to vendors in the discovery process, but take a little cooling down time. Find all those corner cases they said you should care about and evaluate them independently. Important to talk about metrics in advance. Great, they said test case passed, but how will you really gauge whether or not the product is successfully doing what it's supposed to be doing? in the testing but then also in production. Is there a way to measure what's going on externally or do you have to rely on the vendors reporting? I mean that might be fine but just make sure that they can

support exactly what you want to track to gauge success of your program. And if you do care about a specific certification ask to see a list of test cases. What was actually tested in that certification? Besides Seeing the product working in your lab, consider all the operational requirements. Does the vendor have services to get you started? I mean, consider the whole process, the challenges, the operational costs that are hidden, like extra licensing for integrations and things. and ask for the vendor's product and feature roadmap. So it's not uncommon for them to say, "Oh yeah, we'll support that next year," when they have absolutely no intentions of it, and it changes with every single customer they're talking to. Lastly, of course, check out user

communities or forums for that vendor. But just keep in mind it is the internet, right? So it's a double-edged sword just like everywhere else people are going to promote, complain, or troll. You only go to Yelp when you're really happy or you're really pissed off, right? All right, so did everybody keep their last purchase process in mind? Did you have any insights into what techniques were used? So I've just inoculated you against the tactics used in every phase of the sales process, so you're now an insider. So pay attention, and hopefully you'll recognize them. Thank you.