How to write effective & successful SIEM RFP?

 WORD OF CAUTION:

   Never been so drunk that you rode a pig like a horse!!.

 


The topic which is hardly discussed enough and to the level of importance it is required. 2 years back I happen to be involved in the process of writing an RFP for very large and strategic Govt Department. The process was not so simple , I remember keeping / managing 20+ versions of the RFP which was continuously being updated, edited and reviewed by the management. I no way suggest this is the way to go on the road to write an effective RFP for any organization, the number of revisions sometime and more so in my case was due to misunderstanding of the technology concepts then for anything else.

It took almost a year before the RFP was ready for submission to the internal procurement department, to be thereafter published online. Best guys in the dept, the procurement dept just see/ evaluate the document for not its verbosity but the inclusion of right tending process, clauses that corresponds to  organizational to applicable Procurement Policy for Govt Organizations. I'm sure in your organizations, there would be legal and others teams be involved in the process.But my reason to write on this topic is to share my experience and things I have learned on the course of writing a successful and effective RFP. Lets start.....

Strategies

• Compliance focus VS Ops focus VS threat Focus Vs Forensic Focus

You mess here; you mess the entire business alignment. First and foremost is the business angle should the SIEM solution you are trying to bring is expected to deliver / promote your Organization compliance strategy (e.g iso 27001) with better viability and alerting mechanism. Or, threat focus which you want the SIEM solution to program itself in shape,size and form where it can help you detect the different types threats your company faces (e.g virus, access violations, logon abuse).

One of my colleague industry, who happens to be CEO of reputable security service and consultancy firm told me..'SIEM is what you want it to sing; it can be anti-malware, a compliance tool and in some cases a forensic tool (as some vendors technology they try to blur the lines between SIEM technology by adding components of network forensic in their feature set)."

My experience with SIEM tells this is something not hard to achieve (the chaos part), Mgt ask you I want report on firewall denies w.r.t Country e.g china. You add up , tune the SIEM box to receive syslog events. Some other day they see / internal audit reports lot of issues related to privileges abuse now suddenly for Mgt compliance becomes top priority. SIEM engineers are called; sys auditing are enabled logs are shipped through third-party agent (i.e snare) and Analyst start to report on compliance issues.

Ad-hocism prevails to the point , when one -day you receive SNMP trap from SIEM box saying 'license violations device IP address xx.xx.xx.xx version xx has started dropping  logs from pipeline  at a rate of 20keps / minute'. 

As SIEM analyst you in panic mode with fainted breath entire your BOSS office, and explains him the situation. His response...'Get new license; box; call the vendors..' this soon becomes a classic example of Mgt vs technical debacle.

Through undefined strategy ; you have designed and maneuver your SIEM / SOC ops into a tactical and Ops failure. How its simple? Its maths...

Total license procured=3000eps

Compliance logs (unix,win etc)=70%

          Device logs (n/w h/w)=20%

          Others (DB's, NOC)=10%
               
NOTE: Most of you be scared of 70% large number it seems? ou would be used to seeing such number when compliance is done without enforcing any kind of security controls? Here is short example from experience "Dept Head ask me to get him / report on TMG report for bandwidth usage. I ask him nicely Sir, please ask the system-team or the NOC to generate him this report, daily bandwidth usage monitoring is not task of SOC Ops. His response, I can't they have shipped all their logs to your SIEM (another case study transfer responsibility -> transfer risk). Trying to master courage after what I have just heard I take it back the question and asked Sir, have we deployed any controls at first (i.e x amount of bandwidth used in y time by abc user of department xyz). His response No? He said, your task is simple your getting span traffic to SIEM for analysis (Qradar does it effectively) we don't really need TMG logs which is restrictive for L7 inspection. So using the flows data I was to reconstruct a report; for a moment consider myself as TMG admin or a sys-admin who is tasked to take out report of daily login activity. I felt shameless; such mushroom growth scenario was this new use-case by Mgt that very basic idea of security monitoring is killed and overshadowed which is to 'monitor your security controls'. There is a very high cost as any SIEM admin can tell you to report on Give me top sites visited by employees versus give me report for top 5 bandwidth violations. In latter case I would dropped everything from either at the source or the parser level which have logs beside violations. I would also define retention policies in the same regard at the end I achieve a cost-effective a better value for high eps solution.

So, the project which started with the vision for watching threat-landscape of the organization, quickly turned into all around compliance and monitoring function with just ONE REQUIREMENT. I don't want to sound so quote "ANAL" in my review, what so If Mgt think they want to get compliance done using SIEM, its their decision; why I'm to complain? I'm just the Analyst sitting behind screen of 1000's lines of logs? Right.

Unfortunately , purchasing SIEM license is not same as buying online service (you can't buy EPS with credit cards!). With big companies the procurement process is slow, and tiring. A better analogy is when at times you "take your kids to pizza hut ; for a family eat-out and you plan to pay for regular size pizza; but then one of your sons see the big advertisement for the new pizzahut deal which features King size 21 inch pizza of your son favorite Fajita flavour  and also have a free big pitcher of his fav drink along with free nuggets. You as a dad have two options tell him you don't have the cash son; or simply ask your sons to wait for him in restaurant while he drives back to nearest ATM for more cash.".

This going back unlike in the example given above for large organization is not a pleasant feeling esp when unplanned equals disaster.Why?

How do you explain the license space expired in less then 6 months of deployments when I as CTO remember signing off on your requirements ; which said your license will provide the operational capability for SOC for  almost 3+ year straight.

 Because security is always been considered excess baggage; over-budgeting SIEM requirement means quick and very short life of the program. How compliance was not relevant 6 months before (when you were writing SIEM rfp) and now is when you received the audit report? SIEM is NOT a tool to manage your "Mgt fuckups" which is poor gathering of requirements and scope statement.  

Most of us forget that the SIEM is nothing more then "detective tool" no matter what the vendors says and throw you off with their sexy use of jargon; which doesn't really make sense in the battle-field. I have..

• Do you really need to audit your compliance through SIEM?
• What are the existing controls in place to check compliance (i.e admin roles are restricted, o/s patching etc) do these controls guarantee; prevention? (mindful every control should be placed and designed with prevention in mind) ; how much risk is residual? Is the cost of control is higher then the risk you are trying to reduce, the SLE and ARO were these values taken in consideration (i.e $ / eps >> loss ( violation of acceptable use of internet browsing internet,downloading torrents)??.
• Was proper risk analysis / vulnerability assessment carried out in case of audit report for privilege abuse/escalation/misuse.

Planning

Opening statement is

If You Fail to Plan, You Plan to Fail

In this phase I expect you along with management define the what and whats' not of the solution. Which is as simple and sometimes as hard as answering the the famous Five Ws, Five Ws and one H.

Things like what should get logged, what syslog level each log source facility will have, what will be the retention period, do I encrypt the logs, how to create backups...what would be the impact on network in terms of real-time logging etc.

This is what called " readiness" how ready is your organization environment to adopt SIEM technology. My fact guide this course is NIST Log management guide. Link here. There you would find guidelines how to establish effective log-infrastructure ; which is a must have before SIEM. The fact which has been continuously and at time religious advocated by many senior SIEM consultants made more famous by Dr. Anton chuvakin where in his blog he repeatedly emphasis the importance of having a solid 'Log Mgt infra' before SOC /SIEM. Infact it is included as WP (Worst practice) to go for SIEM without having one.

The interesting mention in the NIST guide, is there is not one related to SIEM infact SIEM is discussed as a distinction between syslog server and SIEM. To me , the difference is valid and makes so much sense...'NIST puts SIEM as collection of syslog-servers'. At its core, this is what it really is collection of multiple syslog-server capable of parsing , normalizing events from heterogeneous sources.

The thing to understand in this section, is why these pre-req matters ; why having effective log-management policy and Infra are essentials part for guaranteeing an successful roll-out of SIEM solution. This is why ...

1. From experience vendor ABC spent 70% of its SIEM project deployment, configuration and optimization time in configuring log-sources. Why? You got a nifty sexy use-case for management to see ; the vendor resident engineer on hearing the use-cases gives bunch of requirements...yes our SIEM solution can do this ; IF you do the following:-

USE-CASE: Alert when admin add ANY user to internet policy  group for senior executives

Vendor: You have TMG right? How do you current view logs?  

Company: hm Yes we use TMG as proxy; we use the integrated SQL express edition 2010 for logging.

Vendor: Nice...create a view and role for our SIEM product and open it for remote jdbc polling on port 1433.

Company: But we never did this, will it break our system, what are the guidelines to follow, what versions are supported and whole Pandora of questions?

Vendor: You said ; you are using native setup which comes with default SQL express? Right. In that case; I'm afraid you have to change your deployment to separate SQL server; since the SQL express edition which comes with TMG doesn't have the support for view and audit roles required to configure the remote logging setup 

Company: Sys-admin are unfamilar of this type of auditing, they haven't been trained before to enabling such controls...equally 

Vendor: He is all happy; ask you the remote DB server IP, port and polling frequency. In half hour of configuring he is able to extract the report for the Mgt use-case.

Company: Emails the report, Mgt calls SOC Manager bull-shit him out of his room;..."who asked for a website usage report. I want the audit report". Manager calls his SIEM team ; transfer the bullshit and asked to give him the report he wants. Analyst go back speed-dialing the vendor to bring his @ss back to company office and complete his use-cases implementation.

Vendor: Returns back to office scratching his head; after explained him the situation; calls up his senior who is currently deployed with another client blasts him about the situation he is landed him. "Did you not followed the official guide"." Of course I did".. okay wait a minute ...what the use-case once again...wait ..."let me mail you". Oh shite!. Sorry mate...I think you need to audit 'change firewall policy event'. Few questions..do you know what current roles they have i.e auditor , TMG admin etc? You need to create a custom view? (I know its not in official guide; call me If you need help :P)hmm just talking to you I think the client would love the option of "Tracking configuration changes" using the option they can do more then recording firewall policy changes.Do you know how to extract the /export the logs? You also have to limit the audit logs entries or else it overspill the disk...and Perhaps you may need to write a custom log-fowarder to our SIEM box if it doesn't support jdbc support.....'Hey man are you still on the line...".

Vendor(friend): Yes..before I ask you anything further I want to know the few moments ago I was talking to client sys-admin just told me ..."they have to unplug the sys from network...as just last night the newly raised machine got infected with vulnerability related to unpatched sql-server driver.

Vendor: Thats sucks...Perhaps...."hello Man..you there"...

Vendor(friend):...."beep.....beep....!!!

An experienced Project manager reading the above dialogue can easily point-out list of things which are bad / seriously wrong in such kind of setup. In short, in many deployments I have seen with the introduction of one Project (SIEM) most organizations at PARALLEL (without KNOWING) are creating log-infrastructure in the most chaotic way possible.

BOTTOM LINE: Don't except the SIEM vendor to be your audit consultant.

Another scoop. Before this quick triva TCP versus UDP. Who wins? TCP its all reliable; connection oriented and etc. Two people would disagree it used to voip guys but new in the list is SIEM. Once...we had to deploy syslog agent on AD; the agent is formally known as ALE (Adaptive log exporter)...it was close to end of business hours..In the rush when all was configured we had the option which basically asked the admin the method of transfer via tcp or udp. Syslog natively is udp but in cases TCP is used for reliability. Someone from the vendor project team said TCP...select tcp I have used it in multiple deployments it works its fast and always guarantee traffic to be sent / reach SIEM server. So tcp it was, ....

Next morning...the vendor Point of contact guys greets the sys-admin from the client in a every angry mood; ...'what happened...!..you looked pretty piss*d..'..."Yes I do....I was awaken in the middle of night and had to come to office to see my AD dying...the cpu usage was nearly 95% and corporate uesrs working from home and office were being logged off from their session....And yeah the netstat shows hundreds..of connections in 'fin_wait'....to SIEM server...vendor in his mind holding his heart in his mouth mutters...(the server whose machine I accidentally shutdown before leaving office ....)

The above examples are not examples of bad vendor selection or deployment or Project management...but are faced in regular basis to customer who try...to "both fly and repair the sail while in AIR"

Gathering Requirements

This part is easy to write but easily to get wrong (nearly all the times). In large Org with multiple depts working in decentralized hierarchy means your workload would be greatly increased unless you do have an effective team and plan to gather up these requirements for this new technology.

Also, depending upon how you are going to encompass SIEM with your Organization existing security portfolio as senior fw sec admin,  sys sec administrator their roles might be migrated to positions of mid-level SIEM analysts. HR requirement or need analysis is must for effective running of SOC operations and could mean myriad of whole number of different things if you want to establish something like Security Operations Center and wants to purchase SIEM as one of the component to facilitate and improve your response and detection capability.This would mean from HR, standpoint to develop processes, (i.e incident response, ticketing sys (soc desk)) and how to align the  resources to perform blue and red teaming tasks.

Now, let comes to techincal bits of gathering SOC requirements.

Sizing ...Sizing...and more sizing (cannot emphasis enough). In world of SIEM its all about number unless you want to go and purchase something like ocotopussy or some other open source SIEM solutions where you are not bothered by the fact licenses issues.  Here, the understanding of logging and format becomes crucial, and If you have got a senior log analyst in your Org; now is the time to pick up the phone and ask him to become part of your new Project (this is what all good manager would do). Try to narrow / chances of confusion which will be achieved if you got people who understands how logging works from "bottom-up". For those who are interested in knowing what an eps or epm numbers are how they can be calculate and nice reference would be a similar site dedicated to Qradar / SIEM practices is  http://qradarinsights.com/.  In short, eps is would what vendor would charge you to process the events coming in real-time or as always (near real-time cases). EPS is nothing more then a technical jargon to help bridge the understanding between the vendor and customer to plan out your deployment and also requirements for processing depending upon the volume of security data that could be generated by your devices/systems and the list can include anything from the cctv camera to tape drive anything which has an IP address and O/s have the capability to produce logs one form or another. Also, another distinction worth nothing over here, the capability of device X to produce Y number of events / sec (EPS) given the right syslog configuration w.r.t syslog and facility levels should not mean ALL that rate has to be processed and counted as an "effective eps" rate. In most deployment, the vendor hungry to define functionality will ask the fw admins to configure the syslog setting with highest level of verbosity (which maximizes their chances of getting cool report and trigger alerts). As, as customer you should be keen and both vigilant to prepare your requirements before major SIEM deployment to include , steps to configure these device to send only the Events that are necessary...events which requires you to be alerted on...events for which you have planned USE-CASE.
 
Smart-agents such as win-collect (an agent collect syslog events from multiple windows  stations) have the ability to let you configure using x-path query to limit your data-set to be sent to SIEM in its configuration settings. For. ie. Event ID 1401 to 1500 I need from this system and I already have a rules / in the SIEM rule engine which are waiting for the events from these ID'. Not everything on the system requires auditing, contact your Org compliance team to help you understand even the system which as so passionately involved in getting logs from is worth monitoring for. Whats the CIA rating of the system, what the classification of data that this system manages/  processes, What is its value from BIA?...You don't want to end up in a situation where you want to present a report to senior board members on the performance of your security program by showing daily login stats of a internal houses online web-server which manages the senior executive appointments or worse something as HR portal which is used by internal employees to send birthday greetings. Trust me you don't want to waste expensive license on uncritical.

There are some hyper-management techniques customers sometimes use to introduce fear of uncertainty and fear ; over-visualization security threats is a bad practice; The most important lesson in security that any senior professional would tell you is that " ...of all security should be cost-effective". In above example , some would try to persuade and have an argument of "what ifs". What if my HR portal gets backdoor, what if the web-server is compromised and someone is able to gain access to backed DB through over-looked parameter manipulation vulnerability in your web-form.

My simple response is "what are you doing now". In this very moment as we are talking , all the worst actually start to take place, how would you know, how would you investigate, how would you report, how do you contain and most importantly have you performed any kind of risk / vulnerability assessment for these web-services. If you get the answer to last question as "NO" then other response doesn't matter. You cannot reason FEAR!

Don't except SIEM to be your solution your magic wand to wade of all your security Ills. Without underlying processes like patch management / an effective vulun program you have NO "operational assurance". Having these well-established processes and procedures means from tactical point that you would NOT absorb the energies of your SIEM analyst ; and the avoid the cost of over-license if there exists an already established / acceptable risk statement signed by your company senior management stating ; that current controls are adequate enough and grantees that with the current specified system characteristics they enable the system to remain operational with the measure and acceptable impact to its confidentially , Integrity and availability components.

NOTE:
This is a golden statement for most - organization most simply perish before they reach such conclusions but those who are in disarray addition of SIEM is nothing more then another piece in the jigsaw puzzle!!



Perhaps, the company which wrote this statement has realized that a very skilled security professional have integrated their web-server apache logs with something like a parser (Apache-scalp). Its open-source its free and fully customizable. Loss cost solution to low cost protection requirement (perfect business sense!).

At this point; with the SIEM procurement in near horizon; you can either allows these decentralized security monitoring functions to remain in-tact or PLAN for migrations these points into new SIEM solution. But not before I have an effective use-case to support this..? Something like a internal (disgruntled employee) wants to compromise these servers to make a statement against company exec? Or perhaps able to make use-case as need a alert when company web-servers are compromised by same vulnerability in almost 80% of last patched window.

In first case; reading the profile of the system described above; you as manager shouldn't be so worry (esp when you have mechanism in your controls to track the Internal Ip of that user and the web-site have been pen-tested against defacing attempts).

Secondly, how about you get logs from core fw; you n/w arch is everything from layer 3 switch goes to firewall for screening / policing and then its routed back across the switch to one or more vlans. Why not take logs from IPS from the core-switch? Will this not tell which web-servers are being exploited and sigs.

Through this discussion and examples I want you to make a smart choice when come to make such decisions in your workplace. The SMARTEST ones are those which are informed and researched. Simply means "do your homework".!




Storage

COMING UP SOON!!

Know your Network

COMING UP SOON!!



Litmus test (How do you know you have written an effective RFP)?

COMING UP SOON!