Prototyping CIFv4 - Engineering a Realtime Intelligence Platform

Prototyping CIFv4 - Engineering a Realtime Intelligence Platform

The main focus of the last ~60 hours has been APIs, feeds and real-time streaming. This includes the HTTP REST API the realtime ZeroMQ streaming API and to some extent, WebHooks…

If you're looking to build and deploy your own #ThreatIntel platform, these are the things you should be thinking about.. It should take months, not years.. and you should learn from our mistakes, not just your own.

F your formats, just show me the data- part2

F your formats, just show me the data- part2

The real problem we're trying to solve here is context. We're lifting a bunch of "tokens", that usually have more than 3 characters, surrounding them with context and applying a probability value to them. All this with the express purpose of taking the high value indicators and applying them to our defenses in real-time. Not trivial, but not hard either. I'm not an SKLearn or NLTK expert- but I do know what it feels like to block accidentally netflix.com at the border….

Deploying Threat Intel Platforms, From GitHub.

Deploying Threat Intel Platforms, From GitHub.

The problem wasn't trying to manage and automate the code deployment, as much as it became managing the playbooks that deployed the application(s). We could have kept those playbooks in with the core code, but that's more over-head in the repo and more people touching the core code that didn't need to....

Prototyping CIFv4: Part 1.

Prototyping CIFv4: Part 1.

I've spent about a year thinking about v4 and about 12 hours writing it (most of which has been re-factoring older code and wondering how drunk I was when I wrote it). If you look at the repo today, most of it looks and feeds like v3 but with most of the complexity removed (eg: lots of refactoring for performance and readability). Last night, I was able to get "pings" flowing back and forth between the client and the storage thread, which is good sign...