pinot

Osquery logs to Apache Pinot

This example uses the output of stream in the osquery example. We send this stream to Kafka to be consumed by Apache Pinot.

flowchart TD;
    os{{OSQuery+Extension}}-->Pulsar-->pc([Pulsar_Connection])
    
    pc-->opp[[osquery_processes]]-->SQL:osquery_filter_noise-->fp[[filtered_processes]]

    fp-->SQL:osquery_cleanse-->db[(Apache Pinot)]-->as[/Apache Superset\]
    fp-->sp[SQL:suspicious_processes]-->a>Alert]

    sp-->db

Loading

Getting started

Create an .env file and populate your values

BOOTSTRAP=<< Kafka bootstrap servers >>
CONFLUENT_KEY=<< CONFLUENT KEY >>
CONFLUENT_SECRET=<< CONFLUENT SECRET >>
CLUSTER_ID=<< KAFKA CLUSTER ID >>

SCHEMA_REGISTRY=<< CONFLUENT CLOUD SCHEMA REGISTRY URL >>
CONFLUENT_SR_KEY=<< CONFLUENT SCHEMA REGISTRY KEY >>
CONFLUENT_SR_SECRET=<< CONFLUENT SCHEMA REGSITRY SECRET >>

TOPIC=<< KAFKA PINOT TOPIC >>

CONTROLLER_HOST=<< HOST/IP to PINOT >>
CONTROLLER_PORT=<< PINOT CONTROLLER PORT >>

Install pinot-admin

brew install pinot

Install Confluent Cloud CLI

curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest

Install / Run Pinot EC2 Instance

Install Apache Pinot

PINOT_VERSION=0.10.0 #set to the Pinot version you decide to use

wget https://downloads.apache.org/pinot/apache-pinot-$PINOT_VERSION/apache-pinot-$PINOT_VERSION-bin.tar.gz

# untar it
tar -zxvf apache-pinot-$PINOT_VERSION-bin.tar.gz

# navigate to directory containing the launcher scripts
cd apache-pinot-$PINOT_VERSION-bin

Start Pinot

./bin/quick-start-streaming.sh &

This starts a streaming example built into Apache Pinot.

Start

make topic
make flow
make active

make clean # cleans Decodable