Skip to content

Commit

Permalink
Merge branch 'v1.6'.
Browse files Browse the repository at this point in the history
- Remove Feed Overview dashboard
- Rename The Round Table to OSINT Sweep
- Create psbdmp dashboard
- Disable Ransomware Tracker dashboard
- Use Urlhaus official API
- Modify Greynoise timechart
- Unset all tokens when a new search runs
- Load new config for Phishing Catcher
- Create Hybrid-Analysis dasboard
- Create Malshare dashboard
  • Loading branch information
ecstatic_nobel committed Apr 19, 2019
2 parents 13d93a4 + e454ecb commit c16843b
Show file tree
Hide file tree
Showing 54 changed files with 1,636 additions and 1,105 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2018 ecstatic-nobel
Copyright (c) 2019 ecstatic-nobel

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
43 changes: 22 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,53 +32,54 @@ Before getting started, ensure you have the following:
**Optional Packages**
- Git

Click **[HERE](https://github.com/ecstatic-nobel/osweep/wiki/Setup)** to get started.
Click **[HERE](https://github.com/ecstatic-nobel/OSweep/wiki/Setup)** to get started.

### Gallery
**Feed Overview - Dashboard**
![Feed Overview - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/feedOverview_dashboard.png)

**The Round Table - Dashboard**
![The Round Table - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/theRoundTable_dashboard.png)
**OSINT Sweep - Dashboard**
![OSINT Sweep - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/osintSweep_dashboard.png)

**Certificate Search - Dashboard**
![crtsh - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/crtsh_dashboard.png)
![crtsh - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/crtsh_dashboard.png)

**CyberCrime Tracker - Dashboard**
![CyberCrime Tracker - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/cybercrimeTracker_dashboard.png)
![CyberCrime Tracker - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/cybercrimeTracker_dashboard.png)

**Cymon - Dashboard**
![Cymon - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/cymon_dashboard.png)
![Cymon - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/cymon_dashboard.png)

**GreyNoise - Dashboard**
![GreyNoise - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/greynoise_dashboard.png)
![GreyNoise - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/greynoise_dashboard.png)

**Hybrid-Analysis - Dashboard**
![Hybrid-Analysis - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/hybridAnalysis_dashboard.png)

**MalShare - Dashboard**
![MalShare - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/malshare_dashboard.png)

**Phishing Catcher - Dashboard**
![Phishing Catcher - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/phishingCatcher_dashboard.png)
![Phishing Catcher - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/phishingCatcher_dashboard.png)

**Phishing Kit Tracker - Dashboard**
![Phishing Kit Tracker - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/phishingKitTracker_dashboard.png)
![Phishing Kit Tracker - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/phishingKitTracker_dashboard.png)

**Ransomare Tracker - Dashboard**
![Ransomare Tracker - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/ransomwareTracker_dashboard.png)
**Pastebin Dump - Dashboard**
![Pastebin Dump - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/psbdmp_dashboard.png)

**ThreatCrowd - Dashboard**
![ThreatCrowd - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/threatcrowd_dashboard.png)
![ThreatCrowd - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/threatcrowd_dashboard.png)

**Twitter - Dashboard**
![Twitter - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/twitter_dashboard.png)
![Twitter - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/twitter_dashboard.png)

**URLhaus - Dashboard**
![URLhaus - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/urlhaus_dashboard.png)
![URLhaus - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/urlhaus_dashboard.png)

**<span>urlscan</span>.io - Dashboard**
![urlscanio - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/osweep/master/static/assets/urlscan_dashboard.png)
![urlscanio - Dashboard](https://raw.githubusercontent.com/ecstatic-nobel/OSweep/master/static/assets/urlscan_dashboard.png)

### Dashboards Coming Soon
- Alienvault
- Censys
- Hybrid-Analysis
- Malshare
- Censys
- PulseDive

Please fork, create merge requests, and help make this better.
8 changes: 6 additions & 2 deletions bin/commons.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import sys
import traceback

app_home = "{}/etc/apps/osweep".format(os.environ['SPLUNK_HOME'])
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
import splunk.Intersplunk as InterSplunk
Expand All @@ -34,6 +34,10 @@ def get_apikey(api):
"""Return the API key."""
if api == "greynoise":
return config.greynoise_key
if api == "hybrid-analysis":
return config.hybrid_analysis_apikey
if api == "malshare":
return config.malshare_apikey
if api == "pulsedive":
return config.pulsedive_apikey
if api == "twitter":
Expand Down Expand Up @@ -80,7 +84,7 @@ def return_results(module):
InterSplunk.outputResults(new_results)
return

def deobfuscate_url(provided_ioc):
def deobfuscate_string(provided_ioc):
"""Return deobfuscated URLs."""
pattern = re.compile("^h..p", re.IGNORECASE)
provided_ioc = pattern.sub("http", provided_ioc)
Expand Down
4 changes: 2 additions & 2 deletions bin/crtsh.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
import os
import sys

app_home = "{}/etc/apps/osweep".format(os.environ['SPLUNK_HOME'])
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
import validators
Expand All @@ -56,7 +56,7 @@ def process_iocs(results):
splunk_table = []

for provided_ioc in set(provided_iocs):
provided_ioc = commons.deobfuscate_url(provided_ioc)
provided_ioc = commons.deobfuscate_string(provided_ioc)

if validators.domain(provided_ioc) or validators.ipv4(provided_ioc):
crt_dicts = query_crtsh(provided_ioc, session)
Expand Down
4 changes: 2 additions & 2 deletions bin/cybercrime_tracker.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
import re
import sys

app_home = "{}/etc/apps/osweep".format(os.environ['SPLUNK_HOME'])
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
from bs4 import BeautifulSoup
Expand Down Expand Up @@ -85,7 +85,7 @@ def process_iocs(results):
splunk_table = []

for provided_ioc in set(provided_iocs):
provided_ioc = commons.deobfuscate_url(provided_ioc)
provided_ioc = commons.deobfuscate_string(provided_ioc)

if validators.domain(provided_ioc) or validators.ipv4(provided_ioc):
cct_dicts = query_cct(provided_ioc, session)
Expand Down
4 changes: 2 additions & 2 deletions bin/cymon.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
import os
import sys

app_home = "{}/etc/apps/osweep".format(os.environ['SPLUNK_HOME'])
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
import validators
Expand All @@ -50,7 +50,7 @@ def process_iocs(results):
splunk_table = []

for provided_ioc in provided_iocs:
provided_ioc = commons.deobfuscate_url(provided_ioc)
provided_ioc = commons.deobfuscate_string(provided_ioc)

if validators.ipv4(provided_ioc):
ioc_type = "ip"
Expand Down
4 changes: 2 additions & 2 deletions bin/greynoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
import os
import sys

app_home = "{}/etc/apps/osweep".format(os.environ['SPLUNK_HOME'])
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
import validators
Expand Down Expand Up @@ -135,7 +135,7 @@ def process_iocs(results):
open_file.close()

for provided_ioc in set(provided_iocs):
provided_ioc = commons.deobfuscate_url(provided_ioc)
provided_ioc = commons.deobfuscate_string(provided_ioc)

if not validators.ipv4(provided_ioc) and \
not validators.domain(provided_ioc) and \
Expand Down
161 changes: 161 additions & 0 deletions bin/hybrid_analysis.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
#!/opt/splunk/bin/python
"""
Description: Use Malshare to gather hashes of potentially malicious files. The
script accepts a list of strings (domains, IPs, MD5, or SHA256):
| malshare $ioc$
or input from the pipeline (any field where the value is a domain, IP, MD5 or
SHA256). The first argument is the name of one field:
<search>
| fields <IOC_FIELD>
| malshare <IOC_FIELD>
Source: https://malshare.com/
Instructions:
1. Switch to the Hybrid Analysis dashboard in the OSweep app.
2. Add the list of IOCs to the "Domain, IP, MD5, SHA256 (+)" textbox.
4. Click "Submit".
Rate Limit: None
Results Limit: None
Debugger: open("/tmp/splunk_script.txt", "a").write("{}: <MSG>\n".format(<VAR>))
"""

import os
import re
import sys

app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
tp_modules = "{}/bin/_tp_modules".format(app_home)
sys.path.insert(0, tp_modules)
import validators

import commons


api = "https://www.hybrid-analysis.com/api/v2/search/{}".lower()

def process_iocs(results):
"""Return data formatted for Splunk from Hybrid-Analysis."""
params = [
'authentihash',
'av_detect',
'context',
'country',
'domain',
'env_id',
'filename',
'filetype_desc',
'filetype',
'hash',
'host',
'imp_hash',
'port',
'similar_to',
'ssdeep',
'tag',
'url',
'verdict',
'vx_family'
]

if results != None:
provided_iocs = [y for x in results for y in x.values()]
elif sys.argv[1] == "terms" and sys.argv[2] in params:
if len(sys.argv) > 2:
endpoint = sys.argv[1]
param = sys.argv[2]
provided_iocs = sys.argv[3:]
elif sys.argv[1] == "hash" and sys.argv[2] == "hash":
if len(sys.argv) > 2:
endpoint = sys.argv[1]
param = sys.argv[2]
provided_iocs = sys.argv[3:]

session = commons.create_session()
api_key = commons.get_apikey("hybrid-analysis")
splunk_table = []

for provided_ioc in set(provided_iocs):
provided_ioc = commons.deobfuscate_string(provided_ioc)
provided_ioc = provided_ioc.lower()

ioc_dicts = query_hybridanalysis(endpoint, param, provided_ioc, api_key, session)

for ioc_dict in ioc_dicts:
splunk_table.append(ioc_dict)

session.close()
return splunk_table

def query_hybridanalysis(endpoint, param, provided_ioc, api_key, session):
""" """
ioc_dicts = []

session.headers.update({
"api-key":api_key,
"Accept":"application/json",
"User-Agent":"Falcon Sandbox"
})
resp = session.post(api.format(endpoint), data={param:provided_ioc}, timeout=180)

if resp.status_code == 200 and resp.content != '':
results = resp.json()
else:
ioc_dicts.append({"no data": provided_ioc})
return ioc_dicts

if isinstance(results, dict):
if "result" in results.keys() and len(results["result"]) > 0:
results = results["result"]
else:
ioc_dicts.append({"no data": provided_ioc})
return ioc_dicts

for result in results:
ioc_dict = {}
ioc_dict["type"] = result.get("type", None)
ioc_dict["target_url"] = result.get("target_url", None)
ioc_dict["submit_name"] = result.get("submit_name", None)
ioc_dict["md5"] = result.get("md5", None)
ioc_dict["sha256"] = result.get("sha256", None)
ioc_dict["ssdeep"] = result.get("ssdeep", None)
ioc_dict["imphash"] = result.get("imphash", None)
ioc_dict["av_detect"] = result.get("av_detect", None)
ioc_dict["analysis_start_time"] = result.get("analysis_start_time", None)
ioc_dict["threat_score"] = result.get("threat_score", None)
ioc_dict["interesting"] = result.get("interesting", None)
ioc_dict["threat_level"] = result.get("threat_level", None)
ioc_dict["verdict"] = result.get("verdict", None)
ioc_dict["domains"] = result.get("domains", None)
if ioc_dict["domains"] != None:
ioc_dict["domains"] = "\n".join(ioc_dict["domains"])
ioc_dict["classification_tags"] = result.get("classification_tags", None)
if ioc_dict["classification_tags"] != None:
ioc_dict["classification_tags"] = "\n".join(ioc_dict["classification_tags"])
ioc_dict["compromised_hosts"] = result.get("compromised_hosts", None)
if ioc_dict["compromised_hosts"] != None:
ioc_dict["compromised_hosts"] = "\n".join(ioc_dict["compromised_hosts"])
ioc_dict["hosts"] = result.get("hosts", None)
if ioc_dict["hosts"] != None:
ioc_dict["hosts"] = "\n".join(ioc_dict["hosts"])
ioc_dict["total_network_connections"] = result.get("total_network_connections", None)
ioc_dict["total_processes"] = result.get("total_processes", None)
ioc_dict["extracted_files"] = result.get("extracted_files", None)
if ioc_dict["extracted_files"] != None:
ioc_dict["extracted_files"] = "\n".join(ioc_dict["extracted_files"])
ioc_dict["processes"] = result.get("processes", None)
if ioc_dict["processes"] != None:
ioc_dict["processes"] = "\n".join(ioc_dict["processes"])
ioc_dict["tags"] = result.get("tags", None)
if ioc_dict["tags"] != None:
ioc_dict["tags"] = "\n".join(ioc_dict["tags"])
ioc_dicts.append(ioc_dict)
return ioc_dicts

if __name__ == "__main__":
current_module = sys.modules[__name__]
commons.return_results(current_module)
Loading

0 comments on commit c16843b

Please sign in to comment.