3.0.3 - API Delay && Immediate Mode UI

new

• Add the delay option to the /submit and /submit/single API endpoints. This adds a delay between when the page is considered loading and when a screenshot should be taken. The command line already adds a delay, but this adds it to the API as well (see Swagger docs) (1db81b3)
• Add new web UI for probing using immediate mode. Immediate mode does not record data to the database but instead shows a modal with the results (see screenshots below) (bb5cf4f)

fixed

• Fix API documentation for /submit/single to show the full return object (a9e254d)
• Trim spaces for file targets to help improve reliability of the reader (fd9f55f)

Full Changelog: 3.0.2...3.0.3

ba104c2f84fcebd3f9a6d42417d44afbdfa3baf5  gowitness-3.0.3-darwin-amd64
05a1abce98df0fe5747aceeed91d6253d8e2db93  gowitness-3.0.3-darwin-arm64
c32914a24b777178ed5068d6d10b15dc4e410a5c  gowitness-3.0.3-linux-amd64
d1317177f94c9823b7c948c3a358e7ddbe7ea497  gowitness-3.0.3-linux-arm
c5813ef4fbe4417563c28e24ccf0e4c1b222b8c8  gowitness-3.0.3-linux-arm64
9c1b4d193d3d217446458fb2abc667f812be37a8  gowitness-3.0.3-windows-amd64.exe
ff6078602f02f89b48b3a875404d575f7874e4d3  gowitness-3.0.3-windows-arm64.exe

3.0.2 - File scanner fixes

fixed

• Allow file parser to parse <host>:<port> targets (66eec62)

Full Changelog: 3.0.1...3.0.2

9e13af78bda5322f9f55270fd47b827bdb5f7c2c  gowitness-3.0.2-darwin-amd64
3aeb745f9d8844c221bcba18a10fc6711390b596  gowitness-3.0.2-darwin-arm64
694a150209f71aaa37827be735152689f8d61e8d  gowitness-3.0.2-linux-amd64
155e13bb93aa22f9de8808046b7e2f3c3e8cf9c6  gowitness-3.0.2-linux-arm
5239ec4810b0eeff5ee2b49356e06aabd1eed2ba  gowitness-3.0.2-linux-arm64
c7a93e49fa442301b80ae6cf0ef0f181d4c764da  gowitness-3.0.2-windows-amd64.exe
18de5172ed5ac779e22b613afdb4f89205c72249  gowitness-3.0.2-windows-arm64.exe

3.0.1 - Bug fixes

new

• add /submit/single api endpoint for immediate screenshot results without invoking writers (c4478e2)
• add a link to the swagger api documentation in the ui (f5fd917)

fixes

• dont send completely failed results to writers (5780809)
• handle results without tls information better in the ui (fc06051)
• fix handling of ports specifications in the file reader (and add ipv6 support while im here) (0d3cd8d)
• dont check for file existence when using stdin (d3fbac4)

Full Changelog: 3.0.0...3.0.1

v3.0.0 - The *major* refactor release!

gowitness v3

Gosh, so much work went into this release, I really don't know where to start. v3 has been a while coming, and after I needed some changes for an internal tool, I figured now was as good a time as any to finally fix the stuff that had been bugging me. So, I branched to v3, deleted everything but the README and .gitignore files, and scaffolded a new cobra project.

I had lots of ideas, and a fresh start helped me finally get those in. A lot has changed, been reworked, refactored, upgraded, and more. I had fun building this, and hope you have fun using it!

If this were your typical mobile application, the release notes would have just read "Bug fixes and improvements". While thats true, it's not all there is to say.

overview

If I had to give a TL;DR of what changed, I'd summarise it as:

• Reworked the CLI. Commands are now properly categorised into subcommands. Flags also properly inherit from their parents now.
• Refactored the scanning and screenshotting logic. Most notably, the old preflight logic is removed. It was nice and fast, but when it mattered, it was a huge pain to deal with and came at the cost of result accuracy. Instead, in v3, results are now grabbed from network events like Network.responseReceived.
• Introduced the concept of "drivers," where chromedp was the original (and still default) driver. However, rod is also a driver option now that you can choose using a command-line flag.
• Significantly improved and fixed code quality, concurrency-related issues, and general screenshot reliability.
• Rewrote the report web server frontend in React. It just looks so much better, has significantly more features, and is easier to change now.
• Added an official API, complete with code-generated Swagger documentation!
• Introduced the concept of "writers," which can be used simultaneously. For example, you could write results to an SQLite database, JSON Lines, and stdout all at the same time.
• Fixed perception hashing to use Hamming Distance for grouping.

There's a lot more that's changed, so if you're curious about that and want a bit of story time, feel free to continue reading.

cgo and SQLite

SQLite has been a (required) storage mechanism since version 2. I experimented with using buntdb in version 1, but that didn't last long. It's much nicer using a format other programs could easily build on top of.

Anyway, the problem with SQLite and Golang is that the drivers often compile against the SQLite C headers. This makes cross-compilation in Go harder than it needs to be, where you need a build environment with CGO_ENABLED=1. To deal with this in version 2, I used the Elastic golang-crossbuild Docker images to target different operating systems and architectures for releases.

Thankfully, I have since discovered a pure Go SQLite implementation that comes at an immaterial performance cost! That means no more CGO_ENABLED=1, and easier/faster builds.

architecture

Every day is a school day, and I've learned a lot about Go in the years since gowitness was first released. Like any library in the Go ecosystem, if it's well-structured, anyone can technically include and use it in their own project. Unfortunately, gowitness v1 and v2 were not well-suited for this use case. I mean, you could have imported some gowitness code, but you really shouldn't have! :D

For version 3, I chose to adopt some of the project structure as described in this project-layout project. What's neat about this (apart from learning) is that now, with version 3, you should be able to import and use gowitness as a library in your own project.

Apart from the overall project structure, gowitness also underwent a significant restructuring of the codebase. There used to be one chrome package that mostly had all the important bits. Now, there are a few new concepts which include:

• Drivers: Effectively the libraries that drive Google Chrome using CDP.
• Readers: Functions that read from various sources (files, nmap, nessus, etc.)
• Writers: Functions that write driver results somewhere (SQLite, JSON Lines, stdout, etc.)
• Runner: A core component that "runs" drivers, reading from readers and writing to writers.

These all make up the internals of how an end-to-end probe of a remote website will happen. All of the scan commands use this runner pattern, including the web interface's "New Probe" feature.

scanner Drivers

Version 1 was basically the graduation of a bash script. It literally spawned a shell to run Chrome with the --headless flag. What's even funnier is I started a small https-to-http proxy in a temporary goroutine to get around TLS-related errors too. Good times. It was scruffy, but in the places where I needed it (including that one project where this started), I got screenshots faster and more reliably than I'd ever been able to before!

In version 2 though, I learned about the Chrome DevTools Protocol (CDP) and discovered chromedp as a wrapper for Golang, making it possible to drive Google Chrome without handling all of the shell execution. Sure, you still need to launch Google Chrome, but the library took care of that. I also then learned about --ignore-certificate-errors, which meant the death of my crappy proxy. Overall, using chromedp to do the heavy lifting worked out great. There were (still are?) definitely some bugs in v2, but the most painful issue was the "preflighter" concept that I used. Basically, instead of having Chrome browse to a URL that would inevitably fail (and thereby waste time), I had a simple (and cheap) Golang http.Client perform a so-called preflight to determine if a URL was up. If it was, we'd continue to let Chrome browse there and take a screenshot. The problem though was that the preflighter would often fail (turns out, behaving like a browser can be hard). This meant that results would be incorrectly missed, and that wasn't great.

So, for version 3, I removed the preflighter. Now, Chrome is used for all requests, and results are recorded based on events emitted by CDP. This meant a significant increase in probing accuracy at an acceptable performance cost. Better, but not yet amazing. While spending time on this I discovered rod. Excitedly, I read the documentation and began experimenting. It was... very fast. Like, almost unbelievably fast. At this point, I decided it was time to ditch chromedp and move to rod. It's clearly the winner.

Unfortunately, that win did not last long. I used a cleaned-up version of the Tranco list (removing obvious porn, other NSFW content, etc.) to test scanning with, and found that most websites probed fine from a network perspective, but the screenshots would often fail for no obvious reason. More often than not, actually, and that began my deep dive into why. I'll give you the short version of a few late nights. It seems like if you use a single browser process with tabs, screenshots fail often. Use a fresh browser for each screenshot, and your accuracy goes up significantly. I honestly don't know why. Talking to @singe about this, he asked about per-process ulimits and whatnot, and maybe that could be it. Regardless, because of all the testing I was doing, switching between chromedp and go-rod, I ended up implementing what I'm now calling "drivers". With gowitness v3, you can use gorod as the scanning driver by setting --driver gorod, with chromedp being the default still.

The question though is, what's the difference then? In both driver cases, when the tab strategy is used for screenshots, I'd reliably get poor screenshot accuracy. So, I changed chromedp to spawn a new browser window for each target and kept go-rod using a tabbed strategy. This means, for accuracy use the default driver, chromedp, but for speed (or if resource usage is an issue), use gorod.

readers

Instead of defining how source data should be read in the commands themselves, for version 3, I built the concept of readers which makes them easier to maintain and ultimately reusable and extendable! Readers all implement the same interface, which looks like this at the time of writing:

type Reader interface {
 Read(chan<- string) error
}

The Read method will receive a channel that accepts strings. This means that any reader, regardless of how it sources candidate URLs (i.e., a database, a file, a CIDR parser, Nessus, etc.) should ultimately write a string candidate (as a full, well-formed URL) to the channel. It's a simple concept but powerful when combined with a runner. When a runner is created, it prepares a Targets channel that you can write to. Combining them means we'll have something like this (using the file reader as an exam...

2.5.1

What's Changed

• html a-tag unclosed bug fixed by @gorgiaxx in #187
• Fix database file path location parser by @mr-pmillz in #188
• fix: typo by @testwill in #201
• (fix) MarshalJSON spells with one "l" by @alexandear in #198
• Fixing the missing port issue with Nessus by @quentinpraz in #191

New Contributors

• @gorgiaxx made their first contribution in #187
• @mr-pmillz made their first contribution in #188
• @testwill made their first contribution in #201
• @alexandear made their first contribution in #198
• @quentinpraz made their first contribution in #191

Full Changelog: 2.5.0...2.5.1

1edfe2209731e68621006b3cc0376d6a4b97e85e  gowitness-2.5.1-darwin-amd64
3c817d57d704a3de1dcb084a59b77e684dddb154  gowitness-2.5.1-darwin-arm64
19c75086ac90ae2891f24fa89d76c84748eb3d06  gowitness-2.5.1-linux-amd64
105ecb560afccd5f29fe1d748c32862848f556df  gowitness-2.5.1-linux-arm64
e893957658f560911864eeecec606c0cd9ea8e05  gowitness-2.5.1-linux-armv7
32315a0b50ccd4e28b941a4b0121200b6625eb9e  gowitness-2.5.1-windows-amd64.exe

2.5.0

new

• Make URLs clickable in static report exports (thanks @initstring) (via #172)
• Add ability to execute externally introduced JavaScript via the new --js flag on screenshotted pages. (thanks @djallalzoldik) (via #180)
• Add PostgreSQL support (thanks @habitualdev) (via #166). You should now specify the database to use using a full URI. Eg: sqlite://gowitness.sqlite3 (the default), or postgres://user:pass@host/database.
• Add search API endpoint (docs: search (thanks @habitualdev) (via #166)
• Add ability to store screenshots in the database using a new flag --screenshot-db-store (thanks @habitualdev) (originally via #166 but refactored in 62d6de3). The report viewer will automatically fallback to the filesystem if database screenshots are not available.
• Add HTTP response code filtering, controlling which HTTP codes get screenshotted. (thanks @nickspring) (originally via #137 but implemented in 1503c5b)

fixes

• Fix report server range selector. (thanks @randomactsofsecurity ) (via #163)
• Fix example nessus file syntax (thanks @brettgus) (via #170)
• Prevent duplicate file extensions added to filenames (thanks @maik-s) (via #183)

other

• Bump dependencies and Go version (in 4907cea, 921811d, a7941d8)
• Remove deprecated use of io/ioutil (in f939dec)

new contributors

• @brettgus made their first contribution in #170
• @initstring made their first contribution in #172
• @maik-s made their first contribution in #183
• @djallalzoldik made their first contribution in #180
• @habitualdev made their first contribution in #166

Full Changelog: 2.4.2...2.5.0

72f9578e558527bd5e8c6212d5e63b18867cd0b7  gowitness-2.5.0-darwin-amd64
d3fa213e6e0c8543256e26c1c3b3b71f23175485  gowitness-2.5.0-darwin-arm64
5f76bc689612b0b1ca5266834c76cb0c55a120b8  gowitness-2.5.0-linux-amd64
9c567241d9202689d395e704735fd8f3c1a47cfb  gowitness-2.5.0-linux-arm64
c8005cc40c8a7a9c1c690737e14853c829263acd  gowitness-2.5.0-linux-armv7
d906179afa9b59950a3ded496f9c0ede260c2cb7  gowitness-2.5.0-windows-amd64.exe

2.4.2

fixes

• Improve web UI reverse proxy support when served under a sub directory. A new wiki article was also added (in e904933) (thanks @random-robbie for PR's prompting this).

other

• Bump dependencies and build with Go 1.19 (in 3b2aceb)

8577bca1f581d7f163144b5c9068861fcf401524  gowitness-2.4.2-darwin-amd64
64b7469d97a511650f1efb74499878ad3ef8e76d  gowitness-2.4.2-darwin-arm64
e98b223ae71ef7a8df75ba0a6461b6b772a2176e  gowitness-2.4.2-linux-amd64
18556f2b0b856d90865ddae653c2e05182541e37  gowitness-2.4.2-linux-arm64
adb157143d84ff697c0cf965df3ce04ae34a5a48  gowitness-2.4.2-linux-armv7
6de652cf5ddf8f4f541f6172f127d23747353097  gowitness-2.4.2-windows-amd64.exe

2.4.1

new

• Add ability to specify additional headers when screenshotting via the HTTP api. See the API docs for a usage example (in f5e2aea).
• Add linux/arm and linux/armv7 targets to release binaries (in 12f01a9)

fixes

• Fix nmap command example documentation. (thanks @crypt0rr ) (via #138)

other

• Build release binaries with Go 1.18.4 (in 31c6e64)
• Bump dependencies (in 2041445)
• Fix internal flag name typo (in 86134ed)

8c5dfc2b7f5a66aec4d861522c2d78452e1950ac  gowitness-2.4.1-darwin-amd64
bd4026cad944b6143fbad90b0a9f2e41671509f6  gowitness-2.4.1-darwin-arm64
82c56c41caf8e1474adc851a90df48cb5b7c9ee1  gowitness-2.4.1-linux-amd64
b6391b19c6b5316e74c18f8c05dcec1987186170  gowitness-2.4.1-linux-arm64
9f79dfa11aaf6788a1150da41c12da03185e26af  gowitness-2.4.1-linux-armv7
5ec37979e14290200f0d0919006102cefa89edd2  gowitness-2.4.1-windows-amd64.exe

2.4.0

66 commits later, this is a major release of gowitness with many new features, fixes and overall polish. Some screenshots to see what the updated report server UI looks like is below, followed by the change log for this release. Enjoy!

The new Dashboard view

A dark themed, detailed view:

A light themed, detailed view:

new

• Add application technology identification using Wappalyzer (via #104 and #110 and #127) (in 3a80bf6 and e79214e and 65816c6) (thanks guervild and Ice3man543 and terrabitz)
• Improve performance in HostsInCIDR() method (via #107) (in 71125b2) (thanks NickChillClub)
• Add a Nessus parser (via #123) (in e1923bb) (thanks randomactsofsecurity)
• Add additional header support (via #124) (in c7b874a) (thanks randomactsofsecurity)
• Refactor the webserver to make use of Gin (in 6fad6c3)
• Various web UI dependency upgrades, fixes and layout updates (via 05f3c32)
• Add new API endpoints. API documentation can be found in the Wiki here (via 49c8702 and 6769e88)
• Add a new dashboard view (via 00f9394)
• Restore the ability to export static HTML reports again (via 3e459eb)
• Record both browser console logs and network events as emitted via Chrome. These fields are searchable and make it possible to see the IP's that hosts resolved to at the time of the screenshot (in 396de21 and 3efb7b0)
• Add a simple pager to the detail view (in 0c14e67)
• Add a theme switcher for light mode or dark mode, defaulting to dark mode (in 0cbdafb)
• Add the ability to dump and save the DOM (in 453a0fc)
• Significantly refactor the search feature to more agressively search through collected data. This includes URL's, DOM's, network events, console logs and more. (in 78303cb)
• Add the ability to save screenshots as PDF's instead of PNG's (in 2f87924)
• Add an example docker-compose.yml file to show how the report server could be used when exposed to a larger network.

fixes

• Improve reliability, dealing with some cases where Chrome may hang (via #132) (in 79c4f6e) (thanks rtpt-jonaslieb and randomactsofsecurity)
• Prevent screenshot filenames from becoming overly long (in 19b4e15)

other

• build release binaries with Go 1.18.2 (186f57e) (8b88b47) (19c1827)
• build releases for macOS ARM (96df49e)
• bump dependencies (3092a86) (861543e) (01005bb) (af7a504) (c6da64d) (0002ef9)
• use github actions to build docker image (ff1241d)
• replace vfsgen with native Golang embed for web assets (d98ae0c)
• Bump the default User-Agent string used (via ec73fbb)

76065c1c937630e44ecde32abfc0fd945cb20483  gowitness-2.4.0-darwin-amd64
b556b7f45a1b313a1686843f219cf8b045ad0e48  gowitness-2.4.0-darwin-arm64
d361fe3cdf738b0fe60b204a03017e3b4b38ffb5  gowitness-2.4.0-linux-amd64
0f91805c85dd665758e205dda8e8edf09dacb498  gowitness-2.4.0-windows-amd64.exe

2.3.6

This is primarily a security/hardening release.

fixes

• Limit the allowed URI's that may be submitted to the screenshot or report server to only those starting with http / https by default. You can use the new --allow-insecure-uri / -A flag to disable this. Take note that with the -A flag, it means someone could screenshot file:// URI's and read local files on the host filesystem. To combat some of this abuse, by default the report & screenshot servers listen on localhost only. However, if you are exposing the report or screenshot servers to the Internet (or other untrusted networks), make sure you restrict access to it as other problematic URI's such as localhost and cloud metadata URIs (and any other SSRF vector) will also be reachable this way. (57dffb7) (thanks to Omri Inbar from Checkmarx for reporting the LFI).

other

• go-staticcheck fixes and other code cleanups (1149417)

8a2ca3dc8a58ce3e103aeabd13df7713c0322b2c  gowitness-2.3.6-darwin-amd64
b50938b99af45d7bc209428a648f057b11a6025f  gowitness-2.3.6-linux-amd64
1dcee72acdf074f1850263643ca9297b0d5b38e3  gowitness-2.3.6-windows-amd64.exe