Changelog

Version 1.9.0

Release 2023-05-11

• DeTT&CT now supports Mobile data sources which are introduced in MITRE ATT&CK version 13.

Version 1.8.0

Release 2022-12-21

• DeTT&CT now supports ATT&CK Campaigns. It's included within the Group mode of the DeTT&CT CLI that allows you to make heat maps and overlays for both groups and campaigns. Because of this, we changed the --software-group option to --software to also support campaigns.
• We added a new option to the Group mode: --include-software. Thanks to beerMT. He came up with the idea to include software techniques in the scores of the heat map when a threat actor uses specific software. Until now we only had the option to show what software is used (--software -group), but that option did not influence the score.
• We extended the cache expiry period for ATT&CK information from 24 hours to 7 days.

Version 1.7.0

Release 2022-10-04

• With the financial sponsorship of the Dutch National Police, we added support for ATT&CK Mobile to DeTT&CT.
• Due to overlapping STIX ID's for threat actor groups in multiple ATT&CK matrices, some groups were not to be found. This issue is now solved.
• Due to inconsistent name and alias lists for threat actor groups in ATT&CK STIX data, some groups were not to be found. This issue is now solved.
• Within the Editor DeTT&CT data sources were visible for ICS (having platform=all), while those are not applicable to ICS.

Version 1.6.0

Released 2022-02-08

CLI

• With the financial sponsorship of the Cyber Security Sharing & Analytics (CSSA), we added support for ATT&CK ICS to DeTT&CT.
  • In the current ATT&CK release of ICS, there are inconsistencies between the data on the ICS wiki and the STIX objects. Be aware that the ICS data from STIX is leading for DeTT&CT, and thus not the wiki because that cannot be accessed via an API. For more information see this page: 7. ICS - Inconsistencies.
• Improved the data source statistics (python3 dettect.py ge -ds) by adding:
  • The option to only include data sources for selected platforms.
  • The corresponding ATT&CK platforms per data source in the output.
• Removed the interactive menu. We have decided to do this for the following reasons:
  • Our list of improvements and new features for DeTT&CT is long. Therefore, we want to spend as much time as possible on improving the core of DeTT&CT and less on maintaining functionality that is already provided differently.
  • The interactive menu has not been kept up to date with the latest features and thus features available from the command-line interface.
• Removed the functionality to update the technique administration YAML file to ATT&CK with sub-techniques.
• Numerous small improvements.
• Updated all Python dependencies.
  • Due to Pandas being updated to version 1.4.0, the minimal required Python version is now 3.8.

Editor

• With the financial help of the Cyber Security Sharing & Analytics (CSSA), we added support for ATT&CK ICS to DeTT&CT.
• Numerous small improvements.
• Updated to Node.js version 16
• Updated all JavaScript dependencies.

Version 1.5.0

Released 2021-12-20

Generic

• We've added multiple custom data sources (Web, Email, Internal DNS and DHCP) as an extension to the native ATT&CK data sources. We call these custom data sources: DeTT&CT data sources. These data sources will significantly improve the automatic calculation of your rough visibility based on the number of available data sources. In addition, it provides the capability to score and administrate these important data sources separately. You can find more information here.
  • Please note that your rough visibility score will be lower for some of the techniques because we've added DeTT&CT data sources.
• Sample-data: the technique administration is now in sync again with the data source administration.

CLI

Data sources - Applicable to / type of System

Similar to the technique administration file, we added support for applicable to within the data source administration. The CLI automatically upgrades version 1.0 data source administration files to data format v1.1.

This upgrade can only ensure the data format will be in line with v1.1. But cannot handle how you've recorded information on your data sources. It's therefore advised to put some manual work into the data source administration file after this upgrade. For example, to do things like:

• Assign data sources to the correct type of Systems (which are furthermore linked to ATT&CK platforms).
• As we recommend and explain here, have matching Systems/applicable to between your technique and data source administration YAML file.
• Merge multiple data source files into one single file when you had multiple data source files per ATT&CK platform, type of system, environment, etc. The new v1.1 data format supports combining all of that within the same data source YAML file using the new Systems object.

You can find further information on this new applicable to/type of System functionality here.

Other CLI Changes

• Within the datasource mode, the platform filter argument (-p/--platform) has been replaced by an option to filter on applicable to value (-a/--applicable_to).
• Added a graceful exit of DeTT&CT when MITRE's CTI server could not be reached.
• The following functionality has been removed:

  • Upgrading a technique administration file from version 1.0 to 1.1 and version 1.1. to 1.2.

  • Letting you know that you are missing specific data sources within your data source administration file. This was implemented in the health check and Excel output.

    We noticed that this check could be bothersome when you knew that a data source was missing. We have implemented new features within the Editor to get you informed on relevant data sources.

• Support for DeTT&CT data sources: Web, Email, Internal DNS and DHCP. You can find more information here.
• Updated all Python dependencies.
• Numerous small improvements.

Editor

• Data sources
  • Added support for the data source schema version 1.1, including support for:
    • Editing the Systems object with its applicable to values and corresponding ATT&CK platforms.
    • A drop-down menu to link a data source to one or more Systems/applicable to values.
  • Improved the autofill dropdown for data sources to only show data sources which are not yet administrated and apply to the included ATT&CK platforms.
  • Added a new button to add all data sources at once for the ATT&CK platforms in scope.
    Source of the idea: @SecurePeacock
• Techniques
  • Auto suggest list for applicable to values.
• UI improvement: collapsable file details section (will close on scroll).
  This behaviour can be prevented by using the lock icon.
• Support for DeTT&CT data sources: Web, Email, Internal DNS and DHCP. You can find more information here.
• Updated all JavaScript dependencies. (already published before the release of 1.5.0)
• Numerous small improvements and bug fixes.

Version 1.4.4

Released 2021-10-22

CLI

• Added support for the ATT&CK v10 data sources.
• Navigator layer files now have better default settings for the score aggregation. (already published before the release of 1.4.4)
• Numerous minor bug fixes. (already published before the release of 1.4.4)

Editor

• Added support for the ATT&CK v10 data sources.
• Numerous minor bug fixes. (already published before the release of 1.4.4)
• Numerous small improvements. (already published before the release of 1.4.4)

Version 1.4.3

Released 2021-04-30

CLI

• Added support for the revamped data sources introduced with ATT&CK v9. Please note that this version of DeTT&CT no longer supports the old data source names as they are simply no longer part of the most recent version of ATT&CK. Using them is still possible with version 1.4.2 and a local copy of ATT&CK v8 provided to DeTT&CT with the argument --local-stix-path.
  • You can find more information on ATT&CK v9 and the new data sources on this blog post from MITRE, the data source YAML files (also from MITRE) and on this page on the Wiki.
  • We currently do not yet support data source to technique mapping (to calculate the rough visibility score per technique) for the PRE platform. Support will be added once MITRE has, in a future release of ATT&CK, defined the data sources for this platform.
• Added support for the ATT&CK Navigator version 4.3 and layer version 4.2.
• Bugfix:
  • Issue #40 reported by @sherlon1. A crash could occur in the interactive menu when doing a group overlay. (already pushed to master before the release of 1.4.3)

Editor

• Added support for the revamped data sources introduced with ATT&CK v9.
• Multiple UI improvements. (already published before the release of 1.4.3)
• Updated all JavaScript dependencies. (already published one time before the release of 1.4.3, and for a second time with this release)

Generic

• The sample data source YAML files have not been updated yet to reflect the new data source of ATT&CK v9. We choose to postpone this for a later time to allow a quicker release of v1.4.3.

Version 1.4.2

Released 2020-11-04

CLI

• Added support for the new platforms PRE and Network.
• Updated the data sources per platform mapping.
• Added support for the ATT&CK Navigator version 4.0.
• Removed support for PRE-ATT&CK from the Group menu (PRE-ATT&CK has been replaced by the new platform PRE).
• Bug fixes:
  • all as a platform value for the argument -p/--platform, to include all ATT&CK platforms, was broken.
  • Issue #36 reported by @sherlon1. A crash could occur when retrieving data sources for a technique, in ATT&CK CTI, which had no data sources.
  • Two small bug fixes in the data source and technique administration health check.
• Updated all Python dependencies.

Editor

• Added support for the new platforms PRE and Network.
• Updated all JavaScript dependencies.

Generic

• Updated the file data-sources-empty.yaml to include all ATT&CK data sources.

Version 1.4.1

Released 2020-10-24

CLI

• Added a new argument (-p/--platform) to the data source, detection and visibility menu that allows you to overwrite, when generating a Navigator layer, the platform value(s) as specified in the YAML file.
  • This also improves the group menu, as this now allows you to specify multiple ATT&CK platforms by providing extra -p/--platform arguments.
• Changed how ATT&CK Groups are specified within the group menu. No longer are multiple Groups provided using a double-quoted string in which Groups are separated by commas. Instead, multiple Groups can be provided by additional -g/--group arguments.
• Updated all Python packages.
• Bug fixes:
  • Crash on updating a techniques file based on a data source when having null values in the date key-value pair in the visibility score_logbook. (already pushed to master before the release of 1.4.1)
  • Issue #36 reported by @driesbuyck. DeTT&CT crashed when generating a detection or visibility layer file when having a technique administration file with different Python date formats. (already pushed to master before the release of 1.4.1)
  • Detections with a score of -1, or visibility items with a score of 0 were included in the graph showing the progression of added detection/visibility over time. (already pushed to master before the release of 1.4.1)
  • Within particular circumstances the update of visibility scores, based on updated data sources, would not write the updated technique YAML file to disk.
  • Techniques with a detection score of 0 and a visibility score of 0 where coloured white within a detection/visibility overlay instead of purple.

Editor

• Moved the maximise icon within text fields more to the left to improve the user experience for browsers running on Windows.
• The list editor for the detection's locations no longer shows empty values. This improves the user experience.
• Removed the service worker module to solve a caching problem that could prevent a new version of the Editor from being loaded in the browser.
• Updated all JavaScript dependencies.
• Bug fixes:
  • The detection score slider was missing the score 0 (already pushed to master before the release of 1.4.1)
  • A very long group name would run off the page.

Generic

• Added threat intelligence data from Cisco Talos: 20200901-Cisco-Talos.yaml
  • (already pushed to master before the release of 1.4.1)

Version 1.4.0

Released 2020-07-13

CLI

• Added support for sub-techniques. This includes:
  • A new command-line option (--update-to-sub-techniques) to update the techniques administration YAML file to the new ATT&CK sub-techniques. Most updates are automated using the crosswalk provided by MITRE. Some manual actions are required for techniques which cannot be automatically migrated. These are listed after: Messages that need your attention:.
  • A function that checks if a techniques administration YAML file needs to be updated to ATT&CK sub-techniques.
  • Support for the new Navigator Layer 3.0 format.
• Added a new option (--local-stix-path LOCAL_STIX_PATH) to use local STIX objects instead of using the TAXII server. Can be used to use DeTT&CT offline or to use a specific version of STIX objects.
• When an unknown technique exists in the techniques administration YAML file, DeTT&CT will ignore and continue, but it will also report this.

Editor

• Added support for sub-techniques.
• Added navigation buttons to easily navigate through the list of data sources and techniques.
  • Keyboard shortcut: Ctrl+Shift+Up/Down: go to the next or previous item when editing a data source or technique administration YAML file.
• Updated all JavaScript dependencies.

Generic

• Moved existing threat actor data into a new folder pre-sub-techniques.

Version 1.3.1

Released 2020-06-22

CLI

• All overlays now have shades of colours. When comparing a group with detection coverage the orange colour (a threat actor uses the TTP and you have detection) has shades of orange that reflect the detection level. Also, the green (detection) and blue (visibility) in overlays now have shades of colours.
• New options:
  • The output filename for the data source, visibility, detection and group modes can now be specified: -of OUTPUT_FILENAME, --output-filename OUTPUT_FILENAME
  • The name as shown within a Navigator tab can now be specified: -ln LAYER_NAME, --layer-name LAYER_NAME
• Improved the information displayed within the metadata for all type of overlays. For example, when comparing detection coverage with group data.
• Updated all Python packages.
• Bug fixes:
  • The date format in an auto-updated YAML file conflicted with the date format used in the Editor.
  • The health check crashed when the value for the key-value pair location was not a YAML list.
  • Detections with score=0 (Forensics/Context) were not shown in some layer files.

Editor

• Updated all JavaScript dependencies.
• Added a Notes text field to the File Details section of Data Sources, Techniques and Groups.
• Bug fixes:
  • Within a specific scenario, a YAML file was created with an empty score_logbook.

Generic

• Changed the base image in the Dockerfile to python:3.8-slim-buster resulting in a smaller image and significant decrease in time to build the image.

Version 1.3.0

Released 2020-03-18

• YAML files can now be edited by loading them into the DeTT&CT Editor. It's no longer necessary to edit YAML files using a text editor!
  • All code in the Editor is running within the browser. Therefore, the content of your YAML file is not send to a server.
  • The Editor is hosted on GitHub and can be found here. The Editor can also be run locally using the following command: python dettect.py editor
  • With a few exceptions, all key-value pairs within a data source, techniques or group YAML file can be edited. More info can be found here.
  • Please note that comments (#) within your YAML files are not preserved due to lack of support in the YAML JavaScript library. Put your comments within a key-value pair to keep them. E.g. my-comment-1: your comment goes here.
  • Contributed as a beta tester @rcfontana. Thanks!
• Bug fixes:
  • The logic to determine if a data source was available or not contained several errors.
  • Using a lowercase value for the key-value pair platform in a data source YAML file resulted in an error.

Version 1.2.7

Released 2020-02-10

• The automatic scoring of visibility (based on the number of available data sources) is now more accurate. This was mainly necessary after the introduction of the cloud platforms and data sources in the 2019 October ATT&CK update.
  • On this page you can find which data sources are applicable per platform. We created this specifically for DeTT&CT and is thus not part of ATT&CK.
• Upgraded all used Python packages to their latest version.
• Several small improvements.
• Bug fixes:
  • A data source administration YAML file without the exceptions key-value pair resulted in an error. Reported by @s4vgR.
  • A group YAML file without the software_id key-value pair resulted in an error. Reported by @mavjs.
  • Within specific circumstances, an invalid health error message was shown.

Version 1.2.6

Released 2019-12-17

• It is now possible to perform an EQL search on custom key-value pairs of a technique administration YAML file.
• Added new functionality to support a platform key-value pair in a group YAML file.
• Added a new feature to the data source menu to include all ATT&CK techniques in the generated YAML file (when the argument -y, --yaml is provided) that apply to the platform(s) specified in the data source YAML file:
  • --yaml-all-techniques
• Revoked ATT&CK STIX objects are now removed from the results that are retrieved from the ATT&CK TAXII server.
• Added new functionality to make sure the metadata in a Navigator layer file is compliant with the expected data structure.
• Upgraded all used Python packages to their latest version.
• Several other small improvements.
• Health checks:
  • Added a check for when the data source YAML administration file is missing one of the ATT&CK data sources.
  • Added a check for an empty item in the key-value pair 'location' (in a detection) and 'applicable_to'.
• Bug fixes:
  • A bug that could result in an invalid message in the Excel for a missing ATT&CK data source.
  • An Excel export for a technique administration YAML file would cause a crash when having an empty/None value for a detection or visibility comment. Reported by @Sreemanshanker.
  • Within specific circumstances a wrong colour for visibility was used when detection coverage is overlaid with visibility. Reported by @Sreemanshanker.

Version 1.2.5

Released 2019-11-19

Fixes for two bugs related to the data source administration YAML file:

• Using 'all' for key-value pair 'platform' to generate a technique administration YAML file did not work.
• EQL searches on data source YAML files were broken.

Version 1.2.4

Released 2019-11-14

Fixes for two small bugs that resulted in:

• An invalid Navigator layer file for a group/threat actor heat map, or when overlaid with a group, visibility or detection coverage.
• A crash when generating a Group Navigator layer file overlaid with a non-existing ATT&CK Group.

Version 1.2.3

Released 2019-11-05

• Added the new data sources introduced with the ATT&CK October update to the sample-data file:
  • AWS CloudTrail logs, AWS OS logs, Azure OS logs, Azure activity logs, OAuth audit logs, Office 365 account logs, Office 365 audit logs, Office 365 trace logs, Stackdriver logs.
• Added support for new the platforms introduced with the ATT&CK October update: AWS, GCP, Azure, Azure AD, Office 365, SaaS.
• Added support for using multiple platform values in the data sources administration and techniques administration files.
• Added a health check for an empty or invalid 'platform' value in the techniques administration file.
• Updated to support the ATT&CK Navigator layer version 2.2.
• A small bug fix in the health check

Version 1.2.2

Released 2019-10-17

• Added two new health checks for the data source administration YAML:

  • check on invalid technique IDs in the 'exceptions' list.
  • check on an empty or invalid value for 'platform'.

• Fixed issue #13 reported by @hRun that caused a crash when having empty technique ID entries within the 'exceptions' list of a data source administration YAML file.

Version 1.2.1

Released 2019-09-19

• Fixed a bug within the YAML visibility update functionality.
• Improved the way how EQL is integrated into DeTT&CT.
• Fixed the metadata within the sample file data-sources-endpoints.yaml for the data source "Process command-line parameters".

Version 1.2.0

Released 2019-08-22

• Visibility/Detection score logbook
  It is now possible to keep track of changes in the visibility and detection score for a particular ATT&CK technique. We have therefore introduced version 1.2 of the techniques administration YAML file:

  • Visibility and detection scores are now part of a score object in the YAML file within a score_logbook object. Due to this change, visibility scores are now also having a date key-value pair.
  • The key-value pair date_registered for the detection score is removed. A detection now has a single date key-value pair named date. You can decide, in the upgrade from v1.1 to v1.2, to keep this key-value pair even though DeTT&CT no longer makes use of it.

  For an example of score_logbook with multiple score objects (for both visibility and detection) see technique T1189 in the sample file: techniques-administration-endpoints.yaml

To allow an easy transition, older technique administration YAML file version are automatically upgraded to this new version. DeTT&CT will automatically prompt you on this.

• Automatic update of visibility scores
  The datasource mode has a new option (-u, --update) that helps you to automatically update your rough visibility scores within your technique administration YAML file. So, when you have made changes within your data source administration YAML file (e.g. you have added a new data source) this could result in changes within your visibility scores, which you can now auto-update. Another use-case for the auto-update is when MITRE ATT&CK introduced new techniques, makes changes in the data source listed for a technique or adds new data sources.

  For more info information see: Getting started / How to - Auto-update visibility scores and the use of the score_logbook.

• Exclude/include objects from a YAML file using EQL
  We have integrated EndGame's Event Query Language (EQL) into DeTT&CT. This provides you with powerful options to exclude or include certain objects (detections, visibility or data sources) from your YAML administration files. Some examples of what you can do:

  • Once you have build-up history on when detection/visibility scores have changed (within the score_logbook). You can visualise this change within an ATT&CK Navigator layer file using an EQL query. Of course also without much history, this can be done purely based on when you have added new detections or visibility.
  • You can influence the way how data sources (based on their characteristics you have administrated) are excluded or included in the process to draft a rough overview of your visibility coverage.

  Be aware that the option '-a, --applicable' has been removed and is now replaced by an EQL query. For example, to only include 'client endpoints' the EQL query for that is:

  --search-detection "techniques where arrayContains(detection.applicable_to, 'client endpoints')"

  More information on how to use EQL within DeTT&CT is provided here.

• Several smaller improvements

  • New statistics within the generic mode to get a sorted count on how many ATT&CK Enterprise or Mobile techniques are covered by a Mitigation (python dettect.py generic -m {enterprise,mobile})
  • The Python library PyYAML has been replaced by ruamel.yaml to allow better modification of existing YAML files (to preserve comments and block style and key ordering).
  • The health check performed on a technique administration file now checks for a few more errors and has a notable increase in performance.
  • A health check has been added for the data source administration YAML file.
  • A graph can be generated for the number of visibility objects added through time.
  • Output files (layers, graphs, Excel, YAML) are nog longer overwritten. Instead, a number is added as a suffix to the filename.
  • Numerous other small improvements.

• Other changes

  • Updated the Python package attackcti to version 0.2.7.
  • The Python package eql version 0.7 is added to requirements.txt.

• Bug fix

  • A red colour was shown when the visibility score was equal to 0 in a visibility Navigator JSON layer file (this fix has previously already been pushed to the master branch).

Version 1.1.2

Released 2019-06-16

• Made compatible with the latest version of attackcti (v0.2.6) resulting in the use of ATT&CK STIX objects instead of a custom schema.
• Upgraded all used Python packages to their latest version.
• Fixed a bug that caused the campaign name part of a Group YAML file not to be displayed.

Version 1.1.1

Released 2019-05-23

• Added a new option '--health' to check a technique administration YAML file on possible errors.
• Added the possibility to have a group YAML file type that contains a count on how popular a certain technique is.
  • An example has been added to the directory threat-actor-data for the Red Canary's Threat Detection Report 2019 (1st edition).
• Added both the detection and visibility score in the metadata when doing an overlay of detection/visibility on a group.

Version 1.1

Released 2019-05-02

• Technique administration YAML file version 1.1
  
  New version (1.1) of the technique administration YAML file introducing the following improvements:

  • A technique can now have multiple detection and visibility objects. This allows you to have more detailed scores for a different type of systems by making use of the new key-value pair applicable_to.
  • Added the key-value pair applicable_to to the detection and visibility object. This allows you to specify a list of the type of system(s) to which it applies. For example: crown jewel X, endpoints, etc. You can use the value ['all'] to have the detection or visibility be applicable to all type of systems.
  • Added the key-value pair technique_name, containing the techniques' name (e.g. "Process Injection"), to every technique ID.

  Older technique administration files can be automatically upgraded to this new version. DeTT&CT will prompt you on this if an upgrade is available.

• Excel output
  
  It is now possible to export your technique administration for visibility and detections to Excel:

python dettect.py d -ft sample-data/techniques-administration-endpoints.yaml -fd sample-data/data-sources-endpoints.yaml --excel
File written: output/techniques.xlsx

• Several smaller improvements

  • The detection and visibility layer file contains a score to allow sorting within the ATT&CK Navigator.
  • Added a score for detection/visibility when overlaid with a group to improve the visual comparison.
  • The ATT&CK Navigator's legend is improved when overlaying detection or visibility on a group.
  • Added colours to the Excel output to visualise the scores for data source quality, visibility and detections.
  • Remember the selected path for a YAML administration file in the interactive menu.
  • Added a more detailed error message for invalid YAML files.
  • Constants have been moved to its own file constants.py.

• Bug fixes

  • Fixed a bug reported by @tuckner: issue #3 - product list not appending for visibility ATT&CK layer)
  • Fixed a bug that would cause a crash when doing a software-group using a visibility or detection overlay.
  • Fixed a bug that would cause a crash when the YAML 'score' key-value pair had no value assigned.