September 12, 2019
An improved and flexible model to expire indicators
This blog post aims to give a thorough introduction of a new functionality added in MISP 2.4.116, allowing users and organisations to easily expire information depending on their personalised objectives and targets.
MISP, being a distributed system, enables the sharing of data between various users and organisations, often resulting in the parties involved in an exchange not even knowing one another. Whilst having access to a large trove of information is extremely beneficial for all parties involved, however, it can also introduce a whole new set of challenges to deal with.
In this blog post, we will mainly touch on information quality and freshness, along with other issues such as trust, use-cases and interests. The latter group is partially taken into account, but will not be presented. Nevertheless, these concepts are examined more thoroughly in a paper we published that has served as our basis for the implementation, along with a detailed explanation of the solution we have chosen to tackle these issues.
Our main objective is to provide users with a simple yet customisable system to automatically (or in some cases manually) mark an Indicator Of Compromise (or more generically, an Attribute) as expired.
Before getting started with showing how the model presented in the paper is implemented in MISP, we first need to have a look at some of the basic concepts required to better understand how the various components are working together.
The (potentially) annoying bits of the theory
The solution currently supported in MISP is based on two components: The
base_score and the
score. The idea is to have an initial fixed value called
base_score taking into account the quality of an indicator and a time-dependant
score, which decreases with the passage of time.
A simplified version would look something like this:
score = base_score * P
P is composed of the following
lifetime: The lifetime of the IOC or the point in time at which the score of the Attribute’s score will reach 0
decay_speed: The speed at which the decay happens or the speed at which an Attribute will lose score
⚠ It should be noted that everytime a Sightings is added to an Attribute, the
score is refreshed back to the value of the original
base_score and a new decay is initiated from that point forward.
Polynomial Decaying Model built-in in MISP
We still have to see how the
base_score is actually computed. In the current implementation of the Decaying Model in MISP, the
base_score is computed from the Taxonomies along with the various attached weights. Weights are a means to prioritise extracted
numerical_values from Taxonomies over others.
To give the intuition of how the
base_score computation works, let’s look at two examples. In these examples, the two Taxonomies used are
phishing and admiralty-scale. Both of them contain Tags that have a
numerical_value associated to them:
numerical_value = 100
numerical_value = 25
numerical_value = 75
So, if an Attribute only has a single Tag attached, for example
admiralty-scale:source-reliability="Completely reliable", the
base_score would be:
base_score = 100
Weights come into play when multiple Tags are attached to an Attribute. As a simplification, let’s suppose that both Taxonomies should have the same importance in regards to the Attribute’s score. Thus, the total weight (100) will be shared, assigning both Taxonomies a weight of 50.
admiralty-scale = 50
phishing = 50
If an Attribute has the Tags and attached, the computation steps would look like this:
base_score of this Attribute will be
Now that we’ve seen the basic concepts, let’s have a look at how MISP implements these components. For these examples, we are using the default phishing model model on a test Event.
Integration in MISP
At the Event level, a new filter button has been added, which attaches the real-time computed
score to all Attributes that have been mapped to a Model.
attribute/restSearch endpoint has been updated and now supports four new parameters to help with filtering out expired Attributes or to simply play with the different available models.
includeDecayScore[bool]: Attach the real-time computed
scoreof the Attribute along with the associated Model information
excludeDecayed[bool]: Filter out all expired IOCs
decayingModel[list]: List of Model(s) that will be attached to the given Attributes
modelOverrides[dict]: JSON that can be used to modify Model parameters on-the-fly
// attribute/restSearch query that gets every `ip-src` attributes being tagged with tlp or phishing,
// not being expired,
// with a overriden model threshold of 30 for the two models with id 84 and 12.
"decayingModel": [84, 12],
Default and Custom Models
In MISP, Some Decaying Models, called Default Models, will be supplied by default. Similarly to Taxonomies, Galaxies and misp-objects, Decaying Models will have their own repository along with the possibility to be updated directly from both the API and the UI via a single click. Default Models are available to everyone, meaning that they can been viewed and customised by any user having a presence on the MISP instance.
Custom Models are user-defined models that are shared with other users. However, if desired, they can be kept private to one’s own organisation by turning off the sharing flag, similarly to how the Tag Collection feature handles the same concept.
Decaying Fine Tuning Tool: Setting parameters and mapping model to Attribute types
When creating a new Decaying Model, modifying its parameters and viewing the outcome of the any changes should be as easy and straight forward as possible. In order to do so, several widgets are included with the latest version of MISP.
Customising the lifetime and the decay speed parameters
base_score: Customising Taxonomies’ weights
Viewing scores and simulating the model
Developer perspective: Creating a model using a different algorithm
The Built-in Polynomial Decaying Model implemented in MISP allows users to customise various of its components in order to achieve the desired decay behaviors in a fine-grained manner. Nevertheless, even with the flexibility offered by the system, it is absolutely possible that our model doesn’t encompass your specific use-case’s needs. Thanks to the implemented architecture, any other formulae or algorithms can be added and used in a straightforward way.
Steps to create a new decay algorithm:
- Create a new file
- Extend the Base class as defined in
- Implement the two mandatory functions
isDecayedusing your own formula/algorithm
- Create a Model and set the
class Polynomial extends DecayingModelBase
public const DESCRIPTION = 'The description of your new decaying algorithm';
public function computeScore($model, $attribute, $base_score, $elapsed_time)
// algorithm returning a numerical score
public function isDecayed($model, $attribute, $score)
// algorithm returning a boolean stating
// if the attribute is expired or not
Evaluating the quality and freshness of IOCs is a problem commonly found in Threat Intelligence Platforms. We have tried to solve it using a simple yet customisable system.
Event though the small set of models included in the upcoming MISP release should fit the most common use-cases, we are eagerly awaiting any contributions, fine-tunings or feedback from users. We consider the current implementation to be a foundation, upon which we want to gradually build using both our own findings and the feedback of the community. These would opens us up to plenty of opportunities, including the inclusion of new Models, improvements to the precision of existing Models’, tweaking of the various parameters and even the integration of machine learning algorithms as new Models.
Furthermore, we are not done yet! There are already a bunch of improvements cooking at the MISP-Project kitchen,
- Integration of
- Tweaks to the formula in order to provide better controls in regards to reseting the
base_scoreonce a Sighting has been added
- Per-user overrides for the Taxonomies’
- Weights on a Tag’s predicate level