The objective of this book is to compile the best practices in Threat Intelligence Analysis with the support of the Open Source Threat Intelligence platform MISP. The best practices described here are from Information Sharing communities (ISAC or CSIRT) which are regularly using MISP to support their work and sharing practices.
|Improvement of the analysis process can range from a simple notification of a false-positive or the correction of a typographic error, all the way up to a complete competitive or counter analysis of the original analysis.|
A common difficulty in threat intelligence is to improve existing analyses and especially how to do it efficiently. One of the main questions to ask is: what will be the target audience of the improved analysis and the objective thereof?
Informing the original analyst/author (e.g. a security vendor or a CSIRT) about a specific mistake or error which needs to be corrected.
Improving an existing analysis by performing a complementary analysis or review which will be shared to and used by another group (e.g. a specific constituent, or a team within your organisation or a member of an ISAC, etc).
In the first case, MISP includes a mechanism to propose changes to the original creator, a mechanism we refer to as proposals. By using proposals, you can propose a change to the value or the context of an attribute (such as a typographic error in an IP address, missing contextual information, type of the information, the category or the removal of an IDS flag). The proposal will be sent back to the original author who can decide to accept or discard it.
The advantages of using the proposal system include the lack of a need to create a new event as well as the process itself being very simple and fast. However, it assumes that the party providing the improvements is willing to lose control over the proposed data. This is pretty efficient for small changes but for more comprehensive changes, especially those that include non-attribute information such as galaxy clusters or objects, the event extension is more appropriate.
Apart from being more suitable for more comprehensive changes, the second scenario is also a great fit for the extended event functionality, allowing users wanting to provide additional information or an alternate view-point with the opportunity of creating a self-contained event (which can have its own custom distribution rules) that references the original analysis. This information can be shared back to the original author or kept within a limited distribution scope such as a specific sector, a trust group or as internal information for the organisation providing the additional information.
|For more information about the extended event functionality in MISP, the blog post Introducing The New Extended Events Feature in MISP includes a lot of details.|
What To Share or What Counts As Valuable Information?
|Valuable information is a moving concept and highly depending of the goal of the users sharing and/or using the information. A valuable information can also evolve following the capabilities of an organisation.|
Contribution comes in various shapes and sizes.
Information which is often distributed within sharing communities are the following:
Analysis report of a specific threat (such as security vendor report, blog post) which can be Open Source intelligence or come as limited distribution
Enhanced analysis of an existing report (such as data qualification, competitive or counter analysis)
A post-mortem analysis of an incident
Additional information about existing or known threats (such as adversary techniques, new malware samples or complementary discoveries)
False-positive or false-negative reporting
Asking for contribution or support from the community (such as "have you seen this threat?" or "do you have more samples?")
|By having a look at the object templates or the MISP attribute types, this can help you to discover what is actively shared within other communities. If a type or an object template is not matching your data model, you can easily create new ones.|
|When asking for the support of the community, using a specific taxonomy such as collaborative intelligence to express your needs might help everyone and improve automation.|
Expressing confidence/estimative probability in an analysis
|Expressing the confidence or the lack of it in an analysis is a critical step to help a partner or a third-party to check your hypotheses and conclusions.|
Analysis or reports are often shared together with technical details, but often lack the associated overall confidence level.
Adding confidence or estimative probability have multiple advantages such as:
Allow receiving organisations to filter, classify and score the information in an automated way based on related tags
Information with low-confidence can still be shared and reach communities or organisations interested in such information without impacting organisations filtering out by increased confidence level
Support counter analyses and competitive analyses to validate hypotheses expressed in original reporting
Complement analysis with contrary evidences is also very welcome to ensure the original analysis and the hypotheses are properly evaluated.
|MISP taxonomies contain an exhaustive list of confidence levels including words of estimative probability or confidence in analytic judgment.|
|threat-intelligence.eu includes an overview of the methodologies and process to support threat intelligence.|
How to track and keep the state of an analysis
|Having a workflow to follow and be able to refer to is something useful for the analyst as well as for other people reading or relying on the analysis.|
Keeping track of the advancement of an analysis, of what has been done or still needs to be done is important in order to not forget anything on one side or to ensure work is not performed redundantly by accident. It is essential to have a method to keep these information clear and concise.
One of the possible methodologies is to use tags to mark the information and convey the current state of an analysis.
For instance the MISP Workflow Taxonomy allows the user to describe the state of an analysis, as
incomplete. Moreover, it can be used to clearly specify what still needs to be done using the
todo tags. The workflow taxonomy is separated into two parts. One part is related to the actions to be done (
todo) and the other part is about the current state of the analysis(
state) such as
|For more information on the MISP Workflow Taxonomy, please feel free to read Workflow taxonomy cheat sheet.|
There are several factors for not only successful but efficient intelligence sharing. Certainly, one major aspect is the quality of the indicators (or observable depending on the definition you use), stored as attributes within a MISP event itself. However, it does not stop there. Even the most viable information gained by a shared event can render itself complete useless if not classified and tagged accordingly. One feature which enables a uniformed classification is implemented in MISP as tags. Currently, there are two types of tags, which diverse in the place where they are set. For one, you can add tags to an entire event. These tags should be valid for any individual attribute, thus indicator associated to this specific event. For a more fine-grained specification all of these tags can also be placed at attribute level. This allows the user to put a more detailed and selective view on each attribute.
|In future releases there will also be tagging for MISP Objects. What is somehow an intermediate solution for the two prior mentioned options.|
|MISP Objects in it’s plain concept is a grouping of indicators within one event. These grouped indicators are somehow linked together. The specific relationship is described by the individual object type. A simple file object links e.g. a filename to its observed hash values (md5, sha1, sha256 and many more).|
A frequent use case for placing additional tages on attribute level would be to lower the confidence in certain attributes. If the event is classified with a high confidence tag, some indicators e.g. legit-but-compromised domains or popular filenames should be labeled with a lowered confidence class. There are several real world examples where this or similar attribute specific tagging has proven to be worthwhile.
Most of the tags are organised in a dedicated MISP Taxonomy. This schema dictates how tags should look like and how they are behaving in certain occasions. There are a lot of details in this topic in general which can be read up in the main MISP Taxonomy Gitlab repository. Currently, there are more than 60 different taxonomies available, each of them containing a number of different tags, which is steadily increasing. There are a lot of advantages in having this variety of tags, e.g. there is one tag for each known associated malware type.
However, this sheer huge amount of tags leads to two main concerns, over-tagging and miss-tagging. Beginners can be overwhelmed with the large number of available tags, to miss exactly this taxonomy to properly label the to be shared data. Over-tagging in most cases only leads to an overwhelming visual appearance. Miss-tagging, however, is a critical step into misusage of shared data. The best and most devastating example would be the miss classification of an event. In dedicated and private sharing groups it is quite usual to share intelligence labeled as „for your company only“. This data must not leave the boundaries of this virtual border of the recipient’s firm. To prevent this kind of mistake, the traffic light protocol (aka TLP) and its respective taxonomy can be used.
There are multiple solutions and proposals to solve the issue of missing additional information about the shared content in form of tags. One of these is the following list of tags which should be at least present to either the event itself or the individual attributes (in this order of importance):
Confidence-Tags/Vetting State: There are huge differences in the quality of data, whether it was vetted upon sharing. As this means that the author was confident that the shared data is or at least was a good indicator of compromise.
Origin-Tags: Describes where the information came from, whether it was in an automated fashion or in a manual investigation. This should give an impression how value this intelligence is, as manual investigation should supersede any automatic generation of data.
PAP-Tags: An even more advanced approach of data classification is using the Permissible Actions Protocol. It indicates how the received data can be used to search for compromises within the individual company or constituency.
|The full list of available taxonomies can be found here.|
Authors and Contributors
Information Sharing and Analysis Center
MISP - Open Source Threat Intelligence Platform & Open Standards For Threat Information Sharing