The Dark Crawler is a customizable Web intelligence platform for cybercrime researchers, investigators and concerned organizations.
HOW DOES IT WORK?
AN AUTOMATED SYSTEM
The Dark Crawler is a web‐based application that, given a set of user‐specified starting websites, will recursively follow the links out of that site, both on the Clear-Web Internet and the hidden Dark-Web.
FOR RESEARCHERS
AND INVESTIGATORS
Scalability, versatility and a breadth of available analytical tools make the Dark Crawler a valuable solution for collecting and analyzing digital evidence of online criminal activity.
How DID IT START OUT?
A DATA COLLECTION TOOL
Initiated by Mercur IT Solutions, Inc., the development of the Dark Crawler started in 2010 as a course project in the School of Criminology at Simon Fraser University.
The Crawler was designed to assist in data collection for a course project centered around child mapping child exploitation material (CEM), so the researcher would not have to manually visit and map each potential website.
WIDER APPLICATIONS
The tool has evolved through various research projects in close collaboration between many other participants, including support from the Government of Canada and law enforcement agencies.
Based on the initial successes of the research projects Mercur IT Solutions has continued to develop this tool into a practical, user-friendly product, that can produce tangible results in a variety of settings.
What is it used for?
STUDYING CYBERCRIME
Given its roots in academia, the system is still heavily used for academic research. So far, various applications of the Dark Crawler resulted in 15 academic publications and 21 presentations, including at the United Nations Office on Drugs and Crime in Doha, Qatar (2015).
POLICY AND PRACTICE
The Dark Crawler allows law enforcement to prioritize various online criminal activity within their jurisdiction based on which websites are either hosted or owned within their own jurisdictions.
Why Choose the Dark Crawler?
Scalable architecture allows to connect more computers, increasing performance linearly up to an order of magnitude.
Each client can select the licensing plan most suitable to their goals. The Dark Crawler can be hosted in the “cloud”, or on local hardware and networks, or accessed on a “data as a service” basis.
Stability updates, bug fixes and content patches are released on schedule. Systems running in a Mission Critical environment are given top priority.
Reports are generated to present the data as social network graphs, geolocation maps, sentiment score matrices and many more.