AN AUTOMATED SYSTEM

The Dark Crawler is a web‐based application that, given a set of user‐specified starting websites, will recursively follow the links out of that site, both on the Clear-Web Internet and the hidden Dark-Web.

FOR RESEARCHERS AND INVESTIGATORS

Scalability, versatility and a breadth of available analytical tools make the Dark Crawler a valuable solution for collecting and analyzing digital evidence of online criminal activity.

A DATA COLLECTION TOOL

Initiated by Mercur IT Solutions, Inc., the development of the Dark Crawler started in 2010 as a course project in the School of Criminology at Simon Fraser University.

The Crawler was designed to assist in data collection for a course project centered around child mapping child exploitation material (CEM), so the researcher would not have to manually visit and map each potential website.

WIDER APPLICATIONS

The tool has evolved through various research projects in close collaboration between many other participants, including support from the Government of Canada and law enforcement agencies.

Based on the initial successes of the research projects Mercur IT Solutions has continued to develop this tool into a practical, user-friendly product, that can produce tangible results in a variety of settings.

STUDYING CYBERCRIME

Given its roots in academia, the system is still heavily used for academic research. So far, various applications of the Dark Crawler resulted in 15 academic publications and 21 presentations, including at the United Nations Office on Drugs and Crime in Doha, Qatar (2015).

POLICY AND PRACTICE

The Dark Crawler allows law enforcement to prioritize various online criminal activity within their jurisdiction based on which websites are either hosted or owned within their own jurisdictions.