Skip to content

The Dark Crawler

Exploring the dark corners of the Internet

Menu
  • Home
  • Use Cases
    • Website Analysis
    • Network Exploration
    • Coordinating Reporting
    • Usage Scenarios
  • Features
    • Scalability
    • Maintenance
    • Licensing
    • Reporting Tools
  • Resources
  • Projects
  • Blog
  • Login

Home

The Dark Crawler is a customizable Web intelligence platform for cybercrime researchers, investigators and concerned organizations.

8263362af5eb3c8ae2ca88364a51bfbc_1

HOW DOES IT WORK?

Social network front page
AN AUTOMATED SYSTEM

The Dark Crawler is a web‐based application that, given a set of user‐specified starting websites, will recursively follow the links out of that site, both on the Clear-Web Internet and the hidden Dark-Web. 

FOR RESEARCHERS
AND INVESTIGATORS

Scalability, versatility and a breadth of available analytical tools make the Dark Crawler a valuable solution for collecting and analyzing digital evidence of online criminal activity.

How DID IT START OUT?

DSC_1000_1
A DATA COLLECTION TOOL

Initiated by Mercur IT Solutions, Inc., the development of the Dark Crawler started in 2010 as a course project in the School of Criminology at Simon Fraser University.

The Crawler was designed to assist in data collection for a course project centered around child mapping child exploitation material (CEM), so the researcher would not have to manually visit and map each potential website.

WIDER APPLICATIONS

The tool has evolved through various research projects in close collaboration between many other participants, including support from the Government of Canada and law enforcement agencies.

Based on the initial successes of the research projects Mercur IT Solutions has continued to develop this tool into a practical, user-friendly product, that can produce tangible results in a variety of settings.

What is it used for?

microphone-704255_1920
STUDYING CYBERCRIME

Given its roots in academia, the system is still heavily used for academic research. So far, various applications of the Dark Crawler resulted in 15 academic publications and 21 presentations, including at the United Nations Office on Drugs and Crime in Doha, Qatar (2015).

POLICY AND PRACTICE

The Dark Crawler allows law enforcement to prioritize various online criminal activity within their jurisdiction based on which websites are either hosted or owned within their own jurisdictions.

Why Choose the Dark Crawler?

SCALABILITY

Scalability

Scalable architecture allows to connect more computers, increasing performance linearly up to an order of magnitude.

LICENSING

Licensing

Each client can select the licensing plan most suitable to their goals. The Dark Crawler can be hosted in the “cloud”, or on local hardware and networks, or accessed on a “data as a service” basis. 

MAINTENANCE

Maintenance

Stability updates, bug fixes and content patches are released on schedule. Systems running in a Mission Critical environment are given top priority. 

REPORTING TOOLS

Reporting tools

Reports are generated to present the data as social network graphs, geolocation maps, sentiment score matrices and many more. 

Contact US

The Dark Crawler 2021 . Powered by WordPress

Close