View on GitHub
crawld: a data crawler and repositories fetcher
crawld is a metadata crawler and source code repository fetcher.
crawld comprises two different parts: the crawlers and the fetcher.
crawld focuses on crawling repositories metadata and those of the users
that contributed, or are directly related, to the repositories from code
sharing platforms such as GitHub.
Only a GitHub crawler is currently implemented.
However, the architecture of
crawld has been designed in a way such that new
crawlers (for instance a BitBucket crawler) can be
added without hassle.
All of the collected metadata is stored into a
PostgreSQL database. As
crawld is designed to
be able to crawl several code sharing platforms, common information is stored
in two tables:
repositories. For the rest of the information,
specific tables are created (
gh_organizations for now) and relations are established with the
The table below gives information about what is collected. Bear in mind that
some information might be incomplete (for instance, if a user does not provide
any company information).
||Open issues count
||Last push date
Aside from crawling metadata,
crawld is able to clone and update
repositories, using their clone URL stored into the database.
Cloning and updating can be done regardless of the source code management
system in use (git,
svn, …), however only a
git fetcher is
As source code repositories usually contain a lot of files,
crawld has an
option that allows storing source code repositories as tar archives which makes
things easier for the file system shall you clone a huge number of
ligit2 Go binding for its
git operations. Hence,
libgit2 needs to be installed on your system unless you statically compile it
crawld, run this command in a terminal, assuming
Go is installed:
go get github.com/DevMine/crawld
Or you can download a binary for your platform from the DevMine project’s
You also need to setup a PostgreSQL database.
Look at the
db sub-folder for details.
Usage and configuration
crawld.conf and edit it according to your
needs. The configuration file has several sections:
- database: allows you to configure access to your PostgreSQL
- hostname: hostname of the machine.
- port: PostgreSQL port.
- username: PostgreSQL user that has access to the database.
- password: password of the database user.
- dbname: database name.
- ssl_mode: takes any of these 4 values: “disable”,
“require”, “verify-ca”, “verify-null”. Refer to PostgreSQL
- clone_dir: specify where you would like to clone the
- crawling_time_interval: specify the waiting time between 2
full crawling periods. This is irrelevant for the crawlers where no
limit is specified.
- fetch_time_interval: specify the waiting time between 2 full
repositories cloning/updating periods. You shall preferably choose a
small time period here since the repositories fetcher cannot usually
keep up with the crawlers and you likely want it to update/clone the
- fetch_languages: specify the list of languages the fetcher shall
restrict to. If left empty, all languages are considered.
- tar_repositories: a boolean value indicating whether the repositories
shall be stored as tar archives or not.
- tmp_dir: specify a temporary working directory. If left empty, the
default temporary directory will be used. This directory is used on clone and
update operations when the tar_repositories option is activated. It is
recommended to use a ramdisk for increased performance.
- tmp_dir_file_size_limit: specify the maximum size in GB of an object
to be temporarily placed in tmp_dir for processing. Files of size larger
than this value will not be processed in tmp_dir.
- max_fetcher_workers: specify the maximum number of workers to use for
the fetching task. It defaults to 1 but if your machine has good I/O
throughput and a good CPU, you probably want to increase this conservative
value for performance reasons. Note that fetching is I/O and networked bound
more than CPU bound and hence you probably do not want to increase this
value too much.
- throttler_wait_time: indicates how much time to wait, in seconds,
before resuming normal operation after throttling.
- throttler_sliding_window_size: represents the size of the sliding
window used by the error rate throttler. If you have no idea about what that
means, it is safe to omit it since default value shall be sane.
- throttler_leak_interval: specify the time to wait, in milliseconds,
before taking off a unit from the sliding window. Again, if you have no idea
about what that means, it is safe to omit it since default value shall be
- crawlers: allows you to configure options for the crawlers.
- type: specify crawler type. Currently, only “github” is
- languages: list of programming languages of the repositories
you are interested into. All languages used in a repository are
considered and not only the primary language.
- limit: set this value to 0 to not use a limit. Otherwise,
crawling will stop when “limit” repositories have been fetched.
Note that the behavior is slightly different whether you use the
search API or not. When you use the search API, this limit
correspond to the number of repositories to crawl per language
listed in “languages” . When you do not use the search API, this
is a global limit, regardless of the language.
- since_id: corresponds to the repository ID (eg: GitHub repository ID
in the case of the github crawler) from which to start querying
repositories. Note that this value is ignored when using the search API.
- fork: skip fork repositories if set to false.
- oauth_access_token: your API token. If not provided,
crawld will work but the number of API call is usually limited
to a low number. For instance, in the case of the GitHub
crawler, unauthenticated requests are limited to 60 per hour
where authenticated requests goes up to 5000 per hour.
- use_search_api: specify whether you want to use the search
API or not. Bear in mind that results returned via the search
API are usually limited so you probably not want this option set
to true usually. In the case of the GitHub crawler, when set to
true, the limit is 1000 results per search. This means that you
will get at most the 1000 most popular projects (in terms of
stars count) per language listed in “languages”.
Once the configuration file has been adjusted, you are ready to run
You need to specify the path to the configuration file with the help of the
crawld -c crawld.conf
Some command line options are also available, mainly where to store log files
and whether to disable the data crawlers or repositories fetcher (by default,
the crawlers and the fetcher run in parallel). See
crawld -h for more
crawld consists of two main parts internally: the crawlers (only GitHub for
now) and the repositories fetcher (which only supports git for now).
The metadata crawled are put into the database whereas the repositories are
cloned on physical storage.
Note that the internal architecture of
crawld has been thought to make it easy
to implement new crawlers or VCS backends.