warc repositories

Trending repositories tagged warc, ranked by trending score. Updated daily.

  1. 1
    ArchiveBox/ArchiveBox

    🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

    GitHub · Python · 29,072 stars · 1,634 forks · +13 stars in 24 hours · Trending score 2.06

  2. 2
    webrecorder/replayweb.page

    Serverless replay of web archives directly in the browser

    GitHub · TypeScript · 1,016 stars · 96 forks · 0 stars in 24 hours · Trending score 0.91

  3. 3
    internetarchive/heritrix3

    Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

    GitHub · Java · 3,354 stars · 798 forks · 0 stars in 24 hours · Trending score 0.53

  4. 4
    webrecorder/browsertrix

    Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

    GitHub · TypeScript · 501 stars · 83 forks · −1 stars in 24 hours · Trending score 0.31

  5. 5
    commoncrawl/whirlwind-python

    A whirlwind tour of Common Crawl's data using Python

    GitHub · Python · 47 stars · 10 forks · 0 stars in 24 hours · Trending score 0.24

  6. 6
    webrecorder/browsertrix-crawler

    Run a high-fidelity browser-based web archiving crawler in a single Docker container

    GitHub · TypeScript · 1,190 stars · 155 forks · 0 stars in 24 hours · Trending score 0.03

  7. 7
    commoncrawl/news-crawl

    News crawling with StormCrawler - stores content as WARC

    GitHub · Java · 383 stars · 44 forks · 0 stars in 24 hours · Trending score 0.01

  8. 8
    internetarchive/gowarc

    Read and write WARC files in Go

    GitHub · Go · 56 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    centic9/CommonCrawlDocumentDownload

    A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

    GitHub · Java · 74 stars · 16 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    ruarxive/awesome-digital-preservation

    Awesome list dedicated to digital and data preservation tools, sources, services and so on.

    GitHub · 39 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00

  11. 11
    ArchiveTeam/wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

    GitHub · C · 139 stars · 19 forks · 0 stars in 24 hours · Trending score 0.00

  12. 12
    webrecorder/warcio

    Streaming WARC/ARC library for fast web archive IO

    GitHub · Python · 482 stars · 76 forks · 0 stars in 24 hours · Trending score 0.00