warc repositories
Trending repositories tagged warc, ranked by trending score. Updated daily.
-
1
ArchiveBox/ArchiveBox
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
GitHub · Python · 29,072 stars · 1,634 forks · +13 stars in 24 hours · Trending score 2.06
-
2
webrecorder/replayweb.page
Serverless replay of web archives directly in the browser
GitHub · TypeScript · 1,016 stars · 96 forks · 0 stars in 24 hours · Trending score 0.91
-
3
internetarchive/heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
GitHub · Java · 3,354 stars · 798 forks · 0 stars in 24 hours · Trending score 0.53
-
4
webrecorder/browsertrix
Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
GitHub · TypeScript · 501 stars · 83 forks · −1 stars in 24 hours · Trending score 0.31
-
5
commoncrawl/whirlwind-python
A whirlwind tour of Common Crawl's data using Python
GitHub · Python · 47 stars · 10 forks · 0 stars in 24 hours · Trending score 0.24
-
6
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
GitHub · TypeScript · 1,190 stars · 155 forks · 0 stars in 24 hours · Trending score 0.03
-
7
commoncrawl/news-crawl
News crawling with StormCrawler - stores content as WARC
GitHub · Java · 383 stars · 44 forks · 0 stars in 24 hours · Trending score 0.01
-
8
internetarchive/gowarc
Read and write WARC files in Go
GitHub · Go · 56 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00
-
9
centic9/CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
GitHub · Java · 74 stars · 16 forks · 0 stars in 24 hours · Trending score 0.00
-
10
ruarxive/awesome-digital-preservation
Awesome list dedicated to digital and data preservation tools, sources, services and so on.
GitHub · 39 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00
-
11
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
GitHub · C · 139 stars · 19 forks · 0 stars in 24 hours · Trending score 0.00
-
12
webrecorder/warcio
Streaming WARC/ARC library for fast web archive IO
GitHub · Python · 482 stars · 76 forks · 0 stars in 24 hours · Trending score 0.00