daemon-sec-cheatsheet

archiving-and-evidence.md (40425B)

---

 ---
 title: "Archiving & Evidence Preservation"
 description: "Capture sources so they survive deletion, with the hashes and timestamps that make a capture defensible."
 category: osint
 subcategory: "Archiving & Analysis"
 tags: [osint, archiving, evidence, preservation]
 tools: [wget, browsertrix-crawler, yt-dlp, auto-archiver, wayback, archive-today, opentimestamps, openssl, exiftool, shasum]
 difficulty: intermediate
 updated: 2026-10-04
 references:
   - name: "Bellingcat's Online Investigation Toolkit"
     url: "https://bellingcat.gitbook.io/toolkit"
     author: "Bellingcat"
     license: none
     relation: derived
     note: "Tool catalogue: names, descriptions, cost flags and links for this area."
   - name: "OSINT Newsletter Tools Library"
     url: "https://tools.osintnewsletter.com"
     author: "The OSINT Newsletter"
     license: none
     relation: derived
     note: "Second tool catalogue, cross-checked against the above."
 ---
 
 ## What this covers
 
 Capturing a source so that it still exists after someone deletes it, and so that you can show the
 copy you hold is the copy you took. The capture itself is the easy half; the hash, the third-party
 timestamp and the written provenance are what make it worth anything to an editor, a lawyer or a
 court. Everything below is organised around one question: what does this step actually prove?
 
 ## Method
 
 1. **Check what already exists before you capture.** A Wayback snapshot from before the content
    became interesting is worth more than one you took after. Query the availability and CDX APIs
    first; if there is an older capture, that is your baseline and the diff against today is itself a
    finding.
 2. **Capture to something you do not control, and to something you do.** A third-party archive
    supplies a timestamp nobody can accuse you of forging. A local copy survives the archive being
    rate-limited, robots-excluded or taken down. Neither substitutes for the other.
 3. **Pick the capture tool by what the page is.** Static HTML and media: `wget --warc-file`.
    JavaScript-rendered, infinite-scroll or login-walled: a real browser driving the capture, which
    in practice means `browsertrix-crawler` producing WACZ. Platform video: `yt-dlp`, which gets you
    the metadata sidecar a screen recording never will.
 4. **Hash on acquisition, before you look at the file.** A hash taken after you cropped, converted
    or re-saved proves something about your working copy and nothing about the original.
 5. **Timestamp the manifest, not every file.** One hash over the list of hashes, submitted to
    OpenTimestamps or an RFC 3161 authority, binds the whole collection to a point in time at one
    cost. That is the step that converts "I have a hash" into "I had this hash on that date".
 6. **Write the provenance down in the same commit as the files.** URL, UTC time of capture, the
    tool and version, who handed it to you, and how you reached the page. The route to a finding is
    part of the finding, and it is the part you will forget first.
 7. **Get a second independent capture.** Two archives of the same URL taken by different
    infrastructure, or the same video from a second uploader, is the difference between a claim and
    a corroborated claim. Two copies that both trace to one upload are one copy.
 
 Judgement calls worth naming: submitting a URL to a public archive creates a public record that
 somebody is interested in it, so on a target that watches its referrers and its Wayback entries,
 capture locally first and submit later. And a capture of a page you reached while logged in
 contains your session — strip or redact before you share the WARC.
 
 ## Key tools
 
 ### wget (WARC capture)
 
 The standard way to get a byte-level record of an HTTP exchange rather than a rendered picture of
 it. A WARC stores the request and response headers alongside the body for every resource fetched,
 with a SHA-1 digest per record, so the file is a transcript of the conversation and not just its
 outcome. Use it whenever the page is server-rendered; it is the most portable evidence format in
 this area and every replay tool reads it.
 
 ```bash
 brew install wget                # or: apt install wget
 ```
 
 ```bash
 # the baseline capture: WARC transcript plus a CDX index of what went into it
 wget --warc-file=capture-001 --warc-cdx \
      --page-requisites --adjust-extension --no-parent \
      'https://example.com/page'
 
 # stamp the operator and case into the warcinfo record, so the file self-documents
 wget --warc-file=capture-002 --warc-cdx \
      --warc-header="operator: J. Investigator" \
      --warc-header="description: case-2026-014, post cited in filing" \
      --warc-header="robots: off" \
      --page-requisites --adjust-extension 'https://example.com/page'
 
 # leave it uncompressed while you are inspecting it; grep works on a plain WARC
 wget --warc-file=capture-003 --no-warc-compression 'https://example.com/page'
 
 # pull in the CDN that actually serves the images, or the capture renders blank
 wget --warc-file=capture-004 --warc-cdx \
      --page-requisites --convert-links --adjust-extension \
      --span-hosts --domains example.com,cdn.example.com \
      'https://example.com/page'
 
 # a whole section, politely: the delay is what stops you being blocked mid-capture
 wget --warc-file=capture-005 --warc-cdx --recursive --level=2 --no-parent \
      --wait=2 --random-wait --limit-rate=500k \
      -U 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36' \
      'https://example.com/newsroom/'
 
 # a list of URLs in one WARC, split at 1GB so the files stay movable
 wget --warc-file=capture-006 --warc-max-size=1G --warc-cdx -i urls.txt
 
 # second pass over the same site without re-storing what you already have
 wget --warc-file=capture-007 --warc-dedup=capture-005.cdx --recursive --no-parent \
      'https://example.com/newsroom/'
 
 # read back what the capture contains, without a replay tool
 zcat capture-001.warc.gz | grep -a '^WARC-Target-URI:' | sort -u
 awk 'NR>1 {print $5, $6, $1}' capture-001.cdx   # status, SHA-1 digest, URL
 ```
 
 Read the CDX file: the digest column is a base32 SHA-1 of the response payload, which gives you a
 free per-resource integrity check, and the status column tells you which requisites 404'd. A
 capture whose CDX is full of `302` and `403` rows did not get the page.
 
 What this does not prove: the WARC records what *your* client received from *an* IP at that moment.
 It does not prove the page looked that way to anyone else, does not survive geo-targeted or
 personalised content, and carries no third-party attestation of the time — the `WARC-Date` is your
 own clock. `--convert-links` rewrites the mirrored files on disk, not the WARC records, so the
 transcript stays pristine while the browsable copy is usable; keep both. Above all, `wget` executes
 no JavaScript, so on a modern single-page application it faithfully archives an empty shell. If the
 page needs a browser, use one.
 
 ### browsertrix-crawler
 
 A headless Chrome driven by Webrecorder's crawler, writing WARC and WACZ. This is the live,
 maintained answer for anything JavaScript-rendered, where `wget` returns a loading spinner. It
 scrolls, autoplays, runs site-specific behaviours for the big platforms, and extracts page text for
 search.
 
 The older Python crawlers in this niche have aged out: `wpull` carries a 2013–2016 copyright and no
 release since 2019, and `grab-site` still pins Python 3.7/3.8 and was dropped from nixpkgs after
 23.05. Neither is a reasonable thing to stand behind in 2026. Treat them as read-only history and
 use the crawler below.
 
 ```bash
 docker pull webrecorder/browsertrix-crawler
 ```
 
 ```bash
 # single page, as WACZ, with text extracted for full-text search
 docker run -v "$PWD/crawls:/crawls/" -it webrecorder/browsertrix-crawler crawl \
   --url 'https://example.com/post/123' --scopeType page \
   --generateWACZ --text to-pages --collection case-2026-014
 
 # the page plus whatever it links to on the same host, two levels deep
 docker run -v "$PWD/crawls:/crawls/" -it webrecorder/browsertrix-crawler crawl \
   --url 'https://example.com/newsroom/' --scopeType host --depth 2 \
   --generateWACZ --collection newsroom
 
 # a feed that only loads on scroll: give the behaviours time to finish
 docker run -v "$PWD/crawls:/crawls/" -it webrecorder/browsertrix-crawler crawl \
   --url 'https://example.com/feed' --scopeType page-spa \
   --behaviors autoscroll,autoplay,autofetch,siteSpecific \
   --behaviorTimeout 300 --generateWACZ --collection feed
 
 # several URLs from a file, one collection, so the manifest covers the set
 docker run -v "$PWD/crawls:/crawls/" -it webrecorder/browsertrix-crawler crawl \
   --urlFile /crawls/urls.txt --scopeType page --generateWACZ --collection batch-01
 
 # the WACZ is a zip: look inside before you trust it
 unzip -l crawls/collections/case-2026-014/case-2026-014.wacz
 unzip -p crawls/collections/case-2026-014/case-2026-014.wacz datapackage-digest.json | jq .
 
 # the crawl log is JSON Lines — this is where blocked and failed pages surface
 jq -r 'select(.logLevel=="error") | [.timestamp,.message] | @tsv' \
   crawls/collections/case-2026-014/logs/*.log
 ```
 
 A WACZ is a WARC plus an index, a page list and a signed digest manifest, and it opens in
 [ReplayWeb.page](https://replayweb.page/) with no server. That makes it the format to hand to
 someone who is not going to install anything. Check `datapackage-digest.json` after every crawl:
 if the crawler was blocked, you get a valid WACZ containing a block page, and nothing about the
 file itself says so.
 
 What it does not tell you: a browser capture is a recording of one rendering session, which means
 it includes whatever was personalised, A/B-tested or geo-fenced for that session. Running the same
 crawl from a different exit and getting different content is a finding about the site, not an error.
 Logged-in crawls bake the session into the archive — handle accordingly.
 
 ### yt-dlp
 
 Platform video and audio with its metadata intact. The point is not the video file; a screen
 recording gets you that. The point is the `.info.json` sidecar, which carries uploader ID, upload
 date, duration, view and comment counts, available formats and often the original title and
 description before anybody edited them.
 
 ```bash
 brew install yt-dlp              # or: pipx install yt-dlp
 ```
 
 ```bash
 # the provenance capture: video, metadata, description, thumbnail, subtitles
 yt-dlp --write-info-json --write-description --write-thumbnail \
        --write-subs --sub-langs 'all' --no-mtime \
        -o '%(upload_date)s_%(uploader_id)s_%(id)s.%(ext)s' URL
 
 # what formats exist, before you commit to one
 yt-dlp -F URL
 
 # the best single pre-merged file, so you archive bytes the platform served
 yt-dlp -f 'best[ext=mp4]/best' --no-mtime URL
 
 # metadata only — enough to date and attribute a clip without downloading it
 yt-dlp --skip-download --write-info-json --write-thumbnail URL
 
 # comments too; they routinely date an event more precisely than the upload does
 yt-dlp --write-info-json --write-comments --skip-download URL
 
 # a whole channel, newest first, with an archive file so re-runs are incremental
 yt-dlp --write-info-json --no-mtime --download-archive seen.txt \
        -o '%(upload_date)s_%(id)s.%(ext)s' 'https://example.com/@account/videos'
 
 # the intermediate pages the extractor fetched — useful when an extractor misreads a page
 yt-dlp --write-pages --skip-download URL
 
 # the fields you will actually quote, straight out of the sidecar
 jq -r '[.id,.uploader_id,.upload_date,.timestamp,.duration,.view_count,.title] | @tsv' *.info.json
 ```
 
 `--no-mtime` matters: without it the file's mtime is set from the upload date, which silently
 overwrites your own acquisition timeline with the platform's claim. `timestamp` in the sidecar is
 Unix epoch seconds and is usually more precise than `upload_date`, which is date-only and rendered
 in the platform's timezone of choice.
 
 What the sidecar does not establish: every field in it is the platform's assertion, repeated.
 `upload_date` is when it was uploaded, never when it was filmed, and a re-upload resets it.
 Extractors break when platforms change, so a field that is `null` may mean absent or may mean the
 extractor lost it — check against the page. Rate limits and age or region gates will silently
 truncate a channel pull; compare the count you got against the count the channel claims.
 
 ### Wayback Machine, from the command line
 
 Three separate endpoints, used at three different moments. Availability answers "is there already a
 capture"; CDX answers "what captures exist, and did the content change between them"; Save Page Now
 creates a new one.
 
 ```bash
 # is there a capture at all, and what is the nearest one to a date
 curl -s 'https://archive.org/wayback/available?url=example.com/page&timestamp=20240101' | jq .
 
 # every capture of a URL, with the payload digest that reveals real edits
 curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com/page&output=json&fl=timestamp,original,statuscode,digest,length' | jq -r '.[] | @tsv'
 
 # collapse consecutive identical captures: what is left is the change history
 curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com/page&output=json&fl=timestamp,digest&collapse=digest' | jq -r '.[] | @tsv'
 
 # everything ever captured under a host, which is how you find deleted pages
 curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=timestamp,original,statuscode&limit=500' | jq -r '.[] | @tsv'
 
 # only the captures in a window, for a page that mattered on one specific day
 curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com/page&from=20260301&to=20260331&output=json&fl=timestamp,digest' | jq -r '.[] | @tsv'
 
 # submit a new capture: SPN2, authenticated with Internet Archive S3-style keys
 curl -s -X POST -H 'Accept: application/json' \
      -H "Authorization: LOW $IA_ACCESS_KEY:$IA_SECRET_KEY" \
      -d 'url=https://example.com/page' -d 'capture_all=1' -d 'capture_screenshot=1' \
      'https://web.archive.org/save' | jq .
 
 # poll the job until it reports success, and keep the returned timestamp
 curl -s -H "Authorization: LOW $IA_ACCESS_KEY:$IA_SECRET_KEY" \
      "https://web.archive.org/save/status/$JOB_ID" | jq '{status, timestamp, original_url}'
 
 # fetch the archived copy itself, and hash what you got
 curl -sL 'https://web.archive.org/web/20260301120000id_/https://example.com/page' | shasum -a 256
 ```
 
 Get the key pair from [archive.org/account/s3.php](https://archive.org/account/s3.php) and export
 it as `IA_ACCESS_KEY` / `IA_SECRET_KEY`. **The unauthenticated `GET /save/<url>` form is no longer
 usable for work** — an anonymous request to it returns HTTP 429 rather than a capture. Scripts that
 still rely on it fail silently, because a 429 body looks like a page.
 
 The `id_` infix in that last URL asks for the original bytes without Wayback's navigation banner
 injected, which is what you want if you intend to hash or diff the archived copy. The `digest`
 column in CDX output is the payload hash: two captures sharing a digest are byte-identical, so
 collapsing on it turns a thousand snapshots into the handful of moments the page actually changed.
 That is the single most useful thing the CDX API does.
 
 What Wayback does not give you: permanence. A site can retroactively exclude itself, and captures
 do disappear. It also honours robots directives at crawl time, so absence of a capture is not
 evidence the page did not exist. And a Save Page Now capture is still just a capture — it attests
 that the Internet Archive saw that content at that time, which is a strong third-party claim about
 time and a weak one about authenticity.
 
 ### archive.today
 
 A separate archive with a separate failure mode, which is exactly why you use it alongside Wayback.
 It renders pages that Wayback flattens, ignores robots directives, and has a long record of not
 removing things. There is no API and the submission form sits behind bot protection, so this one is
 a browser job — do not script it.
 
 ```text
 1. Open https://archive.today  (archive.ph and archive.is are the same service;
    if one mirror is blocked where you are, try another)
 2. Paste the URL into the lower box, "My url is alive and I want to archive its content"
 3. Solve the challenge if you get one. Do not automate this step — it is the step
    that gets the service to block your address.
 4. Wait for the capture. The result URL looks like https://archive.ph/AbC12
    and the page header shows the capture time in UTC.
 5. Click "screenshot" in the header to get the full-page render as a separate
    artefact, and save it. The HTML capture and the screenshot fail differently.
 6. Record the short URL AND the capture time. The short URL is your citation.
 7. Before submitting, use the upper box to search for existing captures of the
    same URL — the same reason you query Wayback's CDX API first.
 ```
 
 Two things to know. The service is deliberately opaque about its infrastructure and funding, which
 means you should not treat it as your only copy of anything; keep the local WARC. And it fetches
 the page itself, from its own address, so a submission does not leak your IP to the target — but it
 does create a public, searchable record that someone archived that URL at that minute.
 
 ### Hashing and the manifest
 
 Hashes are the whole evidentiary argument. A hash recorded at acquisition and published or
 timestamped separately lets you show, later, that the file you are producing is the file you took.
 The manifest pattern — one file listing every hash, then one hash of the manifest — scales that to a
 collection without timestamping a thousand files.
 
 ```bash
 # macOS ships shasum; coreutils ships sha256sum. Same output format.
 shasum -a 256 evidence.mp4
 sha256sum evidence.mp4
 ```
 
 ```bash
 # hash the whole capture tree in a stable order, so the manifest is reproducible
 find ./capture -type f -print0 | sort -z | xargs -0 shasum -a 256 > MANIFEST.sha256
 
 # the manifest's own hash: this single line is what you timestamp and publish
 shasum -a 256 MANIFEST.sha256 | tee MANIFEST.sha256.sha256
 
 # verify the collection, any time later
 shasum -a 256 -c MANIFEST.sha256
 
 # show only what failed, which is what you actually want on a large set
 shasum -a 256 -c MANIFEST.sha256 2>/dev/null | grep -v ': OK$'
 
 # a provenance record that travels with the files, in the same directory
 cat > PROVENANCE.txt <<'EOF'
 case:        2026-014
 url:         https://example.com/post/123
 captured_at: 2026-10-04T04:07:56Z   # UTC, from `date -u +%FT%TZ`
 captured_by: J. Investigator
 tooling:     wget 1.25.0 --warc-file; yt-dlp 2026.08.19
 route:       linked from https://example.com/newsroom/ on 2026-10-03
 notes:       page required no login; no personalisation observed
 EOF
 
 # fold the provenance into the manifest so the two cannot drift apart
 shasum -a 256 PROVENANCE.txt >> MANIFEST.sha256
 
 # find duplicates across two collections by hash, not by filename
 sort MANIFEST.sha256 other/MANIFEST.sha256 | awk '{print $1}' | sort | uniq -d
 ```
 
 Use SHA-256. MD5 and SHA-1 still appear in forensic tooling and are fine as identifiers, but a
 collision-prone hash invites an argument you do not need to have. Note that `find | sort` matters:
 an unsorted manifest changes order between runs and filesystems, which makes two manifests of the
 same files look different.
 
 What a hash proves and does not: it proves the bytes have not changed since the hash was taken. It
 says nothing about when they were taken, nothing about where they came from, and nothing about
 whether the content is true. On its own a hash in your own notes is worth very little, because you
 could have written it at any time — which is what the next section fixes.
 
 ### Timestamping: OpenTimestamps and RFC 3161
 
 The step that turns a hash into evidence. A timestamp is a third party's signed assertion that a
 given digest existed before a given moment. Without it, your hash only proves internal
 consistency; with it, you can show you held that exact file on that date and could not have
 produced it later.
 
 ```bash
 pipx install opentimestamps-client     # the `ots` command
 # openssl is already present for the RFC 3161 route
 ```
 
 ```bash
 # OpenTimestamps: anchors the digest in the Bitcoin blockchain, free, no account
 ots stamp MANIFEST.sha256
 
 # what the proof currently contains: pending calendar attestations, then a block
 ots info MANIFEST.sha256.ots
 
 # calendars need an hour or so to get into a block; upgrade the proof afterwards
 ots upgrade MANIFEST.sha256.ots
 
 # verify. Full verification wants a local Bitcoin node; without one you are
 # trusting the calendar, which is weaker but still a third party
 ots verify MANIFEST.sha256.ots
 
 # verify a digest you were given, without holding the file
 ots verify -d 355c357708e4840d12b0d4284d9fe1a911c79c013953874f03068b81127f3363 MANIFEST.sha256.ots
 
 # RFC 3161 route: build a timestamp query over the file's SHA-256
 openssl ts -query -data MANIFEST.sha256 -sha256 -cert -out manifest.tsq
 
 # send it to a Time Stamping Authority and keep the signed reply
 curl -s -H 'Content-Type: application/timestamp-query' \
      --data-binary @manifest.tsq https://freetsa.org/tsr -o manifest.tsr
 
 # read the token: the TSA's time, its identity and the digest it signed over
 openssl ts -reply -in manifest.tsr -text
 
 # verify the token against the TSA's chain — this is the check a reviewer repeats
 curl -sO https://freetsa.org/files/cacert.pem
 curl -sO https://freetsa.org/files/tsa.crt
 openssl ts -verify -data MANIFEST.sha256 -in manifest.tsr \
            -CAfile cacert.pem -untrusted tsa.crt
 ```
 
 A successful verification prints `Verification: OK`, and that line is the thing you cite. Keep the
 `.tsr`, the CA chain you verified against, and the exact file — the token is signed over the digest,
 so a reviewer needs the file to recompute it.
 
 Which route to use: OpenTimestamps costs nothing, needs no account, and anchors to a public
 blockchain, so the proof outlives any single company — but it is coarse (block granularity, roughly
 an hour) and full verification wants a Bitcoin node. An RFC 3161 TSA gives you a precise, signed,
 PKI-rooted token that institutional reviewers already recognise, at the price of trusting that TSA
 and its certificate remaining validatable. Do both; they are cheap and they fail differently.
 
 What a timestamp does not prove: that the content is authentic, that the capture was complete, or
 that you obtained it lawfully. It proves the file existed no later than that moment. That is a
 narrow claim, and stating it narrowly is what makes it credible.
 
 ### Bellingcat Auto Archiver
 
 The pipeline to set up when you are collecting continuously rather than case by case. It reads URLs
 from a feeder, runs a platform-specific extractor, then runs enrichers that hash, thumbnail,
 timestamp and push to Wayback, and writes the results to a database you can read as a case index.
 It is the only tool here that performs the hash-and-timestamp discipline on every item without you
 remembering to.
 
 ```bash
 pipx install auto-archiver
 auto-archiver --help           # the flag list is generated from the installed modules
 ```
 
 Configuration is a single YAML file; the default path is `secrets/orchestration.yaml`. Only modules
 named under `steps` are loaded, so the file is both configuration and a declaration of the pipeline.
 
 ```yaml
 # orchestration.yaml
 steps:
   feeders: [cli_feeder]
   extractors: [generic_extractor]
   enrichers: [hash_enricher, meta_enricher, thumbnail_enricher,
               timestamping_enricher, opentimestamps_enricher]
   databases: [csv_db, console_db]
   storages: [local_storage]
   formatters: [html_formatter]
 
 hash_enricher:
   algorithm: SHA-256          # the only alternative is SHA3-512
 
 timestamping_enricher:
   tsa_urls:
     - http://timestamp.identrust.com
     - http://zeitstempel.dfn.de
   allow_selfsigned: false     # leaving this false is the whole point of the module
 
 local_storage:
   save_to: ./archived
   path_generator: url         # flat | url | random
   filename_generator: static  # names the file after its hash
 
 csv_db:
   csv_file: ./archived/results.csv
 
 logging:
   level: INFO
   file: ./archived/archiver.log
 ```
 
 ```bash
 # one URL through the whole pipeline
 auto-archiver --config orchestration.yaml 'https://example.com/post/123'
 
 # several at once; the CSV gets one row per URL
 auto-archiver --config orchestration.yaml 'https://example.com/a' 'https://example.com/b'
 
 # no config file at all, for a one-off capture with sane defaults
 auto-archiver --mode simple 'https://example.com/post/123'
 
 # override the enricher list without editing the file
 auto-archiver --config orchestration.yaml \
   --enrichers hash_enricher timestamping_enricher opentimestamps_enricher \
   'https://example.com/post/123'
 
 # push to Wayback as part of the run, with the same Internet Archive key pair
 auto-archiver --config orchestration.yaml \
   --enrichers hash_enricher wayback_extractor_enricher \
   --wayback_extractor_enricher.key "$IA_ACCESS_KEY" \
   --wayback_extractor_enricher.secret "$IA_SECRET_KEY" \
   'https://example.com/post/123'
 
 # feed a column of URLs from a spreadsheet export
 auto-archiver --config orchestration.yaml \
   --feeders csv_feeder --csv_feeder.files urls.csv --csv_feeder.column link
 
 # turn up logging when an extractor silently returns nothing
 auto-archiver --config orchestration.yaml --logging.level DEBUG 'https://example.com/post/123'
 
 # the results CSV is the case index: one row per URL, with hashes and archive links
 column -s, -t < ./archived/results.csv | less -S
 ```
 
 Two enrichers do the evidentiary work and they are not interchangeable. `timestamping_enricher`
 aggregates the item's file hashes into one text file and gets an RFC 3161 token over it from each
 TSA in the list; `opentimestamps_enricher` anchors the same digest via the OpenTimestamps
 calendars. Note that `freetsa.org` is deliberately absent from the shipped TSA defaults because its
 certificate does not validate against system authorities — you can add it, but only together with
 `allow_selfsigned: true`, which weakens exactly the property you wanted.
 
 The `gsheet_feeder_db` module is what makes this a team tool: analysts paste URLs into a Google
 Sheet, the archiver writes the hash, the stored path and the archive links back into adjacent
 columns. The `wacz_extractor_enricher` shells out to Docker for browser-based capture, and
 `wayback_extractor_enricher` needs the same key pair as Save Page Now.
 
 Watch the version. The client checks on startup and says so when it is behind, and per-platform
 extractors break often enough that an old release produces empty captures that look like
 successes. After every run, scan the results CSV for rows that carry a hash but no media.
 
 ### ExifTool, for capture-time metadata
 
 Covered in depth on [Image & Video Forensics](/sheets/osint/image-video-forensics); the archiving
 use is narrower. You are not asking whether the file is manipulated, you are recording what the
 file asserted about itself at the moment it entered your custody, so that a later disagreement is
 about the metadata rather than about your handling.
 
 ```bash
 # the full tag dump, archived next to the hash and never edited again
 exiftool -j -g1 -a -u capture/evidence.mp4 > capture/evidence.mp4.exif.json
 
 # the fields that date and attribute a capture, in one line
 exiftool -CreateDate -ModifyDate -FileModifyDate -GPSPosition -Make -Model -Software \
          capture/evidence.mp4
 
 # every timestamp in the file, grouped by where it came from
 exiftool -time:all -a -G1 -s capture/evidence.mp4
 
 # the whole capture tree as one CSV, which doubles as a collection inventory
 exiftool -r -csv -FileName -FileSize -MIMEType -CreateDate -GPSPosition ./capture/ > inventory.csv
 
 # flag files whose structure does not match their declared format
 exiftool -r -validate -warning -a ./capture/
 
 # strip metadata from a working copy before you publish, to protect a source
 cp capture/evidence.jpg publish/evidence.jpg
 exiftool -all= -overwrite_original publish/evidence.jpg
 shasum -a 256 publish/evidence.jpg    # a different file, so a different hash: say so
 ```
 
 Run this on the copy, after hashing the original, and never with `-overwrite_original` on anything
 in the capture tree. Note the last block: a published, stripped file is a *different artefact* with a
 different hash, and conflating the two is a straightforward way to be accused of altering evidence.
 Record both hashes and the relationship between them.
 
 What it does not tell you: metadata is written as easily as it is read, so a helpful
 `CreateDate` is consistent-with and never proof. Most files pulled from platforms have had their
 metadata stripped on upload, and that absence is normal rather than suspicious.
 
 ## Case-management and pipeline tools
 
 | Tool | What it does |
 | --- | --- |
 | [Auto Archiver](https://github.com/bellingcat/auto-archiver) | Bellingcat's pipeline: reads URLs from a spreadsheet, archives pages and media, hashes everything, writes results back. The right tool for continuous collection. |
 | [Hunchly](https://www.hunch.ly/) | Captures every page you visit during a case automatically, with hashes and full-text search. Removes the discipline problem. |
 | [Atlos](https://www.atlos.org/) | Collaborative platform for visual investigations with source tracking and review workflow. |
 | [Lumen](https://lumendatabase.org/) | Archive of takedown notices — sometimes the only record that something existed. |
 | [Web Archives](https://github.com/dessant/web-archives) | Browser extension that queries many archive services at once. |
 
 ## Tool reference
 
 | Tool | What it does | Cost |
 | --- | --- | --- |
 | [Archive.today](https://archive.today) | Archive any webpage and search for archived pages. | free |
 | [Bellingcat TikTok Hashtag Analysis](https://github.com/bellingcat/tiktok-hashtag-analysis) | Archive content and metadata from TikTok posts that contain one or more specified hashtags | free |
 | [Distill](https://distill.io/) | Distill is a website change monitoring tool that allows users to track changes on web pages. | partly free |
 | [Wayback Machine](https://web.archive.org/) | The Wayback Machine is the Internet Archive's free tool for viewing and saving archived web pages, with over a trillion pages captured, widely used for… | free |
 
 ## Pitfalls
 
 - **Screenshots alone are weak.** No metadata, no hash, trivially edited. Capture the file.
 - **Archives honour robots.txt and takedowns.** A site can retroactively remove itself from
   Wayback. Your local copy is what survives.
 - **Dynamic content does not archive well.** Infinite scroll, lazy loading and interactive maps
   frequently fail; verify the capture actually shows what you saw.
 - **Hash after acquisition, not after editing.** A hash of a file you already cropped proves
   nothing about the original.
 - **Archiving can notify.** Submitting a URL to a public archive creates a public record that
   someone is interested in it.
 - **A capture that succeeded is not a capture that worked.** A WARC full of 403s, a WACZ containing
   a block page, and a `yt-dlp` channel pull truncated by rate limiting all exit zero. Read the CDX
   status column, the crawl log and the row count before you file anything.
 - **Old scripts hitting `GET /save/<url>` now get HTTP 429.** Anonymous Save Page Now is no longer
   usable; a 429 body is still a body, so a script that does not check the status silently records a
   failure as a success. Use the authenticated SPN2 POST.
 - **A hash in your own notes is nearly worthless.** You could have written it at any time. The
   third-party timestamp is what makes it an assertion about the past rather than about your memory.
 
 ## Worked example
 
 One datum: a URL to a post carrying a video clip, `https://example.com/@account/post/123`, cited in
 a filing you have been asked to check. The URL, the hashes and the timestamps below are invented;
 the order of the steps and what each one proves are not.
 
 ```bash
 # 1. what already exists, before you touch the page
 curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com/@account/post/123&output=json&fl=timestamp,digest,statuscode&collapse=digest' | jq -r '.[] | @tsv'
 # timestamp        digest                            statuscode
 # 20260228183044   H4XQ...                           200
 # 20260302094112   ZK7B...                           200
 ```
 
 Two distinct payload digests, two days apart. Collapsing on digest is what revealed that: the page
 was edited between 28 February and 2 March, which is a finding in itself and the reason to pull
 both old captures before capturing the live page.
 
 ```bash
 # 2. local capture of the live page, byte-level, with the case in the warcinfo record
 mkdir -p case-2026-014 && cd case-2026-014
 wget --warc-file=live --warc-cdx --no-warc-compression \
      --warc-header="operator: J. Investigator" \
      --warc-header="description: case-2026-014, post cited at para 17" \
      --page-requisites --adjust-extension --no-parent \
      'https://example.com/@account/post/123'
 awk 'NR>1 {print $5, $1}' live.cdx | sort | uniq -c
 #  14 200 https://example.com/...
 #   3 403 https://cdn.example.com/media/...
 ```
 
 Three requisites returned 403, and they are the media files. The `wget` capture has the page text
 and not the clip, which is the common outcome and the reason the next two steps exist rather than
 being optional extras.
 
 ```bash
 # 3. the rendered page, in a browser, because the post body loads via JavaScript
 docker run -v "$PWD/crawls:/crawls/" -it webrecorder/browsertrix-crawler crawl \
   --url 'https://example.com/@account/post/123' --scopeType page \
   --generateWACZ --text to-pages --collection case-2026-014
 unzip -p crawls/collections/case-2026-014/case-2026-014.wacz datapackage-digest.json | jq .
 # { "path": "datapackage.json", "hash": "sha256:9f2c...", "signedData": null }
 ```
 
 `signedData: null` means the WACZ is unsigned — fine, because you are about to timestamp it
 yourself. The digest over `datapackage.json` is the crawler's own integrity check on the archive;
 record it, then stop relying on it and use your own hash.
 
 ```bash
 # 4. the media, with its provenance sidecar, which the WARC never got
 yt-dlp --write-info-json --write-description --write-thumbnail --no-mtime \
        -o '%(upload_date)s_%(uploader_id)s_%(id)s.%(ext)s' \
        'https://example.com/@account/post/123'
 jq -r '[.id,.uploader_id,.upload_date,.timestamp,.duration,.view_count] | @tsv' *.info.json
 # 123   account   20260226   1772120000   47   18422
 ```
 
 The sidecar says the clip was uploaded on 26 February — two days *before* the earliest Wayback
 capture of the post, and before the edit the CDX diff exposed. That is the pivot: the video predates
 the post text it now sits under, so the claim to check is whether the caption was changed, not
 whether the footage is real.
 
 ```bash
 # 5. hash everything, in a stable order, before any further handling
 cd .. && find ./case-2026-014 -type f -print0 | sort -z | xargs -0 shasum -a 256 > MANIFEST.sha256
 wc -l MANIFEST.sha256 && shasum -a 256 MANIFEST.sha256
 # 31 MANIFEST.sha256
 # 4b1c8e... MANIFEST.sha256
 ```
 
 ```bash
 # 6. the step that makes the hash mean something: two independent timestamps
 ots stamp MANIFEST.sha256
 openssl ts -query -data MANIFEST.sha256 -sha256 -cert -out manifest.tsq
 curl -s -H 'Content-Type: application/timestamp-query' \
      --data-binary @manifest.tsq https://freetsa.org/tsr -o manifest.tsr
 openssl ts -reply -in manifest.tsr -text | grep -E 'Status:|Time stamp:'
 # Status: Granted.
 # Time stamp: Oct  4 04:06:07 2026 GMT
 ```
 
 Now the claim is narrow and defensible: this set of 31 files, with these hashes, existed no later
 than 04:06 UTC on 4 October 2026, attested by a TSA and by the Bitcoin blockchain, neither of which
 you control.
 
 ```bash
 # 7. a third-party capture, submitted last so the local copy exists first
 curl -s -X POST -H 'Accept: application/json' \
      -H "Authorization: LOW $IA_ACCESS_KEY:$IA_SECRET_KEY" \
      -d 'url=https://example.com/@account/post/123' -d 'capture_all=1' \
      'https://web.archive.org/save' | jq -r '.job_id'
 # spn2-9f3c...
 ```
 
 Then the same URL by hand into [archive.today](https://archive.today), and its short URL and UTC
 capture time into `PROVENANCE.txt`. Submitting last is deliberate: both submissions are public, so
 if the account notices and deletes the post, you already hold the capture.
 
 What you can assert: the page as it stood at a timestamped moment, the clip with the
 platform's own metadata, and a change history from Wayback showing the post was edited after the
 video was uploaded. What it does not establish: that the footage shows what the caption claims, or
 where or when it was filmed. Those are
 [Image & Video Forensics](/sheets/osint/image-video-forensics) and
 [Geolocation](/sheets/osint/geolocation) questions, and no amount of archiving substitutes for them.
 
 What would falsify it: any manifest entry that fails verification, a timestamp token whose digest
 does not match the manifest, or a WARC/WACZ log showing that the claimed resource failed and only
 an error page was captured. The public archive copy is corroboration; the local capture, hashes
 and independently verifiable timestamps carry the claim.
 
 ## Broader catalogues
 
 - [Archiving OSINT](https://tools.osintnewsletter.com/tool-categories/archiving-osint)
 - [Data Extraction OSINT](https://tools.osintnewsletter.com/tool-categories/data-extraction-osint)
 
 
 ## More tools
 
 Further tools for this area from the OSINT Newsletter Tools Library ([Archiving OSINT](https://tools.osintnewsletter.com/tool-categories/archiving-osint)), excluding those already listed above.
 
 | Tool | What it does |
 | --- | --- |
 | [4plebs](https://4plebs.org/) | A third-party archive and search service for historical 4chan content. |
 | [4shared](https://www.4shared.com/) | File-sharing and cloud-storage platform with a public search function. |
 | [Anna's Archive](https://annas-archive.gl/) | Shadow library search engine aggregating books, papers, and digital texts. |
 | [Archivarix Tube Search](https://tube.archivarix.net/) | A free OSINT tool for discovering historical versions of YouTube videos using archived snapshots from archive data. |
 | [Arctic Shift](https://arctic-shift.photon-reddit.com/) | A Reddit search and archival platform that enables investigators to search historical Reddit posts, comments, deleted content… |
 | [deaditArchive](https://deaditarchive.netlify.app/) | A searchable archive of selected deleted or purged Reddit communities, preserving posts from communities that are no longer… |
 | [Follow That Page](https://www.followthatpage.com/) | A web-monitoring service that tracks specified webpages and sends notifications when changes are detected. |
 | [Free Full PDF](https://www.freefullpdf.com/) | Academic search engine focused on locating freely available full-text scientific PDFs, including journal articles, theses… |
 | [GetProofAnchor](https://getproofanchor.com/) | A web-based tool that captures and preserves online content as verifiable digital evidence. |
 | [Lookyloo](https://lookyloo.circl.lu/capture) | A web forensics tool that captures a webpage and maps every domain, resource, redirect, and third-party connection involved in… |
 | [ProofSnap](https://getproofsnap.com/) | A browser extension that captures a live web page from the rendering browser and seals it as a verifiable archive. |
 | [SearchShared](https://www.searchshared.info/) | File-sharing search engine for locating publicly indexed/shared files across multiple hosting services. |
 
 ## Sources
 
 Both catalogues below are maintained by other people and are considerably larger than
 this page. Use them as the canonical index; this sheet is a working route through them.
 
 - [Bellingcat's Online Investigation Toolkit](https://bellingcat.gitbook.io/toolkit) — ~340 tools, each with its own
   review page covering cost, difficulty, requirements and limitations.
 - [OSINT Newsletter Tools Library](https://tools.osintnewsletter.com) — ~280 tools, organised by investigative goal.
 
 Neither publishes a licence, so nothing here is copied from them: tool names, one-line
 descriptions, cost flags and links are catalogue facts, and the method and commentary are
 this site's own. See [credits](/credits).