daemon-sec-cheatsheet

social-media-monitoring.md (31354B)

---

 ---
 title: "Cross-Platform Monitoring & Collection"
 description: "Track accounts, hashtags and narratives across several platforms at once, and watch pages for changes."
 category: osint
 subcategory: "Social Media"
 tags: [osint, monitoring, collection, social-media]
 tools: [4cat, zeeschuimer, changedetection.io, distill, yt-dlp, rsshub, rss-bridge, arctic-shift]
 difficulty: intermediate
 updated: 2026-10-04
 references:
   - name: "Bellingcat's Online Investigation Toolkit"
     url: "https://bellingcat.gitbook.io/toolkit"
     author: "Bellingcat"
     license: none
     relation: derived
     note: "Tool catalogue: names, descriptions, cost flags and links for this area."
   - name: "OSINT Newsletter Tools Library"
     url: "https://tools.osintnewsletter.com"
     author: "The OSINT Newsletter"
     license: none
     relation: derived
     note: "Second tool catalogue, cross-checked against the above."
 ---
 
 ## What this covers
 
 Watching things over time instead of looking once, and holding what you collect so you can analyse
 it later without re-collecting. Two distinct jobs with different tooling: **bulk collection** of a
 corpus for analysis, and **change detection** on a small number of pages you care about. The third
 thing, which is not optional, is doing either without the pattern of your collection becoming
 visible to the people you are collecting on.
 
 ## Method
 
 1. **Decide what you are watching before you build anything.** A corpus for network analysis and an
    alert on one bio edit need completely different infrastructure, and building the first when you
    needed the second is the usual waste.
 2. **Prefer the platform's own feed.** A native RSS or JSON endpoint is stable, keyless, polite and
    does not attribute activity to an account. Reach for a scraper only when no feed exists.
 3. **Store raw, analyse later.** Keep the untouched response alongside anything you derive from it.
    You will want to re-run the analysis with a different question, and you will not want to
    re-collect under a worse rate limit.
 4. **Record gaps as gaps.** A failed poll is a hole in your data, not a quiet day. Log the failures
    next to the results or your time series will lie to you.
 5. **Set the cadence to the question.** Hourly is almost never justified. A daily poll that runs
    for a year is more valuable, and far less conspicuous, than a five-minute poll that gets you
    blocked in a week.
 6. **Baseline before you conclude.** Coordination, surges and silences only mean something against
    what normal looks like for that topic.
 
 The judgement calls: whether you need the content or only the fact that it changed — the second is
 far cheaper and far quieter. Whether the collection needs an account at all, because the moment it
 does, the collection has an identity and a history. And whether you are allowed to keep what you
 are about to collect, which is a question to settle at the start rather than after you have a
 database of it.
 
 ## Collection hygiene
 
 Monitoring is repeated, scheduled, patterned contact with a target's content. One look is
 invisible. The same request every fifteen minutes from one address for six months is a signature,
 and on some platforms it is a signature attached to a logged-in account.
 
 ```text
 Account
   - never your own, and never one that shares a recovery phone or email with your own
   - a research account per investigation where the platform allows it; one burned
     account should not cost you the others
   - assume every logged-in query is retained and attributable, including search terms
   - some platforms notify a user when a profile is viewed; know which before you look
 
 Rate
   - daily is the default. Justify anything faster to yourself in writing
   - randomise the interval. A poll at exactly :00 every hour is machine-obvious
   - respect the stated limit, and treat a 429 as a signal to back off for hours,
     not to retry in a loop
   - set a real, honest User-Agent that identifies the project. It gets you unblocked
     more often than a spoofed browser string does
 
 Network
   - one address for all of it correlates every collection you run
   - a residential or VPN exit is a trade: less rate limiting, more attribution risk
     to whoever pays for it
   - Tor is blocked by most of these platforms, so it is rarely the answer here
 
 Footprint
   - log every request you make, with timestamp and response code. You need it to
     distinguish "they stopped posting" from "we got blocked"
   - decide retention up front. Bulk personal data attracts obligations even when
     every item was public
 ```
 
 The archiving and provenance side of this — hashes, snapshots, what makes a collected item citable
 later — is on [Archiving & Evidence](/sheets/osint/archiving-and-evidence). Per-platform surfaces,
 query syntax and the specific scrapers are on
 [Social Media Platforms](/sheets/osint/social-media-platforms); this sheet is about running those
 things repeatedly rather than once.
 
 ## Key tools
 
 ### 4CAT
 
 A self-hosted capture-and-analysis platform. It is the serious option because collection and
 analysis live in the same place: the dataset you capture stays on your machine, and you can re-run
 a different analysis over it months later without touching the platform again. That is the thing
 no hosted service gives you.
 
 ```bash
 # the documented install is the compose file plus its .env, not a clone
 mkdir 4cat && cd 4cat
 curl -O https://raw.githubusercontent.com/digitalmethodsinitiative/4cat/master/docker-compose.yml
 curl -O https://raw.githubusercontent.com/digitalmethodsinitiative/4cat/master/.env
 
 # the two settings worth reading before you start it:
 #   SERVER_BIND_ADDRESS=127.0.0.1   localhost only (the shipped default)
 #   PUBLIC_PORT=80                  the port the web UI lands on
 grep -E 'SERVER_BIND_ADDRESS|PUBLIC_PORT|DOCKER_TAG' .env
 
 docker compose up -d
 docker compose logs -f backend      # first start builds indexes; wait for it
 
 # the one-time admin link is printed in the logs, not emailed
 docker compose logs frontend | grep -i 'create a new user\|token'
 ```
 
 Creating a dataset, once it is up:
 
 ```text
 1. http://localhost:80 -> "Create dataset".
 2. Pick the data source. Natively it collects 4chan, 8kun, Bluesky, Telegram, Tumblr
    and TikTok (from a list of URLs). Everything else arrives as an upload.
 3. Set the query and the date range. Date range is the field people leave open and
    then wonder why the job runs for a day -- bound it.
 4. Submit, then leave it. The backend queues and processes; the dataset appears in
    your list with a status. Big Telegram or 4chan pulls take hours.
 5. On the finished dataset, run processors rather than exporting immediately:
    word frequencies, co-word networks, time series, top posters, image downloads
    and clustering. Each produces a child dataset you can chain further processors on.
 6. Export the one you want as CSV or NDJSON, and keep the parent dataset. The parent
    is the thing you cannot re-create.
 ```
 
 The platform coverage is the live constraint and it changes: X/Twitter, Instagram, LinkedIn,
 Threads and Pinterest are no longer collected by 4CAT directly — they come in through Zeeschuimer
 below, or as an upload from elsewhere. Datasets are big, and the Docker volumes will fill a disk
 without warning, so check free space before a large pull. And a 4CAT capture is a snapshot of what
 the platform served at that moment: edits and deletions after the capture are invisible to it,
 which is a feature when you want the original and a trap when you assume it reflects the live site.
 
 ### Zeeschuimer
 
 The companion extension, and the current answer to "how do I get X, Instagram or TikTok data into
 4CAT". It watches the data the platform sends to your browser as you scroll, and keeps it. No API,
 no scraping requests of its own — it records what your normal browsing already fetched.
 
 ```text
 1. Firefox only. Signed .xpi builds are on the project's releases page; Chrome is
    not supported.
 2. Install, then open the extension and enable the platform you are about to browse.
    Supported: TikTok, Instagram, Threads, X/Twitter, Pinterest, Gab, Truth Social,
    9gag, Imgur, Douyin and RedNote/Xiaohongshu.
 3. Browse normally -- a profile, a hashtag, a search. The item counter in the
    extension ticks up as content loads. Content you never scrolled past is not
    captured, because it was never sent to your browser.
 4. Export as NDJSON or CSV, or put your 4CAT URL in the extension and upload straight
    into it as a dataset.
 5. Record how you browsed: which profile, which tab, how far you scrolled. That is
    the sampling frame, and without it the dataset's coverage is undocumented.
 ```
 
 This is manual collection with automatic recording, so it does not scale and it does not run
 unattended — which is also why it survives platform changes better than scrapers do. It captures
 through your logged-in session, so everything you collect is attributable to that account: use a
 research profile in a separate Firefox container or profile, never your own. Platform support needs
 constant maintenance and individual platforms do break; check the releases page before assuming the
 extension is at fault rather than the site.
 
 ### changedetection.io
 
 Self-hosted page watching. This is the one to run when the thing you need to notice is an edit — a
 bio rewritten, a post deleted, an officer removed from a company page, a policy document quietly
 amended, a price or a staff list changing.
 
 ```bash
 # docker, bound to localhost
 docker run -d --restart always -p "127.0.0.1:5000:5000" \
   -v datastore-volume:/datastore \
   --name changedetection.io dgtlmoon/changedetection.io
 
 # or as a package, if you would rather not run a container
 pip3 install changedetection.io
 changedetection.io -d /path/to/empty/data/dir -p 5000
 ```
 
 ```text
 1. http://localhost:5000 -> "Add new watch", paste the URL.
 2. Open "Edit" and set the filter before you set anything else. CSS selector, XPath,
    JSONPath or jq -- point it at the narrowest element carrying the signal. Watching
    a whole page means alerting on rotating ads, view counters and timestamps.
    The "Visual Selector" tab picks the element by clicking it.
 3. Set "Ignore text" for the lines you know churn: relative dates, "N views",
    cookie-banner text.
 4. Recheck time: per-watch. Daily for most things. The default applies to every watch
    you add, so set it low once rather than per watch.
 5. Notifications: Discord, email, Slack, Telegram or a webhook via Apprise. A webhook
    into your own notes or ticketing is the one that leaves a record.
 6. For a page that renders client-side, enable the Playwright/Sockpuppetbrowser
    fetcher for that watch -- the default plain fetch sees an empty shell and will
    report "no change" forever.
 7. "Browser Steps" handles a page behind a form: click, fill, submit, then diff what
    comes back.
 8. Every change is stored as a snapshot with a diff view, which is the part that makes
    it evidence rather than an alert.
 ```
 
 Running it yourself means your watch list is not a third party's business record, which matters
 when the watch list itself is sensitive. The cost is that a JS-heavy watch runs a real browser and
 is far heavier than a text diff — a dozen of those on a small VPS will struggle. And a watch only
 sees what an unauthenticated fetch from your server sees: a page that needs a login, or that
 geo-varies, needs Browser Steps or will silently watch the wrong thing.
 
 ### Distill
 
 The hosted equivalent, for when you will not run infrastructure. Same idea, less setup, and a free
 tier that is genuinely usable for a handful of watches: **25 monitors total but only 5 in the
 cloud, a 6-hour minimum cloud interval, 1,000 cloud checks a month, 2 devices, and 30 email alerts
 a month**. Local monitors in the browser extension are unlimited but only run while the browser is
 open.
 
 ```text
 1. Browser extension or distill.io. On the page you want, click the extension and
    select the region -- it generates the selector for you.
 2. Choose local (runs in your browser, unlimited, only while open) or cloud (runs
    without you, capped as above). For anything that matters, cloud.
 3. Set the check interval. Anything under 6 hours is a paid feature on the free tier,
    and six-hourly is adequate for almost all of this work anyway.
 4. Set the condition, not just "any change" -- Distill supports text conditions, so
    "alert when the number changes" beats "alert when the page differs".
 5. Alerts to email or webhook; the email allowance is the binding constraint on free.
 6. Export the watch list as JSON periodically. It is the only part that is painful to
    rebuild.
 ```
 
 Your watch list and every snapshot live on their servers, which is the trade for not running
 anything. For a target that could plausibly subpoena or compromise a third party, that is the wrong
 trade and `changedetection.io` is the answer instead. The free tier's 6-hour floor also means you
 will miss a post that goes up and comes down inside a window, which is precisely the kind of
 deletion worth catching — if that is the scenario, self-host and poll faster.
 
 ### yt-dlp with a download archive
 
 For recurring capture of a channel's output, the archive file is the whole trick: it records the ID
 of everything already fetched, so the next run picks up only what is new. That turns a one-off
 download into a monitor you can cron.
 
 ```bash
 pipx install yt-dlp
 
 # first run: establish the archive. --break-on-existing stops as soon as it meets
 # something already recorded, so later runs walk only the new items
 yt-dlp --download-archive archive.txt --break-on-existing --lazy-playlist \
   -o '%(upload_date)s-%(id)s.%(ext)s' \
   'https://youtube.com/@channel/videos'
 
 # metadata-only monitoring: no video files, just the record that it existed
 yt-dlp --download-archive seen.txt --break-on-existing \
   --skip-download --write-info-json --write-thumbnail \
   'https://youtube.com/@channel/videos'
 
 # bound it by date instead, for a backfill of a known window
 yt-dlp --dateafter 20260101 --datebefore 20260401 --download-archive archive.txt URL
 
 # subtitles, which turn a channel's output into greppable text as it arrives
 yt-dlp --download-archive seen.txt --break-on-existing --skip-download \
   --write-auto-subs --sub-langs en 'https://youtube.com/@channel/videos'
 
 # throttle it. these two flags are the difference between a monitor and a nuisance
 yt-dlp --sleep-requests 2 --sleep-interval 10 --max-sleep-interval 30 \
   --download-archive archive.txt URL
 
 # several channels in one run, each stopping at its own first-seen item
 yt-dlp --break-per-input --break-on-existing --download-archive archive.txt \
   -a channels.txt
 
 # a cap, so a misconfigured run cannot pull a thousand files overnight
 yt-dlp --max-downloads 50 --download-archive archive.txt URL
 ```
 
 The archive file is state: back it up, and never delete it to "start fresh" unless you mean to
 re-download everything. `--break-on-existing` assumes the listing is newest-first, which it is for
 channels and playlists and is not for some search result pages — on those, drop it and let the
 archive do the skipping. Extractors break when platforms change their pages, so `pipx upgrade
 yt-dlp` before blaming a URL, and a cron job that has silently failed for three weeks is worse than
 no monitoring at all, so alert on non-zero exits. Adding `--cookies-from-browser` attaches a real
 session to every request and makes the whole monitor attributable; avoid it unless the content
 genuinely requires a login. Full metadata usage is on
 [Social Media Platforms](/sheets/osint/social-media-platforms).
 
 ### Native feeds, before you reach for a bridge
 
 Several platforms still publish perfectly good feeds that nobody uses because everyone assumes RSS
 died. These are keyless, stable, cheap to poll and attach to no account — the best monitoring
 surface available, where it exists.
 
 ```bash
 # YouTube channel, by channel ID (the UC... form; @handles do not work here)
 curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw'
 
 # a subreddit's new posts
 curl -s -A 'research-monitor/1.0 (contact: you@example.org)' \
   'https://www.reddit.com/r/osint/new/.rss'
 
 # one Reddit user's activity
 curl -s -A 'research-monitor/1.0 (contact: you@example.org)' \
   'https://www.reddit.com/user/someuser/.rss'
 
 # any Mastodon account, on any instance
 curl -s 'https://mastodon.social/@Gargron.rss'
 
 # a Bluesky profile
 curl -s 'https://bsky.app/profile/bsky.app/rss'
 
 # a GitHub user's public activity, which dates account behaviour precisely
 curl -s 'https://github.com/bellingcat.atom'
 
 # pull just the timestamps, to see cadence without reading content
 curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw' \
   | grep -oE '<published>[^<]+' | sed 's/<published>//'
 ```
 
 Reddit rate-limits these hard and will return HTTP 429 to an anonymous, default-User-Agent client
 within a handful of requests; a descriptive User-Agent and a gap of several seconds between calls
 fixes it, and Reddit's `search.rss` endpoint is throttled more aggressively than the subreddit and
 user feeds. YouTube's feed carries only the most recent entries, so it monitors but does not
 backfill. Bluesky's profile RSS covers posts and not replies or likes — the AT Protocol endpoints
 on [Social Media Platforms](/sheets/osint/social-media-platforms) go deeper. And X/Twitter publishes
 nothing of this kind: `snscrape` has been non-functional against it since the 2023 access changes
 and the public Nitter instances are largely dead, so there is no quiet monitoring surface for X at
 all — only a logged-in session, with everything that implies.
 
 ### RSSHub and RSS-Bridge
 
 For the platforms that killed their feeds. Both are self-hosted services that scrape a site and
 re-emit it as RSS or Atom, so your monitor still speaks one protocol no matter how many platforms
 are behind it.
 
 ```bash
 # RSSHub -- the larger of the two, ~1000 routes
 docker run -d --name rsshub -p 1200:1200 diygod/rsshub
 # or with the compose file, which adds redis caching and a browser for JS sites
 wget https://raw.githubusercontent.com/DIYgod/RSSHub/master/docker-compose.yml
 docker compose up -d
 
 # routes are paths. a Telegram public channel:
 curl -s 'http://localhost:1200/telegram/channel/awesomeRSSHub'
 # the route catalogue lives at docs.rsshub.app/routes/
 
 # RSS-Bridge -- fewer routes (~450) but simpler, and a web UI that builds the URL
 docker create --name=rss-bridge --publish 3000:80 \
   --volume $(pwd)/config:/config rssbridge/rss-bridge
 docker start rss-bridge
 
 # bridges are query parameters rather than paths
 curl -s 'http://localhost:3000/?action=display&bridge=<BridgeName>&format=Atom'
 ```
 
 Self-host both. The public `rsshub.app` demo instance returns HTTP 403 to ordinary requests and is
 not a reliable backend for anything you depend on, and a shared public instance makes your watch
 list someone else's log file either way. Expect individual routes to break: they are scrapers
 wearing an RSS hat, and when a platform changes its markup the route returns an empty feed rather
 than an error — which reads exactly like "the target stopped posting". Check periodically that a
 route still returns items, and never conclude silence from an empty bridge feed without confirming
 against the site.
 
 ### curl and jq on a schedule
 
 For a public JSON endpoint, a few lines beat any framework. The pattern that matters is the
 watermark: store the timestamp of the newest item you have seen, and ask only for things after it.
 
 ```bash
 # poll Arctic Shift for new Reddit posts in a subreddit since the last run.
 # full Arctic Shift usage is on the social media platforms sheet; this is the
 # monitoring shape of it
 AS='https://arctic-shift.photon-reddit.com/api'
 STATE=~/monitor/last_seen_osint
 SINCE=$(cat "$STATE" 2>/dev/null || echo 0)
 
 curl -sS --fail -G "$AS/posts/search" \
   --data-urlencode 'subreddit=osint' \
   --data-urlencode "after=$SINCE" \
   --data-urlencode 'sort=asc' \
   --data-urlencode 'limit=100' \
   -o /tmp/new.json || { echo "poll failed $(date -u +%FT%TZ)" >> ~/monitor/errors.log; exit 1; }
 
 # advance the watermark only on a successful fetch, or a failure silently
 # becomes a gap you never notice
 jq -r '.data[-1].created_utc // empty' /tmp/new.json | grep . && \
   jq -r '.data[-1].created_utc' /tmp/new.json > "$STATE"
 
 # append raw, then derive. the raw file is the thing you cannot regenerate
 cat /tmp/new.json >> ~/monitor/osint-raw.ndjson
 jq -r '.data[] | [.created_utc, .author, .title] | @tsv' /tmp/new.json
 ```
 
 ```bash
 # a Bluesky author feed, keyless, as a second example of the same shape
 curl -sS --fail 'https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=bsky.app&limit=50' \
   | jq -r '.feed[] | [.post.indexedAt, .post.record.text] | @tsv'
 
 # crontab: daily, at a minute that is not :00, with failures mailed to you
 # 37 6 * * *  /home/you/monitor/poll-osint.sh >> /home/you/monitor/run.log 2>&1
 ```
 
 `--fail` is the flag that turns a silent HTML error page into a non-zero exit, and without it your
 monitor will happily append a Cloudflare block page to its dataset for a month. Advance the
 watermark only after a confirmed good response, log every failure with a timestamp, and treat the
 log as part of the dataset — the difference between "they went quiet" and "we were blocked" lives
 there and nowhere else. Note that Bluesky's `getAuthorFeed` is open without a token but
 `searchPosts` on the same public host returns 403 unauthenticated, which is the sort of asymmetry
 worth checking per endpoint before you build a schedule around it.
 
 ### Arctic Shift for Reddit history
 
 The live successor to Pushshift, and the only practical way to get Reddit content that has since
 been deleted or edited. For monitoring it plays a specific role: it is the backfill that gives your
 forward-looking poll a baseline, and the thing you check when a post you captured disappears.
 
 ```bash
 AS='https://arctic-shift.photon-reddit.com/api'
 
 # the baseline: what this account did before you started watching it
 curl -s -G "$AS/comments/search" --data-urlencode 'author=some_user' \
   --data-urlencode 'sort=asc' --data-urlencode 'limit=100' \
   | jq -r '.data[] | [.created_utc, .subreddit] | @tsv'
 
 # posting cadence by hour of day, which is what exposes a coordinated account
 curl -s -G "$AS/posts/search" --data-urlencode 'author=some_user' \
   --data-urlencode 'limit=500' --data-urlencode 'fields=created_utc' \
   | jq -r '.data[].created_utc' \
   | python3 -c 'import sys,datetime,collections; c=collections.Counter(datetime.datetime.fromtimestamp(int(l), datetime.UTC).hour for l in sys.stdin); [print(f"{h:02d} {c[h]}") for h in range(24)]'
 
 # did a post you captured actually get removed, or did you lose it
 curl -s -G "$AS/posts/ids" --data-urlencode 'ids=t3_abc123' | jq '.data[0] | {title, selftext, removed_by_category}'
 ```
 
 Keyword search needs an accompanying `author`, `subreddit`, `link_id` or `parent_id` — bare keyword
 sweeps are refused, and so are very broad author or subreddit queries. The archive holds what it
 ingested at the time, so an edit made after ingestion is invisible and a post deleted within
 seconds may never have been captured at all; a record here proves the text was published, not that
 it is live. Bulk dumps for offline work are published separately from the API. Query syntax in full
 is on [Social Media Platforms](/sheets/osint/social-media-platforms).
 
 ## Narrative and coordination analysis
 
 - **Posting-time clustering** exposes networks. Accounts that post within seconds of each other,
   repeatedly, are coordinated. The `jq` plus histogram pattern above is enough to see it.
 - **Identical phrasing across accounts** is the strongest single indicator of copy-paste campaigns.
 - **Follower overlap** between accounts is more telling than follower count.
 - [The Information Laundromat](https://informationlaundromat.com/) compares content and technical
   metadata — ad IDs, analytics tags, registration details, code structure — across sites, to find
   where the same text is syndicated and which sites share infrastructure. Built by the Alliance for
   Securing Democracy, which merged into the Institute for Strategic Dialogue on 1 January 2026; the
   tool remains live at the address above. The older hyphenated domain no longer resolves.
 - Graphing and clustering what you have collected is on
   [Data Analysis & Visualisation](/sheets/osint/data-analysis-and-visualisation).
 
 ## Tool reference
 
 | Tool | What it does | Cost |
 | --- | --- | --- |
 | [4CAT](https://4cat.nl/) | 4CAT is a tool designed for the easy collection and analysis of online datasets. It allows researchers to uncover patterns and trends in data from social… | free |
 | [Atlos](https://www.atlos.org/) | ATLOS is a platform for collaborative and large-scale open source investigations. | partly free |
 | [Datasette](https://datasette.io) | Open-source “WordPress-for-data” that turns any SQLite database into an interactive website and JSON API in seconds; ideal for publishing, exploring and… | free |
 | [Gephi](https://gephi.org) | Open-source network analysis and visualization software | free |
 | [Maltego Graph](https://www.maltego.com/downloads/) | Maltego Graph is an investigation platform that combines two things at once: (1) It acts as a search tool, and (2) It creates a graph establishing links… | partly free |
 | [Pinpoint](https://journaliststudio.google.com/pinpoint/about) | A tool by Google to catalogue uploaded documents and files, providing automated text recogntion, indexing, audiotranscriptions and other (AI-powered)… | free |
 | [Time.Graphics](https://time.graphics) | A tool for creating, visualizing, and managing timelines online. | partly free |
 | [Zeeschuimer](https://github.com/digitalmethodsinitiative/zeeschuimer) | Firefox extension that records social media data as you browse it, for import into 4CAT. | free |
 | [changedetection.io](https://changedetection.io/) | Self-hosted page-change monitoring with CSS/XPath/jq filters and diff history. | free |
 | [Distill](https://distill.io/) | Hosted page-change monitoring, browser extension plus cloud checks. | partly free |
 
 ## Pitfalls
 
 - **Dead tools still appear in tutorials.** `snscrape` and the public Nitter instances do not work
   against X. Bing's Visual Search API and the rest of the Bing Search family were decommissioned in
   August 2025. Check a tool is alive before you design a collection around it.
 - **Rate limits end collections mid-run.** Check for gaps before you analyse; a missing day looks
   like silence rather than a failure.
 - **An empty feed is ambiguous.** A broken RSSHub route, a changed selector and a target who
   stopped posting all look identical downstream. Monitor your monitors.
 - **Sampling bias reads as a finding.** If a tool only reaches accounts above some follower count,
   or only what you happened to scroll past, its "network" is an artefact of that cutoff.
 - **Coordination needs a baseline.** Fans of the same thing post about it at the same time. Compare
   against normal behaviour for the topic before calling it inauthentic.
 - **Monitoring is contact.** Scheduled requests are a pattern; a logged-in monitor is an
   attributable pattern. Decide what that costs before you start, not after.
 - **Storage and legality.** Bulk personal data attracts data-protection obligations even when every
   individual item was public, and a dataset is harder to delete than to collect.
 
 ## Worked example
 
 One datum: a single Telegram channel name, from a screenshot, alleged to be seeding a story that
 later appeared on a cluster of news-like websites.
 
 1. **Baseline before watching.** Pull the channel's existing history into 4CAT as a Telegram
    dataset with a bounded date range. You now know its normal posting cadence and vocabulary, which
    is what any later claim of a surge has to be measured against.
 2. **Find the forward-looking surface.** Telegram has no public feed, so an RSSHub route
    (`/telegram/channel/<name>` on your own instance) gives you a pollable endpoint. Verify it
    returns items today, so that an empty feed next month means something.
 3. **Schedule it honestly.** A daily `curl` with a stored watermark, a descriptive User-Agent, and
    a failure log. Not hourly — a story that takes days to syndicate does not need fifteen-minute
    resolution, and the slower poll survives longer.
 4. **Watch the downstream sites for edits, not just posts.** Each suspected site gets a
    `changedetection.io` watch filtered to the article body, so a quietly amended paragraph or a
    removed byline raises an alert with a stored diff. This is the part that a collection-only
    approach misses entirely.
 5. **Capture the video claims properly.** The channel posts clips; `yt-dlp --download-archive`
    with `--write-info-json` on its linked channels picks up only what is new each day and records
    `upload_date` for each, giving every clip a latest-possible date.
 6. **Test the syndication claim.** Feed one article URL to the Information Laundromat. Content
    similarity tells you the text is shared; the technical indicators — a common analytics ID across
    four of the sites — tell you something stronger, because wording can be copied by anyone and a
    shared tracking ID usually cannot.
 7. **Cross-check the Reddit leg.** Arctic Shift for posts linking those domains, grouped by
    subreddit and by hour, shows whether the amplification is a handful of accounts on a schedule or
    genuine spread.
 8. **Archive as you go.** Every page you will cite goes to a snapshot at the time you saw it, per
    [Archiving & Evidence](/sheets/osint/archiving-and-evidence) — these sites edit and disappear,
    which is the behaviour you are documenting.
 
 What you can assert: a dated posting history for the channel, dated first appearances on each
 downstream site, stored diffs of any subsequent edits, and a shared technical indicator linking
 some of those sites.
 
 What would falsify it: an RSSHub route that broke silently mid-period, which would turn a real gap
 into an apparent one — so the run log, showing a successful fetch every day, is doing as much
 evidential work as the data. A shared analytics ID that turns out to belong to a common CMS
 template, or to an agency that serves unrelated clients, would break the infrastructure link; check
 what else carries that ID before leaning on it.
 
 ## Broader catalogues
 
 - [Social Media OSINT](https://tools.osintnewsletter.com/tool-categories/social-media-osint)
 
 
 ## Sources
 
 Both catalogues below are maintained by other people and are considerably larger than
 this page. Use them as the canonical index; this sheet is a working route through them.
 
 - [Bellingcat's Online Investigation Toolkit](https://bellingcat.gitbook.io/toolkit) — ~340 tools, each with its own
   review page covering cost, difficulty, requirements and limitations.
 - [OSINT Newsletter Tools Library](https://tools.osintnewsletter.com) — ~280 tools, organised by investigative goal.
 
 Neither publishes a licence, so nothing here is copied from them: tool names, one-line
 descriptions, cost flags and links are catalogue facts, and the method and commentary are
 this site's own. See [credits](/credits).