web-crawler

The saved excerpt is a snapshot from when this skill was listed. The external source remains the complete and most current version.

---
name: web-crawler
version: 2.13.0
description: 'Web scraping plus social data: YouTube, TikTok, Instagram (incl. stories),
  LinkedIn, Reddit, Threads, plus China apps and Asian e-commerce via Apify (Douyin,
  Xiaohongshu, Weibo, Bilibili, Zhihu, Douban, Xueqiu, Taobao, JD, 1688, Xianyu, Shopee,
  Lazada, Temu, Trip.com reviews), plus robust web-page fallback extraction.


  Use when extracting public posts, transcripts, or pages JS-heavy enough to block
  plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages
  behind anti-bot/Cloudflare). Auto-fallback trigger terms: web_fetch failed, 403,
  anti-bot, cloudflare, JS-heavy page.

  '
metadata:
  starchild:
    emoji: "\U0001F577\uFE0F"
    skillKey: web-crawler
    requires:
      bins:
      - python
    tags:
    - scraping
    - social-media
    - web-crawler
    - tiktok
    - instagram
    - youtube
    - linkedin
    - facebook
    - twitter
    - reddit
    - threads
    - bluesky
    - pinterest
    - snapchat
    - twitch
    - truth-social
    - tiktok-shop
    - google
    - ad-library
    - creator-data
    - transcripts
    - trending
    - web_fetch-failed
    - cloudflare
    - anti-bot
    - js-heavy
    - listing-page
    - fallback
    - "\u53CD\u722C"
    - "\u6293\u53D6\u5931\u8D25"
    - "\u81EA\u52A8\u56DE\u9000"
user-invocable: true
disable-model-invocation: false
---

## Preferred entry: call exports.py (don't hand-roll requests)
Ready-made helpers live in `skills/web-crawler/exports.py`. Prefer them over
writing your own `proxied_get`/`proxied_post` calls — they already inject the
proxy credentials, so **there is no API key to find** (don't read
`$SCRAPECREATORS_API_KEY` / `$FIRECRAWL_API_KEY`, don't check `.env`, don't ask
the user).

```python
import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article")          # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID")     # ScrapeCreators
youtube_video("https://youtube.com/watch?v=ID")          # metadata: title/description/uploader (+ transcript)
sc_get("/v1/tiktok/profile", handle="charlidamelio")     # any SC endpoint

from exports import archive_fallback                      # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html")  # archive snapshot
```

### Transcript route — ONE entry for any media link

```python
from core.skill_tools import web_crawler
r = web_crawler.get_transcript(url, caller_id="chat:<thread>")
# {found, kind, source, text, url, title, tried, note}
```

**Design rule: captions / subtitles / published transcripts are METADATA;
the media file is PAYLOAD.** Exhaust every metadata provider first; touch the
media only when the user explicitly agrees. Same rule claude-video, spoken.md
and yt-dlp's `--skip-download` workflow follow. Nothing here is site-specific:

| provider (cost order) | what it is | identity / validity check |
|---|---|---|
| `media_info(url)` | yt-dlp `extract_info(download=False)` — 1800+ sites, returns title, duration, upload date, channel, caption-track URLs. Zero media bytes. | track fetched must be 2xx with a real body |
| RSS `<podcast:transcript>` | Podcasting 2.0 standard, any podcast | feed item matched by duration ±5 % + date ±3 d; transcript URL must be 2xx, > 500 chars, not an error page |
| show's own YouTube upload | channel = the YouTube URL the show declares in its RSS (else search hit whose channel name **equals** the show name minus stopwords) | candidates = duration ±5 % AND date ±3 d; **identity = title similarity ≥ 0.6 OR show-notes overlap** (≥ 6 shared distinctive tokens, Jaccard ≥ 0.15). A candidate that fails identity is returned in `candidate` with `found=False` — never delivered as the episode |
| `scrape_markdown(url)` | any other page (publisher, Snipd, blog) — browser-rendered, survives WAF 403 | > 1000 chars |

`found=False` ⇒ rep