just-scrape

The saved excerpt is a snapshot from when this skill was listed. The external source remains the complete and most current version.

---
name: just-scrape
description: Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes, inspect request history, check ScrapeGraph credits, or validate API setup.
compatibility: "Requires the just-scrape CLI (`npm install -g just-scrape`). Requires `SGAI_API_KEY` for ScrapeGraph AI requests."
license: MIT
allowed-tools: Bash
metadata:
  openclaw:
    requires:
      bins:
        - just-scrape
    install:
      - kind: node
        package: just-scrape
        bins: [just-scrape]
    homepage: https://github.com/ScrapeGraphAI/just-scrape
---

# just-scrape CLI

Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI.

Run `just-scrape --help` or `just-scrape <command> --help` for full option details.

If the task is to integrate ScrapeGraph AI into application code, add `SGAI_API_KEY` to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill.

## Prerequisites

Must be installed and authenticated. Check with `just-scrape validate` and `just-scrape credits`.

```bash
command -v just-scrape >/dev/null 2>&1 || npm install -g just-scrape@latest
just-scrape validate
just-scrape credits
```

- **API key**: Set `SGAI_API_KEY`, use a `.env` file, use `~/.scrapegraphai/config.json`, or complete the interactive prompt.
- **Credits**: Remaining ScrapeGraph AI credits. Each operation consumes credits.

Before doing real work, verify the setup with one small request:

```bash
mkdir -p .just-scrape
just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json
```

```bash
just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json
```

## Workflow

Follow this escalation pattern:

1. **Search** - No specific URL yet. Find pages, answer questions, discover sources.
2. **Scrape** - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding.
3. **Extract** - Need structured JSON from a known URL with an AI prompt and optional schema.
4. **Crawl** - Need bulk content from an entire site section.
5. **Monitor** - Need scheduled page-change tracking with optional webhook notifications.

| Need                        | Command    | When                                       |
| --------------------------- | ---------- | ------------------------------------------ |
| Find pages on a topic       | `search`   | No specific URL yet                        |
| Get a page's content        | `scrape`   | Have a URL, need one or more page formats  |
| AI-powered data extraction  | `extract`  | Need structured data from a known URL      |
| Bulk extract a site section | `crawl`    | Need many pages or docs sections           |
| Track changes over time     | `monitor`  | Need recurring scraping and webhooks       |
| Inspect prior requests      | `history`  | Need past request IDs, status, or payloads |
| Check credit balance        | `credits`  | Need remaining API credits                 |
| Validate API setup          | `validate` | Need health check and API key validation   |

For detailed command reference, run `just-scrape <command> --help`.

**Scrape vs extract:**

- Use `scrape` for raw page formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`.
- Use `scrape -f json -p "<prompt>"` or `extract -p "<prompt>"` for AI-structured output.
- Use `extract` when the task is only structured data. Use `scrape` when mixed formats are needed in one call.

**Avoid redundant fetches:**

- `search -p` can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete.
- `crawl` already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass i