刮擦技巧
- 作者仓库星标 0
- 作者更新于 2026年8月24日 07:33
- 作者仓库 claude-code-skills
- 领域
- 文档
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @daymade · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- Python
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
wc -c article.md
sed -n '1,40p' article.md
第 1 步:诊断安装
首先运行捆绑的诊断脚本:使用结果作为下一步的事实来源。
第三步:选择抓取器
使用这个决策规则:对于普通页面、文章页面和大多数微信公众文章,从`extract get`开始。当静态 HTML 不包含真实内容或页面依赖于 JavaScript 渲染时,请使用`extract fetch`。仅在`fetch`因反机器人或挑战行为而失败后才使用`extract stealthy-fetch`。不要将其设为默认值。
第 4 步:运行最小的有用命令
始终在 shell 命令中引用 URL。当 URL 包含 `?`、`&` 或其他特殊字符时,这在 `zsh` 中是强制的。
第 5 步:验证输出
每次提取后,验证文件而不是假设成功:对于 HTML 输出,检查预期的标题、容器或选择器目标是否实际存在:如果文件很小、为空或缺少预期的容器,则提取不会成功。返回步骤 3 并切换获取器或选择器。
# Scrapling Skill
## Overview
Use Scrapling through its CLI as the default path. Start with the smallest working command, validate the saved output, and only escalate to browser-backed fetching when the static fetch does not contain the real page content.
Do not assume the user's Scrapling install is healthy. Verify it first.
## Default Workflow
Copy this checklist and keep it updated while working:
```text
Scrapling Progress:
- [ ] Step 1: Diagnose the local Scrapling install
- [ ] Step 2: Fix CLI extras or browser runtime if needed
- [ ] Step 3: Choose static or dynamic fetch
- [ ] Step 4: Save output to a file
- [ ] Step 5: Validate file size and extracted content
- [ ] Step 6: Escalate only if the previous path failed
```
## Step 1: Diagnose the Install
Run the bundled diagnostic script first:
```bash
python3 scripts/diagnose_scrapling.py
```
Use the result as the source of truth for the next step.
## Step 2: Fix the Install
### If the CLI was installed without extras
If `scrapling --help` fails with missing `click` or a message about installing Scrapling with extras, reinstall it with the CLI extra:
```bash
uv tool uninstall scrapling
uv tool install 'scrapling[shell]'
```
Do not default to `scrapling[all]` unless the user explicitly needs the broader feature set.
### If browser-backed fetchers are needed
Install the Playwright runtime:
```bash
scrapling install
```
If the install looks slow or opaque, read `references/troubleshooting.md` before guessing. Do not claim success until either:
- `scrapling install` reports that dependencies are already installed, or
- the diagnostic script confirms both Chromium and Chrome Headless Shell are present.
## Step 3: Choose the Fetcher
Use this decision rule:
- Start with `extract get` for normal pages, article pages, and most WeChat public articles.
- Use `extract fetch` when the static HTML does not contain the real content or the page depends on JavaScript rendering.
- Use `extract stealthy-fetch` only after `fetch` still fails because of anti-bot or challenge behavior. Do not make it the default.
## Step 4: Run the Smallest Useful Command
Always quote URLs in shell commands. This is mandatory in `zsh` when the URL contains `?`, `&`, or other special characters.
### Full page to HTML
```bash
scrapling extract get 'https://example.com' page.html
```
### Main content to Markdown
```bash
scrapling extract get 'https://example.com' article.md -s 'main'
```
### JS-rendered page with browser automation
```bash
scrapling extract fetch 'https://example.com' page.html --timeout 20000
```
### WeChat public article body
Use `#js_content` first. This is the default selector for article body extraction on `mp.weixin.qq.com` pages.
```bash
scrapling extract get 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' article.md -s '#js_content'
```
## Step 5: Validate the Output
After every extraction, verify the file instead of assuming success:
```bash
wc -c article.md
sed -n '1,40p' article.md
```
For HTML output, check that the expected title, container, or selector target is actually present:
```bash
rg -n '<title>|js_content|rich_media_title|main' page.html
```
If the file is tiny, empty, or missing the expected container, the extraction did not succeed. Go back to Step 3 and switch fetchers or selectors.
## Step 6: Handle Known Failure Modes
### Local TLS trust store problem
If `extract get` fails with `curl: (60) SSL certificate problem`, treat it as a local trust-store problem first, not a Scrapling content failure.
Retry the same command with:
```bash
--no-verify
```
Only do this after confirming the failure matches the local certificate verification error pattern. Do not silently disable verification by default.
### WeChat article pages
For `mp.weixin.qq.com`:
- Try `extract get` before `extract fetch`
- Use `-s '#js_content'` for the article body
- Validate the saved Markdown or HTML immediately
### Browser-backed fetch failures
If `extract fetch` fails:
1. Re-check the install with `python3 scripts/diagnose_scrapling.py`
2. Confirm Chromium and Chrome Headless Shell are present
3. Retry with a slightly longer timeout
4. Escalate to `stealthy-fetch` only if the site behavior justifies it
## Command Patterns
### Diagnose and smoke test a URL
```bash
python3 scripts/diagnose_scrapling.py --url 'https://example.com'
```
### Diagnose and smoke test a WeChat article body
```bash
python3 scripts/diagnose_scrapling.py \
--url 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' \
--selector '#js_content' \
--no-verify
```
### Diagnose and smoke test a browser-backed fetch
```bash
python3 scripts/diagnose_scrapling.py \
--url 'https://example.com' \
--dynamic
```
## Guardrails
- Do not tell the user to reinstall blindly. Verify first.
- Do not default to the Python library API when the user is clearly asking about the CLI.
- Do not jump to browser-backed fetching unless the static result is missing the real content.
- Do not claim success from exit code alone. Inspect the saved file.
- Do not hardcode user-specific absolute paths into outputs or docs.
## Resources
- Installation and smoke test helper: `scripts/diagnose_scrapling.py`
- Verified failure modes and recovery paths: `references/troubleshooting.md`
章节与文件索引
按说明文件中的顺序排列。
概述
默认工作流程
第 1 步:诊断安装
如果 CLI 是在没有额外安装的情况下安装的
如果需要浏览器支持的获取器
文件/命令 -> scrapling --help · click · scrapling[all] · references/troubleshooting.md · scrapling install · extract get · extract fetch · extract stealthy-fetch
内容校验值 -> 1f7f8190cd0f
当前原文的自动摘译,尚未人工审校。
概述
通过其 CLI 使用 Scrapling 作为默认路径。从最小的工作命令开始,验证保存的输出,并且仅在静态获取不包含真实页面内容时升级到浏览器支持的获取。不要假设用户的 Scrapling 安装是健康的。先验证一下。
默认工作流程
复制此清单并在工作时保持更新:
如果 CLI 是在没有额外安装的情况下安装的
如果
scrapling --help失败并缺少click或出现有关使用附加功能安装 Scrapling 的消息,请使用 CLI 附加功能重新安装它:除非用户明确需要更广泛的功能集,否则不要默认为scrapling[all]。如果需要浏览器支持的获取器
安装 Playwright 运行时:如果安装看起来很慢或不透明,请在猜测之前阅读
references/troubleshooting.md。在以下任一情况之前,请勿声明成功:scrapling install报告依赖项已安装,或者诊断脚本确认 Chromium 和 Chrome Headless Shell 都存在。