paper-fulltext-harvest

Name: paper-fulltext-harvest
Author: jxtse

Data Community

Repo stars 44
Author updated Live
Author repo scientific-research-skills

Fluxly profile Facts only: domain, agents, trust score, runtime, permissions and network

Domain: Data
Compatible agents: Claude Code
Cursor
Cline
Codex
Windsurf
Gemini CLI

+20
Trust score: 88 / 100 · community maintained
Author / version / license: @jxtse · no license declared
Token usage: Lean
Setup complexity: Manual integration
External API key: Required · GitHub
Operating systems: Unspecified (assume cross-platform)
Runtime requirements: Python
Permissions: Read-only
Write / modify
Shell exec
Env read
Network behavior: External requests
Install commands: 26 variants

Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.

Heads up: 未限定 allowed-tools，默认拥有全部工具权限。

Output preview paper-fulltext-harvest.preview

---
name: paper-fulltext-harvest
description: Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources…
category: data
runtime: Python
---

# paper-fulltext-harvest output preview

## PART A: Task fit
- Use case: Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources that exist in 2026: The publisher router (autopaperdownload/publishers.py) recognises **25 DOI requires GitHub API key; runs on Python. Works with Claude Code, Cursor, Cline and 23 more..
- Inputs: target material, constraints, expected output, and acceptance criteria.
- Evidence boundary: follow “Decision tree / §0. Quick start (orchestrated CLI) / §1. Publisher TDM APIs” and do not present inference as author intent.

## PART B: Execution result
- **01** The card summarizes the use case; runtime output centers on “Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources that exist in 2026: The publisher router (autopaperdownload/publishers.py) recognises **25 DOI requires GitHub API key; runs on Python. Works with Claude Code, Cursor, Cline and 23 more.”.
- **02** When the source has headings, the agent prioritizes “Decision tree / §0. Quick start (orchestrated CLI) / §1. Publisher TDM APIs” so the result follows the author’s structure.
- **03** Typical output includes task judgment, concrete steps, required commands or file edits, validation, and follow-up options.
- **04** Risk context follows the fingerprint: read files, write/modify files, run shell commands, read environment variables; may access external network resources; requires GitHub API keys.

## Running Rules
- read files, write/modify files, run shell commands, read environment variables; may access external network resources; requires GitHub API keys.
- Validate with a small sample before expanding scope.
- Return the result, validation criteria, and next iteration options.

SKILL.md SKILL.md

---
name: paper-fulltext-harvest
description: Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources…
category: data
source: jxtse/scientific-research-skills
---

# paper-fulltext-harvest

## When to use
- Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources that exist in 2026: T…
- Use it when the task has clear inputs, repeatable steps, and validation criteria.

## What to provide
- Target material, scope, expected result, and forbidden changes.
- Whether network, commands, file writes, or external services are allowed.

## Execution rules
- Organize steps around “Decision tree / §0. Quick start (orchestrated CLI) / §1. Publisher TDM APIs” and keep inference separate from source facts.
- read files, write/modify files, run shell commands, read environment variables; may access external network resources; requires GitHub API keys.
- Validate with a small sample before expanding the task.

## Output requirements
- Return the deliverable, key evidence, validation method, and next action.
- Mark missing information as unknown; do not invent commands, platforms, or dependencies.

Evidence boundary and execution chain

The author source anchors workflow facts; repository files anchor sources and commands; Fluxly only adds fit, limitations, and quality judgment.

skill "paper-fulltext-harvest" {
  input      -> user goal + target files + boundaries + acceptance criteria
  context    -> Decision tree / §0. Quick start (orchestrated CLI) / §1. Publisher TDM APIs
  rules      -> SKILL.md triggers / order / output contract
  runtime    -> Python | read files, write/modify files, run shell commands, read environment variables | may access external network resources
  guardrails -> requires GitHub API keys + small-sample validation + diff/log review
  output     -> copyable result + checklist + next iteration
}

Interpretation is structured for decision-making; original keeps the upstream SKILL.md unchanged.

Decide Fit First

Core job: Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources that exist in 2026: The publish…
Best fit: Use it when the task has reusable inputs, steps, and validation criteria rather than a one-off answer.
Avoid forcing it: If the source lacks commands, platform support, or external-service evidence, keep those fields unknown instead of guessing.

Design Intent

Structure: The skill is organized around “Decision tree”, “§0. Quick start (orchestrated CLI)”, “§1. Publisher TDM APIs”, “§2. OA fallback (Unpaywall / OpenAlex / Crossref)”, showing how the author expects the agent to judge fit, collect context, and produce verifiable output.
Trigger evidence: Prioritize the author’s wording around when to use it, what context to collect, and what output shape to produce.
Evidence boundary: Author text states facts, repository files prove commands and paths, and Fluxly only adds fit, limits, and usage judgment.

How To Use It

Inputs: Provide target material, scope, expected result, forbidden changes, and validation method.
Invocation: Name paper-fulltext-harvest directly; if the source includes slash commands, start with the command and then add task context.
Validation: Start small and check whether the result follows “Decision tree / §0. Quick start (orchestrated CLI) / §1. Publisher TDM APIs” before expanding.

Boundaries And Review

Dependencies: Prepare GitHub API keys before running a full task.
Permissions: Declared permissions include read / write / shell-exec / env-read; ask the agent to state file, command, and rollback boundaries before acting.
Quality bar: A useful result names the deliverable, evidence, and next action. Generic prose means the task needs tighter context.

Paper Full-text Harvest

Pipeline for downloading academic paper full-text at scale. Handles the three classes of sources that exist in 2026:

Publisher TDM APIs (Elsevier / Wiley / Springer) — for paywalled content where the institution has a subscription
OA aggregators (Unpaywall / OpenAlex / Crossref) — for Open Access copies regardless of publisher
Browser fallback (logged-in user profile) — for paywalled publishers without a TDM API (ACS / RSC / IEEE / AIP / IOP / APS / T&F / many CN journals)

The publisher router (auto_paper_download/publishers.py) recognises 25 DOI prefixes across 19 families, each annotated with the right downstream path (TDM client / OA aggregator / browser fallback) and a support tier. The router is shared with the standalone auto-paper-harvester CLI — see SUPPORTED_PUBLISHERS.md there for the full per-publisher table.

Decision tree

Have a DOI list?
├── DOIs from Elsevier (10.1016, 10.1006, 10.1011)
│   └── Use ElsevierClient (TDM XML API)             → §1
├── DOIs from Wiley (10.1002, 10.1111)
│   └── Use WileyClient (TDM PDF API)                → §1
├── DOIs from Springer/Nature (10.1007, 10.1038, 10.1186, 10.1147)
│   ├── OA papers → SpringerClient OA API            → §1
│   └── Subscription papers → fall through to OA/browser
├── Browser-only publishers without TDM API
│   (10.1021 ACS, 10.1039 RSC, 10.1126 Science, 10.1109 IEEE,
│    10.1063 AIP, 10.1088/10.1143 IOP, 10.1103 APS, 10.1146 Annual Reviews,
│    10.1080 T&F, 10.1116 AVS, 10.1149 ECS, 10.1364 Optica, 10.3938 KPS)
│   ├── Try OA first via Unpaywall/OpenAlex          → §2
│   └── Last resort: browser fallback                → §3
├── OA-leaning publishers (10.1073 PNAS, 10.3762 Beilstein)
│   └── OpenAlex/Unpaywall usually works             → §2
└── Mixed list (typical case)
    └── Use the orchestrated CLI (handles all of the above) → §0

§0. Quick start (orchestrated CLI)

For a typical mixed list of DOIs from Web of Science / Scopus export:

# Setup once
cp scripts/.env.example .env
# Edit .env to fill API keys (see §4 "Configuration")

# Run
python -m auto_paper_download \
    --savedrecs your_export.xls \
    --output-dir ./downloads/ \
    --delay 2.0

The CLI:

Parses DOIs from WoS savedrecs (or pass multiple --savedrecs)
Routes each DOI to the right client by prefix
Handles rate limiting + retries
Per-publisher success summary at end

For resume-safe Elsevier bulk (the most common large run, e.g. 5000+ Elsevier DOIs):

python scripts/redownload_elsevier.py \
    --excel papers.xlsx \
    --output-dir ./elsevier_xml/ \
    --resume \
    --long-pause-every 200 \
    --long-pause-sec 300

§1. Publisher TDM APIs

Read references/tdm-apis.md for full per-publisher details.

Quick reference:

Publisher	API	Auth env var	Output	Rate limit
Elsevier	`api.elsevier.com/content/article/doi/{DOI}?view=FULL`	`ELSEVIER_API_KEY` + `ELSEVIER_INSTTOKEN`	XML (full-text)	~5 req/sec
Wiley	`api.wiley.com/onlinelibrary/tdm/v1/articles/{DOI}`	`WILEY_TDM_TOKEN`	PDF	3 req/sec hard cap
Springer (OA)	`api.springernature.com/openaccess/json`	`SPRINGER_API_KEY`	JSON+text	1 req/sec free
Crossref TDM	URL from `link[]` field with `intended-application: text-mining`	`CR_CLICKTHROUGH_TOKEN`	varies	varies

Critical: All TDM APIs require institutional IP allowlisting — must run from the institution's network or VPN. Test with one DOI before bulk runs.

Instantiate clients directly:

from auto_paper_download.clients import ElsevierClient, WileyClient

elsevier = ElsevierClient()  # reads env vars
xml_path = elsevier.download_structured_full_text(
    doi="10.1016/j.ces.2025.123003",
    article_dir=Path("downloads/10.1016_j.ces.2025.123003"),
)

wiley = WileyClient()
pdf_path = wiley.download_pdf(
    doi="10.1002/anie.202500001",
    article_dir=Path("downloads/10.1002_anie.202500001"),
)

§2. OA fallback (Unpaywall / OpenAlex / Crossref)

For papers that may have OA copies regardless of publisher.

from auto_paper_download.clients import UnpaywallClient, OpenAlexClient, CrossrefClient

# Unpaywall: best OA PDF URL
up = UnpaywallClient()
pdf_path = up.download_pdf(doi=doi, article_dir=Path("downloads/.."))

# OpenAlex: alternative OA source
oa = OpenAlexClient()
pdf_path = oa.download_pdf(doi=doi, article_dir=Path("downloads/.."))

# Crossref: tries to find publisher PDF link
cr = CrossrefClient()
pdf_path = cr.download_pdf(doi=doi, article_dir=Path("downloads/.."))

Always validate downloaded PDFs: First 4 bytes must be %PDF and file size > 50KB. The clients in this skill do this automatically.

Expected hit rate for OA fallback: 40-60% on a generic chemistry/biology list. Recent papers (>2023) have higher OA rates.

§3. Browser fallback (paywalled, no TDM)

For publishers where API isn't available but the user has institutional Cloudflare/SSO access via browser cookies. Slowest path — only use after exhausting §1–§2.

Two routes — pick one

	Route A: OpenClaw `browser` tool	Route B: `auto-paper-harvester` v0.2+ CLI
What	Drive the user's running Chrome via the agent's `browser` capability with `profile="user"`	Standalone CLI with built-in Playwright `launch_persistent_context`
Setup	None — reuses whatever Chrome the user is logged into	`pip install 'auto-paper-download[browser]' && playwright install chromium`
Cookies	User's existing daily-driver Chrome cookies (zero re-login)	Dedicated isolated profile; user logs into SSO once on first run
Selectors	Per-publisher CSS in `references/browser-fallback.md` (ACS / Wiley / RSC / T&F / Nature / AIP / CN journals)	Per-family selectors baked into `browser_fallback.py` (14 publisher families)
Best for	Agent workflows where the user is actively at the keyboard, fewer DOIs (< 100), or one-off rescue runs	Unattended bulk runs (1000+ DOIs), CI/headless servers, when you don't want to lock the user's Chrome
Cost	Ties up user's Chrome for ~5 s/paper	Spawns its own Chromium; user's browser stays free
Surface area	Lives in this skill (`references/browser-fallback.md` + `browser` tool)	Lives in the `auto-paper-harvester` repo (separate install)

Decision rule:

Default to Route A inside this skill (zero install, leverages session the user already has).
Recommend Route B when the run is large (> 500 DOIs), runs unattended, or the user's Chrome shouldn't be locked. Both routes feed into the same downstream validation (PDF magic bytes, file size).

Route A details — OpenClaw `browser` tool

Read references/browser-fallback.md before starting. It covers:

How to drive the user's logged-in Chrome via OpenClaw browser tool with profile="user"
Per-publisher CSS selectors for ACS, Wiley, RSC, T&F, Springer, Nature, AIP, and 3 major Chinese journals
Cloudflare detection + retry strategy
Single-tab reuse pattern (don't open a new tab per DOI — leaks)
Kill-switch via /tmp/stop_scrape

Route B details — auto-paper-harvester CLI

# One-time install (separate from this skill)
git clone https://github.com/jxtse/auto-paper-harvester.git
cd auto-paper-harvester
pip install -e '.[browser]' && playwright install chromium

# Run with same DOI file you'd otherwise feed to this skill
python -m auto_paper_download --savedrecs your_export.xls --use-browser-fallback

It routes every DOI through the same publisher TDM → OA → browser chain as this skill, with the browser pass running automatically against any DOI the API pipeline failed. See its README and SKILL.md for full docs.

Hard reality (applies to both routes)

ACS / Wiley / T&F use Cloudflare. Even with a logged-in profile, expect:

~30% Cloudflare challenges (retry after 10 s usually clears)
Some sites detect headless and hard-block — a real Chrome with an active session (Route A) is more robust than fresh Chromium (Route B) in those cases
Throughput: ~5 sec/paper, ~70-90% success

§4. Configuration

Required env vars (set in .env, see scripts/.env.example):

Variable	Required for	How to get
`ELSEVIER_API_KEY`	Elsevier	https://dev.elsevier.com/ (free key)
`ELSEVIER_INSTTOKEN`	Elsevier institutional access	Contact your library
`WILEY_TDM_TOKEN`	Wiley	https://onlinelibrary.wiley.com/library-info/resources/text-and-datamining (institution must sign TDM agreement)
`SPRINGER_API_KEY`	Springer OA	https://dev.springernature.com/ (free key)
`CROSSREF_MAILTO`	Crossref polite pool (recommended)	Just your email
`OPENALEX_MAILTO`	OpenAlex polite pool (recommended)	Just your email
`UNPAYWALL_EMAIL`	Unpaywall (required)	Just your email

Notes:

All env vars are optional — missing ones simply disable that source
CROSSREF_REQUEST_DELAY / WILEY_REQUEST_DELAY allow tuning per-source delay

Core principles

Cache directory structure: each DOI gets its own folder named <safe_doi>/ (with / replaced by _). This makes resume trivial — check if folder exists with non-empty file.
Cascade sources, cheapest first: TDM API for known publisher → OA aggregator → browser. Each fallback is more expensive (rate, time, fragility).
Respect rate limits: defaults are conservative (--delay 2.0). For long runs use --long-pause-every and --long-pause-sec to avoid cumulative ban.
Don't trust HTTP 200: many publisher APIs return 200 with HTML "subscribe to read" page. Validate content (PDF magic bytes, XML body markers like <ce:para>).
Validate before declaring done: spot-check 5 random files manually before reporting success.

Common pitfalls

Pitfall	Symptom	Fix
Empty PDF/XML directories created on failure	"Downloaded N papers" but files are 0 bytes	Validate file size; remove empty dirs (this code does it via `_cleanup_article_dir`)
Cloudflare blocks headless Playwright	403 / "Just a moment..."	Use OpenClaw `browser` with `profile="user"`, not headless
Rate-limited mid-batch	429s, then permanent block	Increase `--delay`, set `--long-pause-every 200`, respect `Retry-After`
Springer subscription returns HTML "subscribe"	Saved 0-byte or junk PDF	Code checks `%PDF` magic bytes — use the SpringerClient, don't bypass
DOI case sensitivity	Some publishers 404 on uppercase	Code normalizes; if writing your own, always `.lower()`
`.abs` suffix on Crossref DOIs	404 from Crossref	Strip `.abs` before query

When to ask the user

Before running >1000 publisher API requests (institution may have weekly quota)
Before browser scrape loop (will tie up their Chrome for ~5 sec/paper)
When >30% of fetches fail unexpectedly (network / auth problem — investigate before continuing)
When you detect a publisher with no API + no OA — confirm whether to skip or try browser

File layout

paper-fulltext-harvest/
├── SKILL.md                       (this file)
├── references/
│   ├── tdm-apis.md                Per-publisher TDM API details
│   └── browser-fallback.md        Browser scraping guide for paywalled non-TDM publishers (Route A)
└── scripts/
    ├── .env.example               Template for API keys
    ├── pyproject.toml             Dependencies (pip/uv installable)
    ├── redownload_elsevier.py     Resume-safe Elsevier bulk downloader
    └── auto_paper_download/       Main package
        ├── __init__.py
        ├── __main__.py            CLI entrypoint
        ├── publishers.py          25 DOI prefixes → publisher family + handler + support tier (shared with auto-paper-harvester)
        ├── clients.py             ElsevierClient, WileyClient, SpringerClient, CrossrefClient, UnpaywallClient, OpenAlexClient
        ├── downloader.py          Orchestration: parse savedrecs, route by publisher, batch download
        └── supplements.py         Supplementary file downloader

Install command

One-click installPick a runner, click Copy, paste into your terminal, press enter

npx skills add jxtse/scientific-research-skills/skills/paper-fulltext-harvest -y

Open general install guide

View skill source

Works with

Claude Code
Cursor
Cline
Codex
Windsurf
Gemini
OpenCode
Roo Code
AAide
+Augment
GitHub Copilot
OpenClaw
KKiro
Junie
WWarp
Goose
Trae
Kilo Code
OpenHands
Amp
Zencoder
CContinue
AIAider
CodeBuddy
CodeGeeX
Replit Agent

Security audit

88 / 100

Community

未声明 license/许可。
未声明 version。
未限定 allowed-tools，默认拥有全部工具权限。

Metadata

Source: github
jxtse/scientific-research-skills
中文

Category & tags

Category Data