paper fulltext harvest

Data Community

Skill notes

§0. Quick start (orchestrated CLI)

For a typical mixed list of DOIs from Web of Science / Scopus export: The CLI: Parses DOIs from WoS savedrecs (or pass multiple --savedrecs) Routes each DOI to the right client by prefix Handles rate limiting + retries

§1. Publisher TDM APIs

Read references/tdm-apis.md for full per-publisher details. Quick reference: Publisher · API · Auth env var · Output · Rate limit Elsevier · api.elsevier.com/content/article/doi/{DOI}?view=FULL · ELSEVIER_API_KEY + ELSEVIER_INSTTOKEN · XML (full-text) · ~5 req/sec Wiley · api.wiley.com/onlinelibrary/tdm/v1/articles/{DOI} · WILEY_TDM_TOKEN · PDF · 3 req/sec hard cap

§2. OA fallback (Unpaywall / OpenAlex / Crossref)

For papers that may have OA copies regardless of publisher. Always validate downloaded PDFs: First 4 bytes must be %PDF and file size > 50KB. The clients in this skill do this automatically. Expected hit rate for OA fallback: 40-60% on a generic chemistry/biology list. Recent papers (>2023) have higher OA rates.

§3. Browser fallback (paywalled, no TDM)

For publishers where API isn't available but the user has institutional Cloudflare/SSO access via browser cookies. Slowest path — only use after exhausting §1–§2.

Two routes — pick one

Route A: OpenClaw browser tool · Route B: auto-paper-harvester v0.2+ CLI What · Drive the user's running Chrome via the agent's browser capability with profile="user" · Standalone CLI with built-in Playwright launch_persistent_context Setup · None — reuses whatever Chrome the user is logged into · pip install 'auto-paper-download[browser]' && playwright install chromium Cookies · User's existing daily-driver Chrome cookies (zero re-login) · Dedicated isolated profile; user logs into SSO once on first run

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Data
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@jxtse · no license declared
Token usage
Lean
Setup complexity
Manual integration
External API key
Required · GitHub
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
Python
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
  • Env read
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview paper-fulltext-harvest.preview
# §0. Quick start (orchestrated CLI)

# Setup once
cp scripts/.env.example .env
# Edit .env to fill API keys (see §4 "Configuration")

# Run
python -m auto_paper_download \
    --savedrecs your_export.xls \
    --output-dir ./downloads/ \
    --delay 2.0

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.