academic-research

Academic Research

> Nested swiss-knife reference. A modular skill: try the bundled script

> first, then load specific reference files only when you need to escape it.

Try this first

For 80% of "get me this paper" requests, the bundled script is the right answer.

It walks the open-access ladder, falls back automatically, and writes a manifest

the next session can resume from.

# Fetch by any identifier
python3 <skill-path>/scripts/fetch_paper.py 10.1103/PhysRevLett.125.015001
python3 <skill-path>/scripts/fetch_paper.py arXiv:2301.00001
python3 <skill-path>/scripts/fetch_paper.py PMID:12345678

# Batch (one identifier per line in the file)
python3 <skill-path>/scripts/fetch_paper.py --batch dois.txt --out papers/

# Resolve metadata only (no PDF download)
python3 <skill-path>/scripts/fetch_paper.py 10.1038/nature12373 --dry-run

# Skip LibGen (e.g. legal-sensitive environment)
python3 <skill-path>/scripts/fetch_paper.py <id> --no-libgen

Output layout (idempotent — re-runs skip entries with status: ok):

papers/{first-author-year-firstword}/
├── paper.pdf  |  paper.md      # full-text artifact
├── metadata.json                # CrossRef-normalized
└── manifest.json                # {status, tier, source, ts, doi}

Tier ladder (script stops at first hit):

| Tier | Source | Best for |

|------|--------|----------|

| 1 | arXiv direct | Preprints (physics, CS, math, q-bio, econ) |

| 2 | Unpaywall | Publisher-blessed gold/green OA |

| 3 | Europe PMC | Biomedical full-text + PMC mirror |

| 4 | CORE | Institutional repositories (needs $CORE_API_KEY) |

| 5 | Publisher-page extract | Nature/APS/AIP/IOP/Cambridge → in-house extractor (stdlib + requests, no third-party deps). Fetches the already-accessible official article / DOI landing page and parses citation_* metadata + the article body into structured Markdown. No paywall/CAPTCHA bypass, no cookies/credentials — official pages only. A login/paywall page is a clean miss; the ladder then falls through. Opt out with --no-publisher-extract. |

| 5b | Authorized publisher | Licensed/institutional access only — official DOI landing page → same-host publisher PDF → %PDF- validation, full provenance. Recovers paywalled-but-subscribed papers without shadow libraries. Never bypasses paywalls or handles credentials. Opt out with --no-institutional. See authorized-publisher-access.md ↗. |

| 6 | LibGen | Last resort; opt out with --no-libgen |

Set $LINGTAI_RESEARCH_EMAIL to a real address before first use — Unpaywall

rejects placeholder emails with HTTP 422. The default falls back to

[email protected] with a warning.

Read on only if: the script fails on your paper, you need a custom query shape,

or you're composing a multi-step workflow (search → fetch → cite → write).

Before drafting citation-bearing academic writing

> Verify sources before you write prose. If you are about to draft a paper,

> related-work section, literature review, References list, or any author–year /

> citation-bearing manuscript, pass the evidence-verification gate first:

> reference/evidence-verification-gate.md ↗.

> No verified evidence → no confident prose. Peer-reviewed and preprint sources

> must not share the same evidence layer; search results are leads, not citations.

> If the user asks to "go fast," reduce *scope*, not *verification*. Produce a

> verified literature matrix before any submission-like draft.

Escape hatch — quick paths

I'm about to write a paper / references / related-work section → reference/evidence-verification-gate.md (verify FIRST)
The user wants a "fast" / "readable" paper draft → reference/evidence-verification-gate.md (reduce scope, not verification)
I have a DOI                → reference/api-doi-resolver.md → api-crossref.md
I have an arXiv ID          → reference/api-arxiv.md (direct PDF link)
I have a PMID               → reference/api-europe-pmc.md
I have a bibcode            → reference/api-nasa-ads.md (requires free key)
I only have keywords        → reference/decision-tree.md → pick API by discipline
I need a citation network   → reference/api-semantic-scholar.md (single paper) or api-semantic-scholar-batch.md (large corpus) or api-openalex.md
I need to override the PDF ladder → reference/pipeline-obtain-pdf.md
Tier-5 publisher-extract failed and I want to retry it manually → reference/publisher-page-extraction.md
OA + authorized-publisher failed and I have a batch + the user's Zotero → reference/zotero-institutional-fulltext-handoff.md
All OA chains failed        → reference/libgen-fallback.md (last resort)
I need astrophysics         → reference/api-nasa-ads.md
I need high-energy physics  → reference/api-inspire-hep.md
I need biomedical           → reference/api-europe-pmc.md or api-pubmed.md
I need to write/compile a paper → reference/pipeline-latex-writing.md
My empirical draft keeps getting reframed / reviewers "agree" → reference/anti-pattern-text-consistency-vs-data-correspondence.md
I hit an API error          → reference/error-handling.md

Reference index

API references (13)

Each card includes endpoint parameters, runnable code, response shape, rate limits, and fallbacks.

| API | File | Best for | Key? |

|-----|------|----------|------|

| arXiv | api-arxiv.md ↗ | Preprint retrieval | No |

| CrossRef | api-crossref.md ↗ | DOI metadata, funder queries | No (mailto recommended) |

| DOI Resolver | api-doi-resolver.md ↗ | Batch DOI → structured citation | No |

| OpenAlex | api-openalex.md ↗ | Discovery, institution/concept analysis | No |

| Semantic Scholar | api-semantic-scholar.md ↗ | Citation networks, TLDR | No (tight limits) |

| Semantic Scholar Batch | api-semantic-scholar-batch.md ↗ | Citation-graph construction from large ID lists | No (tight limits) |

| CORE | api-core.md ↗ | OA full-text downloads | Optional (recommended) |

| PubMed | api-pubmed.md ↗ | Biomedical search, PMC full text | No |

| Unpaywall | api-unpaywall.md ↗ | OA versions / PDFs | email (real) |

| Google Scholar | api-google-scholar.md ↗ | Broadest discipline coverage | No (needs stealth) |

| Europe PMC | api-europe-pmc.md ↗ | Biomed, PMID, full-text XML | No |

| NASA ADS | api-nasa-ads.md ↗ | Astrophysics, BibTeX export | Yes (free) |

| INSPIRE-HEP | api-inspire-hep.md ↗ | High-energy physics | No |

Pipeline workflows (6)

| Pipeline | File | Purpose |

|----------|------|---------|

| Paper discovery | pipeline-discovery.md ↗ | Keywords → candidate papers |

| PDF acquisition | pipeline-obtain-pdf.md ↗ | Metadata → full text (manual ladder) |

| Citation tracking | pipeline-citation-tracking.md ↗ | Forward/backward citation networks |

| Scholar analysis | pipeline-scholar-analysis.md ↗ | Impact, trends, h-index |

| LaTeX writing | pipeline-latex-writing.md ↗ | Compile, bibliography, figures, debug |

| Decision tree | decision-tree.md ↗ | "I have X — which API should I use?" |

Standalone references

• evidence-verification-gate.md ↗ — verify-before-drafting gate for citation-bearing academic writing: detection triggers, the verified-literature-matrix schema, A/B/C evidence tiers, "go fast = reduce scope not verification," matrix-before-draft artifact convention, pre-draft lint
• publisher-page-extraction.md ↗ — Tier-5 manual escape hatch (Nature/APS/AIP/IOP/Cambridge → structured Markdown)
• authorized-publisher-access.md ↗ — Tier-5b automated probe: official DOI landing page → same-host publisher PDF, licensed access only, no paywall/credential handling
• zotero-institutional-fulltext-handoff.md ↗ — Tier-6a human-in-the-loop: stage a failed batch into Zotero with tags, the human runs Find Full Text (institutional access), agent harvests PDFs back with provenance. No UI automation / TCC bypass / credential handling.
• libgen-fallback.md ↗ — Last-resort PDF source with legal/safety notes
• error-handling.md ↗ — 429 backoff, 403 publisher blocks, timeout patterns
• anti-pattern-text-consistency-vs-data-correspondence.md ↗ — empirical-writing failure mode: prose drifts from the data while reviewer rounds make it more polished. Trigger pattern, re-anchoring steps, detection checklist.

Relationship to built-in web search

• web-search-manual (returned by web_search(action="manual")) is the routing layer ("which tier to use for this URL?")
• academic-research is the deep-dive layer ("how do I write OpenAlex filter parameters? what email does Unpaywall want?")
• The two are complementary. If you're just scraping one publisher page and don't need the OA ladder, the built-in manual's scripts/extract_page.py is lighter.

Known caveats

• Unpaywall's email parameter is required and must be real — placeholder addresses get HTTP 422. Set $LINGTAI_RESEARCH_EMAIL once.
• CORE without an API key is harshly rate-limited (~100/day vs 10,000/day with a free key from https://core.ac.uk/services/api).
• Semantic Scholar free tier is very tight (~100 req/day/IP, ~1 req/s practical). For large-corpus citation graphs use the batch endpoint — api-semantic-scholar-batch.md ↗. Request a key for any serious citation-network work.
• Google Scholar requires a stealth browser (camoufox or playwright-stealth v2); legacy playwright_stealth API does not work.
• arXiv enforces HTTPS — HTTP requests are 301-redirected automatically.
• Library Genesis legality varies by jurisdiction — use is the user's responsibility. Pass --no-libgen to opt out.
• Publisher-page extraction (Tier 5) — in-house stdlib+requests extractor (nothing to install; replaced the broken zhiping0913/Download_paper, issue #136). Fetches the already-accessible official landing page → structured Markdown. No paywall/CAPTCHA bypass, no cookie/credential handling — official pages only; login/paywall/unsupported-prefix/too-short is a clean miss. Heuristic (equations/figures may be lost). --no-publisher-extract to skip. Full contract: reference/publisher-page-extraction.md ↗.
• Authorized-publisher tier (5b) — uses only access you already have: official DOI landing → same-host PDF, %PDF-+Content-Type validated. Never bypasses paywalls/CAPTCHAs, never reads/stores/replays cookies/credentials (cookies/auth never written to provenance). Most institutional access is IP-based, so a plain GET works on a licensed network and harmlessly misses off it. --no-institutional to disable. Full contract: reference/authorized-publisher-access.md ↗.
• Zotero institutional full-text (Tier 6a) is a human-in-the-loop handoff, not an agent tool — for a batch where OA+5b both missed and the user runs Zotero Desktop on an institutional network: the agent stages rows via /connector/saveItems with a dated tag, the human clicks Find Full Text, the agent harvests PDFs back with provenance (copy-not-move, resolved_by: human Find Full Text). Never drive the UI via AppleScript/Accessibility (macOS TCC -1719/-1743, out of policy); no paywall bypass, no credential handling, no auto-LibGen. Full contract: reference/zotero-institutional-fulltext-handoff.md ↗.
• Drafting citation-bearing academic writing without verifying sources first — under "fast"/"readable draft" pressure the model flattens DOI-verified papers, preprints, and search leads into one confident evidence layer, which reads as submission-ready and invites reviewer rejection. Pass the evidence-verification gate ↗ before drafting: verify a literature matrix first, keep A/B/C reliability tiers separate in the prose, and reduce scope (not verification) when speed is requested.
• Writing an empirical paper iteratively can drift from the data — reviewer rounds make prose more polished and internally consistent without verifying it still matches the experiments on disk. Reviewer agreement is text-consistency evidence, not data-correspondence evidence. Anchor every claim to data files/runner code *before* writing, and re-derive (don't just rewrite) when feedback flags confusion. See reference/anti-pattern-text-consistency-vs-data-correspondence.md ↗.

---

> Found a bug or issue? If you encounter any problems with this skill, load the lingtai-issue-report skill and follow its instructions to report it.