Processing and raw access#
Every dataset fetcher takes a process keyword that controls how the downloaded file becomes a DataFrame.
process=True (default) — parquet-cached, processed DataFrame#
The fetcher runs a per-dataset _processor(path) closure that knows the right index_col, header, skiprows, etc. for that table. The resulting DataFrame is cached as a parquet file next to the raw download, so subsequent calls are near-instant and the parquet round-trips MultiIndex headers and dtypes losslessly.
import extracts
df = extracts.fetch_hawkins2017("table1")
df.head()
| 2007 norms | Study 1 Time 1 | Study 1 Time 2 | Amazon Mechanical Turk | TOWER | Ave. recent dream | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | SD | Mean | SD | Mean | SD | Mean | SD | Mean | SD | Mean | SD | |
| Group | ||||||||||||
| Word Count | 13580.00 | 12203.00 | 147.53 | 100.11 | 141.23 | 89.50 | 117.83 | 56.38 | 59.42 | 51.06 | 109.26 | 69.18 |
| Words per Sentence | 24.79 | 67.42 | 19.02 | 8.32 | 18.95 | 9.77 | 18.18 | 11.71 | 19.95 | 16.66 | 19.05 | 12.23 |
| Dic | 82.42 | 4.92 | 91.00 | 4.41 | 91.66 | 4.45 | 92.07 | 4.79 | 91.71 | 10.40 | 91.59 | 6.53 |
| Sixltr | 16.10 | 3.71 | 13.55 | 4.17 | 13.21 | 4.19 | 13.34 | 4.05 | 14.39 | 7.73 | 13.76 | 3.93 |
| funct | 54.85 | 4.99 | 61.20 | 4.60 | 61.54 | 4.47 | 62.07 | 4.36 | 62.21 | 9.28 | 61.83 | 6.08 |
# MultiIndex columns are preserved through the parquet cache
df.columns.nlevels
2
process=False — raw pandas.read_table with your kwargs#
When you need different parsing options, set process=False and pass your own kwargs through:
df = extracts.fetch_barrett2020("table1", process=False, index_col=0)
df.head()
| Pandemic M | Pandemic SD | Normative M | Normative SD | t | p | |
|---|---|---|---|---|---|---|
| LIWC category and content examples | ||||||
| Positive emotions: love, nice, sweet | 1.11 | 1.82 | 1.48 | 1.52 | 4.64 | <.0001*** |
| Negative emotions: hurt, ugly, nasty | 2.31 | 3.32 | 1.40 | 1.47 | 9.14 | <.0001*** |
| Anxiety: worried, fearful, nervous | 0.76 | 2.20 | 0.46 | 0.74 | 5.05 | <.0001*** |
| Anger: hate, furious, annoyed | 0.42 | 1.32 | 0.31 | 0.64 | 2.66 | .0078** |
| Sadness: crying, grief, sad | 0.46 | 1.37 | 0.27 | 0.63 | 4.55 | <.0001*** |
The processor and parquet cache are bypassed; **kwargs are forwarded directly to pandas.read_table.
Just the file path#
If you only need the location of the downloaded raw file (e.g. to hand off to another tool), call fetch_path:
extracts.fetch_path("barrett2020", "table1")
PosixPath('/home/runner/.cache/extracts/barrett2020/table1.tsv')
fetch_path triggers the download if needed, but never parses or caches a processed version.
Special fetchers#
fetch_text(dataset) and fetch_reference(dataset) follow the same pattern:
Default (
process=True): return the parsed content (a string for text, a dict for the BibTeX reference).process=False: return the localPathto the raw file.
entry = extracts.fetch_reference("barrett2020")
entry["fields"].get("title")
Downloading file 'reference.bib' from 'doi:10.5281/zenodo.11300322/reference.bib' to '/home/runner/.cache/extracts/barrett2020'.
'Dreams about COVID-19 versus normative dreams: Trends by gender'