hurovitz1999¶

This dataset comes from DreamBank. It merges the Blind dreamers (F) and Blind dreamers (M) datasets from an archived copy of DreamBank hosted on Zenodo. See the krank GitHub Issue and the hurovitz1999 docs page for more info.

Setup¶

In [1]:

import hashlib
import os
import re
import sys
import textwrap
from copy import copy
from datetime import datetime, timezone

import pandas as pd

from IPython.display import display, IFrame

sys.path.append(os.path.abspath(".."))
import utils_dreambank

import hashlib import os import re import sys import textwrap from copy import copy from datetime import datetime, timezone import pandas as pd from IPython.display import display, IFrame sys.path.append(os.path.abspath("..")) import utils_dreambank

Identify the DreamBank dataset IDs of the datasets that will be included.

In [2]:

DATASETS_IDS = ["blind-f", "blind-m"]

DATASETS_IDS = ["blind-f", "blind-m"]

Load¶

Use utility functions to load in all 3 components (dreams, info, moreinfo) from each dataset.

In [3]:

blind_f_dreams = utils_dreambank.parse_dreams("v2", "blind-f", as_dataframe=True)
blind_m_dreams = utils_dreambank.parse_dreams("v2", "blind-m", as_dataframe=True)

blind_f_info = utils_dreambank.parse_info("v2", "blind-f")
blind_m_info = utils_dreambank.parse_info("v2", "blind-m")

blind_f_moreinfo = utils_dreambank.parse_moreinfo("v2", "blind-f")
blind_m_moreinfo = utils_dreambank.parse_moreinfo("v2", "blind-m")

blind_f_dreams = utils_dreambank.parse_dreams("v2", "blind-f", as_dataframe=True) blind_m_dreams = utils_dreambank.parse_dreams("v2", "blind-m", as_dataframe=True) blind_f_info = utils_dreambank.parse_info("v2", "blind-f") blind_m_info = utils_dreambank.parse_info("v2", "blind-m") blind_f_moreinfo = utils_dreambank.parse_moreinfo("v2", "blind-f") blind_m_moreinfo = utils_dreambank.parse_moreinfo("v2", "blind-m")

Snapshot¶

Take a look at the raw source data.

In [4]:

display(blind_f_dreams.head(n=10))

display(blind_f_dreams.head(n=10))

	dataset_id	dream_id	metadata	word_count	dream_text
0	blind-f	101-01	F-A-T-19/27	231	I was at my parent's house that I grew up in a...
1	blind-f	101-02	F-A-T-19/27	253	I was back in the house that I grew up in and ...
2	blind-f	101-03	F-A-T-19/27	207	Right before waking up I remember standing in ...
3	blind-f	101-04	F-A-T-19/27	358	Last night I dreamed that I went back to schoo...
4	blind-f	101-05	F-A-T-19/27	167	There was a book published, like a book of ess...
5	blind-f	101-06	F-A-T-19/27	672	I had a weird sort of dream, I was stuck somew...
6	blind-f	101-07	F-A-T-19/27	269	[1997-05-08] I was in my parent's old house an...
7	blind-f	101-08	F-A-T-19/27	279	[1997-05-09] I had a dream I was in a mall and...
8	blind-f	101-09	F-A-T-19/27	422	I was in my old high school and I was sitting ...
9	blind-f	101-10	F-A-T-19/27	352	I was with my best friend, X and we were prepa...

In [5]:

display(blind_m_dreams.head(n=10))

display(blind_m_dreams.head(n=10))

	dataset_id	dream_id	metadata	word_count	dream_text
0	blind-m	104-01	M-C-T-45/45	69	[1997-05-18] Dreamed I was in my computer room...
1	blind-m	104-02	M-C-T-45/45	118	[1997-05-18] I dreamed I was riding on a bus, ...
2	blind-m	104-03	M-C-T-45/45	113	[1997-05-19] I was doing the dinner dishes, my...
3	blind-m	104-04	M-C-T-45/45	149	I dreamed I was in a room, I think it was a li...
4	blind-m	104-05	M-C-T-45/45	161	[1997-05-22] The only dream I remember is a ve...
5	blind-m	104-06	M-C-T-45/45	289	[1997-05-24] I was coming out of the subway an...
6	blind-m	104-07	M-C-T-45/45	70	[1997-05-25] Just came home from the grocery s...
7	blind-m	104-08	M-C-T-45/45	622	[1997-05-26] We were on vacation with my cousi...
8	blind-m	104-09	M-C-T-45/45	34	[1997-05-27] A friend of my daughters came to ...
9	blind-m	104-10	M-C-T-45/45	219	[1997-05-28] I was in a mobile home, decided t...

In [6]:

display(blind_f_info)

display(blind_f_info)

{'dataset_id': 'blind-f',
 'name': 'Blind dreamers (F)',
 'timeframe': 'mid-1990s',
 'sex': 'female',
 'n_dreams': 238,
 'description': 'These dreams were collected in the mid-1990s from several men and women ranging from their 20s to their 70s, but most are from 40 to 65. Some are congenitally blind; most of the others have been blind for over 20 years, but one older woman has been blind for only a few years. (For more about this series, including a table with details about each dreamer, click here. For a published article discussing these dream series and summarizing other research on blind dreamers, click here.)'}

In [7]:

display(blind_m_info)

display(blind_m_info)

{'dataset_id': 'blind-m',
 'name': 'Blind dreamers (M)',
 'timeframe': 'mid-1990s',
 'sex': 'male',
 'n_dreams': 143,
 'description': 'These dreams were collected in the mid-1990s from several men and women ranging from their 20s to their 70s, but most are from 40 to 65. Some are congenitally blind; most of the others have been blind for over 20 years, but one older woman has been blind for only a few years. (For more about this series, including a table with details about each dreamer, click here. For a published article discussing these dream series and summarizing other research on blind dreamers, click here.)'}

In [8]:

display(blind_f_moreinfo)

display(blind_f_moreinfo)

{'dataset_id': 'blind-f',
 'name': 'Blind dreamers (F)',
 'sex': 'female',
 'timeframe': 'mid-1990s',
 'n_dreams': 238,
 'moreinfo': 'These dreams have been content analyzed for several major Hall/Van de Castle categories. They have been studied also for mentions of visual, auditory, olfactory, gustatory, and tactile sensations. No contact with the dreamers is possible.\n\n\nCode #SexAgeYears ofEducationOccupationYears Blind(Nature/Degreeof Blindness)# ofdreamreports\n101F2715Unemployed19 (A/T)10\n102F3218Unemployed32 (C/T)10\n103F7312Sales clerk (retired)10 (A/P)39\n111F5019Vocational rehabilitation counselor50 (C/L)59\n114F5212Envelope stuffer52 (C/T)37\n115F6014Cook at a school for the blind60 (A/T)6\n117F4418Factory worker (retired)44 (C/T)32\n120F4918Grant writer for Blind Services47 (A/T)10\n122F5812Unemployed46 (A/T)24\n125F4413Medical transcriptionist44 (C/T)9\n104M4516Human resources management45 (C/T)61\n105M5018Rehabilitation specialist43 (A/P)21\n109M2414.5Software tech worker20 (A/T)20\n113M4513Piano tuner/technician17 (A/T)22\n124M4612Small engine repairs46 (C/T)12\n\n\n\nC = congenital blindness; A = adventitious blindness.\nT = total blindness; P = partial blindness; L = perceives very bright light.',
 'moreinfo_available': True}

In [9]:

display(blind_m_moreinfo)

display(blind_m_moreinfo)

{'dataset_id': 'blind-m',
 'name': 'Blind dreamers (M)',
 'sex': 'male',
 'timeframe': 'mid-1990s',
 'n_dreams': 143,
 'moreinfo': 'These dreams have been content analyzed for several major Hall/Van de Castle categories. They have been studied also for mentions of visual, auditory, olfactory, gustatory, and tactile sensations. No contact with the dreamers is possible.\n\n\nCode #SexAgeYears ofEducationOccupationYears Blind(Nature/Degreeof Blindness)# ofdreamreports\n101F2715Unemployed19 (A/T)10\n102F3218Unemployed32 (C/T)10\n103F7312Sales clerk (retired)10 (A/P)39\n111F5019Vocational rehabilitation counselor50 (C/L)59\n114F5212Envelope stuffer52 (C/T)37\n115F6014Cook at a school for the blind60 (A/T)6\n117F4418Factory worker (retired)44 (C/T)32\n120F4918Grant writer for Blind Services47 (A/T)10\n122F5812Unemployed46 (A/T)24\n125F4413Medical transcriptionist44 (C/T)9\n104M4516Human resources management45 (C/T)61\n105M5018Rehabilitation specialist43 (A/P)21\n109M2414.5Software tech worker20 (A/T)20\n113M4513Piano tuner/technician17 (A/T)22\n124M4612Small engine repairs46 (C/T)12\n\n\n\nC = congenital blindness; A = adventitious blindness.\nT = total blindness; P = partial blindness; L = perceives very bright light.',
 'moreinfo_available': True}

View summary info of the dataframes.

In [10]:

blind_f_dreams.info(memory_usage="deep")

blind_f_dreams.info(memory_usage="deep")

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 238 entries, 0 to 237
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   dataset_id  238 non-null    object
 1   dream_id    238 non-null    object
 2   metadata    238 non-null    object
 3   word_count  238 non-null    int64 
 4   dream_text  238 non-null    object
dtypes: int64(1), object(4)
memory usage: 228.9 KB

In [11]:

blind_m_dreams.info(memory_usage="deep")

blind_m_dreams.info(memory_usage="deep")

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 143 entries, 0 to 142
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   dataset_id  143 non-null    object
 1   dream_id    143 non-null    object
 2   metadata    143 non-null    object
 3   word_count  143 non-null    int64 
 4   dream_text  143 non-null    object
dtypes: int64(1), object(4)
memory usage: 124.3 KB

In [12]:

display(blind_f_dreams.describe(include="number"))

display(blind_f_dreams.describe(include="number"))

	word_count
count	238.000000
mean	150.768908
std	114.980136
min	18.000000
25%	74.000000
50%	116.000000
75%	187.750000
max	782.000000

In [13]:

display(blind_f_dreams.describe(exclude="number"))

display(blind_f_dreams.describe(exclude="number"))

	dataset_id	dream_id	metadata	dream_text
count	238	238	238	238
unique	1	238	9	238
top	blind-f	101-01	F-C-L-50/50	I was at my parent's house that I grew up in a...
freq	238	1	60	1

In [14]:

display(blind_m_dreams.describe(include="number"))

display(blind_m_dreams.describe(include="number"))

	word_count
count	143.000000
mean	130.783217
std	128.247513
min	12.000000
25%	45.000000
50%	103.000000
75%	157.000000
max	898.000000

In [15]:

display(blind_m_dreams.describe(exclude="number"))

display(blind_m_dreams.describe(exclude="number"))

	dataset_id	dream_id	metadata	dream_text
count	143	143	143	143
unique	1	143	5	143
top	blind-m	104-01	M-C-T-45/45	[1997-05-18] Dreamed I was in my computer room...
freq	143	1	61	1

Process¶

Merge the two dreams dataframes.

In [16]:

dreams = pd.concat([blind_f_dreams, blind_m_dreams], ignore_index=True)

dreams = pd.concat([blind_f_dreams, blind_m_dreams], ignore_index=True)

In [17]:

dreams.info()

dreams.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 381 entries, 0 to 380
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   dataset_id  381 non-null    object
 1   dream_id    381 non-null    object
 2   metadata    381 non-null    object
 3   word_count  381 non-null    int64 
 4   dream_text  381 non-null    object
dtypes: int64(1), object(4)
memory usage: 15.0+ KB

In [18]:

display(dreams.describe(include="number"))

display(dreams.describe(include="number"))

	word_count
count	381.000000
mean	143.267717
std	120.355359
min	12.000000
25%	63.000000
50%	110.000000
75%	176.000000
max	898.000000

In [19]:

display(dreams.describe(exclude="number"))

display(dreams.describe(exclude="number"))

	dataset_id	dream_id	metadata	dream_text
count	381	381	381	381
unique	2	381	14	381
top	blind-f	101-01	M-C-T-45/45	I was at my parent's house that I grew up in a...
freq	238	1	61	1

Extract author metadata from tables in moreinfo.

The moreinfo for these datasets includes a table of extensive metadata from each participant/author.

In [20]:

assert blind_f_moreinfo["moreinfo"] == blind_m_moreinfo["moreinfo"]
moreinfo = copy(blind_f_moreinfo)
moreinfo_body = moreinfo["moreinfo"]
moreinfo_body_wrapped = "\n".join(
    [ textwrap.fill(line, width=100) for line in moreinfo_body.splitlines() ]
)
print(moreinfo_body_wrapped)

assert blind_f_moreinfo["moreinfo"] == blind_m_moreinfo["moreinfo"] moreinfo = copy(blind_f_moreinfo) moreinfo_body = moreinfo["moreinfo"] moreinfo_body_wrapped = "\n".join( [ textwrap.fill(line, width=100) for line in moreinfo_body.splitlines() ] ) print(moreinfo_body_wrapped)

These dreams have been content analyzed for several major Hall/Van de Castle categories. They have
been studied also for mentions of visual, auditory, olfactory, gustatory, and tactile sensations. No
contact with the dreamers is possible.


Code #SexAgeYears ofEducationOccupationYears Blind(Nature/Degreeof Blindness)# ofdreamreports
101F2715Unemployed19 (A/T)10
102F3218Unemployed32 (C/T)10
103F7312Sales clerk (retired)10 (A/P)39
111F5019Vocational rehabilitation counselor50 (C/L)59
114F5212Envelope stuffer52 (C/T)37
115F6014Cook at a school for the blind60 (A/T)6
117F4418Factory worker (retired)44 (C/T)32
120F4918Grant writer for Blind Services47 (A/T)10
122F5812Unemployed46 (A/T)24
125F4413Medical transcriptionist44 (C/T)9
104M4516Human resources management45 (C/T)61
105M5018Rehabilitation specialist43 (A/P)21
109M2414.5Software tech worker20 (A/T)20
113M4513Piano tuner/technician17 (A/T)22
124M4612Small engine repairs46 (C/T)12



C = congenital blindness; A = adventitious blindness.
T = total blindness; P = partial blindness; L = perceives very bright light.

View the HTML version of the tables by pulling up the current website.

In [21]:

TABLE_URL_TEMPLATE = "https://dreambank.net/more_info.cgi?further=1&series={dataset_id}"
blind_f_url = TABLE_URL_TEMPLATE.format(dataset_id="blind-f")
blind_m_url = TABLE_URL_TEMPLATE.format(dataset_id="blind-m")
IFrame(blind_f_url, width=800, height=625)

TABLE_URL_TEMPLATE = "https://dreambank.net/more_info.cgi?further=1&series={dataset_id}" blind_f_url = TABLE_URL_TEMPLATE.format(dataset_id="blind-f") blind_m_url = TABLE_URL_TEMPLATE.format(dataset_id="blind-m") IFrame(blind_f_url, width=800, height=625)

Out[21]:

In [22]:

IFrame(blind_m_url, width=800, height=625)

IFrame(blind_m_url, width=800, height=625)

Out[22]:

Extract the table from the raw HTML.

In [23]:

tables = {}
for ds in DATASETS_IDS:
    content = utils_dreambank.extract_file_content("v2", ds, "moreinfo")
    table = pd.read_html(content)[0]
    tables[ds] = table
assert tables["blind-f"].equals(tables["blind-m"])
authors = tables["blind-f"].copy()

tables = {} for ds in DATASETS_IDS: content = utils_dreambank.extract_file_content("v2", ds, "moreinfo") table = pd.read_html(content)[0] tables[ds] = table assert tables["blind-f"].equals(tables["blind-m"]) authors = tables["blind-f"].copy()

In [24]:

display(authors)

display(authors)

	Code #	Sex	Age	Years of Education	Occupation	Years Blind (Nature/Degree of Blindness)	# of dream reports
0	101	F	27	15.0	Unemployed	19 (A/T)	10
1	102	F	32	18.0	Unemployed	32 (C/T)	10
2	103	F	73	12.0	Sales clerk (retired)	10 (A/P)	39
3	111	F	50	19.0	Vocational rehabilitation counselor	50 (C/L)	59
4	114	F	52	12.0	Envelope stuffer	52 (C/T)	37
5	115	F	60	14.0	Cook at a school for the blind	60 (A/T)	6
6	117	F	44	18.0	Factory worker (retired)	44 (C/T)	32
7	120	F	49	18.0	Grant writer for Blind Services	47 (A/T)	10
8	122	F	58	12.0	Unemployed	46 (A/T)	24
9	125	F	44	13.0	Medical transcriptionist	44 (C/T)	9
10	104	M	45	16.0	Human resources management	45 (C/T)	61
11	105	M	50	18.0	Rehabilitation specialist	43 (A/P)	21
12	109	M	24	14.5	Software tech worker	20 (A/T)	20
13	113	M	45	13.0	Piano tuner/technician	17 (A/T)	22
14	124	M	46	12.0	Small engine repairs	46 (C/T)	12

Parse out the 3 different variables from the "Years Blind (Nature/Degree of Blindness)" column.

In [25]:

YEARS_BLIND_RE = r"^(?P<YearsBlind>\d{2}) \((?P<Nature>[A-Z])/(?P<DegreeOfBlindness>[A-Z])\)$"
assert authors["Years Blind (Nature/Degree of Blindness)"].str.match(YEARS_BLIND_RE).all()
years_blind, nature, degree_of_blindness = authors["Years Blind (Nature/Degree of Blindness)"].str.extract(YEARS_BLIND_RE, expand=True).T.values
authors.insert(5, "Years Blind", years_blind.astype(int))
authors.insert(6, "Nature", nature)
authors.insert(7, "Degree of Blindness", degree_of_blindness)
authors = authors.drop(columns=["Years Blind (Nature/Degree of Blindness)"])

YEARS_BLIND_RE = r"^(?P\d{2}) \((?P[A-Z])/(?P[A-Z])\)$" assert authors["Years Blind (Nature/Degree of Blindness)"].str.match(YEARS_BLIND_RE).all() years_blind, nature, degree_of_blindness = authors["Years Blind (Nature/Degree of Blindness)"].str.extract(YEARS_BLIND_RE, expand=True).T.values authors.insert(5, "Years Blind", years_blind.astype(int)) authors.insert(6, "Nature", nature) authors.insert(7, "Degree of Blindness", degree_of_blindness) authors = authors.drop(columns=["Years Blind (Nature/Degree of Blindness)"])

Separate the retired out as it's own variable.

In [26]:

OCCUPATION_RE = r"^(?P<Occupation>.+?)(?: \((?P<Retired>.+?)\))?$"
assert authors["Occupation"].str.match(OCCUPATION_RE).all()
occupation, retired = authors["Occupation"].str.extract(OCCUPATION_RE, expand=True).fillna(False).astype({"Retired": bool}).T.to_numpy()
authors["Occupation"] = occupation
authors.insert(5, "Retired", retired)

OCCUPATION_RE = r"^(?P.+?)(?: \((?P.+?)\))?$" assert authors["Occupation"].str.match(OCCUPATION_RE).all() occupation, retired = authors["Occupation"].str.extract(OCCUPATION_RE, expand=True).fillna(False).astype({"Retired": bool}).T.to_numpy() authors["Occupation"] = occupation authors.insert(5, "Retired", retired)

Sort by Author ID for clarity.

In [27]:

authors = authors.sort_values(by="Code #", ignore_index=True)

authors = authors.sort_values(by="Code #", ignore_index=True)

In [28]:

display(authors)

display(authors)

	Code #	Sex	Age	Years of Education	Occupation	Retired	Years Blind	Nature	Degree of Blindness	# of dream reports
0	101	F	27	15.0	Unemployed	False	19	A	T	10
1	102	F	32	18.0	Unemployed	False	32	C	T	10
2	103	F	73	12.0	Sales clerk	True	10	A	P	39
3	104	M	45	16.0	Human resources management	False	45	C	T	61
4	105	M	50	18.0	Rehabilitation specialist	False	43	A	P	21
5	109	M	24	14.5	Software tech worker	False	20	A	T	20
6	111	F	50	19.0	Vocational rehabilitation counselor	False	50	C	L	59
7	113	M	45	13.0	Piano tuner/technician	False	17	A	T	22
8	114	F	52	12.0	Envelope stuffer	False	52	C	T	37
9	115	F	60	14.0	Cook at a school for the blind	False	60	A	T	6
10	117	F	44	18.0	Factory worker	True	44	C	T	32
11	120	F	49	18.0	Grant writer for Blind Services	False	47	A	T	10
12	122	F	58	12.0	Unemployed	False	46	A	T	24
13	124	M	46	12.0	Small engine repairs	False	46	C	T	12
14	125	F	44	13.0	Medical transcriptionist	False	44	C	T	9

Parse the blindness legend from moreinfo and use to replace in dataframe.

In [29]:

LEGEND_RE = r"\s(?P<key>[A-Z]) = (?P<value>[a-z ]+)"
blind_f_legend = dict(re.findall(LEGEND_RE, blind_f_moreinfo["moreinfo"]))
blind_m_legend = dict(re.findall(LEGEND_RE, blind_m_moreinfo["moreinfo"]))
assert blind_f_legend == blind_m_legend
legend = copy(blind_f_legend)

LEGEND_RE = r"\s(?P[A-Z]) = (?P[a-z ]+)" blind_f_legend = dict(re.findall(LEGEND_RE, blind_f_moreinfo["moreinfo"])) blind_m_legend = dict(re.findall(LEGEND_RE, blind_m_moreinfo["moreinfo"])) assert blind_f_legend == blind_m_legend legend = copy(blind_f_legend)

In [30]:

display(legend)

display(legend)

{'C': 'congenital blindness',
 'A': 'adventitious blindness',
 'T': 'total blindness',
 'P': 'partial blindness',
 'L': 'perceives very bright light'}

In [31]:

authors["Nature"] = authors["Nature"].map(legend)
authors["Degree of Blindness"] = authors["Degree of Blindness"].map(legend)

authors["Nature"] = authors["Nature"].map(legend) authors["Degree of Blindness"] = authors["Degree of Blindness"].map(legend)

In [32]:

display(authors)

display(authors)

	Code #	Sex	Age	Years of Education	Occupation	Retired	Years Blind	Nature	Degree of Blindness	# of dream reports
0	101	F	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	10
1	102	F	32	18.0	Unemployed	False	32	congenital blindness	total blindness	10
2	103	F	73	12.0	Sales clerk	True	10	adventitious blindness	partial blindness	39
3	104	M	45	16.0	Human resources management	False	45	congenital blindness	total blindness	61
4	105	M	50	18.0	Rehabilitation specialist	False	43	adventitious blindness	partial blindness	21
5	109	M	24	14.5	Software tech worker	False	20	adventitious blindness	total blindness	20
6	111	F	50	19.0	Vocational rehabilitation counselor	False	50	congenital blindness	perceives very bright light	59
7	113	M	45	13.0	Piano tuner/technician	False	17	adventitious blindness	total blindness	22
8	114	F	52	12.0	Envelope stuffer	False	52	congenital blindness	total blindness	37
9	115	F	60	14.0	Cook at a school for the blind	False	60	adventitious blindness	total blindness	6
10	117	F	44	18.0	Factory worker	True	44	congenital blindness	total blindness	32
11	120	F	49	18.0	Grant writer for Blind Services	False	47	adventitious blindness	total blindness	10
12	122	F	58	12.0	Unemployed	False	46	adventitious blindness	total blindness	24
13	124	M	46	12.0	Small engine repairs	False	46	congenital blindness	total blindness	12
14	125	F	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	9

Clean up the dreams dataframe.

Handle "dream_id" column. Identify metadata values and parse out.

The dream_id column is just the participant/author ID and the number of their dream. We don't need the latter but can parse out the former.

NOTE: There is one row (323) where the dream number has a letter (109-3a). This might mean it's a part of the previous dream? Not sure but I will treat it as it's own dream to not make that assumption.

In [33]:

DREAM_ID_RE = r"^(?P<Author>\d{3})-\d{2}a?$"
assert dreams["dream_id"].str.match(DREAM_ID_RE).all()
author_id = dreams["dream_id"].str.extract(DREAM_ID_RE)
dreams.insert(1, "Author", author_id)
dreams = dreams.drop(columns=["dream_id"])

DREAM_ID_RE = r"^(?P\d{3})-\d{2}a?$" assert dreams["dream_id"].str.match(DREAM_ID_RE).all() author_id = dreams["dream_id"].str.extract(DREAM_ID_RE) dreams.insert(1, "Author", author_id) dreams = dreams.drop(columns=["dream_id"])

In [34]:

display(dreams.head(n=2))

display(dreams.head(n=2))

	dataset_id	Author	metadata	word_count	dream_text
0	blind-f	101	F-A-T-19/27	231	I was at my parent's house that I grew up in a...
1	blind-f	101	F-A-T-19/27	253	I was back in the house that I grew up in and ...

Handle "metadata" column. Identify metadata values and parse out.

The metadata column has a lot of valuable info about sex, age, and blindness. All can be parsed out into their own columns.

In [35]:

METADATA_RE = r"^(?P<Sex>[MF])-(?P<Nature>[A-Z])-(?P<DegreeOfBlindness>[A-Z])-(?P<YearsBlind>\d{2})/(?P<Age>\d{2})$"
metadata = dreams["metadata"].str.extract(METADATA_RE)

METADATA_RE = r"^(?P[MF])-(?P[A-Z])-(?P[A-Z])-(?P\d{2})/(?P\d{2})$" metadata = dreams["metadata"].str.extract(METADATA_RE)

In [36]:

dreams = pd.concat([dreams, metadata], axis=1, verify_integrity=True)
dreams = dreams.rename(columns={"YearsBlind": "Years Blind", "DegreeOfBlindness": "Degree of Blindness"})
dreams = dreams.drop(columns=["metadata"])
dreams["dream_text"] = dreams.pop("dream_text")

dreams = pd.concat([dreams, metadata], axis=1, verify_integrity=True) dreams = dreams.rename(columns={"YearsBlind": "Years Blind", "DegreeOfBlindness": "Degree of Blindness"}) dreams = dreams.drop(columns=["metadata"]) dreams["dream_text"] = dreams.pop("dream_text")

In [37]:

display(dreams.head(n=2))

display(dreams.head(n=2))

	dataset_id	Author	word_count	Sex	Nature	Degree of Blindness	Years Blind	Age	dream_text
0	blind-f	101	231	F	A	T	19	27	I was at my parent's house that I grew up in a...
1	blind-f	101	253	F	A	T	19	27	I was back in the house that I grew up in and ...

Use the legend to replace metadata values here as was done for authors dataframe.

In [38]:

dreams["Nature"] = dreams["Nature"].map(legend)
dreams["Degree of Blindness"] = dreams["Degree of Blindness"].map(legend)

dreams["Nature"] = dreams["Nature"].map(legend) dreams["Degree of Blindness"] = dreams["Degree of Blindness"].map(legend)

In [39]:

dreams = dreams.drop(columns=["dataset_id"])

dreams = dreams.drop(columns=["dataset_id"])

Check word counts but ultimately remove.

In [40]:

wc_dreambank = dreams["word_count"]
wc_calculated = dreams["dream_text"].str.split().str.len()

wc_dreambank = dreams["word_count"] wc_calculated = dreams["dream_text"].str.split().str.len()

In [41]:

wc_diffs = wc_dreambank - wc_calculated
ax = wc_diffs.plot(kind="hist", figsize=(8, 3), bins=30)
ax.set_yscale("log")

wc_diffs = wc_dreambank - wc_calculated ax = wc_diffs.plot(kind="hist", figsize=(8, 3), bins=30) ax.set_yscale("log")

See what's going on with the really large discrepancies.

In [42]:

with pd.option_context("display.max_colwidth", None):
    display(dreams.loc[wc_diffs.le(-10)])

with pd.option_context("display.max_colwidth", None): display(dreams.loc[wc_diffs.le(-10)])

	Author	word_count	Sex	Nature	Degree of Blindness	Years Blind	Age	dream_text
31	103	31	F	adventitious blindness	partial blindness	10	73	My father was quite incensed because he could not join the state police and I said to him "but dad, no one wants a state policeman who's 86 years old." [In real life my father died at 75 and if he were alive today, he would be 96 not 86] .
262	104	139	M	congenital blindness	total blindness	45	45	[1997-05-13] I went into this nut shop I used to go into to buy peanuts and cashews and things and I went in and thought, "boy this place is really small, they must have changed it around" which was really weird because they had just taken up half the space that used to belong to Wiener World [which in reality is about a half block away on the other side of an alley] and made the nut shop really big, but now its small again. I went in and talked to the person, trying to decide what kind of nuts I wanted and I couldn't think of anything that sounded good because it was early in the morning and I didn't want peanuts and I didn't want any other kind, but I asked him though, if they had anything they could mix with chocolate covered raisins and they said they didn't have anymore chocolate covered raisins.
267	104	898	M	congenital blindness	total blindness	45	45	[1997-05-20] I had to go for a physical and I remember getting there, my wife drove us up, I was supposed to be there at 1:00 and it was 2 minutes to 1:00 when we got there. I checked my watch and said "this is great, we're right on time, its a good thing we left when we did." We got in there and the nurse took me and had me step up on a scale, was weighing me. Then I got up on the table and the doctor came in, he had a deep voice. I remember the sound of his shoes walking, he had a nice firm walk He came in and started talking to me and gave me a shot I asked him what kind of shot it was and he said "It's an ACD shot." My wife is a nurse (in reality) and we were trying to figure out what kind of shot that was. She said she didn't know. He gave me the shot in the upper part of my left arm and it was somewhat painful and I sort of winced a little bit, but I said it didn't hurt that badly. I asked him what it was and he said it was a cardiac shot, maybe hepatitis, I asked what the A was for and he said that was a shot for marijuana. I asked what he meant, and he said "it lessens the effect of marijuana if you smoke it." I laughed and said "Oh great! Now I'll have to smoke twice as much marijuana." And he laughed and I said "no, I don't smoke marijuana" [in reality, not for the last 25 years] . Then, while I was still on the table, there was a loud buzz, a doorbell, but the person rang it excessively long and it was excessively loud, and the doctor groaned "oh boy" but he went to the door and opened it. Then this man with a very effeminate voice came in and said he had flowers to deliver and the doctor said "Oh gosh! Those weren't supposed to be delivered here!" Then he called to his nurse "Bevvv! Come in here and take care of this, these are supposed to be delivered (I think he said) to my wife." So she came in and took care of the man with the flowers. The next thing I remember I was out in my backyard sitting at a picnic table eating dinner with my mother and some other people. My mother was sitting right to my right. I don't remember what we were eating, but the yard was kind of sloped down and we were back towards the end of it and we were sitting there eating and talking. I mentioned to my mother that I had had a shot and said "I bet I could probably drink beer now and I bet I could have a few now." [Reality, I have been a recovering and sober alcoholic for 14 years] And my mother said "Oh, I don't know what I would do if I started to find beer bottles and found out that you were drinking again." I started to laugh and said "well, you know there have been a number of times when I've had a few! It doesn't really bother me if I just drink beer and don't drink too much." She didn't sound too believing but said "oh, that's interesting." My thoughts and been that maybe if this shot for marijuana was effective, it might allow me to drink like a regular drinker without having any bad effects from it. My mother left. They had all left and gone somewhere. I was by myself, and I thought I had better start cleaning up the dishes and I picked up my plate and I had a glass which I left there, and started to walk. I heard a splash and realized that I was right along side of a river and I didn't want to fall in the river, and I couldn't exactly remember which way I had to go to get up to my house. I was pretty sure I could feel that the land sloped a little bit, and I thought I was okay and so I started to walk in generally the right direction, but I sort of veered off to the side of my yard and got tangled up in some bushes that are there. I kept walking and at one point I brushed against a wire antenna that I have in my back yard in reality and I remember thinking "boy this thing is really hanging too low. I should get a pole to prop up the middle with" I kept walking and got myself straightened out and got up to the top near my house, where a patio is and a dishwasher was there, and I started to put my plate inside the dishwasher, and I was going to go back and get some more when I started to hear all these voices from inside my basement. It was my neighbor E from across the street, with a bunch of people and they were talking, it was as if she was giving them a tour of my house. Then she came out and started thanking me and then my mother came out and she thanked us both for being such nice people and being such good help. I remember saying "E you're always welcome." Then she was leaving.
268	104	368	M	congenital blindness	total blindness	45	45	I was in a class, possibly for computer programming. There seemed to be many familiar people in the class with me and the instructor was very friendly and it seemed as if I was sitting in the front of the room and he was sitting in the back, and he came up to me and said "I was going to buy you an "Opticon" but the administration wouldn't let me, they cost $6000 but I brought you a Braille writer and I bought you all this other stuff and I just wondered if you were familiar with it because you are going to need them in this class." I remember thinking "I'm glad he didn't buy an Opticon because I don't remember how to use one. So I began to unpack the box he gave me, and there did appear to be an Opticon and had a little hand held camera with it. A little box that had something similar to a little Braille display, maybe a one line Braille display, maybe an Opticon. I started taking out this thing and all the accessories that came with it and I was amazed at the different plugs and cords and there were adapters like standard serial port adapters for computers, there were some that were offset between the male and female end of it. There were some wires with like plastic keys hanging off of them. My co-worker, M happened to be there then and she said "Oh, this is interesting, you could probably stick this in a car's ignition to start it if you happened to lose your key." We had a laugh about that. I just kept looking at these different pieces. There was one piece that I fitted onto the back of the machine with a little flange, it was some sort of little plug. Throughout the dream everything seemed pretty familiar to me and I seemed to know what everything was for. Then M said she had to go and do an interview and I said "where's everybody else?" and she said "I have no idea, they all left at 8:00 and they just left at 8:00 and I'm left with the work to do." [An Opticon is a device which converts printed material into tactile images that actually feel like the printed letters. It has largely gone out of vogue for the last 15 or 20 years.]
270	104	344	M	congenital blindness	total blindness	45	45	[1997-05-22] I was in the computer room in my basement here and my friend J's daughter was here with me. She was trying to print some stuff and I was helping her with the computer and I was asking her if she ever used Print-Shop. And we were talking about Q&A and WordPerfect and I forgot to run my speech program and was having trouble figuring out where I was and she was kind of helping me. In the dream his daughter, who in reality is 40 sounded like she was 20. Next I remember being at the Guild for the Blind in Hingeville. I was walking through there smoking a pipe and the tobacco smelled really good, like cinnamon [I haven't smoked a pipe in 10 years in reality] . I was walking through there talking to some people and all of a sudden I could hear all these little kids coming in and somebody came over to me and said "do you hear the fire?" At first I said no, but then I heard some sirens off in the distance. They said it was next door at a restaurant which I remember having been in before. Then my wife was there and she was looking and I asked "How bad does it look?" and she said "Oh, it's really a bad fire! A good one, really ablaze." Then she took me over there and we walked into the driveway and we could hear the men up above, working away on the roof. One of the firefighters yelled down "What are you doing here? Get out of here now!" And I responded "I'm just glad I had a chance to hear this." I asked my wife again how it looked and she still said that it looked pretty bad. Then rain started to fall on us but it may have been water from the hose. I said, lets get out of this or we'll have to go home and take a bath. I also remember at some point, being able to smell the smoke, it had a rather acrid smell typical of house fires.
272	104	81	M	congenital blindness	total blindness	45	45	[1997-05-26] [I am a recovering alcoholic who still attends meetings occasionally] I dreamt that I was at a meeting with my sponsor, we generally go on Saturday mornings together. He had spoken during the meeting, it was a discussion meeting, and then 5 minutes before the meeting ended, he repeated, "My name is B and I'm an alcoholic" but then he got all emotional and choked up and couldn't finish what he had wanted to say. He was quite upset and distraught, and I could hear him almost beginning to break down.
281	104	468	M	congenital blindness	total blindness	45	45	[1997-06-05] I was in my office and I was going to go to lunch, walking over toward the coat rack I remembered saying "I think I'm going to blow the rest of this day off." One of my co-workers was kidding me and she said "I heard that!" Then I was supposed to go to a meeting the Cleveland Blind Association and I went to the coat rack and I couldn't find my coat. I was feeling all the different coats, even reached into the pocket of one of them which felt similar to mine, and I found a bunch of balled up papers and different things and I realized that it wasn't my coat. I had to go through the whole coat rack one coat after another until I found mine. They all seemed to feel the same and I had a cab waiting for me so I was really in a hurry to go. I finally did find it, it was way over on the right hand end of the coat rack, probably took five minutes to go through all of the coats [That has never happened to me, I can usually reach right over and feel the texture of my coat.] So I went downstairs and I remember thinking "Oh, at least the cab is coming right here for me I don't have to walk a couple blocks to meet it." The next thing I knew, I was at the Blind Association and I walked in. I couldn't remember exactly which way to go. The more I thought about it the more it seemed like the school for blind children that I went to as a kid, than the blind association. But I got there and was walking and one of the women who worked there was kind of helping me and she said "now we turn to the right here" then she grabbed me and said "its very narrow here" she thought I was going to bump into something. Then we ended up in a big open reception area in the front and again I was waiting for this cab. I don't know how I got to the Blind association but she said, I was right next to this big window in the front of the place, she said "sit here in this prom chair" and I said "boy, I bet I'm less nervous today then when I went to the prom." Then we started talking about the prom and I said that neither of my kids had ever gone to the prom. I remember going to the bath room and then coming out and they said, "Your cab was here but he didn't want to wait so he went to have lunch at the restaurant next door and he'll be back to pick you up." So I waited about 15 or 20 minutes and he never did come back to pick me up.

Makes sense! DreamBank is not including bracketed content, which might systematically be non-dream content. Given the diversity of needs, and that many other dream datasets will inherently include this kind of content, I'll keep it in the dream report. We can drop the word_count column though.

In [43]:

dreams = dreams.drop(columns=["word_count"])

dreams = dreams.drop(columns=["word_count"])

In [44]:

display(dreams.head(n=2))

display(dreams.head(n=2))

	Author	Sex	Nature	Degree of Blindness	Years Blind	Age	dream_text
0	101	F	adventitious blindness	total blindness	19	27	I was at my parent's house that I grew up in a...
1	101	F	adventitious blindness	total blindness	19	27	I was back in the house that I grew up in and ...

Make sure that the author metadata is consistent across authors dataframe (extracted from moreinfo) and dreams dataframe (extracted from metadata columns).

In [45]:

authors_from_dreams = dreams.drop(columns=["dream_text"]).drop_duplicates().sort_values(by="Author", ignore_index=True)

authors_from_dreams = dreams.drop(columns=["dream_text"]).drop_duplicates().sort_values(by="Author", ignore_index=True)

In [46]:

display(authors_from_dreams)

display(authors_from_dreams)

	Author	Sex	Nature	Degree of Blindness	Years Blind	Age
0	101	F	adventitious blindness	total blindness	19	27
1	102	F	congenital blindness	total blindness	32	32
2	103	F	adventitious blindness	partial blindness	10	73
3	104	M	congenital blindness	total blindness	45	45
4	105	M	adventitious blindness	partial blindness	43	50
5	109	M	adventitious blindness	total blindness	20	24
6	111	F	congenital blindness	perceives very bright light	50	50
7	113	M	adventitious blindness	total blindness	17	45
8	114	F	congenital blindness	total blindness	52	52
9	115	F	adventitious blindness	total blindness	60	60
10	117	F	congenital blindness	total blindness	44	44
11	120	F	adventitious blindness	total blindness	47	49
12	122	F	adventitious blindness	total blindness	46	58
13	124	M	congenital blindness	total blindness	46	46
14	125	F	congenital blindness	total blindness	44	44

In [47]:

authors

authors

Out[47]:

	Code #	Sex	Age	Years of Education	Occupation	Retired	Years Blind	Nature	Degree of Blindness	# of dream reports
0	101	F	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	10
1	102	F	32	18.0	Unemployed	False	32	congenital blindness	total blindness	10
2	103	F	73	12.0	Sales clerk	True	10	adventitious blindness	partial blindness	39
3	104	M	45	16.0	Human resources management	False	45	congenital blindness	total blindness	61
4	105	M	50	18.0	Rehabilitation specialist	False	43	adventitious blindness	partial blindness	21
5	109	M	24	14.5	Software tech worker	False	20	adventitious blindness	total blindness	20
6	111	F	50	19.0	Vocational rehabilitation counselor	False	50	congenital blindness	perceives very bright light	59
7	113	M	45	13.0	Piano tuner/technician	False	17	adventitious blindness	total blindness	22
8	114	F	52	12.0	Envelope stuffer	False	52	congenital blindness	total blindness	37
9	115	F	60	14.0	Cook at a school for the blind	False	60	adventitious blindness	total blindness	6
10	117	F	44	18.0	Factory worker	True	44	congenital blindness	total blindness	32
11	120	F	49	18.0	Grant writer for Blind Services	False	47	adventitious blindness	total blindness	10
12	122	F	58	12.0	Unemployed	False	46	adventitious blindness	total blindness	24
13	124	M	46	12.0	Small engine repairs	False	46	congenital blindness	total blindness	12
14	125	F	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	9

In [48]:

assert authors_from_dreams["Author"].eq(authors["Code #"].astype(str)).all()
assert authors_from_dreams["Sex"].eq(authors["Sex"]).all()
assert authors_from_dreams["Age"].eq(authors["Age"].astype(str)).all()
assert authors_from_dreams["Degree of Blindness"].eq(authors["Degree of Blindness"]).all()
assert authors_from_dreams["Nature"].eq(authors["Nature"]).all()
assert authors_from_dreams["Years Blind"].eq(authors["Years Blind"].astype(str)).all()

assert authors_from_dreams["Author"].eq(authors["Code #"].astype(str)).all() assert authors_from_dreams["Sex"].eq(authors["Sex"]).all() assert authors_from_dreams["Age"].eq(authors["Age"].astype(str)).all() assert authors_from_dreams["Degree of Blindness"].eq(authors["Degree of Blindness"]).all() assert authors_from_dreams["Nature"].eq(authors["Nature"]).all() assert authors_from_dreams["Years Blind"].eq(authors["Years Blind"].astype(str)).all()

Drop the author metadata from dreams dataframe since it is all already in authors dataframe, plus authors dataframe has additional metadata.

In [49]:

dreams = dreams.drop(columns=["Sex", "Age", "Degree of Blindness", "Nature", "Years Blind"])

dreams = dreams.drop(columns=["Sex", "Age", "Degree of Blindness", "Nature", "Years Blind"])

In [50]:

n_dreams_from_dreams = dreams["Author"].astype(int).value_counts().sort_index()
n_dreams_from_authors = authors.set_index("Code #")["# of dream reports"].sort_index()

n_dreams_from_dreams = dreams["Author"].astype(int).value_counts().sort_index() n_dreams_from_authors = authors.set_index("Code #")["# of dream reports"].sort_index()

In [51]:

n_dreams = pd.concat([n_dreams_from_dreams, n_dreams_from_authors], axis=1, keys=["From dreams", "From authors"])

n_dreams = pd.concat([n_dreams_from_dreams, n_dreams_from_authors], axis=1, keys=["From dreams", "From authors"])

In [52]:

display(n_dreams)

display(n_dreams)

	From dreams	From authors
101	10	10
102	10	10
103	39	39
104	61	61
105	22	21
109	20	20
111	60	59
113	27	22
114	37	37
115	6	6
117	32	32
120	11	10
122	24	24
124	13	12
125	9	9

There are a few rows where there are more dreams in the dataset than reported from the author table. This makes sense for the "3a" case (see above), but not sure about the others. I will leave it alone for now, as the number of dreams in the dataset is the truth.

In [53]:

display(n_dreams[n_dreams["From dreams"] != n_dreams["From authors"]])

display(n_dreams[n_dreams["From dreams"] != n_dreams["From authors"]])

	From dreams	From authors
105	22	21
111	60	59
113	27	22
120	11	10
124	13	12

Drop number of dreams column since we should just calculate it later as-needed for most accurate values.

In [54]:

authors = authors.drop(columns=["# of dream reports"])

authors = authors.drop(columns=["# of dream reports"])

Clean actual dream reports.

In [55]:

display(dreams)

display(dreams)

	Author	dream_text
0	101	I was at my parent's house that I grew up in a...
1	101	I was back in the house that I grew up in and ...
2	101	Right before waking up I remember standing in ...
3	101	Last night I dreamed that I went back to schoo...
4	101	There was a book published, like a book of ess...
...	...	...
376	124	[1997-07-25] I was going to the store to buy b...
377	124	[1997-08-01] Dreamt we went to an amusement pa...
378	124	[1997-08-07] I remember playing cards with Cla...
379	124	[1997-08-10] This dream had more reality about...
380	124	[1997-08-16] I dreamt of a job I had, years an...

381 rows × 2 columns

First we can remove any of the bracketed dates, pulling them out to their own column (will be empty/NA where not provided).

In [56]:

DREAM_TEXT_RE = r"^(?P<Date>\[\d{4}-\d{2}-\d{2}\])?\s?(?P<Report>.+)$"
assert dreams["dream_text"].str.match(DREAM_TEXT_RE).all()

DREAM_TEXT_RE = r"^(?P\[\d{4}-\d{2}-\d{2}\])?\s?(?P.+)$" assert dreams["dream_text"].str.match(DREAM_TEXT_RE).all()

In [57]:

dreams[["Date", "Report"]] = dreams["dream_text"].str.extract(DREAM_TEXT_RE)
dreams = dreams.drop(columns=["dream_text"])

dreams[["Date", "Report"]] = dreams["dream_text"].str.extract(DREAM_TEXT_RE) dreams = dreams.drop(columns=["dream_text"])

In [58]:

dreams["Date"] = dreams["Date"].str.strip("[]")

dreams["Date"] = dreams["Date"].str.strip("[]")

In [59]:

dreams["Report"] = dreams["Report"].str.strip()

dreams["Report"] = dreams["Report"].str.strip()

Review¶

Rename columns in both dataframes to snake_case.

In [60]:

authors = authors.rename(
    columns={
        "Code #": "author",
        "Age": "age",
        "Sex": "sex",
        "Years of Education": "years_education",
        "Occupation": "occupation",
        "Retired": "retired",
        "Years Blind": "years_blind",
        "Nature": "nature_blindness",
        "Degree of Blindness": "degree_blindness",
        "# of dream reports": "n_reports",
    }
)

authors = authors.rename( columns={ "Code #": "author", "Age": "age", "Sex": "sex", "Years of Education": "years_education", "Occupation": "occupation", "Retired": "retired", "Years Blind": "years_blind", "Nature": "nature_blindness", "Degree of Blindness": "degree_blindness", "# of dream reports": "n_reports", } )

In [61]:

dreams = dreams.rename(
    columns={
        "Author": "author",
        "Date": "date",
        "Report": "report",
    }
)

dreams = dreams.rename( columns={ "Author": "author", "Date": "date", "Report": "report", } )

Merge author info with dreams dataframe for one export file.

In [62]:

authors = authors.astype({"author": str})

authors = authors.astype({"author": str})

In [63]:

df = authors.merge(dreams, on="author", how="right", validate="one_to_many")

df = authors.merge(dreams, on="author", how="right", validate="one_to_many")

In [64]:

df = df.replace({"sex": {"F": "female", "M": "male"}})

df = df.replace({"sex": {"F": "female", "M": "male"}})

In [65]:

df = df.sort_values("author")

df = df.sort_values("author")

In [66]:

assert df["sex"].isin(["female", "male"]).all()
assert not df.drop(columns=["date"]).isna().any(axis=None)
assert (df["report"] == df["report"].str.strip()).all()
assert not df.duplicated().any()
assert not df.duplicated(subset=["report"]).any()
assert df.drop_duplicates(subset=["author", "sex", "age", "occupation", "retired", "years_blind", "nature_blindness", "degree_blindness"]).shape[0] == df["author"].nunique()

assert df["sex"].isin(["female", "male"]).all() assert not df.drop(columns=["date"]).isna().any(axis=None) assert (df["report"] == df["report"].str.strip()).all() assert not df.duplicated().any() assert not df.duplicated(subset=["report"]).any() assert df.drop_duplicates(subset=["author", "sex", "age", "occupation", "retired", "years_blind", "nature_blindness", "degree_blindness"]).shape[0] == df["author"].nunique()

In [67]:

df.info(memory_usage="deep")

df.info(memory_usage="deep")

<class 'pandas.core.frame.DataFrame'>
Index: 381 entries, 0 to 236
Data columns (total 11 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   author            381 non-null    object 
 1   sex               381 non-null    object 
 2   age               381 non-null    int64  
 3   years_education   381 non-null    float64
 4   occupation        381 non-null    object 
 5   retired           381 non-null    object 
 6   years_blind       381 non-null    int64  
 7   nature_blindness  381 non-null    object 
 8   degree_blindness  381 non-null    object 
 9   date              245 non-null    object 
 10  report            381 non-null    object 
dtypes: float64(1), int64(2), object(8)
memory usage: 443.2 KB

In [68]:

df.describe(include="number")

df.describe(include="number")

Out[68]:

	age	years_education	years_blind
count	381.000000	381.000000	381.000000
mean	48.803150	15.259843	38.732283
std	11.275758	2.734899	14.491258
min	24.000000	12.000000	10.000000
25%	45.000000	12.000000	20.000000
50%	49.000000	16.000000	45.000000
75%	52.000000	18.000000	50.000000
max	73.000000	19.000000	60.000000

In [69]:

display(df.describe(exclude="number"))

display(df.describe(exclude="number"))

	author	sex	occupation	retired	nature_blindness	degree_blindness	date	report
count	381	381	381	381	381	381	245	381
unique	15	2	13	2	2	3	90	381
top	104	female	Human resources management	False	congenital blindness	total blindness	1997-05-29	I was at my parent's house that I grew up in a...
freq	61	238	61	310	222	260	7	1

In [70]:

display(df)

display(df)

	author	sex	age	years_education	occupation	retired	years_blind	nature_blindness	degree_blindness	date	report
0	101	female	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	NaN	I was at my parent's house that I grew up in a...
1	101	female	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	NaN	I was back in the house that I grew up in and ...
2	101	female	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	NaN	Right before waking up I remember standing in ...
3	101	female	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	NaN	Last night I dreamed that I went back to schoo...
4	101	female	27	15.0	Unemployed	False	19	adventitious blindness	total blindness	NaN	There was a book published, like a book of ess...
...	...	...	...	...	...	...	...	...	...	...	...
232	125	female	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	NaN	I am in a car with my friend Mark. Mark is tot...
233	125	female	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	NaN	I was at the [childhood school for the blind] ...
234	125	female	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	NaN	I had a dream of a friend of my father's whose...
235	125	female	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	NaN	I dreamt that I was typing medical reports at ...
236	125	female	44	13.0	Medical transcriptionist	False	44	congenital blindness	total blindness	NaN	Dreamt about my dad again. In this dream we we...

381 rows × 11 columns

Export¶

In [71]:

def format_timestamp(unix_timestamp: float) -> str:
    """Convert unix timestamp to UTC-stamped ISO format."""
    dt = datetime.fromtimestamp(unix_timestamp, tz=timezone.utc)
    timestamp = dt.isoformat(timespec="seconds")
    return timestamp

def format_timestamp(unix_timestamp: float) -> str: """Convert unix timestamp to UTC-stamped ISO format.""" dt = datetime.fromtimestamp(unix_timestamp, tz=timezone.utc) timestamp = dt.isoformat(timespec="seconds") return timestamp

In [72]:

OUTPATH = "./output/hurovitz1999.csv"
os.makedirs(os.path.dirname(OUTPATH), exist_ok=True)

TO_CSV_KWARGS = {
    "index": False,
    "na_rep": "N/A",
    "sep": ",",
    "mode": "x",  # Switch to `w` to overwrite existing file
    "encoding": "utf-8-sig",  # Include sig/BOM for better compatibility with Excel
    "lineterminator": "\n",
    "quoting": 2,  # 2 = csv.QUOTE_NONNUMERIC
    "quotechar": '"',
    "doublequote": True,
}
df.to_csv(OUTPATH, **TO_CSV_KWARGS)

print(f"{'file':>10}: {os.path.basename(OUTPATH)}")
print(f"{'size':>10}: {os.path.getsize(OUTPATH) / 1e6} MB")
with open(OUTPATH, "rb") as f:
    print(f"{'md5':>10}: {hashlib.md5(f.read()).hexdigest()}")
    print(f"{'sha256':>10}: {hashlib.sha256(f.read()).hexdigest()}")
print(f"{'Created':>10}: {format_timestamp(os.path.getatime(OUTPATH))}")
print(f"{'Modified':>10}: {format_timestamp(os.path.getmtime(OUTPATH))}")
print(f"{'Accessed':>10}: {format_timestamp(os.path.getatime(OUTPATH))}")
print(f"{'Now':>10}: {datetime.now(timezone.utc).isoformat(timespec='seconds')}")

OUTPATH = "./output/hurovitz1999.csv" os.makedirs(os.path.dirname(OUTPATH), exist_ok=True) TO_CSV_KWARGS = { "index": False, "na_rep": "N/A", "sep": ",", "mode": "x", # Switch to `w` to overwrite existing file "encoding": "utf-8-sig", # Include sig/BOM for better compatibility with Excel "lineterminator": "\n", "quoting": 2, # 2 = csv.QUOTE_NONNUMERIC "quotechar": '"', "doublequote": True, } df.to_csv(OUTPATH, **TO_CSV_KWARGS) print(f"{'file':>10}: {os.path.basename(OUTPATH)}") print(f"{'size':>10}: {os.path.getsize(OUTPATH) / 1e6} MB") with open(OUTPATH, "rb") as f: print(f"{'md5':>10}: {hashlib.md5(f.read()).hexdigest()}") print(f"{'sha256':>10}: {hashlib.sha256(f.read()).hexdigest()}") print(f"{'Created':>10}: {format_timestamp(os.path.getatime(OUTPATH))}") print(f"{'Modified':>10}: {format_timestamp(os.path.getmtime(OUTPATH))}") print(f"{'Accessed':>10}: {format_timestamp(os.path.getatime(OUTPATH))}") print(f"{'Now':>10}: {datetime.now(timezone.utc).isoformat(timespec='seconds')}")

      file: hurovitz1999.csv
      size: 0.315308 MB
       md5: 2af24d8982609d34c62fb5c7eda32f8a
    sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
   Created: 2026-01-09T16:54:14+00:00
  Modified: 2026-01-09T16:54:14+00:00
  Accessed: 2026-01-09T16:54:14+00:00
       Now: 2026-01-09T16:54:14+00:00