HTRC Derived Datasets are structured sets of metadata representing a curated collection of HathiTrust volumes. Read about the basics of our Extracted Features and partner-created datasets here.
HTRC Extracted Features
HTRC Extracted Features datasets consist of metadata and derived data elements that have been extracted from volumes in the HathiTrust Digital
Texts from the HTDL corpus that is not in the public domain are not available for download, which limits its usefulness for research. However, a great deal of fruitful research, especially in the form of text mining, can be performed on the basis of non-consumptive reading using extracted features (features extracted from the text) even when the full text is not available. To this end, the HathiTrust Research Center (HTRC) has started making available a set of page-level features extracted from the HTDL's public domain volumes. These extracted features can be the basis for certain kinds of algorithmic analysis. For example, topic modeling works with bags of words (sets of tokens). Since tokens and their frequencies are provided as features, the EF dataset can enable a user to perform topic modeling with the data.
Worksets and the Extracted Features (EF) Dataset
An EF data file for a volume consists of volume-level metadata, and of the extracted feature data for each page in the volume, in JSON format. The volume-level metadata consists of both metadata about the volume and metadata about the extracted features.
Metadata about the volume consists of the following pieces of data:
Metadata about the extracted features consists of the following pieces of data:
“Extracted features” data:
The extracted features that HTRC currently provides include part-of-speech (POS) -tagged token counts, header and footer identification, and various line-level information. (Providing token information at the page level makes it possible to separate paratext from text — e.g. identify pages of publishers’ ads at the back of a book.) Each page is broken up into three parts: header, body, and footer. Correction of hyphenation of tokens at end of lines has been carried out, but not any additional data cleaning or OCR correction.
The corresponding fields for header, body, and footer are the same, but apply to different parts of the page.
This information is provided to help clarify genre and volume structure; for instance, it can help distinguish poetry from prose, or body text from an index.
A simplified EF data file for basic features, with metadata and features for a single page:
"title":"Shakespeare's Romeo and Juliet,",
"imprint":"Scott Foresman and company, [c1920]"
Library. The dataset is periodically updated, including adding new volumes and adjusting the file schema. When we update the dataset, we create a new version. The current version is v.2.0.
A great deal of useful research can be performed with features extracted from the full text volumes. For this reason, we generate and share a dataset called the HTRC Extracted Features. The current version of the dataset is Extracted Features 2.0. Each Extracted Features file that is generated corresponds to a volume from the HathiTrust Digital Library. The files are in JSON-LD format.
An Extracted Features file has two main parts:
Each file begins with bibliographic and other metadata describing the volume represented by the Extracted Features file.
Features are notable or informative characteristics of the text. The features include:
- Token (word) counts that have been tagged with part-of-speech in order to disambiguate homophones and enable a greater variety of analyses
- Various line-level information, such as the number of lines with text on each page, and a count of characters that start and end lines on each page
- Header and footer identification for cleaner data.
Within each Extracted Features file, features are provided per-page to make it possible to separate text from paratext. For instance, feature information could aid in identifying publishers' ads at the back of a book.
Examples and tutorials
Tools for working with HTRC Extracted Features
Partner-created derived datasets
HTRC has partnered with researchers to create other derived datasets from the HathiTrust corpus. Follow the links below to learn more and access the data.
NovelTM Datasets for English-Language Fiction, 1700-2009 (Ted Underwood, Patrick Kimutis, Jessica Witte)
This dataset is descriptive metadata for 210,305 volumes of English-language fiction in HathiTrust Digital Library. Nineteenth- and twentieth-century fiction are also divided into seven subsets with different emphases (for instance, one where men and women are represented equally, and one composed of only the most prominent and widely-held books). Fiction was identified using a mixed approach of metadata and predictive modeling based on human-assigned ground truth. A full description of the dataset and its creation is available in the dataset report linked below.
Word Frequencies in English-Language Literature, 1700-1922 (Ted Underwood)
This dataset contains the word frequencies for all English-language volumes of fiction, drama, and poetry in the HathiTrust Digital Library from 1700 to 1922. Word counts are aggregated at the volume level, but include only pages tagged as belonging to the relevant literary genre. Fiction was identified using a mixed approach of metadata and predictive modeling based on human-assigned ground truth. A full explanation of the dataset's features, motivation, and creation is available on the dataset documentation page below.
Geographic Locations in English-Language Literature, 1701-2011 (Matthew Wilkens)
The dataset contains volume metadata as well as geographical locations and the number of times the location is mentioned in the text of works of fiction written in English from 1701 - 2011 that are found in the HathiTrust Digital Library. This dataset relied on Ted Underwood’s novelTM dataset to determine which volumes to include, and it is part of Matthew Wilkens' larger Textual Geographies Project. Information about the Textual Geographies Project can be found at the Textual Geographies Project link below. A full explanation of the Textual Geographies in English Literature dataset is available at the documentation link below.