A: HTRC is the research arm of HathiTrust. It is a partnership between Indiana University (IU) Libraries, the Pervasive Technology Institute, and the School of Informatics and Computing at IU, and the University of Illinois, Urbana-Champaign (UIUC) Libraries, and the Graduate School of Library and Information Science at UIUC.
A: We have created a couple of platforms for you to experiment with. The main HTRC services (sometimes referred to as the production stack) gives you a Portal and a Workset Builder.
From the Portal you can log in and run analytic algorithms on a set of predefined collections of volumes. These algorithms, powered by the SEASR toolkit, run against the HathiTrust volumes that are in the public domain (close to 3M).
The Workset Builder is a search interface for the Hathitrust public domain corpus - search results can be saved as a 'workset': a collection of volumes against which the text mining algorithms are run.
In addition to the main services, we also provide a Sandbox stack with the same tools. The sandbox runs against non-Google scanned content (about 260,000 volumes). The advantage of the sandbox is that you can access the index and Data API directly, and so you can write your own algorithms.
A: The HTRC has several overarching paradigms –worksets, algorithms, jobs, and results.
A: HTRC currently has the public domain corpus OCR text from HathiTrust, along with MARC and METS XML.
A: You may sign up for an account by going to the HTRC Production Portal http://htrc2.pti.indiana.edu and choose "Sign up" from the menu.
A: This table lists the HTRC Production Stack entries
Service | Endpoint | Comments |
---|---|---|
Portal | http://htrc2.pti.indiana.edu | The portal allows you to browse volume lists and algorithms, execute algorithms, and view results |
Blacklight | http://sandbox.htrc.illinois.edu:8080/blacklight | The Blacklight search interface allows you to search for volumes, and create volume lists that can be used by algorithms. It provides a GUI interface to our Solr index |
A: You can send an email to htrc-tech-help-l@list.indiana.edu (a list subscribed by HTRC internal staff only) to request for an account, along with your name, your contact information, and indicate that you would like to access the HTRC Sandbox.
A: This table lists the HTRC Sandbox entries
Service | Endpoint | Comments |
Portal | https://sandbox.htrc.illinois.edu/HTRC-UI-Portal2 | The portal allows you to browse volume lists and algorithms, execute algorithms, and view results |
Blacklight | https://sandbox.htrc.illinois.edu/blacklight | The Blacklight search interface allows you to search for volumes, and create volume lists that can be used by algorithms. It provides a GUI interface to our Solr index |
Data API | https://sandbox.htrc.illinois.edu/data-api | The HTRC Data API provides access to the corpus data and METS XML via a RESTful web service |
Solr Proxy | http://sandbox.htrc.illinois.edu/solr | The HTRC Solr Proxy provides access to the Solr index. A sample query is: http://sandbox.htrc.illinois.edu/solr/ocr/select?q=shakespeare please refer to the Solr Guide for more details on query. |
Recent addition: HTRC Bookworm: http://sandbox.htrc.illinois.edu/bookworm
A: This table outlines the differences between the Production Stack and the Sandbox:
Production Stack | Sandbox | |
---|---|---|
purpose | a distributed service oriented cyberinfrastructure to support various digital humanities researches and text analysis of HTRC members | a community asset meant to be open to the community and for interested users to try things out on a smaller scale |
number of machines | 9 | 1 |
corpus | full public domain set | non-Google scanned public domain subset |
number of volumes | 2.7 million | 250,000 |
compute resource | a separate 128-node cluster | local on the Sandbox |
accounts | personal account | pre-defined account pool |
account reclamation | no | yes (reclaimed and reassigned after 30 days of inactivity) |
A: The HTRC Solr Proxy is a thin service in front of Apache Solr services for security and auditing purposes. The Solr Proxy filters requests to allow read-only requests to protect our indices from being modified; other than that, it is fully compatible with Apache Solr. Please see Solr Proxy API User Guide
A: This table outlines the differences between the HTRc Data API and HathiTrust Data API
HTRC Data API | HT Data API | |
---|---|---|
purpose | to serve high-performance large-scale algorithms and programs | to provide public users some volume retrieval capabilities |
throttling enforcement | no | yes |
security | OAuth2 | OAuth |
bulk retrieval of volumes | yes | no |
metadata available | METS | METS, MARC |
A: The HTRC Data Capsule provides a researcher with a virtual machine that they configure as they need. This includes loading necessary software packages and data sets. When they are ready to run their analysis, they switch their data capsule from maintenance mode to "secure mode", and the routines run in a secure mode that does not allow content from the HT repository to leak out. When completed, the researcher receives an email giving them the location from which to download the results. The HTRC Data Capsule is in apha version and undergoing internal testing.
A: Please see HTRC Data API Users Guide
Worksets are collections of volumes from our collection. There are currently two types of workset: basic and labeled. Basic worksets can be created with the Workset Builder or with the upload CSV functionality, labeled worksets can only be added by uploading a CSV.
The easiest way to create a basic workset is to use the Workset Builder. The Workset Builder allows you to search across our collection. In the search results, note that there is a select button:
All the items that you select are kept in the Workset Builder. To review them, click "selected items" in the navigation bar. This is meant as a workspace for building a volume list for the workset, to save a workset of these items: click "Create/Update workset":
When you're saving a workset, note that it can be saved publicly (viewable by all users) or private. After saving a workset, it will be available in the HTRC Portal, for use in analysis or for download.
While a basic workset simply collects volumes in one place, it is possible to add classes to worksets. This allows for use with classification algorithms, such as Naive Bayes.
The CSV can be built in your preferred way. One common approach is to
A labeled workset CSV should follow the following style:
Below is an example of what the CSV file looks like. Given some volumes, classes are assigned to them based on some criteria. For example, here the labels are the names of the authors of the volumes:
volume_id, class
mdp.39015001796500,Austen
uc2.ark:/13960/t42r3rg51,Austen
uc2.ark:/13960/t3dz03x48,Austen
uiuo.ark:/13960/t4km00443,Austen
mdp.39015004997253,Austen
uc2.ark:/13960/t6c24sq2z,Austen
uc2.ark:/13960/t6m041m4z,Dickens
uiuo.ark:/13960/t5cc1pz8f,Dickens
uiuo.ark:/13960/t1wd47104,Dickens
uc2.ark:/13960/t2v40sj3m,Dickens
uiuo.ark:/13960/t3tt5bm1x,Dickens
uiuo.ark:/13960/t6n013296,Dickens
Worksets are uploaded in the HTRC Portal, under Worksets > Upload Workset, or with the '+' button in the workset list view. This is an alternative to the Workset Builder, and currently the only way to add labeled worksets.
Notes:
A: If you want to ensure that results are retained through a restart of the services, then you should save your results.
* The current login timeout is 12 hours.
A: Below are links to some very useful documentation:
A: Yes. All of the HTRC services code modules are open source and are available from SourceForge. Go to http://sourceforge.net/p/htrc/code/ to browse the code, or check out directly from SVN using:
svn co svn://svn.code.sf.net/p/htrc/code/ |
A: Please join the HTRC Usergroup mailing list.
A: HTRC has a GitHub set up for browsing contributed code. It is at https://github.com/htrc
A: To report a bug, please go to http://jira.htrc.illinois.edu/browse/HTRC. You need to create a JIRA login account if you have not done so already. To provide feedback, you may use the "feedback" tab found on the right-hand side of various portal pages to pop up a form.