Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Streamline text as per SHARC-501

...

Info

This documentation has been updated for the newest format of URLs for the Extracted Features dataset, intended for release in August 2016. This format no longer has basic and advanced features described in separate files. If you are looking for information on the earlier format, see revision 32 of this page.

Overview

In this page, we introduce the The Extracted Features functionality (currently under beta release) that has recently been developed by the HathiTrust Research Center (HTRC). This functionality is one of the ways in which users of HTRC's tools can perform non-consumptive analysis of subsets of texts in the HathiTrust Digital Library's corpus, that they have custom-selected by means of the workset mechanism available through the HTRC.. This article explains how you can use the HTRC's workset functionality to download the EF files for a personalized collection. Currently, this functionality is available only for the HathiTrust Digital Library's public domain corpus, consisting of slightly less than 5 million volumes.

Create your workset

This section shows you how to  create a custom workset, for the volume(s) contained in which you will eventually download the corresponding advanced and basic EF data files. Your workset can contain as many volumes as you wish. However, the example workset for this section will consist, for the sake of simplicity, of a single volume from the HathiTrust Digital Library's public domain collection: a published-in-1920 edition of the book of poems titled Buch der Lieder by the German poet Heinrich Heine. Then we show you how you can download the EF data files corresponding to this workset. (One of the use cases for the EF approach to non-consumptive text analysis that we have posted also uses this particular book by Heine to make its point.)

1.1 Navigate to the Portal (https://analytics.hathitrust.org/)

Navigate to the HTRC Portal. Click on the link stating “Sign In” at the upper right corner of the screen.

...


...

 

1.2 Sign in to HTRC Portal

After Step 1, you will reach the  screen shown below. Enter your HTRC username and password at the respective fields, and then click on the “Sign In” button.

...

 

1.3 Verify that you are logged in to the HTRC Portal

...

1.4 Prepare to create a workset

In this step-by-step instruction set, we will show you how to create a new workset using the HTRC Workset Builder. You will be performing a search on the HathiTrust Digital Library’s collection and then selecting some or all of the search results to constitute your workset. For simplicity, we will show you how to create a simple workset consisting of a single volume, for which you will eventually be able to download the basic and advanced feature data files, by the end of these instructions.

Other ways of creating worksets, or of making use of public worksets  created by other users, also exist. For example, if you already have a comma-separated values (CSV) file that specifies the list of HathiTrust volumeIDs corresponding to the volumes that you want your workset to comprise,  you can simply upload it using the "Upload workset" link. For more information about creating, uploading and browsing worksets, you can consult the instructions available at the HTRC Wiki.

Click on 'Create Workset'.

1.5 Access the HTRC Workset Builder

 You should now be at the screen shown below. Click on the “More options” link.

 

Image Removed

 

1.6 Prepare to search the HathiTrust Digital Library

You should now be at the screen shown below. Enter Buch der Lieder in the ‘Title’ field, Heine in the ‘Author’ field and 1920 in the ‘Publish Date’ field as shown below. Then click the “Search” button.

 

...

 

1.7 Select specific volume(s) from the search results

Select the first of the three volumes that show up, by clicking the checkbox next to ‘Select’, as shown below.

...

1.8 Prepare to view the selected volume(s)

Click on “Selected Items” from among the options at the upper right corner of the screen (as shown below).

Image Removed

1.9 Prepare to create a workset consisting of the selected volume(s)

The volume you selected now shows up as a selected item, as shown below. Click on the “Create/Update Workset” link that is within the grey area.

...

1.10 Name and describe the workset that is about to be created

 

Enter a name (for example, 'HeinePoetry') in the ‘Name:’ field as shown below. Enter a description in the ‘Description:’ field and set the availability to ‘Private’ or ‘Public’ as you prefer. Then click on the ‘Create’ button.

 

...

1.11 Verify that the workset has been created

You should see a message, as shown below, stating that the workset has been created.

Image RemovedIf you have not yet created a workset, the Portal & Workset Builder Tutorial for v3.0 includes information on creating an account and building a Workset.

Generate and execute the data file transfer script

 

...

Go to the Algorithms page of the HTRC Portal

...

Click on the ‘Portal’ link, which is at the top right corner of the screen.

...

2.1 Prepare to execute an algorithm

 

...

While logged into the HTRC Portal with the account you created your workset with, click on the 'Algorithms' link , which is near the top of the screen.

 



...

 

From the list of algorithms that shows up, click on EF_Rsync_Script_Generator. This is the algorithm  for generating the 'algorithm' generates a script for downloading the feature data files that correspond to your workset. 

...

Execute the EF_Rsync_Script_Generator algorithm

Specify a job name of your choosing. You also have to select a  workset that the EF_Rsync_Script_Generator algorithm will run against: Check the button saying “Select a workset from my worksets” and select your desired workset.  Your screen should now look like the figure below. At this point, click the ‘Submit’ button.


 

...

Check the status of the EF_Rsync_Script_Generator algorithm's execution

...

You can now see the status of the job, as shown below. The status of the job will initially show as “Staging”. (Refresh the screen after some time and you will see the status to have has changed to “Queued”. )

 

...

Open the completed job

Eventually, the job will have “completed”, and the screen, on refreshing, will look as follows. Click on the link representing the job name.

 

...

Download the results returned by the EF_Rsync_Script_Generator algorithm

At this time, the screen should look like the following:

At the very bottom left of your browser window, you will see a message like the following. (The number you see within the parentheses may vary, depending on how many times you have executed this step before. If doing this step for the first time, there will be no parentheses.) Press the “Keep” button.


 

...

 At this point, the script will be downloaded to your computer’s hard disk, and you will see the message at the bottom left of your browser window be replaced by just the name of the downloaded file:

...

Run the script returned by the EF_Rsync_Script_Generator algorithm

Windows users please note: Before proceeding, Windows users will need to complete additional steps to prepare their machine to work with rsync. Please follow the directions here.

 

After you download the script, from the command line navigate to the directory where the script file is located. This directory will typically be called Downloads, though the location may be different depending on your machine and if you have moved the file. Here is an example:

...

If your workset contained N volumes with HathiTrust volume IDs V1, V2, V3,... VN respectively, then executing the shell script as shown above will cause the following feature data files for the corresponding volumes to be transferred to your computer’s hard disk via rsync: V1.json.bz2, V2.json.bz2, V3.json.bz2, ..., VN.json.bz2

For the The workset in this our example , because it only contained only one volume, the book  Buch der Lieder by Heinrich Heine with the HathiTrust volumeID mdp.39015012864743, the script will transfer one file to your machine: . The corresponding file is called mdp.39015012864743.json.bz2.

...

(Optional) Uncompress the downloaded files

Because the feature data files are compressed, you may need to uncompress them into JSON-formatted text files, depending on your need. The compression used is bzip2. If you are using the files with the HTRC Feature Reader, the library will deal with the compression automatically.