Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Request DescriptionZip Structure LayoutExplanation of Entries

Retrieve volumes

concat=false

volumes.zip

    |-- foo.001122/

    |    |-- 00000001.txt

    |    |-- 00000002.txt

    |    |-- 00000003.txt

    |    |-- 00000004.txt

    |    |-- 00000005.txt

    |    \-- mets.xml

    |-- bar.ark+=13960=t123/

    |    |-- 00000001.txt

    |    |-- 00000002.txt

    |    |-- 00000003.txt

    |    \-- mets.xml

    \-- ERROR.err

 

 

Because the request parameter concat=false, each volume has its own directory, and the pages and metadata documents of each volume are individual files stored under the volume directory.

foo.001122/ is a directory named after the first volume, foo.001122.  The directory name underwent a Pairtree clean process, but since it does not contain any filesystem unsafe characters, the cleaned ID looks the same as the original.

Inside foo.001122/, files 00000001.txt through 00000005.txt are the 5 pages of this volume

mets.xml will also be inside of foo.001122/ if the request parameter mets=true

bar.ark+=13960=t123/ is a directory named after the second volume, bar.ark:/13960/t123.  The directory name underwent a Pairtree clean process so that filesystem-unsafe characters such as colons ':' and slashes '/' are escaped and replaced with filesystem-safe characters.

Inside bar.ark+=13960=t123/, files 00000001.txt through 00000003.txt are the 3 pages of this volume.

mets.xml will also be inside of bar.ark+=13960=t123/ if the request parameter mets=true

ERROR.err is a file at the top level.  It is present if the request encountered some errors and the detailed error information is stored in this file.  In this example, its presence is caused by the request for a non-existent volume gon.000000

Retrieve volumes

concat=true

volumes.zip

    |-- foo.001122.txt

    |-- foo.001122.mets.xml

    |-- bar.ark+=13960=t123.txt

    |-- bar.ark+=13960=t123.mets.xml

    \-- ERROR.err

Because the request parameter concat=true, each volume is a single text file, where the pages of the volume are concatenated into the file in the page order.

foo.001122.txt is the text file entry for the volume foo.001122.  The filename underwent a Pairtree clean process, but since it does not contain any filesystem unsafe characters, the cleaned ID looks identical to the original.

foo.001122.mets.xml will be present if the request parameter mets=true.

bar.ark+=13960=t123.txt is the text file entry for the volume bar.ark:/13960/t123.  The filename underwent a Pairtree clean process, so filesystem-unsafe characters such as colons ':' and slashes '/' are replaced with filesystem-safe characters.

bar.ark+=13960=t123.mets.xml will be present if the request parameter mets=true.

ERROR.err is a file at the top level.  It is present if the request encountered some errors and the detailed error information is stored in this file.  In this example, its presence is caused by the request for a non-existent volume gon.000000

 

 

 

Retrieve pages

concat=false

pages.zip

    |-- foo.001122/

    |    |-- 00000001.txt

    |    |-- 00000002.txt

    |    |-- 00000003.txt

    |    |-- 00000004.txt

    |    |-- 00000005.txt

    |    \-- mets.xml

    |-- bar.ark+=13960=t123/

    |    |-- 00000001.txt

    |    |-- 00000002.txt

    |    |-- 00000003.txt

    |    \-- mets.xml

    \-- ERROR.err

The Zip stream returned from the Data API for page retrieval with the request parameter concat=false is very similar to that returned for volume retrieval with concat=false.  The difference is that only pages requested for will be included.

Retrieve pages

concat=true

pages.zip

    |-- wordseq.txt

    \-- ERROR.err

Because the request parameter concat=true, the returned Zip stream is a "sequence of words" where the content of all pages from all volumes is aggregated into a single text file entry named wordseq.txt.

ERROR.err is a file at the top level.  It is present if the request encountered some errors and the detailed error information is stored in this file.  In this example, its presence is caused by the request for a non-existent volume gon.000000

Note that there is no METS metadata returned because mixing METS metadata and page content into the word sequence could potentially contaminate the information in the word sequence file.

Token count

level=volume

tokencount.zip

    |-- foo.001122.count

    |-- bar.ark+=13960=t123.count

    \-- ERROR.err

Because the request parameter level=volume, the returned Zip stream contains the token count of each volume as an entry, and the name of the entry is the Pairtree cleaned volumeID with ".count" as the extension.

ERROR.err is a file at the top level.  It is present if the request encountered some errors and the detailed error information is stored in this file.  In this example, its presence is caused by the request for a non-existent volume gon.000000

Token count

level=page

tokencount.zip

    |-- foo.001122/

    |    |-- 00000001.count

    |    |-- 00000002.count

    |    |-- 00000003.count

    |    |-- 00000004.count

    |    \-- 00000005.count

    |-- bar.ark+=13960=t123/

    |    |-- 00000001.count

    |    |-- 00000002.count

    |    \-- 00000003.count

    \-- ERROR.err

 

Because the request parameter level= page, in the returned Zip stream, each volume is a directory whose entry name is the Pairtree cleaned volumeID, and each page of the volume is an entry under the directory, and the name of the page is the 8-digit zero-padded page sequence number followed by ".count" extension.

ERROR.err is a file at the top level.  It is present if the request encountered some errors and the detailed error information is stored in this file.  In this example, its presence is caused by the request for a non-existent volume gon.000000

Token Count Output Format and Sorting Order

Each token count output entry is a list of tokens and number of occurrences within the aggregation.  The token and its occurrence count is separated by a space character (0x20), and each token-occurrence pair is a line and is separated from other pairs by a new line character (0x0A).  However, if an aggregation does not contain any texts (e.g. an empty page), that particular entry will be empty.

sortBy & sortOrderToken Count OutputDescription
unspecified

orange 1

banana 2

acorn 2

apple 1

coconut 1

A-team 1

Xylophone 3

if the parameter sortBy is not specified, the returned result does not guarantee any ordering of the token-occurrence pairs, nor does it guarantee the same ordering of these pairs between any 2 runs with the exact same parameters

sortBy=token&

sortOrder=asc

A-team 1

Xylophone 3

acorn 2

apple 1

banana 2

coconut 1

orange 1

with ascending ordering on the tokens, the returned result is sorted using the UTF-8 value of the tokens in ascending order.  In this example, the tokens starting with capital letter "X" come before these starting with lower case letter "a".

sortBy=token&

sortOrder=desc

orange 1

coconut 1

banana 2

apple 1

acorn 2

Xylophone 3

A-team 1

this is the exact reverse of the case above

sortBy=count&

sortOrder=asc

A-team 1

apple 1

coconut 1

orange 1

acorn 2

banana 2

Xylophone 3

with ascending ordering on the occurrence count, the returned result is sorted using the count value in ascending order; however, when multiple tokens have the same count, the order is determined by the ascending ordering of the tokens.

sortBy=count&

sortOrder=desc

Xylophone 3

banana 2

acorn 2

coconut 1

apple 1

A-team 1

this is the exact reserve of the above case, and specifically, when multiple tokens have the same count, the order is determined by the descending ordering of the tokens.

Access Data API with OAuth2

While the Data API by itself does not enforce any security mechanism for authentication and/or authorization, it is typically deployed behind an OAuth2 Servlet Filter.  A client making request to the Data API through the OAuth2 Servlet Filter must first obtain a valid OAuth2 token from the token service, and present the token as an additional HTTP request header to the OAuth2 Servlet Filtered Data API.  Please refer to " Using WSO2 Identity Server as the OAuth2 Provider for HTRC" for details on the usage.