readholdings

Readholdings is a service providing a global view of several accounts used by EbscoHost's HLM tool. The data is saved in an elasticsearch index and can be viewed on kibana dashboards.

Table of content

• Description
• Network-flow
• Installation
  • Development
    • Prerequisites
    • Start
    • Tests
  • Deployment
    • Prerequisites
• Data update

Description

A web interface is used to communicate with an API. It is possible to insert data, track insertions, manage ElasticSearch indexes and manage the application's health status.

Data updates are automated using an application cron job. This launches a job that generates, downloads, reads and transforms the data, then inserts it into the snapshot generated by the Ebsco API in version 2.

Network-flow

readHoldings is made up of several services which are distributed in several docker containers.

Data format

There are 4 types of field :

• id: the id is a concatenation of Portal, VendorID, PackageID, KBID, seperated by "_"
• standard from standard export from Ebsco API
• kbart2 from kbart2 export from Ebsco API
• meta which are custom fields created when data is inserted into elastic
  • access_type: If the holding (KBID) is in DOAJ or DOAB, it will be 'F' (Free); otherwise, it will be 'P' (Paying).
  • BibCNRS: Indicate which CNRS portal this data belongs to
  • createdAt: Date the data was inserted in elastic
  • EmbargoMonth: Unify EmbargoMonth from different time units in month
  • CustomEmbargoMonth: Unify CustomEmbargoMonth from different time units in month
  • ManagedCoverageBegin: Transform the array into a string, replacing 'Present' with the 31st of December of the current year and 'No Access' with '0001-01-01'
  • ManagedCoverageEnd: Transform the array into a string, replacing 'Present' with the 31st of December of the current year and 'No Access' with '0001-01-01'
  • CustomCoverageBegin: Transform the array into a string, replacing 'Present' with the 31st of December of the current year and 'No Access' with '0001-01-01'
  • CustomCoverageEnd: Transform the array into a string, replacing 'Present' with the 31st of December of the current year and 'No Access' with '0001-01-01'

Installation

Development

Prerequisites

The tools you need to let readholdings run are :

• docker
• npm

Commands

# install dependencies
npm i

# Execute this one time
docker-compose -f docker-compose.debug.yml run --rm elastic chown -R elasticsearch /usr/share/elasticsearch/ 

# Start ReadHoldings as daemon
docker-compose -f docker-compose.debug.yml up -d

# Stop ReadHoldings
docker-compose -f docker-compose.debug.yml stop

# Get the status of ReadHoldings services
docker-compose -f docker-compose.debug.yml ps

Deployment

Prerequisites

• docker
• Total extracted data does not exceed 500 000 rows

Environment variables

You can create an environment file named readholdings.local.env.sh and export the following environment variables. You can then source readholdings.env.sh, which contains a set of predefined variables and is overridden by readholdings.local.env.sh.

Adjust system configuration for Elasticsearch

Elasticsearch has some system requirements that you should check.

To avoid memory exceptions, you may have to increase mmaps count. Edit /etc/sysctl.conf and add the following line :

# configuration needed for elastic search
vm.max_map_count=262144

Then apply the changes :

sysctl -p

Start/Stop/Status

Before you start readholdings, make sure all necessary environment variables are set.

# Start readholdings as daemon
docker-compose up -d

# Stop readholdings
docker-compose stop

# Get the status of readholdings services
docker-compose ps

Tests

No test are available (coming soon).