Skip to content

ToolboxInstallation

Jump to bottom Edit New page
Hugo Oliveira edited this page Jan 30, 2020 · 33 revisions

Toolbox Installation

This page provides an overview of the steps required to get the IMOS Toolbox up and running.

Updating your Installation via Git

If you are using git, need to update to the latest stable release, and has no local changes in the toolbox code that need to be kept, you can do the following within the toolbox root folder:

git fetch origin
git checkout 2.6
git pull

If you have local changes, git may block the checkout since some files would be overwritten. Git will warn you about stashing or committing the changes if this is the case. A simple way to circumvent this is to create a new branch to save your changes:

git checkout -b my_local_changes
git add . 
git commit -m 'my local changes before update to version 2.6.1'
git checkout 2.6

After that, you can resolve the conflicts from your 'my_local_changes' branch the way you find more appropriate. For example, a simple diff can be viewed with:

git diff my_local_changes

You can then change the files manually, as in here, or you may want to merge the branch 2.6 into the my_local_branch, and resolve conflicts (if any):

git checkout my_local_changes
git merge 2.6
#Resolve conflicts
git merge --continue

This way, you will be located at your local branch ('my_local_changes') with merged updates from the latest stable release. Please note that conflict resolving may be problematic if changes are too complex - you need to be familiar with diffs, the way git mark the conflicts, and git conflict resolution.

Standalone installation

Requirements

The stable version requires:

  • Windows or Linux (64bit operating system)
  • The Matlab Component Runtime R2018b for your architecture - Win64 or GLNXA64 (Linux). The installer is called MCR_R2018b_{ARCH}_installer, with version tag 9.5 or v95). click here to download
  • An disk partition win long filename support (filename > 160 characters long) (windows only).

For reading metadata from a database:

  • A database with a proper schema and data entries - also known as a deployment database). We provide a basic template here.
  • Java JRE (1.6 or newer) to be able to query the deployment database with the toolbox.

Optional:

  • Microsoft Access 64 bit (the UCanAccess library is used otherwise - e.g. Linux).

The Matlab Component Runtime

The library provides all the hooks and dependencies to use the GUI and other Matlab internal functions within the Toolbox application. A hard-requirement is to install the correct version associated with the toolbox releases:

  • For installations between v2.3 >= imos-toolbox < v2.6.1 , use the MCRInstaller_8.0.
  • For versions >=2.6.1 : MCRInstaller_R2018b_v9.5

You only need to run the installer once.

For Windows, it comes as a straightforward executable.

For Linux, you will need to execute the install application within the zip file. You may want to install the library as root or as a user. During installation, you might receive a warning about requiring the .NET Framework - you may safely ignore it.

Installing with root credentials in the default installation path may automatically include the MCR applications into your PATH (but this is not guaranteed since it can be distribution dependent).

I recommend installing as a user within your local home folder. However, you will need to update your PATH environment. If using the bash shell, you may want to include this into your .bashrc file (or equivalent) after the installation finishes:

MCR_ROOT="$HOME/apps/MATLAB/MATLAB_Runtime/v95"
MCR_APPS="$MCR_ROOT/bin:$MCR_ROOT/glnxa64"
export PATH="$PATH:$MCR_APPS"

Stand-alone application Startup

Windows:

Start the toolbox by double-clicking on imosToolbox_Win64.bat. The script will execute imosToolbox_Win64.exe within a prompt shell, so any messages thrown by the toolbox software are visible. This is the recommended way, since executing the .exe file will not provide any information, warning or error message that may be useful.

Linux:

A sh script imosToolbox_Linux64.sh is provided to setup the correct library paths to execute the stand-alone binary file. You will need to provide the root folder of the Matlab Component Runtime as a first argument to the script:

./imosToolbox_Linux64.sh $MCR_ROOT

where $MCR_ROOT is the deployed MCR root path (e.g. /home/ggalibert/apps/MATLAB/MATLAB_Compiler_Runtime/v95)

The IMOS Toolbox within Matlab (as a library/GUI)

Another usage option is to use the Toolbox within Matlab, as a library and/or GUI. You will need to call the imosToolbox function within the command-line interface to start the graphical interface.

Requirements

For stable releases after v.2.6.1, it's required to have a valid Matlab R2018b or newer installation.

Installation

There are two options:

  • Obtain the latest stable release as a zip file, as required for the Stand-alone application.

  • Checkout the stable, tagged version repository from gitHub, using git:

cd PATH_TO_WHERE_I_WANT_THE_TOOLBOX/
git clone https://github.com/aodn/imos-toolbox.git
cd imos-toolbox
git checkout 2.6

Most of the users should use the first method. However, this requires you to regularly re-download the package at every major (minor) stable (hotfix) release. The latter option, using git, allows you to keep a local copy of the repository and regularly updated to any major/minor release. This requires some familiarity with Git.

Git can be easily installed on Linux, Mac and Windows. Git for Windows comes with Git BASH which provides a BASH environment like in any Linux/Unix/Mac terminal. Hence, the command above can be used to install the repository in both windows and Linux.

Startup within Matlab

You may want to add the root directory to your path:

TOOLBOX_ROOT='/the/path/to/the/imos_toolbox/'
addpath(TOOLBOX_ROOT)

before execute the imosToolbox function. Otherwise, you may go to the root directory and execute the function there.

Since version 2.5 there is no need to add the toolbox directory and it sub-directories to the Matlab path. This is automatically done every time you run the toolbox for the current Matlab session. If you plan to switch from one toolbox version to another, you will have to restart Matlab though.

Notes below are for information only, regarding previous versions or for development/debugging:

The toolbox root, and every subdirectory within the toolbox root must be on the matlab path. In other words, you must add the following list of directories to your matlab path, either by setting the MATLABPATH environment variable, or, within matlab, using the 'Set Path' dialog, which is accessed via 'File'->'Set Path'.

Assuming that the toolbox is installed in C:\imos-toolbox, you must add all the directories and subdirectories within the root path:

  • C:\imos-toolbox\
    • C:\imos-toolbox\AutomaticQC
    • C:\imos-toolbox\DDB
    • C:\imos-toolbox\FlowManager
    • C:\imos-toolbox\GUI
    • C:\imos-toolbox\Graph
      • C:\imos-toolbox\Graph\DepthProfile
      • C:\imos-toolbox\Graph\TimeSeries
      • C:\imos-toolbox\Graph\Transect
    • C:\imos-toolbox\IMOS
    • C:\imos-toolbox\NetCDF
    • C:\imos-toolbox\Parser
    • C:\imos-toolbox\Preprocessing
      • C:\imos-toolbox\Preprocessing\Transform
    • C:\imos-toolbox\Seawater
    • C:\imos-toolbox\Util
      • C:\imos-toolbox\Util\CellUtils
      • C:\imos-toolbox\Util\StructUtils
      • C:\imos-toolbox\Util\Schema

Be carefull not to keep links to several different toolbox versions in your Matlab path.

Regular updates with git

Every quarter and/or minor release, we test the toolbox for stability and merge changes into the current major version branch. If you wish to update to the latest stable release, you have to position yourself in the respective branch and pull the new changes:

git fetch origin
git checkout 2.6
git pull

The above will ensure you are at the "2.6" branch before pulling the changes locally into your directory. Now you have a clone of the 2.6 branch. Any critical fixes will require futures "pulls" or changes to a new branch (for example when a new major version branch happens).

Finally, we only encourange users to use the master branch if you are confortable with git, plan to contribute/develop the toolbox further. We are happy to accept Pull requests.

If you have created a GitHub account, you can watch the toolbox GitHub repo and be notified about anything that is happening related to the toolbox. You can watch for new releases by selecting "Releases Only" in the Watch button in the top-right corner of the main repository page. Websites like coderelease can also provide you with this service.

Conflicts

The toolbox has some configuration files that are open to change by any user. In this scenario, merge conflicts can occur: By pulling new changes, your config files will drift from the default options presented in the repo.

If a conflict appears, you will need to resolve it - see example of conflict.

Configure/Set the Deployment Database

If you seek to pull metadata information from a deployment database, you need to make sure the database is correctly configured. configure your database.

Since imos-toolbox version 2.0, in addition to the MS-Access deployment database (template provided in the downloads page), it is also possible to connect to any user-specific database, as long as its schema complies to minimum requirements.

Since version 2.5.8, a deployment database in CSV file format is supported. A template with example entries can be found in the downloads page).

Limitations have been observed such that it is not possible to open the current 32bit version of the deployment database with a Microsoft Access software 64bit. A workaround may be to convert the .mdb to .accdb. Be aware that once you convert to .accdb, you cannot go back to .mdb without a fair amount of manual copying to rebuild the .mdb.

MS Access database

The toolbox can read the MS Access deployment database using:

  • Microsoft ODBC driver,
  • UCanAccess JDBC driver.

Using the Microsoft Access ODBC drivers

This solution is only recommended when the architecture version of Microsoft Access matches the one of Matlab or standalone binary architecture. For example, if your Microsoft Access software is 32bit, but your Matlab is 64bit, then this won't work.

Operating system Microsof Access Matlab or standalone binary Compatibility
32bit 32bit 32bit Ok
64bit 32bit 32bit Ok
64bit 32bit 64bit KO
64bit 64bit 32bit KO
64bit 64bit 64bit Ok

The Microsoft Access software comes with a Microsoft Access ODBC driver which can be used to turn your MS Access file into an ODBC source following the described steps below:

  • Start
    • Control Panel
      • Administrative Tools
        • Data Sources (ODBC)
          • User DSN
            • Add
              • Driver: "Microsoft Access Driver (*.mdb)"
              • Data Source Name: "imos-ddb" (or whatever you wish to call it)
              • Select -> navigate to the ddb MS Access file

Once you have configured the ddb as an ODBC source, set the toolbox.ddb property (in toolboxProperties.txt) to the ODBC Data Source Name.

With UCanAccess JDBC drivers

Simply set the toolbox.ddb property (in toolboxProperties.txt) to your MS Access file full path.

This is the only solution for Linux operating systems, and when the above solution doesn't work.

The main drawback here is the amount of Java Heap memory needed by the Java Virtual Machine, and one may face a java.lang.OutOfMemoryError: Java heap space error message. The easiest fix is to tell Java to allocate sufficient heap space memory when the Java Virtual Machine starts up. This can be done in two ways:

  • Add a java.opts file in the root directory of the toolbox, that is to say next to imosToolbox.m, which includes (without the double quotes): "-Xmx2g". This way we're telling Java to allocate 2Gb of Heap memory. Then start either Matlab or any standalone binary executable from the root directory of the toolbox. This is the preferred method and only available one for use in standalone.
  • For non-standalone installation, in Matlab, via Preferences, in General -> Java Heap Memory and then set it to the maximum (2Gb is a minimum for AIMS database for example).

CSV files as Deployment databases

A deployment database template (existing entries are provided as examples) is available here and follows the structure of the MS-Access deployment database schema.

Similarly to the case above, You need to inform the toolbox where the Deployment database is by setting the toolbox.ddb entry in the toolboxProperties.txt configuration file:

toolbox.ddb = path_to_the_directory_where_you_keep_the_csv_files

Other databases

Your deployment database architecture has to follow a minimum interface requirements to prevent the toolbox from crashing. Then to be able to query it through the toolbox, you must put the driver (.jar file) used to connect to it in the Java directory and set the relevant toolbox.ddb.* properties (in toolboxProperties.txt) properly. Since version 2.5, postgresql-9.4-1201.jdbc4.jar is included by default since CSIRO uses Posgresql for their deployment database.

Contributing

Please feel free to send any enhancements or proposals in the github issues page.

Pull Requests are also welcome but please be aware that we need to maintain consistency, check compatibility, and test the features before we merge then into the main repository. This may take some time, depending on the proposed changes, code consistency and testing.

If you would like to do further development for Push requests, we recommend you create a fork first and work from there, preferably from the master branch or the stable release (so the number of conflicts/rebases are reduced).

Using the Toolbox

See the overview on using the toolbox to get started.

Add a custom footer
Add a custom sidebar
Clone this wiki locally