Skip to content

Latest commit

 

History

History
455 lines (349 loc) · 15 KB

HACKING

File metadata and controls

455 lines (349 loc) · 15 KB
Raw

This HACKING file describes the development environment. -*- org -*-

Copyright (C) 2008, 2009, 2011 ViewPlus Technologies, Inc. and Abilitiessoft, Inc. Copyright (C) 2012, 2013, 2014,2015 Swiss Library for the Blind, Visually Impaired and Print Disabled

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

This file attempts to describe the maintainer-specific notes to follow when hacking liblouis.

Table of Contents

Developing

Where to get it

The development sources are available through git at github.com:

https://github.com/liblouis/liblouis

Build requirements

To build Automake, Autoconf, and Libtool are used. If you are getting the sources from git (or change configure.ac), you’ll need to have these tools installed to (re)build. Optionally (if you want to generate man pages) you’ll also need help2man. All of these programs are available from ftp://ftp.gnu.org/gnu.

If you want to run the YAML based test suite you will have to install libyaml.

On Mac OS, the programs can be optained with Homebrew (http://brew.sh):

$ brew install automake libtool pkg-config texinfo

Note that if you are using Homebrew to install liblouis (see below), the build dependencies are installed automatically.

Gnulib

Gnulib (http://www.gnu.org/software/gnulib) is used to provide portable basic functionality to programs and libraries. We use two instances of gnulib, one to provide portable functions such as malloc, strndup, etc to the library and another one to provide portable functionality such as getopt, progname or version-etc to the tools.

The first time invocation to import gnulib for the library was

$ gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files \
	      malloc-gnu realloc-gnu strndup

and for the tools

$ gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files \
	      getopt-gnu malloc-gnu progname version-etc

More modules might have been added since. The currently-used gnulib modules and other gnulib information are recorded in gnulib/m4/gnulib-cache.m4 and tools/gnulib/m4/gnulib-cache.m4.

If you want to update from the current gnulib, install gnulib, and then run the following commands in the top-level directory.

$ gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files
$ gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files

How to build

After getting the sources from git, with

$ git clone https://github.com/liblouis/liblouis.git

and installing the tools above, change to the liblouis directory and and bootstrap the project with the following command

$ ./autogen.sh

to do a fresh build. Then run configure as usual:

$ ./configure

You have the choice to compile liblouis for either 16- or 32-bit Unicode. By default it is compiled for the former. To get 32-bit Unicode run configure with --enable-ucs4.

After running configure run make and then make install. You must have root privileges for the installation step.

Install with Homebrew

Homebrew (http://brew.sh) is a package manager for Mac OS X that installs software from source. There is nothing special about the installation process in the sense that under the hood it happens exactly as described above, with the only difference that Homebrew automates it completely.

First, use the brew tap command to add the repository that includes the liblouis formula:

$ brew tap liblouis/liblouis

Now you are ready to install liblouis:

$ brew install liblouis

Docker

Docker (https://www.docker.com) can be useful both for creating a development environment for liblouis, and for shipping the application.

Setting up a developer environment can take long and can be problematic especially for Windows. Thanks to Docker we can set up the environment for you, we can easily distribute it as an image, which can be run by anybody and will behave exactly the same for everybody.

In order to use Docker images of liblouis, first get Docker at http://docs.docker.com/introduction/get-docker. Download the latest liblouis image with:

$ docker pull liblouis/liblouis

Then, enter the development environment by running the image in a Docker container:

$ docker run -it liblouis/liblouis bash

Local files and directories can be “mounted” inside the container, in order to make it easier to edit files and to persist changes across runs. For example, to use local table files:

$ docker run -it -v $(pwd)/tables:/tmp/liblouis/tables liblouis/liblouis bash

See the Docker documentation for more info.

The same Docker image can be used as a development environment and as the application itself. For example, to run the lou_translate tool from inside a Docker container:

$ docker run -it liblouis/liblouis lou_translate en-us-g1.ctb

To rebuild the image yourself, run the following command in the root directory of the liblouis source:

$ docker build -t liblouis/liblouis .

A .dockerignore file is required if you want to compile the source both on te host and in the Docker container. The .dockerignore file can be updated from .gitignore with:

$ make .dockerignore

How to run tests

Tests are run with

$ make check

How to debug

First you have to build liblouis with debugging info enabled.

$ ./configure CFLAGS='-g -O0 -Wall -Wextra'
$ make

Starting the programs under the tools directory within gdb is a little tricky as they are linked with libtool. See the info page of libtool for more information. To start lou_checktable for table wiskunde.ctb for example you’d have to issue the following commands:

$ libtool --mode=execute gdb ./tools/lou_checktable
(gdb) run tables/wiskunde.ctb

How to find memory leaks

Valgrind is a tool that can be used to find memory errors. It is recommended that you compile liblouis without any optimizations and with all warnings enabled before running it through Valgrind:

$ ./configure CFLAGS='-g -O0 -Wall'
$ make

Then use Valgrind to analyze liblouis. For example you can run lou_translate trough Valgrind:

$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --leak-resolution=high --log-file=valgrind.log \
  ./tools/lou_translate en-us-g2.ctb

Type a few words at the prompt, check translation and terminate lou_translate. Now open the file valgrind.log and see if there are any memory leaks reported.

You can also just run lou_checktable for example:

$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --leak-resolution=high --log-file=valgrind.log \
  ./tools/lou_checktable tables/nl-BE-g0.utb

Again open valgrind.log to see if any memory leaks were reported.

For the full experience run lou_allround under Valgrind:

$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --show-reachable=yes \
  --leak-resolution=high --track-origins=yes \
  --log-file=valgrind.log ./tools/lou_allround

AddressSanitizer

AdressSanitizer is a memory error detector for C/C++. It is part of both LLVM and gcc. To check liblouis build it as follows:

$ ./autogen.sh
$ ./configure CFLAGS='-fsanitize=address -O1 -fno-omit-frame-pointer -g'
$ make
$ ./tools/lou_translate tables/en-ueb-g2.ctb

Run a few translations, end the program and marvel at the output of AddressSanitizer. At the time of this writing it complained pretty hard:

SUMMARY: AddressSanitizer: 12384 byte(s) leaked in 12 allocation(s).

Electric Fence

Electric Fence is a tool that helps detect memory access that overruns the boundaries of a malloc() memory allocation, and access to memory that has been released with free().

Under Debian the usage is fairly straightforward:

$ sudo apt install sudo apt install electric-fence

Then compile as above and invoke the debugger as above. Inside gdb set up LOUIS_TABLEPATH and LD_PRELOAD as follows:

$ libtool --mode=execute gdb ./tools/lou_allround
(gdb) set environment LOUIS_TABLEPATH ./tables,./tests/tables,./tests/tables/moreTables,./tests/tablesWithMetadata,./tests/tables/emphclass
(gdb) set environment LD_PRELOAD /usr/lib/libefence.so.0.0
(gdb) run

If there are problems with memory access the program will run into a segmentation fault which you can consequently analyze in the debugger.

Static analysis

scan-build is a command line utility to run a static analyzer over the codebase. It helps to find problems such as memory leaks, dereference of null pointer, etc.

$ make maintainer-clean
$ ./autogen.sh
$ scan-build ./configure
$ scan-build make
...
scan-build: 101 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2019-08-15-141827-5185-1' to examine bug reports.

then look at the generated reports and fix (or report) the issues.

How to analyze performance

Gprof helps you analyze the performance of programs. You have to compile liblouis as follows:

$ ./configure --disable-shared
$ make clean all CFLAGS='-g -O0 -pg' LDFLAGS='-all-static'

Then translate some stuff with a large table:

$ ./tools/lou_translate tests/tables/large.ctb

Finally look at the call profile:

$ libtool --mode=execute gprof ./tools/lou_translate gmon.out

How to build for Windows

See the README.windows file and the windows subdirectory.

How to cross-compile for Windows

To compile for either win32 or win64 two Dockerfiles are provided which set up the right environment to cross-compile liblouis. There is a Makefile target that will build the Docker images, build liblouis and extract the zip files with the windows build artifacts.

To build for both win32 and win64 type the following command:

$ make distwin

This will place liblouis-$(VERSION)-win32.zip and liblouis-$(VERSION)-win64.zip in the current directory. The build is compiled with --enable-ucs4 and contains lou_checkyaml.exe.

How to build for Windows using Cygwin

(possibly use a Vagrantfile as demonstration + explain that Cygwin binaries can not be used outside the Cygwin environment)

How to build for Windows using MinGW

(possibly use a Vagrant file as demonstration)

Release Procedure

These steps describe what a maintainer does to make a release; they are not needed for ordinary patch submission.

Update the NEWS entry

If any new tables were added, renamed or removed please note them with their file name in the NEWS entry. This is usefull for projects like NVDA.

Set the version number

Update the version number in NEWS (with version, date, and release type), configure.ac and windows/include/config.h.

Don’t forget to update the libtool versioning info in configure.ac, i.e. LIBLOUIS_REVISION and possibly LIBLOUIS_CURRENT and LIBLOUIS_AGE. Check the libtool documentation on versioning for an explanation of these values.

Commit and tag

Commit the changes and tag this version

$ git tag -s v2.6.0 -m "Release 2.6.0"
$ git push origin v2.6.0

If you know the exact version number that needs to be tagged use

$ git tag -s v2.6.0 -m "Release 2.6.0" <commit>
$ git push origin v2.6.0

Make the release

Check out a clean copy in a different directory, like /tmp. Run autogen.sh and configure with no special prefixes. Run make distcheck. This will make sure that all needed files are present, and do a general sanity check. Run make dist. This will produce a tarball.

$ ./autogen.sh && ./configure && make && make distcheck && make dist

Upload

Add the tarball to the github liblouis releases page, i.e. add it under https://github.com/liblouis/liblouis/releases with the specific release and add a link to it in $WEBSITE/downloads/index.md. See below for instructions on how to update the web site.

Online documentation

The online documentation is part of the liblouis web site. To add it to the site simply copy doc/liblouis.html to $WEBSITE/documentation/liblouis.html. Make sure you add the proper YAML front matter. Again see below for instructions on how to update the web site.

Web site maintenance

The liblouis web site at liblouis.io is maintained with the help of github pages. To edit the site just check out the repo at https://github.com/liblouis/liblouis.github.io. You’ll need to know a few things about Jekyll and markdown, the markup that is used to edit the content. In order to update the site simply edit, commit and push.

For the new release update the project web site.

  • Add a post containing the current NEWS to the _posts directory and
  • add the download artifacts to the download page

Announce

Send an announcement to the liblouis list liblouis-liblouisxml@freelists.org. See ANNOUNCEMENT for an example.