Installation over TeXLive 2016

[rpspringuel]

When installing Gregorio 4.2 over a TeXLive 2016, one ends up with two versions of the executable on one's system and depending on one's method of usage and PATH settings (both generally and for specific programs) the wrong version may end up getting used.

For the full blown installers, I'm configuring them to move the TL2016 executable aside, but preserve it so that during uninstallation it can be restored to its original location.

I also plan on adding a version check of the installer into the version check scripts which are executed during package loading. Thus even if the above doesn't work, the user will be alerted to the problem in an understandable manner.

I also may add a install-exec-hook so that the installation from source using make install works like the full blown installers.

[rpspringuel]

More details here: https://mail.gna.org/public/gregorio-devel/2016-07/msg00008.html

[henryso]

I would only say---if you do add the install-exec-hook---that it still works for people who use make install (probably with PREFIX set) to install for packaging purposes.

[rpspringuel]

I think I understand what you're concern is, but could you add a bit more detail so that I can be sure.

[henryso]

When distributions package things that are generally installed using make, they will typically call it with make PREFIX=/path/to/some/temp/directory install. This installs the application into /path/to/some/temp/directory which is then turned into the package. Typically, when doing this, you wouldn't want it to mess with any files on your system, like, for example, the TeX Live installation.

[rpspringuel]

That is what I thought you meant. I'm not sure how to get around it, but I'll make sure to pay attention to that issue when I get there. Thanks for the heads-up.

[rpspringuel]

Okay, after looking at Ubuntu, it appears that the gregorio executable from TeXLive 2016 should not be a problem. Therefore this is a specifically Mac issue.

To that end, this is the modification I'm proposing for the Makefile

1. Set up an automake conditional which is true if the platform is Mac (modeled on code here
2. Set up an automake conditional which is true if PREFIX is set to /usr/local (it's default value, we assume that any other value for PREFIX is a packaging installation and should not mess with the local system).
3. Set up an automake conditional which is true if TL2016 is installed.
4. Add a install-exec-hook target which executes mv /Library/TeX/texbin/gregorio /Library/TeX/texbin/gregorio-41 if all three of the above conditionals are true.
5. Add a uninstall-hook target which executes mv /Library/TeX/texbin/gregorio-41 /Library/TeX/texbin/gregorio if all three of the above conditionals are true.

Before I get to work on this, does anyone see any potential problems with this plan?

[henryso]

I think the plan looks OK, but we should also think about perhaps naming the executable (instead) with the version number. It would mean a change for people who run gregorio directly (as opposed to auto-compile) and in the documentation, but we could install a symlink or script to make that easier.

[eroux]

Well, I can't say it's really satisfying, but maybe we don't have a choice... Maybe asking the TeXLive list would be a good idea? Maybe they'll have other ideas?

[eroux]

Discussing with @OldClaudio (who has this problem), it seems that it would be a good idea to be able to configure the absolute path to Gregorio in the TeX document...

[eroux]

another idea is an env variable

[henryso]

Here's another, probably far-fetched, idea: we could put that path to the gregorio executable into a tex or lua file during install.

[eroux]

Hmm, this is an interesting idea... one problem I see is that what's listed in the shell_escape_commands is gregorio, so I don't know how it's going to combine with the idea...

[henryso]

Well, if we're going to allow configuring the path in the TeX document or via an environment variable, it would be the same interaction with shell_escape_commands, no? The only real alternative, then, is to instruct users how to configure their PATH correctly.

[rpspringuel]

The problem isn't simply one of configuring PATH correctly. My PATH is configured so that /usr/local/bin appears before /Library/TeX/texbin but TeXShop still uses /Library/TeX/texbin/gregorio. This makes it clear that TeXShop is rearranging PATH for itself so that it always looks in /Library/TeX/texbin first.

So far I've only seen the problem with TeXShop, but I'll admit that I haven't checked to see if other GUIs (like TeXworks) will have the same problem. If the problem is specific to TeXShop, I'll email its author to find out if there is anything we can do in the hidden settings.

[OldClaudio]

Configuring the path is not that simple; for Mac, for example, the
former installations from 4.0 on suggested to configure a new engine for
TeXShop, so that it could be usable in the "magic comments" so as to
auto configure TeXShop in order to use that engine instead of the other
ones when the user clicks on the Typeset button.
The engine is about like this:

#!/bin/tcsh

set path= ($path /usr/texbin /usr/local/bin)
xelatex -file-line-error -synctex=1 -shell-escape "$1"

Therefore the PATH is modified and configured to use both binary
folders; I switched the order of the path search by modifying the engine
this way (notice that the path setting without using the colon separator
is used also in the other engines that come with the original TeXShop
installation; the same applies to the $path spelling in place of the
usual $PATH one):

#!/bin/tcsh

set path= ($path /usr/local/bin /usr/texbin )
xelatex -file-line-error -synctex=1 -shell-escape "$1"

but without any difference in behavior; moreover the PATH environment by
itself already contains /usr/local/bin in the first position, so that
the one visible in the tcsh procedure is redundant. In facts on my
machine the PATH variable is

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin:/Users/claudio/bin

What I meant with an environment variable was a macro holding and empty
substitution text by default, but that holds the correct path including
a final / (assume it is called \gre@goriopath but the call by
shell_escape_coomands or os.exec luatex command, would have as argument
simply \gre@goriopath gregorio. This wuld be sufficient to pass an
absolute path the the OS so that non patches have to be made to the
local pre installed o newly installed "version-rcN".

It would be a simple action that of defining a non empty substitution
tex for the rc version, while leaving an empty substitution text in the
one distributed with TeXLive.

At the moment I solved my personal problem by changing name to the
TeXLive2016 binaries in both the /usr/local/texlive/2016/bin/
subdirectories. Of course this works fine, but it is a very inelegant
solution.

Claudio

On 02/08/2016 14:09, Henry So wrote:

Well, if we're going to allow configuring the path in the TeX document
or via an environment variable, it would be the same interaction with
|shell_escape_commands|, no? The only real alternative, then, is to
instruct users how to configure their |PATH| correctly.

[OldClaudio]

It certainly is so also with TeXstudio, probably also with Texmaker.

With TeXworks the list of the folders it searches in this order are

/usr/bin /bin /usr/sbin /sbin /usr/local/bin
/usr/local/texlive/2016/bin/x86_64-darwin

therefore this problem should not be present when TeXworks is being used
as the selected GUI.

I have an indirect proof that TeXworks works this way, because the
person I am working with uses TeXworks on a Windows platform. I can't
tell what is the path on that platform because I don't have available a
Windows one.

I agree that contacting Dick Koch is a good idea; he is very reactive
for what concerns user requests about TeXShop.
Please let me know if it might be useful that other Mac users (myself
included) second your request.

Claudio

On 02/08/2016 16:04, Br. Samuel Springuel wrote:

The problem isn't simply one of configuring |PATH| correctly. My
|PATH| is configured so that |/usr/local/bin| appears before
|/Library/TeX/texbin| but TeXShop still uses
|/Library/TeX/texbin/gregorio|. This makes it clear that TeXShop is
rearranging |PATH| for itself so that it always looks in
|/Library/TeX/texbin| first.

So far I've only seen the problem with TeXShop, but I'll admit that I
haven't checked to see if other GUIs (like TeXworks) will have the
same problem. If the problem is specific to TeXShop, I'll email its
author to find out if there is anything we can do in the hidden settings.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtDu3AlMtAcmdbjoFFMYoX3YudIqeD2ks5qb05SgaJpZM4JZFml.

[OldClaudio]

I forgot an important issue connected with my previous message: the
suggested macro could be set to a non empty string by means of a
gregorio.cnf file.
This cnf file should be initialized to an absolute path only when
distributing the RC versions.

This work around by means of a configuration file appears to me as the
simplest way to solve the problem on any platform and with any GUI. If
Dick Koch solves the problem with TeXShop, the problem remains with
other GUIs. The configuration file approach depends neither on the
operating system nor on the specific GUI.

All the best
Claudio

On 02/08/2016 14:09, Henry So wrote:

Well, if we're going to allow configuring the path in the TeX document
or via an environment variable, it would be the same interaction with
|shell_escape_commands|, no? The only real alternative, then, is to
instruct users how to configure their |PATH| correctly.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtDuxmYFnlroDoQc8k3jqRT27xsLSl2ks5qbzNmgaJpZM4JZFml.

[rpspringuel]

I just checked TeXStudio on my primary system (Mac), and it seems to work just fine, picking up the correct binary based on PATH order. TeXmaker and TeXworks, however, exhibit the same problem as TeXShop.

I'm in the middle of creating a Ubuntu 16.04 VM, but when that's done I'll do some more testing on other platforms to try and determine the extent of the problem. That should give us a better idea as to what sort of solution will be necessary to solve it (and who it would be useful to enlist for help).

[rpspringuel]

The following results come from installing 4.2.0-rc2 over TeXLive 2016 without making any customizations to the resulting setup:

On Mac (El Capitan):

• Command line: 4.2
• TeXShop: 4.1 (Using included LuaLaTeX, not our custom LuaLaTeX+se)
• TeXworks: 4.1
• TeXmaker: 4.1
• TeXStudio: 4.2

On Linux (Ubuntu 14.04, the 16.04 install didn't work properly so I have to start over another day):

• Command line: 4.2
• TeXworks: 4.2
• TeXmaker: 4.2
• TeXStudio: 4.2

On Windows (10) the 4.2.0-rc2 installer 1) post-pends the gregorio bin directory to PATH and 2) deletes the 4.1 executable. However, despite this it appears that all methods (command line, TeXworks, TeXmaker, and TeXStudio) are still looking for the 4.1 executable. Since installations over TeXLive 2015 (where our executable didn't exist in the TeXLive bin directory) worked fine, there must be an index of TeXLive executables which isn't getting updated when the installer deletes the included executable. I'm going to have to look more closely at this another time, especially since I know that some of our testers are using this setup just fine.

[OldClaudio]

Dear Br. Samuel, dear Élie, dear all,
I found a simple solution that is also conforming with engineering practice.

When you prepare yearly package for CTAN or an RC version of the github
installable intermediate version, you just call the "gregorio" binary
with a name obtained by merging the word "gregorio" with a string that
contains some form of the version or release candidate or beta
information provided this information is corresponding in a univocal way
with the indications of the tex, sty, lua scripts that the package
contain. Therefore as it is to day, the gregorio binary in the
TeXLive2016 installation has the exact name "gregorio" and the
gregoriotex.lua package calls it with that specific name (not conforming
ot my suggestion, but not contrasting it). The actual second release
candidate to GregorioTeX 4.2.0 contains this information that is also
used to distinguish the various gtex files produced by this software; of
course the string we should merge to the binary name can be 4_2_0-rc2 as
it is done with the gtex files, but it can be shortened, for example to
420rc2; this is the choice I made with my experiments. therefore the
TeXLive2016 binary has a full name of .../texlive/2016/bin/.../gregorio
(the ellipses are plaform dependent but during the lualatex run its the
GUI or kpathsea or the OS that completes the file name with its real
path); next year it might be: .../texlive/2017/bin/.../gregorio423. At
the same time the binary that the release candidate installer puts into
.../local/bin has a full name of .../local/gregorio420rc2; When one runs
lualatex on a given document with the release candidate installed, the
gregoriotex.lua file that will actually execute the binary will call
gregorio420rc2, not gregorio (or next year gregorio423) Not finding it
in the texlive tree it will find it with the OS path in the alternate
path, and no conflicts arise.

I did change the two instances of the luatex statement

local cmd = string.format("gregorio %s -W -o %s -l %s %s",
extra_args, gtex_file, glog_file, gabc_file)
into

local cmd = string.format("gregorio420rc2 %s -W -o %s -l %s %s",
extra_args, gtex_file, glog_file, gabc_file)

and correspondingly I changed the name of the executable in my
/usr/local/bin folder with the same suffix "420rc2", and with TeXShop
and the other GUIs I have, things run smoothly as they are supposted to.

I do not know how to automatically implement this minimal variation on
the luatex code in github (because I don't master the luatex language)
and the other scripts with which the binaries are produced, therefore I
hope that this description is sufficient for you to implement.

What I am suggesting does not depend on platform type or specific
operating system.
The important thong is that the users should not be supposed to do by
hand what I did on my Mac El Capitan platform, where administrator or
root privileges should be acquired before modifying protected files; may
be Linux users who frequently use their command line syntax are aware
of this important point and they know how to avoid fatal errors when
they use the command line with those privileges; probably expert Mac
user can do the same. But Window users generally don't even know that
there are hidden directories, those whose contents is somehow protected
form crazi errors, and would simply work with their mice.
It is evident that the whole production of successive versions of the
beautiful GregorioTeX package must be automatized; meanwile I am very
satisfied of this handmade solution, because it does not change anything
n the TeXLive installation, and it works smoothly; I warmly
suggest to implement it.

The best to all the readers of this message

Claudio

On 03/08/2016 04:19, Br. Samuel Springuel wrote:

The following results come from installing 4.2.0-rc2 over TeXLive 2016
without making any customizations to the resulting setup:

On Mac (El Capitan):

• Command line: 4.2
• TeXShop: 4.1 (Using included LuaLaTeX, not our custom LuaLaTeX+se)
• TeXworks: 4.1
• TeXmaker: 4.1
• TeXStudio: 4.2

On Linux (Ubuntu 14.04, the 16.04 install didn't work properly so I
have to start over another day):

• Command line: 4.2
• TeXworks: 4.2
• TeXmaker: 4.2
• TeXStudio: 4.2

On Windows (10) the 4.2.0-rc2 installer 1) post-pends the gregorio bin
directory to |PATH| and 2) deletes the 4.1 executable. However,
despite this it appears that all methods (command line, TeXworks,
TeXmaker, and TeXStudio) are still looking for the 4.1 executable.
Since installations over TeXLive 2015 (where our executable didn't
exist in the TeXLive bin directory) worked fine, there must be an
index of TeXLive executables which isn't getting updated when the
installer deletes the included executable. I'm going to have to look
more closely at this another time, especially since I know that some
of our testers are using this setup just fine.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtDu4Txyngv5ECbtE32hvnY-NDCU5-Nks5qb_q0gaJpZM4JZFml.

[eroux]

This sounds like a good idea, maybe the gregorio executable from github would have a version name and the TeXLive one would not. That would mean though that users using the github version would have to pass the --shell-escape switch. This sounds like an acceptable limitation...

[rpspringuel]

How about always naming the executables with version number as @OldClaudio suggests and adding a symlink called simply gregorio in texbin which points to the current version. The TeXLive build could just be gregorio to start (thus not making any additional work for the TeXLive people) and then when one of our intermediate installers comes along, it would rename that executable to attach the version number and add the symlink to the executable being installed. This should get around the need for the --shell-escape switch and would leave the lua code untouched.

[henryso]

That's pretty much what I suggested we consider two days ago. Put the version number on the executable and (perhaps) supply a symlink.

Edit: Also, symlinks are weird under Windows, so that needs to be considered as well.

[OldClaudio]

I see some problems with symlinks on vintage versions owWindows
operating systems; I am sure such systems use them from Win8 on, I am
not sure with Win7; earlier versions such as WinXP and WinVista don't.
Claudio

On 03/08/2016 23:30, Henry So wrote:

That's pretty much what I suggested we consider two days ago. Put the
version number on the executable and (perhaps) supply a synlink.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtDu_lF4rEaYqRgGQErr4mWu7jyhb70ks5qcQhugaJpZM4JZFml.

[rpspringuel]

@henryso, I forgot that you had made that suggestion. Sorry for not giving you proper credit.

@OldClaudio, the mklink command is available from Windows Vista onwards, so we'd only be loosing support for Windows XP if we went this route. Given that XP is no longer supported by Microsoft, I'm not adverse to no longer supporting it in our installer.

Even without installer support, XP users could still make use of Gregorio by copying the executable into texbin and renaming it.

[henryso]

mklink also only works under NTFS, but that's really only an issue if trying to install on smaller FAT-based flash drives.

An alternative may be a batch file.

On the opposite side, it may be sufficient to just put the version number on the executable. How many people still run the executable directly? Those who do can probably adapt their scripts with the version number.

[eroux]

TeXLive has a special way to handle symlinks, they build some .exe, I'm affraid we can't really hack into this system, that would take a huge amount of time and disussion... so having a version number in non-TeXLive binaries and ask users to use the --shell-escape looks like the most simple solution

[OldClaudio]

I agree with Èlie's last comment. In facts I tried to use a simlink, but
it did not work on my Mac; I had to restore everything as I wrote
yesterday (with a renamed executable in /usr/local/bin/) and with a
ranemed executable name in both instances of the cmd definition within
the gregoriotex.lua file, and of course using the shell escape.
This in spite of having gregorio listed in the texmfcnf.lua file for
safe execution in protected mode.

Neglecting those who run gregorio from the command line, the RC
installed versions installed from github produce a completely
transparent working environment that requires only the shell escape
setting. Definitely the simplest solution.

Claudio

On 04/08/2016 08:23, Elie Roux wrote:

TeXLive has a special way to handle symlinks, they build some .exe,
I'm affraid we can't really hack into this system, that would take a
huge amount of time and disussion... so having a version number in
non-TeXLive binaries and ask users to use the |--shell-escape| looks
like the most simple solution

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtDu0VRBl4g9xcOJjHSQebvcq9ybDPdks5qcYVtgaJpZM4JZFml.

[henryso]

Is shell_escape_commands the thing preventing us from putting the version number in the executable name for all releases (including TeX Live)?

Edit: Perhaps we should abandon putting gregorio in shell_escape_commands, put the version number in the executable for all releases, and suggest using --shell-escape when using gregorio, for consistency in development, release, and instruction?

[eroux]

I think the gregorio command should stay in the shell_escape_commands, or maybe shell_escape_commands could include gregorio-versionnumber... But I don't really like the idea of removing it completely: having it has two positive aspects:

• it lowers the entry barrier: people can just start using gregorio by installing TeXLive and running their file, no need to mess up with a complex setting such as shell_escape
• it allows stock TeXLive installations to run Gregorio, like ShareLaTeX or Overleef... there's no short term benefit to it, but I believe that in the long term this could have a very positive impact: if we have one day a link on the gregorio website to an overleef template of a booklet, I'm quite sure many more people will be able to use gregorio!

[henryso]

Do you know how much of an ask it is to have the versioned gregorio in shell_escape_commands? It would simplify, most especially, giving people instructions.

[eroux]

I think it's quite ok... but that will be for TeXLive 2017 I'm affraid...

[henryso]

Of course, but for now, we can start tacking on the version number for 4.2, which will ease installation over TeX Live 2016 (and is the exact solution used by @OldClaudio). Then once TeX Live 2017 releases, things will be more standard and easier to explain. I think the number of people running the gregorio executable directly is growing smaller anyway, and those people can always create a script or symlink to deal with it if they don't want to play with their scripts.

[eroux]

That's fine for me, it should be made very explicit in the CHANGELOG and UPGRADE, but no objection

[rpspringuel]

Okay, in order to implement this solution I need to do the following (correct me if I'm wrong):

• Modify the Makefile.am in src so that the compiled executable is named gregorio-<VersionNumber>and register this file with VersionManager.py
• Modify gregoriotex.lua to call gregorio-<VersionNumber> instead of gregorio.
• Since the previous change nullifies the need for the version check of the executable which I had implemented, revert that change.
• Revert the changes to the Mac and Windows installers which were changing the name of the texbin executable.
• Modify installers to account for the version number being attached to the executable.

[henryso]

Sounds about right to me.

[henryso]

Just noting that this fix probably obsoletes the need for a Debian diversion in #1106.

[rpspringuel]

I tried following this StackExchange question in order to change the Makefile (using @FILENAME_VERSION@) but the compiled executable is still coming out as gregorio. Does anyone have any advice for what else I might need to do here?

[henryso]

I think @PACKAGE_VERSION@ is an innate autoconf thing. You can try the third answer, or you can just put the version number directly in the filenames and let VersionManager.py handle changing them. If you get stuck, I can give it a shot tomorrow.

[rpspringuel]

Remaining to be done:

• Modify system-setup.command
• Modify system-setup.bat.

[henryso]

I guess everything for this is done. Please reopen if I'm not correct.