Create a Polymake pexpect interface

[simon-king-jena]

There is PyPolymake, to be added at #21170. But I think it wouldn't hurt to additionally have a pexpect interface to Polymake. And the purpose of this ticket is to create one.

Depends on #22501

CC: @videlec @jplab

Component: packages: optional

Keywords: Polymake, days84

Author: Simon King

Branch: 52a77ee

Reviewer: Vincent Delecroix

Issue created by migration from https://trac.sagemath.org/ticket/22452

[simon-king-jena]

Changed keywords from Polymake, days85 to Polymake, days84

[simon-king-jena]

Branch: u/SimonKing/create_a_polymake_pexpect_interface

[videlec]

Commit: c3add96

[videlec]

Last 10 new commits:

33b53f3	Add documentation to _get_primitive
d78b239	'Implement string representation by single underscore methods': Apply this mantra to interfaces
9a0bcca	Change string representation of invalid interface elements whose parent is None
6910107	Keep the tests that have been removed in the previous commits
cd97ff3	Parse error message of _check_valid() in order to cope with unpredictable future semantical changes
fdaeb2e	Include error message of validity check into string representation of invalid interface elements
cdb085b	Revert introduction of _get_custom_name
b5df135	Actually remove _get_custom_name, not only its applications
6b53229	Reverse addition of _get_primitive. Fix `__cmp__`, bool etc. and their documentation
c3add96	Initial wrapper: providing help, tab completion, accessing members, calling functions, nice(r) string representation

[simon-king-jena]

comment:4

Here I am showcasing some things that work already:

sage: from sage.interfaces.polymake import polymake
sage: polymake.help("objects/Cone/properties/Combinatorics/RAYS_IN_FACETS")
property RAYS_IN_FACETS : IncidenceMatrix<NonSymmetric>
 Ray-facet incidence matrix, with rows corresponding to facets and columns
 to rays. Rays and facets are numbered from 0 to N_RAYS-1 rsp.
 N_FACETS-1, according to their order in RAYS rsp. FACETS.

sage: polymake.help('rand_sphere')
functions/Producing a polytope from scratch/rand_sphere:
rand_sphere(d, n; Options) -> Polytope

 Produce a d-dimensional polytope with n random vertices
 uniformly distributed on the unit sphere.

Arguments:
  Int d the dimension
  Int n the number of random vertices

Options:
  seed => Int controls the outcome of the random number generator;
    fixing a seed number guarantees the same outcome. 

Returns Polytope 

So, there is access to Polymake' help system.

sage: P = polymake.rand<tab key>
                polymake.rand01                polymake.rand_knot              
                polymake.rand_aof              polymake.rand_metric            
                polymake.rand_box              polymake.rand_metric_int       >
                polymake.rand_cyclic           polymake.rand_perm              
                polymake.rand_inner_points     polymake.rand_seed 

So, tab completion of the interface itself works, it yields ALL function names, including those that are not part of the current Polymake application.

sage: P = polymake.rand_sphere(3,10,seed=5)

So, calling a function with parameters works

sage: P
Random spherical polytope of dimension 3; seed=5

So, string representation works, actually nicer than in Polymake (more examples below).

sage: P.known_properties()   # members computed so far
['BOUNDED', 'CONE_AMBIENT_DIM', 'FEASIBLE', 'POINTS']
sage: P.get_schedule('F_VECTOR')

precondition : BOUNDED ( lrs.convex_hull.count: N_FACETS : RAYS | INPUT_RAYS )
lrs.convex_hull.count: N_FACETS : RAYS | INPUT_RAYS
sensitivity check for VertexPerm
cdd.convex_hull.canon: POINTED, RAYS, LINEALITY_SPACE : INPUT_RAYS
CONE_DIM : RAYS | INPUT_RAYS
N_RAYS : RAYS
precondition : POINTED ( LINEALITY_DIM, LINEALITY_SPACE : )
LINEALITY_DIM, LINEALITY_SPACE :
COMBINATORIAL_DIM : CONE_DIM, LINEALITY_DIM
precondition : COMBINATORIAL_DIM ( F_VECTOR : N_FACETS, N_RAYS, COMBINATORIAL_DIM )
F_VECTOR : N_FACETS, N_RAYS, COMBINATORIAL_DIM

So, this shows how the f-vector would be computed for this polytope.

sage: V = P.VER<tab key>
                P.VERTEX_BARYCENTER        P.VERTICES                 
                P.VERTEX_LABELS            P.VERTICES_IN_FACETS       
                P.VERTEX_NORMALS           P.VERTICES_IN_INEQUALITIES 
                P.VERTEX_SIZES             P.VERY_AMPLE               

So, tab completion for Polymake elements works. It shows both members (including those that aren't computed yet) and functions (including those that won't be applicable to the object).

sage: set_verbose(3)
sage: V = P.VERTICES
sage: F = P.F_VECTOR
polymake: used package lrs
  Implementation of the reverse search algorithm of Avis and Fukuda.
  Copyright by David Avis.
  http://cgm.cs.mcgill.ca/~avis/lrs.html

So, when verbosity is set to a positive value, comments printed by Polymake will be printed by Sage.

sage: F
10 24 16
sage: V

1 -549467720344047/2251799813685248 -1666038093248221/2251799813685248 5646952731601237/9007199254740992
1 396443771972991/562949953421312 -3761317241461329/36028797018963968 6325385137097525/9007199254740992
1 8731416515056451/9007199254740992 8225904699783425/36028797018963968 6513541817898829/72057594037927936
1 -6649740931032245/9007199254740992 -316039202166671/562949953421312 -6735311418008155/18014398509481984
1 1027872253618593/1125899906842624 -4941443956760209/18014398509481984 -5443456128447233/18014398509481984
1 3237778007687205/4503599627370496 -3127563668227821/4503599627370496 8478171037802839/288230376151711744
1 4654944693569921/9007199254740992 -2595293938705027/4503599627370496 5702536786492105/9007199254740992
1 -4153090305917205/4503599627370496 2814681273371419/18014398509481984 -3186909849764487/9007199254740992
1 6185041161130579/9007199254740992 -2775723053730255/4503599627370496 -6944451127537363/18014398509481984
1 -6106052881221159/18014398509481984 -2063008243044317/18014398509481984 -8410984913482021/9007199254740992

So, string representation is (at least for the things that are currently wrapped) meaningful.

sage: F[2]
16
sage: V[3][3]
-6735311418008155/18014398509481984

So, attribute access works.

sage: V.foobar()
Traceback (most recent call last)
...
TypeError: polymake:  ERROR: Undefined subroutine &Polymake::User::foobar called at input line 1.
sage: polymake.eval('foobar(4)')
^CInterrupting Polymake...
Traceback (most recent call last)
...
KeyboardInterrupt: Ctrl-c pressed while running Polymake
sage: polymake.eval('foobar(4);')
Traceback (most recent call last)
...
PolymakeError: polymake:  ERROR: Undefined subroutine &Polymake::User::foobar called at input line 1.

So, error handling is implemented.

Meanwhile, more members are known:

sage: P.known_properties()

['AFFINE_HULL',
 'BOUNDED',
 'COMBINATORIAL_DIM',
 'CONE_AMBIENT_DIM',
 'CONE_DIM',
 'FAR_FACE',
 'FEASIBLE',
 'FULL_DIM',
 'F_VECTOR',
 'LINEALITY_DIM',
 'LINEALITY_SPACE',
 'N_FACETS',
 'N_POINTS',
 'N_VERTICES',
 'POINTED',
 'POINTS',
 'VERTICES']

Of course, some things won't work yet. For example, I am not catching warnings yet.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

bfae486

Implement getting length of arrays and hashes

[sagetrac-git]

Changed commit from c3add96 to bfae486

[simon-king-jena]

comment:6

Example for the next commit:

sage: from sage.interfaces.polymake import polymake
sage: P = polymake.rand_sphere(3,10,seed=5)
sage: len(P)   # A polytope is in fact internally represented by an array
14
sage: V = P.VERTICES
sage: len(V)
10
sage: len(V[2])
4

---

New commits:

bfae486

Implement getting length of arrays and hashes

[simon-king-jena]

comment:8

Now I am totally puzzled. After doing some changes, many thinks didn't work. Thus, I reverted all changes. I even deleted my local git branch and downloaded the branch from here. But things still don't work (tab completion etc). Of course I did sage -br.

Do you have any idea what could cause such persisting problems?

[simon-king-jena]

comment:9

It seems that my Sage installation is totally broken. I did "make start" and it seems that it installs python3 without my asking for it.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

c3e7d47

Fix reading from file. Fix KeyboardInterrupt for Polymake

[sagetrac-git]

Changed commit from bfae486 to c3e7d47

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2bd2ac5

add synchronisation code for Polymake interface

[sagetrac-git]

Changed commit from c3e7d47 to 2bd2ac5

[videlec]

comment:12

Replying to @simon-king-jena:

It seems that my Sage installation is totally broken. I did "make start" and it seems that it installs python3 without my asking for it.

Yes. That is normal. python2 and python3 do install by default now (since 2 or 3 beta already).

[simon-king-jena]

comment:13

Replying to @videlec:

Replying to @simon-king-jena:

It seems that my Sage installation is totally broken. I did "make start" and it seems that it installs python3 without my asking for it.

Yes. That is normal. python2 and python3 do install by default now (since 2 or 3 beta already).

Thanks, I didn't notice that before.

I still don't know what went wrong, but it is now again working, including the synchronisation code that I posted in the previous commit.

[sagetrac-git]

Changed commit from 2bd2ac5 to 06e85fd

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

06e85fd

Keep polymake interface in sync. Take docstring from polymake. Fix calling of polymake functions

[simon-king-jena]

comment:15

Finally I found a reasonable source explaining a part of perl that seems particularly relevant: http://perldoc.perl.org/perlref.html

So, I am now rewriting a couple of things in terms of what I learn about perl references.

[simon-king-jena]

comment:16

It is of course conceivable that I am just lacking experience, but I find Perl sucks.

In order to write an interface, it is important to do things uniformly. Hence, all variables created by Sage in Polymake should be "the same". In the current branch, "the same" means that they are all references (which in Perl means they are SCALAR).

So, how to assign any object x in Polymake to a variable var?

If you define a SCALAR "var" and assign to "x", you of course do not get a reference to x but some scalar interpretation of whatever is in x. Thus, I need to create a reference to x. Sounds trivial (and perhaps it is). However, all what I found is that one needs to do $var=\@x, $var=\%x or $var=\*x depending on whether x is an array, a hash or an io stream. OK, sounds easy. But so far I found no way to make Perl tell me what x really is.

Instead, let var uniformly be an array whose single entry is x. Thus:

polytope > @P = [rand_sphere(4,33, seed=>5)];

polytope > print @P[0];

polymake:  ERROR: Scalar value @Polymake::User::P[0] better written as $Polymake::User::P[0] at input line 1.

WTF?? I should better replace something that I did not write??? From what I thought I have learnt about Perl syntax, @P[0] should give the first entry in the array P!

[simon-king-jena]

comment:17

Aha, my mistake. Apparently it works slightly differently:

polytope > @P = (rand_sphere(4,33, seed=>5));
polytope > print $P[0];
Polymake::polytope::Polytope__Rational=ARRAY(0x80d0370)
polytope > print $P[0][1];
Random spherical polytope of dimension 4; seed=5

So, creating a new variable that points to the content of the old variable should work like this, and it does:

polytope > @Q = ($P[0]);

polytope > print $Q[0];
Polymake::polytope::Polytope__Rational=ARRAY(0x80d0370)
polytope > print $Q[0][1];
Random spherical polytope of dimension 4; seed=5

Access to members also works as expected:

polytope > @V = ($Q[0]->VERTICES);
...
polytope > print $V[0];
...
1 -2185542006599419/2251799813685248 -5220912692532935/36028797018963968 2637694986535851/72057594037927936 3401062251409073/18014398509481984
polytope > @S = ($Q[0]->get_schedule("H_VECTOR"));
polymake: used package ppl
  The Parma Polyhedra Library (PPL): A C++ library for convex polyhedra
  and other numerical abstractions.
  http://www.cs.unipr.it/ppl/


polytope > print join(", ", $S[0]->list);
precondition : N_RAYS | N_INPUT_RAYS ( ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS ), sensitivity check for FacetPerm, ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS, RAYS_IN_FACETS : RAYS, FACETS, GRAPH.ADJACENCY : RAYS_IN_FACETS, DUAL_GRAPH.ADJACENCY : RAYS_IN_FACETS, SIMPLICIAL : COMBINATORIAL_DIM, RAYS_IN_FACETS, N_EDGES : ADJACENCY ( applied to DUAL_GRAPH ), N_EDGES : ADJACENCY ( applied to GRAPH ), N_RAYS : RAYS, N_FACETS : FACETS, precondition : COMBINATORIAL_DIM ( F_VECTOR : N_FACETS, N_RAYS, GRAPH.N_EDGES, DUAL_GRAPH.N_EDGES, COMBINATORIAL_DIM ), F_VECTOR : N_FACETS, N_RAYS, GRAPH.N_EDGES, DUAL_GRAPH.N_EDGES, COMBINATORIAL_DIM, precondition : SIMPLICIAL ( H_VECTOR : F_VECTOR ), H_VECTOR : F_VECTOR

So, I'll rewrite my branch accordingly.

[simon-king-jena]

comment:18

Argh. The same trick won't work when I want a list in a list:

polytope > print join(", ", $L[0]);
precondition : N_RAYS | N_INPUT_RAYS ( ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS )

[simon-king-jena]

comment:19

Next thing to try: Use a reference to an array. According to http://perldoc.perl.org/5.20.0/perllol.html, I expect it to work as follows:

polytope > $P = [rand_sphere(4,33, seed=>5)];
polytope > print $P->[0];
Polymake::polytope::Polytope__Rational=ARRAY(0x7c9bae0)
polytope > print $P->[0][1];
Random spherical polytope of dimension 4; seed=5
polytope > $Q = [$P->[0]];
polytope > print $Q->[0];
Polymake::polytope::Polytope__Rational=ARRAY(0x7c9bae0)
polytope > print $Q->[0][1];
Random spherical polytope of dimension 4; seed=5
polytope > $V = [$Q->[0]->VERTICES];
polytope > print $V->[0];
...
polytope > $S = [$Q->[0]->get_schedule("H_VECTOR")];
polytope > print join(", ", $S->[0]->list);
precondition : N_RAYS | N_INPUT_RAYS ( ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS ), sensitivity check for FacetPerm, ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS, RAYS_IN_FACETS : RAYS, FACETS, GRAPH.ADJACENCY : RAYS_IN_FACETS, DUAL_GRAPH.ADJACENCY : RAYS_IN_FACETS, SIMPLICIAL : COMBINATORIAL_DIM, RAYS_IN_FACETS, N_EDGES : ADJACENCY ( applied to DUAL_GRAPH ), N_EDGES : ADJACENCY ( applied to GRAPH ), N_RAYS : RAYS, N_FACETS : FACETS, precondition : COMBINATORIAL_DIM ( F_VECTOR : N_FACETS, N_RAYS, GRAPH.N_EDGES, DUAL_GRAPH.N_EDGES, COMBINATORIAL_DIM ), F_VECTOR : N_FACETS, N_RAYS, GRAPH.N_EDGES, DUAL_GRAPH.N_EDGES, COMBINATORIAL_DIM, precondition : SIMPLICIAL ( H_VECTOR : F_VECTOR ), H_VECTOR : F_VECTOR
polytope > $L = [$S->[0]->list];
polytope > print join(", ", $L->[0]);
precondition : N_RAYS | N_INPUT_RAYS ( ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS )

So, for wrapping lists, it won't work either.

[simon-king-jena]

comment:20

It seems that the problem with nested arrays is known and is discussed here.

[simon-king-jena]

comment:21

Apparently there are no arrays of arrays in Perl. Instead, each item in an array has to be a scalar. Hence, an array of arrays would in fact be implemented as an array of references to arrays.

And it seems that Perl would automagically create such references if you assign something to an entry of an existing array, according to the reference given in the previous post. So, I expect that things would finally work as follows:

polytope > $P = []; $P->[0] = rand_sphere(4,33,seed=>5);
polytope > print $P->[0][1];
Random spherical polytope of dimension 4; seed=5
polytope > $Q = []; $Q->[0] = $P->[0];
polytope > print $Q->[0][1];
Random spherical polytope of dimension 4; seed=5
polytope > $V = []; $V->[0] = $Q->[0]->VERTICES;
polytope > print $V->[0];
...  # output as expected
polytope > $S = []; $S->[0] = $Q->[0]->get_schedule("H_VECTOR");
polytope > print join("\n ", $S->[0]->list);
precondition : N_RAYS | N_INPUT_RAYS ( ppl.convex_hull.primal: FACETS, LINEAR_SPAN : RAYS | INPUT_RAYS )
 sensitivity check for FacetPerm
...   # output as expected
polytope > $L = []; $L->[0] = $S->[0]->list;

polytope > print join("\n ", $L->[0]);
15

WTF??

I suppose the problem here is that in the line $L->[0] = $S->[0]->list; perl puts a scalar value into the first entry of the list that L points to. In contrast to what I thought perl would do, it does not put a reference (which is a scalar) to $S->[0]->list into the first entry of L, but it puts there a scalar interpretation of the array $S->[0]->list --- which is the length of that array.

Crap.

[simon-king-jena]

comment:22

I will now try this model:

• A variable created by Sage in the interface are arrays.
• Such variable either corresponds to a genuine Perl array, or it is an array of length one whose single entry is a reference to some Polymake object.
• I only need to distinguish one thing: If var is an array of length>0, then it is guaranteed to be a genuine Perl array, and we consistently use var as @var. Otherwise, we use it in the form $var[0].
• Since the length of the array is determined at the time of its creation, I can store an attribute in the Sage wrapper that tells what case we'll have to deal with.
• As far as I can see, the only problem would be a genuine Perl array x of length 1, that we would mistakenly interpret as a Polymake object wrapped in x; this would only be problematic if the array x would later be modified.

Anyway. I will next try to use this approach to assign expr in Polymake to a variable:

• @SAGE0 = (expr);, and "SAGE0" is the autogenerated name of a Sage wrapper for expr.
• Test the length of @SAGE0. If the length is 1, then the sage wrapper stores the information that to access the wrapped value it has to use $SAGE0[0]. Otherwise, it stores the information that to access the wrapped value it has to use @SAGE0.
• Do not care about the case that expr is an array of length one.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

52226e2	Do not _check_valid in `_repr_`, since it is done in __repr__
726d3fd	Merge branch 't/22501/make_it_easier_to_customize_pexpect_interfaces_new' into t/22452/create_a_polymake_pexpect_interface
185430f	Use print function, not print statement, in doctests

[sagetrac-git]

Changed commit from a8157a4 to 185430f

[simon-king-jena]

Changed work issues from use print function in doctests to none

[simon-king-jena]

comment:93

The doctests are now using print function, not print statements. And for completeness, I have merged with the current version of #22501. Needs review, again.

[vbraun]

comment:95

sage -t --long src/sage/tests/py3_syntax.py
**********************************************************************
File "src/sage/tests/py3_syntax.py", line 17, in sage.tests.py3_syntax
Failed example:
    py3_syntax.run_tests('.py')   # long time
Expected nothing
Got:
    Invalid Python 3 syntax found:
      File "src/sage/interfaces/polymake.py", line 1503
        except PolymakeError, msg:
                            ^
    SyntaxError: invalid syntax
    <BLANKLINE>
    <BLANKLINE>
**********************************************************************

[mkoeppe]

Changed branch from u/SimonKing/create_a_polymake_pexpect_interface to u/mkoeppe/create_a_polymake_pexpect_interface

[mkoeppe]

New commits:

db26ccf	Merge tag '7.6' into t/22452/create_a_polymake_pexpect_interface
52a77ee	PolymakeElement._member_list: Fix for python 3 syntax

[mkoeppe]

Changed commit from 185430f to 52a77ee

[vbraun]

Changed branch from u/mkoeppe/create_a_polymake_pexpect_interface to 52a77ee

[jplab]

comment:101

Great!!!!! Thanks so much for all this work!!! I will let polymake people know and start showing it around!

As a follow up, I guess it would make sense to make a thematic tutorial on the interface?!?! I can volunteer to do it since I was playing a lot with the doc lately. I will continue to bug you with questions when I have them!

Thanks a lot again!

[jplab]

Changed commit from 52a77ee to none

[videlec]

comment:102

Indeed this is good news! I think it would be appropriate to show one example of a computation with polymake inside the thematic tutorial that you started at #22572. It can be a follow up ticket if you do not want to delay it.