[PPL-devel] [SWIPL] Re: GMP memory allocation problem in SWI-Prolog 5.6.38 and following versions

[Jan Wielemaker]

On Wednesday 03 October 2007 14:37, Roberto Bagnara wrote:
> Jan Wielemaker wrote:

> > Most of the time it is, certainly with open source :-)
>
> True, but having to change code just for that...
> Think about global variables.  If you have something like
>
> mpz_class n;
>
> in C++ code I don't know a reliable way to initialize Prolog before the
> constructor is run.

I never said I liked C++ :-)

> > Actually it never calls pl_abort(), unless something is really wrong. It
> > does call longjmp() back to the Prolog context :-) Note that it only
> > does so if it cannot allocate.
>
> Yes. But I think we disagree with our definition of "really wrong."
> The PPL is exception-safe, meaning that the application can be programmed
> in a way that, whenever allocation fails (something that does happen
> when working with [big] collections of [big] polyhedra) all the memory
> used by the failed (sub)computation is automatically discarded, a
> simplification takes place, and computation restart.  This is only possible
> if the overall application is the only entity allowed to change GMP's
> allocation function.
>
> > The only thing you are allowed to do from
> > the GMP allocation functions is stop the process.
>
> This is not my understanding: where did you get this restriction from?

Quoting from the GMP manual (GMP 4.2.1, SuSE RPM):

----------------------------------------------------------------
   There's currently no defined way for the allocation functions to
recover from an error such as out of memory, they must terminate
program execution.  A `longjmp' or throwing a C++ exception will have
undefined results.  This may change in the future.
----------------------------------------------------------------

So, I wonder how you safely recover. Surely if you call Prolog, which
calls a GMP function, which gets you using longjmp or a C++ exception
back to the C++ program, you haven't made Prolog particulary happy.
Seems from the rest that you are calling C++ from Prolog, which might
make things a bit better ...

> > Prolog knows a few
> > cases where it can safely do better (discussed on the GMP mailinglist),
> > by throwing a Prolog resource_error exception.
>
> I am not contending this.  What I am saying is that in a complex
> application, Prolog can be one of several components all using GMP.  Thus,
> the decision about which allocation functions are adequate can only be
> taken at the application level, not at the component level.

True. The way GMP is designed doesn't make it easy for us though.
Ideally I would of course call GMP functions, check some return code
and map that to a Prolog exception if there is a problem.  GMP however
doesn't give error codes :-(

> > Anyway for the others, I added
> >
> > 	PL_action(PL_GMP_SET_ALLOC_FUNCTIONS, TRUE)
> >
> > if you want Prolog's GMP allocation initialised *now* (without
> > initializing the rest of SWI-Prolog).  or
> >
> > 	PL_action(PL_GMP_SET_ALLOC_FUNCTIONS, FALSE)
> >
> > If you want to stop Prolog from setting the allocation functions.  Must
> > be called before PL_initialise(), of course.
>
> I see how to use that in the foreign-calls-Prolog case.
> How can I achieve that in the Prolog-calls-foreign case (which is the one
> I am mostly interested in)?

You mean you load a shared object into Prolog? Good! That means however
that the C++ constructors are executed *after* the Prolog
initialization, so Prolog can (and already has) changed the allocation
functions. In fact it also created a couple of GMP numbers, but as these
are used read-only throughout the process, this shouldn't matter. You
should be able to change the allocation functions using the
mp_set_memory_functions() at any point in time as long as Prolog is not
doing any active GMP computation at the same time (which should be fine
as long as you have no threads running). Prolog does GMP computation,
but at the end it copies the result to the Prolog stack in a format that
allows for direct use as *input* argument to GMP functions. Except during
computation, it *never* has normally allocated GMP numbers around.

This is also the trick around errors: if it gets a GMP allocation error
it will deallocate all GMP allocation for that thread.  Maybe we should
improve here, ensuring the Prolog allocation functions only do their work
if the GMP operation is called from Prolog.

> > I also added some comments to the section of the manual dealing with GMP
> > and Prolog.  On Windows the situation is entirely different :-(
>
> Well, I am not so worried about Windows :-)

Lucky you!

> > Please install from CVS.  Do not forget a "make clean"!
>
> Right.  I hope people packaging SWI-Prolog for the various distributions
> do not select a version in the range 5.6.38-5.6.43.

Some of these problems cannot be avoided.  Let us just try to fix them.
I'm don't think we have right now :-(  Hope you understand the Prolog
viewpoint a bit better now.

	Cheers --- Jan