[kea-dev] Strange behaviour with Option 53 - what am I missing?

[Dave Cole]

It is pretty easy to work around in the Python case.  Here is a bit of my code:

- - %< - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
static int
findSymbol(void *lib, void **sym, const char *name) {
    *sym = dlsym(lib, name);
    if (*sym == NULL) {
        LOG_INFO(logger, PYTHON_LOAD_FAILED).arg(string("could not dlsym ") + name);
        return (1);
    }
    return (0);
}

static void   (*dl_Py_Initialize)(void);
static void * (*dl_PyCapsule_Import)(const char *name, int no_block);

int
loadPythonLibrary(const string &filename) {
    void *lib = dlopen(filename.c_str(), RTLD_NOW | RTLD_GLOBAL);
    if (lib == NULL) {
        LOG_INFO(logger, PYTHON_LOAD_FAILED).arg("could not dlopen " + filename);
        return (1);
    }

    if (findSymbol(lib, (void **) &dl_PyCapsule_Import, "PyCapsule_Import")
        || findSymbol(lib, (void **) &dl_Py_Initialize, "Py_Initialize")) {
        return (1);
    }
    return (0);
}

static bool kea_loaded = false;

int
load(LibraryHandle& handle) {
    // Load the embedded libpython interpreter.
    ConstElementPtr libpython_elem = handle.getParameter("libpython");
    if (!libpython_elem || libpython_elem->getType() != Element::string) {
        LOG_ERROR(logger, PYTHON_LOAD_FAILED).arg("expected libpython configuration parameter");
        return (1);
    }
    if (loadPythonLibrary(libpython_elem->stringValue())) {
        return (1);
    }
    // Get the name of the Python module implementing the hook.
    ConstElementPtr module_elem = handle.getParameter("module");
    if (!module_elem || module_elem->getType() != Element::string) {
        LOG_ERROR(logger, PYTHON_LOAD_FAILED).arg("expected module configuration parameter");
        return (1);
    }
    const string &module = module_elem->stringValue();

    // Initialise the Python interpreter.
    dl_Py_Initialize();

    // Load the kea module C API.
    PyKea_API = (void **) dl_PyCapsule_Import("kea._C_API", 0);
    if (PyKea_API == NULL) {
        LOG_ERROR(logger, PYTHON_LOAD_FAILED).arg("could not import the kea module _C_API");
        return (1);
    }
    // Bootstrap the kea module.
    PyKea_SetLogger(logger, PYTHON_LOG);
    if (PyKea_Load(handle, module)) {
        return (1);
    }

    kea_loaded = true;
    return (0);
}
- - %< - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The PyKea_ stuff is simply an implementation of a Python Capsule:
   https://docs.python.org/2/extending/extending.html#using-capsules

All of the actual Kea binding code is inside a separate library that is a vanilla Python extension module.  This means that the problems caused by RTLD_LOCAL are minimised to just the code above.

The idea is that while running inside Kea the kea module uses the capabilities inside Kea for logging, and uses Kea supplied instances of LibraryHandle and CalloutHandle and logging.  For unit test code I have simple wrapper of CalloutManager so I can construct a CalloutHandle.  Then in my unittests I subclass the kea.LibraryHandle to replace getParameter(), and so on...
  
From: Francis Dupont <fdupont at isc.org>
Sent: Thursday, 1 February 2018 1:14:20 AM
To: Dave Cole
Cc: Francis Dupont; kea-dev at lists.isc.org
Subject: Re: [kea-dev] Strange behaviour with Option 53 - what am I missing?
  

Dave Cole writes:
> I was hoping to be able to minimise the amount of extra work done by the bi=
> nding, so eliminating unnecessary object allocation / destruction would be =
> a good thing.  I recently had a play with exposing the shared_ptr use_count=
>  to confirm a behaviour I was seeking.  It turns out that the shared_ptr wo=
> rks very nicely with Python wrapping.

=> from the time (more than 30 years ago) I was a CS researcher working
on garbage collection I got a bad opinion about reference counting:
 - expensive in space (need to store the reference count in each object)
 - expensive in time (need to increment and decrement the reference count
  with an atomic operation at each reference change)
 - unbound delay when releasing the last reference
 - too easy to break with cycles
 - fragment memory (vs compact memory)
So it is clearly the worst garbage collection system. But both shared pointers
and C Python are based on reference counting so they work together without
problems...

BTW I didn't forget your concern about hook support in Kea using dlopen()
with RTLD_LOCAL (vs RTLD_GLOBAL) flag. I'll try to do something about this
in 1.4-final or 1.5 milestone...

Thanks

Francis Dupont <fdupont at isc.org>