From 18bce9fa01a558f5efecf68a3b21cfc1ce733db4 Mon Sep 17 00:00:00 2001 From: marcelvanherk Date: Sat, 17 Dec 2016 16:06:31 +0000 Subject: [PATCH 01/22] Update README.md --- README.md | 408 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 408 insertions(+) diff --git a/README.md b/README.md index ce66561..9a0c666 100644 --- a/README.md +++ b/README.md @@ -16,3 +16,411 @@ do, use the recommended methods (```pip```, ```easy-install```, etc) to install This version has been modified to compile under Ubuntu. I haven't tested it under other distributions, your mileage may vary. + +Introduction +------------ + +Lunatic Python is a two-way bridge between Python and Lua, allowing these languages to intercommunicate. Being two-way means that it allows Lua inside Python, Python inside Lua, Lua inside Python inside Lua, Python inside Lua inside Python, and so on. + +Why? + +Even though the project was born as an experiment, it's already being used in real world projects to integrate features from both languages. Please, let me know if you use it in real world projects. + +Examples + +Lua inside Python +A basic example. + +```>>> import lua +>>> lg = lua.globals() +>>> lg.string + +>>> lg.string.lower + +>>> lg.string.lower("Hello world!") +'hello world!' +``` + +Now, let's put a local object into Lua space. + +```>>> d = {} +>>> lg.d = d +>>> lua.execute("d['key'] = 'value'") +>>> d +{'key': 'value'} +Can we get the reference back from Lua space? +>>> d2 = lua.eval("d") +>>> d is d2 +True +``` + +Good! + +Is the python interface available inside the Lua interpreter? + +```>>> lua.eval("python") + +``` + +Yes, it looks so. Let's nest some evaluations and see a local reference passing through. + +```>>> class MyClass: pass +... +>>> obj = MyClass() +>>> obj +<__main__.MyClass instance at 0x403ccb4c> +>>> lua.eval(r"python.eval('lua.eval(\"python.eval(\'obj\')\")')") +<__main__.MyClass instance at 0x403ccb4c> +``` + +Are you still following me? Good. Then you've probably noticed that the Python interpreter state inside the Lua interpreter state is the same as the outside Python we're running. Let's see that in a more comfortable way. + +```>>> lua.execute("pg = python.globals()") +>>> lua.eval("pg.obj") +<__main__.MyClass instance at 0x403ccb4c> +``` + +Things get more interesting when we start to really mix Lua and Python code. + +```>>> table = lua.eval("table") +>>> def show(key, value): +... print "key is %s and value is %s" % (`key`, `value`) +... +>>> t = lua.eval("{a=1, b=2, c=3}") +>>> table.foreach(t, show) +key is 'a' and value is 1 +key is 'c' and value is 3 +key is 'b' and value is 2 +>>> +``` + +Of course, in this case the same could be achieved easily with Python. + +```>>> def show(key, value): +... print "key is %s and value is %s" % (`key`, `value`) +... +>>> t = lua.eval("{a=1, b=2, c=3}") +>>> for k in t: +... show(k, t[k]) +... +key is 'a' and value is 1 +key is 'c' and value is 3 +key is 'b' and value is 2 +``` + +Python inside Lua +----------------- + +Now, let's have a look from another perspective. The basic idea is exactly the same. + +```> require("python") +> python.execute("import string") +> pg = python.globals() +> =pg.string + +> =pg.string.lower("Hello world!") +hello world! +``` + +As Lua is mainly an embedding language, getting access to the batteries included in Python may be interesting. + +```> re = python.import("re") +> pattern = re.compile("^Hel(lo) world!") +> match = pattern.match("Hello world!") +> =match.group(1) +lo +``` + +Just like in the Python example, let's put a local object in Python space. + +```> d = {} +> pg.d = d +> python.execute("d['key'] = 'value'") +> table.foreach(d, print) +key value +``` + +Again, let's grab back the reference from Python space. + +```> d2 = python.eval("d") +> print(d == d2) +true +``` + +Is the Lua interface available to Python? + +```> =python.eval("lua") + +``` + +Good. So let's do the nested trick in Lua as well. + +```> t = {} +> =t +table: 0x80fbdb8 +> =python.eval("lua.eval('python.eval(\"lua.eval(\\'t\\')\")')") +table: 0x80fbdb8 +> +``` + +It means that the Lua interpreter state inside the Python interpreter is the same as the outside Lua interpreter state. Let's show that in a more obvious way. + +```> python.execute("lg = lua.globals()") +> =python.eval("lg.t") +table: 0x80fbdb8 +``` + +Now for the mixing example. + +```> function notthree(num) +>> return (num ~= 3) +>> end +> l = python.eval("[1, 2, 3, 4, 5]") +> filter = python.eval("filter") +> =filter(notthree, l) +[1, 2, 4, 5] +``` + +Documentation +============= + +Theory +------ +The bridging mechanism consists of creating the missing interpreter state inside the host interpreter. That is, when you run the bridging system inside Python, a Lua interpreter is created; when you run the system inside Lua, a Python interpreter is created. +Once both interpreter states are available, these interpreters are provided with the necessary tools to interact freely with each other. The given tools offer not only the ability of executing statements inside the alien interpreter, but also to acquire individual objects and interact with them inside the native state. This magic is done by two special object types, which act bridging native object access to the alien interpreter state. + +Almost every object which is passed between Python and Lua is encapsulated in the language specific bridging object type. The only types which are not encapsulated are strings and numbers, which are converted to the native equivalent objects. +Besides that, the Lua side also has special treatment for encapsulated Python functions and methods. The most obvious way to implement calling of Python objects inside the Lua interpreter is to implement a ```__call``` function in the bridging object metatable. Unfortunately this mechanism is not supported in certain situations, since some places test if the object type is a function, which is not the case of the bridging object. To overwhelm these problems, Python functions and methods are automatically converted to native Lua function closures, becoming accessible in every Lua context. Callable object instances which are not functions nor methods, on the other hand, will still use the metatable mechanism. Luckily, they may also be converted in a native function closure using the ```asfunc()``` function, if necessary. + +Attribute vs. Subscript object access +------------------------------------- + +Accessing an attribute or using the subscript operator in Lua give access to the same information. This behavior is reflected in the Python special object that encapsulates Lua objects, allowing Lua tables to be accessed in a more comfortable way, and also giving access to objects which use protected Python keywords (such as the print function). For example: + +```>>> string = lua.eval("string") +>>> string.lower + +>>> string["lower"] + +``` + +Using Python from the Lua side requires a little bit more attention, since Python has a more strict syntax than Lua. The later makes no distinction between attribute and subscript access, so we need some way to know what kind of access is desired at a given moment. This control is provided using two functions: ```asindx()``` and ```asattr()```. These functions receive a single Python object as parameter, and return the same object with the given access discipline. Notice that dictionaries and lists use the index discipline by default, while other objects use the attribute discipline. For example: + +```> dict = python.eval("{}") +> =dict.keys +nil +> dict.keys = 10 +> print(dict["keys"]) +10 +> =dict +{'keys': 10} +> =dict.keys = 10 +n.asattr(dict) +> =dict.keys +function: 0x80fa9b8 +> =dict.keys() +['keys'] +``` + +Lua inside Python +----------------- + +When executing Python as the host language, the Lua functionality is accessed by importing the lua module. When Lua is the host language, the lua module will already be available in the global Python scope. +Below is a description of the functions available in the lua module. + +`lua.execute(statement)` + +This function will execute the given statement inside the Lua interpreter state. +Examples: + +```>>> lua.execute("foo = 'bar'") +lua.eval(expression) +``` + +This function will evaluate the given expression inside the Lua interpreter state, and return the result. It may be used to acquire any object from the Lua interpreter state. +Examples: + +```>>> lua.eval("'foo'..2") +'foo2' +>>> lua.eval('string') + +>>> string = lua.eval('string') +>>> string.lower("Hello world!") +'hello world!' +``` + +`lua.globals()` + +Return the Lua global scope from the interpreter state. + +Examples: + +```>>> lg = lua.globals() +>>> lg.string.lower("Hello world!") +'hello world!' +>>> lg["string"].lower("Hello world!") +'hello world!' +>>> lg["print"] + +>>> lg["print"]("Hello world!") +Hello world! +``` + +`lua.require(name)` + +Executes the require() Lua function, importing the given module. + +Examples: +```>>> lua.require("testmod") +True +>>> lua.execute("func()") +I'm func in testmod! +``` + +Python inside Lua +----------------- + +Unlike Python, Lua has no default path to its modules. Thus, the default path of the real Lua module of Lunatic Python is together with the Python module, and a python.lua stub is provided. This stub must be placed in a path accessible by the Lua require() mechanism, and once imported it will locate the real module and load it. + +Unfortunately, there's a minor inconvenience for our purposes regarding the Lua system which imports external shared objects. The hardcoded behavior of the loadlib() function is to load shared objects without exporting their symbols. This is usually not a problem in the Lua world, but we're going a little beyond their usual requirements here. We're loading the Python interpreter as a shared object, and the Python interpreter may load its own external modules which are compiled as shared objects as well, and these will want to link back to the symbols in the Python interpreter. Luckily, fixing this problem is easier than explaining the problem. It's just a matter of replacing the flag RTLD_NOW in the loadlib.c file of the Lua distribution by the or'ed version RTLD_NOW|RTLD_GLOBAL. This will avoid "undefined symbol" errors which could eventually happen. +Below is a description of the functions available in the python module. +python.execute(statement) + +This function will execute the given statement inside the Python interpreter state. +Examples: + +```> python.execute("foo = 'bar'") +``` + +`python.eval(expression)` + +This function will evaluate the given expression inside the Python interpreter state, and return the result. It may be used to acquire any object from the Python interpreter state. + +Examples: + +```> python.execute("import string") +> =python.eval("string") + +> string = python.eval("string") +> =string.lower("Hello world!") +hello world! +``` + +`python.globals()` + +Return the Python global scope dictionary from the interpreter state. +Examples: + +```> python.execute("import string") +> pg = python.globals() +> =pg.string.lower("Hello world!") +hello world! +> =pg["string"].lower("Hello world!") +hello world! +``` + +`python.locals()` + +Return the Python local scope dictionary from the interpreter state. +Examples: + +```> function luafunc() +>> print(python.globals().var) +>> print(python.locals().var) +>> end +> python.execute("def func():\n var = 'value'\n lua.execute('luafunc()')") +> python.execute("func()") +nil +value +``` + +`python.builtins()` + +Return the Python builtins module dictionary from the interpreter state. +Examples: + +```> pb = python.builtins() +> =pb.len("Hello world!") +12 +``` + +`python.import(name)` + +Imports and returns the given Python module. +Examples: + +```> os = python.import("os") +> =os.getcwd() +/home/niemeyer/src/lunatic-python +``` + +`python.asattr(pyobj)` + +Return a copy of the given Python object with an attribute access discipline. +Examples: + +```> dict = python.eval("{}") +> =dict.keys +nil +> dict.keys = 10 +> print(dict["keys"]) +10 +> =dict +{'keys': 10} +> =dict.keys = 10 +n.asattr(dict) +> =dict.keys +function: 0x80fa9b8 +> =dict.keys() +['keys'] +``` + +`python.asindx(pyobj)` + +Return a copy of the given Python object with an index access discipline. +Examples: + +```> buffer = python.eval("buffer('foobar')") +> =buffer[0] +stdin:1: unknown attribute in python object +stack traceback: + [C]: ? + stdin:1: in main chunk + [C]: ? +> buffer = python.asindx(buffer) +> =buffer[0] +f +``` + +`python.asfunc(pyobj)` + +Return a copy of the given Python object enclosed in a Lua function closure. This is useful to use Python callable instances in places that require a Lua function. Python methods and functions are automatically converted to Lua functions, and don't require to be explicitly converted. +Examples: + +```> python.execute("class Join:\n def __call__(self, *args):\n return '-'.join(args)") +> join = python.eval("Join()") +> =join +<__main__.Join instance at 0x403a864c> +> =join('foo', 'bar') +foo-bar +> =table.foreach({foo='bar'}, join) +stdin:1: bad argument #2 to `foreach' (function expected, got userdata) +stack traceback: + [C]: in function `foreach' + stdin:1: in main chunk + [C]: ? +> =table.foreach({foo='bar'}, python.asfunc(join)) +foo-bar +``` + +License +------- + +Lunatic Python is available under the LGPL license. + +Download +-------- +Available files: +• lunatic-python-1.0.tar.bz2 +Author +Gustavo Niemeyer From 7bb5bb5103966e338a98f6ef2411f127bb7ba4f5 Mon Sep 17 00:00:00 2001 From: Danni Moiseyev Date: Sun, 23 Feb 2020 15:07:33 +0200 Subject: [PATCH 02/22] Use LUA 5.1 if 5.2 doesn't exists on the system --- setup.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 2fee368..9f0496d 100755 --- a/setup.py +++ b/setup.py @@ -18,8 +18,15 @@ if os.path.isfile("MANIFEST"): os.unlink("MANIFEST") +presult, poutput = commands.getstatusoutput("pkg-config --exists lua5.2") +HAS_LUA5_2 = (presult == 0) + # You may have to change these -LUAVERSION = "5.2" +if HAS_LUA5_2: + LUAVERSION = "5.2" +else: + LUAVERSION = "5.1" + PYTHONVERSION = get_python_version() PYLIBS = ["python" + get_python_version(), "pthread", "util"] PYLIBDIR = [get_python_lib(standard_lib=True) + "/config"] From e0419c6f1700baf844f3f1680f47f7f3ab35a815 Mon Sep 17 00:00:00 2001 From: marcelvanherk Date: Mon, 16 Mar 2020 14:54:52 +0000 Subject: [PATCH 03/22] Fix python calls with only named arguments and reading attribute --- src/pythoninlua.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index f5b5bd5..c7b25ae 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -115,6 +115,7 @@ static int py_object_call(lua_State *L) // passing a single table forces named keyword call style, e.g. plt.plot{x, y, c='red'} if (nargs==1 && lua_istable(L, 2)) { lua_pushnil(L); /* first key */ + nargs=0; while (lua_next(L, 2) != 0) { if (lua_isnumber(L, -2)) { int i = lua_tointeger(L, -2); @@ -271,6 +272,9 @@ static int _p_object_index_get(lua_State *L, py_object *obj, int keyn) } item = PyObject_GetItem(obj->o, key); + if (!item) { + item = PyObject_GetAttr(obj->o, key); + } Py_DECREF(key); From 658f1ff459b9ebf3c85f389decf9f8dca8e8be6a Mon Sep 17 00:00:00 2001 From: Greatwolf Date: Mon, 27 Apr 2020 23:20:49 -0700 Subject: [PATCH 04/22] Updated to use xenial. Updated ppa repo links. Added Python 3.8 to test matrix. --- .travis.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index a59d666..9b914f1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,7 +1,7 @@ language: c sudo: required -dist: trusty +dist: xenial compiler: - gcc @@ -11,11 +11,12 @@ env: - LUA_ENV=lua5.3 PY_ENV=python2.7 m_SUFFIX= - LUA_ENV=lua5.1 PY_ENV=python3.6 m_SUFFIX=m - LUA_ENV=lua5.3 PY_ENV=python3.6 m_SUFFIX=m + - LUA_ENV=lua5.1 PY_ENV=python3.8 m_SUFFIX= + - LUA_ENV=lua5.3 PY_ENV=python3.8 m_SUFFIX= before_install: - - sudo add-apt-repository -y "deb http://ppa.launchpad.net/grilo-team/travis/ubuntu trusty main" - - sudo add-apt-repository -y "deb http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu trusty main" + - sudo add-apt-repository -y ppa:deadsnakes/ppa - sudo apt-get update -qq install: From 28b6b9d53000b6c3082516f7f825b7e14c06bcad Mon Sep 17 00:00:00 2001 From: Greatwolf Date: Tue, 28 Apr 2020 01:02:51 -0700 Subject: [PATCH 05/22] Have CMake pass in the name of the python runtime being built with rather than guessing it in the source. This should also fix issue #74. --- src/CMakeLists.txt | 5 +++++ src/pythoninlua.c | 10 +++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index b4664b5..38c58a9 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -9,6 +9,11 @@ if (WIN32) target_compile_definitions(src PRIVATE LUA_BUILD_AS_DLL) endif (WIN32) +if (UNIX) + get_filename_component(PYTHON_LIBRT ${PYTHON_LIBRARIES} NAME) + target_compile_definitions(src PRIVATE PYTHON_LIBRT=${PYTHON_LIBRT}) +endif (UNIX) + if (CMAKE_COMPILER_IS_GNUCC) target_compile_options(src PUBLIC -Wall -pedantic -std=c99) endif (CMAKE_COMPILER_IS_GNUCC) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index c7b25ae..97e3ed9 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -634,12 +634,12 @@ LUA_API int luaopen_python(lua_State *L) */ #if defined(__linux__) # define STR(s) #s -#if PY_MAJOR_VERSION < 3 -# define PYLIB_STR(major, minor) "libpython" STR(major) "." STR(minor) ".so" -#else -# define PYLIB_STR(major, minor) "libpython" STR(major) "." STR(minor) "m.so" +# define PYLIB_STR(s) STR(s) +#if !defined(PYTHON_LIBRT) +# error PYTHON_LIBRT must be defined when building under Linux! #endif - dlopen(PYLIB_STR(PY_MAJOR_VERSION, PY_MINOR_VERSION), RTLD_NOW | RTLD_GLOBAL); + void *ok = dlopen(PYLIB_STR(PYTHON_LIBRT), RTLD_NOW | RTLD_GLOBAL); + assert(ok); (void) ok; #endif Py_Initialize(); From 31df26b1b8668a6437afaf566bc8cacf37cc8179 Mon Sep 17 00:00:00 2001 From: Greatwolf Date: Tue, 28 Apr 2020 19:45:36 -0700 Subject: [PATCH 06/22] Added Lua 5.2 to test matrix. --- .travis.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.travis.yml b/.travis.yml index 9b914f1..e3160e5 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,10 +8,13 @@ compiler: env: - LUA_ENV=lua5.1 PY_ENV=python2.7 m_SUFFIX= + - LUA_ENV=lua5.2 PY_ENV=python2.7 m_SUFFIX= - LUA_ENV=lua5.3 PY_ENV=python2.7 m_SUFFIX= - LUA_ENV=lua5.1 PY_ENV=python3.6 m_SUFFIX=m + - LUA_ENV=lua5.2 PY_ENV=python3.6 m_SUFFIX=m - LUA_ENV=lua5.3 PY_ENV=python3.6 m_SUFFIX=m - LUA_ENV=lua5.1 PY_ENV=python3.8 m_SUFFIX= + - LUA_ENV=lua5.2 PY_ENV=python3.8 m_SUFFIX= - LUA_ENV=lua5.3 PY_ENV=python3.8 m_SUFFIX= From 44437b24c0f16a9eafa6cf14d5c580d4eeaae9ec Mon Sep 17 00:00:00 2001 From: Greatwolf Date: Wed, 29 Apr 2020 10:12:49 -0700 Subject: [PATCH 07/22] Fixed warning. --- src/pythoninlua.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index 97e3ed9..cbfc433 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -358,7 +358,7 @@ static int py_object_tostring(lua_State *L) if (!repr) { char buf[256]; - snprintf(buf, 256, "python object: %p", obj->o); + snprintf(buf, 256, "python object: %p", (void *)obj->o); lua_pushstring(L, buf); PyErr_Clear(); } From b2e44a2f59e2af9e11d8a28a8f6246ea3ce11661 Mon Sep 17 00:00:00 2001 From: Greatwolf Date: Mon, 4 May 2020 13:14:18 -0700 Subject: [PATCH 08/22] Added project title and license badge. --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index f341c95..41e344c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,7 @@ +# Lunatic-Python + [![Build Status](https://travis-ci.org/bastibe/lunatic-python.svg?branch=master)](https://travis-ci.org/bastibe/lunatic-python) +[![License: LGP-L2.1](https://img.shields.io/badge/license-LGPL%202.1-blue.svg)](https://opensource.org/licenses/LGPL-2.1) Details ======= From 936dea9db8c6e2e5b8a0ecb6dfa5fb936900b5a3 Mon Sep 17 00:00:00 2001 From: user202729 <25191436+user202729@users.noreply.github.com> Date: Tue, 2 Nov 2021 12:20:32 +0700 Subject: [PATCH 09/22] Update README.md to make code blocks render correctly on GitHub --- README.md | 100 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 41e344c..9f7de3e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Installing ---------- To install, you will need to have the Python and Lua development libraries on your system. If you -do, use the recommended methods (```pip```, ```easy-install```, etc) to install lunatic-python. +do, use the recommended methods (`pip`, `easy-install`, etc) to install lunatic-python. This version has been modified to compile under Ubuntu. I haven't tested it under other distributions, your mileage may vary. @@ -36,7 +36,8 @@ Examples Lua inside Python A basic example. -```>>> import lua +``` +>>> import lua >>> lg = lua.globals() >>> lg.string @@ -48,7 +49,8 @@ A basic example. Now, let's put a local object into Lua space. -```>>> d = {} +``` +>>> d = {} >>> lg.d = d >>> lua.execute("d['key'] = 'value'") >>> d @@ -63,13 +65,15 @@ Good! Is the python interface available inside the Lua interpreter? -```>>> lua.eval("python") +``` +>>> lua.eval("python") ``` Yes, it looks so. Let's nest some evaluations and see a local reference passing through. -```>>> class MyClass: pass +``` +>>> class MyClass: pass ... >>> obj = MyClass() >>> obj @@ -80,14 +84,16 @@ Yes, it looks so. Let's nest some evaluations and see a local reference passing Are you still following me? Good. Then you've probably noticed that the Python interpreter state inside the Lua interpreter state is the same as the outside Python we're running. Let's see that in a more comfortable way. -```>>> lua.execute("pg = python.globals()") +``` +>>> lua.execute("pg = python.globals()") >>> lua.eval("pg.obj") <__main__.MyClass instance at 0x403ccb4c> ``` Things get more interesting when we start to really mix Lua and Python code. -```>>> table = lua.eval("table") +``` +>>> table = lua.eval("table") >>> def show(key, value): ... print "key is %s and value is %s" % (`key`, `value`) ... @@ -101,7 +107,8 @@ key is 'b' and value is 2 Of course, in this case the same could be achieved easily with Python. -```>>> def show(key, value): +``` +>>> def show(key, value): ... print "key is %s and value is %s" % (`key`, `value`) ... >>> t = lua.eval("{a=1, b=2, c=3}") @@ -118,7 +125,8 @@ Python inside Lua Now, let's have a look from another perspective. The basic idea is exactly the same. -```> require("python") +``` +> require("python") > python.execute("import string") > pg = python.globals() > =pg.string @@ -129,7 +137,8 @@ hello world! As Lua is mainly an embedding language, getting access to the batteries included in Python may be interesting. -```> re = python.import("re") +``` +> re = python.import("re") > pattern = re.compile("^Hel(lo) world!") > match = pattern.match("Hello world!") > =match.group(1) @@ -138,7 +147,8 @@ lo Just like in the Python example, let's put a local object in Python space. -```> d = {} +``` +> d = {} > pg.d = d > python.execute("d['key'] = 'value'") > table.foreach(d, print) @@ -147,20 +157,23 @@ key value Again, let's grab back the reference from Python space. -```> d2 = python.eval("d") +``` +> d2 = python.eval("d") > print(d == d2) true ``` Is the Lua interface available to Python? -```> =python.eval("lua") +``` +> =python.eval("lua") ``` Good. So let's do the nested trick in Lua as well. -```> t = {} +``` +> t = {} > =t table: 0x80fbdb8 > =python.eval("lua.eval('python.eval(\"lua.eval(\\'t\\')\")')") @@ -170,14 +183,16 @@ table: 0x80fbdb8 It means that the Lua interpreter state inside the Python interpreter is the same as the outside Lua interpreter state. Let's show that in a more obvious way. -```> python.execute("lg = lua.globals()") +``` +> python.execute("lg = lua.globals()") > =python.eval("lg.t") table: 0x80fbdb8 ``` Now for the mixing example. -```> function notthree(num) +``` +> function notthree(num) >> return (num ~= 3) >> end > l = python.eval("[1, 2, 3, 4, 5]") @@ -191,27 +206,30 @@ Documentation Theory ------ + The bridging mechanism consists of creating the missing interpreter state inside the host interpreter. That is, when you run the bridging system inside Python, a Lua interpreter is created; when you run the system inside Lua, a Python interpreter is created. Once both interpreter states are available, these interpreters are provided with the necessary tools to interact freely with each other. The given tools offer not only the ability of executing statements inside the alien interpreter, but also to acquire individual objects and interact with them inside the native state. This magic is done by two special object types, which act bridging native object access to the alien interpreter state. Almost every object which is passed between Python and Lua is encapsulated in the language specific bridging object type. The only types which are not encapsulated are strings and numbers, which are converted to the native equivalent objects. -Besides that, the Lua side also has special treatment for encapsulated Python functions and methods. The most obvious way to implement calling of Python objects inside the Lua interpreter is to implement a ```__call``` function in the bridging object metatable. Unfortunately this mechanism is not supported in certain situations, since some places test if the object type is a function, which is not the case of the bridging object. To overwhelm these problems, Python functions and methods are automatically converted to native Lua function closures, becoming accessible in every Lua context. Callable object instances which are not functions nor methods, on the other hand, will still use the metatable mechanism. Luckily, they may also be converted in a native function closure using the ```asfunc()``` function, if necessary. +Besides that, the Lua side also has special treatment for encapsulated Python functions and methods. The most obvious way to implement calling of Python objects inside the Lua interpreter is to implement a `__call` function in the bridging object metatable. Unfortunately this mechanism is not supported in certain situations, since some places test if the object type is a function, which is not the case of the bridging object. To overwhelm these problems, Python functions and methods are automatically converted to native Lua function closures, becoming accessible in every Lua context. Callable object instances which are not functions nor methods, on the other hand, will still use the metatable mechanism. Luckily, they may also be converted in a native function closure using the `asfunc()` function, if necessary. Attribute vs. Subscript object access ------------------------------------- Accessing an attribute or using the subscript operator in Lua give access to the same information. This behavior is reflected in the Python special object that encapsulates Lua objects, allowing Lua tables to be accessed in a more comfortable way, and also giving access to objects which use protected Python keywords (such as the print function). For example: -```>>> string = lua.eval("string") +``` +>>> string = lua.eval("string") >>> string.lower >>> string["lower"] ``` -Using Python from the Lua side requires a little bit more attention, since Python has a more strict syntax than Lua. The later makes no distinction between attribute and subscript access, so we need some way to know what kind of access is desired at a given moment. This control is provided using two functions: ```asindx()``` and ```asattr()```. These functions receive a single Python object as parameter, and return the same object with the given access discipline. Notice that dictionaries and lists use the index discipline by default, while other objects use the attribute discipline. For example: +Using Python from the Lua side requires a little bit more attention, since Python has a more strict syntax than Lua. The later makes no distinction between attribute and subscript access, so we need some way to know what kind of access is desired at a given moment. This control is provided using two functions: `asindx()` and `asattr()`. These functions receive a single Python object as parameter, and return the same object with the given access discipline. Notice that dictionaries and lists use the index discipline by default, while other objects use the attribute discipline. For example: -```> dict = python.eval("{}") +``` +> dict = python.eval("{}") > =dict.keys nil > dict.keys = 10 @@ -238,14 +256,16 @@ Below is a description of the functions available in the lua module. This function will execute the given statement inside the Lua interpreter state. Examples: -```>>> lua.execute("foo = 'bar'") +``` +>>> lua.execute("foo = 'bar'") lua.eval(expression) ``` This function will evaluate the given expression inside the Lua interpreter state, and return the result. It may be used to acquire any object from the Lua interpreter state. Examples: -```>>> lua.eval("'foo'..2") +``` +>>> lua.eval("'foo'..2") 'foo2' >>> lua.eval('string') @@ -260,7 +280,8 @@ Return the Lua global scope from the interpreter state. Examples: -```>>> lg = lua.globals() +``` +>>> lg = lua.globals() >>> lg.string.lower("Hello world!") 'hello world!' >>> lg["string"].lower("Hello world!") @@ -276,7 +297,8 @@ Hello world! Executes the require() Lua function, importing the given module. Examples: -```>>> lua.require("testmod") +``` +>>> lua.require("testmod") True >>> lua.execute("func()") I'm func in testmod! @@ -289,12 +311,14 @@ Unlike Python, Lua has no default path to its modules. Thus, the default path of Unfortunately, there's a minor inconvenience for our purposes regarding the Lua system which imports external shared objects. The hardcoded behavior of the loadlib() function is to load shared objects without exporting their symbols. This is usually not a problem in the Lua world, but we're going a little beyond their usual requirements here. We're loading the Python interpreter as a shared object, and the Python interpreter may load its own external modules which are compiled as shared objects as well, and these will want to link back to the symbols in the Python interpreter. Luckily, fixing this problem is easier than explaining the problem. It's just a matter of replacing the flag RTLD_NOW in the loadlib.c file of the Lua distribution by the or'ed version RTLD_NOW|RTLD_GLOBAL. This will avoid "undefined symbol" errors which could eventually happen. Below is a description of the functions available in the python module. -python.execute(statement) + +`python.execute(statement)` This function will execute the given statement inside the Python interpreter state. Examples: -```> python.execute("foo = 'bar'") +``` +> python.execute("foo = 'bar'") ``` `python.eval(expression)` @@ -303,7 +327,8 @@ This function will evaluate the given expression inside the Python interpreter s Examples: -```> python.execute("import string") +``` +> python.execute("import string") > =python.eval("string") > string = python.eval("string") @@ -316,7 +341,8 @@ hello world! Return the Python global scope dictionary from the interpreter state. Examples: -```> python.execute("import string") +``` +> python.execute("import string") > pg = python.globals() > =pg.string.lower("Hello world!") hello world! @@ -329,7 +355,8 @@ hello world! Return the Python local scope dictionary from the interpreter state. Examples: -```> function luafunc() +``` +> function luafunc() >> print(python.globals().var) >> print(python.locals().var) >> end @@ -344,7 +371,8 @@ value Return the Python builtins module dictionary from the interpreter state. Examples: -```> pb = python.builtins() +``` +> pb = python.builtins() > =pb.len("Hello world!") 12 ``` @@ -354,7 +382,8 @@ Examples: Imports and returns the given Python module. Examples: -```> os = python.import("os") +``` +> os = python.import("os") > =os.getcwd() /home/niemeyer/src/lunatic-python ``` @@ -364,7 +393,8 @@ Examples: Return a copy of the given Python object with an attribute access discipline. Examples: -```> dict = python.eval("{}") +``` +> dict = python.eval("{}") > =dict.keys nil > dict.keys = 10 @@ -385,7 +415,8 @@ function: 0x80fa9b8 Return a copy of the given Python object with an index access discipline. Examples: -```> buffer = python.eval("buffer('foobar')") +``` +> buffer = python.eval("buffer('foobar')") > =buffer[0] stdin:1: unknown attribute in python object stack traceback: @@ -402,7 +433,8 @@ f Return a copy of the given Python object enclosed in a Lua function closure. This is useful to use Python callable instances in places that require a Lua function. Python methods and functions are automatically converted to Lua functions, and don't require to be explicitly converted. Examples: -```> python.execute("class Join:\n def __call__(self, *args):\n return '-'.join(args)") +``` +> python.execute("class Join:\n def __call__(self, *args):\n return '-'.join(args)") > join = python.eval("Join()") > =join <__main__.Join instance at 0x403a864c> From 82fd9b61db2aef7f479b8a3e914f01627342eb41 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Thu, 4 Jul 2024 10:55:55 +0000 Subject: [PATCH 10/22] Modified function error handling after calling python function in py_object_call. Captured any error string from the called function and added it to the lua error being thrown. --- src/pythoninlua.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index cbfc433..e3df724 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -20,6 +20,7 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ + #include #if defined(__linux__) # include @@ -183,8 +184,23 @@ static int py_object_call(lua_State *L) ret = py_convert(L, value); Py_DECREF(value); } else { - PyErr_Print(); - luaL_error(L, "error calling python function"); + char s_err[1024]; + memset(s_err, 0, 1024); + + PyObject *exc_type, *exc_value, *exc_traceback; + PyErr_Fetch(&exc_type, &exc_value, &exc_traceback); + PyObject *exc_str = PyObject_Str(exc_value); + + // Need not be garbage collected as per documentation of PyUnicode_AsUTF8 + const char *exc_cstr = PyUnicode_AsUTF8(exc_str); + + strncpy(s_err, (!(exc_cstr)?"UNKNOWN ERROR":exc_cstr), 1023); + Py_XDECREF(exc_type); + Py_XDECREF(exc_value); + Py_XDECREF(exc_traceback); + Py_XDECREF(exc_str); + + luaL_error(L, "error calling python function [%s]", s_err); } return ret; From c3367ad8092c1fbae99b0ee7eecc09503363f95b Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Thu, 4 Jul 2024 15:10:59 +0000 Subject: [PATCH 11/22] Used traceback.format_exception so that the formatting is original. --- src/pythoninlua.c | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index e3df724..fc58a95 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -184,23 +184,54 @@ static int py_object_call(lua_State *L) ret = py_convert(L, value); Py_DECREF(value); } else { - char s_err[1024]; - memset(s_err, 0, 1024); + char s_exc[1024]; + char s_traceback[1280]; + memset(s_exc, 0, 1024); + memset(s_traceback, 0, 1280); PyObject *exc_type, *exc_value, *exc_traceback; PyErr_Fetch(&exc_type, &exc_value, &exc_traceback); + PyErr_NormalizeException(&exc_type, &exc_value, &exc_traceback); + PyObject *exc_str = PyObject_Str(exc_value); // Need not be garbage collected as per documentation of PyUnicode_AsUTF8 - const char *exc_cstr = PyUnicode_AsUTF8(exc_str); + const char *exc_cstr = (exc_str)?PyUnicode_AsUTF8(exc_str):""; + strncpy(s_exc, (!(exc_cstr)?"UNKNOWN ERROR":exc_cstr), 1023); + + if (exc_value != NULL && exc_traceback != NULL) { + PyObject *traceback_module = PyImport_ImportModule("traceback"); + if (traceback_module != NULL) { + PyObject *traceback_list = PyObject_CallMethod(traceback_module, + "format_exception", "OOO", exc_type, exc_value, exc_traceback); + if (traceback_list != NULL) { + PyObject *traceback_str = PyUnicode_Join(PyUnicode_FromString(""), traceback_list); + if (traceback_str != NULL) { + // Need not be garbage collected as per documentation of PyUnicode_AsUTF8 + const char *traceback_cstr = PyUnicode_AsUTF8(traceback_str); + if (traceback_cstr != NULL) { + strncpy(s_traceback, traceback_cstr, 1023); + } + Py_XDECREF(traceback_str); + } + Py_XDECREF(traceback_list); + } + Py_XDECREF(traceback_module); + } + } + + if (*s_traceback == '\0') { + strcpy(s_traceback, "Exception: "); + strncat(s_traceback, s_exc, (1280 - strlen("Exception: "))); + } + - strncpy(s_err, (!(exc_cstr)?"UNKNOWN ERROR":exc_cstr), 1023); Py_XDECREF(exc_type); Py_XDECREF(exc_value); Py_XDECREF(exc_traceback); Py_XDECREF(exc_str); - luaL_error(L, "error calling python function [%s]", s_err); + luaL_error(L, "error calling python function:\n%s", s_traceback); } return ret; From 3946a029ee0a422e1286f4ae8f8009dbc86e7624 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Fri, 5 Jul 2024 01:19:56 +0000 Subject: [PATCH 12/22] Further edit to previous commit, based on review --- src/pythoninlua.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index fc58a95..6029b4d 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -184,10 +184,8 @@ static int py_object_call(lua_State *L) ret = py_convert(L, value); Py_DECREF(value); } else { - char s_exc[1024]; - char s_traceback[1280]; - memset(s_exc, 0, 1024); - memset(s_traceback, 0, 1280); + char s_exc[1024] = {0}; + char s_traceback[1280] = {0}; PyObject *exc_type, *exc_value, *exc_traceback; PyErr_Fetch(&exc_type, &exc_value, &exc_traceback); From f484faceb37d96ba26d5db48c4256eb6ad49aa1c Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Sun, 7 Jul 2024 12:51:18 +0000 Subject: [PATCH 13/22] Changes realted to impact due to introduction of #define PY_SSIZE_T_CLEAN --- src/luainpython.c | 113 ++++++++++++++++++++++++++++++---------------- 1 file changed, 74 insertions(+), 39 deletions(-) diff --git a/src/luainpython.c b/src/luainpython.c index c55c6ab..6922d3c 100644 --- a/src/luainpython.c +++ b/src/luainpython.c @@ -20,6 +20,7 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ +#define PY_SSIZE_T_CLEAN #include /* need this to build with Lua 5.2: enables lua_strlen() macro */ @@ -396,9 +397,17 @@ static PyObject *LuaObject_iternext(LuaObject *obj) return ret; } +#ifdef PY_SSIZE_T_CLEAN +static Py_ssize_t LuaObject_length(LuaObject *obj) +#else static int LuaObject_length(LuaObject *obj) +#endif { +#ifdef PY_SSIZE_T_CLEAN + Py_ssize_t len; +#else int len; +#endif lua_rawgeti(LuaState, LUA_REGISTRYINDEX, ((LuaObject*)obj)->ref); len = luaL_len(LuaState, -1); lua_settop(LuaState, 0); @@ -428,46 +437,68 @@ static PyMappingMethods LuaObject_as_mapping = { PyTypeObject LuaObject_Type = { PyVarObject_HEAD_INIT(NULL, 0) - "lua.custom", /*tp_name*/ - sizeof(LuaObject), /*tp_basicsize*/ - 0, /*tp_itemsize*/ - (destructor)LuaObject_dealloc, /*tp_dealloc*/ - 0, /*tp_print*/ - 0, /*tp_getattr*/ - 0, /*tp_setattr*/ - 0, /*tp_compare*/ - LuaObject_str, /*tp_repr*/ - 0, /*tp_as_number*/ - 0, /*tp_as_sequence*/ - &LuaObject_as_mapping, /*tp_as_mapping*/ - 0, /*tp_hash*/ - (ternaryfunc)LuaObject_call, /*tp_call*/ - LuaObject_str, /*tp_str*/ - LuaObject_getattr, /*tp_getattro*/ - LuaObject_setattr, /*tp_setattro*/ - 0, /*tp_as_buffer*/ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ - "custom lua object", /*tp_doc*/ - 0, /*tp_traverse*/ - 0, /*tp_clear*/ - LuaObject_richcmp, /*tp_richcompare*/ - 0, /*tp_weaklistoffset*/ - PyObject_SelfIter, /*tp_iter*/ - (iternextfunc)LuaObject_iternext, /*tp_iternext*/ - 0, /*tp_methods*/ - 0, /*tp_members*/ - 0, /*tp_getset*/ - 0, /*tp_base*/ - 0, /*tp_dict*/ - 0, /*tp_descr_get*/ - 0, /*tp_descr_set*/ - 0, /*tp_dictoffset*/ - 0, /*tp_init*/ - PyType_GenericAlloc, /*tp_alloc*/ - PyType_GenericNew, /*tp_new*/ - PyObject_Del, /*tp_free*/ - 0, /*tp_is_gc*/ + .tp_name = "lua.custom", + .tp_basicsize = sizeof(LuaObject), + .tp_dealloc = (destructor)LuaObject_dealloc, + .tp_repr = LuaObject_str, + .tp_as_mapping = &LuaObject_as_mapping, + .tp_call = (ternaryfunc)LuaObject_call, + .tp_str = LuaObject_str, + .tp_getattro = LuaObject_getattr, + .tp_setattro = LuaObject_setattr, + .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, + .tp_doc = "custom lua object", + .tp_richcompare = LuaObject_richcmp, + .tp_iter = PyObject_SelfIter, + .tp_iternext = (iternextfunc)LuaObject_iternext, + .tp_alloc = PyType_GenericAlloc, + .tp_new = PyType_GenericNew, + .tp_free = PyObject_Del, }; +/* +PyTypeObject LuaObject_Type = { + PyVarObject_HEAD_INIT(NULL, 0) + "lua.custom", //tp_name + sizeof(LuaObject), //tp_basicsize + 0, //tp_itemsize + (destructor)LuaObject_dealloc, //tp_dealloc + 0, //tp_print + 0, //tp_getattr + 0, //tp_setattr + 0, //tp_compare + LuaObject_str, //tp_repr + 0, //tp_as_number + 0, //tp_as_sequence + &LuaObject_as_mapping, //tp_as_mapping + 0, //tp_hash + (ternaryfunc)LuaObject_call, //tp_call + LuaObject_str, //tp_str + LuaObject_getattr, //tp_getattro + LuaObject_setattr, //tp_setattro + 0, //tp_as_buffer + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, //tp_flags + "custom lua object", //tp_doc + 0, //tp_traverse + 0, //tp_clear + LuaObject_richcmp, //tp_richcompare + 0, //tp_weaklistoffset + PyObject_SelfIter, //tp_iter + (iternextfunc)LuaObject_iternext, //tp_iternext + 0, //tp_methods + 0, //tp_members + 0, //tp_getset + 0, //tp_base + 0, //tp_dict + 0, //tp_descr_get + 0, //tp_descr_set + 0, //tp_dictoffset + 0, //tp_init + PyType_GenericAlloc, //tp_alloc + PyType_GenericNew, //tp_new + PyObject_Del, //tp_free + 0, //tp_is_gc +}; +*/ PyObject *Lua_run(PyObject *args, int eval) @@ -475,7 +506,11 @@ PyObject *Lua_run(PyObject *args, int eval) PyObject *ret; char *buf = NULL; char *s; +#ifdef PY_SSIZE_T_CLEAN + Py_ssize_t len; +#else int len; +#endif if (!PyArg_ParseTuple(args, "s#", &s, &len)) return NULL; From d73cfbb41502bb7ffe8806ee0edcc047129fd6a4 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Sun, 7 Jul 2024 16:34:05 +0000 Subject: [PATCH 14/22] Defect fix: function LuaObject_richcmp was returning instance of Py_True/Py_False, without incrementing the reference. Fixed it by calling LuaConvert --- src/luainpython.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/src/luainpython.c b/src/luainpython.c index 6922d3c..7a1a8cc 100644 --- a/src/luainpython.c +++ b/src/luainpython.c @@ -111,7 +111,12 @@ static PyObject *LuaCall(lua_State *L, PyObject *args) { PyObject *ret = NULL; PyObject *arg; - int nargs, rc, i; +#ifdef PY_SSIZE_T_CLEAN + Py_ssize_t nargs, i; +#else + int nargs, i; +#endif + int rc; if (!PyTuple_Check(args)) { PyErr_SetString(PyExc_TypeError, "tuple expected"); @@ -359,7 +364,8 @@ static PyObject* LuaObject_richcmp(PyObject *lhs, PyObject *rhs, int op) PyErr_SetString(PyExc_RuntimeError, lua_tostring(LuaState, -1)); return NULL; } - return lua_toboolean(LuaState, -1) ? Py_True : Py_False; + //return lua_toboolean(LuaState, -1) ? Py_True : Py_False; + return LuaConvert(LuaState, -1); } static PyObject *LuaObject_call(PyObject *obj, PyObject *args) @@ -455,6 +461,7 @@ PyTypeObject LuaObject_Type = { .tp_new = PyType_GenericNew, .tp_free = PyObject_Del, }; + /* PyTypeObject LuaObject_Type = { PyVarObject_HEAD_INIT(NULL, 0) @@ -462,7 +469,7 @@ PyTypeObject LuaObject_Type = { sizeof(LuaObject), //tp_basicsize 0, //tp_itemsize (destructor)LuaObject_dealloc, //tp_dealloc - 0, //tp_print + 0, //tp_vectorcall_offset 0, //tp_getattr 0, //tp_setattr 0, //tp_compare @@ -497,6 +504,16 @@ PyTypeObject LuaObject_Type = { PyType_GenericNew, //tp_new PyObject_Del, //tp_free 0, //tp_is_gc + 0, // tp_bases + 0, // tp_mro + 0, // tp_cache + 0, // tp_subclasses + 0, //tp_weaklist + 0, //tp_del + 0, //tp_version_tag + 0, //tp_finalize + 0, //tp_vectorcall + 0, //tp_watched }; */ From 406b46e3c6d55c34a2bba3b29b80c50187928576 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Mon, 8 Jul 2024 06:05:49 +0000 Subject: [PATCH 15/22] Edited code to remove unnecessary commented lines --- src/luainpython.c | 57 ----------------------------------------------- 1 file changed, 57 deletions(-) diff --git a/src/luainpython.c b/src/luainpython.c index 7a1a8cc..02a3ed9 100644 --- a/src/luainpython.c +++ b/src/luainpython.c @@ -364,7 +364,6 @@ static PyObject* LuaObject_richcmp(PyObject *lhs, PyObject *rhs, int op) PyErr_SetString(PyExc_RuntimeError, lua_tostring(LuaState, -1)); return NULL; } - //return lua_toboolean(LuaState, -1) ? Py_True : Py_False; return LuaConvert(LuaState, -1); } @@ -462,62 +461,6 @@ PyTypeObject LuaObject_Type = { .tp_free = PyObject_Del, }; -/* -PyTypeObject LuaObject_Type = { - PyVarObject_HEAD_INIT(NULL, 0) - "lua.custom", //tp_name - sizeof(LuaObject), //tp_basicsize - 0, //tp_itemsize - (destructor)LuaObject_dealloc, //tp_dealloc - 0, //tp_vectorcall_offset - 0, //tp_getattr - 0, //tp_setattr - 0, //tp_compare - LuaObject_str, //tp_repr - 0, //tp_as_number - 0, //tp_as_sequence - &LuaObject_as_mapping, //tp_as_mapping - 0, //tp_hash - (ternaryfunc)LuaObject_call, //tp_call - LuaObject_str, //tp_str - LuaObject_getattr, //tp_getattro - LuaObject_setattr, //tp_setattro - 0, //tp_as_buffer - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, //tp_flags - "custom lua object", //tp_doc - 0, //tp_traverse - 0, //tp_clear - LuaObject_richcmp, //tp_richcompare - 0, //tp_weaklistoffset - PyObject_SelfIter, //tp_iter - (iternextfunc)LuaObject_iternext, //tp_iternext - 0, //tp_methods - 0, //tp_members - 0, //tp_getset - 0, //tp_base - 0, //tp_dict - 0, //tp_descr_get - 0, //tp_descr_set - 0, //tp_dictoffset - 0, //tp_init - PyType_GenericAlloc, //tp_alloc - PyType_GenericNew, //tp_new - PyObject_Del, //tp_free - 0, //tp_is_gc - 0, // tp_bases - 0, // tp_mro - 0, // tp_cache - 0, // tp_subclasses - 0, //tp_weaklist - 0, //tp_del - 0, //tp_version_tag - 0, //tp_finalize - 0, //tp_vectorcall - 0, //tp_watched -}; -*/ - - PyObject *Lua_run(PyObject *args, int eval) { PyObject *ret; From 1930603f996df3050e22b69949f8c5cf882ae713 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Tue, 9 Jul 2024 01:58:15 +0000 Subject: [PATCH 16/22] Refactored code for error formatting --- src/pythoninlua.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index 6029b4d..8ca9467 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -219,8 +219,7 @@ static int py_object_call(lua_State *L) } if (*s_traceback == '\0') { - strcpy(s_traceback, "Exception: "); - strncat(s_traceback, s_exc, (1280 - strlen("Exception: "))); + snprintf(s_traceback, 1280, "Exception: %s", s_exc); } From e6837bf1fd1f4b75c9e069f497cf7f5029e835b3 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Tue, 9 Jul 2024 06:19:20 +0000 Subject: [PATCH 17/22] Refactored code for error formatting --- src/pythoninlua.c | 18 +++++++++--------- tests/test_py.lua | 25 +++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/src/pythoninlua.c b/src/pythoninlua.c index 8ca9467..464be43 100644 --- a/src/pythoninlua.c +++ b/src/pythoninlua.c @@ -185,7 +185,7 @@ static int py_object_call(lua_State *L) Py_DECREF(value); } else { char s_exc[1024] = {0}; - char s_traceback[1280] = {0}; + char s_traceback[1024] = {0}; PyObject *exc_type, *exc_value, *exc_traceback; PyErr_Fetch(&exc_type, &exc_value, &exc_traceback); @@ -195,7 +195,7 @@ static int py_object_call(lua_State *L) // Need not be garbage collected as per documentation of PyUnicode_AsUTF8 const char *exc_cstr = (exc_str)?PyUnicode_AsUTF8(exc_str):""; - strncpy(s_exc, (!(exc_cstr)?"UNKNOWN ERROR":exc_cstr), 1023); + strncpy(s_exc, (!(exc_cstr)?"UNKNOWN ERROR\n":exc_cstr), 1023); if (exc_value != NULL && exc_traceback != NULL) { PyObject *traceback_module = PyImport_ImportModule("traceback"); @@ -217,18 +217,18 @@ static int py_object_call(lua_State *L) Py_XDECREF(traceback_module); } } - - if (*s_traceback == '\0') { - snprintf(s_traceback, 1280, "Exception: %s", s_exc); - } - - Py_XDECREF(exc_type); Py_XDECREF(exc_value); Py_XDECREF(exc_traceback); Py_XDECREF(exc_str); - luaL_error(L, "error calling python function:\n%s", s_traceback); + if (*s_traceback == '\0') { + luaL_error(L, "error calling python function:\nException: %s", s_exc); + } + else { + luaL_error(L, "error calling python function:\nException: %s", s_traceback); + } + } return ret; diff --git a/tests/test_py.lua b/tests/test_py.lua index a5f4d28..b945528 100644 --- a/tests/test_py.lua +++ b/tests/test_py.lua @@ -40,3 +40,28 @@ bar = 2 assert(python.globals().foo == 1) assert(python.globals().bar == 2) + +python.execute +[[ +def throw_exc(): + raise Exception("THIS EXCEPTION") +]] + +local status, exc = pcall(python.globals().throw_exc) +assert(status == false) +assert(exc == +[[error calling python function: +Exception: Traceback (most recent call last): + File "", line 2, in throw_exc +Exception: THIS EXCEPTION +]], exc) + +local b, e = string.find(exc, "Exception: ", 1); +local ob, oe = b, e; +while (b ~= nil) do + ob, oe = b, e + b, e = string.find(exc, "Exception: ", e+1) +end +local exc_s = (string.sub(exc, oe+1)) +assert((require "pl.stringx").strip(exc_s) == "THIS EXCEPTION"); + From d521cba316c64d49e06b386d628de50c18bc7ba3 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Tue, 9 Jul 2024 08:54:11 +0000 Subject: [PATCH 18/22] removed dependency on penlight lua library --- tests/test_py.lua | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_py.lua b/tests/test_py.lua index b945528..2467596 100644 --- a/tests/test_py.lua +++ b/tests/test_py.lua @@ -63,5 +63,5 @@ while (b ~= nil) do b, e = string.find(exc, "Exception: ", e+1) end local exc_s = (string.sub(exc, oe+1)) -assert((require "pl.stringx").strip(exc_s) == "THIS EXCEPTION"); +--assert((require "pl.stringx").strip(exc_s) == "THIS EXCEPTION"); From 38c0ccda53ed6198f388fde5495b955e8b7f29f6 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Thu, 11 Jul 2024 10:54:25 +0000 Subject: [PATCH 19/22] modified cmake build to pick python3 if available else pick python2 --- CMakeLists.txt | 60 ++++++++++++++++++++++------------------------ src/CMakeLists.txt | 38 ++++++++++++++--------------- 2 files changed, 48 insertions(+), 50 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1422914..10af951 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,31 +1,29 @@ -cmake_minimum_required(VERSION 3.0.0) - -set(CMAKE_BUILD_TYPE_INIT "Release") -set(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR}/bin) -set(LIBRARY_OUTPUT_PATH ${EXECUTABLE_OUTPUT_PATH}) - - -project(Lunatic) - -find_package(Lua 5.1 REQUIRED) -find_package(PythonLibs 2.7 REQUIRED) - - -add_subdirectory(src) - -add_library(python MODULE $) -set_target_properties(python PROPERTIES - PREFIX "") - -add_library(lua MODULE $) -if (WIN32) - set_target_properties(lua PROPERTIES - PREFIX "" - SUFFIX ".pyd") -else (WIN32) - set_target_properties(lua PROPERTIES - PREFIX "") -endif (WIN32) - -target_link_libraries(lua ${LUA_LIBRARIES} ${PYTHON_LIBRARIES}) -target_link_libraries(python ${LUA_LIBRARIES} ${PYTHON_LIBRARIES}) +cmake_minimum_required(VERSION 3.0.0) + +set(CMAKE_BUILD_TYPE_INIT "Release") +set(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR}/bin) +set(LIBRARY_OUTPUT_PATH ${EXECUTABLE_OUTPUT_PATH}) + +project(Lunatic) + +find_package(Lua 5.1 REQUIRED) +find_package(Python REQUIRED COMPONENTS Interpreter Development) + +add_subdirectory(src) + +add_library(python MODULE $) +set_target_properties(python PROPERTIES + PREFIX "") + +add_library(lua MODULE $) +if (WIN32) + set_target_properties(lua PROPERTIES + PREFIX "" + SUFFIX ".pyd") +else (WIN32) + set_target_properties(lua PROPERTIES + PREFIX "") +endif (WIN32) + +target_link_libraries(lua ${LUA_LIBRARIES} ${Python_LIBRARIES}) +target_link_libraries(python ${LUA_LIBRARIES} ${Python_LIBRARIES}) diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 38c58a9..3c84b54 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -1,19 +1,19 @@ -add_library(src OBJECT luainpython.c pythoninlua.c) -set_target_properties(src PROPERTIES - POSITION_INDEPENDENT_CODE TRUE) - -target_include_directories(src PRIVATE ${LUA_INCLUDE_DIR} ${PYTHON_INCLUDE_DIR}) - -target_compile_definitions(src PRIVATE LUA_LIB) -if (WIN32) - target_compile_definitions(src PRIVATE LUA_BUILD_AS_DLL) -endif (WIN32) - -if (UNIX) - get_filename_component(PYTHON_LIBRT ${PYTHON_LIBRARIES} NAME) - target_compile_definitions(src PRIVATE PYTHON_LIBRT=${PYTHON_LIBRT}) -endif (UNIX) - -if (CMAKE_COMPILER_IS_GNUCC) - target_compile_options(src PUBLIC -Wall -pedantic -std=c99) -endif (CMAKE_COMPILER_IS_GNUCC) +add_library(src OBJECT luainpython.c pythoninlua.c) +set_target_properties(src PROPERTIES + POSITION_INDEPENDENT_CODE TRUE) + +target_include_directories(src PRIVATE ${LUA_INCLUDE_DIR} ${Python_INCLUDE_DIRS}) + +target_compile_definitions(src PRIVATE LUA_LIB) +if (WIN32) + target_compile_definitions(src PRIVATE LUA_BUILD_AS_DLL) +endif (WIN32) + +if (UNIX) + get_filename_component(PYTHON_LIBRT ${Python_LIBRARIES} NAME) + target_compile_definitions(src PRIVATE PYTHON_LIBRT=${PYTHON_LIBRT}) +endif (UNIX) + +if (CMAKE_COMPILER_IS_GNUCC) + target_compile_options(src PUBLIC -Wall -pedantic -std=c99) +endif (CMAKE_COMPILER_IS_GNUCC) From f875b166f1ff9d5e3e2dd843561bee6c74a07558 Mon Sep 17 00:00:00 2001 From: Sudheer Hebbale Date: Fri, 12 Jul 2024 04:59:05 +0000 Subject: [PATCH 20/22] Changed minimum cmake version to 3.12 --- CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 10af951..9dabdfa 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,4 +1,4 @@ -cmake_minimum_required(VERSION 3.0.0) +cmake_minimum_required(VERSION 3.12.0) set(CMAKE_BUILD_TYPE_INIT "Release") set(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR}/bin) From 5dae46aeabdddea77dce81b8bf4ff9b78e184ec7 Mon Sep 17 00:00:00 2001 From: Shmuel Zeigerman Date: Fri, 4 Apr 2025 23:46:43 +0300 Subject: [PATCH 21/22] Improve compatibility with LuaJIT --- src/luainpython.c | 3 +++ src/luainpython.h | 3 +++ 2 files changed, 6 insertions(+) diff --git a/src/luainpython.c b/src/luainpython.c index 02a3ed9..52adb9b 100644 --- a/src/luainpython.c +++ b/src/luainpython.c @@ -308,6 +308,9 @@ static PyObject *LuaObject_str(PyObject *obj) } #if LUA_VERSION_NUM == 501 +#ifdef LUA_OK // defined in LuaJIT +#undef LUA_OK +#endif enum { LUA_OK, LUA_OPEQ, LUA_OPLT, LUA_OPLE, diff --git a/src/luainpython.h b/src/luainpython.h index 73ea65c..c5a841e 100644 --- a/src/luainpython.h +++ b/src/luainpython.h @@ -26,6 +26,9 @@ #if LUA_VERSION_NUM == 501 #define luaL_len lua_objlen #define luaL_setfuncs(L, l, nup) luaL_register(L, NULL, (l)) + #ifdef luaL_newlib // defined in LuaJIT + #undef luaL_newlib + #endif #define luaL_newlib(L, l) (lua_newtable(L), luaL_register(L, NULL, (l))) #endif From 1228580d896936c73bbdbd45fb52e65a3ec317a5 Mon Sep 17 00:00:00 2001 From: ShalokShalom Date: Wed, 7 May 2025 10:16:31 +0200 Subject: [PATCH 22/22] Add syntax highlight I add syntax highlighting to all examples. I cared to always choose the right one (Lua/Python) --- README.md | 108 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 9f7de3e..d5052b4 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Examples Lua inside Python A basic example. -``` +```lua >>> import lua >>> lg = lua.globals() >>> lg.string @@ -49,7 +49,7 @@ A basic example. Now, let's put a local object into Lua space. -``` +```lua >>> d = {} >>> lg.d = d >>> lua.execute("d['key'] = 'value'") @@ -65,14 +65,14 @@ Good! Is the python interface available inside the Lua interpreter? -``` +```lua >>> lua.eval("python") ``` Yes, it looks so. Let's nest some evaluations and see a local reference passing through. -``` +```python >>> class MyClass: pass ... >>> obj = MyClass() @@ -84,7 +84,7 @@ Yes, it looks so. Let's nest some evaluations and see a local reference passing Are you still following me? Good. Then you've probably noticed that the Python interpreter state inside the Lua interpreter state is the same as the outside Python we're running. Let's see that in a more comfortable way. -``` +```python >>> lua.execute("pg = python.globals()") >>> lua.eval("pg.obj") <__main__.MyClass instance at 0x403ccb4c> @@ -92,7 +92,7 @@ Are you still following me? Good. Then you've probably noticed that the Python Things get more interesting when we start to really mix Lua and Python code. -``` +```python >>> table = lua.eval("table") >>> def show(key, value): ... print "key is %s and value is %s" % (`key`, `value`) @@ -107,7 +107,7 @@ key is 'b' and value is 2 Of course, in this case the same could be achieved easily with Python. -``` +```python >>> def show(key, value): ... print "key is %s and value is %s" % (`key`, `value`) ... @@ -125,7 +125,7 @@ Python inside Lua Now, let's have a look from another perspective. The basic idea is exactly the same. -``` +```python > require("python") > python.execute("import string") > pg = python.globals() @@ -137,7 +137,7 @@ hello world! As Lua is mainly an embedding language, getting access to the batteries included in Python may be interesting. -``` +```python > re = python.import("re") > pattern = re.compile("^Hel(lo) world!") > match = pattern.match("Hello world!") @@ -147,7 +147,7 @@ lo Just like in the Python example, let's put a local object in Python space. -``` +```python > d = {} > pg.d = d > python.execute("d['key'] = 'value'") @@ -157,7 +157,7 @@ key value Again, let's grab back the reference from Python space. -``` +```python > d2 = python.eval("d") > print(d == d2) true @@ -165,14 +165,14 @@ true Is the Lua interface available to Python? -``` +```pythom > =python.eval("lua") ``` Good. So let's do the nested trick in Lua as well. -``` +```python > t = {} > =t table: 0x80fbdb8 @@ -183,7 +183,7 @@ table: 0x80fbdb8 It means that the Lua interpreter state inside the Python interpreter is the same as the outside Lua interpreter state. Let's show that in a more obvious way. -``` +```lua > python.execute("lg = lua.globals()") > =python.eval("lg.t") table: 0x80fbdb8 @@ -191,7 +191,7 @@ table: 0x80fbdb8 Now for the mixing example. -``` +```lua > function notthree(num) >> return (num ~= 3) >> end @@ -218,7 +218,7 @@ Attribute vs. Subscript object access Accessing an attribute or using the subscript operator in Lua give access to the same information. This behavior is reflected in the Python special object that encapsulates Lua objects, allowing Lua tables to be accessed in a more comfortable way, and also giving access to objects which use protected Python keywords (such as the print function). For example: -``` +```python >>> string = lua.eval("string") >>> string.lower @@ -228,7 +228,7 @@ Accessing an attribute or using the subscript operator in Lua give access to the Using Python from the Lua side requires a little bit more attention, since Python has a more strict syntax than Lua. The later makes no distinction between attribute and subscript access, so we need some way to know what kind of access is desired at a given moment. This control is provided using two functions: `asindx()` and `asattr()`. These functions receive a single Python object as parameter, and return the same object with the given access discipline. Notice that dictionaries and lists use the index discipline by default, while other objects use the attribute discipline. For example: -``` +```python > dict = python.eval("{}") > =dict.keys nil @@ -251,12 +251,14 @@ Lua inside Python When executing Python as the host language, the Lua functionality is accessed by importing the lua module. When Lua is the host language, the lua module will already be available in the global Python scope. Below is a description of the functions available in the lua module. -`lua.execute(statement)` +```python +lua.execute(statement) +``` This function will execute the given statement inside the Lua interpreter state. Examples: -``` +```lua >>> lua.execute("foo = 'bar'") lua.eval(expression) ``` @@ -264,7 +266,7 @@ lua.eval(expression) This function will evaluate the given expression inside the Lua interpreter state, and return the result. It may be used to acquire any object from the Lua interpreter state. Examples: -``` +```lua >>> lua.eval("'foo'..2") 'foo2' >>> lua.eval('string') @@ -274,13 +276,15 @@ Examples: 'hello world!' ``` -`lua.globals()` +```lua +lua.globals() +``` Return the Lua global scope from the interpreter state. Examples: -``` +```lua >>> lg = lua.globals() >>> lg.string.lower("Hello world!") 'hello world!' @@ -292,12 +296,14 @@ Examples: Hello world! ``` -`lua.require(name)` +```lua +lua.require(name) +``` Executes the require() Lua function, importing the given module. Examples: -``` +```lua >>> lua.require("testmod") True >>> lua.execute("func()") @@ -312,22 +318,26 @@ Unlike Python, Lua has no default path to its modules. Thus, the default path of Unfortunately, there's a minor inconvenience for our purposes regarding the Lua system which imports external shared objects. The hardcoded behavior of the loadlib() function is to load shared objects without exporting their symbols. This is usually not a problem in the Lua world, but we're going a little beyond their usual requirements here. We're loading the Python interpreter as a shared object, and the Python interpreter may load its own external modules which are compiled as shared objects as well, and these will want to link back to the symbols in the Python interpreter. Luckily, fixing this problem is easier than explaining the problem. It's just a matter of replacing the flag RTLD_NOW in the loadlib.c file of the Lua distribution by the or'ed version RTLD_NOW|RTLD_GLOBAL. This will avoid "undefined symbol" errors which could eventually happen. Below is a description of the functions available in the python module. -`python.execute(statement)` +```python +python.execute(statement) +``` This function will execute the given statement inside the Python interpreter state. Examples: -``` +```python > python.execute("foo = 'bar'") ``` -`python.eval(expression)` +```python +python.eval(expression) +``` This function will evaluate the given expression inside the Python interpreter state, and return the result. It may be used to acquire any object from the Python interpreter state. Examples: -``` +```python > python.execute("import string") > =python.eval("string") @@ -336,12 +346,14 @@ Examples: hello world! ``` -`python.globals()` +```python +python.globals() +``` Return the Python global scope dictionary from the interpreter state. Examples: -``` +```python > python.execute("import string") > pg = python.globals() > =pg.string.lower("Hello world!") @@ -350,12 +362,14 @@ hello world! hello world! ``` -`python.locals()` +```python +python.locals() +``` Return the Python local scope dictionary from the interpreter state. Examples: -``` +```python > function luafunc() >> print(python.globals().var) >> print(python.locals().var) @@ -366,34 +380,40 @@ nil value ``` -`python.builtins()` +```python +python.builtins() +``` Return the Python builtins module dictionary from the interpreter state. Examples: -``` +```python > pb = python.builtins() > =pb.len("Hello world!") 12 ``` -`python.import(name)` +```python +python.import(name) +``` Imports and returns the given Python module. Examples: -``` +```python > os = python.import("os") > =os.getcwd() /home/niemeyer/src/lunatic-python ``` -`python.asattr(pyobj)` +```python +python.asattr(pyobj) +``` Return a copy of the given Python object with an attribute access discipline. Examples: -``` +```python > dict = python.eval("{}") > =dict.keys nil @@ -410,12 +430,14 @@ function: 0x80fa9b8 ['keys'] ``` -`python.asindx(pyobj)` +```python +python.asindx(pyobj) +``` Return a copy of the given Python object with an index access discipline. Examples: -``` +```python > buffer = python.eval("buffer('foobar')") > =buffer[0] stdin:1: unknown attribute in python object @@ -428,12 +450,14 @@ stack traceback: f ``` -`python.asfunc(pyobj)` +```python +python.asfunc(pyobj) +``` Return a copy of the given Python object enclosed in a Lua function closure. This is useful to use Python callable instances in places that require a Lua function. Python methods and functions are automatically converted to Lua functions, and don't require to be explicitly converted. Examples: -``` +```python > python.execute("class Join:\n def __call__(self, *args):\n return '-'.join(args)") > join = python.eval("Join()") > =join