Dynamic array and array view types

[capr]

Here's two basic libs that we could use as a starting point for a discussion:

https://github.com/luapower/dynarray
https://github.com/luapower/arrayview

arrayview is a struct with just a pointer & length that maps the idea of a finite array over a given buffer so that you can apply bound-checked access, copying, sorting, etc. over that buffer. dynarray is a malloc'ed arrayview, so basically an arrayview with additional methods and overloads for inserting and removing elements, etc. (and it actually uses arrayview instead of duplicating its code). The APIs are documented in the source code for now for many reasons (for one, this code was already been rewritten once and it will be further modified with increased usage).

Two points I want to make on these libs.

1. The necessity of a tier-0 lib

Even though they are the most basic libs you can imagine, they are necessarily not dependency-free, as they both depend on the low module (https://github.com/luapower/low) which is my tier-0 lib as discussed in #3. Hopefully it's now easier to see why that kind of lib is needed and what's in it. Here's an incomplete checklist:

• addmethods(T, func) - pattern for declaring methods lazily for containers, see How to implement recursive types? terra#348
• addproperties(T) - adds T.properties which can be quotes or macros so that t.prop redirects to t.properties.prop.
• after_getmethod(), before_entrymissing() etc. - allows a metamethod to be assigned multiple independent handlers -- this way you can have say addmethods() on a type which assigns __getmethod and still assign __getmethod afterwards for a different purpose.
• gettersandsetters() - calls t:get_<name>() for t.<name> and t:set_<name>(val) for t.<name> = <val>
• iif(), min(), max(), assert() - ...
• copy(dst, src, len) - typed memmove
• equal(a, b) - typed memcmp
• alloc() - typed realloc
• hash(uint32|uint64, buf, len, hash) - default hash function
• C headers are included with a wrapper around includec() so that 1) the calls are memoized, 2) symbols are dumped into a single table which the low module inherits so that setfenv(1, 'low') gives unprefixed access to functions like memset etc.

Some of these are one-liners that can be copy-pasted to remove the dependency on the tier-0 lib, but some are not, so we need to talk about #3.

2. Number of users is the only reliable indicator of API quality

As I mentioned before, I think the best way (IMO the only way) to make a good lib is to first have an use case for it and have the lib emerge out of that. I strongly believe that an API is only as good as the number of calls to it that are made from a variety of contexts. I can't think of a stronger indicator of quality than that. An API that's not heavily used by at least 2-3 apps/libs is not ready for prime-time no matter how much design effort is put into it. My libs only have 1-2 users so far and already have undergone major refactorings and even rewrites. I'm not sure what the action point is here, rather than the advice to not waste time with "designing" APIs for imagined use cases: nobody's going to use them, and for good reason.

[aiverson]

I've read through the code you mentioned, and I have some thoughts.

Firstly, there is a comment about adding attributes to methods and functions in order to remove the necessity of a hack in low.t. I wrote the attributes for entries in a struct, so I know how to write the attributes for a function. It shouldn't be difficult.

before_entrymissing, after_entrymissing, etc. are good. We should definitely include them. They might be worth putting in a separate module with a few more metaprogramming utilities.

gettersandsetters should be replaced with usages of the specialentries implementation that I put in the test for struct entry attributes, or something derived from that.

I think hash should be provided with an API like https://doc.rust-lang.org/std/hash/index.html.
The simple buffer hash you provide would be a part of that, but since that hash may include padding bits or union fields that aren't valid, it doesn't produce nearly as good of a hash as a slightly expanded interface.

I very strongly don't want to use setfenv(1, whatever) in the standard library. It obscures the origin of symbols, makes it harder to remember what the environment is and what global variables are available, and it is incompatible with current versions of PUCLua and future versions of LuaJIT/RaptorJIT which creates porting burden for when Terra gets support for a new version of Lua.

The memoization of includec calls is nice, but I don't know how much it buys you since the types from two different header files that are included separately but include a common header can't get merged.

Alloc and realloc I think should be part of a separate module for allocators. I've got a design for this, I can throw it together pretty quickly.

[capr]

You're right about the memoization of includec. It would be nice to have a version of includec which could be called repeatedly with the same environment arg so that header guards could work. Personally I don't even include C headers directly but use the cleaned-up ones that I already use in ffi.cdef which makes loading them 10x faster (down from 0.5s to 0.05s in my case).

[aiverson]

If you are going to transcribe them anyways, it might be worth writing them as Terra externs for even faster load time. It might be possible to partially automate this. Besides that, putting them in a package that gets required already provides one level of memoization. I tend to write my code such that included or ffi cdecls are combined into files with little else in them, possibly a thin convenience wrapper over the raw API, and which have disjoint contents. Then, I put a higher level API on top of that in a different file. That seems to work well enough.

[capr]

Well, I had them already in my many LuaJIT bindings, eg. https://github.com/luapower/cairo/blob/master/cairo_h.lua

And since I already had them like this, I wrote a little wrapper to have them in Terra too:

--forward ffi.cdef() calls to includecstring() so that Terra can use ffi
--cdefs from LuaJIT C bindings instead of loading original header files
--which can load up to 10x slower.
local ffi_builtin_types = [[

typedef          char      int8_t;
typedef unsigned char      uint8_t;
typedef          short     int16_t;
typedef unsigned short     uint16_t;
typedef          int       int32_t;
typedef unsigned int       uint32_t;
typedef          long long int64_t;
typedef unsigned long long uint64_t;

typedef uint64_t size_t;
typedef int64_t  ptrdiff_t;

]]
function low.require_h(...)
	local cdef = ffi.cdef
	ffi.cdef = function(s)
		C(ffi_builtin_types..s,
			--enums are anonymized in some headers because they are boxed in
			--LuaJIT, but C complains about that.
			{'-Wno-missing-declarations'})
	end
	for i=1,select('#',...) do
		require((select(i,...)))
	end
	ffi.cdef = cdef
end

[capr]

Btw, I was wondering when you're going to comment on my use of setfenv :) So let me clarify: I have no intention of imposing these patterns on others and so I have little interest in arguing for them, that is, unless someone actually asks or provides some good counter-arguments, none of which happened here AFAICS.