compiles? ship it!

OK, the lock-free hash table for Lua is working! after a little cleanup of the code, i announced it on the Lua list. Let's hope it stirs some new ideas. interested? get it here!

The code is small, so reading it all and comprehending isn't so hard... the next step should be writing some real tests to see if it really works. Bracing for embarrassment...

Lock-free concurrent Lua?

For some time now I've been thinking about how to make Lua concurrent. That is, having several threads of execution, all in the same Lua space, sharing all data; but without the Big Language Lock that effectively sequentializes execution of most of the code.

Of course, the problem is that LuaThreads (the only Lua multithreading extension that puts all threads on the same space) uses just one lock per space, and all threads fight for this one resource.

One 'obvious' solution would be to use one lock per object. A little better might be to use readers/writers locks, to allow concurrent readings. yep, should work.

But a couple of days ago I saw this: Scalable Nonblocking Data Structures on Slashdot about a guy that got lock-free hashtables with great scalability on 700+ threads (with even more cores than threads!). The article was very simplistic, and seemingly inconsistent; but one /.er pointed to the Google Talk. Now that's inspiring!

I've previously read about lock-free algorithms and data structures, but this guy (that turned to be the Cliff Click)

So, after a couple of days of mulling about it, I'm hacking some code!

The original idea was to see if it's possible to lock-free'ize Lua tables. It's no easy task. The main stumbling block is that it seems that the CAS primitives available in gcc operate on pointer-sized values; since Lua tables store values directly in the hash table (and specially on the array part of the table), it's important to swap values atomically.

Much easier, and what I'm doing now, is to write new table code, lock-free from the start.

Of course, I'm using lots of naïve code in several parts, but hope to get the main ideas set, and clean silly things later (initialization right now looks very slow). Also, I still have no idea how to do the array part, and there are some unresolved issues on table resizing; but they all seem doable. And the code is coming surprisingly clean.

:-)

Getting Lua in Apache

Used the few hours i had available this weekend trying to see what options are there to get Lua running as an Apache2 module. Shortly put, i knew about two routes: Kepler's mod2 and mod_wombat (which is supposed to be part of apache some day)

Asked about the source for mod_wombat and promptly got what i needed (thanks Brian!). looked around in the docs. looks really similar to mod_python (and i guess to mod_perl??), it can keep a pool of LuaStates, with some code preexecuted on each process, and call a specified function to process client requests. just what i hoped.

now the bad news... it says "You may user (and I encourage you to!) the threaded MPMs"; but PHP5 doesn't run in a threaded apache. ??? i had no idea it was so limited! so i guess the preforked MPM is the most used apache build. who runs apache without PHP?

then came Kepler's mod2 time. i didn't have high hopes on this because the "build everything from scratch for each request" architecture in Kepler was what motivated me to write what later became Xavante.

looking around in the mod2 launcher code confirmed it: there's a small Lua layer that is clearly called with a new, clean LuaState for each request. useless.

but i also found a nice preprocessor switch: some #if LUA_STATE_PER_REQUEST and #if LUA_STATE_PER_PROCESS to change where and when are the LuaStates created! could that be what i was looking for? of course, i was sure that code would be far less tested than the 'standard' LUA_STATE_PER_REQUEST; but still it would be easier to debug it with the kepler guy's help. but... no, it can pre-create the LuaState, but it doesn't execute _any_ code! still waits for the client's request to load, compile and execute the main Lua files. Ugh.

in conclusion: it will be mod_wombat (as i could have guessed from the start); but it's a pity not to use it how the author "encourages" to do.