least significant

Back when I was young child, my brothers and I used to play a game called Moonstone. So it was with a bit of nostalgic glee that I went about spending a few hours or so in evening poking around the innards of the game. I wish I had bothered to take a look at the time, as I managed to remove the unbelievably long, unskippable intro that the game has, which would of saved my brothers and I a few hours of pain, enduring the intro whilst waiting for the game to start. You can find the game itself on various abandonware sites and it runs fine in DOSBox. The game comes with three executables, MS.EXE (the executable to play the game), INTR.EXE (the executable that runs the intro sequence) and MAIN.EXE (the main game itself). DOSBox has a debugger build that allows you to examine memory, step through execution, set breakpoints and more. There's a guide to help you along. The freeware version of IDA, thankfully supports dos exes, so I could open up all three to have a poke around. ...

A while back a group of friends and I went through a spate of playing TTT(Trouble in Terrorist Town), to the point where we ended up playing during that Christmas day post dinner lull. For me, TTT scratches the traitor game mechanic that you can find in board games like The Resistance, Battlestar Galactica and Shadows Over Camelot which you don't really get in any other computer game. Unlike these games, the number of traitors is rigidly defined by the total number of players by a formula. local function GetTraitorCount (ply_count) -- get number of traitors: pct of players rounded down local traitor_count = math.floor (ply_count * GetConVar( "ttt_traitor_pct" ):GetFloat()) -- make sure there is at least 1 traitor traitor_count = math.Clamp(traitor_count, 1 , GetConVar( "ttt_traitor_max" ):GetInt()) return traitor_count end This means that for a given number of players, the number of traitors/innocents is the same each round. This i...

So we had a report about a memory leak when using the ckan datastore extension, where large queries to the datastore would leak large amounts of memory per request. It wasn't simple to get to the bottom of it, at first I couldn't recreate it the leak at all. The test data I was using was the STAR experiment csv files, which I found when I googled 'Large example csv files'. The reporter Alice Heaton, had kindly written a script that would recreate the leak. Even with this, I could not recreate the problem, until I upped the number of rows fetched by a factor of ten. I suspect that Alice has more data per column with perhaps large text fields instead of the mainly numeric data of the STAR experiment data I was using. Once I could reliably recreate the problem, I ended up poking around using heapy , which I've used previously to track down similar problems and inserted some code to setup heapy and an ipdb breakpoint from guppy import hpy hp = hpy() heap ...

If you've taken a look at the ckan source code, you'll have come across 'context' as the first parameter in many of the functions. It basically contains all threadlocal information required for a function to execute. It's taken me far too long to understand why they exist and I currently think contexts in their current state are pointless. I'm assuming contexts only contain model, session and user. I get why context exists, it basically comes down to whether you prefer passing parameters for the request/session/user around into functions, so that the parameters define exactly what the function needs to execute. If we do this though, these are 'different' to other parameters which is why they've been seperated into context/data_dict. Your functions do not have to refer to some magical global object to execute. Or whether you are happy using threadlocals, so that if your logic function, calls another function that calls another, that calls ....

At some point at university I borrowed, read a tiny amount of and got fined for the late return of Large Scale C++ software design. I think somewhere at the beginning there was a section on using headers to avoid circular imports. ckan is a large enough project that we can learn a thing or two here. So we have a ckan python style guide , where we used to recommend avoiding from blah import banana style imports and instead recommended import blah.banana as banana Eventually we decided to allow from imports , because the above is ugly, but the docs were updated to include some additional rules about what you can and cannot import to avoid circular imports. ckan.model often gets imported all over the shop and I believe it was the cause of many circular import problems. Stuff like get_action ended up creating stuff to get around it. A simple example of ckan.model ending up everywhere would be importing from ckan.logic import schema schema would contain all the action fu...

We use sqlalchemy in ckan and often we run into problems with DetachedInstanceErrors. For this reason, we normally jam all the data from a model object into a dictionary to avoid getting the DetachedInstanceError. This isn't sqlachemy's fault, it's more than likely that we are just incapable of using it properly. For a while, we kept on running into issues where paster db clean would hang and I spent a while trying to figure out what the hell was going on and what was wrong with the session. As it turned out, there was nothing wrong session really, it was just that we were handling it badly for command line functions. There were basically 2 problems. Not closing transactions that are implicity opened by sqlalchemy Closing and removing a session and then attempting to use the sqlalchemy objects from that session causing DetachedInstanceErrors To demonstrate this, we're going to use do some fiddling with ckan from the ipython command line shell. We'll st...

(e:feb 2015. I originally wrote this years ago when I was looking for a job and 'having a blog' was a question you were asked at interviews. This is nothing revolutionary, hell I'd rewrite the snippet now so it would yield the rows instead of constructing the entire table and returning it. But I felt good coming up with it back in the day.) Handling time series data using python’s builtin datastructures is a bitch. I guess the best way would be to just use a library like pandas , but a colleague came across the way I’d handled it in this code snippet using itertools and functools and asked me to share it with the rest of the team. I’ll just concentrate on the parts which require more explanation. The most complex line to understand is line 24 List comprehensions and python ‘switch’ row = dict([ (option, function_dict[option](series['list'])) for option in table_options ]) The dict function can build dictionaries in a variety of ways, a list pairs in m...