Posts

Showing posts from 2015

SQLTap and CKAN

Are you a CKAN developer? Have you had a trip or fall at work involving a slow running sql query that wasn't your fault? Do you terribly miss the django debug toolbar , pyramid debugtoolbar , flask  debugtoolbar or similar? Well you're basically screwed, but to help alleviate the pain I made a tiny ckan extension that integrates the SQLTap middleware with ckan, once you've added 'sqltap' to your list of plugins in your development init you'll find the sqltap interface at http://localhost:5000/__sqltap__ When you run the paster development server. Please don't cry when you see the results there, we'll get round to implementing dogpile caching eventually

Translating ckan extensions using the ITranslations

From ckan 2.5 onwards you will be able to translate the strings in ckan extensions in a much more friendly and easy way. Unless there are any major issues in the code review, this pull request should make your life easier. Previously, there were a few, not ideal solutions (see ckan#959 ) which involved having a paster command or script that would munge all the po/mo files together using gettext's msgcat command. This has the downside that the sysadmin of any ckan instance would have to run this script and whatever other series of texts whenever they had a new ckan extension that they wanted to add. The new pull request allows extension writers to provide a translations that will automatically be included without any additional steps other than the standard step specifying the plugin in the ckan.plugins in the configuration file. To do this you'll need to copy the example plugin and edit your plugin so it looks a bit like ... from ckan.lib.plugins import DefaultTr

An Evening With Assembly

Image
Back when I was young child, my brothers and I used to play a game called Moonstone. So it was with a bit of nostalgic glee that I went about spending a few hours or so in evening poking around the innards of the game. I wish I had bothered to take a look at the time, as I managed to remove the unbelievably long, unskippable intro that the game has, which would of saved my brothers and I a few hours of pain, enduring the intro whilst waiting for the game to start. You can find the game itself on various abandonware sites and it runs fine in DOSBox. The game comes with three executables, MS.EXE (the executable to play the game), INTR.EXE (the executable that runs the intro sequence) and MAIN.EXE (the main game itself). DOSBox has a debugger build that allows you to examine memory, step through execution, set breakpoints and more. There's a guide to help you along. The freeware version of IDA, thankfully supports dos exes, so I could open up all three to have a poke around.

Randomising traitor numbers in Trouble in Terrorist Town

A while back a group of friends and I went through a spate of playing TTT(Trouble in Terrorist Town), to the point where we ended up playing during that Christmas day post dinner lull. For me, TTT scratches the traitor game mechanic that you can find in board games like The Resistance, Battlestar Galactica and Shadows Over Camelot which you don't really get in any other computer game. Unlike these games, the number of traitors is rigidly defined by the total number of players by a formula. local function GetTraitorCount (ply_count) -- get number of traitors: pct of players rounded down local traitor_count = math.floor (ply_count * GetConVar( "ttt_traitor_pct" ):GetFloat()) -- make sure there is at least 1 traitor traitor_count = math.Clamp(traitor_count, 1 , GetConVar( "ttt_traitor_max" ):GetInt()) return traitor_count end This means that for a given number of players, the number of traitors/innocents is the same each round. This i

Digging into python memory issues in ckan with heapy

Image
So we had a report about a memory leak when using the ckan datastore extension, where large queries to the datastore would leak large amounts of memory per request. It wasn't simple to get to the bottom of it, at first I couldn't recreate it the leak at all. The test data I was using was the STAR experiment csv files, which I found when I googled 'Large example csv files'. The reporter Alice Heaton, had kindly written a script that would recreate the leak. Even with this, I could not recreate the problem, until I upped the number of rows fetched by a factor of ten. I suspect that Alice has more data per column with perhaps large text fields instead of the mainly numeric data of the STAR experiment data I was using. Once I could reliably recreate the problem, I ended up poking around using heapy , which I've used previously to track down similar problems and inserted some code to setup heapy and an ipdb breakpoint from guppy import hpy hp = hpy() heap

wtf is this context thing in ckan?

If you've taken a look at the ckan source code, you'll have come across 'context' as the first parameter in many of the functions. It basically contains all threadlocal information required for a function to execute. It's taken me far too long to understand why they exist and I currently think contexts in their current state are pointless. I'm assuming contexts only contain model, session and user. I get why context exists, it basically comes down to whether you prefer passing parameters for the request/session/user around into functions, so that the parameters define exactly what the function needs to execute. If we do this though, these are 'different' to other parameters which is why they've been seperated into context/data_dict. Your functions do not have to refer to some magical global object to execute. Or whether you are happy using threadlocals, so that if your logic function, calls another function that calls another, that calls .

python programmer discovers C headers.

Image
At some point at university I borrowed, read a tiny amount of and got fined for the late return of Large Scale C++ software design. I think somewhere at the beginning there was a section on using headers to avoid circular imports. ckan is a large enough project that we can learn a thing or two here. So we have a ckan python style guide , where we used to recommend avoiding from blah import banana style imports and instead recommended import blah.banana as banana Eventually we decided to allow from imports , because the above is ugly, but the docs were updated to include some additional rules about what you can and cannot import to avoid circular imports. ckan.model often gets imported all over the shop and I believe it was the cause of many circular import problems. Stuff like get_action ended up creating stuff to get around it. A simple example of ckan.model ending up everywhere would be importing from ckan.logic import schema schema would contain all the action fu