Running kdb+ on Windows

In addition to our cloud hosted solution, Tapaas can be deployed on premise. We are currently working with an FX Broker who is a Windows Shop. We are deploying an FX Tick Data analytics environment based on kdb+tick from Kx Systems.  Because all my experience has been deploying kdb+ on Linux, I thought there are probably many others who would benefit from my sharing the how-to and gotchas when running kdb+ on Windows.

Here is a list of key decisions that we had to make:

• Do we use  a Framework for management of Kdb or just roll our own?
• How do we run Kdb as a Service on Windows?
• Do I really have to use CMD.EXE (dreaded .bat files) or PowerShell to automate things?
• Is there any way my scripts can run on both Linux and Windows without change?

You might also want to read my Linux Dev’s Windows Survival Guide post where I share what I found to make life in Windows bearable.

AquaQ TORQ Framework

First off, kdb+ is quite sparse by design.  Therefore, most sites will require addition of several utility libraries and functions which provide a sort of Aspect Oriented Programming or AOP.  For example, consistent logging, multiple timers, security and usage and performance logging.  Because all of this adds overhead, kdb+ does not include any of this out of the box.  However, there is a rich collection of code contributed by kdb+ users that can be used.

The folks at AquaQ have built the TORQ framework by curating and socialising what they believe are the essential bits from the contributed code and bundled it into a concise framework.

Config changes required with TORQ

TORQ is pretty well ready to run with very few changes.  However, like anything, these bootstrap settings might not be appropriate for your environment or use case.  They have selected sensible defaults.  But for best results, you should familiarise yourself with the features of TORQ and review these settings to assure they are appropriate.

1. Set the path and environment variables – in the startup script, there are
2. Define your schema – this is an important task.  Stay tuned for more details on some critical schema design decisions.
3. Modify the launch scripts as the schema name is a command line argument for the tickerplant.q
4. Edit the <installdir>/config/process.csv file.  This file has hosts and ports for various

Running kdb+ as a Windows Service

As it turns out, there are a number of choices for running arbitrary programs and scripts as a Service on Windows.

SrvAny – RKTools.exe

The kx.com cookbook has one suggestion which uses the SrvAny.exe program that was provided by Microsoft in rktools.exe.  I retrieved the following details that weren’t included in the kx cookbook, from stackoverflow:

For Windows Home Server or Windows Server 2003 (works with WinXP too), the Windows Server 2003 Resource Kit Tools comes with utilities that can be used in tandem for this, called instsrv.exe and srvany.exe. See this Microsoft KB article KB137890 for details on how to use these utils.

This option does work… But it’s not that nice to have to edit registry keys for your command line options, etc..

Cygrunsrv

As you suspected, this is part of the Cygwin platform.  Of course it requires install of Cygwin.  But it is quite easy to use, especially if you are used to Unix.  I learned about this when looking at how to run sshd on Windows – which is essential if you want to manage Windows remotely using familiar tools like ssh and rsync.

This option might not be acceptable to your client’s Windows Administrators.  It certainly met with a serious cringe at my last client.  So I’m shying away from it.

Python Wrapper

This is probably my favourite because it kills two birds with one stone.  Since Python is a powerful scripting language that is cross platform, if you write your Kdb management scripts with Python you can run your scripts and Kdb of course, cross platform as well.  Python has a fabulous library call ‘os’ which has methods that adapt annoying platform differences like flipping your ‘/’ to ‘\’ in paths, etc..  You might need to write a few:

if os.name = ‘nt’:

But that’s easy compared to maintaining two sets of scripts.

Further, there is a nice solution for writing a Python Windows Service in Stackoverflow.

Non-Sucking Service Manager (nssm)

Lastly, there is the Non-Sucking Service Manager.  Apparently, the authors believe the existing offerings (like SrvAny.exe) suck.. So they wrote this.  I haven’t tried it because I basically prefer the Python option above.  If you try it and it’s great, please come back and leave a comment.