I’ll be presenting Tapaas Alpha at Finance Magnates Summit. Tapaas Alpha is our Real-time Timeseries Data Analytics platform for Risk Management which we’ve built specifically for Retail FX Brokers and Prime Brokers.
Continue readingWhy we use Kdb+ from Kx Systems
When it came time to choose technologies for the implementation of Tapaas, there was really no decision needed. We knew we had to use KDB+ as the core engine to give us the real-time performance and flexibility that we needed in our product.
Tapaas provides streaming as well as historical analytics so we knew we needed query processing and data storage layers that worked well together in both those use cases. Dave and I had plenty of experience with Kdb+ in our previous lives at big bulge bracket firms on workloads far more demanding than what is envisioned for Tapaas. And we are both big fans of the product.
Ever since I started working with KDB+ more than 10 years ago, I found the power and simplicity of KDB+ compelling for both real-time streaming and historical time series applications. For a start, it has brute data handling speed which is essential for delivering true low latency streaming analytics in environments where data volumes seem to grow daily. But perhaps its greatest asset is its query language Q which is tightly woven into its data architecture. Q is based on a functional programming model, which allows even the most complex logic to be expressed succinctly and efficiently directly in the database. There are no data access layers, no adaptor logic, no unnecessary data translations between database semantics and analytics semantics. The data is just readily amenable to be processed by our logic through a comprehensive set of powerful functions. And given the scale of KDB+ applications we have worked with previously, we have no doubts we have a great growth path as Tapaas clients seek to gain more and more insights into their data.
Running kdb+ on Windows
In addition to our cloud hosted solution, Tapaas can be deployed on premise. We are currently working with an FX Broker who is a Windows Shop. We are deploying an FX Tick Data analytics environment based on kdb+tick from Kx Systems. Because all my experience has been deploying kdb+ on Linux, I thought there are probably many others who would benefit from my sharing the how-to and gotchas when running kdb+ on Windows.
Here is a list of key decisions that we had to make:
- Do we use a Framework for management of Kdb or just roll our own?
- How do we run Kdb as a Service on Windows?
- Do I really have to use CMD.EXE (dreaded .bat files) or PowerShell to automate things?
- Is there any way my scripts can run on both Linux and Windows without change?
You might also want to read my Linux Dev’s Windows Survival Guide post where I share what I found to make life in Windows bearable.
AquaQ TORQ Framework
First off, kdb+ is quite sparse by design. Therefore, most sites will require addition of several utility libraries and functions which provide a sort of Aspect Oriented Programming or AOP. For example, consistent logging, multiple timers, security and usage and performance logging. Because all of this adds overhead, kdb+ does not include any of this out of the box. However, there is a rich collection of code contributed by kdb+ users that can be used.
The folks at AquaQ have built the TORQ framework by curating and socialising what they believe are the essential bits from the contributed code and bundled it into a concise framework.
Config changes required with TORQ
TORQ is pretty well ready to run with very few changes. However, like anything, these bootstrap settings might not be appropriate for your environment or use case. They have selected sensible defaults. But for best results, you should familiarise yourself with the features of TORQ and review these settings to assure they are appropriate.
- Set the path and environment variables – in the startup script, there are
- Define your schema – this is an important task. Stay tuned for more details on some critical schema design decisions.
- Modify the launch scripts as the schema name is a command line argument for the tickerplant.q
- Edit the <installdir>/config/process.csv file. This file has hosts and ports for various
Running kdb+ as a Windows Service
As it turns out, there are a number of choices for running arbitrary programs and scripts as a Service on Windows.
SrvAny – RKTools.exe
The kx.com cookbook has one suggestion which uses the SrvAny.exe program that was provided by Microsoft in rktools.exe. I retrieved the following details that weren’t included in the kx cookbook, from stackoverflow:
For Windows Home Server or Windows Server 2003 (works with WinXP too), the Windows Server 2003 Resource Kit Tools comes with utilities that can be used in tandem for this, called instsrv.exe and srvany.exe. See this Microsoft KB article KB137890 for details on how to use these utils.
This option does work… But it’s not that nice to have to edit registry keys for your command line options, etc..
Cygrunsrv
As you suspected, this is part of the Cygwin platform. Of course it requires install of Cygwin. But it is quite easy to use, especially if you are used to Unix. I learned about this when looking at how to run sshd on Windows – which is essential if you want to manage Windows remotely using familiar tools like ssh and rsync.
This option might not be acceptable to your client’s Windows Administrators. It certainly met with a serious cringe at my last client. So I’m shying away from it.
Python Wrapper
This is probably my favourite because it kills two birds with one stone. Since Python is a powerful scripting language that is cross platform, if you write your Kdb management scripts with Python you can run your scripts and Kdb of course, cross platform as well. Python has a fabulous library call ‘os’ which has methods that adapt annoying platform differences like flipping your ‘/’ to ‘\’ in paths, etc.. You might need to write a few:
if os.name = ‘nt’:
But that’s easy compared to maintaining two sets of scripts.
Further, there is a nice solution for writing a Python Windows Service in Stackoverflow.
Non-Sucking Service Manager (nssm)
Lastly, there is the Non-Sucking Service Manager. Apparently, the authors believe the existing offerings (like SrvAny.exe) suck.. So they wrote this. I haven’t tried it because I basically prefer the Python option above. If you try it and it’s great, please come back and leave a comment.
The Analytics Platform as a Service
Tapaas, The Analytics Platform as a Service, is a platform designed to help you profit from insight.
Continue reading