Big Data and High-Frequency Data with kdb+/q
Time series databases are increasingly being recognized as a powerful way to manage big data. They can be used to instrument, learn, and automate applications and systems, enable real-time and predictive analytics across manufacturing processes, and provide operational intelligence to make improved and faster decisions based on what is occurring now, what has occurred in the past, and what is predicted to take place in the future.
Kdb+ is a time series database optimized for big data analytics. The columnar design of kdb+ means it offers greater speed and efficiency than typical relational databases and its native support for time series operations vastly improves both the speed and performance of queries, aggregation, and analysis of structured data.
Kdb+ is also different from other popular databases because it has built-in proprietary languages, k and q, allowing it to operate directly on the data in the database, removing the need to ship data to other applications for analysis. Kdb+ and q make full use of the intrinsic power of modern multi-core hardware architectures. Kdb+ supports a number of different interfaces with a very simple API, for easy connectivity to external graphical reporting, and legacy systems.
In the world of high-frequency trading, the kdb+ database and its underlying programming languages, k and (especially) q, have risen to the top of the ranks as tools for implementing quantitative analyses of all types.
“A lot of kdb+ jobs are project-based, and very well paid,” says Olly Thompson, an electronic trading systems recruiter at GQR Global Markets. “The reason for that is that there are simply not a lot of kdb+ developers out there.”
A senior quant developer at U.S. bank, who works in kdb+, confirms that he is one of a select tribe. “Skilled kdb+ engineers are hard to come by–and skilled kdb+ quants are even more rare.”
Although kdb+/q has a steep learning curve, once mastered, it becomes an exceedingly fast, convenient, and indispensable tool.
As the volume of data and speed at which it arrives continue to grow, traditional relational database management systems are facing ever more challenging workloads that they were never designed to support. For completely independent and audited performance benchmarks, the Security Technology Analysis Center (STAC) Benchmark Council has a number of tests comparing low-latency, high-volume technologies, and kdb+ features well in STAC results.
Przemek Tomczak explains:
“Kdb+ is ideally suited for these demands because of its unique combination of a higher-performance in-memory, columnar and relational database with an integrated vector-oriented programming system. Our customers are using kdb+ to get significant improvements to the performance and scalability of their applications in the face of these data volumes, particularly for supervisory control and data acquisition, data historians, fault detection and prediction, advanced data warehouses, and capital markets trading and surveillance systems.”
- Introduction to kdb+/q
- Installing kdb+/q
- Variables, types, and operators
- Control flow
- Functional forms
- Input and output
- Splayed and partitioned tables
- Inter-process communication and integration
- Kdb+/q devops
- Level 2 data
- Introduction to kdb+/tick
- Your course provides a solid foundation in the q programming language underlying the kdb+ database.
- It is designed to make you a proficient kdb+ user and developer in the shortest possible time.
- We have over a decade of experience teaching kdb+/q both in-house and to wider audiences.
- Your instructors are known for their clear, in-depth explanations and attention to detail.
- We know the challenges that you will encounter on this path and carefully delineate all caveats.
- We provide numerous examples of realistic kdb+/q usage.
- Numerous interactive exercises help you learn and integrate the material.
- You learn how to apply your knowledge to a real production high-frequency database of cryptocurrency data.
|Time||Day 1||Day 2||Day 3||Day 4|
|09:30 - 10:00||Registration and welcome||Registration and welcome||Registration and welcome||Registration and welcome|
|10:00 - 10:45||Introduction to kdb+/q||Control flow||Joins||Parallelization|
|10:45 - 11:30||Installing kdb+/q||Functions||Functional forms||Kdb+/q devops|
|11:30 - 12:15||Variables, types, and operators||Iterators||Input and output||Level 2 data|
|12:15 - 13:00||Lists||Tables||Splayed and partitioned tables||Simulation|
|13:00 - 13:45||Dictionaries||Q-sql||Inter-process communication and integration||Introduction to kdb+/tick|
|13:45 - 14:00||Test||Test||Test||Test|
Paul Bilokon, PhD
CEO and Founder of Thalesians Ltd. Previously served as Director and Head of global credit and core e-trading quants at Deutsche Bank, the teams that he helped set up with Jason Batt and Martin Zinkin. Having also worked at Morgan Stanley, Lehman Brothers, and Nomura, Paul pioneered electronic trading in credit with Rob Smith and William Osborn at Citigroup.
Paul has graduated from Christ Church, University of Oxford, with a distinction and Best Overall Performance prize. He has also graduated twice from Imperial College London.
Paul’s lectures at Imperial College London in machine learning for MSc students in mathematics and finance and his courses consistently achieve top rankings among the students.
Paul has made contributions to mathematical logic, domain theory, and stochastic filtering theory, and, with Abbas Edalat, has published a prestigious LICS paper. Paul’s books are being published by Wiley and Springer.
Dr Bilokon is a Member of the British Computer Society, Institution of Engineering and Technology, and European Complex Systems Society.
Paul is a frequent speaker at premier conferences such as Global Derivatives/QuantMinds, WBS QuanTech, AI, and Quantitative Finance conferences, alphascope, LICS, and Domains.
WBS Training Ltd organizes workshops and conferences for the capital markets and treasury divisions of investment companies worldwide, with all our efforts centred solely on the education of our clients. WBS Training does not operate to present dozens of events every year. Instead we select only the most innovative, pertinent and dynamic subjects, thus bridging the gap between the latest theoretical developments through to proven practical trading floor requirements. Therefore, we aim to ensure that such requirements can be effectively implemented in the real financial world.
Our depth of experience within the training environment provides us with a greater knowledge and understanding of what our clients require from financial business training. This promotes the unique position of us delivering the quality and service that is crucial to our client’s continued success and competitive advantage in the market place.
We have a flexible approach which allows for a more personal relationship with our clients. Furthermore, we do listen to what our clients want. WBS Training looks forward to meeting your company’s ever changing requirements, and welcomes you to our website.