Big Data and high-frequency data with kdb+/q

Servers

Overview

q is a programming language for array processing, developed by Arthur Whitney on the basis of Kenneth E. Iverson’s APL. The kdb+ database built on top of q is a de facto standard technology for dealing with rapidly arriving, high-frequency, big data.

kdb+/q has taken the world of electronic, including algorithmic, trading by storm. It is used by numerous sell-side and buy-side institutions, including some of the most successful hedge funds and electronic market makers.

Beyong the world of electronic trading, kdb+/q is used in retail, gaming, manufacturing, telco, IoT, life sciences, utilities, and aerospace industries.

Your course will take you through the foundations of kdb+/q and explain why it is a language of choice for Big Data, high-frequency data, and real-time event processing.

We shall explain how to work with tables and q-sql effectively, how to set up tickerplants, real-time, and historical instances, and how to apply kdb+/q to machine learning problems.

We shall consider advanced applications to tree-based regression and classification, random forests, deep learning, Google DeepMind and Monte Carlo search, producing demonstrations on real-life data examples.

Instructors

  • Paul Alexander Bilokon, PhD
  • Jan Novotny, PhD

Venue

The course will take place on Level39.

Schedule

TimeActivity
08:30 – 09:00Registration and welcome, a tour of Level39
09:00 – 10:00Lecture 1: Foundations of kdb+ and the q programming language
10:00 – 10:30Tutorial 1
10:30 – 11:00Coffee break
11:00 – 12:00Lecture 2: Working with tables and q-sql
12:00 – 12:30Tutorial 2
12:30 – 13:30Lunch
13:30 – 14:30Lecture 3: Big data in kdb+/q
14:30 – 15:00Tutorial 3
15:00 – 15:30Coffee break
15:30 – 16:30Lecture 4: Tickerplant architecture for data captures
16:30 – 17:00Tutorial 4
17:00 – 18:00Lab

Syllabus

  • Foundations of kdb+ and the q programming language
  • Working with tables and q-sql
  • Big data in kdb+/q
  • Tickerplant architecture for data captures

Bigliography

Your course is designed to be self-contained. However, if you would like to read up on the content before, during, and/or after the course, we recommend the following books:

  • Jeffry A. Borror. q For Mortals Version 3: An Introduction to q Programming. q4m LLC, 2015.
  • Nick Psaris. Q Tips: Fast, Scalable and Maintainable Kdb+. Vector Sigma, 2015.
  • Jan Novotny, Paul Bilokon, Aris Galiotos, Frederic Deleze. Machine Learning and Big Data with kdb+/q. Wiley, 2019.