Python libraries for working with data¶

Motivation¶

Two third-party (but de facto standard) Python libraries play a particular role in Data Analysis: pandas and NumPy.

The data types defined in pandas — DataFrame and Series — enable the data scientist to work effectively with tables. pandas provides some of the functionality that we find in SQL databases and spreadsheets. Since much of the data that we deal with comes in the form of tables, pandas is extremely useful.

NumPy provides a lower-level type — the multidimensional array. This data type is used as an underlying implementation by pandas DataFrames and Series. NumPy arrays are used wherever bulk operations on numbers are needed in Python, which is what much of Data Science is about. NumPy arrays can be seen as a Python implementation of the key linear algebra object — the matrix.

Objectives¶

1. To introduce pandas DataFrames and Series.
2. To show how… …to quickly inspect a pandas DataFrame.
3. …pandas DataFrames can be indexed using loc and iloc.
4. …DataFrame‘s columns can be indexed.
5. …to iterate through columns of a DataFrame.
6. …a DataFrame can be effectively summarised.
7. …to add a column to a DataFrame.
8. …to overwrite a column in a DataFrame.
9. …to rearrange columns in a DataFrame.
10. …to delete a column from a DataFrame.
11. …to apply a function to a column in a DataFrame.
12. …to filter data in a DataFrame based on some boolean expression.
13. …to deal with missing data (NaNs).
14. …to use groupby.
15. …to append to a DataFrame.
16. …to join on a DataFrame.
17. …to get at the NumPy arrays behind the pandas DataFrame and Series.
18. …to define one-dimensional (flat) NumPy arrays.
19. …to use np.arange, np.linspace, and np.logspace.
20. To explain the reasons to prefer NumPy arrays over standard Python data structures, such as lists.
21. To show how… …to define two-dimensional NumPy arrays, thus implementing matrices.
22. …to create NumPy arrays using np.zeros, np.ones, np.full, np.empty, np.tile.
23. …to use 32-bit floating point numbers (float32) instead of 64-bit floating point numbers in NumPy arrays.
24. …to generate random matrices.
25. …to reshape matrices.
26. …to multiply matrices by scalars.
27. …to add matrices together.
28. …to multiply matrices together.
29. …to mix arrays and scalars in arithmetic operations; to explain broadcasting.
30. …to transpose matrices.
31. …to invert matrices.
32. …to create identity matrices.
33. …to stack NumPy arrays horizontally.
34. …to stack NumPy arrays vertically.
35. …to index NumPy arrays, including boolean indices.
36. …to compare NumPy arrays.
37. …to use some useful functions: np.cumsum, np.cumprod, np.min, np.max, np.argmin, np.argmax, np.mean, np.var, np.std.
38. …to apply a function to each row/column in a NumPy array.
39. …to use NumPy array flags.
40. …to make NumPy arrays immutable (kind of).
41. …to find out more about the configuration of the NumPy library, e.g. which BLAS it is using.

pandas DataFrames and Series¶

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. We can import it with

The key data type provided by pandas is DataFrame. Let us create one:

pandas DataFrames in many ways resemble SQL database tables and worksheets in spreadsheet applications.

How to quickly inspect a DataFrame¶

Let’s examine our newly created DataFrame:

In practice, we may be dealing with DataFrames containing thousands of rows of data (not fifteen, as in our example). So instead of looking at the entire DataFrame we may look at its…

We may wish to look at more rows from the DataFrame‘s head:

Or examine its tail:

The index of the DataFrame, loc and iloc¶

Notice the numbers in bold on the left. These constitute the index of the DataFrame:

We can access the list of columns of the data frame through

Thus the index indexes the rows, whereas the column names index the columns. We can access individual rows of the DataFrame through

Notice that the index of the data frame doesn’t have to be formed from integers starting with zero, as in our case

We could have chosen to index the rows with strings, 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O'. Since the DataFrame is a mutable object, i.e. it can be changed after construction, we can replace its index accordingly:

Now we can index the rows with

Whatever the index of the DataFrame, we can also refer to the rows through their integer offsets from the top, using iloc instead of loc:

Whereas the type of the entire object is

The type of df.loc['D'] (equivalently, of df.iloc[3]) is

Series is the second most important type defined in the pandas package (after DataFrame).

Indexing columns¶

As we said above, the list of columns in the dataframe is accessible through

Individual columns can be indexed using

The type of the object thus obtained is

You can also access multiple columns at once using

The type of the resulting object is DataFrame, rather than Series:

Iterating through columns¶

Let us demonstrate how we can iterate through the columns of a DataFrame. In this example we shall store the first (0th) value from each column in a dictionary:

Summarising the DataFrame¶

Another quick way to obtain a high-level view of a (potentially very large) DataFrame is with

whereas the following will present summary statistics for each column:

Adding a column¶

To add a column, assuming the column 'house age' does not yet exist:

Overwriting a column¶

Similarly, if the column 'house age' already exists,

Rearranging the columns¶

To rearrange the columns:

Deleting a column¶

Applying a function to a column¶

Notice that the original column hasn’t been overwritten:

To overwrite it,

To reverse what we have just done:

Filtering data¶

Suppose that we want to consider only those rows where the value in the column 'number of convenience stores' is greater than or equal to 7. We can then index df by

obtaining, as a result,

If we want to consider those rows where the 'number of convenience stores' is greater than or equal to seven or the distance to the nearest MRT station is less than 500, we index the dataframe by

The result being

If instead we want to consider those rows where the 'number of convenience stores' is greater than or equal to seven and the distance to the nearest MRT station is less than 500, we index the dataframe by

The result being

Suppose that we set df1 to the above…

We get a SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. The slice (the view of a subset of the original DataFrame) has changed:

Whereas the original has not changed:

For more information on this behaviour, read http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

Dealing with missing data¶

Let us examine what happens if we have missing data in the DataFrame.

To this end, let us take a copy

and modify it like so:

The special np.nan value indicates that the data is missing or invalid (NaN stands for “not a number”). NaNs often result from numerical calculations. Their presence may interfere with further numerical work. One quick way to address the issue of NaNs is by dropping (removing) them:

This has returned a copy of the DataFrame with all rows containing NaNs removed. df_copy itself has not been changed:

But we could overwrite it with the result of dropna():

groupby¶

It may be useful to summarise the pandas DataFrame for a given value of a particular column. This can be achieved using groupby. For example, we may wish to look at the mean of each column for each possible value of 'number of convenience stores':

It looks like the house price per unit area may increase with the number of convenience stores, although we aren’t particularly certain about this conclusion, whereas the distance to the nearest MRT station does seem to decrease as the number of convenience stores increases.

Instead of the mean, we may consider the minimum:

Or maximum:

Appending to a DataFrame¶

Let us again consider our DataFrame,

Suppose that we have another DataFrame,

We can append df1 to the end of df using

Joining on a DataFrame¶

Let us again consider our DataFrame,

Suppose we have another DataFrame,

We can join df_comments onto df based on the matching indices:

We may wish to replace the NaNs that have resulted in the 'comments' column with blank strings:

Exercise¶

Consider the DataFrame

Replace the default index with the parsed dates from the 'date' column. Once this is done, remove the 'date' column.

Exercise¶

Consider again the DataFrame

Find the mean, minimum, and maximum 'value'. Use the library functions numpy.mean, numpy.min, and numpy.max.

Exercise¶

Consider the DataFrame

Join it with another DataFrame so that, for dates 2019-09-03 and 2019-09-06 the comment 'missing data' is added in the 'comments' column.

Exercise¶

Consider the DataFrame

Two new data points become available. 9.89 for 2019-09-08 and 3.89 for 2019-09-09. Append them to the DataFrame.

Exercise¶

In the DataFrame resulting from appending the data in the previous exercise, find the mean value for each day of the weak (Monday, Tuesday, Wednesday, etc.). You can get a weekday from a date using

NumPy arrays¶

Many Python libraries (such as pandas) rely on NumPy under the hood. NumPy, imported with

implements multidimensional arrays.

NumPy arrays behind the pandas DataFrame and Series¶

To access the underlying NumPy array of a pandas DataFrame, we can use values:

This gives us the raw numerical data. Similarly, we can access the NumPy array behind the Series representing a specific column:

Or row:

Defining one-dimensional (flat) NumPy arrays¶

Defining a one-dimensional (flat) array is easy: you can simply wrap a Python list:

np.arange, np.linspace, np.logspace¶

Often we want to obtain a result similar to that of range but for floating point numbers (rather than integers); we want to obtain an array of floating point numbers at equally spaced intervals (e.g. for plotting). In NumPy this is attained using linspace. Thus to obtain an array of fifteen equally spaced floating point numbers starting at -5. (inclusive), finishing at 10. (inclusive), we can use

Notice that the arguments specifying the start and end of the interval are both inclusive: the first value is exactly -5., the last value is exactly 10.. This is unlike the arguments of Python’s standard range: the start of the range is inclusive, but the end of the range is exclusive:

Also note that the third argument of np.linspace is the total number of points to be produced, whereas the third argument of range is the step size. NumPy has a function np.arange, which is similar to range, but works with floating point arguments and produces NumPy arrays:

np.linspace and np.arange are particularly useful for plotting:

We need the following cell magic to enable plotting in a Jupyter notebook:

And then we plot using the library Matplotlib (also known as pyplot):

Instead of linearly (equally) spaced points, we can produce points spaced logarithmically using np.logspace:

Here we start with the zeroth power of ten and end with the fifth power of 10.

Reasons to prefer NumPy arrays over standard Python data structures, such as lists¶

Suppose that we want to compute the sine of a number of regularly spaced angles (perhaps with a view of plotting the resulting graph):

We can time the creation of each of the arrays, xs and ys, using the timeit cell magic:

The creation of xs took about 6 milliseconds.

The creation of ys took about 7 milliseconds.

We can replace the above Python lists with NumPy arrays and initialise them using vectorised operations (vectorised operations work on many values in one go), np.linspace and np.sin:

Let us time these operations.

Vectorised operations on NumPy arrays take hundreds of microseconds, as opposed to milliseconds for Python lists, thus we have an order of magnitude improvement in speed.

Performance is one reason to prefer NumPy arrays over standard Python data structures, such as lists.

Defining two-dimensional NumPy arrays (implementing matrices)¶

The real power of NumPy is in supporting two-dimensional, arrays, which can implement matrices:

This particular two-dimensional NumPy array (matrix) is 2 by 3:

Other ways of creating NumPy arrays¶

Instead of creating NumPy arrays from (either nested or flat) Python lists, we coul