Skip to main content
`Ctrl`+`K`

# Sparse data structures

pandas provides data structures for efficiently storing sparse data. These are not necessarily sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (`NaN` / missing value, though any value can be chosen, including 0) is omitted. The compressed values are not actually stored in the array.

    In [1]: arr = np.random.randn(10)

    In [2]: arr[2:-2] = np.nan

    In [3]: ts = pd.Series(pd.arrays.SparseArray(arr))

    In [4]: ts
    Out[4]:
    0    0.469112
    1   -0.282863
    2         NaN
    3         NaN
    4         NaN
    5         NaN
    6         NaN
    7         NaN
    8   -0.861849
    9   -2.104569
    dtype: Sparse[float64, nan]

Notice the dtype, `Sparse[float64, nan]`. The `nan` means that elements in the array that are `nan` aren’t actually stored, only the non-`nan` elements are. Those non-`nan` elements have a `float64` dtype.

The sparse objects exist for memory efficiency reasons. Suppose you had a large, mostly NA `DataFrame`:

    In [5]: df = pd.DataFrame(np.random.randn(10000, 4))

    In [6]: df.iloc[:9998] = np.nan

    In [7]: sdf = df.astype(pd.SparseDtype("float", np.nan))

    In [8]: sdf.head()
    Out[8]:
         0    1    2    3
    0  NaN  NaN  NaN  NaN
    1  NaN  NaN  NaN  NaN
    2  NaN  NaN  NaN  NaN
    3  NaN  NaN  NaN  NaN
    4  NaN  NaN  NaN  NaN

    In [9]: sdf.dtypes
    Out[9]:
    0    Sparse[float64, nan]
    1    Sparse[float64, nan]
    2    Sparse[float64, nan]
    3    Sparse[float64, nan]
    dtype: object

    In [10]: sdf.sparse.density
    Out[10]: np.float64(0.0002)

As you can see, the density (% of values that have not been “compressed”) is extremely low. This sparse object takes up much less memory on disk (pickled) and in the Python interpreter.

    In [11]: f'dense: {df.memory_usage().sum()} bytes'
    Out[11]: 'dense: 320132 bytes'

    In [12]: f'sparse: {sdf.memory_usage().sum()} bytes'
    Out[12]: 'sparse: 228 bytes'

Functionally, their behavior should be nearly identical to their dense counterparts.

## SparseArray

`arrays.SparseArray` is a `ExtensionArray` for storing an array of sparse values (see dtypes for more on extension arrays). It is a 1-dimensional ndarray-like object storing only values distinct from the `fill_value`:

    In [13]: arr = np.random.randn(10)

    In [14]: arr[2:5] = np.nan

    In [15]: arr[7:8] = np.nan

    In [16]: sparr = pd.arrays.SparseArray(arr)

    In [17]: sparr
    Out[17]:
    [-1.9556635297215477, -1.6588664275960427, nan, nan, nan, 1.1589328886422277, 0.14529711373305043, nan, 0.6060271905134522, 1.3342113401317768]
    Fill: nan
    IntIndex
    Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

A sparse array can be converted to a regular (dense) ndarray with `numpy.asarray()`

    In [18]: np.asarray(sparr)
    Out[18]:
    array([-1.9557, -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,
               nan,  0.606 ,  1.3342])

## SparseDtype

The `SparseArray.dtype` property stores two pieces of information

1. The dtype of the non-sparse values

2. The scalar fill value

    In [19]: sparr.dtype
    Out[19]: Sparse[float64, nan]

A `SparseDtype` may be constructed by passing only a dtype

    In [20]: pd.SparseDtype(np.dtype('datetime64[ns]'))
    Out[20]: Sparse[datetime64[ns], np.datetime64('NaT')]

in which case a default fill value will be used (for NumPy dtypes this is often the “missing” value for that dtype). To override this default an explicit fill value may be passed instead

    In [21]: pd.SparseDtype(np.dtype('datetime64[ns]'),
       ....:                fill_value=pd.Timestamp('2017-01-01'))
       ....:
    Out[21]: Sparse[datetime64[ns], Timestamp('2017-01-01 00:00:00')]

Finally, the string alias `'Sparse[dtype]'` may be used to specify a sparse dtype in many places

    In [22]: pd.array([1, 0, 0, 2], dtype='Sparse[int]')
    Out[22]:
    [1, 0, 0, 2]
    Fill: 0
    IntIndex
    Indices: array([0, 3], dtype=int32)

## Sparse accessor

pandas provides a `.sparse` accessor, similar to `.str` for string data, `.cat` for categorical data, and `.dt` for datetime-like data. This namespace provides attributes and methods that are specific to sparse data.

    In [23]: s = pd.Series([0, 0, 1, 2], dtype="Sparse[int]")

    In [24]: s.sparse.density
    Out[24]: 0.5

    In [25]: s.sparse.fill_value
    Out[25]: 0

This accessor is available only on data with `SparseDtype`, and on the `Series` class itself for creating a Series with sparse data from a scipy COO matrix with.

A `.sparse` accessor has been added for `DataFrame` as well. See Sparse accessor for more.

## Sparse calculation

You can apply NumPy ufuncs to `arrays.SparseArray` and get a `arrays.SparseArray` as a result.

    In [26]: arr = pd.arrays.SparseArray([1., np.nan, np.nan, -2., np.nan])

    In [27]: np.abs(arr)
    Out[27]:
    [1.0, nan, nan, 2.0, nan]
    Fill: nan
    IntIndex
    Indices: array([0, 3], dtype=int32)

The ufunc is also applied to `fill_value`. This is needed to get the correct dense result.

    In [28]: arr = pd.arrays.SparseArray([1., -1, -1, -2., -1], fill_value=-1)

    In [29]: np.abs(arr)
    Out[29]:
    [1, 1, 1, 2.0, 1]
    Fill: 1
    IntIndex
    Indices: array([3], dtype=int32)

    In [30]: np.abs(arr).to_dense()
    Out[30]: array([1., 1., 1., 2., 1.])

Conversion

To convert data from sparse to dense, use the `.sparse` accessors

    In [31]: sdf.sparse.to_dense()
    Out[31]:
                 0         1         2         3
    0          NaN       NaN       NaN       NaN
    1          NaN       NaN       NaN       NaN
    2          NaN       NaN       NaN       NaN
    3          NaN       NaN       NaN       NaN
    4          NaN       NaN       NaN       NaN
    ...        ...       ...       ...       ...
    9995       NaN       NaN       NaN       NaN
    9996       NaN       NaN       NaN       NaN
    9997       NaN       NaN       NaN       NaN
    9998  0.509184 -0.774928 -1.369894 -0.382141
    9999  0.280249 -1.648493  1.490865 -0.890819

    [10000 rows x 4 columns]

From dense to sparse, use `DataFrame.astype()` with a `SparseDtype`.

    In [32]: dense = pd.DataFrame({"A": [1, 0, 0, 1]})

    In [33]: dtype = pd.SparseDtype(int, fill_value=0)

    In [34]: dense.astype(dtype)
    Out[34]:
       A
    0  1
    1  0
    2  0
    3  1

## Interaction with scipy.sparse

Use `DataFrame.sparse.from_spmatrix()` to create a `DataFrame` with sparse values from a sparse matrix.

    In [35]: from scipy.sparse import csr_matrix

    In [36]: arr = np.random.random(size=(1000, 5))

    In [37]: arr[arr < .9] = 0

    In [38]: sp_arr = csr_matrix(arr)

    In [39]: sp_arr
    Out[39]:
    <Compressed Sparse Row sparse matrix of dtype 'float64'
    	with 517 stored elements and shape (1000, 5)>

    In [40]: sdf = pd.DataFrame.sparse.from_spmatrix(sp_arr)

    In [41]: sdf.head()
    Out[41]:
              0    1    2         3    4
    0   0.95638  NaN  NaN       NaN  NaN
    1       NaN  NaN  NaN       NaN  NaN
    2       NaN  NaN  NaN       NaN  NaN
    3       NaN  NaN  NaN       NaN  NaN
    4  0.999552  NaN  NaN  0.956153  NaN

    In [42]: sdf.dtypes
    Out[42]:
    0    Sparse[float64, nan]
    1    Sparse[float64, nan]
    2    Sparse[float64, nan]
    3    Sparse[float64, nan]
    4    Sparse[float64, nan]
    dtype: object

All sparse formats are supported, but matrices that are not in `COOrdinate` format will be converted, copying data as needed. To convert back to sparse SciPy matrix in COO format, you can use the `DataFrame.sparse.to_coo()` method:

    In [43]: sdf.sparse.to_coo()
    Out[43]:
    <COOrdinate sparse matrix of dtype 'float64'
    	with 517 stored elements and shape (1000, 5)>

`Series.sparse.to_coo()` is implemented for transforming a `Series` with sparse values indexed by a `MultiIndex` to a `scipy.sparse.coo_matrix`.

The method requires a `MultiIndex` with two or more levels.

    In [44]: s = pd.Series([3.0, np.nan, 1.0, 3.0, np.nan, np.nan])

    In [45]: s.index = pd.MultiIndex.from_tuples(
       ....:     [
       ....:         (1, 2, "a", 0),
       ....:         (1, 2, "a", 1),
       ....:         (1, 1, "b", 0),
       ....:         (1, 1, "b", 1),
       ....:         (2, 1, "b", 0),
       ....:         (2, 1, "b", 1),
       ....:     ],
       ....:     names=["A", "B", "C", "D"],
       ....: )
       ....:

    In [46]: ss = s.astype('Sparse')

    In [47]: ss
    Out[47]:
    A  B  C  D
    1  2  a  0    3.0
             1    NaN
       1  b  0    1.0
             1    3.0
    2  1  b  0    NaN
             1    NaN
    dtype: Sparse[float64, nan]

In the example below, we transform the `Series` to a sparse representation of a 2-d array by specifying that the first and second `MultiIndex` levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation.

    In [48]: A, rows, columns = ss.sparse.to_coo(
       ....:     row_levels=["A", "B"], column_levels=["C", "D"], sort_labels=True
       ....: )
       ....:

    In [49]: A
    Out[49]:
    <COOrdinate sparse matrix of dtype 'float64'
    	with 3 stored elements and shape (3, 4)>

    In [50]: A.todense()
    Out[50]:
    matrix([[0., 0., 1., 3.],
            [3., 0., 0., 0.],
            [0., 0., 0., 0.]])

    In [51]: rows
    Out[51]: [(1, 1), (1, 2), (2, 1)]

    In [52]: columns
    Out[52]: [('a', 0), ('a', 1), ('b', 0), ('b', 1)]

Specifying different row and column labels (and not sorting them) yields a different sparse matrix:

    In [53]: A, rows, columns = ss.sparse.to_coo(
       ....:     row_levels=["A", "B", "C"], column_levels=["D"], sort_labels=False
       ....: )
       ....:

    In [54]: A
    Out[54]:
    <COOrdinate sparse matrix of dtype 'float64'
    	with 3 stored elements and shape (3, 2)>

    In [55]: A.todense()
    Out[55]:
    matrix([[3., 0.],
            [1., 3.],
            [0., 0.]])

    In [56]: rows
    Out[56]: [(1, 2, 'a'), (1, 1, 'b'), (2, 1, 'b')]

    In [57]: columns
    Out[57]: [(0,), (1,)]

A convenience method `Series.sparse.from_coo()` is implemented for creating a `Series` with sparse values from a `scipy.sparse.coo_matrix`.

    In [58]: from scipy import sparse

    In [59]: A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])), shape=(3, 4))

    In [60]: A
    Out[60]:
    <COOrdinate sparse matrix of dtype 'float64'
    	with 3 stored elements and shape (3, 4)>

    In [61]: A.todense()
    Out[61]:
    matrix([[0., 0., 1., 2.],
            [3., 0., 0., 0.],
            [0., 0., 0., 0.]])

The default behaviour (with `dense_index=False`) simply returns a `Series` containing only the non-null entries.

    In [62]: ss = pd.Series.sparse.from_coo(A)

    In [63]: ss
    Out[63]:
    0  2    1.0
       3    2.0
    1  0    3.0
    dtype: Sparse[float64, nan]

Specifying `dense_index=True` will result in an index that is the Cartesian product of the row and columns coordinates of the matrix. Note that this will consume a significant amount of memory (relative to `dense_index=False`) if the sparse matrix is large (and sparse) enough.

    In [64]: ss_dense = pd.Series.sparse.from_coo(A, dense_index=True)

    In [65]: ss_dense
    Out[65]:
    1  0    3.0
       2    NaN
       3    NaN
    0  0    NaN
       2    1.0
       3    2.0
       0    NaN
       2    1.0
       3    2.0
    dtype: Sparse[float64, nan]