Sparse data structures

We have implemented “sparse” versions of Series, DataFrame, and Panel. These are not sparse in the typical “mostly 0”. You can view these objects as being “compressed” where any data matching a specific value (NaN/missing by default, though any value can be chosen) is omitted. A special SparseIndex object tracks where data has been “sparsified”. This will make much more sense in an example. All of the standard pandas data structures have a to_sparse method:

In [824]: ts = Series(randn(10))

In [825]: ts[2:-2] = np.nan

In [826]: sts = ts.to_sparse()

In [827]: sts
Out[827]: 
0    0.469112
1   -0.282863
2         NaN
3         NaN
4         NaN
5         NaN
6         NaN
7         NaN
8   -0.861849
9   -2.104569
BlockIndex
Block locations: array([0, 8], dtype=int32)
Block lengths: array([2, 2], dtype=int32)

The to_sparse method takes a kind argument (for the sparse index, see below) and a fill_value. So if we had a mostly zero Series, we could convert it to sparse with fill_value=0:

In [828]: ts.fillna(0).to_sparse(fill_value=0)
Out[828]: 
0    0.469112
1   -0.282863
2    0.000000
3    0.000000
4    0.000000
5    0.000000
6    0.000000
7    0.000000
8   -0.861849
9   -2.104569
BlockIndex
Block locations: array([0, 8], dtype=int32)
Block lengths: array([2, 2], dtype=int32)

The sparse objects exist for memory efficiency reasons. Suppose you had a large, mostly NA DataFrame:

In [829]: df = DataFrame(randn(10000, 4))

In [830]: df.ix[:9998] = np.nan

In [831]: sdf = df.to_sparse()

In [832]: sdf
Out[832]: 
<class 'pandas.sparse.frame.SparseDataFrame'>
Int64Index: 10000 entries, 0 to 9999
Columns: 4 entries, 0 to 3
dtypes: float64(4)

In [833]: sdf.density
Out[833]: 0.0001

As you can see, the density (% of values that have not been “compressed”) is extremely low. This sparse object takes up much less memory on disk (pickled) and in the Python interpreter. Functionally, their behavior should be nearly identical to their dense counterparts.

Any sparse object can be converted back to the standard dense form by calling to_dense:

In [834]: sts.to_dense()
Out[834]: 
0    0.469112
1   -0.282863
2         NaN
3         NaN
4         NaN
5         NaN
6         NaN
7         NaN
8   -0.861849
9   -2.104569

SparseArray

SparseArray is the base layer for all of the sparse indexed data structures. It is a 1-dimensional ndarray-like object storing only values distinct from the fill_value:

In [835]: arr = np.random.randn(10)

In [836]: arr[2:5] = np.nan; arr[7:8] = np.nan

In [837]: sparr = SparseArray(arr)

In [838]: sparr
Out[838]: 
SparseArray([-1.9557, -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,
           nan,  0.606 ,  1.3342])
IntIndex
Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

Like the indexed objects (SparseSeries, SparseDataFrame, SparsePanel), a SparseArray can be converted back to a regular ndarray by calling to_dense:

In [839]: sparr.to_dense()
Out[839]: 
array([-1.9557, -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,
           nan,  0.606 ,  1.3342])

SparseList

SparseList is a list-like data structure for managing a dynamic collection of SparseArrays. To create one, simply call the SparseList constructor with a fill_value (defaulting to NaN):

In [840]: spl = SparseList()

In [841]: spl
Out[841]: 
<pandas.sparse.list.SparseList object at 0x10a7e64d0>

The two important methods are append and to_array. append can accept scalar values or any 1-dimensional sequence:

In [842]: spl.append(np.array([1., nan, nan, 2., 3.]))

In [843]: spl.append(5)

In [844]: spl.append(sparr)

In [845]: spl
Out[845]: 
<pandas.sparse.list.SparseList object at 0x10a7e64d0>
SparseArray([ 1., nan, nan,  2.,  3.])
IntIndex
Indices: array([0, 3, 4], dtype=int32)
SparseArray([ 5.])
IntIndex
Indices: array([0], dtype=int32)
SparseArray([-1.9557, -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,
           nan,  0.606 ,  1.3342])
IntIndex
Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

As you can see, all of the contents are stored internally as a list of memory-efficient SparseArray objects. Once you’ve accumulated all of the data, you can call to_array to get a single SparseArray with all the data:

In [846]: spl.to_array()
Out[846]: 
SparseArray([ 1.    ,     nan,     nan,  2.    ,  3.    ,  5.    , -1.9557,
       -1.6589,     nan,     nan,     nan,  1.1589,  0.1453,     nan,
        0.606 ,  1.3342])
IntIndex
Indices: array([ 0,  3,  4,  5,  6,  7, 11, 12, 14, 15], dtype=int32)

SparseIndex objects

Two kinds of SparseIndex are implemented, block and integer. We recommend using block as it’s more memory efficient. The integer format keeps an arrays of all of the locations where the data are not equal to the fill value. The block format tracks only the locations and sizes of blocks of data.