Internals¶
This section will provide a look into some of pandas internals. It’s primarily intended for developers of pandas itself.
Indexing¶
In pandas there are a few objects implemented which can serve as valid containers for the axis labels:
Index
: the generic “ordered set” object, an ndarray of object dtype assuming nothing about its contents. The labels must be hashable (and likely immutable) and unique. Populates a dict of label to location in Cython to doO(1)
lookups.Int64Index
: a version ofIndex
highly optimized for 64-bit integer data, such as time stampsFloat64Index
: a version ofIndex
highly optimized for 64-bit float dataMultiIndex
: the standard hierarchical index objectDatetimeIndex
: An Index object withTimestamp
boxed elements (impl are the int64 values)TimedeltaIndex
: An Index object withTimedelta
boxed elements (impl are the in64 values)PeriodIndex
: An Index object with Period elements
There are functions that make the creation of a regular index easy:
date_range
: fixed frequency date range generated from a time rule or DateOffset. An ndarray of Python datetime objectsperiod_range
: fixed frequency date range generated from a time rule or DateOffset. An ndarray ofPeriod
objects, representing Timespans
The motivation for having an Index
class in the first place was to enable
different implementations of indexing. This means that it’s possible for you,
the user, to implement a custom Index
subclass that may be better suited to
a particular application than the ones provided in pandas.
From an internal implementation point of view, the relevant methods that an
Index
must define are one or more of the following (depending on how
incompatible the new object internals are with the Index
functions):
get_loc
: returns an “indexer” (an integer, or in some cases a slice object) for a labelslice_locs
: returns the “range” to slice between two labelsget_indexer
: Computes the indexing vector for reindexing / data alignment purposes. See the source / docstrings for more on thisget_indexer_non_unique
: Computes the indexing vector for reindexing / data alignment purposes when the index is non-unique. See the source / docstrings for more on thisreindex
: Does any pre-conversion of the input index then callsget_indexer
union
,intersection
: computes the union or intersection of two Index objectsinsert
: Inserts a new label into an Index, yielding a new objectdelete
: Delete a label, yielding a new objectdrop
: Deletes a set of labelstake
: Analogous to ndarray.take
MultiIndex¶
Internally, the MultiIndex
consists of a few things: the levels, the
integer labels, and the level names:
In [1]: index = pd.MultiIndex.from_product([range(3), ['one', 'two']], names=['first', 'second'])
In [2]: index
Out[2]:
MultiIndex(levels=[[0, 1, 2], ['one', 'two']],
labels=[[0, 0, 1, 1, 2, 2], [0, 1, 0, 1, 0, 1]],
names=['first', 'second'])
In [3]: index.levels