pandas 0.9.1 documentation

Intro to Data Structures

We’ll start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started. The fundamental behavior about data types, indexing, and axis labeling / alignment apply across all of the objects. To get started, import numpy and load pandas into your namespace:

In [269]: import numpy as np

# will use a lot in examples
In [270]: randn = np.random.randn

In [271]: from pandas import *

Here is a basic tenet to keep in mind: data alignment is intrinsic. The link between labels and data will not be broken unless done so explicitly by you.

We’ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.

When using pandas, we recommend the following import convention:

import pandas as pd

Series

Series is a one-dimensional labeled array (technically a subclass of ndarray) capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:

>>> s = Series(data, index=index)

Here, data can be many different things:

  • a Python dict
  • an ndarray
  • a scalar value (like 5)

The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is:

From ndarray

If data is an ndarray, index must be the same length as data. If no index is passed, one will be created having values [0, ..., len(data) - 1].

In [272]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])

In [273]: s
Out[273]: 
a    0.664
b   -0.487
c   -0.504
d    0.307
e    1.570

In [274]: s.index
Out[274]: Index([a, b, c, d, e], dtype=object)

In [275]: Series(randn(5))
Out[275]: 
0   -0.431
1   -0.705
2    0.555
3    0.939
4    0.722

Note

Starting in v0.8.0, pandas supports non-unique index values. In previous version, if the index values are not unique an exception will not be raised immediately, but attempting any operation involving the index will later result in an exception. In other words, the Index object containing the labels “lazily” checks whether the values are unique. The reason for being lazy is nearly all performance-based (there are many instances in computations, like parts of GroupBy, where the index is not used).

From dict

If data is a dict, if index is passed the values in data corresponding to the labels in the index will be pulled out. Otherwise, an index will be constructed from the sorted keys of the dict, if possible.

In [276]: d = {'a' : 0., 'b' : 1., 'c' : 2.}

In [277]: Series(d)
Out[277]: 
a    0
b    1
c    2

In [278]: Series(d, index=['b', 'c', 'd', 'a'])
Out[278]: 
b     1
c     2
d   NaN
a     0

Note

NaN (not a number) is the standard missing data marker used in pandas

From scalar value If data is a scalar value, an index must be provided. The value will be repeated to match the length of index

In [279]: Series(5., index=['a', 'b', 'c', 'd', 'e'])
Out[279]: 
a    5
b    5
c    5
d    5
e    5

Series is ndarray-like

As a subclass of ndarray, Series is a valid argument to most NumPy functions and behaves similarly to a NumPy array. However, things like slicing also slice the index.

In [280]: s[0]
Out[280]: 0.66444516201494186

In [281]: s[:3]
Out[281]: 
a    0.664
b   -0.487
c   -0.504

In [282]: s[s > s.median()]
Out[282]: 
a    0.664
e    1.570

In [283]: s[[4, 3, 1]]
Out[283]: 
e    1.570
d    0.307
b   -0.487

In [284]: np.exp(s)
Out[284]: 
a    1.943
b    0.614
c    0.604
d    1.359
e    4.807

We will address array-based indexing in a separate section.

Series is dict-like

A Series is alike a fixed-size dict in that you can get and set values by index label:

In [285]: s['a']
Out[285]: 0.66444516201494186

In [286]: s['e'] = 12.

In [287]: s
Out[287]: 
a     0.664
b    -0.487
c    -0.504
d     0.307
e    12.000

In [288]: 'e' in s
Out[288]: True

In [289]: 'f' in s
Out[289]: False

If a label is not contained, an exception is raised:

>>> s['f']
KeyError: 'f'

Using the get method, a missing label will return None or specified default:

In [290]: s.get('f')

In [291]: s.get('f', np.nan)
Out[291]: nan

Vectorized operations and label alignment with Series

When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary. Series can be also be passed into most NumPy methods expecting an ndarray.

In [292]: s + s
Out[292]: 
a     1.329
b    -0.974
c    -1.009
d     0.613
e    24.000

In [293]: s * 2
Out[293]: 
a     1.329
b    -0.974
c    -1.009
d     0.613
e    24.000

In [294]: np.exp(s)
Out[294]: 
a         1.943
b         0.614
c         0.604
d         1.359
e    162754.791

A key difference between Series and ndarray is that operations between Series automatically align the data based on label. Thus, you can write computations without giving consideration to whether the Series involved have the same labels.

In [295]: s[1:] + s[:-1]
Out[295]: 
a      NaN
b   -0.974
c   -1.009
d    0.613
e      NaN

The result of an operation between unaligned Series will have the union of the indexes involved. If a label is not found in one Series or the other, the result will be marked as missing (NaN). Being able to write code without doing any explicit data alignment grants immense freedom and flexibility in interactive data analysis and research. The integrated data alignment features of the pandas data structures set pandas apart from the majority of related tools for working with labeled data.

Note

In general, we chose to make the default result of operations between differently indexed objects yield the union of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is typically important information as part of a computation. You of course have the option of dropping labels with missing data via the dropna function.

Name attribute

Series can also have a name attribute:

In [296]: s = Series(np.random.randn(5), name='something')

In [297]: s
Out[297]: 
0    0.015
1    1.987
2   -0.259
3    0.111
4    1.012
Name: something

In [298]: s.name
Out[298]: 'something'

The Series name will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as you will see below.

DataFrame

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:

  • Dict of 1D ndarrays, lists, dicts, or Series
  • 2-D numpy.ndarray
  • Structured or record ndarray
  • A Series
  • Another DataFrame

Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.

If axis labels are not passed, they will be constructed from the input data based on common sense rules.

From dict of Series or dicts

The result index will be the union of the indexes of the various Series. If there are any nested dicts, these will be first converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.

In [299]: d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']),
   .....:      'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
   .....:

In [300]: df = DataFrame(d)

In [301]: df
Out[301]: 
   one  two
a    1    1
b    2    2
c    3    3
d  NaN    4

In [302]: DataFrame(d, index=['d', 'b', 'a'])
Out[302]: 
   one  two
d  NaN    4
b    2    2
a    1    1

In [303]: DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
Out[303]: 
   two three
d    4   NaN
b    2   NaN
a    1   NaN

The row and column labels can be accessed respectively by accessing the index and columns attributes:

Note

When a particular set of columns is passed along with a dict of data, the passed columns override the keys in the dict.

In [304]: df.index
Out[304]: Index([a, b, c, d], dtype=object)

In [305]: df.columns
Out[305]: Index([one, two], dtype=object)

From dict of ndarrays / lists

The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays. If no index is passed, the result will be range(n), where n is the array length.

In [306]: d = {'one' : [1., 2., 3., 4.],
   .....:      'two' : [4., 3., 2., 1.]}
   .....:

In [307]: DataFrame(d)
Out[307]: 
   one  two
0    1    4
1    2    3
2    3    2
3    4    1

In [308]: DataFrame(d, index=['a', 'b', 'c', 'd'])
Out[308]: 
   one  two
a    1    4
b    2    3
c    3    2
d    4    1

From structured or record array

This case is handled identically to a dict of arrays.

In [309]: data = np.zeros((2,),dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])

In [310]: data[:] = [(1,2.,'Hello'),(2,3.,"World")]

In [311]: DataFrame(data)
Out[311]: 
   A  B      C
0  1  2  Hello
1  2  3  World

In [312]: DataFrame(data, index=['first', 'second'])
Out[312]: 
        A  B      C
first   1  2  Hello
second  2  3  World

In [313]: DataFrame(data, columns=['C', 'A', 'B'])
Out[313]: 
       C  A  B
0  Hello  1  2
1  World  2  3

Note

DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.

From a list of dicts

In [314]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

In [315]: DataFrame(data2)
Out[315]: 
   a   b   c
0  1   2 NaN
1  5  10  20

In [316]: DataFrame(data2, index=['first', 'second'])
Out[316]: 
        a   b   c
first   1   2 NaN
second  5  10  20

In [317]: DataFrame(data2, columns=['a', 'b'])
Out[317]: 
   a   b
0  1   2
1  5  10

From a Series

The result will be a DataFrame with the same index as the input Series, and with one column whose name is the original name of the Series (only if no other column name provided).

Missing Data

Much more will be said on this topic in the Missing data section. To construct a DataFrame with missing data, use np.nan for those values which are missing. Alternatively, you may pass a numpy.MaskedArray as the data argument to the DataFrame constructor, and its masked entries will be considered missing.

Alternate Constructors

DataFrame.from_dict

DataFrame.from_dict takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates like the DataFrame constructor except for the orient parameter which is 'columns' by default, but which can be set to 'index' in order to use the dict keys as row labels.

DataFrame.from_records

DataFrame.from_records takes a list of tuples or an ndarray with structured dtype. Works analogously to the normal DataFrame constructor, except that index maybe be a specific field of the structured dtype to use as the index. For example:

In [318]: data
Out[318]: 
array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], 
      dtype=[('A', '<i4'), ('B', '<f4'), ('C', '|S10')])

In [319]: DataFrame.from_records(data, index='C')
Out[319]: 
       A  B
C          
Hello  1  2
World  2  3

DataFrame.from_items

DataFrame.from_items works analogously to the form of the dict constructor that takes a sequence of (key, value) pairs, where the keys are column (or row, in the case of orient='index') names, and the value are the column values (or row values). This can be useful for constructing a DataFrame with the columns in a particular order without having to pass an explicit list of columns:

In [320]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])])
Out[320]: 
   A  B
0  1  4
1  2  5
2  3  6

If you pass orient='index', the keys will be the row labels. But in this case you must also pass the desired column names:

In [321]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])],
   .....:                      orient='index', columns=['one', 'two', 'three'])
   .....:
Out[321]: 
   one  two  three
A    1    2      3
B    4    5      6

Column selection, addition, deletion

You can treat a DataFrame semantically like a dict of like-indexed Series objects. Getting, setting, and deleting columns works with the same syntax as the analogous dict operations:

In [322]: df['one']
Out[322]: 
a     1
b     2
c     3
d   NaN
Name: one

In [323]: df['three'] = df['one'] * df['two']

In [324]: df['flag'] = df['one'] > 2

In [325]: df
Out[325]: 
   one  two  three   flag
a    1    1      1  False
b    2    2      4  False
c    3    3      9   True
d  NaN    4    NaN  False

Columns can be deleted or popped like with a dict:

In [326]: del df['two']

In [327]: three = df.pop('three')

In [328]: df
Out[328]: 
   one   flag
a    1  False
b    2  False
c    3   True
d  NaN  False

When inserting a scalar value, it will naturally be propagated to fill the column:

In [329]: df['foo'] = 'bar'

In [330]: df
Out[330]: 
   one   flag  foo
a    1  False  bar
b    2  False  bar
c    3   True  bar
d  NaN  False  bar

When inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrame’s index:

In [331]: df['one_trunc'] = df['one'][:2]

In [332]: df
Out[332]: 
   one   flag  foo  one_trunc
a    1  False  bar          1
b    2  False  bar          2
c    3   True  bar        NaN
d  NaN  False  bar        NaN

You can insert raw ndarrays but their length must match the length of the DataFrame’s index.

By default, columns get inserted at the end. The insert function is available to insert at a particular location in the columns:

In [333]: df.insert(1, 'bar', df['one'])

In [334]: df
Out[334]: 
   one  bar   flag  foo  one_trunc
a    1    1  False  bar          1
b    2    2  False  bar          2
c    3    3   True  bar        NaN
d  NaN  NaN  False  bar        NaN

Indexing / Selection

The basics of indexing are as follows:

Operation Syntax Result
Select column df[col] Series
Select row by label df.xs(label) or df.ix[label] Series
Select row by location (int) df.ix[loc] Series
Slice rows df[5:10] DataFrame
Select rows by boolean vector df[bool_vec] DataFrame

Row selection, for example, returns a Series whose index is the columns of the DataFrame:

In [335]: df.xs('b')
Out[335]: 
one              2
bar              2
flag         False
foo            bar
one_trunc        2
Name: b

In [336]: df.ix[2]
Out[336]: 
one             3
bar             3
flag         True
foo           bar
one_trunc     NaN
Name: c

Note if a DataFrame contains columns of multiple dtypes, the dtype of the row will be chosen to accommodate all of the data types (dtype=object is the most general).

For a more exhaustive treatment of more sophisticated label-based indexing and slicing, see the section on indexing. We will address the fundamentals of reindexing / conforming to new sets of lables in the section on reindexing.

Data alignment and arithmetic

Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). Again, the resulting object will have the union of the column and row labels.

In [337]: df = DataFrame(randn(10, 4), columns=['A', 'B', 'C', 'D'])

In [338]: df2 = DataFrame(randn(7, 3), columns=['A', 'B', 'C'])

In [339]: df + df2
Out[339]: 
       A      B      C   D
0  2.752 -0.429  0.702 NaN
1  0.067 -3.397  1.775 NaN
2 -0.499 -1.138 -1.277 NaN
3  0.731  0.988  0.505 NaN
4 -0.538 -1.828 -1.974 NaN
5 -0.100 -2.885  1.676 NaN
6  1.405 -1.078  0.320 NaN
7    NaN    NaN    NaN NaN
8    NaN    NaN    NaN NaN
9    NaN    NaN    NaN NaN

When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting row-wise. For example:

In [340]: df - df.ix[0]
Out[340]: 
       A      B      C      D
0  0.000  0.000  0.000  0.000
1 -0.586 -3.234 -1.119 -2.876
2 -1.136 -3.727 -2.066 -1.500
3 -0.799 -0.803 -1.860 -2.038
4 -1.632 -2.216 -2.384 -1.367
5 -0.485 -2.545 -1.044 -1.190
6 -0.168  0.395 -1.333 -1.060
7 -2.367 -0.799 -3.479 -2.653
8 -2.953 -1.565 -2.251 -2.978
9 -1.615 -1.712 -2.521 -2.106

In the special case of working with time series data, if the Series is a TimeSeries (which it will be automatically if the index contains datetime objects), and the DataFrame index also contains dates, the broadcasting will be column-wise:

In [341]: index = date_range('1/1/2000', periods=8)

In [342]: df = DataFrame(randn(8, 3), index=index,
   .....:                columns=['A', 'B', 'C'])
   .....:

In [343]: df
Out[343]: 
                A      B      C
2000-01-01 -1.209 -1.257 -0.500
2000-01-02  0.430 -0.242 -0.724
2000-01-03  1.257 -0.871 -0.544
2000-01-04 -0.766 -0.219  0.663
2000-01-05 -1.566  1.780 -2.139
2000-01-06 -0.593 -1.059  0.119
2000-01-07 -0.123  1.306 -0.129
2000-01-08 -0.389  0.143 -1.715

In [344]: type(df['A'])
Out[344]: pandas.core.series.TimeSeries

In [345]: df - df['A']
Out[345]: 
            A      B      C
2000-01-01  0 -0.048  0.709
2000-01-02  0 -0.672 -1.154
2000-01-03  0 -2.128 -1.801
2000-01-04  0  0.547  1.429
2000-01-05  0  3.346 -0.572
2000-01-06  0 -0.466  0.711
2000-01-07  0  1.429 -0.006
2000-01-08  0  0.532 -1.326

Technical purity aside, this case is so common in practice that supporting the special case is preferable to the alternative of forcing the user to transpose and do column-based alignment like so:

In [346]: (df.T - df['A']).T
Out[346]: 
            A      B      C
2000-01-01  0 -0.048  0.709
2000-01-02  0 -0.672 -1.154
2000-01-03  0 -2.128 -1.801
2000-01-04  0  0.547  1.429
2000-01-05  0  3.346 -0.572
2000-01-06  0 -0.466  0.711
2000-01-07  0  1.429 -0.006
2000-01-08  0  0.532 -1.326

For explicit control over the matching and broadcasting behavior, see the section on flexible binary operations.

Operations with scalars are just as you would expect:

In [347]: df * 5 + 2
Out[347]: 
                A       B      C
2000-01-01 -4.043  -4.285 -0.499
2000-01-02  4.149   0.789 -1.619
2000-01-03  8.286  -2.355 -0.719
2000-01-04 -1.830   0.907  5.314
2000-01-05 -5.831  10.900 -8.693
2000-01-06 -0.963  -3.296  2.594
2000-01-07  1.385   8.528  1.354
2000-01-08  0.054   2.716 -6.575

In [348]: 1 / df
Out[348]: 
                A      B      C
2000-01-01 -0.827 -0.796 -2.001
2000-01-02  2.327 -4.129 -1.382
2000-01-03  0.795 -1.148 -1.839
2000-01-04 -1.305 -4.574  1.509
2000-01-05 -0.638  0.562 -0.468
2000-01-06 -1.687 -0.944  8.416
2000-01-07 -8.128  0.766 -7.746
2000-01-08 -2.570  6.983 -0.583

In [349]: df ** 4
Out[349]: 
                A       B       C
2000-01-01  2.133   2.497   0.062
2000-01-02  0.034   0.003   0.275
2000-01-03  2.499   0.576   0.087
2000-01-04  0.344   0.002   0.193
2000-01-05  6.018  10.038  20.918
2000-01-06  0.123   1.258   0.000
2000-01-07  0.000   2.906   0.000
2000-01-08  0.023   0.000   8.652

Boolean operators work as well:

In [350]: df1 = DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)

In [351]: df2 = DataFrame({'a' : [0, 1, 1], 'b' : [1, 1, 0] }, dtype=bool)

In [352]: df1 & df2
Out[352]: 
       a      b
0  False  False
1  False   True
2   True  False

In [353]: df1 | df2
Out[353]: 
      a     b
0  True  True
1  True  True
2  True  True

In [354]: df1 ^ df2
Out[354]: 
       a      b
0   True   True
1   True  False
2  False   True

In [355]: -df1
Out[355]: 
       a      b
0  False   True
1   True  False
2  False  False

Transposing

To transpose, access the T attribute (also the transpose function), similar to an ndarray:

# only show the first 5 rows
In [356]: df[:5].T
Out[356]: 
   2000-01-01  2000-01-02  2000-01-03  2000-01-04  2000-01-05
A      -1.209       0.430       1.257      -0.766      -1.566
B      -1.257      -0.242      -0.871      -0.219       1.780
C      -0.500      -0.724      -0.544       0.663      -2.139

DataFrame interoperability with NumPy functions

Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions can be used with no issues on DataFrame, assuming the data within are numeric:

In [357]: np.exp(df)
Out[357]: 
                A      B      C
2000-01-01  0.299  0.285  0.607
2000-01-02  1.537  0.785  0.485
2000-01-03  3.516  0.419  0.581
2000-01-04  0.465  0.804  1.940
2000-01-05  0.209  5.930  0.118
2000-01-06  0.553  0.347  1.126
2000-01-07  0.884  3.690  0.879
2000-01-08  0.678  1.154  0.180

In [358]: np.asarray(df)
Out[358]: 
array([[-1.2085, -1.257 , -0.4997],
       [ 0.4298, -0.2422, -0.7238],
       [ 1.2573, -0.871 , -0.5437],
       [-0.7661, -0.2186,  0.6628],
       [-1.5663,  1.78  , -2.1386],
       [-0.5927, -1.0591,  0.1188],
       [-0.123 ,  1.3056, -0.1291],
       [-0.3892,  0.1432, -1.715 ]])

The dot method on DataFrame implements matrix multiplication:

In [359]: df.T.dot(df)
Out[359]: 
       A      B      C
A  6.784 -1.889  3.064
B -1.889  8.460 -3.214
C  3.064 -3.214  9.055

Similarly, the dot method on Series implements dot product:

In [360]: s1 = Series(np.arange(5,10))

In [361]: s1.dot(s1)
Out[361]: 255

DataFrame is not intended to be a drop-in replacement for ndarray as its indexing semantics are quite different in places from a matrix.

Console display

For very large DataFrame objects, only a summary will be printed to the console (here I am reading a CSV version of the baseball dataset from the plyr R package):

In [362]: baseball = read_csv('data/baseball.csv')

In [363]: print baseball
<class 'pandas.core.frame.DataFrame'>
Int64Index: 100 entries, 88641 to 89534
Data columns:
id       100  non-null values
year     100  non-null values
stint    100  non-null values
team     100  non-null values
lg       100  non-null values
g        100  non-null values
ab       100  non-null values
r        100  non-null values
h        100  non-null values
X2b      100  non-null values
X3b      100  non-null values
hr       100  non-null values
rbi      100  non-null values
sb       100  non-null values
cs       100  non-null values
bb       100  non-null values
so       100  non-null values
ibb      100  non-null values
hbp      100  non-null values
sh       100  non-null values
sf       100  non-null values
gidp     100  non-null values
dtypes: float64(9), int64(10), object(3)

However, using to_string will return a string representation of the DataFrame in tabular form, though it won’t always fit the console width:

In [364]: print baseball.ix[-20:, :12].to_string()
              id  year  stint team  lg    g   ab    r    h  X2b  X3b  hr
88641  womacto01  2006      2  CHN  NL   19   50    6   14    1    0   1
88643  schilcu01  2006      1  BOS  AL   31    2    0    1    0    0   0
88645  myersmi01  2006      1  NYA  AL   62    0    0    0    0    0   0
88649  helliri01  2006      1  MIL  NL   20    3    0    0    0    0   0
88650  johnsra05  2006      1  NYA  AL   33    6    0    1    0    0   0
88652  finlest01  2006      1  SFN  NL  139  426   66  105   21   12   6
88653  gonzalu01  2006      1  ARI  NL  153  586   93  159   52    2  15
88662   seleaa01  2006      1  LAN  NL   28   26    2    5    1    0   0
89177  francju01  2007      2  ATL  NL   15   40    1   10    3    0   0
89178  francju01  2007      1  NYN  NL   40   50    7   10    0    0   1
89330   zaungr01  2007      1  TOR  AL  110  331   43   80   24    1  10
89333  witasja01  2007      1  TBA  AL    3    0    0    0    0    0   0
89334  williwo02  2007      1  HOU  NL   33   59    3    6    0    0   1
89335  wickmbo01  2007      2  ARI  NL    8    0    0    0    0    0   0
89336  wickmbo01  2007      1  ATL  NL   47    0    0    0    0    0   0
89337  whitero02  2007      1  MIN  AL   38  109    8   19    4    0   4
89338  whiteri01  2007      1  HOU  NL   20    1    0    0    0    0   0
89339  wellsda01  2007      2  LAN  NL    7   15    2    4    1    0   0
89340  wellsda01  2007      1  SDN  NL   22   38    1    4    0    0   0
89341  weathda01  2007      1  CIN  NL   67    0    0    0    0    0   0
89343  walketo04  2007      1  OAK  AL   18   48    5   13    1    0   0
89345  wakefti01  2007      1  BOS  AL    1    2    0    0    0    0   0
89347  vizquom01  2007      1  SFN  NL  145  513   54  126   18    3   4
89348  villoro01  2007      1  NYA  AL    6    0    0    0    0    0   0
89352  valenjo03  2007      1  NYN  NL   51  166   18   40   11    1   3
89354  trachst01  2007      2  CHN  NL    4    7    0    1    0    0   0
89355  trachst01  2007      1  BAL  AL    3    5    0    0    0    0   0
89359  timlimi01  2007      1  BOS  AL    4    0    0    0    0    0   0
89360  thomeji01  2007      1  CHA  AL  130  432   79  119   19    0  35
89361  thomafr04  2007      1  TOR  AL  155  531   63  147   30    0  26
89363  tavarju01  2007      1  BOS  AL    2    4    0    1    0    0   0
89365  sweenma01  2007      2  LAN  NL   30   33    2    9    1    0   0
89366  sweenma01  2007      1  SFN  NL   76   90   18   23    8    0   2
89367  suppaje01  2007      1  MIL  NL   33   61    4    8    0    0   0
89368  stinnke01  2007      1  SLN  NL   26   82    7   13    3    0   1
89370  stantmi02  2007      1  CIN  NL   67    2    0    0    0    0   0
89371  stairma01  2007      1  TOR  AL  125  357   58  103   28    1  21
89372  sprinru01  2007      1  SLN  NL   72    1    0    0    0    0   0
89374   sosasa01  2007      1  TEX  AL  114  412   53  104   24    1  21
89375  smoltjo01  2007      1  ATL  NL   30   54    1    5    1    0   0
89378  sheffga01  2007      1  DET  AL  133  494  107  131   20    1  25
89381   seleaa01  2007      1  NYN  NL   31    4    0    0    0    0   0
89382  seaneru01  2007      1  LAN  NL   68    1    0    0    0    0   0
89383  schmija01  2007      1  LAN  NL    6    7    1    1    0    0   1
89384  schilcu01  2007      1  BOS  AL    1    2    0    1    0    0   0
89385  sandere02  2007      1  KCA  AL   24   73   12   23    7    0   2
89388  rogerke01  2007      1  DET  AL    1    2    0    0    0    0   0
89389  rodriiv01  2007      1  DET  AL  129  502   50  141   31    3  11
89396  ramirma02  2007      1  BOS  AL  133  483   84  143   33    1  20
89398  piazzmi01  2007      1  OAK  AL   83  309   33   85   17    1   8
89400  perezne01  2007      1  DET  AL   33   64    5   11    3    0   1
89402   parkch01  2007      1  NYN  NL    1    1    0    0    0    0   0
89406  oliveda02  2007      1  LAA  AL    5    0    0    0    0    0   0
89410  myersmi01  2007      1  NYA  AL    6    1    0    0    0    0   0
89411  mussimi01  2007      1  NYA  AL    2    2    0    0    0    0   0
89412  moyerja01  2007      1  PHI  NL   33   73    4    9    2    0   0
89420   mesajo01  2007      1  PHI  NL   38    0    0    0    0    0   0
89421  martipe02  2007      1  NYN  NL    5    9    1    1    1    0   0
89425  maddugr01  2007      1  SDN  NL   33   62    2    9    2    0   0
89426  mabryjo01  2007      1  COL  NL   28   34    4    4    1    0   1
89429  loftoke01  2007      2  CLE  AL   52  173   24   49    9    3   0
89430  loftoke01  2007      1  TEX  AL   84  317   62   96   16    3   7
89431  loaizes01  2007      1  LAN  NL    5    7    0    1    0    0   0
89438  kleskry01  2007      1  SFN  NL  116  362   51   94   27    3   6
89439   kentje01  2007      1  LAN  NL  136  494   78  149   36    1  20
89442  jonesto02  2007      1  DET  AL    5    0    0    0    0    0   0
89445  johnsra05  2007      1  ARI  NL   10   15    0    1    0    0   0
89450  hoffmtr01  2007      1  SDN  NL   60    0    0    0    0    0   0
89451  hernaro01  2007      2  LAN  NL   22    0    0    0    0    0   0
89452  hernaro01  2007      1  CLE  AL    2    0    0    0    0    0   0
89460  guarded01  2007      1  CIN  NL   15    0    0    0    0    0   0
89462  griffke02  2007      1  CIN  NL  144  528   78  146   24    1  30
89463  greensh01  2007      1  NYN  NL  130  446   62  130   30    1  10
89464  graffto01  2007      1  MIL  NL   86  231   34   55    8    0   9
89465  gordoto01  2007      1  PHI  NL   44    0    0    0    0    0   0
89466  gonzalu01  2007      1  LAN  NL  139  464   70  129   23    2  15
89467  gomezch02  2007      2  CLE  AL   19   53    4   15    2    0   0
89468  gomezch02  2007      1  BAL  AL   73  169   17   51   10    1   1
89469  glavito02  2007      1  NYN  NL   33   56    3   12    1    0   0
89473  floydcl01  2007      1  CHN  NL  108  282   40   80   10    1   9
89474  finlest01  2007      1  COL  NL   43   94    9   17    3    0   1
89480  embreal01  2007      1  OAK  AL    4    0    0    0    0    0   0
89481  edmonji01  2007      1  SLN  NL  117  365   39   92   15    2  12
89482  easleda01  2007      1  NYN  NL   76  193   24   54    6    0  10
89489  delgaca01  2007      1  NYN  NL  139  538   71  139   30    0  24
89493  cormirh01  2007      1  CIN  NL    6    0    0    0    0    0   0
89494  coninje01  2007      2  NYN  NL   21   41    2    8    2    0   0
89495  coninje01  2007      1  CIN  NL   80  215   23   57   11    1   6
89497  clemero02  2007      1  NYA  AL    2    2    0    1    0    0   0
89498  claytro01  2007      2  BOS  AL    8    6    1    0    0    0   0
89499  claytro01  2007      1  TOR  AL   69  189   23   48   14    0   1
89501  cirilje01  2007      2  ARI  NL   28   40    6    8    4    0   0
89502  cirilje01  2007      1  MIN  AL   50  153   18   40    9    2   2
89521  bondsba01  2007      1  SFN  NL  126  340   75   94   14    0  28
89523  biggicr01  2007      1  HOU  NL  141  517   68  130   31    3  10
89525  benitar01  2007      2  FLO  NL   34    0    0    0    0    0   0
89526  benitar01  2007      1  SFN  NL   19    0    0    0    0    0   0
89530  ausmubr01  2007      1  HOU  NL  117  349   38   82   16    3   3
89533   aloumo01  2007      1  NYN  NL   87  328   51  112   19    1  13
89534  alomasa02  2007      1  NYN  NL    8   22    1    3    1    0   0

DataFrame column types

The four main types stored in pandas objects are float, int, boolean, and object. A convenient dtypes attribute return a Series with the data type of each column:

In [365]: baseball.dtypes
Out[365]: 
id        object
year       int64
stint      int64
team      object
lg        object
g          int64
ab         int64
r          int64
h          int64
X2b        int64
X3b        int64
hr         int64
rbi      float64
sb       float64
cs       float64
bb         int64
so       float64
ibb      float64
hbp      float64
sh       float64
sf       float64
gidp     float64

The related method get_dtype_counts will return the number of columns of each type:

In [366]: baseball.get_dtype_counts()
Out[366]: 
float64     9
int64      10
object      3

DataFrame column attribute access and IPython completion

If a DataFrame column label is a valid Python variable name, the column can be accessed like attributes:

In [367]: df = DataFrame({'foo1' : np.random.randn(5),
   .....:                 'foo2' : np.random.randn(5)})
   .....:

In [368]: df
Out[368]: 
       foo1      foo2
0  0.759091 -0.648742
1 -0.050457  0.209870
2  0.959219 -0.325391
3 -0.817600 -1.978199
4 -0.200407 -0.211127

In [369]: df.foo1
Out[369]: 
0    0.759091
1   -0.050457
2    0.959219
3   -0.817600
4   -0.200407
Name: foo1

The columns are also connected to the IPython completion mechanism so they can be tab-completed:

In [5]: df.fo<TAB>
df.foo1  df.foo2

Panel

Panel is a somewhat less-used, but still important container for 3-dimensional data. The term panel data is derived from econometrics and is partially responsible for the name pandas: pan(el)-da(ta)-s. The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data and, in particular, econometric analysis of panel data. However, for the strict purposes of slicing and dicing a collection of DataFrame objects, you may find the axis names slightly arbitrary:

  • items: axis 0, each item corresponds to a DataFrame contained inside
  • major_axis: axis 1, it is the index (rows) of each of the DataFrames
  • minor_axis: axis 2, it is the columns of each of the DataFrames

Construction of Panels works about like you would expect:

From 3D ndarray with optional axis labels

In [370]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
   .....:            major_axis=date_range('1/1/2000', periods=5),
   .....:            minor_axis=['A', 'B', 'C', 'D'])
   .....:

In [371]: wp
Out[371]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 5 (major) x 4 (minor)
Items: Item1 to Item2
Major axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor axis: A to D

From dict of DataFrame objects

In [372]: data = {'Item1' : DataFrame(randn(4, 3)),
   .....:         'Item2' : DataFrame(randn(4, 2))}
   .....:

In [373]: Panel(data)
Out[373]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major) x 3 (minor)
Items: Item1 to Item2
Major axis: 0 to 3
Minor axis: 0 to 2

Note that the values in the dict need only be convertible to DataFrame. Thus, they can be any of the other valid inputs to DataFrame as per above.

One helpful factory method is Panel.from_dict, which takes a dictionary of DataFrames as above, and the following named parameters:

Parameter Default Description
intersect False drops elements whose indices do not align
orient items use minor to use DataFrames’ columns as panel items

For example, compare to the construction above:

In [374]: Panel.from_dict(data, orient='minor')
Out[374]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 4 (major) x 2 (minor)
Items: 0 to 2
Major axis: 0 to 3
Minor axis: Item1 to Item2

Orient is especially useful for mixed-type DataFrames. If you pass a dict of DataFrame objects with mixed-type columns, all of the data will get upcasted to dtype=object unless you pass orient='minor':

In [375]: df = DataFrame({'a': ['foo', 'bar', 'baz'],
   .....:                 'b': np.random.randn(3)})
   .....:

In [376]: df
Out[376]: 
     a         b
0  foo  0.080597
1  bar -0.000185
2  baz -0.264704

In [377]: data = {'item1': df, 'item2': df}

In [378]: panel = Panel.from_dict(data, orient='minor')

In [379]: panel['a']
Out[379]: 
  item1 item2
0   foo   foo
1   bar   bar
2   baz   baz

In [380]: panel['b']
Out[380]: 
      item1     item2
0  0.080597  0.080597
1 -0.000185 -0.000185
2 -0.264704 -0.264704

In [381]: panel['b'].dtypes
Out[381]: 
item1    float64
item2    float64

Note

Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected feature-wise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on, of course, in future releases. And faster if you join me in working on the codebase.

From DataFrame using to_panel method

This method was introduced in v0.7 to replace LongPanel.to_long, and converts a DataFrame with a two-level index to a Panel.

In [382]: midx = MultiIndex(levels=[['one', 'two'], ['x','y']], labels=[[1,1,0,0],[1,0,1,0]])

In [383]: df = DataFrame({'A' : [1, 2, 3, 4], 'B': [5, 6, 7, 8]}, index=midx)

In [384]: df.to_panel()
Out[384]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 2 (major) x 2 (minor)
Items: A to B
Major axis: one to two
Minor axis: x to y

Item selection / addition / deletion

Similar to DataFrame functioning as a dict of Series, Panel is like a dict of DataFrames:

In [385]: wp['Item1']
Out[385]: 
                   A         B         C         D
2000-01-01 -0.519332 -1.765523 -0.966196 -0.890524
2000-01-02 -1.314597 -1.458515 -0.919663 -0.699091
2000-01-03  1.357258 -0.098278 -0.987183 -1.362030
2000-01-04 -1.309989 -1.153000  0.606382 -0.681101
2000-01-05 -0.289724 -0.996632 -1.407699  1.014104

In [386]: wp['Item3'] = wp['Item1'] / wp['Item2']

The API for insertion and deletion is the same as for DataFrame. And as with DataFrame, if the item is a valid python identifier, you can access it as an attribute and tab-complete it in IPython.

Transposing

A Panel can be rearranged using its transpose method (which does not make a copy by default unless the data are heterogeneous):

In [387]: wp.transpose(2, 0, 1)
Out[387]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 3 (major) x 5 (minor)
Items: A to D
Major axis: Item1 to Item3
Minor axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00

Indexing / Selection

Operation Syntax Result
Select item wp[item] DataFrame
Get slice at major_axis label wp.major_xs(val) DataFrame
Get slice at minor_axis label wp.minor_xs(val) DataFrame

For example, using the earlier example data, we could do:

In [388]: wp['Item1']
Out[388]: 
                   A         B         C         D
2000-01-01 -0.519332 -1.765523 -0.966196 -0.890524
2000-01-02 -1.314597 -1.458515 -0.919663 -0.699091
2000-01-03  1.357258 -0.098278 -0.987183 -1.362030
2000-01-04 -1.309989 -1.153000  0.606382 -0.681101
2000-01-05 -0.289724 -0.996632 -1.407699  1.014104

In [389]: wp.major_xs(wp.major_axis[2])
Out[389]: 
      Item1     Item2     Item3
A  1.357258 -0.177665 -7.639427
B -0.098278  0.490838 -0.200224
C -0.987183 -1.360102  0.725815
D -1.362030  1.592456 -0.855302

In [390]: wp.minor_axis
Out[390]: Index([A, B, C, D], dtype=object)

In [391]: wp.minor_xs('C')
Out[391]: 
               Item1     Item2      Item3
2000-01-01 -0.966196  0.071823 -13.452418
2000-01-02 -0.919663  0.214910  -4.279288
2000-01-03 -0.987183 -1.360102   0.725815
2000-01-04  0.606382 -1.890591  -0.320737
2000-01-05 -1.407699 -0.151652   9.282439

Conversion to DataFrame

A Panel can be represented in 2D form as a hierarchically indexed DataFrame. See the section hierarchical indexing for more on this. To convert a Panel to a DataFrame, use the to_frame method:

In [392]: panel = Panel(np.random.randn(3, 5, 4), items=['one', 'two', 'three'],
   .....:               major_axis=date_range('1/1/2000', periods=5),
   .....:               minor_axis=['a', 'b', 'c', 'd'])
   .....:

In [393]: panel.to_frame()
Out[393]: 
                       one       two     three
major      minor                              
2000-01-01 a     -0.566820  0.597468  0.716659
           b     -1.643966 -0.491240 -0.919717
           c      1.471262  1.281674 -0.024595
           d      0.677634 -0.099685  0.068997
2000-01-02 a     -0.485743 -1.823043  0.601797
           b     -0.342272 -0.779213  0.866615
           c     -1.042291 -0.949327  0.092911
           d     -0.611457  0.768043 -2.606892
2000-01-03 a     -0.141224 -0.054860  0.309303
           b      0.007220 -1.493561 -0.548401
           c     -0.516147  0.106004 -2.044772
           d      0.446161 -0.903513 -1.666264
2000-01-04 a      0.483368 -0.719875 -1.439775
           b      0.186405  0.301945  1.326361
           c     -1.439567  1.112546  0.221680
           d     -0.503782 -0.542770  1.840992
2000-01-05 a      0.890769 -2.695540  1.165150
           b     -0.777798  0.431284 -1.420521
           c     -0.552820 -0.431092  1.616679
           d     -1.428744  1.666631 -1.030912