Intro to Data Structures¶
We’ll start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started. The fundamental behavior about data types, indexing, and axis labeling / alignment apply across all of the objects. To get started, import numpy and load pandas into your namespace:
In [267]: import numpy as np
# will use a lot in examples
In [268]: randn = np.random.randn
In [269]: from pandas import *
Here is a basic tenet to keep in mind: data alignment is intrinsic. Link between labels and data will not be broken unless done so explicitly by you.
We’ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.
When using pandas, we recommend the following import convention:
import pandas as pd
Series¶
Series is a one-dimensional labeled array (technically a subclass of ndarray) capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:
>>> s = Series(data, index=index)
Here, data can be many different things:
- a Python dict
- an ndarray
- a scalar value (like 5)
The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is:
From ndarray
If data is an ndarray, index must be the same length as data. If no index is passed, one will be created having values [0, ..., len(data) - 1].
In [270]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [271]: s
Out[271]:
a 0.664
b -0.487
c -0.504
d 0.307
e 1.570
In [272]: s.index
Out[272]: Index([a, b, c, d, e], dtype=object)
In [273]: Series(randn(5))
Out[273]:
0 -0.431
1 -0.705
2 0.555
3 0.939
4 0.722
Note
Starting in v0.8.0, pandas supports non-unique index values. In previous version, if the index values are not unique an exception will not be raised immediately, but attempting any operation involving the index will later result in an exception. In other words, the Index object containing the labels “lazily” checks whether the values are unique. The reason for being lazy is nearly all performance-based (there are many instances in computations, like parts of GroupBy, where the index is not used).
From dict
If data is a dict, if index is passed the values in data corresponding to the labels in the index will be pulled out. Otherwise, an index will be constructed from the sorted keys of the dict, if possible.
In [274]: d = {'a' : 0., 'b' : 1., 'c' : 2.}
In [275]: Series(d)
Out[275]:
a 0
b 1
c 2
In [276]: Series(d, index=['b', 'c', 'd', 'a'])
Out[276]:
b 1
c 2
d NaN
a 0
Note
NaN (not a number) is the standard missing data marker used in pandas
From scalar value If data is a scalar value, an index must be provided. The value will be repeated to match the length of index
In [277]: Series(5., index=['a', 'b', 'c', 'd', 'e'])
Out[277]:
a 5
b 5
c 5
d 5
e 5
Series is ndarray-like¶
As a subclass of ndarray, Series is a valid argument to most NumPy functions and behaves similarly to a NumPy array. However, things like slicing also slice the index.
In [278]: s[0]
Out[278]: 0.66444516201494186
In [279]: s[:3]
Out[279]:
a 0.664
b -0.487
c -0.504
In [280]: s[s > s.median()]
Out[280]:
a 0.664
e 1.570
In [281]: s[[4, 3, 1]]
Out[281]:
e 1.570
d 0.307
b -0.487
In [282]: np.exp(s)
Out[282]:
a 1.943
b 0.614
c 0.604
d 1.359
e 4.807
We will address array-based indexing in a separate section.
Series is dict-like¶
A Series is alike a fixed-size dict in that you can get and set values by index label:
In [283]: s['a']
Out[283]: 0.66444516201494186
In [284]: s['e'] = 12.
In [285]: s
Out[285]:
a 0.664
b -0.487
c -0.504
d 0.307
e 12.000
In [286]: 'e' in s
Out[286]: True
In [287]: 'f' in s
Out[287]: False
If a label is not contained, an exception is raised:
>>> s['f']
KeyError: 'f'
Using the get method, a missing label will return None or specified default:
In [288]: s.get('f')
In [289]: s.get('f', np.nan)
Out[289]: nan
Vectorized operations and label alignment with Series¶
When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary. Series can be also be passed into most NumPy methods expecting an ndarray.
In [290]: s + s
Out[290]:
a 1.329
b -0.974
c -1.009
d 0.613
e 24.000
In [291]: s * 2
Out[291]:
a 1.329
b -0.974
c -1.009
d 0.613
e 24.000
In [292]: np.exp(s)
Out[292]:
a 1.943
b 0.614
c 0.604
d 1.359
e 162754.791
A key difference between Series and ndarray is that operations between Series automatically align the data based on label. Thus, you can write computations without giving consideration to whether the Series involved have the same labels.
In [293]: s[1:] + s[:-1]
Out[293]:
a NaN
b -0.974
c -1.009
d 0.613
e NaN
The result of an operation between unaligned Series will have the union of the indexes involved. If a label is not found in one Series or the other, the result will be marked as missing (NaN). Being able to write code without doing any explicit data alignment grants immense freedom and flexibility in interactive data analysis and research. The integrated data alignment features of the pandas data structures set pandas apart from the majority of related tools for working with labeled data.
Note
In general, we chose to make the default result of operations between differently indexed objects yield the union of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is typically important information as part of a computation. You of course have the option of dropping labels with missing data via the dropna function.
Name attribute¶
Series can also have a name attribute:
In [294]: s = Series(np.random.randn(5), name='something')
In [295]: s
Out[295]:
0 0.015
1 1.987
2 -0.259
3 0.111
4 1.012
Name: something
In [296]: s.name
Out[296]: 'something'
The Series name will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as you will see below.
DataFrame¶
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:
- Dict of 1D ndarrays, lists, dicts, or Series
- 2-D numpy.ndarray
- Structured or record ndarray
- A Series
- Another DataFrame
Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.
If axis labels are not passed, they will be constructed from the input data based on common sense rules.
From dict of Series or dicts¶
The result index will be the union of the indexes of the various Series. If there are any nested dicts, these will be first converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.
In [297]: d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']),
.....: 'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
.....:
In [298]: df = DataFrame(d)
In [299]: df
Out[299]:
one two
a 1 1
b 2 2
c 3 3
d NaN 4
In [300]: DataFrame(d, index=['d', 'b', 'a'])
Out[300]:
one two
d NaN 4
b 2 2
a 1 1
In [301]: DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
Out[301]:
two three
d 4 NaN
b 2 NaN
a 1 NaN
The row and column labels can be accessed respectively by accessing the index and columns attributes:
Note
When a particular set of columns is passed along with a dict of data, the passed columns override the keys in the dict.
In [302]: df.index
Out[302]: Index([a, b, c, d], dtype=object)
In [303]: df.columns
Out[303]: Index([one, two], dtype=object)
From dict of ndarrays / lists¶
The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays. If no index is passed, the result will be range(n), where n is the array length.
In [304]: d = {'one' : [1., 2., 3., 4.],
.....: 'two' : [4., 3., 2., 1.]}
.....:
In [305]: DataFrame(d)
Out[305]:
one two
0 1 4
1 2 3
2 3 2
3 4 1
In [306]: DataFrame(d, index=['a', 'b', 'c', 'd'])
Out[306]:
one two
a 1 4
b 2 3
c 3 2
d 4 1
From structured or record array¶
This case is handled identically to a dict of arrays.
In [307]: data = np.zeros((2,),dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])
In [308]: data[:] = [(1,2.,'Hello'),(2,3.,"World")]
In [309]: DataFrame(data)
Out[309]:
A B C
0 1 2 Hello
1 2 3 World
In [310]: DataFrame(data, index=['first', 'second'])
Out[310]:
A B C
first 1 2 Hello
second 2 3 World
In [311]: DataFrame(data, columns=['C', 'A', 'B'])
Out[311]:
C A B
0 Hello 1 2
1 World 2 3
Note
DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.
From a list of dicts¶
In [312]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]
In [313]: DataFrame(data2)
Out[313]:
a b c
0 1 2 NaN
1 5 10 20
In [314]: DataFrame(data2, index=['first', 'second'])
Out[314]:
a b c
first 1 2 NaN
second 5 10 20
In [315]: DataFrame(data2, columns=['a', 'b'])
Out[315]:
a b
0 1 2
1 5 10
From a Series¶
The result will be a DataFrame with the same index as the input Series, and with one column whose name is the original name of the Series (only if no other column name provided).
Missing Data
Much more will be said on this topic in the Missing data section. To construct a DataFrame with missing data, use np.nan for those values which are missing. Alternatively, you may pass a numpy.MaskedArray as the data argument to the DataFrame constructor, and its masked entries will be considered missing.
Alternate Constructors¶
DataFrame.from_dict
DataFrame.from_dict takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates like the DataFrame constructor except for the orient parameter which is 'columns' by default, but which can be set to 'index' in order to use the dict keys as row labels.
DataFrame.from_records
DataFrame.from_records takes a list of tuples or an ndarray with structured dtype. Works analogously to the normal DataFrame constructor, except that index maybe be a specific field of the structured dtype to use as the index. For example:
In [316]: data
Out[316]:
array([(1, 2.0, 'Hello'), (2, 3.0, 'World')],
dtype=[('A', '<i4'), ('B', '<f4'), ('C', '|S10')])
In [317]: DataFrame.from_records(data, index='C')
Out[317]:
A B
C
Hello 1 2
World 2 3
DataFrame.from_items
DataFrame.from_items works analogously to the form of the dict constructor that takes a sequence of (key, value) pairs, where the keys are column (or row, in the case of orient='index') names, and the value are the column values (or row values). This can be useful for constructing a DataFrame with the columns in a particular order without having to pass an explicit list of columns:
In [318]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])])
Out[318]:
A B
0 1 4
1 2 5
2 3 6
If you pass orient='index', the keys will be the row labels. But in this case you must also pass the desired column names:
In [319]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])],
.....: orient='index', columns=['one', 'two', 'three'])
.....:
Out[319]:
one two three
A 1 2 3
B 4 5 6
Column selection, addition, deletion¶
You can treat a DataFrame semantically like a dict of like-indexed Series objects. Getting, setting, and deleting columns works with the same syntax as the analogous dict operations:
In [320]: df['one']
Out[320]:
a 1
b 2
c 3
d NaN
Name: one
In [321]: df['three'] = df['one'] * df['two']
In [322]: df['flag'] = df['one'] > 2
In [323]: df
Out[323]:
one two three flag
a 1 1 1 False
b 2 2 4 False
c 3 3 9 True
d NaN 4 NaN False
Columns can be deleted or popped like with a dict:
In [324]: del df['two']
In [325]: three = df.pop('three')
In [326]: df
Out[326]:
one flag
a 1 False
b 2 False
c 3 True
d NaN False
When inserting a scalar value, it will naturally be propagated to fill the column:
In [327]: df['foo'] = 'bar'
In [328]: df
Out[328]:
one flag foo
a 1 False bar
b 2 False bar
c 3 True bar
d NaN False bar
When inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrame’s index:
In [329]: df['one_trunc'] = df['one'][:2]
In [330]: df
Out[330]:
one flag foo one_trunc
a 1 False bar 1
b 2 False bar 2
c 3 True bar NaN
d NaN False bar NaN
You can insert raw ndarrays but their length must match the length of the DataFrame’s index.
By default, columns get inserted at the end. The insert function is available to insert at a particular location in the columns:
In [331]: df.insert(1, 'bar', df['one'])
In [332]: df
Out[332]:
one bar flag foo one_trunc
a 1 1 False bar 1
b 2 2 False bar 2
c 3 3 True bar NaN
d NaN NaN False bar NaN
Indexing / Selection¶
The basics of indexing are as follows:
| Operation | Syntax | Result |
|---|---|---|
| Select column | df[col] | Series |
| Select row by label | df.xs(label) or df.ix[label] | Series |
| Select row by location (int) | df.ix[loc] | Series |
| Slice rows | df[5:10] | DataFrame |
| Select rows by boolean vector | df[bool_vec] | DataFrame |
Row selection, for example, returns a Series whose index is the columns of the DataFrame:
In [333]: df.xs('b')
Out[333]:
one 2
bar 2
flag False
foo bar
one_trunc 2
Name: b
In [334]: df.ix[2]
Out[334]:
one 3
bar 3
flag True
foo bar
one_trunc NaN
Name: c
Note if a DataFrame contains columns of multiple dtypes, the dtype of the row will be chosen to accommodate all of the data types (dtype=object is the most general).
For a more exhaustive treatment of more sophisticated label-based indexing and slicing, see the section on indexing. We will address the fundamentals of reindexing / conforming to new sets of lables in the section on reindexing.
Data alignment and arithmetic¶
Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). Again, the resulting object will have the union of the column and row labels.
In [335]: df = DataFrame(randn(10, 4), columns=['A', 'B', 'C', 'D'])
In [336]: df2 = DataFrame(randn(7, 3), columns=['A', 'B', 'C'])
In [337]: df + df2
Out[337]:
A B C D
0 2.752 -0.429 0.702 NaN
1 0.067 -3.397 1.775 NaN
2 -0.499 -1.138 -1.277 NaN
3 0.731 0.988 0.505 NaN
4 -0.538 -1.828 -1.974 NaN
5 -0.100 -2.885 1.676 NaN
6 1.405 -1.078 0.320 NaN
7 NaN NaN NaN NaN
8 NaN NaN NaN NaN
9 NaN NaN NaN NaN
When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting row-wise. For example:
In [338]: df - df.ix[0]
Out[338]:
A B C D
0 0.000 0.000 0.000 0.000
1 -0.586 -3.234 -1.119 -2.876
2 -1.136 -3.727 -2.066 -1.500
3 -0.799 -0.803 -1.860 -2.038
4 -1.632 -2.216 -2.384 -1.367
5 -0.485 -2.545 -1.044 -1.190
6 -0.168 0.395 -1.333 -1.060
7 -2.367 -0.799 -3.479 -2.653
8 -2.953 -1.565 -2.251 -2.978
9 -1.615 -1.712 -2.521 -2.106
In the special case of working with time series data, if the Series is a TimeSeries (which it will be automatically if the index contains datetime objects), and the DataFrame index also contains dates, the broadcasting will be column-wise:
In [339]: index = date_range('1/1/2000', periods=8)
In [340]: df = DataFrame(randn(8, 3), index=index,
.....: columns=['A', 'B', 'C'])
.....:
In [341]: df
Out[341]:
A B C
2000-01-01 -1.209 -1.257 -0.500
2000-01-02 0.430 -0.242 -0.724
2000-01-03 1.257 -0.871 -0.544
2000-01-04 -0.766 -0.219 0.663
2000-01-05 -1.566 1.780 -2.139
2000-01-06 -0.593 -1.059 0.119
2000-01-07 -0.123 1.306 -0.129
2000-01-08 -0.389 0.143 -1.715
In [342]: type(df['A'])
Out[342]: pandas.core.series.TimeSeries
In [343]: df - df['A']
Out[343]:
A B C
2000-01-01 0 -0.048 0.709
2000-01-02 0 -0.672 -1.154
2000-01-03 0 -2.128 -1.801
2000-01-04 0 0.547 1.429
2000-01-05 0 3.346 -0.572
2000-01-06 0 -0.466 0.711
2000-01-07 0 1.429 -0.006
2000-01-08 0 0.532 -1.326
Technical purity aside, this case is so common in practice that supporting the special case is preferable to the alternative of forcing the user to transpose and do column-based alignment like so:
In [344]: (df.T - df['A']).T
Out[344]:
A B C
2000-01-01 0 -0.048 0.709
2000-01-02 0 -0.672 -1.154
2000-01-03 0 -2.128 -1.801
2000-01-04 0 0.547 1.429
2000-01-05 0 3.346 -0.572
2000-01-06 0 -0.466 0.711
2000-01-07 0 1.429 -0.006
2000-01-08 0 0.532 -1.326
For explicit control over the matching and broadcasting behavior, see the section on flexible binary operations.
Operations with scalars are just as you would expect:
In [345]: df * 5 + 2
Out[345]:
A B C
2000-01-01 -4.043 -4.285 -0.499
2000-01-02 4.149 0.789 -1.619
2000-01-03 8.286 -2.355 -0.719
2000-01-04 -1.830 0.907 5.314
2000-01-05 -5.831 10.900 -8.693
2000-01-06 -0.963 -3.296 2.594
2000-01-07 1.385 8.528 1.354
2000-01-08 0.054 2.716 -6.575
In [346]: 1 / df
Out[346]:
A B C
2000-01-01 -0.827 -0.796 -2.001
2000-01-02 2.327 -4.129 -1.382
2000-01-03 0.795 -1.148 -1.839
2000-01-04 -1.305 -4.574 1.509
2000-01-05 -0.638 0.562 -0.468
2000-01-06 -1.687 -0.944 8.416
2000-01-07 -8.128 0.766 -7.746
2000-01-08 -2.570 6.983 -0.583
In [347]: df ** 4
Out[347]:
A B C
2000-01-01 2.133 2.497 0.062
2000-01-02 0.034 0.003 0.275
2000-01-03 2.499 0.576 0.087
2000-01-04 0.344 0.002 0.193
2000-01-05 6.018 10.038 20.918
2000-01-06 0.123 1.258 0.000
2000-01-07 0.000 2.906 0.000
2000-01-08 0.023 0.000 8.652
Boolean operators work as well:
In [348]: df1 = DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)
In [349]: df2 = DataFrame({'a' : [0, 1, 1], 'b' : [1, 1, 0] }, dtype=bool)
In [350]: df1 & df2
Out[350]:
a b
0 False False
1 False True
2 True False
In [351]: df1 | df2
Out[351]:
a b
0 True True
1 True True
2 True True
In [352]: df1 ^ df2
Out[352]:
a b
0 True True
1 True False
2 False True
In [353]: -df1
Out[353]:
a b
0 False True
1 True False
2 False False
Transposing¶
To transpose, access the T attribute (also the transpose function), similar to an ndarray:
# only show the first 5 rows
In [354]: df[:5].T
Out[354]:
2000-01-01 2000-01-02 2000-01-03 2000-01-04 2000-01-05
A -1.209 0.430 1.257 -0.766 -1.566
B -1.257 -0.242 -0.871 -0.219 1.780
C -0.500 -0.724 -0.544 0.663 -2.139
DataFrame interoperability with NumPy functions¶
Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions can be used with no issues on DataFrame, assuming the data within are numeric:
In [355]: np.exp(df)
Out[355]:
A B C
2000-01-01 0.299 0.285 0.607
2000-01-02 1.537 0.785 0.485
2000-01-03 3.516 0.419 0.581
2000-01-04 0.465 0.804 1.940
2000-01-05 0.209 5.930 0.118
2000-01-06 0.553 0.347 1.126
2000-01-07 0.884 3.690 0.879
2000-01-08 0.678 1.154 0.180
In [356]: np.asarray(df)
Out[356]:
array([[-1.2085, -1.257 , -0.4997],
[ 0.4298, -0.2422, -0.7238],
[ 1.2573, -0.871 , -0.5437],
[-0.7661, -0.2186, 0.6628],
[-1.5663, 1.78 , -2.1386],
[-0.5927, -1.0591, 0.1188],
[-0.123 , 1.3056, -0.1291],
[-0.3892, 0.1432, -1.715 ]])
The dot method on DataFrame implements matrix multiplication:
In [357]: df.T.dot(df)
Out[357]:
A B C
A 6.784 -1.889 3.064
B -1.889 8.460 -3.214
C 3.064 -3.214 9.055
Similarly, the dot method on Series implements dot product:
In [358]: s1 = Series(np.arange(5,10))
In [359]: s1.dot(s1)
Out[359]: 255
DataFrame is not intended to be a drop-in replacement for ndarray as its indexing semantics are quite different in places from a matrix.
Console display¶
For very large DataFrame objects, only a summary will be printed to the console (here I am reading a CSV version of the baseball dataset from the plyr R package):
In [360]: baseball = read_csv('data/baseball.csv')
In [361]: print baseball
<class 'pandas.core.frame.DataFrame'>
Int64Index: 100 entries, 88641 to 89534
Data columns:
id 100 non-null values
year 100 non-null values
stint 100 non-null values
team 100 non-null values
lg 100 non-null values
g 100 non-null values
ab 100 non-null values
r 100 non-null values
h 100 non-null values
X2b 100 non-null values
X3b 100 non-null values
hr 100 non-null values
rbi 100 non-null values
sb 100 non-null values
cs 100 non-null values
bb 100 non-null values
so 100 non-null values
ibb 100 non-null values
hbp 100 non-null values
sh 100 non-null values
sf 100 non-null values
gidp 100 non-null values
dtypes: float64(9), int64(10), object(3)
However, using to_string will return a string representation of the DataFrame in tabular form, though it won’t always fit the console width:
In [362]: print baseball.ix[-20:, :12].to_string()
id year stint team lg g ab r h X2b X3b hr
88641 womacto01 2006 2 CHN NL 19 50 6 14 1 0 1
88643 schilcu01 2006 1 BOS AL 31 2 0 1 0 0 0
88645 myersmi01 2006 1 NYA AL 62 0 0 0 0 0 0
88649 helliri01 2006 1 MIL NL 20 3 0 0 0 0 0
88650 johnsra05 2006 1 NYA AL 33 6 0 1 0 0 0
88652 finlest01 2006 1 SFN NL 139 426 66 105 21 12 6
88653 gonzalu01 2006 1 ARI NL 153 586 93 159 52 2 15
88662 seleaa01 2006 1 LAN NL 28 26 2 5 1 0 0
89177 francju01 2007 2 ATL NL 15 40 1 10 3 0 0
89178 francju01 2007 1 NYN NL 40 50 7 10 0 0 1
89330 zaungr01 2007 1 TOR AL 110 331 43 80 24 1 10
89333 witasja01 2007 1 TBA AL 3 0 0 0 0 0 0
89334 williwo02 2007 1 HOU NL 33 59 3 6 0 0 1
89335 wickmbo01 2007 2 ARI NL 8 0 0 0 0 0 0
89336 wickmbo01 2007 1 ATL NL 47 0 0 0 0 0 0
89337 whitero02 2007 1 MIN AL 38 109 8 19 4 0 4
89338 whiteri01 2007 1 HOU NL 20 1 0 0 0 0 0
89339 wellsda01 2007 2 LAN NL 7 15 2 4 1 0 0
89340 wellsda01 2007 1 SDN NL 22 38 1 4 0 0 0
89341 weathda01 2007 1 CIN NL 67 0 0 0 0 0 0
89343 walketo04 2007 1 OAK AL 18 48 5 13 1 0 0
89345 wakefti01 2007 1 BOS AL 1 2 0 0 0 0 0
89347 vizquom01 2007 1 SFN NL 145 513 54 126 18 3 4
89348 villoro01 2007 1 NYA AL 6 0 0 0 0 0 0
89352 valenjo03 2007 1 NYN NL 51 166 18 40 11 1 3
89354 trachst01 2007 2 CHN NL 4 7 0 1 0 0 0
89355 trachst01 2007 1 BAL AL 3 5 0 0 0 0 0
89359 timlimi01 2007 1 BOS AL 4 0 0 0 0 0 0
89360 thomeji01 2007 1 CHA AL 130 432 79 119 19 0 35
89361 thomafr04 2007 1 TOR AL 155 531 63 147 30 0 26
89363 tavarju01 2007 1 BOS AL 2 4 0 1 0 0 0
89365 sweenma01 2007 2 LAN NL 30 33 2 9 1 0 0
89366 sweenma01 2007 1 SFN NL 76 90 18 23 8 0 2
89367 suppaje01 2007 1 MIL NL 33 61 4 8 0 0 0
89368 stinnke01 2007 1 SLN NL 26 82 7 13 3 0 1
89370 stantmi02 2007 1 CIN NL 67 2 0 0 0 0 0
89371 stairma01 2007 1 TOR AL 125 357 58 103 28 1 21
89372 sprinru01 2007 1 SLN NL 72 1 0 0 0 0 0
89374 sosasa01 2007 1 TEX AL 114 412 53 104 24 1 21
89375 smoltjo01 2007 1 ATL NL 30 54 1 5 1 0 0
89378 sheffga01 2007 1 DET AL 133 494 107 131 20 1 25
89381 seleaa01 2007 1 NYN NL 31 4 0 0 0 0 0
89382 seaneru01 2007 1 LAN NL 68 1 0 0 0 0 0
89383 schmija01 2007 1 LAN NL 6 7 1 1 0 0 1
89384 schilcu01 2007 1 BOS AL 1 2 0 1 0 0 0
89385 sandere02 2007 1 KCA AL 24 73 12 23 7 0 2
89388 rogerke01 2007 1 DET AL 1 2 0 0 0 0 0
89389 rodriiv01 2007 1 DET AL 129 502 50 141 31 3 11
89396 ramirma02 2007 1 BOS AL 133 483 84 143 33 1 20
89398 piazzmi01 2007 1 OAK AL 83 309 33 85 17 1 8
89400 perezne01 2007 1 DET AL 33 64 5 11 3 0 1
89402 parkch01 2007 1 NYN NL 1 1 0 0 0 0 0
89406 oliveda02 2007 1 LAA AL 5 0 0 0 0 0 0
89410 myersmi01 2007 1 NYA AL 6 1 0 0 0 0 0
89411 mussimi01 2007 1 NYA AL 2 2 0 0 0 0 0
89412 moyerja01 2007 1 PHI NL 33 73 4 9 2 0 0
89420 mesajo01 2007 1 PHI NL 38 0 0 0 0 0 0
89421 martipe02 2007 1 NYN NL 5 9 1 1 1 0 0
89425 maddugr01 2007 1 SDN NL 33 62 2 9 2 0 0
89426 mabryjo01 2007 1 COL NL 28 34 4 4 1 0 1
89429 loftoke01 2007 2 CLE AL 52 173 24 49 9 3 0
89430 loftoke01 2007 1 TEX AL 84 317 62 96 16 3 7
89431 loaizes01 2007 1 LAN NL 5 7 0 1 0 0 0
89438 kleskry01 2007 1 SFN NL 116 362 51 94 27 3 6
89439 kentje01 2007 1 LAN NL 136 494 78 149 36 1 20
89442 jonesto02 2007 1 DET AL 5 0 0 0 0 0 0
89445 johnsra05 2007 1 ARI NL 10 15 0 1 0 0 0
89450 hoffmtr01 2007 1 SDN NL 60 0 0 0 0 0 0
89451 hernaro01 2007 2 LAN NL 22 0 0 0 0 0 0
89452 hernaro01 2007 1 CLE AL 2 0 0 0 0 0 0
89460 guarded01 2007 1 CIN NL 15 0 0 0 0 0 0
89462 griffke02 2007 1 CIN NL 144 528 78 146 24 1 30
89463 greensh01 2007 1 NYN NL 130 446 62 130 30 1 10
89464 graffto01 2007 1 MIL NL 86 231 34 55 8 0 9
89465 gordoto01 2007 1 PHI NL 44 0 0 0 0 0 0
89466 gonzalu01 2007 1 LAN NL 139 464 70 129 23 2 15
89467 gomezch02 2007 2 CLE AL 19 53 4 15 2 0 0
89468 gomezch02 2007 1 BAL AL 73 169 17 51 10 1 1
89469 glavito02 2007 1 NYN NL 33 56 3 12 1 0 0
89473 floydcl01 2007 1 CHN NL 108 282 40 80 10 1 9
89474 finlest01 2007 1 COL NL 43 94 9 17 3 0 1
89480 embreal01 2007 1 OAK AL 4 0 0 0 0 0 0
89481 edmonji01 2007 1 SLN NL 117 365 39 92 15 2 12
89482 easleda01 2007 1 NYN NL 76 193 24 54 6 0 10
89489 delgaca01 2007 1 NYN NL 139 538 71 139 30 0 24
89493 cormirh01 2007 1 CIN NL 6 0 0 0 0 0 0
89494 coninje01 2007 2 NYN NL 21 41 2 8 2 0 0
89495 coninje01 2007 1 CIN NL 80 215 23 57 11 1 6
89497 clemero02 2007 1 NYA AL 2 2 0 1 0 0 0
89498 claytro01 2007 2 BOS AL 8 6 1 0 0 0 0
89499 claytro01 2007 1 TOR AL 69 189 23 48 14 0 1
89501 cirilje01 2007 2 ARI NL 28 40 6 8 4 0 0
89502 cirilje01 2007 1 MIN AL 50 153 18 40 9 2 2
89521 bondsba01 2007 1 SFN NL 126 340 75 94 14 0 28
89523 biggicr01 2007 1 HOU NL 141 517 68 130 31 3 10
89525 benitar01 2007 2 FLO NL 34 0 0 0 0 0 0
89526 benitar01 2007 1 SFN NL 19 0 0 0 0 0 0
89530 ausmubr01 2007 1 HOU NL 117 349 38 82 16 3 3
89533 aloumo01 2007 1 NYN NL 87 328 51 112 19 1 13
89534 alomasa02 2007 1 NYN NL 8 22 1 3 1 0 0
DataFrame column types¶
The four main types stored in pandas objects are float, int, boolean, and object. A convenient dtypes attribute return a Series with the data type of each column:
In [363]: baseball.dtypes
Out[363]:
id object
year int64
stint int64
team object
lg object
g int64
ab int64
r int64
h int64
X2b int64
X3b int64
hr int64
rbi float64
sb float64
cs float64
bb int64
so float64
ibb float64
hbp float64
sh float64
sf float64
gidp float64
The related method get_dtype_counts will return the number of columns of each type:
In [364]: baseball.get_dtype_counts()
Out[364]:
float64 9
int64 10
object 3
DataFrame column attribute access and IPython completion¶
If a DataFrame column label is a valid Python variable name, the column can be accessed like attributes:
In [365]: df = DataFrame({'foo1' : np.random.randn(5),
.....: 'foo2' : np.random.randn(5)})
.....:
In [366]: df
Out[366]:
foo1 foo2
0 0.759091 -0.648742
1 -0.050457 0.209870
2 0.959219 -0.325391
3 -0.817600 -1.978199
4 -0.200407 -0.211127
In [367]: df.foo1
Out[367]:
0 0.759091
1 -0.050457
2 0.959219
3 -0.817600
4 -0.200407
Name: foo1
The columns are also connected to the IPython completion mechanism so they can be tab-completed:
In [5]: df.fo<TAB>
df.foo1 df.foo2
Panel¶
Panel is a somewhat less-used, but still important container for 3-dimensional data. The term panel data is derived from econometrics and is partially responsible for the name pandas: pan(el)-da(ta)-s. The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data and, in particular, econometric analysis of panel data. However, for the strict purposes of slicing and dicing a collection of DataFrame objects, you may find the axis names slightly arbitrary:
- items: axis 0, each item corresponds to a DataFrame contained inside
- major_axis: axis 1, it is the index (rows) of each of the DataFrames
- minor_axis: axis 2, it is the columns of each of the DataFrames
Construction of Panels works about like you would expect:
From 3D ndarray with optional axis labels¶
In [368]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
.....: major_axis=date_range('1/1/2000', periods=5),
.....: minor_axis=['A', 'B', 'C', 'D'])
.....:
In [369]: wp
Out[369]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 5 (major) x 4 (minor)
Items: Item1 to Item2
Major axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor axis: A to D
From dict of DataFrame objects¶
In [370]: data = {'Item1' : DataFrame(randn(4, 3)),
.....: 'Item2' : DataFrame(randn(4, 2))}
.....:
In [371]: Panel(data)
Out[371]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major) x 3 (minor)
Items: Item1 to Item2
Major axis: 0 to 3
Minor axis: 0 to 2
Note that the values in the dict need only be convertible to DataFrame. Thus, they can be any of the other valid inputs to DataFrame as per above.
One helpful factory method is Panel.from_dict, which takes a dictionary of DataFrames as above, and the following named parameters:
| Parameter | Default | Description |
|---|---|---|
| intersect | False | drops elements whose indices do not align |
| orient | items | use minor to use DataFrames’ columns as panel items |
For example, compare to the construction above:
In [372]: Panel.from_dict(data, orient='minor')
Out[372]:
<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 4 (major) x 2 (minor)
Items: 0 to 2
Major axis: 0 to 3
Minor axis: Item1 to Item2
Orient is especially useful for mixed-type DataFrames. If you pass a dict of DataFrame objects with mixed-type columns, all of the data will get upcasted to dtype=object unless you pass orient='minor':
In [373]: df = DataFrame({'a': ['foo', 'bar', 'baz'],
.....: 'b': np.random.randn(3)})
.....:
In [374]: df
Out[374]:
a b
0 foo 0.080597
1 bar -0.000185
2 baz -0.264704
In [375]: data = {'item1': df, 'item2': df}
In [376]: panel = Panel.from_dict(data, orient='minor')
In [377]: panel['a']
Out[377]:
item1 item2
0 foo foo
1 bar bar
2 baz baz
In [378]: panel['b']
Out[378]:
item1 item2
0 0.080597 0.080597
1 -0.000185 -0.000185
2 -0.264704 -0.264704
In [379]: panel['b'].dtypes
Out[379]:
item1 float64
item2 float64
Note
Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected feature-wise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on, of course, in future releases. And faster if you join me in working on the codebase.
From DataFrame using to_panel method¶
This method was introduced in v0.7 to replace LongPanel.to_long, and converts a DataFrame with a two-level index to a Panel.
In [380]: midx = MultiIndex(levels=[['one', 'two'], ['x','y']], labels=[[1,1,0,0],[1,0,1,0]])
In [381]: df = DataFrame({'A' : [1, 2, 3, 4], 'B': [5, 6, 7, 8]}, index=midx)
In [382]: df.to_panel()
Out[382]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 2 (major) x 2 (minor)
Items: A to B
Major axis: one to two
Minor axis: x to y
Item selection / addition / deletion¶
Similar to DataFrame functioning as a dict of Series, Panel is like a dict of DataFrames:
In [383]: wp['Item1']
Out[383]:
A B C D
2000-01-01 -0.519332 -1.765523 -0.966196 -0.890524
2000-01-02 -1.314597 -1.458515 -0.919663 -0.699091
2000-01-03 1.357258 -0.098278 -0.987183 -1.362030
2000-01-04 -1.309989 -1.153000 0.606382 -0.681101
2000-01-05 -0.289724 -0.996632 -1.407699 1.014104
In [384]: wp['Item3'] = wp['Item1'] / wp['Item2']
The API for insertion and deletion is the same as for DataFrame. And as with DataFrame, if the item is a valid python identifier, you can access it as an attribute and tab-complete it in IPython.
Transposing¶
A Panel can be rearranged using its transpose method (which does not make a copy by default unless the data are heterogeneous):
In [385]: wp.transpose(2, 0, 1)
Out[385]:
<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 3 (major) x 5 (minor)
Items: A to D
Major axis: Item1 to Item3
Minor axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Indexing / Selection¶
| Operation | Syntax | Result |
|---|---|---|
| Select item | wp[item] | DataFrame |
| Get slice at major_axis label | wp.major_xs(val) | DataFrame |
| Get slice at minor_axis label | wp.minor_xs(val) | DataFrame |
For example, using the earlier example data, we could do:
In [386]: wp['Item1']
Out[386]:
A B C D
2000-01-01 -0.519332 -1.765523 -0.966196 -0.890524
2000-01-02 -1.314597 -1.458515 -0.919663 -0.699091
2000-01-03 1.357258 -0.098278 -0.987183 -1.362030
2000-01-04 -1.309989 -1.153000 0.606382 -0.681101
2000-01-05 -0.289724 -0.996632 -1.407699 1.014104
In [387]: wp.major_xs(wp.major_axis[2])
Out[387]:
Item1 Item2 Item3
A 1.357258 -0.177665 -7.639427
B -0.098278 0.490838 -0.200224
C -0.987183 -1.360102 0.725815
D -1.362030 1.592456 -0.855302
In [388]: wp.minor_axis
Out[388]: Index([A, B, C, D], dtype=object)
In [389]: wp.minor_xs('C')
Out[389]:
Item1 Item2 Item3
2000-01-01 -0.966196 0.071823 -13.452418
2000-01-02 -0.919663 0.214910 -4.279288
2000-01-03 -0.987183 -1.360102 0.725815
2000-01-04 0.606382 -1.890591 -0.320737
2000-01-05 -1.407699 -0.151652 9.282439
Conversion to DataFrame¶
A Panel can be represented in 2D form as a hierarchically indexed DataFrame. See the section hierarchical indexing for more on this. To convert a Panel to a DataFrame, use the to_frame method:
In [390]: panel = Panel(np.random.randn(3, 5, 4), items=['one', 'two', 'three'],
.....: major_axis=date_range('1/1/2000', periods=5),
.....: minor_axis=['a', 'b', 'c', 'd'])
.....:
In [391]: panel.to_frame()
Out[391]:
one two three
major minor
2000-01-01 a -0.566820 0.597468 0.716659
b -1.643966 -0.491240 -0.919717
c 1.471262 1.281674 -0.024595
d 0.677634 -0.099685 0.068997
2000-01-02 a -0.485743 -1.823043 0.601797
b -0.342272 -0.779213 0.866615
c -1.042291 -0.949327 0.092911
d -0.611457 0.768043 -2.606892
2000-01-03 a -0.141224 -0.054860 0.309303
b 0.007220 -1.493561 -0.548401
c -0.516147 0.106004 -2.044772
d 0.446161 -0.903513 -1.666264
2000-01-04 a 0.483368 -0.719875 -1.439775
b 0.186405 0.301945 1.326361
c -1.439567 1.112546 0.221680
d -0.503782 -0.542770 1.840992
2000-01-05 a 0.890769 -2.695540 1.165150
b -0.777798 0.431284 -1.420521
c -0.552820 -0.431092 1.616679
d -1.428744 1.666631 -1.030912