Version 0.15.0 (October 18, 2014)¶
This is a major release from 0.14.1 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.
Warning
pandas >= 0.15.0 will no longer support compatibility with NumPy versions < 1.7.0. If you want to use the latest versions of pandas, please upgrade to NumPy >= 1.7.0 (GH7711)
Highlights include:
The
Categorical
type was integrated as a first-class pandas type, see hereNew scalar type
Timedelta
, and a new index typeTimedeltaIndex
, see hereNew datetimelike properties accessor
.dt
for Series, see Datetimelike PropertiesNew DataFrame default display for
df.info()
to include memory usage, see Memory Usageread_csv
will now by default ignore blank lines when parsing, see hereAPI change in using Indexes in set operations, see here
Enhancements in the handling of timezones, see here
A lot of improvements to the rolling and expanding moment functions, see here
Internal refactoring of the
Index
class to no longer sub-classndarray
, see Internal Refactoringdropping support for
PyTables
less than version 3.0.0, andnumexpr
less than version 2.1 (GH7990)Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing
Split out string methods documentation into Working with Text Data
Check the API Changes and deprecations before updating
Warning
In 0.15.0 Index
has internally been refactored to no longer sub-class ndarray
but instead subclass PandasObject
, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be
a transparent change with only very limited API implications (See the Internal Refactoring)
Warning
The refactoring in Categorical
changed the two argument constructor from
“codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use
Categorical
directly, please audit your code before updating to this pandas
version and change it to use the from_codes()
constructor. See more on Categorical
here
New features¶
Categoricals in Series/DataFrame¶
Categorical
can now be included in Series
and DataFrames
and gained new
methods to manipulate. Thanks to Jan Schulz for much of this API/implementation. (GH3943, GH5313, GH5314,
GH7444, GH7839, GH7848, GH7864, GH7914, GH7768, GH8006, GH3678,
GH8075, GH8076, GH8143, GH8453, GH8518).
For full docs, see the categorical introduction and the API documentation.
In [1]: df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
...: "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})
...:
In [2]: df["grade"] = df["raw_grade"].astype("category")
In [3]: df["grade"]
Out[3]:
0 a
1 b
2 b
3 a
4 a
5 e
Name: grade, Length: 6, dtype: category
Categories (3, object): ['a', 'b', 'e']
# Rename the categories
In [4]: df["grade"].cat.categories = ["very good", "good", "very bad"]
# Reorder the categories and simultaneously add the missing categories
In [5]: df["grade"] = df["grade"].cat.set_categories(["very bad", "bad",
...: "medium", "good", "very good"])
...:
In [6]: df["grade"]
Out[6]:
0 very good
1 good
2 good
3 very good
4 very good
5 very bad
Name: grade, Length: 6, dtype: category
Categories (5, object): ['very bad', 'bad', 'medium', 'good', 'very good']
In [7]: df.sort_values("grade")
Out[7]:
id raw_grade grade
5 6 e very bad
1 2 b good
2 3 b good
0 1 a very good
3 4 a very good
4 5 a very good
[6 rows x 3 columns]
In [8]: df.groupby("grade").size()
Out[8]:
grade
very bad 1
bad 0
medium 0
good 2
very good 3
Length: 5, dtype: int64
pandas.core.group_agg
andpandas.core.factor_agg
were removed. As an alternative, construct a dataframe and usedf.groupby(<group>).agg(<func>)
.Supplying “codes/labels and levels” to the
Categorical
constructor is not supported anymore. Supplying two arguments to the constructor is now interpreted as “values and levels (now called ‘categories’)”. Please change your code to use thefrom_codes()
constructor.The
Categorical.labels
attribute was renamed toCategorical.codes
and is read only. If you want to manipulate codes, please use one of the API methods on Categoricals.The
Categorical.levels
attribute is renamed toCategorical.categories
.
TimedeltaIndex/scalar¶
We introduce a new scalar type Timedelta
, which is a subclass of datetime.timedelta
, and behaves in a similar manner,
but allows compatibility with np.timedelta64
types as well as a host of custom representation, parsing, and attributes.
This type is very similar to how Timestamp
works for datetimes
. It is a nice-API box for the type. See the docs.
(GH3009, GH4533, GH8209, GH8187, GH8190, GH7869, GH7661, GH8345, GH8471)
Warning
Timedelta
scalars (and TimedeltaIndex
) component fields are not the same as the component fields on a datetime.timedelta
object. For example, .seconds
on a datetime.timedelta
object returns the total number of seconds combined between hours
, minutes
and seconds
. In contrast, the pandas Timedelta
breaks out hours, minutes, microseconds and nanoseconds separately.
# Timedelta accessor
In [9]: tds = pd.Timedelta('31 days 5 min 3 sec')
In [10]: tds.minutes
Out[10]: 5L
In [11]: tds.seconds
Out[11]: 3L
# datetime.timedelta accessor
# this is 5 minutes * 60 + 3 seconds
In [12]: tds.to_pytimedelta().seconds
Out[12]: 303
Note: this is no longer true starting from v0.16.0, where full
compatibility with datetime.timedelta
is introduced. See the
0.16.0 whatsnew entry
Warning
Prior to 0.15.0 pd.to_timedelta
would return a Series
for list-like/Series input, and a np.timedelta64
for scalar input.
It will now return a TimedeltaIndex
for list-like input, Series
for Series input, and Timedelta
for scalar input.
The arguments to pd.to_timedelta
are now (arg,unit='ns',box=True,coerce=False)
, previously were (arg,box=True,unit='ns')
as these are more logical.
Construct a scalar
In [9]: pd.Timedelta('1 days 06:05:01.00003')
Out[9]: Timedelta('1 days 06:05:01.000030')
In [10]: pd.Timedelta('15.5us')
Out[10]: Timedelta('0 days 00:00:00.000015500')
In [11]: pd.Timedelta('1 hour 15.5us')
Out[11]: Timedelta('0 days 01:00:00.000015500')
# negative Timedeltas have this string repr
# to be more consistent with datetime.timedelta conventions
In [12]: pd.Timedelta('-1us')
Out[12]: Timedelta('-1 days +23:59:59.999999')
# a NaT
In [13]: pd.Timedelta('nan')
Out[13]: NaT
Access fields for a Timedelta
In [14]: td = pd.Timedelta('1 hour 3m 15.5us')
In [15]: td.seconds
Out[15]: 3780
In [16]: td.microseconds
Out[16]: 15
In [17]: td.nanoseconds
Out[17]: 500
Construct a TimedeltaIndex
In [18]: pd.TimedeltaIndex(['1 days', '1 days, 00:00:05',
....: np.timedelta64(2, 'D'),
....: datetime.timedelta(days=2, seconds=2)])
....:
Out[18]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:00:05', '2 days 00:00:00',
'2 days 00:00:02'],
dtype='timedelta64[ns]', freq=None)
Constructing a TimedeltaIndex
with a regular range
In [19]: pd.timedelta_range('1 days', periods=5, freq='D')
Out[19]: TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq='D')
In [20]: pd.timedelta_range(start='1 days', end='2 days', freq='30T')
Out[20]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:30:00', '1 days 01:00:00',
'1 days 01:30:00', '1 days 02:00:00', '1 days 02:30:00',
'1 days 03:00:00', '1 days 03:30:00', '1 days 04:00:00',
'1 days 04:30:00', '1 days 05:00:00', '1 days 05:30:00',
'1 days 06:00:00', '1 days 06:30:00', '1 days 07:00:00',
'1 days 07:30:00', '1 days 08:00:00', '1 days 08:30:00',
'1 days 09:00:00', '1 days 09:30:00', '1 days 10:00:00',
'1 days 10:30:00', '1 days 11:00:00', '1 days 11:30:00',
'1 days 12:00:00', '1 days 12:30:00', '1 days 13:00:00',
'1 days 13:30:00', '1 days 14:00:00', '1 days 14:30:00',
'1 days 15:00:00', '1 days 15:30:00', '1 days 16:00:00',
'1 days 16:30:00', '1 days 17:00:00', '1 days 17:30:00',
'1 days 18:00:00', '1 days 18:30:00', '1 days 19:00:00',
'1 days 19:30:00', '1 days 20:00:00', '1 days 20:30:00',
'1 days 21:00:00', '1 days 21:30:00', '1 days 22:00:00',
'1 days 22:30:00', '1 days 23:00:00', '1 days 23:30:00',
'2 days 00:00:00'],
dtype='timedelta64[ns]', freq='30T')
You can now use a TimedeltaIndex
as the index of a pandas object
In [21]: s = pd.Series(np.arange(5),
....: index=pd.timedelta_range('1 days', periods=5, freq='s'))
....:
In [22]: s
Out[22]:
1 days 00:00:00 0
1 days 00:00:01 1
1 days 00:00:02 2
1 days 00:00:03 3
1 days 00:00:04 4
Freq: S, Length: 5, dtype: int64
You can select with partial string selections
In [23]: s['1 day 00:00:02']
Out[23]: 2
In [24]: s['1 day':'1 day 00:00:02']
Out[24]:
1 days 00:00:00 0
1 days 00:00:01 1
1 days 00:00:02 2
Freq: S, Length: 3, dtype: int64
Finally, the combination of TimedeltaIndex
with DatetimeIndex
allow certain combination operations that are NaT
preserving:
In [25]: tdi = pd.TimedeltaIndex(['1 days', pd.NaT, '2 days'])
In [26]: tdi.tolist()
Out[26]: [Timedelta('1 days 00:00:00'), NaT, Timedelta('2 days 00:00:00')]
In [27]: dti = pd.date_range('20130101', periods=3)
In [28]: dti.tolist()
Out[28]:
[Timestamp('2013-01-01 00:00:00', freq='D'),
Timestamp('2013-01-02 00:00:00', freq='D'),
Timestamp('2013-01-03 00:00:00', freq='D')]
In [29]: (dti + tdi).tolist()
Out[29]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]
In [30]: (dti - tdi).tolist()
Out[30]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]
iteration of a
Series
e.g.list(Series(...))
oftimedelta64[ns]
would prior to v0.15.0 returnnp.timedelta64
for each element. These will now be wrapped inTimedelta
.
Memory usage¶
Implemented methods to find memory usage of a DataFrame. See the FAQ for more. (GH6852).
A new display option display.memory_usage
(see Options and settings) sets the default behavior of the memory_usage
argument in the df.info()
method. By default display.memory_usage
is True
.
In [31]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
....: 'complex128', 'object', 'bool']
....:
In [32]: n = 5000
In [33]: data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}
In [34]: df = pd.DataFrame(data)
In [35]: df['categorical'] = df['object'].astype('category')
In [36]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 int64 5000 non-null int64
1 float64 5000 non-null float64
2 datetime64[ns] 5000 non-null datetime64[ns]
3 timedelta64[ns] 5000 non-null timedelta64[ns]
4 complex128 5000 non-null complex128
5 object 5000 non-null object
6 bool 5000 non-null bool
7 categorical 5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 288.2+ KB
Additionally memory_usage()
is an available method for a dataframe object which returns the memory usage of each column.
In [37]: df.memory_usage(index=True)
Out[37]:
Index 128
int64 40000
float64 40000
datetime64[ns] 40000
timedelta64[ns] 40000
complex128 80000
object 40000
bool 5000
categorical 9968
Length: 9, dtype: int64
Series.dt accessor¶
Series
has gained an accessor to succinctly return datetime like properties for the values of the Series, if its a datetime/period like Series. (GH7207)
This will return a Series, indexed like the existing Series. See the docs
# datetime
In [38]: s = pd.Series(pd.date_range('20130101 09:10:12', periods=4))
In [39]: s
Out[39]:
0 2013-01-01 09:10:12
1 2013-01-02 09:10:12
2 2013-01-03 09:10:12
3 2013-01-04 09:10:12
Length: 4, dtype: datetime64[ns]
In [40]: s.dt.hour
Out[40]:
0 9
1 9
2 9
3 9
Length: 4, dtype: int64
In [41]: s.dt.second
Out[41]:
0 12
1 12
2 12
3 12
Length: 4, dtype: int64
In [42]: s.dt.day
Out[42]:
0 1
1 2
2 3
3 4
Length: 4, dtype: int64
In [43]: s.dt.freq
Out[43]: 'D'
This enables nice expressions like this:
In [44]: s[s.dt.day == 2]
Out[44]:
1 2013-01-02 09:10:12
Length: 1, dtype: datetime64[ns]
You can easily produce tz aware transformations:
In [45]: stz = s.dt.tz_localize('US/Eastern')
In [46]: stz
Out[46]:
0 2013-01-01 09:10:12-05:00
1 2013-01-02 09:10:12-05:00
2 2013-01-03 09:10:12-05:00
3 2013-01-04 09:10:12-05:00
Length: 4, dtype: datetime64[ns, US/Eastern]
In [47]: stz.dt.tz
Out[47]: <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
You can also chain these types of operations:
In [48]: s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
Out[48]:
0 2013-01-01 04:10:12-05:00
1 2013-01-02 04:10:12-05:00
2 2013-01-03 04:10:12-05:00
3 2013-01-04 04:10:12-05:00
Length: 4, dtype: datetime64[ns, US/Eastern]
The .dt
accessor works for period and timedelta dtypes.
# period
In [49]: s = pd.Series(pd.period_range('20130101', periods=4, freq='D'))
In [50]: s
Out[50]:
0 2013-01-01
1 2013-01-02
2 2013-01-03
3 2013-01-04
Length: 4, dtype: period[D]
In [51]: s.dt.year
Out[51]:
0 2013
1 2013
2 2013
3 2013
Length: 4, dtype: int64
In [52]: s.dt.day
Out[52]:
0 1
1 2
2 3
3 4
Length: 4, dtype: int64
# timedelta
In [53]: s = pd.Series(pd.timedelta_range('1 day 00:00:05', periods=4, freq='s'))
In [54]: s
Out[54]:
0 1 days 00:00:05
1 1 days 00:00:06
2 1 days 00:00:07
3 1 days 00:00:08
Length: 4, dtype: timedelta64[ns]
In [55]: s.dt.days
Out[55]:
0 1
1 1
2 1
3 1
Length: 4, dtype: int64
In [56]: s.dt.seconds
Out[56]:
0 5
1 6
2 7
3 8
Length: 4, dtype: int64
In [57]: s.dt.components
Out[57]:
days hours minutes seconds milliseconds microseconds nanoseconds
0 1 0 0 5 0 0 0
1 1 0 0 6 0 0 0
2 1 0 0 7 0 0 0
3 1 0 0 8 0 0 0
[4 rows x 7 columns]
Timezone handling improvements¶
tz_localize(None)
for tz-awareTimestamp
andDatetimeIndex
now removes timezone holding local time, previously this resulted inException
orTypeError
(GH7812)In [58]: ts = pd.Timestamp('2014-08-01 09:00', tz='US/Eastern') In [59]: ts Out[59]: Timestamp('2014-08-01 09:00:00-0400', tz='US/Eastern') In [60]: ts.tz_localize(None) Out[60]: Timestamp('2014-08-01 09:00:00') In [61]: didx = pd.date_range(start='2014-08-01 09:00', freq='H', ....: periods=10, tz='US/Eastern') ....: In [62]: didx Out[62]: DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00', '2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00', '2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00', '2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00', '2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'], dtype='datetime64[ns, US/Eastern]', freq='H') In [63]: didx.tz_localize(None) Out[63]: DatetimeIndex(['2014-08-01 09:00:00', '2014-08-01 10:00:00', '2014-08-01 11:00:00', '2014-08-01 12:00:00', '2014-08-01 13:00:00', '2014-08-01 14:00:00', '2014-08-01 15:00:00', '2014-08-01 16:00:00', '2014-08-01 17:00:00', '2014-08-01 18:00:00'], dtype='datetime64[ns]', freq=None)
tz_localize
now accepts theambiguous
keyword which allows for passing an array of bools indicating whether the date belongs in DST or not, ‘NaT’ for setting transition times to NaT, ‘infer’ for inferring DST/non-DST, and ‘raise’ (default) for anAmbiguousTimeError
to be raised. See the docs for more details (GH7943)DataFrame.tz_localize
andDataFrame.tz_convert
now accepts an optionallevel
argument for localizing a specific level of a MultiIndex (GH7846)Timestamp.tz_localize
andTimestamp.tz_convert
now raiseTypeError
in error cases, rather thanException
(GH8025)a timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone (rather than being a naive
datetime64[ns]
) asobject
dtype (GH8411)Timestamp.__repr__
displaysdateutil.tz.tzoffset
info (GH7907)
Rolling/expanding moments improvements¶
rolling_min()
,rolling_max()
,rolling_cov()
, androlling_corr()
now return objects with allNaN
whenlen(arg) < min_periods <= window
rather than raising. (This makes all rolling functions consistent in this behavior). (GH7766)Prior to 0.15.0
In [64]: s = pd.Series([10, 11, 12, 13])
In [15]: pd.rolling_min(s, window=10, min_periods=5) ValueError: min_periods (5) must be <= window (4)
New behavior
In [4]: pd.rolling_min(s, window=10, min_periods=5) Out[4]: 0 NaN 1 NaN 2 NaN 3 NaN dtype: float64
rolling_max()
,rolling_min()
,rolling_sum()
,rolling_mean()
,rolling_median()
,rolling_std()
,rolling_var()
,rolling_skew()
,rolling_kurt()
,rolling_quantile()
,rolling_cov()
,rolling_corr()
,rolling_corr_pairwise()
,rolling_window()
, androlling_apply()
withcenter=True
previously would return a result of the same structure as the inputarg
withNaN
in the final(window-1)/2
entries.Now the final
(window-1)/2
entries of the result are calculated as if the inputarg
were followed by(window-1)/2
NaN
values (or with shrinking windows, in the case ofrolling_apply()
). (GH7925, GH8269)Prior behavior (note final value is
NaN
):In [7]: pd.rolling_sum(Series(range(4)), window=3, min_periods=0, center=True) Out[7]: 0 1 1 3 2 6 3 NaN dtype: float64
New behavior (note final value is
5 = sum([2, 3, NaN])
):In [7]: pd.rolling_sum(pd.Series(range(4)), window=3, ....: min_periods=0, center=True) Out[7]: 0 1 1 3 2 6 3 5 dtype: float64
rolling_window()
now normalizes the weights properly in rolling mean mode (mean=True) so that the calculated weighted means (e.g. ‘triang’, ‘gaussian’) are distributed about the same means as those calculated without weighting (i.e. ‘boxcar’). See the note on normalization for further details. (GH7618)In [65]: s = pd.Series([10.5, 8.8, 11.4, 9.7, 9.3])
Behavior prior to 0.15.0:
In [39]: pd.rolling_window(s, window=3, win_type='triang', center=True) Out[39]: 0 NaN 1 6.583333 2 6.883333 3 6.683333 4 NaN dtype: float64
New behavior
In [10]: pd.rolling_window(s, window=3, win_type='triang', center=True) Out[10]: 0 NaN 1 9.875 2 10.325 3 10.025 4 NaN dtype: float64
Removed
center
argument from allexpanding_
functions (see list), as the results produced whencenter=True
did not make much sense. (GH7925)Added optional
ddof
argument toexpanding_cov()
androlling_cov()
. The default value of1
is backwards-compatible. (GH8279)Documented the
ddof
argument toexpanding_var()
,expanding_std()
,rolling_var()
, androlling_std()
. These functions’ support of addof
argument (with a default value of1
) was previously undocumented. (GH8064)ewma()
,ewmstd()
,ewmvol()
,ewmvar()
,ewmcov()
, andewmcorr()
now interpretmin_periods
in the same manner that therolling_*()
andexpanding_*()
functions do: a given result entry will beNaN
if the (expanding, in this case) window does not contain at leastmin_periods
values. The previous behavior was to set toNaN
themin_periods
entries starting with the first non-NaN
value. (GH7977)Prior behavior (note values start at index
2
, which ismin_periods
after index0
(the index of the first non-empty value)):In [66]: s = pd.Series([1, None, None, None, 2, 3])
In [51]: ewma(s, com=3., min_periods=2) Out[51]: 0 NaN 1 NaN 2 1.000000 3 1.000000 4 1.571429 5 2.189189 dtype: float64
New behavior (note values start at index
4
, the location of the 2nd (sincemin_periods=2
) non-empty value):In [2]: pd.ewma(s, com=3., min_periods=2) Out[2]: 0 NaN 1 NaN 2 NaN 3 NaN 4 1.759644 5 2.383784 dtype: float64
ewmstd()
,ewmvol()
,ewmvar()
,ewmcov()
, andewmcorr()
now have an optionaladjust
argument, just likeewma()
does, affecting how the weights are calculated. The default value ofadjust
isTrue
, which is backwards-compatible. See Exponentially weighted moment functions for details. (GH7911)ewma()
,ewmstd()
,ewmvol()
,ewmvar()
,ewmcov()
, andewmcorr()
now have an optionalignore_na
argument. Whenignore_na=False
(the default), missing values are taken into account in the weights calculation. Whenignore_na=True
(which reproduces the pre-0.15.0 behavior), missing values are ignored in the weights calculation. (GH7543)In [7]: pd.ewma(pd.Series([None, 1., 8.]), com=2.) Out[7]: 0 NaN 1 1.0 2 5.2 dtype: float64 In [8]: pd.ewma(pd.Series([1., None, 8.]), com=2., ....: ignore_na=True) # pre-0.15.0 behavior Out[8]: 0 1.0 1 1.0 2 5.2 dtype: float64 In [9]: pd.ewma(pd.Series([1., None, 8.]), com=2., ....: ignore_na=False) # new default Out[9]: 0 1.000000 1 1.000000 2 5.846154 dtype: float64
Warning
By default (
ignore_na=False
) theewm*()
functions’ weights calculation in the presence of missing values is different than in pre-0.15.0 versions. To reproduce the pre-0.15.0 calculation of weights in the presence of missing values one must specify explicitlyignore_na=True
.Bug in
expanding_cov()
,expanding_corr()
,rolling_cov()
,rolling_cor()
,ewmcov()
, andewmcorr()
returning results with columns sorted by name and producing an error for non-unique columns; now handles non-unique columns and returns columns in original order (except for the case of two DataFrames withpairwise=False
, where behavior is unchanged) (GH7542)Bug in
rolling_count()
andexpanding_*()
functions unnecessarily producing error message for zero-length data (GH8056)Bug in
rolling_apply()
andexpanding_apply()
interpretingmin_periods=0
asmin_periods=1
(GH8080)Bug in
expanding_std()
andexpanding_var()
for a single value producing a confusing error message (GH7900)Bug in
rolling_std()
androlling_var()
for a single value producing0
rather thanNaN
(GH7900)Bug in
ewmstd()
,ewmvol()
,ewmvar()
, andewmcov()
calculation of de-biasing factors whenbias=False
(the default). Previously an incorrect constant factor was used, based onadjust=True
,ignore_na=True
, and an infinite number of observations. Now a different factor is used for each entry, based on the actual weights (analogous to the usualN/(N-1)
factor). In particular, for a single point a value ofNaN
is returned whenbias=False
, whereas previously a value of (approximately)0
was returned.For example, consider the following pre-0.15.0 results for
ewmvar(..., bias=False)
, and the corresponding debiasing factors:In [67]: s = pd.Series([1., 2., 0., 4.])
In [89]: ewmvar(s, com=2., bias=False) Out[89]: 0 -2.775558e-16 1 3.000000e-01 2 9.556787e-01 3 3.585799e+00 dtype: float64 In [90]: ewmvar(s, com=2., bias=False) / ewmvar(s, com=2., bias=True) Out[90]: 0 1.25 1 1.25 2 1.25 3 1.25 dtype: float64
Note that entry
0
is approximately 0, and the debiasing factors are a constant 1.25. By comparison, the following 0.15.0 results have aNaN
for entry0
, and the debiasing factors are decreasing (towards 1.25):In [14]: pd.ewmvar(s, com=2., bias=False) Out[14]: 0 NaN 1 0.500000 2 1.210526 3 4.089069 dtype: float64 In [15]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True) Out[15]: 0 NaN 1 2.083333 2 1.583333 3 1.425439 dtype: float64
See Exponentially weighted moment functions for details. (GH7912)
Improvements in the SQL IO module¶
Added support for a
chunksize
parameter toto_sql
function. This allows DataFrame to be written in chunks and avoid packet-size overflow errors (GH8062).Added support for a
chunksize
parameter toread_sql
function. Specifying this argument will return an iterator through chunks of the query result (GH2908).Added support for writing
datetime.date
anddatetime.time
object columns withto_sql
(GH6932).Added support for specifying a
schema
to read from/write to withread_sql_table
andto_sql
(GH7441, GH7952). For example:df.to_sql('table', engine, schema='other_schema') # noqa F821 pd.read_sql_table('table', engine, schema='other_schema') # noqa F821
Added support for writing
NaN
values withto_sql
(GH2754).Added support for writing datetime64 columns with
to_sql
for all database flavors (GH7103).
Backwards incompatible API changes¶
Breaking changes¶
API changes related to Categorical
(see here
for more details):
The
Categorical
constructor with two arguments changed from “codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you useCategorical
directly, please audit your code by changing it to use thefrom_codes()
constructor.An old function call like (prior to 0.15.0):
pd.Categorical([0,1,0,2,1], levels=['a', 'b', 'c'])
will have to adapted to the following to keep the same behaviour:
In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c']) Out[2]: [a, b, a, c, b] Categories (3, object): [a, b, c]
API changes related to the introduction of the Timedelta
scalar (see
above for more details):
Prior to 0.15.0
to_timedelta()
would return aSeries
for list-like/Series input, and anp.timedelta64
for scalar input. It will now return aTimedeltaIndex
for list-like input,Series
for Series input, andTimedelta
for scalar input.
For API changes related to the rolling and expanding functions, see detailed overview above.
Other notable API changes:
Consistency when indexing with
.loc
and a list-like indexer when no values are found.In [68]: df = pd.DataFrame([['a'], ['b']], index=[1, 2]) In [69]: df Out[69]: 0 1 a 2 b [2 rows x 1 columns]
In prior versions there was a difference in these two constructs:
df.loc[[3]]
would return a frame reindexed by 3 (with allnp.nan
values)df.loc[[3],:]
would raiseKeyError
.
Both will now raise a
KeyError
. The rule is that at least 1 indexer must be found when using a list-like and.loc
(GH7999)Furthermore in prior versions these were also different:
df.loc[[1,3]]
would return a frame reindexed by [1,3]df.loc[[1,3],:]
would raiseKeyError
.
Both will now return a frame reindex by [1,3]. E.g.
In [3]: df.loc[[1, 3]] Out[3]: 0 1 a 3 NaN In [4]: df.loc[[1, 3], :] Out[4]: 0 1 a 3 NaN
This can also be seen in multi-axis indexing with a
Panel
.>>> p = pd.Panel(np.arange(2 * 3 * 4).reshape(2, 3, 4), ... items=['ItemA', 'ItemB'], ... major_axis=[1, 2, 3], ... minor_axis=['A', 'B', 'C', 'D']) >>> p <class 'pandas.core.panel.Panel'> Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis) Items axis: ItemA to ItemB Major_axis axis: 1 to 3 Minor_axis axis: A to D
The following would raise
KeyError
prior to 0.15.0:In [5]: Out[5]: ItemA ItemD 1 3 NaN 2 7 NaN 3 11 NaN
Furthermore,
.loc
will raise If no values are found in a MultiIndex with a list-like indexer:In [70]: s = pd.Series(np.arange(3, dtype='int64'), ....: index=pd.MultiIndex.from_product([['A'], ....: ['foo', 'bar', 'baz']], ....: names=['one', 'two']) ....: ).sort_index() ....: In [71]: s Out[71]: one two A bar 1 baz 2 foo 0 Length: 3, dtype: int64 In [72]: try: ....: s.loc[['D']] ....: except KeyError as e: ....: print("KeyError: " + str(e)) ....: KeyError: "['D'] not in index"
Assigning values to
None
now considers the dtype when choosing an ‘empty’ value (GH7941).Previously, assigning to
None
in numeric containers changed the dtype to object (or errored, depending on the call). It now usesNaN
:In [73]: s = pd.Series([1, 2, 3]) In [74]: s.loc[0] = None In [75]: s Out[75]: 0 NaN 1 2.0 2 3.0 Length: 3, dtype: float64
NaT
is now used similarly for datetime containers.For object containers, we now preserve
None
values (previously these were converted toNaN
values).In [76]: s = pd.Series(["a", "b", "c"]) In [77]: s.loc[0] = None In [78]: s Out[78]: 0 None 1 b 2 c Length: 3, dtype: object
To insert a
NaN
, you must explicitly usenp.nan
. See the docs.In prior versions, updating a pandas object inplace would not reflect in other python references to this object. (GH8511, GH5104)
In [79]: s = pd.Series([1, 2, 3]) In [80]: s2 = s In [81]: s += 1.5
Behavior prior to v0.15.0
# the original object In [5]: s Out[5]: 0 2.5 1 3.5 2 4.5 dtype: float64 # a reference to the original object In [7]: s2 Out[7]: 0 1 1 2 2 3 dtype: int64
This is now the correct behavior
# the original object In [82]: s Out[82]: 0 2.5 1 3.5 2 4.5 Length: 3, dtype: float64 # a reference to the original object In [83]: s2 Out[83]: 0 2.5 1 3.5 2 4.5 Length: 3, dtype: float64
Made both the C-based and Python engines for
read_csv
andread_table
ignore empty lines in input as well as white space-filled lines, as long assep
is not white space. This is an API change that can be controlled by the keyword parameterskip_blank_lines
. See the docs (GH4466)A timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone and inserted as
object
dtype rather than being converted to a naivedatetime64[ns]
(GH8411).Bug in passing a
DatetimeIndex
with a timezone that was not being retained in DataFrame construction from a dict (GH7822)In prior versions this would drop the timezone, now it retains the timezone, but gives a column of
object
dtype:In [84]: i = pd.date_range('1/1/2011', periods=3, freq='10s', tz='US/Eastern') In [85]: i Out[85]: DatetimeIndex(['2011-01-01 00:00:00-05:00', '2011-01-01 00:00:10-05:00', '2011-01-01 00:00:20-05:00'], dtype='datetime64[ns, US/Eastern]', freq='10S') In [86]: df = pd.DataFrame({'a': i}) In [87]: df Out[87]: a 0 2011-01-01 00:00:00-05:00 1 2011-01-01 00:00:10-05:00 2 2011-01-01 00:00:20-05:00 [3 rows x 1 columns] In [88]: df.dtypes Out[88]: a datetime64[ns, US/Eastern] Length: 1, dtype: object
Previously this would have yielded a column of
datetime64
dtype, but without timezone info.The behaviour of assigning a column to an existing dataframe as
df['a'] = i
remains unchanged (this already returned anobject
column with a timezone).When passing multiple levels to
stack()
, it will now raise aValueError
when the levels aren’t all level names or all level numbers (GH7660). See Reshaping by stacking and unstacking.Raise a
ValueError
indf.to_hdf
with ‘fixed’ format, ifdf
has non-unique columns as the resulting file will be broken (GH7761)SettingWithCopy
raise/warnings (according to the optionmode.chained_assignment
) will now be issued when setting a value on a sliced mixed-dtype DataFrame using chained-assignment. (GH7845, GH7950)In [1]: df = pd.DataFrame(np.arange(0, 9), columns=['count']) In [2]: df['group'] = 'b' In [3]: df.iloc[0:5]['group'] = 'a' /usr/local/bin/ipython:1: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
merge
,DataFrame.merge
, andordered_merge
now return the same type as theleft
argument (GH7737).Previously an enlargement with a mixed-dtype frame would act unlike
.append
which will preserve dtypes (related GH2578, GH8176):In [89]: df = pd.DataFrame([[True, 1], [False, 2]], ....: columns=["female", "fitness"]) ....: In [90]: df Out[90]: female fitness 0 True 1 1 False 2 [2 rows x 2 columns] In [91]: df.dtypes Out[91]: female bool fitness int64 Length: 2, dtype: object # dtypes are now preserved In [92]: df.loc[2] = df.loc[1] In [93]: df Out[93]: female fitness 0 True 1 1 False 2 2 False 2 [3 rows x 2 columns] In [94]: df.dtypes Out[94]: female bool fitness int64 Length: 2, dtype: object
Series.to_csv()
now returns a string whenpath=None
, matching the behaviour ofDataFrame.to_csv()
(GH8215).read_hdf
now raisesIOError
when a file that doesn’t exist is passed in. Previously, a new, empty file was created, and aKeyError
raised (GH7715).DataFrame.info()
now ends its output with a newline character (GH8114)Concatenating no objects will now raise a
ValueError
rather than a bareException
.Merge errors will now be sub-classes of
ValueError
rather than rawException
(GH8501)DataFrame.plot
andSeries.plot
keywords are now have consistent orders (GH8037)
Internal refactoring¶
In 0.15.0 Index
has internally been refactored to no longer sub-class ndarray
but instead subclass PandasObject
, similarly to the rest of the pandas objects. This
change allows very easy sub-classing and creation of new index types. This should be
a transparent change with only very limited API implications (GH5080, GH7439, GH7796, GH8024, GH8367, GH7997, GH8522):
you may need to unpickle pandas version < 0.15.0 pickles using
pd.read_pickle
rather thanpickle.load
. See pickle docswhen plotting with a
PeriodIndex
, the matplotlib internal axes will now be arrays ofPeriod
rather than aPeriodIndex
(this is similar to how aDatetimeIndex
passes arrays ofdatetimes
now)MultiIndexes will now raise similarly to other pandas objects w.r.t. truth testing, see here (GH7897).
When plotting a DatetimeIndex directly with matplotlib’s
plot
function, the axis labels will no longer be formatted as dates but as integers (the internal representation of adatetime64
). UPDATE This is fixed in 0.15.1, see here.
Deprecations¶
The attributes
Categorical
labels
andlevels
attributes are deprecated and renamed tocodes
andcategories
.The
outtype
argument topd.DataFrame.to_dict
has been deprecated in favor oforient
. (GH7840)The
convert_dummies
method has been deprecated in favor ofget_dummies
(GH8140)The
infer_dst
argument intz_localize
will be deprecated in favor ofambiguous
to allow for more flexibility in dealing with DST transitions. Replaceinfer_dst=True
withambiguous='infer'
for the same behavior (GH7943). See the docs for more details.The top-level
pd.value_range
has been deprecated and can be replaced by.describe()
(GH8481)
The
Index
set operations+
and-
were deprecated in order to provide these for numeric type operations on certain index types.+
can be replaced by.union()
or|
, and-
by.difference()
. Further the method nameIndex.diff()
is deprecated and can be replaced byIndex.difference()
(GH8226)# + pd.Index(['a', 'b', 'c']) + pd.Index(['b', 'c', 'd']) # should be replaced by pd.Index(['a', 'b', 'c']).union(pd.Index(['b', 'c', 'd']))
# - pd.Index(['a', 'b', 'c']) - pd.Index(['b', 'c', 'd']) # should be replaced by pd.Index(['a', 'b', 'c']).difference(pd.Index(['b', 'c', 'd']))
The
infer_types
argument toread_html()
now has no effect and is deprecated (GH7762, GH7032).
Removal of prior version deprecations/changes¶
Remove
DataFrame.delevel
method in favor ofDataFrame.reset_index
Enhancements¶
Enhancements in the importing/exporting of Stata files:
Added support for bool, uint8, uint16 and uint32 data types in
to_stata
(GH7097, GH7365)Added conversion option when importing Stata files (GH8527)
DataFrame.to_stata
andStataWriter
check string length for compatibility with limitations imposed in dta files where fixed-width strings must contain 244 or fewer characters. Attempting to write Stata dta files with strings longer than 244 characters raises aValueError
. (GH7858)read_stata
andStataReader
can import missing data information into aDataFrame
by setting the argumentconvert_missing
toTrue
. When using this options, missing values are returned asStataMissingValue
objects and columns containing missing values haveobject
data type. (GH8045)
Enhancements in the plotting functions:
Added
layout
keyword toDataFrame.plot
. You can pass a tuple of(rows, columns)
, one of which can be-1
to automatically infer (GH6667, GH8071).Allow to pass multiple axes to
DataFrame.plot
,hist
andboxplot
(GH5353, GH6970, GH7069)Added support for
c
,colormap
andcolorbar
arguments forDataFrame.plot
withkind='scatter'
(GH7780)Histogram from
DataFrame.plot
withkind='hist'
(GH7809), See the docs.Boxplot from
DataFrame.plot
withkind='box'
(GH7998), See the docs.
Other:
read_csv
now has a keyword parameterfloat_precision
which specifies which floating-point converter the C engine should use during parsing, see here (GH8002, GH8044)Added
searchsorted
method toSeries
objects (GH7447)describe()
on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via theinclude
/exclude
arguments. See the docs (GH8164).In [95]: df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8, ....: 'catB': ['a', 'b', 'c', 'd'] * 6, ....: 'numC': np.arange(24), ....: 'numD': np.arange(24.) + .5}) ....: In [96]: df.describe(include=["object"]) Out[96]: catA catB count 24 24 unique 2 4 top foo a freq 16 6 [4 rows x 2 columns] In [97]: df.describe(include=["number", "object"], exclude=["float"]) Out[97]: catA catB numC count 24 24 24.000000 unique 2 4 NaN top foo a NaN freq 16 6 NaN mean NaN NaN 11.500000 std NaN NaN 7.071068 min NaN NaN 0.000000 25% NaN NaN 5.750000 50% NaN NaN 11.500000 75% NaN NaN 17.250000 max NaN NaN 23.000000 [11 rows x 3 columns]
Requesting all columns is possible with the shorthand ‘all’
In [98]: df.describe(include='all') Out[98]: catA catB numC numD count 24 24 24.000000 24.000000 unique 2 4 NaN NaN top foo a NaN NaN freq 16 6 NaN NaN mean NaN NaN 11.500000 12.000000 std NaN NaN 7.071068 7.071068 min NaN NaN 0.000000 0.500000 25% NaN NaN 5.750000 6.250000 50% NaN NaN 11.500000 12.000000 75% NaN NaN 17.250000 17.750000 max NaN NaN 23.000000 23.500000 [11 rows x 4 columns]
Without those arguments,
describe
will behave as before, including only numerical columns or, if none are, only categorical columns. See also the docsAdded
split
as an option to theorient
argument inpd.DataFrame.to_dict
. (GH7840)The
get_dummies
method can now be used on DataFrames. By default only categorical columns are encoded as 0’s and 1’s, while other columns are left untouched.In [99]: df = pd.DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'], ....: 'C': [1, 2, 3]}) ....: In [100]: pd.get_dummies(df) Out[100]: C A_a A_b B_b B_c 0 1 1 0 0 1 1 2 0 1 0 1 2 3 1 0 1 0 [3 rows x 5 columns]
PeriodIndex
supportsresolution
as the same asDatetimeIndex
(GH7708)pandas.tseries.holiday
has added support for additional holidays and ways to observe holidays (GH7070)pandas.tseries.holiday.Holiday
now supports a list of offsets in Python3 (GH7070)pandas.tseries.holiday.Holiday
now supports a days_of_week parameter (GH7070)GroupBy.nth()
now supports selecting multiple nth values (GH7910)In [101]: business_dates = pd.date_range(start='4/1/2014', end='6/30/2014', freq='B') In [102]: df = pd.DataFrame(1, index=business_dates, columns=['a', 'b']) # get the first, 4th, and last date index for each month In [103]: df.groupby([df.index.year, df.index.month]).nth([0, 3, -1]) Out[103]: a b 2014 4 1 1 4 1 1 4 1 1 5 1 1 5 1 1 5 1 1 6 1 1 6 1 1 6 1 1 [9 rows x 2 columns]
Period
andPeriodIndex
supports addition/subtraction withtimedelta
-likes (GH7966)If
Period
freq isD
,H
,T
,S
,L
,U
,N
,Timedelta
-like can be added if the result can have same freq. Otherwise, only the sameoffsets
can be added.In [104]: idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H') In [105]: idx Out[105]: PeriodIndex(['2014-07-01 09:00', '2014-07-01 10:00', '2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00'], dtype='period[H]') In [106]: idx + pd.offsets.Hour(2) Out[106]: PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00', '2014-07-01 14:00', '2014-07-01 15:00'], dtype='period[H]') In [107]: idx + pd.Timedelta('120m') Out[107]: PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00', '2014-07-01 14:00', '2014-07-01 15:00'], dtype='period[H]') In [108]: idx = pd.period_range('2014-07', periods=5, freq='M') In [109]: idx Out[109]: PeriodIndex(['2014-07', '2014-08', '2014-09', '2014-10', '2014-11'], dtype='period[M]') In [110]: idx + pd.offsets.MonthEnd(3) Out[110]: PeriodIndex(['2014-10', '2014-11', '2014-12', '2015-01', '2015-02'], dtype='period[M]')
Added experimental compatibility with
openpyxl
for versions >= 2.0. TheDataFrame.to_excel
methodengine
keyword now recognizesopenpyxl1
andopenpyxl2
which will explicitly require openpyxl v1 and v2 respectively, failing if the requested version is not available. Theopenpyxl
engine is a now a meta-engine that automatically uses whichever version of openpyxl is installed. (GH7177)DataFrame.fillna
can now accept aDataFrame
as a fill value (GH8377)Passing multiple levels to
stack()
will now work when multiple level numbers are passed (GH7660). See Reshaping by stacking and unstacking.set_names()
,set_labels()
, andset_levels()
methods now take an optionallevel
keyword argument to all modification of specific level(s) of a MultiIndex. Additionallyset_names()
now accepts a scalar string value when operating on anIndex
or on a specific level of aMultiIndex
(GH7792)In [111]: idx = pd.MultiIndex.from_product([['a'], range(3), list("pqr")], .....: names=['foo', 'bar', 'baz']) .....: In [112]: idx.set_names('qux', level=0) Out[112]: MultiIndex([('a', 0, 'p'), ('a', 0, 'q'), ('a', 0, 'r'), ('a', 1, 'p'), ('a', 1, 'q'), ('a', 1, 'r'), ('a', 2, 'p'), ('a', 2, 'q'), ('a', 2, 'r')], names=['qux', 'bar', 'baz']) In [113]: idx.set_names(['qux', 'corge'], level=[0, 1]) Out[113]: MultiIndex([('a', 0, 'p'), ('a', 0, 'q'), ('a', 0, 'r'), ('a', 1, 'p'), ('a', 1, 'q'), ('a', 1, 'r'), ('a', 2, 'p'), ('a', 2, 'q'), ('a', 2, 'r')], names=['qux', 'corge', 'baz']) In [114]: idx.set_levels(['a', 'b', 'c'], level='bar') Out[114]: MultiIndex([('a', 'a', 'p'), ('a', 'a', 'q'), ('a', 'a', 'r'), ('a', 'b', 'p'), ('a', 'b', 'q'), ('a', 'b', 'r'), ('a', 'c', 'p'), ('a', 'c', 'q'), ('a', 'c', 'r')], names=['foo', 'bar', 'baz']) In [115]: idx.set_levels([['a', 'b', 'c'], [1, 2, 3]], level=[1, 2]) Out[115]: MultiIndex([('a', 'a', 1), ('a', 'a', 2), ('a', 'a', 3), ('a', 'b', 1), ('a', 'b', 2), ('a', 'b', 3), ('a', 'c', 1), ('a', 'c', 2), ('a', 'c', 3)], names=['foo', 'bar', 'baz'])
Index.isin
now supports alevel
argument to specify which index level to use for membership tests (GH7892, GH7890)In [1]: idx = pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']]) In [2]: idx.values Out[2]: array([(0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), (1, 'c')], dtype=object) In [3]: idx.isin(['a', 'c', 'e'], level=1) Out[3]: array([ True, False, True, True, False, True], dtype=bool)
Index
now supportsduplicated
anddrop_duplicates
. (GH4060)In [116]: idx = pd.Index([1, 2, 3, 4, 1, 2]) In [117]: idx Out[117]: Int64Index([1, 2, 3, 4, 1, 2], dtype='int64') In [118]: idx.duplicated() Out[118]: array([False, False, False, False, True, True]) In [119]: idx.drop_duplicates() Out[119]: Int64Index([1, 2, 3, 4], dtype='int64')
add
copy=True
argument topd.concat
to enable pass through of complete blocks (GH8252)Added support for numpy 1.8+ data types (
bool_
,int_
,float_
,string_
) for conversion to R dataframe (GH8400)
Performance¶
Performance improvements in
DatetimeIndex.__iter__
to allow faster iteration (GH7683)Performance improvements in
Period
creation (andPeriodIndex
setitem) (GH5155)Improvements in Series.transform for significant performance gains (revised) (GH6496)
Performance improvements in
StataReader
when reading large files (GH8040, GH8073)Performance improvements in
StataWriter
when writing large files (GH8079)Performance and memory usage improvements in multi-key
groupby
(GH8128)Performance improvements in groupby
.agg
and.apply
where builtins max/min were not mapped to numpy/cythonized versions (GH7722)Performance improvement in writing to sql (
to_sql
) of up to 50% (GH8208).Performance benchmarking of groupby for large value of ngroups (GH6787)
Performance improvement in
CustomBusinessDay
,CustomBusinessMonth
(GH8236)Performance improvement for
MultiIndex.values
for multi-level indexes containing datetimes (GH8543)
Bug fixes¶
Bug in pivot_table, when using margins and a dict aggfunc (GH8349)
Bug in
read_csv
wheresqueeze=True
would return a view (GH8217)Bug in checking of table name in
read_sql
in certain cases (GH7826).Bug in
DataFrame.groupby
whereGrouper
does not recognize level when frequency is specified (GH7885)Bug in multiindexes dtypes getting mixed up when DataFrame is saved to SQL table (GH8021)
Bug in
Series
0-division with a float and integer operand dtypes (GH7785)Bug in
Series.astype("unicode")
not callingunicode
on the values correctly (GH7758)Bug in
DataFrame.as_matrix()
with mixeddatetime64[ns]
andtimedelta64[ns]
dtypes (GH7778)Bug in
HDFStore.select_column()
not preserving UTC timezone info when selecting aDatetimeIndex
(GH7777)Bug in
to_datetime
whenformat='%Y%m%d'
andcoerce=True
are specified, where previously an object array was returned (rather than a coerced time-series withNaT
), (GH7930)Bug in
DatetimeIndex
andPeriodIndex
in-place addition and subtraction cause different result from normal one (GH6527)Bug in adding and subtracting
PeriodIndex
withPeriodIndex
raiseTypeError
(GH7741)Bug in
combine_first
withPeriodIndex
data raisesTypeError
(GH3367)Bug in MultiIndex slicing with missing indexers (GH7866)
Bug in MultiIndex slicing with various edge cases (GH8132)
Regression in MultiIndex indexing with a non-scalar type object (GH7914)
Bug in
Timestamp
comparisons with==
andint64
dtype (GH8058)Bug in pickles contains
DateOffset
may raiseAttributeError
whennormalize
attribute is referred internally (GH7748)Bug in
Panel
when usingmajor_xs
andcopy=False
is passed (deprecation warning fails because of missingwarnings
) (GH8152).Bug in pickle deserialization that failed for pre-0.14.1 containers with dup items trying to avoid ambiguity when matching block and manager items, when there’s only one block there’s no ambiguity (GH7794)
Bug in putting a
PeriodIndex
into aSeries
would convert toint64
dtype, rather thanobject
ofPeriods
(GH7932)Bug in
HDFStore
iteration when passing a where (GH8014)Bug in
DataFrameGroupby.transform
when transforming with a passed non-sorted key (GH8046, GH8430)Bug in repeated timeseries line and area plot may result in
ValueError
or incorrect kind (GH7733)Bug in inference in a
MultiIndex
withdatetime.date
inputs (GH7888)Bug in
get
where anIndexError
would not cause the default value to be returned (GH7725)Bug in
offsets.apply
,rollforward
androllback
may reset nanosecond (GH7697)Bug in
offsets.apply
,rollforward
androllback
may raiseAttributeError
ifTimestamp
hasdateutil
tzinfo (GH7697)Bug in sorting a MultiIndex frame with a
Float64Index
(GH8017)Bug in inconsistent panel setitem with a rhs of a
DataFrame
for alignment (GH7763)Bug in
is_superperiod
andis_subperiod
cannot handle higher frequencies thanS
(GH7760, GH7772, GH7803)Bug in 32-bit platforms with
Series.shift
(GH8129)Bug in
PeriodIndex.unique
returns int64np.ndarray
(GH7540)Bug in
groupby.apply
with a non-affecting mutation in the function (GH8467)Bug in
DataFrame.reset_index
which hasMultiIndex
containsPeriodIndex
orDatetimeIndex
with tz raisesValueError
(GH7746, GH7793)Bug in
DataFrame.plot
withsubplots=True
may draw unnecessary minor xticks and yticks (GH7801)Bug in
StataReader
which did not read variable labels in 117 files due to difference between Stata documentation and implementation (GH7816)Bug in
StataReader
where strings were always converted to 244 characters-fixed width irrespective of underlying string size (GH7858)Bug in
DataFrame.plot
andSeries.plot
may ignorerot
andfontsize
keywords (GH7844)Bug in
DatetimeIndex.value_counts
doesn’t preserve tz (GH7735)Bug in
PeriodIndex.value_counts
results inInt64Index
(GH7735)Bug in
DataFrame.join
when doing left join on index and there are multiple matches (GH5391)Bug in
GroupBy.transform()
where int groups with a transform that didn’t preserve the index were incorrectly truncated (GH7972).Bug in
groupby
where callable objects without name attributes would take the wrong path, and produce aDataFrame
instead of aSeries
(GH7929)Bug in
groupby
error message when a DataFrame grouping column is duplicated (GH7511)Bug in
read_html
where theinfer_types
argument forced coercion of date-likes incorrectly (GH7762, GH7032).Bug in
Series.str.cat
with an index which was filtered as to not include the first item (GH7857)Bug in
Timestamp
cannot parsenanosecond
from string (GH7878)Bug in
Timestamp
with string offset andtz
results incorrect (GH7833)Bug in
tslib.tz_convert
andtslib.tz_convert_single
may return different results (GH7798)Bug in
DatetimeIndex.intersection
of non-overlapping timestamps with tz raisesIndexError
(GH7880)Bug in alignment with TimeOps and non-unique indexes (GH8363)
Bug in
GroupBy.filter()
where fast path vs. slow path made the filter return a non scalar value that appeared valid but wasn’t (GH7870).Bug in
date_range()
/DatetimeIndex()
when the timezone was inferred from input dates yet incorrect times were returned when crossing DST boundaries (GH7835, GH7901).Bug in
to_excel()
where a negative sign was being prepended to positive infinity and was absent for negative infinity (GH7949)Bug in area plot draws legend with incorrect
alpha
whenstacked=True
(GH8027)Period
andPeriodIndex
addition/subtraction withnp.timedelta64
results in incorrect internal representations (GH7740)Bug in
Holiday
with no offset or observance (GH7987)Bug in
DataFrame.to_latex
formatting when columns or index is aMultiIndex
(GH7982).Bug in
DateOffset
around Daylight Savings Time produces unexpected results (GH5175).Bug in
DataFrame.shift
where empty columns would throwZeroDivisionError
on numpy 1.7 (GH8019)Bug in installation where
html_encoding/*.html
wasn’t installed and therefore some tests were not running correctly (GH7927).Bug in
read_html
wherebytes
objects were not tested for in_read
(GH7927).Bug in
DataFrame.stack()
when one of the column levels was a datelike (GH8039)Bug in broadcasting numpy scalars with
DataFrame
(GH8116)Bug in
pivot_table
performed with namelessindex
andcolumns
raisesKeyError
(GH8103)Bug in
DataFrame.plot(kind='scatter')
draws points and errorbars with different colors when the color is specified byc
keyword (GH8081)Bug in
Float64Index
whereiat
andat
were not testing and were failing (GH8092).Bug in
DataFrame.boxplot()
where y-limits were not set correctly when producing multiple axes (GH7528, GH5517).Bug in
read_csv
where line comments were not handled correctly given a custom line terminator ordelim_whitespace=True
(GH8122).Bug in
read_html
where empty tables caused aStopIteration
(GH7575)Bug in casting when setting a column in a same-dtype block (GH7704)
Bug in accessing groups from a
GroupBy
when the original grouper was a tuple (GH8121).Bug in
.at
that would accept integer indexers on a non-integer index and do fallback (GH7814)Bug with kde plot and NaNs (GH8182)
Bug in
GroupBy.count
with float32 data type were nan values were not excluded (GH8169).Bug with stacked barplots and NaNs (GH8175).
Bug in resample with non evenly divisible offsets (e.g. ‘7s’) (GH8371)
Bug in interpolation methods with the
limit
keyword when no values needed interpolating (GH7173).Bug where
col_space
was ignored inDataFrame.to_string()
whenheader=False
(GH8230).Bug with
DatetimeIndex.asof
incorrectly matching partial strings and returning the wrong date (GH8245).Bug in plotting methods modifying the global matplotlib rcParams (GH8242).
Bug in
DataFrame.__setitem__
that caused errors when setting a dataframe column to a sparse array (GH8131)Bug where
Dataframe.boxplot()
failed when entire column was empty (GH8181).Bug with messed variables in
radviz
visualization (GH8199).Bug in interpolation methods with the
limit
keyword when no values needed interpolating (GH7173).Bug where
col_space
was ignored inDataFrame.to_string()
whenheader=False
(GH8230).Bug in
to_clipboard
that would clip long column data (GH8305)Bug in
DataFrame
terminal display: Setting max_column/max_rows to zero did not trigger auto-resizing of dfs to fit terminal width/height (GH7180).Bug in OLS where running with “cluster” and “nw_lags” parameters did not work correctly, but also did not throw an error (GH5884).
Bug in
DataFrame.dropna
that interpreted non-existent columns in the subset argument as the ‘last column’ (GH8303)Bug in
Index.intersection
on non-monotonic non-unique indexes (GH8362).Bug in masked series assignment where mismatching types would break alignment (GH8387)
Bug in
NDFrame.equals
gives false negatives with dtype=object (GH8437)Bug in assignment with indexer where type diversity would break alignment (GH8258)
Bug in
NDFrame.loc
indexing when row/column names were lost when target was a list/ndarray (GH6552)Regression in
NDFrame.loc
indexing when rows/columns were converted to Float64Index if target was an empty list/ndarray (GH7774)Bug in
Series
that allows it to be indexed by aDataFrame
which has unexpected results. Such indexing is no longer permitted (GH8444)Bug in item assignment of a
DataFrame
with MultiIndex columns where right-hand-side columns were not aligned (GH7655)Suppress FutureWarning generated by NumPy when comparing object arrays containing NaN for equality (GH7065)
Bug in
DataFrame.eval()
where the dtype of thenot
operator (~
) was not correctly inferred asbool
.
Contributors¶
A total of 80 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.
Aaron Schumacher +
Adam Greenhall
Andy Hayden
Anthony O’Brien +
Artemy Kolchinsky +
Ben Schiller +
Benedikt Sauer
Benjamin Thyreau +
BorisVerk +
Chris Reynolds +
Chris Stoafer +
DSM
Dav Clark +
FragLegs +
German Gomez-Herrero +
Hsiaoming Yang +
Huan Li +
Hyungtae Kim +
Isaac Slavitt +
Jacob Schaer
Jacob Wasserman +
Jan Schulz
Jeff Reback
Jeff Tratner
Jesse Farnham +
Joe Bradish +
Joerg Rittinger +
John W. O’Brien
Joris Van den Bossche
Kevin Sheppard
Kyle Meyer
Max Chang +
Michael Mueller
Michael W Schatzow +
Mike Kelly
Mortada Mehyar
Nathan Sanders +
Nathan Typanski +
Paul Masurel +
Phillip Cloud
Pietro Battiston
RenzoBertocchi +
Ross Petchler +
Shahul Hameed +
Shashank Agarwal +
Stephan Hoyer
Tom Augspurger
TomAugspurger
Tony Lorenzo +
Wes Turner
Wilfred Hughes +
Yevgeniy Grechka +
Yoshiki Vázquez Baeza +
behzad nouri +
benjamin
bjonen +
dlovell +
dsm054
hunterowens +
immerrr
ischwabacher
jmorris0x0 +
jnmclarty +
jreback
klonuo +
lexual
mcjcode +
mtrbean +
onesandzeroes
rockg
seth-p
sinhrks
someben +
stahlous +
stas-sl +
thatneat +
tom-alcorn +
unknown
unutbu
zachcp +