Version 0.15.0 (October 18, 2014)#

This is a major release from 0.14.1 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Warning

pandas >= 0.15.0 will no longer support compatibility with NumPy versions < 1.7.0. If you want to use the latest versions of pandas, please upgrade to NumPy >= 1.7.0 (GH 7711)

Warning

In 0.15.0 Index has internally been refactored to no longer sub-class ndarray but instead subclass PandasObject, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be a transparent change with only very limited API implications (See the Internal Refactoring)

Warning

The refactoring in Categorical changed the two argument constructor from “codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use Categorical directly, please audit your code before updating to this pandas version and change it to use the from_codes() constructor. See more on Categorical here

New features#

Categoricals in Series/DataFrame#

Categorical can now be included in Series and DataFrames and gained new methods to manipulate. Thanks to Jan Schulz for much of this API/implementation. (GH 3943, GH 5313, GH 5314, GH 7444, GH 7839, GH 7848, GH 7864, GH 7914, GH 7768, GH 8006, GH 3678, GH 8075, GH 8076, GH 8143, GH 8453, GH 8518).

For full docs, see the categorical introduction and the API documentation.

In [1]: df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
   ...:                    "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})
   ...: 

In [2]: df["grade"] = df["raw_grade"].astype("category")

In [3]: df["grade"]
Out[3]: 
0    a
1    b
2    b
3    a
4    a
5    e
Name: grade, Length: 6, dtype: category
Categories (3, object): ['a', 'b', 'e']

# Rename the categories
In [4]: df["grade"] = df["grade"].cat.rename_categories(["very good", "good", "very bad"])

# Reorder the categories and simultaneously add the missing categories
In [5]: df["grade"] = df["grade"].cat.set_categories(["very bad", "bad",
   ...:                                               "medium", "good", "very good"])
   ...: 

In [6]: df["grade"]
Out[6]: 
0    very good
1         good
2         good
3    very good
4    very good
5     very bad
Name: grade, Length: 6, dtype: category
Categories (5, object): ['very bad', 'bad', 'medium', 'good', 'very good']

In [7]: df.sort_values("grade")
Out[7]: 
   id raw_grade      grade
5   6         e   very bad
1   2         b       good
2   3         b       good
0   1         a  very good
3   4         a  very good
4   5         a  very good

[6 rows x 3 columns]

In [8]: df.groupby("grade", observed=False).size()
Out[8]: 
grade
very bad     1
bad          0
medium       0
good         2
very good    3
Length: 5, dtype: int64

  • pandas.core.group_agg and pandas.core.factor_agg were removed. As an alternative, construct a dataframe and use df.groupby(<group>).agg(<func>).

  • Supplying “codes/labels and levels” to the Categorical constructor is not supported anymore. Supplying two arguments to the constructor is now interpreted as “values and levels (now called ‘categories’)”. Please change your code to use the from_codes() constructor.

  • The Categorical.labels attribute was renamed to Categorical.codes and is read only. If you want to manipulate codes, please use one of the API methods on Categoricals.

  • The Categorical.levels attribute is renamed to Categorical.categories.

TimedeltaIndex/scalar#

We introduce a new scalar type Timedelta, which is a subclass of datetime.timedelta, and behaves in a similar manner, but allows compatibility with np.timedelta64 types as well as a host of custom representation, parsing, and attributes. This type is very similar to how Timestamp works for datetimes. It is a nice-API box for the type. See the docs. (GH 3009, GH 4533, GH 8209, GH 8187, GH 8190, GH 7869, GH 7661, GH 8345, GH 8471)

Warning

Timedelta scalars (and TimedeltaIndex) component fields are not the same as the component fields on a datetime.timedelta object. For example, .seconds on a datetime.timedelta object returns the total number of seconds combined between hours, minutes and seconds. In contrast, the pandas Timedelta breaks out hours, minutes, microseconds and nanoseconds separately.

# Timedelta accessor
In [9]: tds = pd.Timedelta('31 days 5 min 3 sec')

In [10]: tds.minutes
Out[10]: 5L

In [11]: tds.seconds
Out[11]: 3L

# datetime.timedelta accessor
# this is 5 minutes * 60 + 3 seconds
In [12]: tds.to_pytimedelta().seconds
Out[12]: 303

Note: this is no longer true starting from v0.16.0, where full compatibility with datetime.timedelta is introduced. See the 0.16.0 whatsnew entry

Warning

Prior to 0.15.0 pd.to_timedelta would return a Series for list-like/Series input, and a np.timedelta64 for scalar input. It will now return a TimedeltaIndex for list-like input, Series for Series input, and Timedelta for scalar input.

The arguments to pd.to_timedelta are now (arg,unit='ns',box=True,coerce=False), previously were (arg,box=True,unit='ns') as these are more logical.

Construct a scalar

In [9]: pd.Timedelta('1 days 06:05:01.00003')
Out[9]: Timedelta('1 days 06:05:01.000030')

In [10]: pd.Timedelta('15.5us')
Out[10]: Timedelta('0 days 00:00:00.000015500')

In [11]: pd.Timedelta('1 hour 15.5us')
Out[11]: Timedelta('0 days 01:00:00.000015500')

# negative Timedeltas have this string repr
# to be more consistent with datetime.timedelta conventions
In [12]: pd.Timedelta('-1us')
Out[12]: Timedelta('-1 days +23:59:59.999999')

# a NaT
In [13]: pd.Timedelta('nan')
Out[13]: NaT

Access fields for a Timedelta

In [14]: td = pd.Timedelta('1 hour 3m 15.5us')

In [15]: td.seconds
Out[15]: 3780

In [16]: td.microseconds
Out[16]: 15

In [17]: td.nanoseconds
Out[17]: 500

Construct a TimedeltaIndex

In [18]: pd.TimedeltaIndex(['1 days', '1 days, 00:00:05',
   ....:                    np.timedelta64(2, 'D'),
   ....:                    datetime.timedelta(days=2, seconds=2)])
   ....: 
Out[18]: 
TimedeltaIndex(['1 days 00:00:00', '1 days 00:00:05', '2 days 00:00:00',
                '2 days 00:00:02'],
               dtype='timedelta64[ns]', freq=None)

Constructing a TimedeltaIndex with a regular range

In [19]: pd.timedelta_range('1 days', periods=5, freq='D')
Out[19]: TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq='D')

In [20]: pd.timedelta_range(start='1 days', end='2 days', freq='30T')
Out[20]: 
TimedeltaIndex(['1 days 00:00:00', '1 days 00:30:00', '1 days 01:00:00',
                '1 days 01:30:00', '1 days 02:00:00', '1 days 02:30:00',
                '1 days 03:00:00', '1 days 03:30:00', '1 days 04:00:00',
                '1 days 04:30:00', '1 days 05:00:00', '1 days 05:30:00',
                '1 days 06:00:00', '1 days 06:30:00', '1 days 07:00:00',
                '1 days 07:30:00', '1 days 08:00:00', '1 days 08:30:00',
                '1 days 09:00:00', '1 days 09:30:00', '1 days 10:00:00',
                '1 days 10:30:00', '1 days 11:00:00', '1 days 11:30:00',
                '1 days 12:00:00', '1 days 12:30:00', '1 days 13:00:00',
                '1 days 13:30:00', '1 days 14:00:00', '1 days 14:30:00',
                '1 days 15:00:00', '1 days 15:30:00', '1 days 16:00:00',
                '1 days 16:30:00', '1 days 17:00:00', '1 days 17:30:00',
                '1 days 18:00:00', '1 days 18:30:00', '1 days 19:00:00',
                '1 days 19:30:00', '1 days 20:00:00', '1 days 20:30:00',
                '1 days 21:00:00', '1 days 21:30:00', '1 days 22:00:00',
                '1 days 22:30:00', '1 days 23:00:00', '1 days 23:30:00',
                '2 days 00:00:00'],
               dtype='timedelta64[ns]', freq='30T')

You can now use a TimedeltaIndex as the index of a pandas object

In [21]: s = pd.Series(np.arange(5),
   ....:               index=pd.timedelta_range('1 days', periods=5, freq='s'))
   ....: 

In [22]: s
Out[22]: 
1 days 00:00:00    0
1 days 00:00:01    1
1 days 00:00:02    2
1 days 00:00:03    3
1 days 00:00:04    4
Freq: S, Length: 5, dtype: int64

You can select with partial string selections

In [23]: s['1 day 00:00:02']
Out[23]: 2

In [24]: s['1 day':'1 day 00:00:02']
Out[24]: 
1 days 00:00:00    0
1 days 00:00:01    1
1 days 00:00:02    2
Freq: S, Length: 3, dtype: int64

Finally, the combination of TimedeltaIndex with DatetimeIndex allow certain combination operations that are NaT preserving:

In [25]: tdi = pd.TimedeltaIndex(['1 days', pd.NaT, '2 days'])

In [26]: tdi.tolist()
Out[26]: [Timedelta('1 days 00:00:00'), NaT, Timedelta('2 days 00:00:00')]

In [27]: dti = pd.date_range('20130101', periods=3)

In [28]: dti.tolist()
Out[28]: 
[Timestamp('2013-01-01 00:00:00'),
 Timestamp('2013-01-02 00:00:00'),
 Timestamp('2013-01-03 00:00:00')]

In [29]: (dti + tdi).tolist()
Out[29]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]

In [30]: (dti - tdi).tolist()
Out[30]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]

  • iteration of a Series e.g. list(Series(...)) of timedelta64[ns] would prior to v0.15.0 return np.timedelta64 for each element. These will now be wrapped in Timedelta.

Memory usage#

Implemented methods to find memory usage of a DataFrame. See the FAQ for more. (GH 6852).

A new display option display.memory_usage (see Options and settings) sets the default behavior of the memory_usage argument in the df.info() method. By default display.memory_usage is True.

In [31]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
   ....:           'complex128', 'object', 'bool']
   ....: 

In [32]: n = 5000

In [33]: data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}

In [34]: df = pd.DataFrame(data)

In [35]: df['categorical'] = df['object'].astype('category')

In [36]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
 #   Column           Non-Null Count  Dtype          
---  ------           --------------  -----          
 0   int64            5000 non-null   int64          
 1   float64          5000 non-null   float64        
 2   datetime64[ns]   5000 non-null   datetime64[ns] 
 3   timedelta64[ns]  5000 non-null   timedelta64[ns]
 4   complex128       5000 non-null   complex128     
 5   object           5000 non-null   object         
 6   bool             5000 non-null   bool           
 7   categorical      5000 non-null   category       
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 288.2+ KB

Additionally memory_usage() is an available method for a dataframe object which returns the memory usage of each column.

In [37]: df.memory_usage(index=True)
Out[37]: 
Index                128
int64              40000
float64            40000
datetime64[ns]     40000
timedelta64[ns]    40000
complex128         80000
object             40000
bool                5000
categorical         9968
Length: 9, dtype: int64

Series.dt accessor#

Series has gained an accessor to succinctly return datetime like properties for the values of the Series, if its a datetime/period like Series. (GH 7207) This will return a Series, indexed like the existing Series. See the docs

# datetime
In [38]: s = pd.Series(pd.date_range('20130101 09:10:12', periods=4))

In [39]: s
Out[39]: 
0   2013-01-01 09:10:12
1   2013-01-02 09:10:12
2   2013-01-03 09:10:12
3   2013-01-04 09:10:12
Length: 4, dtype: datetime64[ns]

In [40]: s.dt.hour
Out[40]: 
0    9
1    9
2    9
3    9
Length: 4, dtype: int32

In [41]: s.dt.second
Out[41]: 
0    12
1    12
2    12
3    12
Length: 4, dtype: int32

In [42]: s.dt.day
Out[42]: 
0    1
1    2
2    3
3    4
Length: 4, dtype: int32

In [43]: s.dt.freq
Out[43]: 'D'

This enables nice expressions like this:

In [44]: s[s.dt.day == 2]
Out[44]: 
1   2013-01-02 09:10:12
Length: 1, dtype: datetime64[ns]

You can easily produce tz aware transformations:

In [45]: stz = s.dt.tz_localize('US/Eastern')

In [46]: stz
Out[46]: 
0   2013-01-01 09:10:12-05:00
1   2013-01-02 09:10:12-05:00
2   2013-01-03 09:10:12-05:00
3   2013-01-04 09:10:12-05:00
Length: 4, dtype: datetime64[ns, US/Eastern]

In [47]: stz.dt.tz
Out[47]: <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>

You can also chain these types of operations:

In [48]: s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
Out[48]: 
0   2013-01-01 04:10:12-05:00
1   2013-01-02 04:10:12-05:00
2   2013-01-03 04:10:12-05:00
3   2013-01-04 04:10:12-05:00
Length: 4, dtype: datetime64[ns, US/Eastern]

The .dt accessor works for period and timedelta dtypes.

# period
In [49]: s = pd.Series(pd.period_range('20130101', periods=4, freq='D'))

In [50]: s
Out[50]: 
0    2013-01-01
1    2013-01-02
2    2013-01-03
3    2013-01-04
Length: 4, dtype: period[D]

In [51]: s.dt.year
Out[51]: 
0    2013
1    2013
2    2013
3    2013
Length: 4, dtype: int64

In [52]: s.dt.day
Out[52]: 
0    1
1    2
2    3
3    4
Length: 4, dtype: int64

# timedelta
In [53]: s = pd.Series(pd.timedelta_range('1 day 00:00:05', periods=4, freq='s'))

In [54]: s
Out[54]: 
0   1 days 00:00:05
1   1 days 00:00:06
2   1 days 00:00:07
3   1 days 00:00:08
Length: 4, dtype: timedelta64[ns]

In [55]: s.dt.days
Out[55]: 
0    1
1    1
2    1
3    1
Length: 4, dtype: int64

In [56]: s.dt.seconds
Out[56]: 
0    5
1    6
2    7
3    8
Length: 4, dtype: int32

In [57]: s.dt.components
Out[57]: 
   days  hours  minutes  seconds  milliseconds  microseconds  nanoseconds
0     1      0        0        5             0             0            0
1     1      0        0        6             0             0            0
2     1      0        0        7             0             0            0
3     1      0        0        8             0             0            0

[4 rows x 7 columns]

Timezone handling improvements#

  • tz_localize(None) for tz-aware Timestamp and DatetimeIndex now removes timezone holding local time, previously this resulted in Exception or TypeError (GH 7812)

    In [58]: ts = pd.Timestamp('2014-08-01 09:00', tz='US/Eastern')

In [59]: ts
Out[59]: Timestamp('2014-08-01 09:00:00-0400', tz='US/Eastern')

In [60]: ts.tz_localize(None)
Out[60]: Timestamp('2014-08-01 09:00:00')

In [61]: didx = pd.date_range(start='2014-08-01 09:00', freq='H',
   ....:                      periods=10, tz='US/Eastern')
   ....: 

In [62]: didx
Out[62]: 
DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00',
               '2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00',
               '2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00',
               '2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00',
               '2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'],
              dtype='datetime64[ns, US/Eastern]', freq='H')

In [63]: didx.tz_localize(None)
Out[63]: 
DatetimeIndex(['2014-08-01 09:00:00', '2014-08-01 10:00:00',
               '2014-08-01 11:00:00', '2014-08-01 12:00:00',
               '2014-08-01 13:00:00', '2014-08-01 14:00:00',
               '2014-08-01 15:00:00', '2014-08-01 16:00:00',
               '2014-08-01 17:00:00', '2014-08-01 18:00:00'],
              dtype='datetime64[ns]', freq=None)

  • tz_localize now accepts the ambiguous keyword which allows for passing an array of bools indicating whether the date belongs in DST or not, ‘NaT’ for setting transition times to NaT, ‘infer’ for inferring DST/non-DST, and ‘raise’ (default) for an AmbiguousTimeError to be raised. See the docs for more details (GH 7943)

  • DataFrame.tz_localize and DataFrame.tz_convert now accepts an optional level argument for localizing a specific level of a MultiIndex (GH 7846)

  • Timestamp.tz_localize and Timestamp.tz_convert now raise TypeError in error cases, rather than Exception (GH 8025)

  • a timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone (rather than being a naive datetime64[ns]) as object dtype (GH 8411)

  • Timestamp.__repr__ displays dateutil.tz.tzoffset info (GH 7907)

Rolling/expanding moments improvements#

  • rolling_min(), rolling_max(), rolling_cov(), and rolling_corr() now return objects with all NaN when len(arg) < min_periods <= window rather than raising. (This makes all rolling functions consistent in this behavior). (GH 7766)

    Prior to 0.15.0

    In [64]: s = pd.Series([10, 11, 12, 13])

    In [15]: pd.rolling_min(s, window=10, min_periods=5)
ValueError: min_periods (5) must be <= window (4)

    New behavior

    In [4]: pd.rolling_min(s, window=10, min_periods=5)
Out[4]:
0   NaN
1   NaN
2   NaN
3   NaN
dtype: float64

  • rolling_max(), rolling_min(), rolling_sum(), rolling_mean(), rolling_median(), rolling_std(), rolling_var(), rolling_skew(), rolling_kurt(), rolling_quantile(), rolling_cov(), rolling_corr(), rolling_corr_pairwise(), rolling_window(), and rolling_apply() with center=True previously would return a result of the same structure as the input arg with NaN in the final (window-1)/2 entries.

    Now the final (window-1)/2 entries of the result are calculated as if the input arg were followed by (window-1)/2 NaN values (or with shrinking windows, in the case of rolling_apply()). (GH 7925, GH 8269)

    Prior behavior (note final value is NaN):

    In [7]: pd.rolling_sum(Series(range(4)), window=3, min_periods=0, center=True)
Out[7]:
0     1
1     3
2     6
3   NaN
dtype: float64

    New behavior (note final value is 5 = sum([2, 3, NaN])):

    In [7]: pd.rolling_sum(pd.Series(range(4)), window=3,
  ....:                min_periods=0, center=True)
Out[7]:
0    1
1    3
2    6
3    5
dtype: float64

  • rolling_window() now normalizes the weights properly in rolling mean mode (mean=True) so that the calculated weighted means (e.g. ‘triang’, ‘gaussian’) are distributed about the same means as those calculated without weighting (i.e. ‘boxcar’). See the note on normalization for further details. (GH 7618)

    In [65]: s = pd.Series([10.5, 8.8, 11.4, 9.7, 9.3])

    Behavior prior to 0.15.0:

    In [39]: pd.rolling_window(s, window=3, win_type='triang', center=True)
Out[39]:
0         NaN
1    6.583333
2    6.883333
3    6.683333
4         NaN
dtype: float64

    New behavior

    In [10]: pd.rolling_window(s, window=3, win_type='triang', center=True)
Out[10]:
0       NaN
1     9.875
2    10.325
3    10.025
4       NaN
dtype: float64

  • Removed center argument from all expanding_ functions (see list), as the results produced when center=True did not make much sense. (GH 7925)

  • Added optional ddof argument to expanding_cov() and rolling_cov(). The default value of 1 is backwards-compatible. (GH 8279)

  • Documented the ddof argument to expanding_var(), expanding_std(), rolling_var(), and rolling_std(). These functions’ support of a ddof argument (with a default value of 1) was previously undocumented. (GH 8064)

  • ewma(), ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now interpret min_periods in the same manner that the rolling_*() and expanding_*() functions do: a given result entry will be NaN if the (expanding, in this case) window does not contain at least min_periods values. The previous behavior was to set to NaN the min_periods entries starting with the first non- NaN value. (GH 7977)

    Prior behavior (note values start at index 2, which is min_periods after index 0 (the index of the first non-empty value)):

    In [66]: s  = pd.Series([1, None, None, None, 2, 3])

    In [51]: pd.ewma(s, com=3., min_periods=2)
Out[51]:
0         NaN
1         NaN
2    1.000000
3    1.000000
4    1.571429
5    2.189189
dtype: float64

    New behavior (note values start at index 4, the location of the 2nd (since min_periods=2) non-empty value):

    In [2]: pd.ewma(s, com=3., min_periods=2)
Out[2]:
0         NaN
1         NaN
2         NaN
3         NaN
4    1.759644
5    2.383784
dtype: float64

  • ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now have an optional adjust argument, just like ewma() does, affecting how the weights are calculated. The default value of adjust is True, which is backwards-compatible. See Exponentially weighted moment functions for details. (GH 7911)

  • ewma(), ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now have an optional ignore_na argument. When ignore_na=False (the default), missing values are taken into account in the weights calculation. When ignore_na=True (which reproduces the pre-0.15.0 behavior), missing values are ignored in the weights calculation. (GH 7543)

    In [7]: pd.ewma(pd.Series([None, 1., 8.]), com=2.)
Out[7]:
0    NaN
1    1.0
2    5.2
dtype: float64

In [8]: pd.ewma(pd.Series([1., None, 8.]), com=2.,
  ....:         ignore_na=True)  # pre-0.15.0 behavior
Out[8]:
0    1.0
1    1.0
2    5.2
dtype: float64

In [9]: pd.ewma(pd.Series([1., None, 8.]), com=2.,
  ....:         ignore_na=False)  # new default
Out[9]:
0    1.000000
1    1.000000
2    5.846154
dtype: float64

    Warning

    By default (ignore_na=False) the ewm*() functions’ weights calculation in the presence of missing values is different than in pre-0.15.0 versions. To reproduce the pre-0.15.0 calculation of weights in the presence of missing values one must specify explicitly ignore_na=True.

  • Bug in expanding_cov(), expanding_corr(), rolling_cov(), rolling_cor(), ewmcov(), and ewmcorr() returning results with columns sorted by name and producing an error for non-unique columns; now handles non-unique columns and returns columns in original order (except for the case of two DataFrames with pairwise=False, where behavior is unchanged) (GH 7542)

  • Bug in rolling_count() and expanding_*() functions unnecessarily producing error message for zero-length data (GH 8056)

  • Bug in rolling_apply() and expanding_apply() interpreting min_periods=0 as min_periods=1 (GH 8080)

  • Bug in expanding_std() and expanding_var() for a single value producing a confusing error message (GH 7900)

  • Bug in rolling_std() and rolling_var() for a single value producing 0 rather than NaN (GH 7900)

  • Bug in ewmstd(), ewmvol(), ewmvar(), and ewmcov() calculation of de-biasing factors when bias=False (the default). Previously an incorrect constant factor was used, based on adjust=True, ignore_na=True, and an infinite number of observations. Now a different factor is used for each entry, based on the actual weights (analogous to the usual N/(N-1) factor). In particular, for a single point a value of NaN is returned when bias=False, whereas previously a value of (approximately) 0 was returned.

    For example, consider the following pre-0.15.0 results for ewmvar(..., bias=False), and the corresponding debiasing factors:

    In [67]: s = pd.Series([1., 2., 0., 4.])

    In [89]: pd.ewmvar(s, com=2., bias=False)
Out[89]:
0   -2.775558e-16
1    3.000000e-01
2    9.556787e-01
3    3.585799e+00
dtype: float64

In [90]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True)
Out[90]:
0    1.25
1    1.25
2    1.25
3    1.25
dtype: float64

    Note that entry 0 is approximately 0, and the debiasing factors are a constant 1.25. By comparison, the following 0.15.0 results have a NaN for entry 0, and the debiasing factors are decreasing (towards 1.25):

    In [14]: pd.ewmvar(s, com=2., bias=False)
Out[14]:
0         NaN
1    0.500000
2    1.210526
3    4.089069
dtype: float64

In [15]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True)
Out[15]:
0         NaN
1    2.083333
2    1.583333
3    1.425439
dtype: float64

    See Exponentially weighted moment functions for details. (GH 7912)

Improvements in the SQL IO module#

  • Added support for a chunksize parameter to to_sql function. This allows DataFrame to be written in chunks and avoid packet-size overflow errors (GH 8062).

  • Added support for a chunksize parameter to read_sql function. Specifying this argument will return an iterator through chunks of the query result (GH 2908).

  • Added support for writing datetime.date and datetime.time object columns with to_sql (GH 6932).

  • Added support for specifying a schema to read from/write to with read_sql_table and to_sql (GH 7441, GH 7952). For example:

    df.to_sql('table', engine, schema='other_schema')  # noqa F821
pd.read_sql_table('table', engine, schema='other_schema')  # noqa F821

  • Added support for writing NaN values with to_sql (GH 2754).

  • Added support for writing datetime64 columns with to_sql for all database flavors (GH 7103).

Backwards incompatible API changes#

Breaking changes#

API changes related to Categorical (see here for more details):

  • The Categorical constructor with two arguments changed from “codes/labels and levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use Categorical directly, please audit your code by changing it to use the from_codes() constructor.

    An old function call like (prior to 0.15.0):

    pd.Categorical([0,1,0,2,1], levels=['a', 'b', 'c'])

    will have to adapted to the following to keep the same behaviour:

    In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c'])
Out[2]:
[a, b, a, c, b]
Categories (3, object): [a, b, c]

API changes related to the introduction of the Timedelta scalar (see above for more details):

  • Prior to 0.15.0 to_timedelta() would return a Series for list-like/Series input, and a np.timedelta64 for scalar input. It will now return a TimedeltaIndex for list-like input, Series for Series input, and Timedelta for scalar input.

For API changes related to the rolling and expanding functions, see detailed overview above.

Other notable API changes:

  • Consistency when indexing with .loc and a list-like indexer when no values are found.

    In [68]: df = pd.DataFrame([['a'], ['b']], index=[1, 2])

In [69]: df
Out[69]: 
   0
1  a
2  b

[2 rows x 1 columns]

    In prior versions there was a difference in these two constructs:

    • df.loc[[3]] would return a frame reindexed by 3 (with all np.nan values)

    • df.loc[[3],:] would raise KeyError.

    Both will now raise a KeyError. The rule is that at least 1 indexer must be found when using a list-like and .loc (GH 7999)

    Furthermore in prior versions these were also different:

    • df.loc[[1,3]] would return a frame reindexed by [1,3]

    • df.loc[[1,3],:] would raise KeyError.

    Both will now return a frame reindex by [1,3]. E.g.

    In [3]: df.loc[[1, 3]]
Out[3]:
     0
1    a
3  NaN

In [4]: df.loc[[1, 3], :]
Out[4]:
     0
1    a
3  NaN

    This can also be seen in multi-axis indexing with a Panel.

    >>> p = pd.Panel(np.arange(2 * 3 * 4).reshape(2, 3, 4),
...              items=['ItemA', 'ItemB'],
...              major_axis=[1, 2, 3],
...              minor_axis=['A', 'B', 'C', 'D'])
>>> p
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemB
Major_axis axis: 1 to 3
Minor_axis axis: A to D

    The following would raise KeyError prior to 0.15.0:

    In [5]:
Out[5]:
   ItemA  ItemD
1      3    NaN
2      7    NaN
3     11    NaN

    Furthermore, .loc will raise If no values are found in a MultiIndex with a list-like indexer:

    In [70]: s = pd.Series(np.arange(3, dtype='int64'),
   ....:               index=pd.MultiIndex.from_product([['A'],
   ....:                                                ['foo', 'bar', 'baz']],
   ....:                                                names=['one', 'two'])
   ....:               ).sort_index()
   ....: 

In [71]: s
Out[71]: 
one  two
A    bar    1
     baz    2
     foo    0
Length: 3, dtype: int64

In [72]: try:
   ....:     s.loc[['D']]
   ....: except KeyError as e:
   ....:     print("KeyError: " + str(e))
   ....: 
KeyError: "['D'] not in index"

  • Assigning values to None now considers the dtype when choosing an ‘empty’ value (GH 7941).

    Previously, assigning to None in numeric containers changed the dtype to object (or errored, depending on the call). It now uses NaN:

    In [73]: s = pd.Series([1., 2., 3.])

In [74]: s.loc[0] = None

In [75]: s
Out[75]: 
0    NaN
1    2.0
2    3.0
Length: 3, dtype: float64

    NaT is now used similarly for datetime containers.

    For object containers, we now preserve None values (previously these were converted to NaN values).

    In [76]: s = pd.Series(["a", "b", "c"])

In [77]: s.loc[0] = None

In [78]: s
Out[78]: 
0    None
1       b
2       c
Length: 3, dtype: object

    To insert a NaN, you must explicitly use np.nan. See the docs.

  • In prior versions, updating a pandas object inplace would not reflect in other python references to this object. (GH 8511, GH 5104)

    In [79]: s = pd.Series([1, 2, 3])

In [80]: s2 = s

In [81]: s += 1.5

    Behavior prior to v0.15.0

    # the original object
In [5]: s
Out[5]:
0    2.5
1    3.5
2    4.5
dtype: float64


# a reference to the original object
In [7]: s2
Out[7]:
0    1
1    2
2    3
dtype: int64

    This is now the correct behavior

    # the original object
In [82]: s
Out[82]: 
0    2.5
1    3.5
2    4.5
Length: 3, dtype: float64

# a reference to the original object
In [83]: s2
Out[83]: 
0    2.5
1    3.5
2    4.5
Length: 3, dtype: float64

  • Made both the C-based and Python engines for read_csv and read_table ignore empty lines in input as well as white space-filled lines, as long as sep is not white space. This is an API change that can be controlled by the keyword parameter skip_blank_lines. See the docs (GH 4466)

  • A timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone and inserted as object dtype rather than being converted to a naive datetime64[ns] (GH 8411).

  • Bug in passing a DatetimeIndex with a timezone that was not being retained in DataFrame construction from a dict (GH 7822)

    In prior versions this would drop the timezone, now it retains the timezone, but gives a column of object dtype:

    In [84]: i = pd.date_range('1/1/2011', periods=3, freq='10s', tz='US/Eastern')

In [85]: i
Out[85]: 
DatetimeIndex(['2011-01-01 00:00:00-05:00', '2011-01-01 00:00:10-05:00',
               '2011-01-01 00:00:20-05:00'],
              dtype='datetime64[ns, US/Eastern]', freq='10S')

In [86]: df = pd.DataFrame({'a': i})

In [87]: df
Out[87]: 
                          a
0 2011-01-01 00:00:00-05:00
1 2011-01-01 00:00:10-05:00
2 2011-01-01 00:00:20-05:00

[3 rows x 1 columns]

In [88]: df.dtypes
Out[88]: 
a    datetime64[ns, US/Eastern]
Length: 1, dtype: object

    Previously this would have yielded a column of datetime64 dtype, but without timezone info.

    The behaviour of assigning a column to an existing dataframe as df['a'] = i remains unchanged (this already returned an object column with a timezone).

  • When passing multiple levels to stack(), it will now raise a ValueError when the levels aren’t all level names or all level numbers (GH 7660). See Reshaping by stacking and unstacking.

  • Raise a ValueError in df.to_hdf with ‘fixed’ format, if df has non-unique columns as the resulting file will be broken (GH 7761)

  • SettingWithCopy raise/warnings (according to the option mode.chained_assignment) will now be issued when setting a value on a sliced mixed-dtype DataFrame using chained-assignment. (GH 7845, GH 7950)

    In [1]: df = pd.DataFrame(np.arange(0, 9), columns=['count'])

In [2]: df['group'] = 'b'

In [3]: df.iloc[0:5]['group'] = 'a'
/usr/local/bin/ipython:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

  • merge, DataFrame.merge, and ordered_merge now return the same type as the left argument (GH 7737).

  • Previously an enlargement with a mixed-dtype frame would act unlike .append which will preserve dtypes (related GH 2578, GH 8176):

    In [89]: df = pd.DataFrame([[True, 1], [False, 2]],
   ....:                   columns=["female", "fitness"])
   ....: 

In [90]: df
Out[90]: 
   female  fitness
0    True        1
1   False        2

[2 rows x 2 columns]

In [91]: df.dtypes
Out[91]: 
female      bool
fitness    int64
Length: 2, dtype: object

# dtypes are now preserved
In [92]: df.loc[2] = df.loc[1]

In [93]: df
Out[93]: 
   female  fitness
0    True        1
1   False        2
2   False        2

[3 rows x 2 columns]

In [94]: df.dtypes
Out[94]: 
female      bool
fitness    int64
Length: 2, dtype: object

  • Series.to_csv() now returns a string when path=None, matching the behaviour of DataFrame.to_csv() (GH 8215).

  • read_hdf now raises IOError when a file that doesn’t exist is passed in. Previously, a new, empty file was created, and a KeyError raised (GH 7715).

  • DataFrame.info() now ends its output with a newline character (GH 8114)

  • Concatenating no objects will now raise a ValueError rather than a bare Exception.

  • Merge errors will now be sub-classes of ValueError rather than raw Exception (GH 8501)

  • DataFrame.plot and Series.plot keywords are now have consistent orders (GH 8037)

Internal refactoring#

In 0.15.0 Index has internally been refactored to no longer sub-class ndarray but instead subclass PandasObject, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be a transparent change with only very limited API implications (GH 5080, GH 7439, GH 7796, GH 8024, GH 8367, GH 7997, GH 8522):

  • you may need to unpickle pandas version < 0.15.0 pickles using pd.read_pickle rather than pickle.load. See pickle docs

  • when plotting with a PeriodIndex, the matplotlib internal axes will now be arrays of Period rather than a PeriodIndex (this is similar to how a DatetimeIndex passes arrays of datetimes now)

  • MultiIndexes will now raise similarly to other pandas objects w.r.t. truth testing, see here (GH 7897).

  • When plotting a DatetimeIndex directly with matplotlib’s plot function, the axis labels will no longer be formatted as dates but as integers (the internal representation of a datetime64). UPDATE This is fixed in 0.15.1, see here.

Deprecations#

  • The attributes Categorical labels and levels attributes are deprecated and renamed to codes and categories.

  • The outtype argument to pd.DataFrame.to_dict has been deprecated in favor of orient. (GH 7840)

  • The convert_dummies method has been deprecated in favor of get_dummies (GH 8140)

  • The infer_dst argument in tz_localize will be deprecated in favor of ambiguous to allow for more flexibility in dealing with DST transitions. Replace infer_dst=True with ambiguous='infer' for the same behavior (GH 7943). See the docs for more details.

  • The top-level pd.value_range has been deprecated and can be replaced by .describe() (GH 8481)

  • The Index set operations + and - were deprecated in order to provide these for numeric type operations on certain index types. + can be replaced by .union() or |, and - by .difference(). Further the method name Index.diff() is deprecated and can be replaced by Index.difference() (GH 8226)

    # +
pd.Index(['a', 'b', 'c']) + pd.Index(['b', 'c', 'd'])

# should be replaced by
pd.Index(['a', 'b', 'c']).union(pd.Index(['b', 'c', 'd']))

    # -
pd.Index(['a', 'b', 'c']) - pd.Index(['b', 'c', 'd'])

# should be replaced by
pd.Index(['a', 'b', 'c']).difference(pd.Index(['b', 'c', 'd']))

  • The infer_types argument to read_html() now has no effect and is deprecated (GH 7762, GH 7032).

Removal of prior version deprecations/changes#

  • Remove DataFrame.delevel method in favor of DataFrame.reset_index

Enhancements#

Enhancements in the importing/exporting of Stata files:

  • Added support for bool, uint8, uint16 and uint32 data types in to_stata (GH 7097, GH 7365)

  • Added conversion option when importing Stata files (GH 8527)

  • DataFrame.to_stata and StataWriter check string length for compatibility with limitations imposed in dta files where fixed-width strings must contain 244 or fewer characters. Attempting to write Stata dta files with strings longer than 244 characters raises a ValueError. (GH 7858)

  • read_stata and StataReader can import missing data information into a DataFrame by setting the argument convert_missing to True. When using this options, missing values are returned as StataMissingValue objects and columns containing missing values have object data type. (GH 8045)

Enhancements in the plotting functions:

  • Added layout keyword to DataFrame.plot. You can pass a tuple of (rows, columns), one of which can be -1 to automatically infer (GH 6667, GH 8071).

  • Allow to pass multiple axes to DataFrame.plot, hist and boxplot (GH 5353, GH 6970, GH 7069)

  • Added support for c, colormap and colorbar arguments for DataFrame.plot with kind='scatter' (GH 7780)

  • Histogram from DataFrame.plot with kind='hist' (GH 7809), See the docs.

  • Boxplot from DataFrame.plot with kind='box' (GH 7998), See the docs.

Other:

  • read_csv now has a keyword parameter float_precision which specifies which floating-point converter the C engine should use during parsing, see here (GH 8002, GH 8044)

  • Added searchsorted method to Series objects (GH 7447)

  • describe() on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via the include/exclude arguments. See the docs (GH 8164).

    In [95]: df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,
   ....:                    'catB': ['a', 'b', 'c', 'd'] * 6,
   ....:                    'numC': np.arange(24),
   ....:                    'numD': np.arange(24.) + .5})
   ....: 

In [96]: df.describe(include=["object"])
Out[96]: 
       catA catB
count    24   24
unique    2    4
top     foo    a
freq     16    6

[4 rows x 2 columns]

In [97]: df.describe(include=["number", "object"], exclude=["float"])
Out[97]: 
       catA catB       numC
count    24   24  24.000000
unique    2    4        NaN
top     foo    a        NaN
freq     16    6        NaN
mean    NaN  NaN  11.500000
std     NaN  NaN   7.071068
min     NaN  NaN   0.000000
25%     NaN  NaN   5.750000
50%     NaN  NaN  11.500000
75%     NaN  NaN  17.250000
max     NaN  NaN  23.000000

[11 rows x 3 columns]

    Requesting all columns is possible with the shorthand ‘all’

    In [98]: df.describe(include='all')
Out[98]: 
       catA catB       numC       numD
count    24   24  24.000000  24.000000
unique    2    4        NaN        NaN
top     foo    a        NaN        NaN
freq     16    6        NaN        NaN
mean    NaN  NaN  11.500000  12.000000
std     NaN  NaN   7.071068   7.071068
min     NaN  NaN   0.000000   0.500000
25%     NaN  NaN   5.750000   6.250000
50%     NaN  NaN  11.500000  12.000000
75%     NaN  NaN  17.250000  17.750000
max     NaN  NaN  23.000000  23.500000

[11 rows x 4 columns]

    Without those arguments, describe will behave as before, including only numerical columns or, if none are, only categorical columns. See also the docs

  • Added split as an option to the orient argument in pd.DataFrame.to_dict. (GH 7840)

  • The get_dummies method can now be used on DataFrames. By default only categorical columns are encoded as 0’s and 1’s, while other columns are left untouched.

    In [99]: df = pd.DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],
   ....:                 'C': [1, 2, 3]})
   ....: 

In [100]: pd.get_dummies(df)
Out[100]: 
   C    A_a    A_b    B_b    B_c
0  1   True  False  False   True
1  2  False   True  False   True
2  3   True  False   True  False

[3 rows x 5 columns]

  • PeriodIndex supports resolution as the same as DatetimeIndex (GH 7708)

  • pandas.tseries.holiday has added support for additional holidays and ways to observe holidays (GH 7070)

  • pandas.tseries.holiday.Holiday now supports a list of offsets in Python3 (GH 7070)

  • pandas.tseries.holiday.Holiday now supports a days_of_week parameter (GH 7070)

  • GroupBy.nth() now supports selecting multiple nth values (GH 7910)

    In [101]: business_dates = pd.date_range(start='4/1/2014', end='6/30/2014', freq='B')

In [102]: df = pd.DataFrame(1, index=business_dates, columns=['a', 'b'])

# get the first, 4th, and last date index for each month
In [103]: df.groupby([df.index.year, df.index.month]).nth([0, 3, -1])
Out[103]: 
            a  b
2014-04-01  1  1
2014-04-04  1  1
2014-04-30  1  1
2014-05-01  1  1
2014-05-06  1  1
2014-05-30  1  1
2014-06-02  1  1
2014-06-05  1  1
2014-06-30  1  1

[9 rows x 2 columns]

  • Period and PeriodIndex supports addition/subtraction with timedelta-likes (GH 7966)

    If Period freq is D, H, T, S, L, U, N, Timedelta-like can be added if the result can have same freq. Otherwise, only the same offsets can be added.

    In [104]: idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H')

In [105]: idx
Out[105]: 
PeriodIndex(['2014-07-01 09:00', '2014-07-01 10:00', '2014-07-01 11:00',
             '2014-07-01 12:00', '2014-07-01 13:00'],
            dtype='period[H]')

In [106]: idx + pd.offsets.Hour(2)
Out[106]: 
PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
             '2014-07-01 14:00', '2014-07-01 15:00'],
            dtype='period[H]')

In [107]: idx + pd.Timedelta('120m')
Out[107]: 
PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
             '2014-07-01 14:00', '2014-07-01 15:00'],
            dtype='period[H]')

In [108]: idx = pd.period_range('2014-07', periods=5, freq='M')

In [109]: idx
Out[109]: PeriodIndex(['2014-07', '2014-08', '2014-09', '2014-10', '2014-11'], dtype='period[M]')

In [110]: idx + pd.offsets.MonthEnd(3)
Out[110]: PeriodIndex(['2014-10', '2014-11', '2014-12', '2015-01', '2015-02'], dtype='period[M]')

  • Added experimental compatibility with openpyxl for versions >= 2.0. The DataFrame.to_excel method engine keyword now recognizes openpyxl1 and openpyxl2 which will explicitly require openpyxl v1 and v2 respectively, failing if the requested version is not available. The openpyxl engine is a now a meta-engine that automatically uses whichever version of openpyxl is installed. (GH 7177)

  • DataFrame.fillna can now accept a DataFrame as a fill value (GH 8377)

  • Passing multiple levels to stack() will now work when multiple level numbers are passed (GH 7660). See Reshaping by stacking and unstacking.

  • set_names(), set_labels(), and set_levels() methods now take an optional level keyword argument to all modification of specific level(s) of a MultiIndex. Additionally set_names() now accepts a scalar string value when operating on an Index or on a specific level of a MultiIndex (GH 7792)

    In [111]: idx = pd.MultiIndex.from_product([['a'], range(3), list("pqr")],
   .....:                                  names=['foo', 'bar', 'baz'])
   .....: 

In [112]: idx.set_names('qux', level=0)
Out[112]: 
MultiIndex([('a', 0, 'p'),
            ('a', 0, 'q'),
            ('a', 0, 'r'),
            ('a', 1, 'p'),
            ('a', 1, 'q'),
            ('a', 1, 'r'),
            ('a', 2, 'p'),
            ('a', 2, 'q'),
            ('a', 2, 'r')],
           names=['qux', 'bar', 'baz'])

In [113]: idx.set_names(['qux', 'corge'], level=[0, 1])
Out[113]: 
MultiIndex([('a', 0, 'p'),
            ('a', 0, 'q'),
            ('a', 0, 'r'),
            ('a', 1, 'p'),
            ('a', 1, 'q'),
            ('a', 1, 'r'),
            ('a', 2, 'p'),
            ('a', 2, 'q'),
            ('a', 2, 'r')],
           names=['qux', 'corge', 'baz'])

In [114]: idx.set_levels(['a', 'b', 'c'], level='bar')
Out[114]: 
MultiIndex([('a', 'a', 'p'),
            ('a', 'a', 'q'),
            ('a', 'a', 'r'),
            ('a', 'b', 'p'),
            ('a', 'b', 'q'),
            ('a', 'b', 'r'),
            ('a', 'c', 'p'),
            ('a', 'c', 'q'),
            ('a', 'c', 'r')],
           names=['foo', 'bar', 'baz'])

In [115]: idx.set_levels([['a', 'b', 'c'], [1, 2, 3]], level=[1, 2])
Out[115]: 
MultiIndex([('a', 'a', 1),
            ('a', 'a', 2),
            ('a', 'a', 3),
            ('a', 'b', 1),
            ('a', 'b', 2),
            ('a', 'b', 3),
            ('a', 'c', 1),
            ('a', 'c', 2),
            ('a', 'c', 3)],
           names=['foo', 'bar', 'baz'])

  • Index.isin now supports a level argument to specify which index level to use for membership tests (GH 7892, GH 7890)

    In [1]: idx = pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']])

In [2]: idx.values
Out[2]: array([(0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), (1, 'c')], dtype=object)

In [3]: idx.isin(['a', 'c', 'e'], level=1)
Out[3]: array([ True, False,  True,  True, False,  True], dtype=bool)

  • Index now supports duplicated and drop_duplicates. (GH 4060)

    In [116]: idx = pd.Index([1, 2, 3, 4, 1, 2])

In [117]: idx
Out[117]: Index([1, 2, 3, 4, 1, 2], dtype='int64')

In [118]: idx.duplicated()
Out[118]: array([False, False, False, False,  True,  True])

In [119]: idx.drop_duplicates()
Out[119]: Index([1, 2, 3, 4], dtype='int64')

  • add copy=True argument to pd.concat to enable pass through of complete blocks (GH 8252)

  • Added support for numpy 1.8+ data types (bool_, int_, float_, string_) for conversion to R dataframe (GH 8400)

Performance#

  • Performance improvements in DatetimeIndex.__iter__ to allow faster iteration (GH 7683)

  • Performance improvements in Period creation (and PeriodIndex setitem) (GH 5155)

  • Improvements in Series.transform for significant performance gains (revised) (GH 6496)

  • Performance improvements in StataReader when reading large files (GH 8040, GH 8073)

  • Performance improvements in StataWriter when writing large files (GH 8079)

  • Performance and memory usage improvements in multi-key groupby (GH 8128)

  • Performance improvements in groupby .agg and .apply where builtins max/min were not mapped to numpy/cythonized versions (GH 7722)

  • Performance improvement in writing to sql (to_sql) of up to 50% (GH 8208).

  • Performance benchmarking of groupby for large value of ngroups (GH 6787)

  • Performance improvement in CustomBusinessDay, CustomBusinessMonth (GH 8236)

  • Performance improvement for MultiIndex.values for multi-level indexes containing datetimes (GH 8543)

Bug fixes#

  • Bug in pivot_table, when using margins and a dict aggfunc (GH 8349)

  • Bug in read_csv where squeeze=True would return a view (GH 8217)

  • Bug in checking of table name in read_sql in certain cases (GH 7826).

  • Bug in DataFrame.groupby where Grouper does not recognize level when frequency is specified (GH 7885)

  • Bug in multiindexes dtypes getting mixed up when DataFrame is saved to SQL table (GH 8021)

  • Bug in Series 0-division with a float and integer operand dtypes (GH 7785)

  • Bug in Series.astype("unicode") not calling unicode on the values correctly (GH 7758)

  • Bug in DataFrame.as_matrix() with mixed datetime64[ns] and timedelta64[ns] dtypes (GH 7778)

  • Bug in HDFStore.select_column() not preserving UTC timezone info when selecting a DatetimeIndex (GH 7777)

  • Bug in to_datetime when format='%Y%m%d' and coerce=True are specified, where previously an object array was returned (rather than a coerced time-series with NaT), (GH 7930)

  • Bug in DatetimeIndex and PeriodIndex in-place addition and subtraction cause different result from normal one (GH 6527)

  • Bug in adding and subtracting PeriodIndex with PeriodIndex raise TypeError (GH 7741)

  • Bug in combine_first with PeriodIndex data raises TypeError (GH 3367)

  • Bug in MultiIndex slicing with missing indexers (GH 7866)

  • Bug in MultiIndex slicing with various edge cases (GH 8132)

  • Regression in MultiIndex indexing with a non-scalar type object (GH 7914)

  • Bug in Timestamp comparisons with == and int64 dtype (GH 8058)

  • Bug in pickles contains DateOffset may raise AttributeError when normalize attribute is referred internally (GH 7748)

  • Bug in Panel when using major_xs and copy=False is passed (deprecation warning fails because of missing warnings) (GH 8152).

  • Bug in pickle deserialization that failed for pre-0.14.1 containers with dup items trying to avoid ambiguity when matching block and manager items, when there’s only one block there’s no ambiguity (GH 7794)

  • Bug in putting a PeriodIndex into a Series would convert to int64 dtype, rather than object of Periods (GH 7932)

  • Bug in HDFStore iteration when passing a where (GH 8014)

  • Bug in DataFrameGroupby.transform when transforming with a passed non-sorted key (GH 8046, GH 8430)

  • Bug in repeated timeseries line and area plot may result in ValueError or incorrect kind (GH 7733)

  • Bug in inference in a MultiIndex with datetime.date inputs (GH 7888)

  • Bug in get where an IndexError would not cause the default value to be returned (GH 7725)

  • Bug in offsets.apply, rollforward and rollback may reset nanosecond (GH 7697)

  • Bug in offsets.apply, rollforward and rollback may raise AttributeError if Timestamp has dateutil tzinfo (GH 7697)

  • Bug in sorting a MultiIndex frame with a Float64Index (GH 8017)

  • Bug in inconsistent panel setitem with a rhs of a DataFrame for alignment (GH 7763)

  • Bug in is_superperiod and is_subperiod cannot handle higher frequencies than S (GH 7760, GH 7772, GH 7803)

  • Bug in 32-bit platforms with Series.shift (GH 8129)

  • Bug in PeriodIndex.unique returns int64 np.ndarray (GH 7540)

  • Bug in groupby.apply with a non-affecting mutation in the function (GH 8467)

  • Bug in DataFrame.reset_index which has MultiIndex contains PeriodIndex or DatetimeIndex with tz raises ValueError (GH 7746, GH 7793)

  • Bug in DataFrame.plot with subplots=True may draw unnecessary minor xticks and yticks (GH 7801)

  • Bug in StataReader which did not read variable labels in 117 files due to difference between Stata documentation and implementation (GH 7816)

  • Bug in StataReader where strings were always converted to 244 characters-fixed width irrespective of underlying string size (GH 7858)

  • Bug in DataFrame.plot and Series.plot may ignore rot and fontsize keywords (GH 7844)

  • Bug in DatetimeIndex.value_counts doesn’t preserve tz (GH 7735)

  • Bug in PeriodIndex.value_counts results in Int64Index (GH 7735)

  • Bug in DataFrame.join when doing left join on index and there are multiple matches (GH 5391)

  • Bug in GroupBy.transform() where int groups with a transform that didn’t preserve the index were incorrectly truncated (GH 7972).

  • Bug in groupby where callable objects without name attributes would take the wrong path, and produce a DataFrame instead of a Series (GH 7929)

  • Bug in groupby error message when a DataFrame grouping column is duplicated (GH 7511)

  • Bug in read_html where the infer_types argument forced coercion of date-likes incorrectly (GH 7762, GH 7032).

  • Bug in Series.str.cat with an index which was filtered as to not include the first item (GH 7857)

  • Bug in Timestamp cannot parse nanosecond from string (GH 7878)

  • Bug in Timestamp with string offset and tz results incorrect (GH 7833)

  • Bug in tslib.tz_convert and tslib.tz_convert_single may return different results (GH 7798)

  • Bug in DatetimeIndex.intersection of non-overlapping timestamps with tz raises IndexError (GH 7880)

  • Bug in alignment with TimeOps and non-unique indexes (GH 8363)

  • Bug in GroupBy.filter() where fast path vs. slow path made the filter return a non scalar value that appeared valid but wasn’t (GH 7870).

  • Bug in date_range()/DatetimeIndex() when the timezone was inferred from input dates yet incorrect times were returned when crossing DST boundaries (GH 7835, GH 7901).

  • Bug in to_excel() where a negative sign was being prepended to positive infinity and was absent for negative infinity (GH 7949)

  • Bug in area plot draws legend with incorrect alpha when stacked=True (GH 8027)

  • Period and PeriodIndex addition/subtraction with np.timedelta64 results in incorrect internal representations (GH 7740)

  • Bug in Holiday with no offset or observance (GH 7987)

  • Bug in DataFrame.to_latex formatting when columns or index is a MultiIndex (GH 7982).

  • Bug in DateOffset around Daylight Savings Time produces unexpected results (GH 5175).

  • Bug in DataFrame.shift where empty columns would throw ZeroDivisionError on numpy 1.7 (GH 8019)

  • Bug in installation where html_encoding/*.html wasn’t installed and therefore some tests were not running correctly (GH 7927).

  • Bug in read_html where bytes objects were not tested for in _read (GH 7927).

  • Bug in DataFrame.stack() when one of the column levels was a datelike (GH 8039)

  • Bug in broadcasting numpy scalars with DataFrame (GH 8116)

  • Bug in pivot_table performed with nameless index and columns raises KeyError (GH 8103)

  • Bug in DataFrame.plot(kind='scatter') draws points and errorbars with different colors when the color is specified by c keyword (GH 8081)

  • Bug in Float64Index where iat and at were not testing and were failing (GH 8092).

  • Bug in DataFrame.boxplot() where y-limits were not set correctly when producing multiple axes (GH 7528, GH 5517).

  • Bug in read_csv where line comments were not handled correctly given a custom line terminator or delim_whitespace=True (GH 8122).

  • Bug in read_html where empty tables caused a StopIteration (GH 7575)

  • Bug in casting when setting a column in a same-dtype block (GH 7704)

  • Bug in accessing groups from a GroupBy when the original grouper was a tuple (GH 8121).

  • Bug in .at that would accept integer indexers on a non-integer index and do fallback (GH 7814)

  • Bug with kde plot and NaNs (GH 8182)

  • Bug in GroupBy.count with float32 data type were nan values were not excluded (GH 8169).

  • Bug with stacked barplots and NaNs (GH 8175).

  • Bug in resample with non evenly divisible offsets (e.g. ‘7s’) (GH 8371)

  • Bug in interpolation methods with the limit keyword when no values needed interpolating (GH 7173).

  • Bug where col_space was ignored in DataFrame.to_string() when header=False (GH 8230).

  • Bug with DatetimeIndex.asof incorrectly matching partial strings and returning the wrong date (GH 8245).

  • Bug in plotting methods modifying the global matplotlib rcParams (GH 8242).

  • Bug in DataFrame.__setitem__ that caused errors when setting a dataframe column to a sparse array (GH 8131)

  • Bug where Dataframe.boxplot() failed when entire column was empty (GH 8181).

  • Bug with messed variables in radviz visualization (GH 8199).

  • Bug in interpolation methods with the limit keyword when no values needed interpolating (GH 7173).

  • Bug where col_space was ignored in DataFrame.to_string() when header=False (GH 8230).

  • Bug in to_clipboard that would clip long column data (GH 8305)

  • Bug in DataFrame terminal display: Setting max_column/max_rows to zero did not trigger auto-resizing of dfs to fit terminal width/height (GH 7180).

  • Bug in OLS where running with “cluster” and “nw_lags” parameters did not work correctly, but also did not throw an error (GH 5884).

  • Bug in DataFrame.dropna that interpreted non-existent columns in the subset argument as the ‘last column’ (GH 8303)

  • Bug in Index.intersection on non-monotonic non-unique indexes (GH 8362).

  • Bug in masked series assignment where mismatching types would break alignment (GH 8387)

  • Bug in NDFrame.equals gives false negatives with dtype=object (GH 8437)

  • Bug in assignment with indexer where type diversity would break alignment (GH 8258)

  • Bug in NDFrame.loc indexing when row/column names were lost when target was a list/ndarray (GH 6552)

  • Regression in NDFrame.loc indexing when rows/columns were converted to Float64Index if target was an empty list/ndarray (GH 7774)

  • Bug in Series that allows it to be indexed by a DataFrame which has unexpected results. Such indexing is no longer permitted (GH 8444)

  • Bug in item assignment of a DataFrame with MultiIndex columns where right-hand-side columns were not aligned (GH 7655)

  • Suppress FutureWarning generated by NumPy when comparing object arrays containing NaN for equality (GH 7065)

  • Bug in DataFrame.eval() where the dtype of the not operator (~) was not correctly inferred as bool.

Contributors#

A total of 80 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

  • Aaron Schumacher +

  • Adam Greenhall

  • Andy Hayden

  • Anthony O’Brien +

  • Artemy Kolchinsky +

  • Ben Schiller +

  • Benedikt Sauer

  • Benjamin Thyreau +

  • BorisVerk +

  • Chris Reynolds +

  • Chris Stoafer +

  • DSM

  • Dav Clark +

  • FragLegs +

  • German Gomez-Herrero +

  • Hsiaoming Yang +

  • Huan Li +

  • Hyungtae Kim +

  • Isaac Slavitt +

  • Jacob Schaer

  • Jacob Wasserman +

  • Jan Schulz

  • Jeff Reback

  • Jeff Tratner

  • Jesse Farnham +

  • Joe Bradish +

  • Joerg Rittinger +

  • John W. O’Brien

  • Joris Van den Bossche

  • Kevin Sheppard

  • Kyle Meyer

  • Max Chang +

  • Michael Mueller

  • Michael W Schatzow +

  • Mike Kelly

  • Mortada Mehyar

  • Nathan Sanders +

  • Nathan Typanski +

  • Paul Masurel +

  • Phillip Cloud

  • Pietro Battiston

  • RenzoBertocchi +

  • Ross Petchler +

  • Shahul Hameed +

  • Shashank Agarwal +

  • Stephan Hoyer

  • Tom Augspurger

  • TomAugspurger

  • Tony Lorenzo +

  • Wes Turner

  • Wilfred Hughes +

  • Yevgeniy Grechka +

  • Yoshiki Vázquez Baeza +

  • behzad nouri +

  • benjamin

  • bjonen +

  • dlovell +

  • dsm054

  • hunterowens +

  • immerrr

  • ischwabacher

  • jmorris0x0 +

  • jnmclarty +

  • jreback

  • klonuo +

  • lexual

  • mcjcode +

  • mtrbean +

  • onesandzeroes

  • rockg

  • seth-p

  • sinhrks

  • someben +

  • stahlous +

  • stas-sl +

  • thatneat +

  • tom-alcorn +

  • unknown

  • unutbu

  • zachcp +