Version 0.22.0 (December 29, 2017)#

This is a major release from 0.21.1 and includes a single, API-breaking change. We recommend that all users upgrade to this version after carefully reading the release note (singular!).

Backwards incompatible API changes#

pandas 0.22.0 changes the handling of empty and all-NA sums and products. The summary is that

  • The sum of an empty or all-NA Series is now 0

  • The product of an empty or all-NA Series is now 1

  • We’ve added a min_count parameter to .sum() and .prod() controlling the minimum number of valid values for the result to be valid. If fewer than min_count non-NA values are present, the result is NA. The default is 0. To return NaN, the 0.21 behavior, use min_count=1.

Some background: In pandas 0.21, we fixed a long-standing inconsistency in the return value of all-NA series depending on whether or not bottleneck was installed. See Sum/prod of all-NaN or empty Series/DataFrames is now consistently NaN. At the same time, we changed the sum and prod of an empty Series to also be NaN.

Based on feedback, we’ve partially reverted those changes.

Arithmetic operations#

The default sum for empty or all-NA Series is now 0.

pandas 0.21.x

In [1]: pd.Series([]).sum()
Out[1]: nan

In [2]: pd.Series([np.nan]).sum()
Out[2]: nan

pandas 0.22.0

In [1]: pd.Series([]).sum()
Out[1]: 0

In [2]: pd.Series([np.nan]).sum()
Out[2]: 0.0

The default behavior is the same as pandas 0.20.3 with bottleneck installed. It also matches the behavior of NumPy’s np.nansum on empty and all-NA arrays.

To have the sum of an empty series return NaN (the default behavior of pandas 0.20.3 without bottleneck, or pandas 0.21.x), use the min_count keyword.

In [3]: pd.Series([]).sum(min_count=1)
Out[3]: nan

Thanks to the skipna parameter, the .sum on an all-NA series is conceptually the same as the .sum of an empty one with skipna=True (the default).

In [4]: pd.Series([np.nan]).sum(min_count=1)  # skipna=True by default
Out[4]: nan

The min_count parameter refers to the minimum number of non-null values required for a non-NA sum or product.

Series.prod() has been updated to behave the same as Series.sum(), returning 1 instead.

In [5]: pd.Series([]).prod()
Out[5]: 1

In [6]: pd.Series([np.nan]).prod()
Out[6]: 1.0

In [7]: pd.Series([]).prod(min_count=1)
Out[7]: nan

These changes affect DataFrame.sum() and DataFrame.prod() as well. Finally, a few less obvious places in pandas are affected by this change.

Grouping by a Categorical#

Grouping by a Categorical and summing now returns 0 instead of NaN for categories with no observations. The product now returns 1 instead of NaN.

pandas 0.21.x

In [8]: grouper = pd.Categorical(['a', 'a'], categories=['a', 'b'])

In [9]: pd.Series([1, 2]).groupby(grouper, observed=False).sum()
Out[9]:
a    3.0
b    NaN
dtype: float64

pandas 0.22

In [8]: grouper = pd.Categorical(["a", "a"], categories=["a", "b"])

In [9]: pd.Series([1, 2]).groupby(grouper, observed=False).sum()
Out[9]: 
a    3
b    0
dtype: int64

To restore the 0.21 behavior of returning NaN for unobserved groups, use min_count>=1.

In [10]: pd.Series([1, 2]).groupby(grouper, observed=False).sum(min_count=1)
Out[10]: 
a    3.0
b    NaN
dtype: float64

Resample#

The sum and product of all-NA bins has changed from NaN to 0 for sum and 1 for product.

pandas 0.21.x

In [11]: s = pd.Series([1, 1, np.nan, np.nan],
   ....:               index=pd.date_range('2017', periods=4))
   ....: s
Out[11]:
2017-01-01    1.0
2017-01-02    1.0
2017-01-03    NaN
2017-01-04    NaN
Freq: D, dtype: float64

In [12]: s.resample('2d').sum()
Out[12]:
2017-01-01    2.0
2017-01-03    NaN
Freq: 2D, dtype: float64

pandas 0.22.0

In [11]: s = pd.Series([1, 1, np.nan, np.nan],
   ....:               index=pd.date_range("2017", periods=4))

In [12]: s.resample("2d").sum()
Out[12]:
2017-01-01    2.0
2017-01-03    0.0
Freq: 2D, Length: 2, dtype: float64

To restore the 0.21 behavior of returning NaN, use min_count>=1.

In [13]: s.resample("2d").sum(min_count=1)
Out[13]:
2017-01-01    2.0
2017-01-03    NaN
Freq: 2D, Length: 2, dtype: float64

In particular, upsampling and taking the sum or product is affected, as upsampling introduces missing values even if the original series was entirely valid.

pandas 0.21.x

In [14]: idx = pd.DatetimeIndex(['2017-01-01', '2017-01-02'])

In [15]: pd.Series([1, 2], index=idx).resample('12H').sum()
Out[15]:
2017-01-01 00:00:00    1.0
2017-01-01 12:00:00    NaN
2017-01-02 00:00:00    2.0
Freq: 12H, dtype: float64

pandas 0.22.0

In [14]: idx = pd.DatetimeIndex(["2017-01-01", "2017-01-02"])
In [15]: pd.Series([1, 2], index=idx).resample("12H").sum()
Out[15]:
2017-01-01 00:00:00    1
2017-01-01 12:00:00    0
2017-01-02 00:00:00    2
Freq: 12H, Length: 3, dtype: int64

Once again, the min_count keyword is available to restore the 0.21 behavior.

In [16]: pd.Series([1, 2], index=idx).resample("12H").sum(min_count=1)
Out[16]:
2017-01-01 00:00:00    1.0
2017-01-01 12:00:00    NaN
2017-01-02 00:00:00    2.0
Freq: 12H, Length: 3, dtype: float64

Rolling and expanding#

Rolling and expanding already have a min_periods keyword that behaves similar to min_count. The only case that changes is when doing a rolling or expanding sum with min_periods=0. Previously this returned NaN, when fewer than min_periods non-NA values were in the window. Now it returns 0.

pandas 0.21.1

In [17]: s = pd.Series([np.nan, np.nan])

In [18]: s.rolling(2, min_periods=0).sum()
Out[18]:
0   NaN
1   NaN
dtype: float64

pandas 0.22.0

In [11]: s = pd.Series([np.nan, np.nan])

In [12]: s.rolling(2, min_periods=0).sum()
Out[12]: 
0    0.0
1    0.0
dtype: float64

The default behavior of min_periods=None, implying that min_periods equals the window size, is unchanged.

Compatibility#

If you maintain a library that should work across pandas versions, it may be easiest to exclude pandas 0.21 from your requirements. Otherwise, all your sum() calls would need to check if the Series is empty before summing.

With setuptools, in your setup.py use:

install_requires=['pandas!=0.21.*', ...]

With conda, use

requirements:
  run:
    - pandas !=0.21.0,!=0.21.1

Note that the inconsistency in the return value for all-NA series is still there for pandas 0.20.3 and earlier. Avoiding pandas 0.21 will only help with the empty case.

Contributors#

A total of 1 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

  • Tom Augspurger