Version 0.13.0 (January 3, 2014)#

This is a major release from 0.12.0 and includes a number of API changes, several new features and enhancements along with a large number of bug fixes.

Highlights include:

  • support for a new index type Float64Index, and other Indexing enhancements

  • HDFStore has a new string based syntax for query specification

  • support for new methods of interpolation

  • updated timedelta operations

  • a new string manipulation method extract

  • Nanosecond support for Offsets

  • isin for DataFrames

Several experimental features are added, including:

  • new eval/query methods for expression evaluation

  • support for msgpack serialization

  • an i/o interface to Google’s BigQuery

Their are several new or updated docs sections including:

Warning

In 0.13.0 Series has internally been refactored to no longer sub-class ndarray but instead subclass NDFrame, similar to the rest of the pandas containers. This should be a transparent change with only very limited API implications. See Internal Refactoring

API changes#

  • read_excel now supports an integer in its sheetname argument giving the index of the sheet to read in (GH 4301).

  • Text parser now treats anything that reads like inf (“inf”, “Inf”, “-Inf”, “iNf”, etc.) as infinity. (GH 4220, GH 4219), affecting read_table, read_csv, etc.

  • pandas now is Python 2/3 compatible without the need for 2to3 thanks to @jtratner. As a result, pandas now uses iterators more extensively. This also led to the introduction of substantive parts of the Benjamin Peterson’s six library into compat. (GH 4384, GH 4375, GH 4372)

  • pandas.util.compat and pandas.util.py3compat have been merged into pandas.compat. pandas.compat now includes many functions allowing 2/3 compatibility. It contains both list and iterator versions of range, filter, map and zip, plus other necessary elements for Python 3 compatibility. lmap, lzip, lrange and lfilter all produce lists instead of iterators, for compatibility with numpy, subscripting and pandas constructors.(GH 4384, GH 4375, GH 4372)

  • Series.get with negative indexers now returns the same as [] (GH 4390)

  • Changes to how Index and MultiIndex handle metadata (levels, labels, and names) (GH 4039):

    # previously, you would have set levels or labels directly
>>> pd.index.levels = [[1, 2, 3, 4], [1, 2, 4, 4]]

# now, you use the set_levels or set_labels methods
>>> index = pd.index.set_levels([[1, 2, 3, 4], [1, 2, 4, 4]])

# similarly, for names, you can rename the object
# but setting names is not deprecated
>>> index = pd.index.set_names(["bob", "cranberry"])

# and all methods take an inplace kwarg - but return None
>>> pd.index.set_names(["bob", "cranberry"], inplace=True)

  • All division with NDFrame objects is now truedivision, regardless of the future import. This means that operating on pandas objects will by default use floating point division, and return a floating point dtype. You can use // and floordiv to do integer division.

    Integer division

    In [3]: arr = np.array([1, 2, 3, 4])

In [4]: arr2 = np.array([5, 3, 2, 1])

In [5]: arr / arr2
Out[5]: array([0, 0, 1, 4])

In [6]: pd.Series(arr) // pd.Series(arr2)
Out[6]:
0    0
1    0
2    1
3    4
dtype: int64

    True Division

    In [7]: pd.Series(arr) / pd.Series(arr2)  # no future import required
Out[7]:
0    0.200000
1    0.666667
2    1.500000
3    4.000000
dtype: float64

  • Infer and downcast dtype if downcast='infer' is passed to fillna/ffill/bfill (GH 4604)

  • __nonzero__ for all NDFrame objects, will now raise a ValueError, this reverts back to (GH 1073, GH 4633) behavior. See gotchas for a more detailed discussion.

    This prevents doing boolean comparison on entire pandas objects, which is inherently ambiguous. These all will raise a ValueError.

    >>> df = pd.DataFrame({'A': np.random.randn(10),
...                    'B': np.random.randn(10),
...                    'C': pd.date_range('20130101', periods=10)
...                    })
...
>>> if df:
...     pass
...
Traceback (most recent call last):
    ...
ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,
a.bool(), a.item(), a.any() or a.all().

>>> df1 = df
>>> df2 = df
>>> df1 and df2
Traceback (most recent call last):
    ...
ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,
a.bool(), a.item(), a.any() or a.all().

>>> d = [1, 2, 3]
>>> s1 = pd.Series(d)
>>> s2 = pd.Series(d)
>>> s1 and s2
Traceback (most recent call last):
    ...
ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,
a.bool(), a.item(), a.any() or a.all().

    Added the .bool() method to NDFrame objects to facilitate evaluating of single-element boolean Series:

    >>> pd.Series([True]).bool()
 True
>>> pd.Series([False]).bool()
 False
>>> pd.DataFrame([[True]]).bool()
 True
>>> pd.DataFrame([[False]]).bool()
 False

  • All non-Index NDFrames (Series, DataFrame, Panel, Panel4D, SparsePanel, etc.), now support the entire set of arithmetic operators and arithmetic flex methods (add, sub, mul, etc.). SparsePanel does not support pow or mod with non-scalars. (GH 3765)

  • Series and DataFrame now have a mode() method to calculate the statistical mode(s) by axis/Series. (GH 5367)

  • Chained assignment will now by default warn if the user is assigning to a copy. This can be changed with the option mode.chained_assignment, allowed options are raise/warn/None. See the docs.

    In [1]: dfc = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]})

In [2]: pd.set_option('chained_assignment', 'warn')

    The following warning / exception will show if this is attempted.

    In [3]: dfc.loc[0]['A'] = 1111

    Traceback (most recent call last)
   ...
SettingWithCopyWarning:
   A value is trying to be set on a copy of a slice from a DataFrame.
   Try using .loc[row_index,col_indexer] = value instead

    Here is the correct method of assignment.

    In [4]: dfc.loc[0, 'A'] = 11

In [5]: dfc
Out[5]: 
     A  B
0   11  1
1  bbb  2
2  ccc  3
  • Panel.reindex has the following call signature Panel.reindex(items=None, major_axis=None, minor_axis=None, **kwargs)

    to conform with other NDFrame objects. See Internal Refactoring for more information.

  • Series.argmin and Series.argmax are now aliased to Series.idxmin and Series.idxmax. These return the index of the

    min or max element respectively. Prior to 0.13.0 these would return the position of the min / max element. (GH 6214)

Prior version deprecations/changes#

These were announced changes in 0.12 or prior that are taking effect as of 0.13.0

  • Remove deprecated Factor (GH 3650)

  • Remove deprecated set_printoptions/reset_printoptions (GH 3046)

  • Remove deprecated _verbose_info (GH 3215)

  • Remove deprecated read_clipboard/to_clipboard/ExcelFile/ExcelWriter from pandas.io.parsers (GH 3717) These are available as functions in the main pandas namespace (e.g. pd.read_clipboard)

  • default for tupleize_cols is now False for both to_csv and read_csv. Fair warning in 0.12 (GH 3604)

  • default for display.max_seq_len is now 100 rather than None. This activates truncated display (”…”) of long sequences in various places. (GH 3391)

Deprecations#

Deprecated in 0.13.0

  • deprecated iterkv, which will be removed in a future release (this was an alias of iteritems used to bypass 2to3’s changes). (GH 4384, GH 4375, GH 4372)

  • deprecated the string method match, whose role is now performed more idiomatically by extract. In a future release, the default behavior of match will change to become analogous to contains, which returns a boolean indexer. (Their distinction is strictness: match relies on re.match while contains relies on re.search.) In this release, the deprecated behavior is the default, but the new behavior is available through the keyword argument as_indexer=True.

Indexing API changes#

Prior to 0.13, it was impossible to use a label indexer (.loc/.ix) to set a value that was not contained in the index of a particular axis. (GH 2578). See the docs

In the Series case this is effectively an appending operation

In [6]: s = pd.Series([1, 2, 3])

In [7]: s
Out[7]: 
0    1
1    2
2    3
dtype: int64

In [8]: s[5] = 5.

In [9]: s
Out[9]: 
0    1.0
1    2.0
2    3.0
5    5.0
dtype: float64

In [10]: dfi = pd.DataFrame(np.arange(6).reshape(3, 2),
   ....:                    columns=['A', 'B'])
   ....: 

In [11]: dfi
Out[11]: 
   A  B
0  0  1
1  2  3
2  4  5

This would previously KeyError

In [12]: dfi.loc[:, 'C'] = dfi.loc[:, 'A']

In [13]: dfi
Out[13]: 
   A  B  C
0  0  1  0
1  2  3  2
2  4  5  4

This is like an append operation.

In [14]: dfi.loc[3] = 5

In [15]: dfi
Out[15]: 
   A  B  C
0  0  1  0
1  2  3  2
2  4  5  4
3  5  5  5

A Panel setting operation on an arbitrary axis aligns the input to the Panel

In [20]: p = pd.Panel(np.arange(16).reshape(2, 4, 2),
   ....:              items=['Item1', 'Item2'],
   ....:              major_axis=pd.date_range('2001/1/12', periods=4),
   ....:              minor_axis=['A', 'B'], dtype='float64')
   ....:

In [21]: p
Out[21]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to B

In [22]: p.loc[:, :, 'C'] = pd.Series([30, 32], index=p.items)

In [23]: p
Out[23]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to C

In [24]: p.loc[:, :, 'C']
Out[24]:
            Item1  Item2
2001-01-12   30.0   32.0
2001-01-13   30.0   32.0
2001-01-14   30.0   32.0
2001-01-15   30.0   32.0

Float64Index API change#

  • Added a new index type, Float64Index. This will be automatically created when passing floating values in index creation. This enables a pure label-based slicing paradigm that makes [],ix,loc for scalar indexing and slicing work exactly the same. (GH 263)

    Construction is by default for floating type values.

    In [16]: index = pd.Index([1.5, 2, 3, 4.5, 5])

In [17]: index
Out[17]: Index([1.5, 2.0, 3.0, 4.5, 5.0], dtype='float64')

In [18]: s = pd.Series(range(5), index=index)

In [19]: s
Out[19]: 
1.5    0
2.0    1
3.0    2
4.5    3
5.0    4
dtype: int64

    Scalar selection for [],.ix,.loc will always be label based. An integer will match an equal float index (e.g. 3 is equivalent to 3.0)

    In [20]: s[3]
Out[20]: 2

In [21]: s.loc[3]
Out[21]: 2

    The only positional indexing is via iloc

    In [22]: s.iloc[3]
Out[22]: 3

    A scalar index that is not found will raise KeyError

    Slicing is ALWAYS on the values of the index, for [],ix,loc and ALWAYS positional with iloc

    In [23]: s[2:4]
Out[23]: 
2.0    1
3.0    2
dtype: int64

In [24]: s.loc[2:4]
Out[24]: 
2.0    1
3.0    2
dtype: int64

In [25]: s.iloc[2:4]
Out[25]: 
3.0    2
4.5    3
dtype: int64

    In float indexes, slicing using floats are allowed

    In [26]: s[2.1:4.6]
Out[26]: 
3.0    2
4.5    3
dtype: int64

In [27]: s.loc[2.1:4.6]
Out[27]: 
3.0    2
4.5    3
dtype: int64

  • Indexing on other index types are preserved (and positional fallback for [],ix), with the exception, that floating point slicing on indexes on non Float64Index will now raise a TypeError.

    In [1]: pd.Series(range(5))[3.5]
TypeError: the label [3.5] is not a proper indexer for this index type (Int64Index)

In [1]: pd.Series(range(5))[3.5:4.5]
TypeError: the slice start [3.5] is not a proper indexer for this index type (Int64Index)

    Using a scalar float indexer will be deprecated in a future version, but is allowed for now.

    In [3]: pd.Series(range(5))[3.0]
Out[3]: 3

HDFStore API changes#

  • Query Format Changes. A much more string-like query format is now supported. See the docs.

    In [28]: path = 'test.h5'

In [29]: dfq = pd.DataFrame(np.random.randn(10, 4),
   ....:                    columns=list('ABCD'),
   ....:                    index=pd.date_range('20130101', periods=10))
   ....: 

In [30]: dfq.to_hdf(path, 'dfq', format='table', data_columns=True)

    Use boolean expressions, with in-line function evaluation.

    In [31]: pd.read_hdf(path, 'dfq',
   ....:             where="index>Timestamp('20130104') & columns=['A', 'B']")
   ....: 
Out[31]: 
                   A         B
2013-01-05 -0.424972  0.567020
2013-01-06 -0.673690  0.113648
2013-01-07  0.404705  0.577046
2013-01-08 -0.370647 -1.157892
2013-01-09  1.075770 -0.109050
2013-01-10  0.357021 -0.674600

    Use an inline column reference

    In [32]: pd.read_hdf(path, 'dfq',
   ....:             where="A>0 or C>0")
   ....: 
Out[32]: 
                   A         B         C         D
2013-01-01  0.469112 -0.282863 -1.509059 -1.135632
2013-01-02  1.212112 -0.173215  0.119209 -1.044236
2013-01-04  0.721555 -0.706771 -1.039575  0.271860
2013-01-05 -0.424972  0.567020  0.276232 -1.087401
2013-01-07  0.404705  0.577046 -1.715002 -1.039268
2013-01-09  1.075770 -0.109050  1.643563 -1.469388
2013-01-10  0.357021 -0.674600 -1.776904 -0.968914

  • the format keyword now replaces the table keyword; allowed values are fixed(f) or table(t) the same defaults as prior < 0.13.0 remain, e.g. put implies fixed format and append implies table format. This default format can be set as an option by setting io.hdf.default_format.

    In [33]: path = 'test.h5'

In [34]: df = pd.DataFrame(np.random.randn(10, 2))

In [35]: df.to_hdf(path, 'df_table', format='table')

In [36]: df.to_hdf(path, 'df_table2', append=True)

In [37]: df.to_hdf(path, 'df_fixed')

In [38]: with pd.HDFStore(path) as store:
   ....:     print(store)
   ....: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5

  • Significant table writing performance improvements

  • handle a passed Series in table format (GH 4330)

  • can now serialize a timedelta64[ns] dtype in a table (GH 3577), See the docs.

  • added an is_open property to indicate if the underlying file handle is_open; a closed store will now report ‘CLOSED’ when viewing the store (rather than raising an error) (GH 4409)

  • a close of a HDFStore now will close that instance of the HDFStore but will only close the actual file if the ref count (by PyTables) w.r.t. all of the open handles are 0. Essentially you have a local instance of HDFStore referenced by a variable. Once you close it, it will report closed. Other references (to the same file) will continue to operate until they themselves are closed. Performing an action on a closed file will raise ClosedFileError

    In [39]: path = 'test.h5'

In [40]: df = pd.DataFrame(np.random.randn(10, 2))

In [41]: store1 = pd.HDFStore(path)

In [42]: store2 = pd.HDFStore(path)

In [43]: store1.append('df', df)

In [44]: store2.append('df2', df)

In [45]: store1
Out[45]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5

In [46]: store2
Out[46]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5

In [47]: store1.close()

In [48]: store2
Out[48]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5

In [49]: store2.close()

In [50]: store2
Out[50]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5

  • removed the _quiet attribute, replace by a DuplicateWarning if retrieving duplicate rows from a table (GH 4367)

  • removed the warn argument from open. Instead a PossibleDataLossError exception will be raised if you try to use mode='w' with an OPEN file handle (GH 4367)

  • allow a passed locations array or mask as a where condition (GH 4467). See the docs for an example.

  • add the keyword dropna=True to append to change whether ALL nan rows are not written to the store (default is True, ALL nan rows are NOT written), also settable via the option io.hdf.dropna_table (GH 4625)

  • pass through store creation arguments; can be used to support in-memory stores

DataFrame repr changes#

The HTML and plain text representations of DataFrame now show a truncated view of the table once it exceeds a certain size, rather than switching to the short info view (GH 4886, GH 5550). This makes the representation more consistent as small DataFrames get larger.

Truncated HTML representation of a DataFrame

To get the info view, call DataFrame.info(). If you prefer the info view as the repr for large DataFrames, you can set this by running set_option('display.large_repr', 'info').

Enhancements#

  • df.to_clipboard() learned a new excel keyword that let’s you paste df data directly into excel (enabled by default). (GH 5070).

  • read_html now raises a URLError instead of catching and raising a ValueError (GH 4303, GH 4305)

  • Added a test for read_clipboard() and to_clipboard() (GH 4282)

  • Clipboard functionality now works with PySide (GH 4282)

  • Added a more informative error message when plot arguments contain overlapping color and style arguments (GH 4402)

  • to_dict now takes records as a possible out type. Returns an array of column-keyed dictionaries. (GH 4936)

  • NaN handing in get_dummies (GH 4446) with dummy_na

    # previously, nan was erroneously counted as 2 here
# now it is not counted at all
In [51]: pd.get_dummies([1, 2, np.nan])
Out[51]: 
     1.0    2.0
0   True  False
1  False   True
2  False  False

# unless requested
In [52]: pd.get_dummies([1, 2, np.nan], dummy_na=True)
Out[52]: 
     1.0    2.0    NaN
0   True  False  False
1  False   True  False
2  False  False   True

  • timedelta64[ns] operations. See the docs.

    Warning

    Most of these operations require numpy >= 1.7

    Using the new top-level to_timedelta, you can convert a scalar or array from the standard timedelta format (produced by to_csv) into a timedelta type (np.timedelta64 in nanoseconds).

    In [53]: pd.to_timedelta('1 days 06:05:01.00003')
Out[53]: Timedelta('1 days 06:05:01.000030')

In [54]: pd.to_timedelta('15.5us')
Out[54]: Timedelta('0 days 00:00:00.000015500')

In [55]: pd.to_timedelta(['1 days 06:05:01.00003', '15.5us', 'nan'])
Out[55]: TimedeltaIndex(['1 days 06:05:01.000030', '0 days 00:00:00.000015500', NaT], dtype='timedelta64[ns]', freq=None)

In [56]: pd.to_timedelta(np.arange(5), unit='s')
Out[56]: 
TimedeltaIndex(['0 days 00:00:00', '0 days 00:00:01', '0 days 00:00:02',
                '0 days 00:00:03', '0 days 00:00:04'],
               dtype='timedelta64[ns]', freq=None)

In [57]: pd.to_timedelta(np.arange(5), unit='d')
Out[57]: TimedeltaIndex(['0 days', '1 days', '2 days', '3 days', '4 days'], dtype='timedelta64[ns]', freq=None)

    A Series of dtype timedelta64[ns] can now be divided by another timedelta64[ns] object, or astyped to yield a float64 dtyped Series. This is frequency conversion. See the docs for the docs.

    In [58]: import datetime

In [59]: td = pd.Series(pd.date_range('20130101', periods=4)) - pd.Series(
   ....:     pd.date_range('20121201', periods=4))
   ....: 

In [60]: td[2] += np.timedelta64(datetime.timedelta(minutes=5, seconds=3))

In [61]: td[3] = np.nan

In [62]: td
Out[62]: 
0   31 days 00:00:00
1   31 days 00:00:00
2   31 days 00:05:03
3                NaT
dtype: timedelta64[ns]

# to days
In [63]: td / np.timedelta64(1, 'D')
Out[63]: 
0    31.000000
1    31.000000
2    31.003507
3          NaN
dtype: float64

In [64]: td.astype('timedelta64[D]')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[64], line 1
----> 1 td.astype('timedelta64[D]')

File ~/work/pandas/pandas/pandas/core/generic.py:6534, in NDFrame.astype(self, dtype, copy, errors)
   6530     results = [ser.astype(dtype, copy=copy) for _, ser in self.items()]
   6532 else:
   6533     # else, only a single dtype is given
-> 6534     new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   6535     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6536     return res.__finalize__(self, method="astype")

File ~/work/pandas/pandas/pandas/core/internals/managers.py:414, in BaseBlockManager.astype(self, dtype, copy, errors)
    411 elif using_copy_on_write():
    412     copy = False
--> 414 return self.apply(
    415     "astype",
    416     dtype=dtype,
    417     copy=copy,
    418     errors=errors,
    419     using_cow=using_copy_on_write(),
    420 )

File ~/work/pandas/pandas/pandas/core/internals/managers.py:354, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    352         applied = b.apply(f, **kwargs)
    353     else:
--> 354         applied = getattr(b, f)(**kwargs)
    355     result_blocks = extend_blocks(applied, result_blocks)
    357 out = type(self).from_blocks(result_blocks, self.axes)

File ~/work/pandas/pandas/pandas/core/internals/blocks.py:616, in Block.astype(self, dtype, copy, errors, using_cow)
    596 """
    597 Coerce to the new dtype.
    598 
   (...)
    612 Block
    613 """
    614 values = self.values
--> 616 new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
    618 new_values = maybe_coerce_values(new_values)
    620 refs = None

File ~/work/pandas/pandas/pandas/core/dtypes/astype.py:238, in astype_array_safe(values, dtype, copy, errors)
    235     dtype = dtype.numpy_dtype
    237 try:
--> 238     new_values = astype_array(values, dtype, copy=copy)
    239 except (ValueError, TypeError):
    240     # e.g. _astype_nansafe can fail on object-dtype of strings
    241     #  trying to convert to float
    242     if errors == "ignore":

File ~/work/pandas/pandas/pandas/core/dtypes/astype.py:180, in astype_array(values, dtype, copy)
    176     return values
    178 if not isinstance(values, np.ndarray):
    179     # i.e. ExtensionArray
--> 180     values = values.astype(dtype, copy=copy)
    182 else:
    183     values = _astype_nansafe(values, dtype, copy=copy)

File ~/work/pandas/pandas/pandas/core/arrays/timedeltas.py:380, in TimedeltaArray.astype(self, dtype, copy)
    376         return type(self)._simple_new(
    377             res_values, dtype=res_values.dtype, freq=self.freq
    378         )
    379     else:
--> 380         raise ValueError(
    381             f"Cannot convert from {self.dtype} to {dtype}. "
    382             "Supported resolutions are 's', 'ms', 'us', 'ns'"
    383         )
    385 return dtl.DatetimeLikeArrayMixin.astype(self, dtype, copy=copy)

ValueError: Cannot convert from timedelta64[ns] to timedelta64[D]. Supported resolutions are 's', 'ms', 'us', 'ns'

# to seconds
In [65]: td / np.timedelta64(1, 's')
Out[65]: 
0    2678400.0
1    2678400.0
2    2678703.0
3          NaN
dtype: float64

In [66]: td.astype('timedelta64[s]')
Out[66]: 
0   31 days 00:00:00
1   31 days 00:00:00
2   31 days 00:05:03
3                NaT
dtype: timedelta64[s]

    Dividing or multiplying a timedelta64[ns] Series by an integer or integer Series

    In [67]: td * -1
Out[67]: 
0   -31 days +00:00:00
1   -31 days +00:00:00
2   -32 days +23:54:57
3                  NaT
dtype: timedelta64[ns]

In [68]: td * pd.Series([1, 2, 3, 4])
Out[68]: 
0   31 days 00:00:00
1   62 days 00:00:00
2   93 days 00:15:09
3                NaT
dtype: timedelta64[ns]

    Absolute DateOffset objects can act equivalently to timedeltas

    In [69]: from pandas import offsets

In [70]: td + offsets.Minute(5) + offsets.Milli(5)
Out[70]: 
0   31 days 00:05:00.005000
1   31 days 00:05:00.005000
2   31 days 00:10:03.005000
3                       NaT
dtype: timedelta64[ns]

    Fillna is now supported for timedeltas

    In [71]: td.fillna(pd.Timedelta(0))
Out[71]: 
0   31 days 00:00:00
1   31 days 00:00:00
2   31 days 00:05:03
3    0 days 00:00:00
dtype: timedelta64[ns]

In [72]: td.fillna(datetime.timedelta(days=1, seconds=5))
Out[72]: 
0   31 days 00:00:00
1   31 days 00:00:00
2   31 days 00:05:03
3    1 days 00:00:05
dtype: timedelta64[ns]

    You can do numeric reduction operations on timedeltas.

    In [73]: td.mean()
Out[73]: Timedelta('31 days 00:01:41')

In [74]: td.quantile(.1)
Out[74]: Timedelta('31 days 00:00:00')

  • plot(kind='kde') now accepts the optional parameters bw_method and ind, passed to scipy.stats.gaussian_kde() (for scipy >= 0.11.0) to set the bandwidth, and to gkde.evaluate() to specify the indices at which it is evaluated, respectively. See scipy docs. (GH 4298)

  • DataFrame constructor now accepts a numpy masked record array (GH 3478)

  • The new vectorized string method extract return regular expression matches more conveniently.

    In [75]: pd.Series(['a1', 'b2', 'c3']).str.extract('[ab](\\d)')
Out[75]: 
     0
0    1
1    2
2  NaN

    Elements that do not match return NaN. Extracting a regular expression with more than one group returns a DataFrame with one column per group.

    In [76]: pd.Series(['a1', 'b2', 'c3']).str.extract('([ab])(\\d)')
Out[76]: 
     0    1
0    a    1
1    b    2
2  NaN  NaN

    Elements that do not match return a row of NaN. Thus, a Series of messy strings can be converted into a like-indexed Series or DataFrame of cleaned-up or more useful strings, without necessitating get() to access tuples or re.match objects.

    Named groups like

    In [77]: pd.Series(['a1', 'b2', 'c3']).str.extract(
   ....:     '(?P<letter>[ab])(?P<digit>\\d)')
   ....: 
Out[77]: 
  letter digit
0      a     1
1      b     2
2    NaN   NaN

    and optional groups can also be used.

    In [78]: pd.Series(['a1', 'b2', '3']).str.extract(
   ....:      '(?P<letter>[ab])?(?P<digit>\\d)')
   ....: 
Out[78]: 
  letter digit
0      a     1
1      b     2
2    NaN     3

  • read_stata now accepts Stata 13 format (GH 4291)

  • read_fwf now infers the column specifications from the first 100 rows of the file if the data has correctly separated and properly aligned columns using the delimiter provided to the function (GH 4488).

  • support for nanosecond times as an offset

    Warning

    These operations require numpy >= 1.7

    Period conversions in the range of seconds and below were reworked and extended up to nanoseconds. Periods in the nanosecond range are now available.

    In [79]: pd.date_range('2013-01-01', periods=5, freq='5N')
Out[79]: 
DatetimeIndex([          '2013-01-01 00:00:00',
               '2013-01-01 00:00:00.000000005',
               '2013-01-01 00:00:00.000000010',
               '2013-01-01 00:00:00.000000015',
               '2013-01-01 00:00:00.000000020'],
              dtype='datetime64[ns]', freq='5N')

    or with frequency as offset

    In [80]: pd.date_range('2013-01-01', periods=5, freq=pd.offsets.Nano(5))
Out[80]: 
DatetimeIndex([          '2013-01-01 00:00:00',
               '2013-01-01 00:00:00.000000005',
               '2013-01-01 00:00:00.000000010',
               '2013-01-01 00:00:00.000000015',
               '2013-01-01 00:00:00.000000020'],
              dtype='datetime64[ns]', freq='5N')

    Timestamps can be modified in the nanosecond range

    In [81]: t = pd.Timestamp('20130101 09:01:02')

In [82]: t + pd.tseries.offsets.Nano(123)
Out[82]: Timestamp('2013-01-01 09:01:02.000000123')

  • A new method, isin for DataFrames, which plays nicely with boolean indexing. The argument to isin, what we’re comparing the DataFrame to, can be a DataFrame, Series, dict, or array of values. See the docs for more.

    To get the rows where any of the conditions are met:

    In [83]: dfi = pd.DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'f', 'n']})

In [84]: dfi
Out[84]: 
   A  B
0  1  a
1  2  b
2  3  f
3  4  n

In [85]: other = pd.DataFrame({'A': [1, 3, 3, 7], 'B': ['e', 'f', 'f', 'e']})

In [86]: mask = dfi.isin(other)

In [87]: mask
Out[87]: 
       A      B
0   True  False
1  False  False
2   True   True
3  False  False

In [88]: dfi[mask.any(axis=1)]
Out[88]: 
   A  B
0  1  a
2  3  f

  • Series now supports a to_frame method to convert it to a single-column DataFrame (GH 5164)

  • All R datasets listed here http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html can now be loaded into pandas objects

    # note that pandas.rpy was deprecated in v0.16.0
import pandas.rpy.common as com
com.load_data('Titanic')

  • tz_localize can infer a fall daylight savings transition based on the structure of the unlocalized data (GH 4230), see the docs

  • DatetimeIndex is now in the API documentation, see the docs

  • json_normalize() is a new method to allow you to create a flat table from semi-structured JSON data. See the docs (GH 1067)

  • Added PySide support for the qtpandas DataFrameModel and DataFrameWidget.

  • Python csv parser now supports usecols (GH 4335)

  • Frequencies gained several new offsets:

  • DataFrame has a new interpolate method, similar to Series (GH 4434, GH 1892)

    In [89]: df = pd.DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8],
   ....:                   'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]})
   ....: 

In [90]: df.interpolate()
Out[90]: 
     A      B
0  1.0   0.25
1  2.1   1.50
2  3.4   2.75
3  4.7   4.00
4  5.6  12.20
5  6.8  14.40

    Additionally, the method argument to interpolate has been expanded to include 'nearest', 'zero', 'slinear', 'quadratic', 'cubic', 'barycentric', 'krogh', 'piecewise_polynomial', 'pchip', 'polynomial', 'spline' The new methods require scipy. Consult the Scipy reference guide and documentation for more information about when the various methods are appropriate. See the docs.

    Interpolate now also accepts a limit keyword argument. This works similar to fillna’s limit:

    In [91]: ser = pd.Series([1, 3, np.nan, np.nan, np.nan, 11])

In [92]: ser.interpolate(limit=2)
Out[92]: 
0     1.0
1     3.0
2     5.0
3     7.0
4     NaN
5    11.0
dtype: float64

  • Added wide_to_long panel data convenience function. See the docs.

    In [93]: np.random.seed(123)

In [94]: df = pd.DataFrame({"A1970" : {0 : "a", 1 : "b", 2 : "c"},
   ....:                    "A1980" : {0 : "d", 1 : "e", 2 : "f"},
   ....:                    "B1970" : {0 : 2.5, 1 : 1.2, 2 : .7},
   ....:                    "B1980" : {0 : 3.2, 1 : 1.3, 2 : .1},
   ....:                    "X"     : dict(zip(range(3), np.random.randn(3)))
   ....:                   })
   ....: 

In [95]: df["id"] = df.index

In [96]: df
Out[96]: 
  A1970 A1980  B1970  B1980         X  id
0     a     d    2.5    3.2 -1.085631   0
1     b     e    1.2    1.3  0.997345   1
2     c     f    0.7    0.1  0.282978   2

In [97]: pd.wide_to_long(df, ["A", "B"], i="id", j="year")
Out[97]: 
                X  A    B
id year                  
0  1970 -1.085631  a  2.5
1  1970  0.997345  b  1.2
2  1970  0.282978  c  0.7
0  1980 -1.085631  d  3.2
1  1980  0.997345  e  1.3
2  1980  0.282978  f  0.1

  • to_csv now takes a date_format keyword argument that specifies how output datetime objects should be formatted. Datetimes encountered in the index, columns, and values will all have this formatting applied. (GH 4313)

  • DataFrame.plot will scatter plot x versus y by passing kind='scatter' (GH 2215)

  • Added support for Google Analytics v3 API segment IDs that also supports v2 IDs. (GH 5271)

Experimental#

  • The new eval() function implements expression evaluation using numexpr behind the scenes. This results in large speedups for complicated expressions involving large DataFrames/Series. For example,

    In [98]: nrows, ncols = 20000, 100

In [99]: df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols))
   ....:                       for _ in range(4)]
   ....: 

    # eval with NumExpr backend
In [100]: %timeit pd.eval('df1 + df2 + df3 + df4')
10.9 ms +- 489 us per loop (mean +- std. dev. of 7 runs, 100 loops each)

    # pure Python evaluation
In [101]: %timeit df1 + df2 + df3 + df4
10.7 ms +- 600 us per loop (mean +- std. dev. of 7 runs, 100 loops each)

    For more details, see the the docs

  • Similar to pandas.eval, DataFrame has a new DataFrame.eval method that evaluates an expression in the context of the DataFrame. For example,

    In [102]: df = pd.DataFrame(np.random.randn(10, 2), columns=['a', 'b'])

In [103]: df.eval('a + b')
Out[103]: 
0   -0.685204
1    1.589745
2    0.325441
3   -1.784153
4   -0.432893
5    0.171850
6    1.895919
7    3.065587
8   -0.092759
9    1.391365
dtype: float64

  • query() method has been added that allows you to select elements of a DataFrame using a natural query syntax nearly identical to Python syntax. For example,

    In [104]: n = 20

In [105]: df = pd.DataFrame(np.random.randint(n, size=(n, 3)), columns=['a', 'b', 'c'])

In [106]: df.query('a < b < c')
Out[106]: 
    a   b   c
11  1   5   8
15  8  16  19

    selects all the rows of df where a < b < c evaluates to True. For more details see the the docs.

  • pd.read_msgpack() and pd.to_msgpack() are now a supported method of serialization of arbitrary pandas (and python objects) in a lightweight portable binary format. See the docs

    Warning

    Since this is an EXPERIMENTAL LIBRARY, the storage format may not be stable until a future release.

    df = pd.DataFrame(np.random.rand(5, 2), columns=list('AB'))
df.to_msgpack('foo.msg')
pd.read_msgpack('foo.msg')

s = pd.Series(np.random.rand(5), index=pd.date_range('20130101', periods=5))
pd.to_msgpack('foo.msg', df, s)
pd.read_msgpack('foo.msg')

    You can pass iterator=True to iterator over the unpacked results

    for o in pd.read_msgpack('foo.msg', iterator=True):
    print(o)

  • pandas.io.gbq provides a simple way to extract from, and load data into, Google’s BigQuery Data Sets by way of pandas DataFrames. BigQuery is a high performance SQL-like database service, useful for performing ad-hoc queries against extremely large datasets. See the docs

    from pandas.io import gbq

# A query to select the average monthly temperatures in the
# in the year 2000 across the USA. The dataset,
# publicata:samples.gsod, is available on all BigQuery accounts,
# and is based on NOAA gsod data.

query = """SELECT station_number as STATION,
month as MONTH, AVG(mean_temp) as MEAN_TEMP
FROM publicdata:samples.gsod
WHERE YEAR = 2000
GROUP BY STATION, MONTH
ORDER BY STATION, MONTH ASC"""

# Fetch the result set for this query

# Your Google BigQuery Project ID
# To find this, see your dashboard:
# https://console.developers.google.com/iam-admin/projects?authuser=0
projectid = 'xxxxxxxxx'
df = gbq.read_gbq(query, project_id=projectid)

# Use pandas to process and reshape the dataset

df2 = df.pivot(index='STATION', columns='MONTH', values='MEAN_TEMP')
df3 = pd.concat([df2.min(), df2.mean(), df2.max()],
                axis=1, keys=["Min Tem", "Mean Temp", "Max Temp"])

    The resulting DataFrame is:

    > df3
            Min Tem  Mean Temp    Max Temp
 MONTH
 1     -53.336667  39.827892   89.770968
 2     -49.837500  43.685219   93.437932
 3     -77.926087  48.708355   96.099998
 4     -82.892858  55.070087   97.317240
 5     -92.378261  61.428117  102.042856
 6     -77.703334  65.858888  102.900000
 7     -87.821428  68.169663  106.510714
 8     -89.431999  68.614215  105.500000
 9     -86.611112  63.436935  107.142856
 10    -78.209677  56.880838   92.103333
 11    -50.125000  48.861228   94.996428
 12    -50.332258  42.286879   94.396774

    Warning

    To use this module, you will need a BigQuery account. See <https://cloud.google.com/products/big-query> for details.

    As of 10/10/13, there is a bug in Google’s API preventing result sets from being larger than 100,000 rows. A patch is scheduled for the week of 10/14/13.

Internal refactoring#

In 0.13.0 there is a major refactor primarily to subclass Series from NDFrame, which is the base class currently for DataFrame and Panel, to unify methods and behaviors. Series formerly subclassed directly from ndarray. (GH 4080, GH 3862, GH 816)

Warning

There are two potential incompatibilities from < 0.13.0

  • Using certain numpy functions would previously return a Series if passed a Series as an argument. This seems only to affect np.ones_like, np.empty_like, np.diff and np.where. These now return ndarrays.

    In [107]: s = pd.Series([1, 2, 3, 4])

    Numpy Usage

    In [108]: np.ones_like(s)
Out[108]: array([1, 1, 1, 1])

In [109]: np.diff(s)
Out[109]: array([1, 1, 1])

In [110]: np.where(s > 1, s, np.nan)
Out[110]: array([nan,  2.,  3.,  4.])

    Pandonic Usage

    In [111]: pd.Series(1, index=s.index)
Out[111]: 
0    1
1    1
2    1
3    1
dtype: int64

In [112]: s.diff()
Out[112]: 
0    NaN
1    1.0
2    1.0
3    1.0
dtype: float64

In [113]: s.where(s > 1)
Out[113]: 
0    NaN
1    2.0
2    3.0
3    4.0
dtype: float64

  • Passing a Series directly to a cython function expecting an ndarray type will no long work directly, you must pass Series.values, See Enhancing Performance

  • Series(0.5) would previously return the scalar 0.5, instead this will return a 1-element Series

  • This change breaks rpy2<=2.3.8. an Issue has been opened against rpy2 and a workaround is detailed in GH 5698. Thanks @JanSchulz.

  • Pickle compatibility is preserved for pickles created prior to 0.13. These must be unpickled with pd.read_pickle, see Pickling.

  • Refactor of series.py/frame.py/panel.py to move common code to generic.py

    • added _setup_axes to created generic NDFrame structures

    • moved methods

      • from_axes,_wrap_array,axes,ix,loc,iloc,shape,empty,swapaxes,transpose,pop

      • __iter__,keys,__contains__,__len__,__neg__,__invert__

      • convert_objects,as_blocks,as_matrix,values

      • __getstate__,__setstate__ (compat remains in frame/panel)

      • __getattr__,__setattr__

      • _indexed_same,reindex_like,align,where,mask

      • fillna,replace (Series replace is now consistent with DataFrame)

      • filter (also added axis argument to selectively filter on a different axis)

      • reindex,reindex_axis,take

      • truncate (moved to become part of NDFrame)

  • These are API changes which make Panel more consistent with DataFrame

    • swapaxes on a Panel with the same axes specified now return a copy

    • support attribute access for setting

    • filter supports the same API as the original DataFrame filter

  • Reindex called with no arguments will now return a copy of the input object

  • TimeSeries is now an alias for Series. the property is_time_series can be used to distinguish (if desired)

  • Refactor of Sparse objects to use BlockManager

    • Created a new block type in internals, SparseBlock, which can hold multi-dtypes and is non-consolidatable. SparseSeries and SparseDataFrame now inherit more methods from there hierarchy (Series/DataFrame), and no longer inherit from SparseArray (which instead is the object of the SparseBlock)

    • Sparse suite now supports integration with non-sparse data. Non-float sparse data is supportable (partially implemented)

    • Operations on sparse structures within DataFrames should preserve sparseness, merging type operations will convert to dense (and back to sparse), so might be somewhat inefficient

    • enable setitem on SparseSeries for boolean/integer/slices

    • SparsePanels implementation is unchanged (e.g. not using BlockManager, needs work)

  • added ftypes method to Series/DataFrame, similar to dtypes, but indicates if the underlying is sparse/dense (as well as the dtype)

  • All NDFrame objects can now use __finalize__() to specify various values to propagate to new objects from an existing one (e.g. name in Series will follow more automatically now)

  • Internal type checking is now done via a suite of generated classes, allowing isinstance(value, klass) without having to directly import the klass, courtesy of @jtratner

  • Bug in Series update where the parent frame is not updating its cache based on changes (GH 4080) or types (GH 3217), fillna (GH 3386)

  • Indexing with dtype conversions fixed (GH 4463, GH 4204)

  • Refactor Series.reindex to core/generic.py (GH 4604, GH 4618), allow method= in reindexing on a Series to work

  • Series.copy no longer accepts the order parameter and is now consistent with NDFrame copy

  • Refactor rename methods to core/generic.py; fixes Series.rename for (GH 4605), and adds rename with the same signature for Panel

  • Refactor clip methods to core/generic.py (GH 4798)

  • Refactor of _get_numeric_data/_get_bool_data to core/generic.py, allowing Series/Panel functionality

  • Series (for index) / Panel (for items) now allow attribute access to its elements (GH 1903)

    In [114]: s = pd.Series([1, 2, 3], index=list('abc'))

In [115]: s.b
Out[115]: 2

In [116]: s.a = 5

In [117]: s
Out[117]: 
a    5
b    2
c    3
dtype: int64

Bug fixes#

  • HDFStore

    • raising an invalid TypeError rather than ValueError when appending with a different block ordering (GH 4096)

    • read_hdf was not respecting as passed mode (GH 4504)

    • appending a 0-len table will work correctly (GH 4273)

    • to_hdf was raising when passing both arguments append and table (GH 4584)

    • reading from a store with duplicate columns across dtypes would raise (GH 4767)

    • Fixed a bug where ValueError wasn’t correctly raised when column names weren’t strings (GH 4956)

    • A zero length series written in Fixed format not deserializing properly. (GH 4708)

    • Fixed decoding perf issue on pyt3 (GH 5441)

    • Validate levels in a MultiIndex before storing (GH 5527)

    • Correctly handle data_columns with a Panel (GH 5717)

  • Fixed bug in tslib.tz_convert(vals, tz1, tz2): it could raise IndexError exception while trying to access trans[pos + 1] (GH 4496)

  • The by argument now works correctly with the layout argument (GH 4102, GH 4014) in *.hist plotting methods

  • Fixed bug in PeriodIndex.map where using str would return the str representation of the index (GH 4136)

  • Fixed test failure test_time_series_plot_color_with_empty_kwargs when using custom matplotlib default colors (GH 4345)

  • Fix running of stata IO tests. Now uses temporary files to write (GH 4353)

  • Fixed an issue where DataFrame.sum was slower than DataFrame.mean for integer valued frames (GH 4365)

  • read_html tests now work with Python 2.6 (GH 4351)

  • Fixed bug where network testing was throwing NameError because a local variable was undefined (GH 4381)

  • In to_json, raise if a passed orient would cause loss of data because of a duplicate index (GH 4359)

  • In to_json, fix date handling so milliseconds are the default timestamp as the docstring says (GH 4362).

  • as_index is no longer ignored when doing groupby apply (GH 4648, GH 3417)

  • JSON NaT handling fixed, NaTs are now serialized to null (GH 4498)

  • Fixed JSON handling of escapable characters in JSON object keys (GH 4593)

  • Fixed passing keep_default_na=False when na_values=None (GH 4318)

  • Fixed bug with values raising an error on a DataFrame with duplicate columns and mixed dtypes, surfaced in (GH 4377)

  • Fixed bug with duplicate columns and type conversion in read_json when orient='split' (GH 4377)

  • Fixed JSON bug where locales with decimal separators other than ‘.’ threw exceptions when encoding / decoding certain values. (GH 4918)

  • Fix .iat indexing with a PeriodIndex (GH 4390)

  • Fixed an issue where PeriodIndex joining with self was returning a new instance rather than the same instance (GH 4379); also adds a test for this for the other index types

  • Fixed a bug with all the dtypes being converted to object when using the CSV cparser with the usecols parameter (GH 3192)

  • Fix an issue in merging blocks where the resulting DataFrame had partially set _ref_locs (GH 4403)

  • Fixed an issue where hist subplots were being overwritten when they were called using the top level matplotlib API (GH 4408)

  • Fixed a bug where calling Series.astype(str) would truncate the string (GH 4405, GH 4437)

  • Fixed a py3 compat issue where bytes were being repr’d as tuples (GH 4455)

  • Fixed Panel attribute naming conflict if item is named ‘a’ (GH 3440)

  • Fixed an issue where duplicate indexes were raising when plotting (GH 4486)

  • Fixed an issue where cumsum and cumprod didn’t work with bool dtypes (GH 4170, GH 4440)

  • Fixed Panel slicing issued in xs that was returning an incorrect dimmed object (GH 4016)

  • Fix resampling bug where custom reduce function not used if only one group (GH 3849, GH 4494)

  • Fixed Panel assignment with a transposed frame (GH 3830)

  • Raise on set indexing with a Panel and a Panel as a value which needs alignment (GH 3777)

  • frozenset objects now raise in the Series constructor (GH 4482, GH 4480)

  • Fixed issue with sorting a duplicate MultiIndex that has multiple dtypes (GH 4516)

  • Fixed bug in DataFrame.set_values which was causing name attributes to be lost when expanding the index. (GH 3742, GH 4039)

  • Fixed issue where individual names, levels and labels could be set on MultiIndex without validation (GH 3714, GH 4039)

  • Fixed (GH 3334) in pivot_table. Margins did not compute if values is the index.

  • Fix bug in having a rhs of np.timedelta64 or np.offsets.DateOffset when operating with datetimes (GH 4532)

  • Fix arithmetic with series/datetimeindex and np.timedelta64 not working the same (GH 4134) and buggy timedelta in NumPy 1.6 (GH 4135)

  • Fix bug in pd.read_clipboard on windows with PY3 (GH 4561); not decoding properly

  • tslib.get_period_field() and tslib.get_period_field_arr() now raise if code argument out of range (GH 4519, GH 4520)

  • Fix boolean indexing on an empty series loses index names (GH 4235), infer_dtype works with empty arrays.

  • Fix reindexing with multiple axes; if an axes match was not replacing the current axes, leading to a possible lazy frequency inference issue (GH 3317)

  • Fixed issue where DataFrame.apply was reraising exceptions incorrectly (causing the original stack trace to be truncated).

  • Fix selection with ix/loc and non_unique selectors (GH 4619)

  • Fix assignment with iloc/loc involving a dtype change in an existing column (GH 4312, GH 5702) have internal setitem_with_indexer in core/indexing to use Block.setitem

  • Fixed bug where thousands operator was not handled correctly for floating point numbers in csv_import (GH 4322)

  • Fix an issue with CacheableOffset not properly being used by many DateOffset; this prevented the DateOffset from being cached (GH 4609)

  • Fix boolean comparison with a DataFrame on the lhs, and a list/tuple on the rhs (GH 4576)

  • Fix error/dtype conversion with setitem of None on Series/DataFrame (GH 4667)

  • Fix decoding based on a passed in non-default encoding in pd.read_stata (GH 4626)

  • Fix DataFrame.from_records with a plain-vanilla ndarray. (GH 4727)

  • Fix some inconsistencies with Index.rename and MultiIndex.rename, etc. (GH 4718, GH 4628)

  • Bug in using iloc/loc with a cross-sectional and duplicate indices (GH 4726)

  • Bug with using QUOTE_NONE with to_csv causing Exception. (GH 4328)

  • Bug with Series indexing not raising an error when the right-hand-side has an incorrect length (GH 2702)

  • Bug in MultiIndexing with a partial string selection as one part of a MultIndex (GH 4758)

  • Bug with reindexing on the index with a non-unique index will now raise ValueError (GH 4746)

  • Bug in setting with loc/ix a single indexer with a MultiIndex axis and a NumPy array, related to (GH 3777)

  • Bug in concatenation with duplicate columns across dtypes not merging with axis=0 (GH 4771, GH 4975)

  • Bug in iloc with a slice index failing (GH 4771)

  • Incorrect error message with no colspecs or width in read_fwf. (GH 4774)

  • Fix bugs in indexing in a Series with a duplicate index (GH 4548, GH 4550)

  • Fixed bug with reading compressed files with read_fwf in Python 3. (GH 3963)

  • Fixed an issue with a duplicate index and assignment with a dtype change (GH 4686)

  • Fixed bug with reading compressed files in as bytes rather than str in Python 3. Simplifies bytes-producing file-handling in Python 3 (GH 3963, GH 4785).

  • Fixed an issue related to ticklocs/ticklabels with log scale bar plots across different versions of matplotlib (GH 4789)

  • Suppressed DeprecationWarning associated with internal calls issued by repr() (GH 4391)

  • Fixed an issue with a duplicate index and duplicate selector with .loc (GH 4825)

  • Fixed an issue with DataFrame.sort_index where, when sorting by a single column and passing a list for ascending, the argument for ascending was being interpreted as True (GH 4839, GH 4846)

  • Fixed Panel.tshift not working. Added freq support to Panel.shift (GH 4853)

  • Fix an issue in TextFileReader w/ Python engine (i.e. PythonParser) with thousands != “,” (GH 4596)

  • Bug in getitem with a duplicate index when using where (GH 4879)

  • Fix Type inference code coerces float column into datetime (GH 4601)

  • Fixed _ensure_numeric does not check for complex numbers (GH 4902)

  • Fixed a bug in Series.hist where two figures were being created when the by argument was passed (GH 4112, GH 4113).

  • Fixed a bug in convert_objects for > 2 ndims (GH 4937)

  • Fixed a bug in DataFrame/Panel cache insertion and subsequent indexing (GH 4939, GH 5424)

  • Fixed string methods for FrozenNDArray and FrozenList (GH 4929)

  • Fixed a bug with setting invalid or out-of-range values in indexing enlargement scenarios (GH 4940)

  • Tests for fillna on empty Series (GH 4346), thanks @immerrr

  • Fixed copy() to shallow copy axes/indices as well and thereby keep separate metadata. (GH 4202, GH 4830)

  • Fixed skiprows option in Python parser for read_csv (GH 4382)

  • Fixed bug preventing cut from working with np.inf levels without explicitly passing labels (GH 3415)

  • Fixed wrong check for overlapping in DatetimeIndex.union (GH 4564)

  • Fixed conflict between thousands separator and date parser in csv_parser (GH 4678)

  • Fix appending when dtypes are not the same (error showing mixing float/np.datetime64) (GH 4993)

  • Fix repr for DateOffset. No longer show duplicate entries in kwds. Removed unused offset fields. (GH 4638)

  • Fixed wrong index name during read_csv if using usecols. Applies to c parser only. (GH 4201)

  • Timestamp objects can now appear in the left hand side of a comparison operation with a Series or DataFrame object (GH 4982).

  • Fix a bug when indexing with np.nan via iloc/loc (GH 5016)

  • Fixed a bug where low memory c parser could create different types in different chunks of the same file. Now coerces to numerical type or raises warning. (GH 3866)

  • Fix a bug where reshaping a Series to its own shape raised TypeError (GH 4554) and other reshaping issues.

  • Bug in setting with ix/loc and a mixed int/string index (GH 4544)

  • Make sure series-series boolean comparisons are label based (GH 4947)

  • Bug in multi-level indexing with a Timestamp partial indexer (GH 4294)

  • Tests/fix for MultiIndex construction of an all-nan frame (GH 4078)

  • Fixed a bug where read_html() wasn’t correctly inferring values of tables with commas (GH 5029)

  • Fixed a bug where read_html() wasn’t providing a stable ordering of returned tables (GH 4770, GH 5029).

  • Fixed a bug where read_html() was incorrectly parsing when passed index_col=0 (GH 5066).

  • Fixed a bug where read_html() was incorrectly inferring the type of headers (GH 5048).

  • Fixed a bug where DatetimeIndex joins with PeriodIndex caused a stack overflow (GH 3899).

  • Fixed a bug where groupby objects didn’t allow plots (GH 5102).

  • Fixed a bug where groupby objects weren’t tab-completing column names (GH 5102).

  • Fixed a bug where groupby.plot() and friends were duplicating figures multiple times (GH 5102).

  • Provide automatic conversion of object dtypes on fillna, related (GH 5103)

  • Fixed a bug where default options were being overwritten in the option parser cleaning (GH 5121).

  • Treat a list/ndarray identically for iloc indexing with list-like (GH 5006)

  • Fix MultiIndex.get_level_values() with missing values (GH 5074)

  • Fix bound checking for Timestamp() with datetime64 input (GH 4065)

  • Fix a bug where TestReadHtml wasn’t calling the correct read_html() function (GH 5150).

  • Fix a bug with NDFrame.replace() which made replacement appear as though it was (incorrectly) using regular expressions (GH 5143).

  • Fix better error message for to_datetime (GH 4928)

  • Made sure different locales are tested on travis-ci (GH 4918). Also adds a couple of utilities for getting locales and setting locales with a context manager.

  • Fixed segfault on isnull(MultiIndex) (now raises an error instead) (GH 5123, GH 5125)

  • Allow duplicate indices when performing operations that align (GH 5185, GH 5639)

  • Compound dtypes in a constructor raise NotImplementedError (GH 5191)

  • Bug in comparing duplicate frames (GH 4421) related

  • Bug in describe on duplicate frames

  • Bug in to_datetime with a format and coerce=True not raising (GH 5195)

  • Bug in loc setting with multiple indexers and a rhs of a Series that needs broadcasting (GH 5206)

  • Fixed bug where inplace setting of levels or labels on MultiIndex would not clear cached values property and therefore return wrong values. (GH 5215)

  • Fixed bug where filtering a grouped DataFrame or Series did not maintain the original ordering (GH 4621).

  • Fixed Period with a business date freq to always roll-forward if on a non-business date. (GH 5203)

  • Fixed bug in Excel writers where frames with duplicate column names weren’t written correctly. (GH 5235)

  • Fixed issue with drop and a non-unique index on Series (GH 5248)

  • Fixed segfault in C parser caused by passing more names than columns in the file. (GH 5156)

  • Fix Series.isin with date/time-like dtypes (GH 5021)

  • C and Python Parser can now handle the more common MultiIndex column format which doesn’t have a row for index names (GH 4702)

  • Bug when trying to use an out-of-bounds date as an object dtype (GH 5312)

  • Bug when trying to display an embedded PandasObject (GH 5324)

  • Allows operating of Timestamps to return a datetime if the result is out-of-bounds related (GH 5312)

  • Fix return value/type signature of initObjToJSON() to be compatible with numpy’s import_array() (GH 5334, GH 5326)

  • Bug when renaming then set_index on a DataFrame (GH 5344)

  • Test suite no longer leaves around temporary files when testing graphics. (GH 5347) (thanks for catching this @yarikoptic!)

  • Fixed html tests on win32. (GH 4580)

  • Make sure that head/tail are iloc based, (GH 5370)

  • Fixed bug for PeriodIndex string representation if there are 1 or 2 elements. (GH 5372)

  • The GroupBy methods transform and filter can be used on Series and DataFrames that have repeated (non-unique) indices. (GH 4620)

  • Fix empty series not printing name in repr (GH 4651)

  • Make tests create temp files in temp directory by default. (GH 5419)

  • pd.to_timedelta of a scalar returns a scalar (GH 5410)

  • pd.to_timedelta accepts NaN and NaT, returning NaT instead of raising (GH 5437)

  • performance improvements in isnull on larger size pandas objects

  • Fixed various setitem with 1d ndarray that does not have a matching length to the indexer (GH 5508)

  • Bug in getitem with a MultiIndex and iloc (GH 5528)

  • Bug in delitem on a Series (GH 5542)

  • Bug fix in apply when using custom function and objects are not mutated (GH 5545)

  • Bug in selecting from a non-unique index with loc (GH 5553)

  • Bug in groupby returning non-consistent types when user function returns a None, (GH 5592)

  • Work around regression in numpy 1.7.0 which erroneously raises IndexError from ndarray.item (GH 5666)

  • Bug in repeated indexing of object with resultant non-unique index (GH 5678)

  • Bug in fillna with Series and a passed series/dict (GH 5703)

  • Bug in groupby transform with a datetime-like grouper (GH 5712)

  • Bug in MultiIndex selection in PY3 when using certain keys (GH 5725)

  • Row-wise concat of differing dtypes failing in certain cases (GH 5754)

Contributors#

A total of 77 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

  • Agustín Herranz +

  • Alex Gaudio +

  • Alex Rothberg +

  • Andreas Klostermann +

  • Andreas Würl +

  • Andy Hayden

  • Ben Alex +

  • Benedikt Sauer +

  • Brad Buran

  • Caleb Epstein +

  • Chang She

  • Christopher Whelan

  • DSM +

  • Dale Jung +

  • Dan Birken

  • David Rasch +

  • Dieter Vandenbussche

  • Gabi Davar +

  • Garrett Drapala

  • Goyo +

  • Greg Reda +

  • Ivan Smirnov +

  • Jack Kelly +

  • Jacob Schaer +

  • Jan Schulz +

  • Jeff Tratner

  • Jeffrey Tratner

  • John McNamara +

  • John W. O’Brien +

  • Joris Van den Bossche

  • Justin Bozonier +

  • Kelsey Jordahl

  • Kevin Stone

  • Kieran O’Mahony

  • Kyle Hausmann +

  • Kyle Kelley +

  • Kyle Meyer

  • Mike Kelly

  • Mortada Mehyar +

  • Nick Foti +

  • Olivier Harris +

  • Ondřej Čertík +

  • PKEuS

  • Phillip Cloud

  • Pierre Haessig +

  • Richard T. Guy +

  • Roman Pekar +

  • Roy Hyunjin Han

  • Skipper Seabold

  • Sten +

  • Thomas A Caswell +

  • Thomas Kluyver

  • Tiago Requeijo +

  • TomAugspurger

  • Trent Hauck

  • Valentin Haenel +

  • Viktor Kerkez +

  • Vincent Arel-Bundock

  • Wes McKinney

  • Wes Turner +

  • Weston Renoud +

  • Yaroslav Halchenko

  • Zach Dwiel +

  • chapman siu +

  • chappers +

  • d10genes +

  • danielballan

  • daydreamt +

  • engstrom +

  • jreback

  • monicaBee +

  • prossahl +

  • rockg +

  • unutbu +

  • westurner +

  • y-p

  • zach powers