Version 0.19.0 (October 2, 2016)#

This is a major release from 0.18.1 and includes number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Highlights include:

merge_asof() for asof-style time-series joining, see here
.rolling() is now time-series aware, see here
read_csv() now supports parsing Categorical data, see here
A function union_categorical() has been added for combining categoricals, see here
PeriodIndex now has its own period dtype, and changed to be more consistent with other Index classes. See here
Sparse data structures gained enhanced support of int and bool dtypes, see here
Comparison operations with Series no longer ignores the index, see here for an overview of the API changes.
Introduction of a pandas development API for utility functions, see here.
Deprecation of Panel4D and PanelND. We recommend to represent these types of n-dimensional data with the xarray package.
Removal of the previously deprecated modules pandas.io.data, pandas.io.wb, pandas.tools.rplot.

Warning

pandas >= 0.19.0 will no longer silence numpy ufunc warnings upon import, see here.

New features#

Function `merge_asof` for asof-style time-series joining#

A long-time requested feature has been added through the merge_asof() function, to support asof style joining of time-series (GH 1870, GH 13695, GH 13709, GH 13902). Full documentation is here.

The merge_asof() performs an asof merge, which is similar to a left-join except that we match on nearest key rather than equal keys.

In [1]: left = pd.DataFrame({"a": [1, 5, 10], "left_val": ["a", "b", "c"]})

In [2]: right = pd.DataFrame({"a": [1, 2, 3, 6, 7], "right_val": [1, 2, 3, 6, 7]})

In [3]: left
Out[3]: 
    a left_val
0   1        a
1   5        b
2  10        c

[3 rows x 2 columns]

In [4]: right
Out[4]: 
   a  right_val
0  1          1
1  2          2
2  3          3
3  6          6
4  7          7

[5 rows x 2 columns]

We typically want to match exactly when possible, and use the most recent value otherwise.

In [5]: pd.merge_asof(left, right, on="a")
Out[5]: 
    a left_val  right_val
0   1        a          1
1   5        b          3
2  10        c          7

[3 rows x 3 columns]

We can also match rows ONLY with prior data, and not an exact match.

In [6]: pd.merge_asof(left, right, on="a", allow_exact_matches=False)
Out[6]: 
    a left_val  right_val
0   1        a        NaN
1   5        b        3.0
2  10        c        7.0

[3 rows x 3 columns]

In a typical time-series example, we have trades and quotes and we want to asof-join them. This also illustrates using the by parameter to group data before merging.

In [7]: trades = pd.DataFrame(
   ...:     {
   ...:         "time": pd.to_datetime(
   ...:             [
   ...:                 "20160525 13:30:00.023",
   ...:                 "20160525 13:30:00.038",
   ...:                 "20160525 13:30:00.048",
   ...:                 "20160525 13:30:00.048",
   ...:                 "20160525 13:30:00.048",
   ...:             ]
   ...:         ),
   ...:         "ticker": ["MSFT", "MSFT", "GOOG", "GOOG", "AAPL"],
   ...:         "price": [51.95, 51.95, 720.77, 720.92, 98.00],
   ...:         "quantity": [75, 155, 100, 100, 100],
   ...:     },
   ...:     columns=["time", "ticker", "price", "quantity"],
   ...: )
   ...: 

In [8]: quotes = pd.DataFrame(
   ...:     {
   ...:         "time": pd.to_datetime(
   ...:             [
   ...:                 "20160525 13:30:00.023",
   ...:                 "20160525 13:30:00.023",
   ...:                 "20160525 13:30:00.030",
   ...:                 "20160525 13:30:00.041",
   ...:                 "20160525 13:30:00.048",
   ...:                 "20160525 13:30:00.049",
   ...:                 "20160525 13:30:00.072",
   ...:                 "20160525 13:30:00.075",
   ...:             ]
   ...:         ),
   ...:         "ticker": ["GOOG", "MSFT", "MSFT", "MSFT", "GOOG", "AAPL", "GOOG", "MSFT"],
   ...:         "bid": [720.50, 51.95, 51.97, 51.99, 720.50, 97.99, 720.50, 52.01],
   ...:         "ask": [720.93, 51.96, 51.98, 52.00, 720.93, 98.01, 720.88, 52.03],
   ...:     },
   ...:     columns=["time", "ticker", "bid", "ask"],
   ...: )
   ...: 

In [9]: trades
Out[9]: 
                     time ticker   price  quantity
0 2016-05-25 13:30:00.023   MSFT   51.95        75
1 2016-05-25 13:30:00.038   MSFT   51.95       155
2 2016-05-25 13:30:00.048   GOOG  720.77       100
3 2016-05-25 13:30:00.048   GOOG  720.92       100
4 2016-05-25 13:30:00.048   AAPL   98.00       100

[5 rows x 4 columns]

In [10]: quotes
Out[10]: 
                     time ticker     bid     ask
0 2016-05-25 13:30:00.023   GOOG  720.50  720.93
1 2016-05-25 13:30:00.023   MSFT   51.95   51.96
2 2016-05-25 13:30:00.030   MSFT   51.97   51.98
3 2016-05-25 13:30:00.041   MSFT   51.99   52.00
4 2016-05-25 13:30:00.048   GOOG  720.50  720.93
5 2016-05-25 13:30:00.049   AAPL   97.99   98.01
6 2016-05-25 13:30:00.072   GOOG  720.50  720.88
7 2016-05-25 13:30:00.075   MSFT   52.01   52.03

[8 rows x 4 columns]

An asof merge joins on the on, typically a datetimelike field, which is ordered, and in this case we are using a grouper in the by field. This is like a left-outer join, except that forward filling happens automatically taking the most recent non-NaN value.

In [11]: pd.merge_asof(trades, quotes, on="time", by="ticker")
Out[11]: 
                     time ticker   price  quantity     bid     ask
0 2016-05-25 13:30:00.023   MSFT   51.95        75   51.95   51.96
1 2016-05-25 13:30:00.038   MSFT   51.95       155   51.97   51.98
2 2016-05-25 13:30:00.048   GOOG  720.77       100  720.50  720.93
3 2016-05-25 13:30:00.048   GOOG  720.92       100  720.50  720.93
4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN

[5 rows x 6 columns]

This returns a merged DataFrame with the entries in the same order as the original left passed DataFrame (trades in this case), with the fields of the quotes merged.

Method `.rolling()` is now time-series aware#

.rolling() objects are now time-series aware and can accept a time-series offset (or convertible) for the window argument (GH 13327, GH 12995). See the full documentation here.

In [12]: dft = pd.DataFrame(
   ....:     {"B": [0, 1, 2, np.nan, 4]},
   ....:     index=pd.date_range("20130101 09:00:00", periods=5, freq="s"),
   ....: )
   ....: 

In [13]: dft
Out[13]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:01  1.0
2013-01-01 09:00:02  2.0
2013-01-01 09:00:03  NaN
2013-01-01 09:00:04  4.0

[5 rows x 1 columns]

This is a regular frequency index. Using an integer window parameter works to roll along the window frequency.

In [14]: dft.rolling(2).sum()
Out[14]: 
                       B
2013-01-01 09:00:00  NaN
2013-01-01 09:00:01  1.0
2013-01-01 09:00:02  3.0
2013-01-01 09:00:03  NaN
2013-01-01 09:00:04  NaN

[5 rows x 1 columns]

In [15]: dft.rolling(2, min_periods=1).sum()
Out[15]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:01  1.0
2013-01-01 09:00:02  3.0
2013-01-01 09:00:03  2.0
2013-01-01 09:00:04  4.0

[5 rows x 1 columns]

Specifying an offset allows a more intuitive specification of the rolling frequency.

In [16]: dft.rolling("2s").sum()
Out[16]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:01  1.0
2013-01-01 09:00:02  3.0
2013-01-01 09:00:03  2.0
2013-01-01 09:00:04  4.0

[5 rows x 1 columns]

Using a non-regular, but still monotonic index, rolling with an integer window does not impart any special calculation.

In [17]: dft = pd.DataFrame(
   ....:     {"B": [0, 1, 2, np.nan, 4]},
   ....:     index=pd.Index(
   ....:         [
   ....:             pd.Timestamp("20130101 09:00:00"),
   ....:             pd.Timestamp("20130101 09:00:02"),
   ....:             pd.Timestamp("20130101 09:00:03"),
   ....:             pd.Timestamp("20130101 09:00:05"),
   ....:             pd.Timestamp("20130101 09:00:06"),
   ....:         ],
   ....:         name="foo",
   ....:     ),
   ....: )
   ....: 

In [18]: dft
Out[18]: 
                       B
foo                     
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  2.0
2013-01-01 09:00:05  NaN
2013-01-01 09:00:06  4.0

[5 rows x 1 columns]

In [19]: dft.rolling(2).sum()
Out[19]: 
                       B
foo                     
2013-01-01 09:00:00  NaN
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  NaN
2013-01-01 09:00:06  NaN

[5 rows x 1 columns]

Using the time-specification generates variable windows for this sparse data.

In [20]: dft.rolling("2s").sum()
Out[20]: 
                       B
foo                     
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  NaN
2013-01-01 09:00:06  4.0

[5 rows x 1 columns]

Furthermore, we now allow an optional on parameter to specify a column (rather than the default of the index) in a DataFrame.

In [21]: dft = dft.reset_index()

In [22]: dft
Out[22]: 
                  foo    B
0 2013-01-01 09:00:00  0.0
1 2013-01-01 09:00:02  1.0
2 2013-01-01 09:00:03  2.0
3 2013-01-01 09:00:05  NaN
4 2013-01-01 09:00:06  4.0

[5 rows x 2 columns]

In [23]: dft.rolling("2s", on="foo").sum()
Out[23]: 
                  foo    B
0 2013-01-01 09:00:00  0.0
1 2013-01-01 09:00:02  1.0
2 2013-01-01 09:00:03  3.0
3 2013-01-01 09:00:05  NaN
4 2013-01-01 09:00:06  4.0

[5 rows x 2 columns]

Method `read_csv` has improved support for duplicate column names#

Duplicate column names are now supported in read_csv() whether they are in the file or passed in as the names parameter (GH 7160, GH 9424)

In [24]: data = "0,1,2\n3,4,5"

In [25]: names = ["a", "b", "a"]

Previous behavior:

In [2]: pd.read_csv(StringIO(data), names=names)
Out[2]:
   a  b  a
0  2  1  2
1  5  4  5

The first a column contained the same data as the second a column, when it should have contained the values [0, 3].

New behavior:

In [26]: pd.read_csv(StringIO(data), names=names)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[26], line 1
----> 1 pd.read_csv(StringIO(data), names=names)

File ~/work/pandas/pandas/pandas/io/parsers/readers.py:853, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, skip_blank_lines, parse_dates, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, low_memory, memory_map, float_precision, storage_options, dtype_backend)
    841 kwds_defaults = _refine_defaults_read(
    842     dialect,
    843     delimiter,
   (...)
    849     dtype_backend=dtype_backend,
    850 )
    851 kwds.update(kwds_defaults)
--> 853 return _read(filepath_or_buffer, kwds)

File ~/work/pandas/pandas/pandas/io/parsers/readers.py:710, in _read(filepath_or_buffer, kwds)
    707 nrows = kwds.get("nrows", None)
    709 # Check for duplicates in names.
--> 710 _validate_names(kwds.get("names", None))
    712 # Create the parser.
    713 parser = TextFileReader(filepath_or_buffer, **kwds)

File ~/work/pandas/pandas/pandas/io/parsers/readers.py:664, in _validate_names(names)
    662 if names is not None:
    663     if len(names) != len(set(names)):
--> 664         raise ValueError("Duplicate names are not allowed.")
    665     if not (
    666         is_list_like(names, allow_sets=False) or isinstance(names, abc.KeysView)
    667     ):
    668         raise ValueError("Names should be an ordered collection.")

ValueError: Duplicate names are not allowed.

Method `read_csv` supports parsing `Categorical` directly#

The read_csv() function now supports parsing a Categorical column when specified as a dtype (GH 10153). Depending on the structure of the data, this can result in a faster parse time and lower memory usage compared to converting to Categorical after parsing. See the io docs here.

In [27]: data = """
   ....: col1,col2,col3
   ....: a,b,1
   ....: a,b,2
   ....: c,d,3
   ....: """
   ....: 

In [28]: pd.read_csv(StringIO(data))
Out[28]: 
  col1 col2  col3
0    a    b     1
1    a    b     2
2    c    d     3

[3 rows x 3 columns]

In [29]: pd.read_csv(StringIO(data)).dtypes
Out[29]: 
col1    object
col2    object
col3     int64
Length: 3, dtype: object

In [30]: pd.read_csv(StringIO(data), dtype="category").dtypes
Out[30]: 
col1    category
col2    category
col3    category
Length: 3, dtype: object

Individual columns can be parsed as a Categorical using a dict specification

In [31]: pd.read_csv(StringIO(data), dtype={"col1": "category"}).dtypes
Out[31]: 
col1    category
col2      object
col3       int64
Length: 3, dtype: object

Note

The resulting categories will always be parsed as strings (object dtype). If the categories are numeric they can be converted using the to_numeric() function, or as appropriate, another converter such as to_datetime().

In [32]: df = pd.read_csv(StringIO(data), dtype="category")

In [33]: df.dtypes
Out[33]: 
col1    category
col2    category
col3    category
Length: 3, dtype: object

In [34]: df["col3"]
Out[34]: 
0    1
1    2
2    3
Name: col3, Length: 3, dtype: category
Categories (3, object): ['1', '2', '3']

In [35]: new_categories = pd.to_numeric(df["col3"].cat.categories)

In [36]: df["col3"] = df["col3"].cat.rename_categories(new_categories)

In [37]: df["col3"]
Out[37]: 
0    1
1    2
2    3
Name: col3, Length: 3, dtype: category
Categories (3, int64): [1, 2, 3]

Categorical concatenation#

A function union_categoricals() has been added for combining categoricals, see Unioning Categoricals (GH 13361, GH 13763, GH 13846, GH 14173)

In [38]: from pandas.api.types import union_categoricals

In [39]: a = pd.Categorical(["b", "c"])

In [40]: b = pd.Categorical(["a", "b"])

In [41]: union_categoricals([a, b])
Out[41]: 
['b', 'c', 'a', 'b']
Categories (3, object): ['b', 'c', 'a']

concat and append now can concat category dtypes with different categories as object dtype (GH 13524)

In [42]: s1 = pd.Series(["a", "b"], dtype="category")

In [43]: s2 = pd.Series(["b", "c"], dtype="category")

Previous behavior:

In [1]: pd.concat([s1, s2])
ValueError: incompatible categories in categorical concat

New behavior:

In [44]: pd.concat([s1, s2])
Out[44]: 
0    a
1    b
0    b
1    c
Length: 4, dtype: object

Semi-month offsets#

pandas has gained new frequency offsets, SemiMonthEnd (‘SM’) and SemiMonthBegin (‘SMS’). These provide date offsets anchored (by default) to the 15th and end of month, and 15th and 1st of month respectively. (GH 1543)

In [45]: from pandas.tseries.offsets import SemiMonthEnd, SemiMonthBegin

SemiMonthEnd:

In [46]: pd.Timestamp("2016-01-01") + SemiMonthEnd()
Out[46]: Timestamp('2016-01-15 00:00:00')

In [47]: pd.date_range("2015-01-01", freq="SM", periods=4)
Out[47]: DatetimeIndex(['2015-01-15', '2015-01-31', '2015-02-15', '2015-02-28'], dtype='datetime64[ns]', freq='SM-15')

SemiMonthBegin:

In [46]: pd.Timestamp("2016-01-01") + SemiMonthBegin()
Out[46]: Timestamp('2016-01-15 00:00:00')

In [47]: pd.date_range("2015-01-01", freq="SMS", periods=4)
Out[47]: DatetimeIndex(['2015-01-01', '2015-01-15', '2015-02-01', '2015-02-15'], dtype='datetime64[ns]', freq='SMS-15')

Using the anchoring suffix, you can also specify the day of month to use instead of the 15th.

In [50]: pd.date_range("2015-01-01", freq="SMS-16", periods=4)
Out[50]: DatetimeIndex(['2015-01-01', '2015-01-16', '2015-02-01', '2015-02-16'], dtype='datetime64[ns]', freq='SMS-16')

In [51]: pd.date_range("2015-01-01", freq="SM-14", periods=4)
Out[51]: DatetimeIndex(['2015-01-14', '2015-01-31', '2015-02-14', '2015-02-28'], dtype='datetime64[ns]', freq='SM-14')

New Index methods#

The following methods and options are added to Index, to be more consistent with the Series and DataFrame API.

Index now supports the .where() function for same shape indexing (GH 13170)

In [48]: idx = pd.Index(["a", "b", "c"])

In [49]: idx.where([True, False, True])
Out[49]: Index(['a', None, 'c'], dtype='object')

Index now supports .dropna() to exclude missing values (GH 6194)

In [50]: idx = pd.Index([1, 2, np.nan, 4])

In [51]: idx.dropna()
Out[51]: Index([1.0, 2.0, 4.0], dtype='float64')

For MultiIndex, values are dropped if any level is missing by default. Specifying how='all' only drops values where all levels are missing.

In [52]: midx = pd.MultiIndex.from_arrays([[1, 2, np.nan, 4], [1, 2, np.nan, np.nan]])

In [53]: midx
Out[53]: 
MultiIndex([(1.0, 1.0),
            (2.0, 2.0),
            (nan, nan),
            (4.0, nan)],
           )

In [54]: midx.dropna()
Out[54]: 
MultiIndex([(1, 1),
            (2, 2)],
           )

In [55]: midx.dropna(how="all")
Out[55]: 
MultiIndex([(1, 1.0),
            (2, 2.0),
            (4, nan)],
           )

Index now supports .str.extractall() which returns a DataFrame, see the docs here (GH 10008, GH 13156)

In [56]: idx = pd.Index(["a1a2", "b1", "c1"])

In [57]: idx.str.extractall(r"[ab](?P<digit>\d)")
Out[57]: 
        digit
  match      
0 0         1
  1         2
1 0         1

[3 rows x 1 columns]

Index.astype() now accepts an optional boolean argument copy, which allows optional copying if the requirements on dtype are satisfied (GH 13209)

Google BigQuery enhancements#

The read_gbq() method has gained the dialect argument to allow users to specify whether to use BigQuery’s legacy SQL or BigQuery’s standard SQL. See the docs for more details (GH 13615).
The to_gbq() method now allows the DataFrame column order to differ from the destination table schema (GH 11359).

Fine-grained NumPy errstate#

Previous versions of pandas would permanently silence numpy’s ufunc error handling when pandas was imported. pandas did this in order to silence the warnings that would arise from using numpy ufuncs on missing data, which are usually represented as NaN s. Unfortunately, this silenced legitimate warnings arising in non-pandas code in the application. Starting with 0.19.0, pandas will use the numpy.errstate context manager to silence these warnings in a more fine-grained manner, only around where these operations are actually used in the pandas code base. (GH 13109, GH 13145)

After upgrading pandas, you may see new RuntimeWarnings being issued from your code. These are likely legitimate, and the underlying cause likely existed in the code when using previous versions of pandas that simply silenced the warning. Use numpy.errstate around the source of the RuntimeWarning to control how these conditions are handled.

Method `get_dummies` now returns integer dtypes#

The pd.get_dummies function now returns dummy-encoded columns as small integers, rather than floats (GH 8725). This should provide an improved memory footprint.

Previous behavior:

In [1]: pd.get_dummies(['a', 'b', 'a', 'c']).dtypes

Out[1]:
a    float64
b    float64
c    float64
dtype: object

New behavior:

In [58]: pd.get_dummies(["a", "b", "a", "c"]).dtypes
Out[58]: 
a    bool
b    bool
c    bool
Length: 3, dtype: object

Downcast values to smallest possible dtype in `to_numeric`#

pd.to_numeric() now accepts a downcast parameter, which will downcast the data if possible to smallest specified numerical dtype (GH 13352)

In [59]: s = ["1", 2, 3]

In [60]: pd.to_numeric(s, downcast="unsigned")
Out[60]: array([1, 2, 3], dtype=uint8)

In [61]: pd.to_numeric(s, downcast="integer")
Out[61]: array([1, 2, 3], dtype=int8)

pandas development API#

As part of making pandas API more uniform and accessible in the future, we have created a standard sub-package of pandas, pandas.api to hold public API’s. We are starting by exposing type introspection functions in pandas.api.types. More sub-packages and officially sanctioned API’s will be published in future versions of pandas (GH 13147, GH 13634)

The following are now part of this API:

In [62]: import pprint

In [63]: from pandas.api import types

In [64]: funcs = [f for f in dir(types) if not f.startswith("_")]

In [65]: pprint.pprint(funcs)
['CategoricalDtype',
 'DatetimeTZDtype',
 'IntervalDtype',
 'PeriodDtype',
 'infer_dtype',
 'is_any_real_numeric_dtype',
 'is_array_like',
 'is_bool',
 'is_bool_dtype',
 'is_categorical_dtype',
 'is_complex',
 'is_complex_dtype',
 'is_datetime64_any_dtype',
 'is_datetime64_dtype',
 'is_datetime64_ns_dtype',
 'is_datetime64tz_dtype',
 'is_dict_like',
 'is_dtype_equal',
 'is_extension_array_dtype',
 'is_file_like',
 'is_float',
 'is_float_dtype',
 'is_hashable',
 'is_int64_dtype',
 'is_integer',
 'is_integer_dtype',
 'is_interval_dtype',
 'is_iterator',
 'is_list_like',
 'is_named_tuple',
 'is_number',
 'is_numeric_dtype',
 'is_object_dtype',
 'is_period_dtype',
 'is_re',
 'is_re_compilable',
 'is_scalar',
 'is_signed_integer_dtype',
 'is_sparse',
 'is_string_dtype',
 'is_timedelta64_dtype',
 'is_timedelta64_ns_dtype',
 'is_unsigned_integer_dtype',
 'pandas_dtype',
 'union_categoricals']

Note

Calling these functions from the internal module pandas.core.common will now show a DeprecationWarning (GH 13990)

Other enhancements#

Timestamp can now accept positional and keyword parameters similar to datetime.datetime() (GH 10758, GH 11630)

In [66]: pd.Timestamp(2012, 1, 1)
Out[66]: Timestamp('2012-01-01 00:00:00')

In [67]: pd.Timestamp(year=2012, month=1, day=1, hour=8, minute=30)
Out[67]: Timestamp('2012-01-01 08:30:00')

The .resample() function now accepts a on= or level= parameter for resampling on a datetimelike column or MultiIndex level (GH 13500)

In [68]: df = pd.DataFrame(
   ....:     {"date": pd.date_range("2015-01-01", freq="W", periods=5), "a": np.arange(5)},
   ....:     index=pd.MultiIndex.from_arrays(
   ....:         [[1, 2, 3, 4, 5], pd.date_range("2015-01-01", freq="W", periods=5)],
   ....:         names=["v", "d"],
   ....:     ),
   ....: )
   ....: 

In [69]: df
Out[69]: 
                   date  a
v d                       
1 2015-01-04 2015-01-04  0
2 2015-01-11 2015-01-11  1
3 2015-01-18 2015-01-18  2
4 2015-01-25 2015-01-25  3
5 2015-02-01 2015-02-01  4

[5 rows x 2 columns]

In [74]: df.resample("M", on="date")[["a"]].sum()
Out[74]:
            a
date
2015-01-31  6
2015-02-28  4

[2 rows x 1 columns]

In [75]: df.resample("M", level="d")[["a"]].sum()
Out[75]:
            a
d
2015-01-31  6
2015-02-28  4

[2 rows x 1 columns]

The .get_credentials() method of GbqConnector can now first try to fetch the application default credentials. See the docs for more details (GH 13577).
The .tz_localize() method of DatetimeIndex and Timestamp has gained the errors keyword, so you can potentially coerce nonexistent timestamps to NaT. The default behavior remains to raising a NonExistentTimeError (GH 13057)
.to_hdf/read_hdf() now accept path objects (e.g. pathlib.Path, py.path.local) for the file path (GH 11773)
The pd.read_csv() with engine='python' has gained support for the decimal (GH 12933), na_filter (GH 13321) and the memory_map option (GH 13381).
Consistent with the Python API, pd.read_csv() will now interpret +inf as positive infinity (GH 13274)
The pd.read_html() has gained support for the na_values, converters, keep_default_na options (GH 13461)
Categorical.astype() now accepts an optional boolean argument copy, effective when dtype is categorical (GH 13209)
DataFrame has gained the .asof() method to return the last non-NaN values according to the selected subset (GH 13358)
The DataFrame constructor will now respect key ordering if a list of OrderedDict objects are passed in (GH 13304)
pd.read_html() has gained support for the decimal option (GH 12907)
Series has gained the properties .is_monotonic, .is_monotonic_increasing, .is_monotonic_decreasing, similar to Index (GH 13336)
DataFrame.to_sql() now allows a single value as the SQL type for all columns (GH 11886).
Series.append now supports the ignore_index option (GH 13677)
.to_stata() and StataWriter can now write variable labels to Stata dta files using a dictionary to make column names to labels (GH 13535, GH 13536)
.to_stata() and StataWriter will automatically convert datetime64[ns] columns to Stata format %tc, rather than raising a ValueError (GH 12259)
read_stata() and StataReader raise with a more explicit error message when reading Stata files with repeated value labels when convert_categoricals=True (GH 13923)
DataFrame.style will now render sparsified MultiIndexes (GH 11655)
DataFrame.style will now show column level names (e.g. DataFrame.columns.names) (GH 13775)

DataFrame has gained support to re-order the columns based on the values in a row using df.sort_values(by='...', axis=1) (GH 10806)

In [70]: df = pd.DataFrame({"A": [2, 7], "B": [3, 5], "C": [4, 8]}, index=["row1", "row2"])

In [71]: df
Out[71]: 
      A  B  C
row1  2  3  4
row2  7  5  8

[2 rows x 3 columns]

In [72]: df.sort_values(by="row2", axis=1)
Out[72]: 
      B  A  C
row1  3  2  4
row2  5  7  8

[2 rows x 3 columns]

Added documentation to I/O regarding the perils of reading in columns with mixed dtypes and how to handle it (GH 13746)
to_html() now has a border argument to control the value in the opening <table> tag. The default is the value of the html.border option, which defaults to 1. This also affects the notebook HTML repr, but since Jupyter’s CSS includes a border-width attribute, the visual effect is the same. (GH 11563).
Raise ImportError in the sql functions when sqlalchemy is not installed and a connection string is used (GH 11920).
Compatibility with matplotlib 2.0. Older versions of pandas should also work with matplotlib 2.0 (GH 13333)
Timestamp, Period, DatetimeIndex, PeriodIndex and .dt accessor have gained a .is_leap_year property to check whether the date belongs to a leap year. (GH 13727)
astype() will now accept a dict of column name to data types mapping as the dtype argument. (GH 12086)
The pd.read_json and DataFrame.to_json has gained support for reading and writing json lines with lines option see Line delimited json (GH 9180)
read_excel() now supports the true_values and false_values keyword arguments (GH 13347)
groupby() will now accept a scalar and a single-element list for specifying level on a non-MultiIndex grouper. (GH 13907)
Non-convertible dates in an excel date column will be returned without conversion and the column will be object dtype, rather than raising an exception (GH 10001).
pd.Timedelta(None) is now accepted and will return NaT, mirroring pd.Timestamp (GH 13687)
pd.read_stata() can now handle some format 111 files, which are produced by SAS when generating Stata dta files (GH 11526)
Series and Index now support divmod which will return a tuple of series or indices. This behaves like a standard binary operator with regards to broadcasting rules (GH 14208).

API changes#

`Series.tolist()` will now return Python types#

Series.tolist() will now return Python types in the output, mimicking NumPy .tolist() behavior (GH 10904)

In [73]: s = pd.Series([1, 2, 3])

Previous behavior:

In [7]: type(s.tolist()[0])
Out[7]:
 <class 'numpy.int64'>

New behavior:

In [74]: type(s.tolist()[0])
Out[74]: int

`Series` operators for different indexes#

Following Series operators have been changed to make all operators consistent, including DataFrame (GH 1134, GH 4581, GH 13538)

Series comparison operators now raise ValueError when index are different.
Series logical operators align both index of left and right hand side.

Warning

Until 0.18.1, comparing Series with the same length, would succeed even if the .index are different (the result ignores .index). As of 0.19.0, this will raises ValueError to be more strict. This section also describes how to keep previous behavior or align different indexes, using the flexible comparison methods like .eq.

As a result, Series and DataFrame operators behave as below:

Arithmetic operators#

Arithmetic operators align both index (no changes).

In [75]: s1 = pd.Series([1, 2, 3], index=list("ABC"))

In [76]: s2 = pd.Series([2, 2, 2], index=list("ABD"))

In [77]: s1 + s2
Out[77]: 
A    3.0
B    4.0
C    NaN
D    NaN
Length: 4, dtype: float64

In [78]: df1 = pd.DataFrame([1, 2, 3], index=list("ABC"))

In [79]: df2 = pd.DataFrame([2, 2, 2], index=list("ABD"))

In [80]: df1 + df2
Out[80]: 
     0
A  3.0
B  4.0
C  NaN
D  NaN

[4 rows x 1 columns]

Comparison operators#

Comparison operators raise ValueError when .index are different.

Previous behavior (Series):

Series compared values ignoring the .index as long as both had the same length:

In [1]: s1 == s2
Out[1]:
A    False
B     True
C    False
dtype: bool

New behavior (Series):

In [2]: s1 == s2
Out[2]:
ValueError: Can only compare identically-labeled Series objects

Note

To achieve the same result as previous versions (compare values based on locations ignoring .index), compare both .values.

In [81]: s1.values == s2.values
Out[81]: array([False,  True, False])

If you want to compare Series aligning its .index, see flexible comparison methods section below:

In [82]: s1.eq(s2)
Out[82]: 
A    False
B     True
C    False
D    False
Length: 4, dtype: bool

Current behavior (DataFrame, no change):

In [3]: df1 == df2
Out[3]:
ValueError: Can only compare identically-labeled DataFrame objects

Logical operators#

Logical operators align both .index of left and right hand side.

Previous behavior (Series), only left hand side index was kept:

In [4]: s1 = pd.Series([True, False, True], index=list('ABC'))
In [5]: s2 = pd.Series([True, True, True], index=list('ABD'))
In [6]: s1 & s2
Out[6]:
A     True
B    False
C    False
dtype: bool

New behavior (Series):

In [83]: s1 = pd.Series([True, False, True], index=list("ABC"))

In [84]: s2 = pd.Series([True, True, True], index=list("ABD"))

In [85]: s1 & s2
Out[85]: 
A     True
B    False
C    False
D    False
Length: 4, dtype: bool

Note

Series logical operators fill a NaN result with False.

Note

To achieve the same result as previous versions (compare values based on only left hand side index), you can use reindex_like:

In [86]: s1 & s2.reindex_like(s1)
Out[86]: 
A     True
B    False
C    False
Length: 3, dtype: bool

Current behavior (DataFrame, no change):

In [87]: df1 = pd.DataFrame([True, False, True], index=list("ABC"))

In [88]: df2 = pd.DataFrame([True, True, True], index=list("ABD"))

In [89]: df1 & df2
Out[89]: 
       0
A   True
B  False
C  False
D  False

[4 rows x 1 columns]

Flexible comparison methods#

Series flexible comparison methods like eq, ne, le, lt, ge and gt now align both index. Use these operators if you want to compare two Series which has the different index.

In [90]: s1 = pd.Series([1, 2, 3], index=["a", "b", "c"])

In [91]: s2 = pd.Series([2, 2, 2], index=["b", "c", "d"])

In [92]: s1.eq(s2)
Out[92]: 
a    False
b     True
c    False
d    False
Length: 4, dtype: bool

In [93]: s1.ge(s2)
Out[93]: 
a    False
b     True
c     True
d    False
Length: 4, dtype: bool

Previously, this worked the same as comparison operators (see above).

`Series` type promotion on assignment#

A Series will now correctly promote its dtype for assignment with incompat values to the current dtype (GH 13234)

In [94]: s = pd.Series()

Previous behavior:

In [2]: s["a"] = pd.Timestamp("2016-01-01")

In [3]: s["b"] = 3.0
TypeError: invalid type promotion

New behavior:

In [95]: s["a"] = pd.Timestamp("2016-01-01")

In [96]: s["b"] = 3.0

In [97]: s
Out[97]: 
a    2016-01-01 00:00:00
b                    3.0
Length: 2, dtype: object

In [98]: s.dtype
Out[98]: dtype('O')

Function `.to_datetime()` changes#

Previously if .to_datetime() encountered mixed integers/floats and strings, but no datetimes with errors='coerce' it would convert all to NaT.

Previous behavior:

In [2]: pd.to_datetime([1, 'foo'], errors='coerce')
Out[2]: DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)

Current behavior:

This will now convert integers/floats with the default unit of ns.

In [99]: pd.to_datetime([1, "foo"], errors="coerce")
Out[99]: DatetimeIndex(['1970-01-01 00:00:00.000000001', 'NaT'], dtype='datetime64[ns]', freq=None)

Bug fixes related to .to_datetime():

Bug in pd.to_datetime() when passing integers or floats, and no unit and errors='coerce' (GH 13180).
Bug in pd.to_datetime() when passing invalid data types (e.g. bool); will now respect the errors keyword (GH 13176)
Bug in pd.to_datetime() which overflowed on int8, and int16 dtypes (GH 13451)
Bug in pd.to_datetime() raise AttributeError with NaN and the other string is not valid when errors='ignore' (GH 12424)
Bug in pd.to_datetime() did not cast floats correctly when unit was specified, resulting in truncated datetime (GH 13834)

Merging changes#

Merging will now preserve the dtype of the join keys (GH 8596)

In [100]: df1 = pd.DataFrame({"key": [1], "v1": [10]})

In [101]: df1
Out[101]: 
   key  v1
0    1  10

[1 rows x 2 columns]

In [102]: df2 = pd.DataFrame({"key": [1, 2], "v1": [20, 30]})

In [103]: df2
Out[103]: 
   key  v1
0    1  20
1    2  30

[2 rows x 2 columns]

Previous behavior:

In [5]: pd.merge(df1, df2, how='outer')
Out[5]:
   key    v1
0  1.0  10.0
1  1.0  20.0
2  2.0  30.0

In [6]: pd.merge(df1, df2, how='outer').dtypes
Out[6]:
key    float64
v1     float64
dtype: object

New behavior:

We are able to preserve the join keys

In [104]: pd.merge(df1, df2, how="outer")
Out[104]: 
   key  v1
0    1  10
1    1  20
2    2  30

[3 rows x 2 columns]

In [105]: pd.merge(df1, df2, how="outer").dtypes
Out[105]: 
key    int64
v1     int64
Length: 2, dtype: object

Of course if you have missing values that are introduced, then the resulting dtype will be upcast, which is unchanged from previous.

In [106]: pd.merge(df1, df2, how="outer", on="key")
Out[106]: 
   key  v1_x  v1_y
0    1  10.0    20
1    2   NaN    30

[2 rows x 3 columns]

In [107]: pd.merge(df1, df2, how="outer", on="key").dtypes
Out[107]: 
key       int64
v1_x    float64
v1_y      int64
Length: 3, dtype: object

Method `.describe()` changes#

Percentile identifiers in the index of a .describe() output will now be rounded to the least precision that keeps them distinct (GH 13104)

In [108]: s = pd.Series([0, 1, 2, 3, 4])

In [109]: df = pd.DataFrame([0, 1, 2, 3, 4])

Previous behavior:

The percentiles were rounded to at most one decimal place, which could raise ValueError for a data frame if the percentiles were duplicated.

In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[3]:
count     5.000000
mean      2.000000
std       1.581139
min       0.000000
0.0%      0.000400
0.1%      0.002000
0.1%      0.004000
50%       2.000000
99.9%     3.996000
100.0%    3.998000
100.0%    3.999600
max       4.000000
dtype: float64

In [4]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[4]:
...
ValueError: cannot reindex from a duplicate axis

New behavior:

In [110]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[110]: 
count     5.000000
mean      2.000000
std       1.581139
min       0.000000
0.01%     0.000400
0.05%     0.002000
0.1%      0.004000
50%       2.000000
99.9%     3.996000
99.95%    3.998000
99.99%    3.999600
max       4.000000
Length: 12, dtype: float64

In [111]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[111]: 
               0
count   5.000000
mean    2.000000
std     1.581139
min     0.000000
0.01%   0.000400
0.05%   0.002000
0.1%    0.004000
50%     2.000000
99.9%   3.996000
99.95%  3.998000
99.99%  3.999600
max     4.000000

[12 rows x 1 columns]

Furthermore:

Passing duplicated percentiles will now raise a ValueError.
Bug in .describe() on a DataFrame with a mixed-dtype column index, which would previously raise a TypeError (GH 13288)

`Period` changes#

The `PeriodIndex` now has `period` dtype#

PeriodIndex now has its own period dtype. The period dtype is a pandas extension dtype like category or the timezone aware dtype (datetime64[ns, tz]) (GH 13941). As a consequence of this change, PeriodIndex no longer has an integer dtype:

Previous behavior:

In [1]: pi = pd.PeriodIndex(['2016-08-01'], freq='D')

In [2]: pi
Out[2]: PeriodIndex(['2016-08-01'], dtype='int64', freq='D')

In [3]: pd.api.types.is_integer_dtype(pi)
Out[3]: True

In [4]: pi.dtype
Out[4]: dtype('int64')

New behavior:

In [112]: pi = pd.PeriodIndex(["2016-08-01"], freq="D")

In [113]: pi
Out[113]: PeriodIndex(['2016-08-01'], dtype='period[D]')

In [114]: pd.api.types.is_integer_dtype(pi)
Out[114]: False

In [115]: pd.api.types.is_period_dtype(pi)
Out[115]: True

In [116]: pi.dtype
Out[116]: period[D]

In [117]: type(pi.dtype)
Out[117]: pandas.PeriodDtype

`Period('NaT')` now returns `pd.NaT`#

Previously, Period has its own Period('NaT') representation different from pd.NaT. Now Period('NaT') has been changed to return pd.NaT. (GH 12759, GH 13582)

Previous behavior:

In [5]: pd.Period('NaT', freq='D')
Out[5]: Period('NaT', 'D')

New behavior:

These result in pd.NaT without providing freq option.

In [118]: pd.Period("NaT")
Out[118]: NaT

In [119]: pd.Period(None)
Out[119]: NaT

To be compatible with Period addition and subtraction, pd.NaT now supports addition and subtraction with int. Previously it raised ValueError.

Previous behavior:

In [5]: pd.NaT + 1
...
ValueError: Cannot add integral value to Timestamp without freq.

New behavior:

In [120]: pd.NaT + 1
Out[120]: NaT

In [121]: pd.NaT - 1
Out[121]: NaT

`PeriodIndex.values` now returns array of `Period` object#

.values is changed to return an array of Period objects, rather than an array of integers (GH 13988).

Previous behavior:

In [6]: pi = pd.PeriodIndex(['2011-01', '2011-02'], freq='M')
In [7]: pi.values
Out[7]: array([492, 493])

New behavior:

In [122]: pi = pd.PeriodIndex(["2011-01", "2011-02"], freq="M")

In [123]: pi.values
Out[123]: array([Period('2011-01', 'M'), Period('2011-02', 'M')], dtype=object)

Index `+` / `-` no longer used for set operations#

Addition and subtraction of the base Index type and of DatetimeIndex (not the numeric index types) previously performed set operations (set union and difference). This behavior was already deprecated since 0.15.0 (in favor using the specific .union() and .difference() methods), and is now disabled. When possible, + and - are now used for element-wise operations, for example for concatenating strings or subtracting datetimes (GH 8227, GH 14127).

Previous behavior:

In [1]: pd.Index(['a', 'b']) + pd.Index(['a', 'c'])
FutureWarning: using '+' to provide set union with Indexes is deprecated, use '|' or .union()
Out[1]: Index(['a', 'b', 'c'], dtype='object')

New behavior: the same operation will now perform element-wise addition:

In [124]: pd.Index(["a", "b"]) + pd.Index(["a", "c"])
Out[124]: Index(['aa', 'bc'], dtype='object')

Note that numeric Index objects already performed element-wise operations. For example, the behavior of adding two integer Indexes is unchanged. The base Index is now made consistent with this behavior.

In [125]: pd.Index([1, 2, 3]) + pd.Index([2, 3, 4])
Out[125]: Index([3, 5, 7], dtype='int64')

Further, because of this change, it is now possible to subtract two DatetimeIndex objects resulting in a TimedeltaIndex:

Previous behavior:

In [1]: (pd.DatetimeIndex(['2016-01-01', '2016-01-02'])
   ...:  - pd.DatetimeIndex(['2016-01-02', '2016-01-03']))
FutureWarning: using '-' to provide set differences with datetimelike Indexes is deprecated, use .difference()
Out[1]: DatetimeIndex(['2016-01-01'], dtype='datetime64[ns]', freq=None)

New behavior:

In [126]: (
   .....:     pd.DatetimeIndex(["2016-01-01", "2016-01-02"])
   .....:     - pd.DatetimeIndex(["2016-01-02", "2016-01-03"])
   .....: )
   .....: 
Out[126]: TimedeltaIndex(['-1 days', '-1 days'], dtype='timedelta64[s]', freq=None)

`Index.difference` and `.symmetric_difference` changes#

Index.difference and Index.symmetric_difference will now, more consistently, treat NaN values as any other values. (GH 13514)

In [127]: idx1 = pd.Index([1, 2, 3, np.nan])

In [128]: idx2 = pd.Index([0, 1, np.nan])

Previous behavior:

In [3]: idx1.difference(idx2)
Out[3]: Float64Index([nan, 2.0, 3.0], dtype='float64')

In [4]: idx1.symmetric_difference(idx2)
Out[4]: Float64Index([0.0, nan, 2.0, 3.0], dtype='float64')

New behavior:

In [129]: idx1.difference(idx2)
Out[129]: Index([2.0, 3.0], dtype='float64')

In [130]: idx1.symmetric_difference(idx2)
Out[130]: Index([0.0, 2.0, 3.0], dtype='float64')

`Index.unique` consistently returns `Index`#

Index.unique() now returns unique values as an Index of the appropriate dtype. (GH 13395). Previously, most Index classes returned np.ndarray, and DatetimeIndex, TimedeltaIndex and PeriodIndex returned Index to keep metadata like timezone.

Previous behavior:

In [1]: pd.Index([1, 2, 3]).unique()
Out[1]: array([1, 2, 3])

In [2]: pd.DatetimeIndex(['2011-01-01', '2011-01-02',
   ...:                   '2011-01-03'], tz='Asia/Tokyo').unique()
Out[2]:
DatetimeIndex(['2011-01-01 00:00:00+09:00', '2011-01-02 00:00:00+09:00',
               '2011-01-03 00:00:00+09:00'],
              dtype='datetime64[ns, Asia/Tokyo]', freq=None)

New behavior:

In [131]: pd.Index([1, 2, 3]).unique()
Out[131]: Index([1, 2, 3], dtype='int64')

In [132]: pd.DatetimeIndex(
   .....:     ["2011-01-01", "2011-01-02", "2011-01-03"], tz="Asia/Tokyo"
   .....: ).unique()
   .....: 
Out[132]: 
DatetimeIndex(['2011-01-01 00:00:00+09:00', '2011-01-02 00:00:00+09:00',
               '2011-01-03 00:00:00+09:00'],
              dtype='datetime64[s, Asia/Tokyo]', freq=None)

`MultiIndex` constructors, `groupby` and `set_index` preserve categorical dtypes#

MultiIndex.from_arrays and MultiIndex.from_product will now preserve categorical dtype in MultiIndex levels (GH 13743, GH 13854).

In [133]: cat = pd.Categorical(["a", "b"], categories=list("bac"))

In [134]: lvl1 = ["foo", "bar"]

In [135]: midx = pd.MultiIndex.from_arrays([cat, lvl1])

In [136]: midx
Out[136]: 
MultiIndex([('a', 'foo'),
            ('b', 'bar')],
           )

Previous behavior:

In [4]: midx.levels[0]
Out[4]: Index(['b', 'a', 'c'], dtype='object')

In [5]: midx.get_level_values[0]
Out[5]: Index(['a', 'b'], dtype='object')

New behavior: the single level is now a CategoricalIndex:

In [137]: midx.levels[0]
Out[137]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category')

In [138]: midx.get_level_values(0)
Out[138]: CategoricalIndex(['a', 'b'], categories=['b', 'a', 'c'], ordered=False, dtype='category')

An analogous change has been made to MultiIndex.from_product. As a consequence, groupby and set_index also preserve categorical dtypes in indexes

In [139]: df = pd.DataFrame({"A": [0, 1], "B": [10, 11], "C": cat})

In [140]: df_grouped = df.groupby(by=["A", "C"], observed=False).first()

In [141]: df_set_idx = df.set_index(["A", "C"])

Previous behavior:

In [11]: df_grouped.index.levels[1]
Out[11]: Index(['b', 'a', 'c'], dtype='object', name='C')
In [12]: df_grouped.reset_index().dtypes
Out[12]:
A      int64
C     object
B    float64
dtype: object

In [13]: df_set_idx.index.levels[1]
Out[13]: Index(['b', 'a', 'c'], dtype='object', name='C')
In [14]: df_set_idx.reset_index().dtypes
Out[14]:
A      int64
C     object
B      int64
dtype: object

New behavior:

In [142]: df_grouped.index.levels[1]
Out[142]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category', name='C')

In [143]: df_grouped.reset_index().dtypes
Out[143]: 
A       int64
C    category
B     float64
Length: 3, dtype: object

In [144]: df_set_idx.index.levels[1]
Out[144]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category', name='C')

In [145]: df_set_idx.reset_index().dtypes
Out[145]: 
A       int64
C    category
B       int64
Length: 3, dtype: object

Function `read_csv` will progressively enumerate chunks#

When read_csv() is called with chunksize=n and without specifying an index, each chunk used to have an independently generated index from 0 to n-1. They are now given instead a progressive index, starting from 0 for the first chunk, from n for the second, and so on, so that, when concatenated, they are identical to the result of calling read_csv() without the chunksize= argument (GH 12185).

In [146]: data = "A,B\n0,1\n2,3\n4,5\n6,7"

Previous behavior:

In [2]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))
Out[2]:
   A  B
0  0  1
1  2  3
0  4  5
1  6  7

New behavior:

In [147]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))
Out[147]: 
   A  B
0  0  1
1  2  3
2  4  5
3  6  7

[4 rows x 2 columns]

Sparse changes#

These changes allow pandas to handle sparse data with more dtypes, and for work to make a smoother experience with data handling.

Types `int64` and `bool` support enhancements#

Sparse data structures now gained enhanced support of int64 and bool dtype (GH 667, GH 13849).

Previously, sparse data were float64 dtype by default, even if all inputs were of int or bool dtype. You had to specify dtype explicitly to create sparse data with int64 dtype. Also, fill_value had to be specified explicitly because the default was np.nan which doesn’t appear in int64 or bool data.

In [1]: pd.SparseArray([1, 2, 0, 0])
Out[1]:
[1.0, 2.0, 0.0, 0.0]
Fill: nan
IntIndex
Indices: array([0, 1, 2, 3], dtype=int32)

# specifying int64 dtype, but all values are stored in sp_values because
# fill_value default is np.nan
In [2]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64)
Out[2]:
[1, 2, 0, 0]
Fill: nan
IntIndex
Indices: array([0, 1, 2, 3], dtype=int32)

In [3]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64, fill_value=0)
Out[3]:
[1, 2, 0, 0]
Fill: 0
IntIndex
Indices: array([0, 1], dtype=int32)

As of v0.19.0, sparse data keeps the input dtype, and uses more appropriate fill_value defaults (0 for int64 dtype, False for bool dtype).

In [148]: pd.arrays.SparseArray([1, 2, 0, 0], dtype=np.int64)
Out[148]: 
[1, 2, 0, 0]
Fill: 0
IntIndex
Indices: array([0, 1], dtype=int32)

In [149]: pd.arrays.SparseArray([True, False, False, False])
Out[149]: 
[True, False, False, False]
Fill: False
IntIndex
Indices: array([0], dtype=int32)

See the docs for more details.

Operators now preserve dtypes#

Sparse data structure now can preserve dtype after arithmetic ops (GH 13848)

s = pd.SparseSeries([0, 2, 0, 1], fill_value=0, dtype=np.int64)
s.dtype

s + 1

Sparse data structure now support astype to convert internal dtype (GH 13900)

s = pd.SparseSeries([1.0, 0.0, 2.0, 0.0], fill_value=0)
s
s.astype(np.int64)

astype fails if data contains values which cannot be converted to specified dtype. Note that the limitation is applied to fill_value which default is np.nan.

In [7]: pd.SparseSeries([1., np.nan, 2., np.nan], fill_value=np.nan).astype(np.int64)
Out[7]:
ValueError: unable to coerce current fill_value nan to int64 dtype

Other sparse fixes#

Subclassed SparseDataFrame and SparseSeries now preserve class types when slicing or transposing. (GH 13787)
SparseArray with bool dtype now supports logical (bool) operators (GH 14000)
Bug in SparseSeries with MultiIndex [] indexing may raise IndexError (GH 13144)
Bug in SparseSeries with MultiIndex [] indexing result may have normal Index (GH 13144)
Bug in SparseDataFrame in which axis=None did not default to axis=0 (GH 13048)
Bug in SparseSeries and SparseDataFrame creation with object dtype may raise TypeError (GH 11633)
Bug in SparseDataFrame doesn’t respect passed SparseArray or SparseSeries ‘s dtype and fill_value (GH 13866)
Bug in SparseArray and SparseSeries don’t apply ufunc to fill_value (GH 13853)
Bug in SparseSeries.abs incorrectly keeps negative fill_value (GH 13853)
Bug in single row slicing on multi-type SparseDataFrame s, types were previously forced to float (GH 13917)
Bug in SparseSeries slicing changes integer dtype to float (GH 8292)
Bug in SparseDataFarme comparison ops may raise TypeError (GH 13001)
Bug in SparseDataFarme.isnull raises ValueError (GH 8276)
Bug in SparseSeries representation with bool dtype may raise IndexError (GH 13110)
Bug in SparseSeries and SparseDataFrame of bool or int64 dtype may display its values like float64 dtype (GH 13110)
Bug in sparse indexing using SparseArray with bool dtype may return incorrect result (GH 13985)
Bug in SparseArray created from SparseSeries may lose dtype (GH 13999)
Bug in SparseSeries comparison with dense returns normal Series rather than SparseSeries (GH 13999)

Indexer dtype changes#

Note

This change only affects 64 bit python running on Windows, and only affects relatively advanced indexing operations

Methods such as Index.get_indexer that return an indexer array, coerce that array to a “platform int”, so that it can be directly used in 3rd party library operations like numpy.take. Previously, a platform int was defined as np.int_ which corresponds to a C integer, but the correct type, and what is being used now, is np.intp, which corresponds to the C integer size that can hold a pointer (GH 3033, GH 13972).

These types are the same on many platform, but for 64 bit python on Windows, np.int_ is 32 bits, and np.intp is 64 bits. Changing this behavior improves performance for many operations on that platform.

Previous behavior:

In [1]: i = pd.Index(['a', 'b', 'c'])

In [2]: i.get_indexer(['b', 'b', 'c']).dtype
Out[2]: dtype('int32')

New behavior:

In [1]: i = pd.Index(['a', 'b', 'c'])

In [2]: i.get_indexer(['b', 'b', 'c']).dtype
Out[2]: dtype('int64')

Other API changes#

Timestamp.to_pydatetime will issue a UserWarning when warn=True, and the instance has a non-zero number of nanoseconds, previously this would print a message to stdout (GH 14101).
Series.unique() with datetime and timezone now returns return array of Timestamp with timezone (GH 13565).
Panel.to_sparse() will raise a NotImplementedError exception when called (GH 13778).
Index.reshape() will raise a NotImplementedError exception when called (GH 12882).
.filter() enforces mutual exclusion of the keyword arguments (GH 12399).
eval’s upcasting rules for float32 types have been updated to be more consistent with NumPy’s rules. New behavior will not upcast to float64 if you multiply a pandas float32 object by a scalar float64 (GH 12388).
An UnsupportedFunctionCall error is now raised if NumPy ufuncs like np.mean are called on groupby or resample objects (GH 12811).
__setitem__ will no longer apply a callable rhs as a function instead of storing it. Call where directly to get the previous behavior (GH 13299).
Calls to .sample() will respect the random seed set via numpy.random.seed(n) (GH 13161)
Styler.apply is now more strict about the outputs your function must return. For axis=0 or axis=1, the output shape must be identical. For axis=None, the output must be a DataFrame with identical columns and index labels (GH 13222).
Float64Index.astype(int) will now raise ValueError if Float64Index contains NaN values (GH 13149)
TimedeltaIndex.astype(int) and DatetimeIndex.astype(int) will now return Int64Index instead of np.array (GH 13209)
Passing Period with multiple frequencies to normal Index now returns Index with object dtype (GH 13664)
PeriodIndex.fillna with Period has different freq now coerces to object dtype (GH 13664)
Faceted boxplots from DataFrame.boxplot(by=col) now return a Series when return_type is not None. Previously these returned an OrderedDict. Note that when return_type=None, the default, these still return a 2-D NumPy array (GH 12216, GH 7096).
pd.read_hdf will now raise a ValueError instead of KeyError, if a mode other than r, r+ and a is supplied. (GH 13623)
pd.read_csv(), pd.read_table(), and pd.read_hdf() raise the builtin FileNotFoundError exception for Python 3.x when called on a nonexistent file; this is back-ported as IOError in Python 2.x (GH 14086)
More informative exceptions are passed through the csv parser. The exception type would now be the original exception type instead of CParserError (GH 13652).
pd.read_csv() in the C engine will now issue a ParserWarning or raise a ValueError when sep encoded is more than one character long (GH 14065)
DataFrame.values will now return float64 with a DataFrame of mixed int64 and uint64 dtypes, conforming to np.find_common_type (GH 10364, GH 13917)
.groupby.groups will now return a dictionary of Index objects, rather than a dictionary of np.ndarray or lists (GH 14293)

Deprecations#

Series.reshape and Categorical.reshape have been deprecated and will be removed in a subsequent release (GH 12882, GH 12882)
PeriodIndex.to_datetime has been deprecated in favor of PeriodIndex.to_timestamp (GH 8254)
Timestamp.to_datetime has been deprecated in favor of Timestamp.to_pydatetime (GH 8254)
Index.to_datetime and DatetimeIndex.to_datetime have been deprecated in favor of pd.to_datetime (GH 8254)
pandas.core.datetools module has been deprecated and will be removed in a subsequent release (GH 14094)
SparseList has been deprecated and will be removed in a future version (GH 13784)
DataFrame.to_html() and DataFrame.to_latex() have dropped the colSpace parameter in favor of col_space (GH 13857)
DataFrame.to_sql() has deprecated the flavor parameter, as it is superfluous when SQLAlchemy is not installed (GH 13611)
Deprecated read_csv keywords:
- compact_ints and use_unsigned have been deprecated and will be removed in a future version (GH 13320)
- buffer_lines has been deprecated and will be removed in a future version (GH 13360)
- as_recarray has been deprecated and will be removed in a future version (GH 13373)
- skip_footer has been deprecated in favor of skipfooter and will be removed in a future version (GH 13349)
top-level pd.ordered_merge() has been renamed to pd.merge_ordered() and the original name will be removed in a future version (GH 13358)
Timestamp.offset property (and named arg in the constructor), has been deprecated in favor of freq (GH 12160)
pd.tseries.util.pivot_annual is deprecated. Use pivot_table as alternative, an example is here (GH 736)
pd.tseries.util.isleapyear has been deprecated and will be removed in a subsequent release. Datetime-likes now have a .is_leap_year property (GH 13727)
Panel4D and PanelND constructors are deprecated and will be removed in a future version. The recommended way to represent these types of n-dimensional data are with the xarray package. pandas provides a to_xarray() method to automate this conversion (GH 13564).
pandas.tseries.frequencies.get_standard_freq is deprecated. Use pandas.tseries.frequencies.to_offset(freq).rule_code instead (GH 13874)
pandas.tseries.frequencies.to_offset’s freqstr keyword is deprecated in favor of freq (GH 13874)
Categorical.from_array has been deprecated and will be removed in a future version (GH 13854)

Removal of prior version deprecations/changes#

The SparsePanel class has been removed (GH 13778)
The pd.sandbox module has been removed in favor of the external library pandas-qt (GH 13670)
The pandas.io.data and pandas.io.wb modules are removed in favor of the pandas-datareader package (GH 13724).
The pandas.tools.rplot module has been removed in favor of the seaborn package (GH 13855)
DataFrame.to_csv() has dropped the engine parameter, as was deprecated in 0.17.1 (GH 11274, GH 13419)
DataFrame.to_dict() has dropped the outtype parameter in favor of orient (GH 13627, GH 8486)
pd.Categorical has dropped setting of the ordered attribute directly in favor of the set_ordered method (GH 13671)
pd.Categorical has dropped the levels attribute in favor of categories (GH 8376)
DataFrame.to_sql() has dropped the mysql option for the flavor parameter (GH 13611)
Panel.shift() has dropped the lags parameter in favor of periods (GH 14041)
pd.Index has dropped the diff method in favor of difference (GH 13669)
pd.DataFrame has dropped the to_wide method in favor of to_panel (GH 14039)
Series.to_csv has dropped the nanRep parameter in favor of na_rep (GH 13804)
Series.xs, DataFrame.xs, Panel.xs, Panel.major_xs, and Panel.minor_xs have dropped the copy parameter (GH 13781)
str.split has dropped the return_type parameter in favor of expand (GH 13701)
Removal of the legacy time rules (offset aliases), deprecated since 0.17.0 (this has been alias since 0.8.0) (GH 13590, GH 13868). Now legacy time rules raises ValueError. For the list of currently supported offsets, see here.
The default value for the return_type parameter for DataFrame.plot.box and DataFrame.boxplot changed from None to "axes". These methods will now return a matplotlib axes by default instead of a dictionary of artists. See here (GH 6581).
The tquery and uquery functions in the pandas.io.sql module are removed (GH 5950).

Performance improvements#

Improved performance of sparse IntIndex.intersect (GH 13082)
Improved performance of sparse arithmetic with BlockIndex when the number of blocks are large, though recommended to use IntIndex in such cases (GH 13082)
Improved performance of DataFrame.quantile() as it now operates per-block (GH 11623)
Improved performance of float64 hash table operations, fixing some very slow indexing and groupby operations in python 3 (GH 13166, GH 13334)
Improved performance of DataFrameGroupBy.transform (GH 12737)
Improved performance of Index and Series .duplicated (GH 10235)
Improved performance of Index.difference (GH 12044)
Improved performance of RangeIndex.is_monotonic_increasing and is_monotonic_decreasing (GH 13749)
Improved performance of datetime string parsing in DatetimeIndex (GH 13692)
Improved performance of hashing Period (GH 12817)
Improved performance of factorize of datetime with timezone (GH 13750)
Improved performance of by lazily creating indexing hashtables on larger Indexes (GH 14266)
Improved performance of groupby.groups (GH 14293)
Unnecessary materializing of a MultiIndex when introspecting for memory usage (GH 14308)

Bug fixes#

Bug in groupby().shift(), which could cause a segfault or corruption in rare circumstances when grouping by columns with missing values (GH 13813)
Bug in groupby().cumsum() calculating cumprod when axis=1. (GH 13994)
Bug in pd.to_timedelta() in which the errors parameter was not being respected (GH 13613)
Bug in io.json.json_normalize(), where non-ascii keys raised an exception (GH 13213)
Bug when passing a not-default-indexed Series as xerr or yerr in .plot() (GH 11858)
Bug in area plot draws legend incorrectly if subplot is enabled or legend is moved after plot (matplotlib 1.5.0 is required to draw area plot legend properly) (GH 9161, GH 13544)
Bug in DataFrame assignment with an object-dtyped Index where the resultant column is mutable to the original object. (GH 13522)
Bug in matplotlib AutoDataFormatter; this restores the second scaled formatting and re-adds micro-second scaled formatting (GH 13131)
Bug in selection from a HDFStore with a fixed format and start and/or stop specified will now return the selected range (GH 8287)
Bug in Categorical.from_codes() where an unhelpful error was raised when an invalid ordered parameter was passed in (GH 14058)
Bug in Series construction from a tuple of integers on windows not returning default dtype (int64) (GH 13646)
Bug in TimedeltaIndex addition with a Datetime-like object where addition overflow was not being caught (GH 14068)
Bug in .groupby(..).resample(..) when the same object is called multiple times (GH 13174)
Bug in .to_records() when index name is a unicode string (GH 13172)
Bug in calling .memory_usage() on object which doesn’t implement (GH 12924)
Regression in Series.quantile with nans (also shows up in .median() and .describe() ); furthermore now names the Series with the quantile (GH 13098, GH 13146)
Bug in SeriesGroupBy.transform with datetime values and missing groups (GH 13191)
Bug where empty Series were incorrectly coerced in datetime-like numeric operations (GH 13844)
Bug in Categorical constructor when passed a Categorical containing datetimes with timezones (GH 14190)
Bug in Series.str.extractall() with str index raises ValueError (GH 13156)
Bug in Series.str.extractall() with single group and quantifier (GH 13382)
Bug in DatetimeIndex and Period subtraction raises ValueError or AttributeError rather than TypeError (GH 13078)
Bug in Index and Series created with NaN and NaT mixed data may not have datetime64 dtype (GH 13324)
Bug in Index and Series may ignore np.datetime64('nat') and np.timdelta64('nat') to infer dtype (GH 13324)
Bug in PeriodIndex and Period subtraction raises AttributeError (GH 13071)
Bug in PeriodIndex construction returning a float64 index in some circumstances (GH 13067)
Bug in .resample(..) with a PeriodIndex not changing its freq appropriately when empty (GH 13067)
Bug in .resample(..) with a PeriodIndex not retaining its type or name with an empty DataFrame appropriately when empty (GH 13212)
Bug in groupby(..).apply(..) when the passed function returns scalar values per group (GH 13468).
Bug in groupby(..).resample(..) where passing some keywords would raise an exception (GH 13235)
Bug in .tz_convert on a tz-aware DateTimeIndex that relied on index being sorted for correct results (GH 13306)
Bug in .tz_localize with dateutil.tz.tzlocal may return incorrect result (GH 13583)
Bug in DatetimeTZDtype dtype with dateutil.tz.tzlocal cannot be regarded as valid dtype (GH 13583)
Bug in pd.read_hdf() where attempting to load an HDF file with a single dataset, that had one or more categorical columns, failed unless the key argument was set to the name of the dataset. (GH 13231)
Bug in .rolling() that allowed a negative integer window in construction of the Rolling() object, but would later fail on aggregation (GH 13383)
Bug in Series indexing with tuple-valued data and a numeric index (GH 13509)
Bug in printing pd.DataFrame where unusual elements with the object dtype were causing segfaults (GH 13717)
Bug in ranking Series which could result in segfaults (GH 13445)
Bug in various index types, which did not propagate the name of passed index (GH 12309)
Bug in DatetimeIndex, which did not honour the copy=True (GH 13205)
Bug in DatetimeIndex.is_normalized returns incorrectly for normalized date_range in case of local timezones (GH 13459)
Bug in pd.concat and .append may coerces datetime64 and timedelta to object dtype containing python built-in datetime or timedelta rather than Timestamp or Timedelta (GH 13626)
Bug in PeriodIndex.append may raises AttributeError when the result is object dtype (GH 13221)
Bug in CategoricalIndex.append may accept normal list (GH 13626)
Bug in pd.concat and .append with the same timezone get reset to UTC (GH 7795)
Bug in Series and DataFrame .append raises AmbiguousTimeError if data contains datetime near DST boundary (GH 13626)
Bug in DataFrame.to_csv() in which float values were being quoted even though quotations were specified for non-numeric values only (GH 12922, GH 13259)
Bug in DataFrame.describe() raising ValueError with only boolean columns (GH 13898)
Bug in MultiIndex slicing where extra elements were returned when level is non-unique (GH 12896)
Bug in .str.replace does not raise TypeError for invalid replacement (GH 13438)
Bug in MultiIndex.from_arrays which didn’t check for input array lengths matching (GH 13599)
Bug in cartesian_product and MultiIndex.from_product which may raise with empty input arrays (GH 12258)
Bug in pd.read_csv() which may cause a segfault or corruption when iterating in large chunks over a stream/file under rare circumstances (GH 13703)
Bug in pd.read_csv() which caused errors to be raised when a dictionary containing scalars is passed in for na_values (GH 12224)
Bug in pd.read_csv() which caused BOM files to be incorrectly parsed by not ignoring the BOM (GH 4793)
Bug in pd.read_csv() with engine='python' which raised errors when a numpy array was passed in for usecols (GH 12546)
Bug in pd.read_csv() where the index columns were being incorrectly parsed when parsed as dates with a thousands parameter (GH 14066)
Bug in pd.read_csv() with engine='python' in which NaN values weren’t being detected after data was converted to numeric values (GH 13314)
Bug in pd.read_csv() in which the nrows argument was not properly validated for both engines (GH 10476)
Bug in pd.read_csv() with engine='python' in which infinities of mixed-case forms were not being interpreted properly (GH 13274)
Bug in pd.read_csv() with engine='python' in which trailing NaN values were not being parsed (GH 13320)
Bug in pd.read_csv() with engine='python' when reading from a tempfile.TemporaryFile on Windows with Python 3 (GH 13398)
Bug in pd.read_csv() that prevents usecols kwarg from accepting single-byte unicode strings (GH 13219)
Bug in pd.read_csv() that prevents usecols from being an empty set (GH 13402)
Bug in pd.read_csv() in the C engine where the NULL character was not being parsed as NULL (GH 14012)
Bug in pd.read_csv() with engine='c' in which NULL quotechar was not accepted even though quoting was specified as None (GH 13411)
Bug in pd.read_csv() with engine='c' in which fields were not properly cast to float when quoting was specified as non-numeric (GH 13411)
Bug in pd.read_csv() in Python 2.x with non-UTF8 encoded, multi-character separated data (GH 3404)
Bug in pd.read_csv(), where aliases for utf-xx (e.g. UTF-xx, UTF_xx, utf_xx) raised UnicodeDecodeError (GH 13549)
Bug in pd.read_csv, pd.read_table, pd.read_fwf, pd.read_stata and pd.read_sas where files were opened by parsers but not closed if both chunksize and iterator were None. (GH 13940)
Bug in StataReader, StataWriter, XportReader and SAS7BDATReader where a file was not properly closed when an error was raised. (GH 13940)
Bug in pd.pivot_table() where margins_name is ignored when aggfunc is a list (GH 13354)
Bug in pd.Series.str.zfill, center, ljust, rjust, and pad when passing non-integers, did not raise TypeError (GH 13598)
Bug in checking for any null objects in a TimedeltaIndex, which always returned True (GH 13603)
Bug in Series arithmetic raises TypeError if it contains datetime-like as object dtype (GH 13043)
Bug Series.isnull() and Series.notnull() ignore Period('NaT') (GH 13737)
Bug Series.fillna() and Series.dropna() don’t affect to Period('NaT') (GH 13737
Bug in .fillna(value=np.nan) incorrectly raises KeyError on a category dtyped Series (GH 14021)
Bug in extension dtype creation where the created types were not is/identical (GH 13285)
Bug in .resample(..) where incorrect warnings were triggered by IPython introspection (GH 13618)
Bug in NaT - Period raises AttributeError (GH 13071)
Bug in Series comparison may output incorrect result if rhs contains NaT (GH 9005)
Bug in Series and Index comparison may output incorrect result if it contains NaT with object dtype (GH 13592)
Bug in Period addition raises TypeError if Period is on right hand side (GH 13069)
Bug in Period and Series or Index comparison raises TypeError (GH 13200)
Bug in pd.set_eng_float_format() that would prevent NaN and Inf from formatting (GH 11981)
Bug in .unstack with Categorical dtype resets .ordered to True (GH 13249)
Clean some compile time warnings in datetime parsing (GH 13607)
Bug in factorize raises AmbiguousTimeError if data contains datetime near DST boundary (GH 13750)
Bug in .set_index raises AmbiguousTimeError if new index contains DST boundary and multi levels (GH 12920)
Bug in .shift raises AmbiguousTimeError if data contains datetime near DST boundary (GH 13926)
Bug in pd.read_hdf() returns incorrect result when a DataFrame with a categorical column and a query which doesn’t match any values (GH 13792)
Bug in .iloc when indexing with a non lexsorted MultiIndex (GH 13797)
Bug in .loc when indexing with date strings in a reverse sorted DatetimeIndex (GH 14316)
Bug in Series comparison operators when dealing with zero dim NumPy arrays (GH 13006)
Bug in .combine_first may return incorrect dtype (GH 7630, GH 10567)
Bug in groupby where apply returns different result depending on whether first result is None or not (GH 12824)
Bug in groupby(..).nth() where the group key is included inconsistently if called after .head()/.tail() (GH 12839)
Bug in .to_html, .to_latex and .to_string silently ignore custom datetime formatter passed through the formatters key word (GH 10690)
Bug in DataFrame.iterrows(), not yielding a Series subclasse if defined (GH 13977)
Bug in pd.to_numeric when errors='coerce' and input contains non-hashable objects (GH 13324)
Bug in invalid Timedelta arithmetic and comparison may raise ValueError rather than TypeError (GH 13624)
Bug in invalid datetime parsing in to_datetime and DatetimeIndex may raise TypeError rather than ValueError (GH 11169, GH 11287)
Bug in Index created with tz-aware Timestamp and mismatched tz option incorrectly coerces timezone (GH 13692)
Bug in DatetimeIndex with nanosecond frequency does not include timestamp specified with end (GH 13672)
Bug in Series when setting a slice with a np.timedelta64 (GH 14155)
Bug in Index raises OutOfBoundsDatetime if datetime exceeds datetime64[ns] bounds, rather than coercing to object dtype (GH 13663)
Bug in Index may ignore specified datetime64 or timedelta64 passed as dtype (GH 13981)
Bug in RangeIndex can be created without no arguments rather than raises TypeError (GH 13793)
Bug in .value_counts() raises OutOfBoundsDatetime if data exceeds datetime64[ns] bounds (GH 13663)
Bug in DatetimeIndex may raise OutOfBoundsDatetime if input np.datetime64 has other unit than ns (GH 9114)
Bug in Series creation with np.datetime64 which has other unit than ns as object dtype results in incorrect values (GH 13876)
Bug in resample with timedelta data where data was casted to float (GH 13119).
Bug in pd.isnull() pd.notnull() raise TypeError if input datetime-like has other unit than ns (GH 13389)
Bug in pd.merge() may raise TypeError if input datetime-like has other unit than ns (GH 13389)
Bug in HDFStore/read_hdf() discarded DatetimeIndex.name if tz was set (GH 13884)
Bug in Categorical.remove_unused_categories() changes .codes dtype to platform int (GH 13261)
Bug in groupby with as_index=False returns all NaN’s when grouping on multiple columns including a categorical one (GH 13204)
Bug in df.groupby(...)[...] where getitem with Int64Index raised an error (GH 13731)
Bug in the CSS classes assigned to DataFrame.style for index names. Previously they were assigned "col_heading level<n> col<c>" where n was the number of levels + 1. Now they are assigned "index_name level<n>", where n is the correct level for that MultiIndex.
Bug where pd.read_gbq() could throw ImportError: No module named discovery as a result of a naming conflict with another python package called apiclient (GH 13454)
Bug in Index.union returns an incorrect result with a named empty index (GH 13432)
Bugs in Index.difference and DataFrame.join raise in Python3 when using mixed-integer indexes (GH 13432, GH 12814)
Bug in subtract tz-aware datetime.datetime from tz-aware datetime64 series (GH 14088)
Bug in .to_excel() when DataFrame contains a MultiIndex which contains a label with a NaN value (GH 13511)
Bug in invalid frequency offset string like “D1”, “-2-3H” may not raise ValueError (GH 13930)
Bug in concat and groupby for hierarchical frames with RangeIndex levels (GH 13542).
Bug in Series.str.contains() for Series containing only NaN values of object dtype (GH 14171)
Bug in agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 (GH 12821)
Bug in using NumPy ufunc with PeriodIndex to add or subtract integer raise IncompatibleFrequency. Note that using standard operator like + or - is recommended, because standard operators use more efficient path (GH 13980)
Bug in operations on NaT returning float instead of datetime64[ns] (GH 12941)
Bug in Series flexible arithmetic methods (like .add()) raises ValueError when axis=None (GH 13894)
Bug in DataFrame.to_csv() with MultiIndex columns in which a stray empty line was added (GH 6618)
Bug in DatetimeIndex, TimedeltaIndex and PeriodIndex.equals() may return True when input isn’t Index but contains the same values (GH 13107)
Bug in assignment against datetime with timezone may not work if it contains datetime near DST boundary (GH 14146)
Bug in pd.eval() and HDFStore query truncating long float literals with python 2 (GH 14241)
Bug in Index raises KeyError displaying incorrect column when column is not in the df and columns contains duplicate values (GH 13822)
Bug in Period and PeriodIndex creating wrong dates when frequency has combined offset aliases (GH 13874)
Bug in .to_string() when called with an integer line_width and index=False raises an UnboundLocalError exception because idx referenced before assignment.
Bug in eval() where the resolvers argument would not accept a list (GH 14095)
Bugs in stack, get_dummies, make_axis_dummies which don’t preserve categorical dtypes in (multi)indexes (GH 13854)
PeriodIndex can now accept list and array which contains pd.NaT (GH 13430)
Bug in df.groupby where .median() returns arbitrary values if grouped dataframe contains empty bins (GH 13629)
Bug in Index.copy() where name parameter was ignored (GH 14302)

Contributors#

A total of 117 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

Adrien Emery +
Alex Alekseyev
Alex Vig +
Allen Riddell +
Amol +
Amol Agrawal +
Andy R. Terrel +
Anthonios Partheniou
Ben Kandel +
Bob Baxley +
Brett Rosen +
Camilo Cota +
Chris
Chris Grinolds
Chris Warth
Christian Hudon
Christopher C. Aycock
Daniel Siladji +
Douglas McNeil
Drewrey Lupton +
Eduardo Blancas Reyes +
Elliot Marsden +
Evan Wright
Felix Marczinowski +
Francis T. O’Donovan
Geraint Duck +
Giacomo Ferroni +
Grant Roch +
Gábor Lipták
Haleemur Ali +
Hassan Shamim +
Iulius Curt +
Ivan Nazarov +
Jeff Reback
Jeffrey Gerard +
Jenn Olsen +
Jim Crist
Joe Jevnik
John Evans +
John Freeman
John Liekezer +
John W. O’Brien
John Zwinck +
Johnny Gill +
Jordan Erenrich +
Joris Van den Bossche
Josh Howes +
Jozef Brandys +
Ka Wo Chen
Kamil Sindi +
Kerby Shedden
Kernc +
Kevin Sheppard
Matthieu Brucher +
Maximilian Roos
Michael Scherer +
Mike Graham +
Mortada Mehyar
Muhammad Haseeb Tariq +
Nate George +
Neil Parley +
Nicolas Bonnotte
OXPHOS
Pan Deng / Zora +
Paul +
Paul Mestemaker +
Pauli Virtanen
Pawel Kordek +
Pietro Battiston
Piotr Jucha +
Ravi Kumar Nimmi +
Robert Gieseke
Robert Kern +
Roger Thomas
Roy Keyes +
Russell Smith +
Sahil Dua +
Sanjiv Lobo +
Sašo Stanovnik +
Shawn Heide +
Sinhrks
Stephen Kappel +
Steve Choi +
Stewart Henderson +
Sudarshan Konge +
Thomas A Caswell
Tom Augspurger
Tom Bird +
Uwe Hoffmann +
WillAyd +
Xiang Zhang +
YG-Riku +
Yadunandan +
Yaroslav Halchenko
Yuichiro Kaneko +
adneu
agraboso +
babakkeyvani +
c123w +
chris-b1
cmazzullo +
conquistador1492 +
cr3 +
dsm054
gfyoung
harshul1610 +
iamsimha +
jackieleng +
mpuels +
pijucha +
priyankjain +
sinhrks
wcwagner +
yui-knk +
zhangjinjie +
znmean +
颜发才（Yan Facai） +

Version 0.19.0 (October 2, 2016)#

New features#

Function merge_asof for asof-style time-series joining#

Method .rolling() is now time-series aware#

Method read_csv has improved support for duplicate column names#

Method read_csv supports parsing Categorical directly#

Categorical concatenation#

Semi-month offsets#

New Index methods#

Google BigQuery enhancements#

Fine-grained NumPy errstate#

Method get_dummies now returns integer dtypes#

Downcast values to smallest possible dtype in to_numeric#

pandas development API#

Other enhancements#

API changes#

Series.tolist() will now return Python types#

Series operators for different indexes#

Arithmetic operators#

Comparison operators#

Logical operators#

Flexible comparison methods#

Series type promotion on assignment#

Function .to_datetime() changes#

Merging changes#

Method .describe() changes#

Period changes#

The PeriodIndex now has period dtype#

Period('NaT') now returns pd.NaT#

PeriodIndex.values now returns array of Period object#

Index + / - no longer used for set operations#

Index.difference and .symmetric_difference changes#

Index.unique consistently returns Index#

MultiIndex constructors, groupby and set_index preserve categorical dtypes#

Function read_csv will progressively enumerate chunks#

Sparse changes#

Types int64 and bool support enhancements#

Operators now preserve dtypes#

Other sparse fixes#

Indexer dtype changes#

Other API changes#

Deprecations#

Removal of prior version deprecations/changes#

Performance improvements#

Bug fixes#

Contributors#

This Page

Function `merge_asof` for asof-style time-series joining#

Method `.rolling()` is now time-series aware#

Method `read_csv` has improved support for duplicate column names#

Method `read_csv` supports parsing `Categorical` directly#

Method `get_dummies` now returns integer dtypes#

Downcast values to smallest possible dtype in `to_numeric`#

`Series.tolist()` will now return Python types#

`Series` operators for different indexes#

`Series` type promotion on assignment#

Function `.to_datetime()` changes#

Method `.describe()` changes#

`Period` changes#

The `PeriodIndex` now has `period` dtype#

`Period('NaT')` now returns `pd.NaT`#

`PeriodIndex.values` now returns array of `Period` object#

Index `+` / `-` no longer used for set operations#

`Index.difference` and `.symmetric_difference` changes#

`Index.unique` consistently returns `Index`#

`MultiIndex` constructors, `groupby` and `set_index` preserve categorical dtypes#

Function `read_csv` will progressively enumerate chunks#

Types `int64` and `bool` support enhancements#