What’s new in 2.2.0 (Month XX, 2024)

These are the changes in pandas 2.2.0. See Release notes for a full changelog including other versions of pandas.

Enhancements

ADBC Driver support in to_sql and read_sql

read_sql() and to_sql() now work with Apache Arrow ADBC drivers. Compared to traditional drivers used via SQLAlchemy, ADBC drivers should provide significant performance improvements, better type support and cleaner nullability handling.

import adbc_driver_postgresql.dbapi as pg_dbapi

df = pd.DataFrame(
    [
        [1, 2, 3],
        [4, 5, 6],
    ],
    columns=['a', 'b', 'c']
)
uri = "postgresql://postgres:postgres@localhost/postgres"
with pg_dbapi.connect(uri) as conn:
    df.to_sql("pandas_table", conn, index=False)

# for roundtripping
with pg_dbapi.connect(uri) as conn:
    df2 = pd.read_sql("pandas_table", conn)

The Arrow type system offers a wider array of types that can more closely match what databases like PostgreSQL can offer. To illustrate, note this (non-exhaustive) listing of types available in different databases and pandas backends:

numpy/pandas	arrow	postgres	sqlite
int16/Int16	int16	SMALLINT	INTEGER
int32/Int32	int32	INTEGER	INTEGER
int64/Int64	int64	BIGINT	INTEGER
float32	float32	REAL	REAL
float64	float64	DOUBLE PRECISION	REAL
object	string	TEXT	TEXT
bool	bool_	BOOLEAN
datetime64[ns]	timestamp(us)	TIMESTAMP
datetime64[ns,tz]	timestamp(us,tz)	TIMESTAMPTZ
	date32	DATE
	month_day_nano_interval	INTERVAL
	binary	BINARY	BLOB
	decimal128	DECIMAL [1]
	list	ARRAY [1]
	struct	COMPOSITE TYPE [1]

Footnotes

[1] (1,2,3)

Not implemented as of writing, but theoretically possible

If you are interested in preserving database types as best as possible throughout the lifecycle of your DataFrame, users are encouraged to leverage the dtype_backend="pyarrow" argument of read_sql()

# for roundtripping
with pg_dbapi.connect(uri) as conn:
    df2 = pd.read_sql("pandas_table", conn, dtype_backend="pyarrow")

This will prevent your data from being converted to the traditional pandas/NumPy type system, which often converts SQL types in ways that make them impossible to round-trip.

For a full list of ADBC drivers and their development status, see the ADBC Driver Implementation Status documentation.

ExtensionArray.to_numpy converts to suitable NumPy dtype

ExtensionArray.to_numpy() will now convert to a suitable NumPy dtype instead of object dtype for nullable extension dtypes.

Old behavior:

In [1]: ser = pd.Series([1, 2, 3], dtype="Int64")
In [2]: ser.to_numpy()
Out[2]: array([1, 2, 3], dtype=object)

New behavior:

In [1]: ser = pd.Series([1, 2, 3], dtype="Int64")

In [2]: ser.to_numpy()
Out[2]: array([1, 2, 3])

The default NumPy dtype (without any arguments) is determined as follows:

• float dtypes are cast to NumPy floats

• integer dtypes without missing values are cast to NumPy integer dtypes

• integer dtypes with missing values are cast to NumPy float dtypes and NaN is used as missing value indicator

• boolean dtypes without missing values are cast to NumPy bool dtype

• boolean dtypes with missing values keep object dtype

Series.struct accessor for PyArrow structured data

The Series.struct accessor provides attributes and methods for processing data with struct[pyarrow] dtype Series. For example, Series.struct.explode() converts PyArrow structured data to a pandas DataFrame. (GH 54938)

In [3]: import pyarrow as pa

In [4]: series = pd.Series(
   ...:     [
   ...:         {"project": "pandas", "version": "2.2.0"},
   ...:         {"project": "numpy", "version": "1.25.2"},
   ...:         {"project": "pyarrow", "version": "13.0.0"},
   ...:     ],
   ...:     dtype=pd.ArrowDtype(
   ...:         pa.struct([
   ...:             ("project", pa.string()),
   ...:             ("version", pa.string()),
   ...:         ])
   ...:     ),
   ...: )
   ...: 

In [5]: series.struct.explode()
Out[5]: 
   project version
0   pandas   2.2.0
1    numpy  1.25.2
2  pyarrow  13.0.0

Series.list accessor for PyArrow list data

The Series.list accessor provides attributes and methods for processing data with list[pyarrow] dtype Series. For example, Series.list.__getitem__() allows indexing pyarrow lists in a Series. (GH 55323)

In [6]: import pyarrow as pa

In [7]: series = pd.Series(
   ...:     [
   ...:         [1, 2, 3],
   ...:         [4, 5],
   ...:         [6],
   ...:     ],
   ...:     dtype=pd.ArrowDtype(
   ...:         pa.list_(pa.int64())
   ...:     ),
   ...: )
   ...: 

In [8]: series.list[0]
Out[8]: 
0    1
1    4
2    6
dtype: int64[pyarrow]

Calamine engine for read_excel()

The calamine engine was added to read_excel(). It uses python-calamine, which provides Python bindings for the Rust library calamine. This engine supports Excel files (.xlsx, .xlsm, .xls, .xlsb) and OpenDocument spreadsheets (.ods) (GH 50395).

There are two advantages of this engine:

1. Calamine is often faster than other engines, some benchmarks show results up to 5x faster than ‘openpyxl’, 20x - ‘odf’, 4x - ‘pyxlsb’, and 1.5x - ‘xlrd’. But, ‘openpyxl’ and ‘pyxlsb’ are faster in reading a few rows from large files because of lazy iteration over rows.

2. Calamine supports the recognition of datetime in .xlsb files, unlike ‘pyxlsb’ which is the only other engine in pandas that can read .xlsb files.

pd.read_excel("path_to_file.xlsb", engine="calamine")

For more, see Calamine (Excel and ODS files) in the user guide on IO tools.

Other enhancements

• to_sql() with method parameter set to multi works with Oracle on the backend

• Series.attrs / DataFrame.attrs now uses a deepcopy for propagating attrs (GH 54134).

• read_csv() now supports on_bad_lines parameter with engine="pyarrow". (GH 54480)

• read_sas() returns datetime64 dtypes with resolutions better matching those stored natively in SAS, and avoids returning object-dtype in cases that cannot be stored with datetime64[ns] dtype (GH 56127)

• read_spss() now returns a DataFrame that stores the metadata in DataFrame.attrs. (GH 54264)

• tseries.api.guess_datetime_format() is now part of the public API (GH 54727)

• ExtensionArray._explode() interface method added to allow extension type implementations of the explode method (GH 54833)

• ExtensionArray.duplicated() added to allow extension type implementations of the duplicated method (GH 55255)

• Allow passing read_only, data_only and keep_links arguments to openpyxl using engine_kwargs of read_excel() (GH 55027)

• DataFrame.apply now allows the usage of numba (via engine="numba") to JIT compile the passed function, allowing for potential speedups (GH 54666)

• Implement masked algorithms for Series.value_counts() (GH 54984)

• Implemented Series.str.extract() for ArrowDtype (GH 56268)

• Improved error message that appears in DatetimeIndex.to_period() with frequencies which are not supported as period frequencies, such as “BMS” (GH 56243)

• Improved error message when constructing Period with invalid offsets such as “QS” (GH 55785)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

merge() and DataFrame.join() now consistently follow documented sort behavior

In previous versions of pandas, merge() and DataFrame.join() did not always return a result that followed the documented sort behavior. pandas now follows the documented sort behavior in merge and join operations (GH 54611).

As documented, sort=True sorts the join keys lexicographically in the resulting DataFrame. With sort=False, the order of the join keys depends on the join type (how keyword):

• how="left": preserve the order of the left keys

• how="right": preserve the order of the right keys

• how="inner": preserve the order of the left keys

• how="outer": sort keys lexicographically

One example with changing behavior is inner joins with non-unique left join keys and sort=False:

In [9]: left = pd.DataFrame({"a": [1, 2, 1]})

In [10]: right = pd.DataFrame({"a": [1, 2]})

In [11]: result = pd.merge(left, right, how="inner", on="a", sort=False)

Old Behavior

In [5]: result
Out[5]:
   a
0  1
1  1
2  2

New Behavior

In [12]: result
Out[12]: 
   a
0  1
1  2
2  1

merge() and DataFrame.join() no longer reorder levels when levels differ

In previous versions of pandas, merge() and DataFrame.join() would reorder index levels when joining on two indexes with different levels (GH 34133).

In [13]: left = pd.DataFrame({"left": 1}, index=pd.MultiIndex.from_tuples([("x", 1), ("x", 2)], names=["A", "B"]))

In [14]: right = pd.DataFrame({"right": 2}, index=pd.MultiIndex.from_tuples([(1, 1), (2, 2)], names=["B", "C"]))

In [15]: result = left.join(right)

Old Behavior

In [5]: result
Out[5]:
       left  right
B A C
1 x 1     1      2
2 x 2     1      2

New Behavior

In [16]: result
Out[16]: 
       left  right
A B C             
x 1 1     1      2
  2 2     1      2

Backwards incompatible API changes

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package	Minimum Version	Required	Changed
		X	X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
		X

See Dependencies and Optional dependencies for more.

Other API changes

• check_exact now only takes effect for floating-point dtypes in testing.assert_frame_equal() and testing.assert_series_equal(). In particular, integer dtypes are always checked exactly (GH 55882)

Deprecations

Deprecate aliases M, Q, Y, etc. in favour of ME, QE, YE, etc. for offsets

Deprecated the following frequency aliases (GH 9586):

offsets	deprecated aliases	new aliases
MonthEnd	M	ME
BusinessMonthEnd	BM	BME
SemiMonthEnd	SM	SME
CustomBusinessMonthEnd	CBM	CBME
QuarterEnd	Q	QE
BQuarterEnd	BQ	BQE
YearEnd	Y	YE
BYearEnd	BY	BYE

For example:

Previous behavior:

In [8]: pd.date_range('2020-01-01', periods=3, freq='Q-NOV')
Out[8]:
DatetimeIndex(['2020-02-29', '2020-05-31', '2020-08-31'],
              dtype='datetime64[ns]', freq='Q-NOV')

Future behavior:

In [17]: pd.date_range('2020-01-01', periods=3, freq='QE-NOV')
Out[17]: DatetimeIndex(['2020-02-29', '2020-05-31', '2020-08-31'], dtype='datetime64[ns]', freq='QE-NOV')

Deprecated automatic downcasting

Deprecated the automatic downcasting of object dtype results in a number of methods. These would silently change the dtype in a hard to predict manner since the behavior was value dependent. Additionally, pandas is moving away from silent dtype changes (GH 54710, GH 54261).

These methods are:

• Series.replace() and DataFrame.replace()

• DataFrame.fillna(), Series.fillna()

• DataFrame.ffill(), Series.ffill()

• DataFrame.bfill(), Series.bfill()

Explicitly call DataFrame.infer_objects() to replicate the current behavior in the future.

result = result.infer_objects(copy=False)

Set the following option to opt into the future behavior:

In [9]: pd.set_option("future.no_silent_downcasting", True)

Other Deprecations

• Changed Timedelta.resolution_string() to return h, min, s, ms, us, and ns instead of H, T, S, L, U, and N, for compatibility with respective deprecations in frequency aliases (GH 52536)

• Deprecated pandas.api.types.is_interval() and pandas.api.types.is_period(), use isinstance(obj, pd.Interval) and isinstance(obj, pd.Period) instead (GH 55264)

• Deprecated read_gbq() and DataFrame.to_gbq(). Use pandas_gbq.read_gbq and pandas_gbq.to_gbq instead https://pandas-gbq.readthedocs.io/en/latest/api.html (GH 55525)

• Deprecated DataFrameGroupBy.fillna() and SeriesGroupBy.fillna(); use DataFrameGroupBy.ffill(), DataFrameGroupBy.bfill() for forward and backward filling or DataFrame.fillna() to fill with a single value (or the Series equivalents) (GH 55718)

• Deprecated Index.format(), use index.astype(str) or index.map(formatter) instead (GH 55413)

• Deprecated Series.ravel(), the underlying array is already 1D, so ravel is not necessary (GH 52511)

• Deprecated Series.view(), use Series.astype() instead to change the dtype (GH 20251)

• Deprecated core.internals members Block, ExtensionBlock, and DatetimeTZBlock, use public APIs instead (GH 55139)

• Deprecated year, month, quarter, day, hour, minute, and second keywords in the PeriodIndex constructor, use PeriodIndex.from_fields() instead (GH 55960)

• Deprecated allowing non-integer periods argument in date_range(), timedelta_range(), period_range(), and interval_range() (GH 56036)

• Deprecated allowing non-keyword arguments in DataFrame.to_clipboard(). (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_csv() except path_or_buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_dict(). (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_excel() except excel_writer. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_gbq() except destination_table. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_hdf() except path_or_buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_html() except buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_json() except path_or_buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_latex() except buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_markdown() except buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_parquet() except path. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_pickle() except path. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_string() except buf. (GH 54229)

• Deprecated allowing non-keyword arguments in DataFrame.to_xml() except path_or_buffer. (GH 54229)

• Deprecated allowing passing BlockManager objects to DataFrame or SingleBlockManager objects to Series (GH 52419)

• Deprecated downcasting behavior in Series.where(), DataFrame.where(), Series.mask(), DataFrame.mask(), Series.clip(), DataFrame.clip(); in a future version these will not infer object-dtype columns to non-object dtype, or all-round floats to integer dtype. Call result.infer_objects(copy=False) on the result for object inference, or explicitly cast floats to ints. To opt in to the future version, use pd.set_option("future.no_silent_downcasting", True) (GH 53656)

• Deprecated including the groups in computations when using DataFrameGroupBy.apply() and DataFrameGroupBy.resample(); pass include_groups=False to exclude the groups (GH 7155)

• Deprecated indexing an Index with a boolean indexer of length zero (GH 55820)

• Deprecated not passing a tuple to DataFrameGroupBy.get_group or SeriesGroupBy.get_group when grouping by a length-1 list-like (GH 25971)

• Deprecated string AS denoting frequency in YearBegin and strings AS-DEC, AS-JAN, etc. denoting annual frequencies with various fiscal year starts (GH 54275)

• Deprecated string A denoting frequency in YearEnd and strings A-DEC, A-JAN, etc. denoting annual frequencies with various fiscal year ends (GH 54275)

• Deprecated string BAS denoting frequency in BYearBegin and strings BAS-DEC, BAS-JAN, etc. denoting annual frequencies with various fiscal year starts (GH 54275)

• Deprecated string BA denoting frequency in BYearEnd and strings BA-DEC, BA-JAN, etc. denoting annual frequencies with various fiscal year ends (GH 54275)

• Deprecated strings H, BH, and CBH denoting frequencies in Hour, BusinessHour, CustomBusinessHour (GH 52536)

• Deprecated strings H, S, U, and N denoting units in to_timedelta() (GH 52536)

• Deprecated strings H, T, S, L, U, and N denoting units in Timedelta (GH 52536)

• Deprecated strings T, S, L, U, and N denoting frequencies in Minute, Second, Milli, Micro, Nano (GH 52536)

• Deprecated the BaseGrouper attributes group_keys_seq and reconstructed_codes; these will be removed in a future version of pandas (GH 56148)

• Deprecated the Grouping attributes group_index, result_index, and group_arraylike; these will be removed in a future version of pandas (GH 56148)

• Deprecated the errors="ignore" option in to_datetime(), to_timedelta(), and to_numeric(); explicitly catch exceptions instead (GH 54467)

• Deprecated the fastpath keyword in the Series constructor (GH 20110)

• Deprecated the kind keyword in Series.resample() and DataFrame.resample(), explicitly cast the object’s index instead (GH 55895)

• Deprecated the ordinal keyword in PeriodIndex, use PeriodIndex.from_ordinals() instead (GH 55960)

• Deprecated the unit keyword in TimedeltaIndex construction, use to_timedelta() instead (GH 55499)

• Deprecated the behavior of Series.value_counts() and Index.value_counts() with object dtype; in a future version these will not perform dtype inference on the resulting Index, do result.index = result.index.infer_objects() to retain the old behavior (GH 56161)

• Deprecated the default of observed=False in DataFrame.pivot_table(); will be True in a future version (GH 56236)

• Deprecated the extension test classes BaseNoReduceTests, BaseBooleanReduceTests, and BaseNumericReduceTests, use BaseReduceTests instead (GH 54663)

• Deprecated the option mode.data_manager and the ArrayManager; only the BlockManager will be available in future versions (GH 55043)

• Deprecated the previous implementation of DataFrame.stack; specify future_stack=True to adopt the future version (GH 53515)

Performance improvements

• Performance improvement in testing.assert_frame_equal() and testing.assert_series_equal() (GH 55949, GH 55971)

• Performance improvement in concat() with axis=1 and objects with unaligned indexes (GH 55084)

• Performance improvement in get_dummies() (GH 56089)

• Performance improvement in merge_asof() when by is not None (GH 55580, GH 55678)

• Performance improvement in read_stata() for files with many variables (GH 55515)

• Performance improvement in DataFrame.groupby() when aggregating pyarrow timestamp and duration dtypes (GH 55031)

• Performance improvement in DataFrame.join() when joining on unordered categorical indexes (GH 56345)

• Performance improvement in DataFrame.loc() and Series.loc() when indexing with a MultiIndex (GH 56062)

• Performance improvement in DataFrame.sort_index() and Series.sort_index() when indexed by a MultiIndex (GH 54835)

• Performance improvement in DataFrame.to_dict() on converting DataFrame to dictionary (GH 50990)

• Performance improvement in Index.difference() (GH 55108)

• Performance improvement in Index.sort_values() when index is already sorted (GH 56128)

• Performance improvement in MultiIndex.get_indexer() when method is not None (GH 55839)

• Performance improvement in Series.duplicated() for pyarrow dtypes (GH 55255)

• Performance improvement in Series.str.get_dummies() when dtype is "string[pyarrow]" or "string[pyarrow_numpy]" (GH 56110)

• Performance improvement in Series.str() methods (GH 55736)

• Performance improvement in Series.value_counts() and Series.mode() for masked dtypes (GH 54984, GH 55340)

• Performance improvement in DataFrameGroupBy.nunique() and SeriesGroupBy.nunique() (GH 55972)

• Performance improvement in SeriesGroupBy.idxmax(), SeriesGroupBy.idxmin(), DataFrameGroupBy.idxmax(), DataFrameGroupBy.idxmin() (GH 54234)

• Performance improvement when indexing into a non-unique index (GH 55816)

• Performance improvement when indexing with more than 4 keys (GH 54550)

• Performance improvement when localizing time to UTC (GH 55241)

Bug fixes

Categorical

• Categorical.isin() raising InvalidIndexError for categorical containing overlapping Interval values (GH 34974)

• Bug in CategoricalDtype.__eq__() returning false for unordered categorical data with mixed types (GH 55468)

Datetimelike

• Bug in DatetimeIndex construction when passing both a tz and either dayfirst or yearfirst ignoring dayfirst/yearfirst (GH 55813)

• Bug in DatetimeIndex when passing an object-dtype ndarray of float objects and a tz incorrectly localizing the result (GH 55780)

• Bug in concat() raising AttributeError when concatenating all-NA DataFrame with DatetimeTZDtype dtype DataFrame. (GH 52093)

• Bug in testing.assert_extension_array_equal() that could use the wrong unit when comparing resolutions (GH 55730)

• Bug in to_datetime() and DatetimeIndex when passing a list of mixed-string-and-numeric types incorrectly raising (GH 55780)

• Bug in to_datetime() and DatetimeIndex when passing mixed-type objects with a mix of timezones or mix of timezone-awareness failing to raise ValueError (GH 55693)

• Bug in DatetimeIndex.shift() with non-nanosecond resolution incorrectly returning with nanosecond resolution (GH 56117)

• Bug in DatetimeIndex.union() returning object dtype for tz-aware indexes with the same timezone but different units (GH 55238)

• Bug in Index.is_monotonic_increasing() and Index.is_monotonic_decreasing() always caching Index.is_unique() as True when first value in index is NaT (GH 55755)

• Bug in Index.view() to a datetime64 dtype with non-supported resolution incorrectly raising (GH 55710)

• Bug in Series.dt.round() with non-nanosecond resolution and NaT entries incorrectly raising OverflowError (GH 56158)

• Bug in Tick.delta() with very large ticks raising OverflowError instead of OutOfBoundsTimedelta (GH 55503)

• Bug in Timestamp.unit() being inferred incorrectly from an ISO8601 format string with minute or hour resolution and a timezone offset (GH 56208)

• Bug in .astype converting from a higher-resolution datetime64 dtype to a lower-resolution datetime64 dtype (e.g. datetime64[us]->datetim64[ms]) silently overflowing with values near the lower implementation bound (GH 55979)

• Bug in adding or subtracting a Week offset to a datetime64 Series, Index, or DataFrame column with non-nanosecond resolution returning incorrect results (GH 55583)

• Bug in addition or subtraction of BusinessDay offset with offset attribute to non-nanosecond Index, Series, or DataFrame column giving incorrect results (GH 55608)

• Bug in addition or subtraction of DateOffset objects with microsecond components to datetime64 Index, Series, or DataFrame columns with non-nanosecond resolution (GH 55595)

• Bug in addition or subtraction of very large Tick objects with Timestamp or Timedelta objects raising OverflowError instead of OutOfBoundsTimedelta (GH 55503)

• Bug in creating a Index, Series, or DataFrame with a non-nanosecond DatetimeTZDtype and inputs that would be out of bounds with nanosecond resolution incorrectly raising OutOfBoundsDatetime (GH 54620)

• Bug in creating a Index, Series, or DataFrame with a non-nanosecond datetime64 (or DatetimeTZDtype) from mixed-numeric inputs treating those as nanoseconds instead of as multiples of the dtype’s unit (which would happen with non-mixed numeric inputs) (GH 56004)

• Bug in creating a Index, Series, or DataFrame with a non-nanosecond datetime64 dtype and inputs that would be out of bounds for a datetime64[ns] incorrectly raising OutOfBoundsDatetime (GH 55756)

• Bug in parsing datetime strings with nanosecond resolution with non-ISO8601 formats incorrectly truncating sub-microsecond components (GH 56051)

• Bug in parsing datetime strings with sub-second resolution and trailing zeros incorrectly inferring second or millisecond resolution (GH 55737)

• Bug in the results of to_datetime() with an floating-dtype argument with unit not matching the pointwise results of Timestamp (GH 56037)

Timedelta

• Bug in Timedelta construction raising OverflowError instead of OutOfBoundsTimedelta (GH 55503)

• Bug in rendering (__repr__) of TimedeltaIndex and Series with timedelta64 values with non-nanosecond resolution entries that are all multiples of 24 hours failing to use the compact representation used in the nanosecond cases (GH 55405)

Timezones

• Bug in AbstractHolidayCalendar where timezone data was not propagated when computing holiday observances (GH 54580)

• Bug in Timestamp construction with an ambiguous value and a pytz timezone failing to raise pytz.AmbiguousTimeError (GH 55657)

• Bug in Timestamp.tz_localize() with nonexistent="shift_forward around UTC+0 during DST (GH 51501)

Numeric

• Bug in read_csv() with engine="pyarrow" causing rounding errors for large integers (GH 52505)

• Bug in Series.pow() not filling missing values correctly (GH 55512)

Conversion

• Bug in DataFrame.astype() when called with str on unpickled array - the array might change in-place (GH 54654)

• Bug in DataFrame.astype() where errors="ignore" had no effect for extension types (GH 54654)

• Bug in Series.convert_dtypes() not converting all NA column to null[pyarrow] (GH 55346)

Strings

• Bug in pandas.api.types.is_string_dtype() while checking object array with no elements is of the string dtype (GH 54661)

• Bug in DataFrame.apply() failing when engine="numba" and columns or index have StringDtype (GH 56189)

• Bug in Series.__mul__() for ArrowDtype with pyarrow.string dtype and string[pyarrow] for the pyarrow backend (GH 51970)

• Bug in Series.str.startswith() and Series.str.endswith() with arguments of type tuple[str, ...] for string[pyarrow] (GH 54942)

Interval

• Bug in Interval __repr__ not displaying UTC offsets for Timestamp bounds. Additionally the hour, minute and second components will now be shown. (GH 55015)

• Bug in IntervalIndex.factorize() and Series.factorize() with IntervalDtype with datetime64 or timedelta64 intervals not preserving non-nanosecond units (GH 56099)

• Bug in IntervalIndex.from_arrays() when passed datetime64 or timedelta64 arrays with mismatched resolutions constructing an invalid IntervalArray object (GH 55714)

• Bug in IntervalIndex.get_indexer() with datetime or timedelta intervals incorrectly matching on integer targets (GH 47772)

• Bug in IntervalIndex.get_indexer() with timezone-aware datetime intervals incorrectly matching on a sequence of timezone-naive targets (GH 47772)

• Bug in setting values on a Series with an IntervalIndex using a slice incorrectly raising (GH 54722)

Indexing

• Bug in DataFrame.loc() when setting Series with extension dtype into NumPy dtype (GH 55604)

• Bug in Index.difference() not returning a unique set of values when other is empty or other is considered non-comparable (GH 55113)

• Bug in setting Categorical values into a DataFrame with numpy dtypes raising RecursionError (GH 52927)

Missing

• Bug in DataFrame.update() wasn’t updating in-place for tz-aware datetime64 dtypes (GH 56227)

MultiIndex

• Bug in MultiIndex.get_indexer() not raising ValueError when method provided and index is non-monotonic (GH 53452)

I/O

• Bug in read_csv() where engine="python" did not respect chunksize arg when skiprows was specified. (GH 56323)

• Bug in read_csv() where engine="python" was causing a TypeError when a callable skiprows and a chunk size was specified. (GH 55677)

• Bug in read_csv() where on_bad_lines="warn" would write to stderr instead of raise a Python warning. This now yields a errors.ParserWarning (GH 54296)

• Bug in read_csv() with engine="pyarrow" where quotechar was ignored (GH 52266)

• Bug in read_csv() with engine="pyarrow" where usecols wasn’t working with a csv with no headers (GH 54459)

• Bug in read_excel(), with engine="xlrd" (xls files) erroring when file contains NaNs/Infs (GH 54564)

• Bug in read_json() not handling dtype conversion properly if infer_string is set (GH 56195)

• Bug in DataFrame.to_excel(), with OdsWriter (ods files) writing boolean/string value (GH 54994)

• Bug in DataFrame.to_hdf() and read_hdf() with datetime64 dtypes with non-nanosecond resolution failing to round-trip correctly (GH 55622)

• Bug in read_excel() with engine="odf" (ods files) when string contains annotation (GH 55200)

• Bug in read_excel() with an ODS file without cached formatted cell for float values (GH 55219)

• Bug where DataFrame.to_json() would raise an OverflowError instead of a TypeError with unsupported NumPy types (GH 55403)

Period

• Bug in PeriodIndex construction when more than one of data, ordinal and **fields are passed failing to raise ValueError (GH 55961)

• Bug in Period addition silently wrapping around instead of raising OverflowError (GH 55503)

• Bug in casting from PeriodDtype with astype to datetime64 or DatetimeTZDtype with non-nanosecond unit incorrectly returning with nanosecond unit (GH 55958)

Plotting

• Bug in DataFrame.plot.box() with vert=False and a matplotlib Axes created with sharey=True (GH 54941)

• Bug in DataFrame.plot.scatter() discaring string columns (GH 56142)

• Bug in Series.plot() when reusing an ax object failing to raise when a how keyword is passed (GH 55953)

Groupby/resample/rolling

• Bug in Rolling where duplicate datetimelike indexes are treated as consecutive rather than equal with closed='left' and closed='neither' (GH 20712)

• Bug in DataFrameGroupBy.idxmin(), DataFrameGroupBy.idxmax(), SeriesGroupBy.idxmin(), and SeriesGroupBy.idxmax() would not retain Categorical dtype when the index was a CategoricalIndex that contained NA values (GH 54234)

• Bug in DataFrameGroupBy.transform() and SeriesGroupBy.transform() when observed=False and f="idxmin" or f="idxmax" would incorrectly raise on unobserved categories (GH 54234)

• Bug in DataFrameGroupBy.value_counts() and SeriesGroupBy.value_count() could result in incorrect sorting if the columns of the DataFrame or name of the Series are integers (GH 55951)

• Bug in DataFrameGroupBy.value_counts() and SeriesGroupBy.value_count() would not respect sort=False in DataFrame.groupby() and Series.groupby() (GH 55951)

• Bug in DataFrameGroupBy.value_counts() and SeriesGroupBy.value_count() would sort by proportions rather than frequencies when sort=True and normalize=True (GH 55951)

• Bug in DataFrame.asfreq() and Series.asfreq() with a DatetimeIndex with non-nanosecond resolution incorrectly converting to nanosecond resolution (GH 55958)

• Bug in DataFrame.ewm() when passed times with non-nanosecond datetime64 or DatetimeTZDtype dtype (GH 56262)

• Bug in DataFrame.resample() not respecting closed and label arguments for BusinessDay (GH 55282)

• Bug in DataFrame.resample() where bin edges were not correct for BusinessDay (GH 55281)

• Bug in DataFrame.resample() where bin edges were not correct for MonthBegin (GH 55271)

Reshaping

• Bug in concat() ignoring sort parameter when passed DatetimeIndex indexes (GH 54769)

• Bug in merge_asof() raising TypeError when by dtype is not object, int64, or uint64 (GH 22794)

• Bug in merge() returning columns in incorrect order when left and/or right is empty (GH 51929)

• Bug in DataFrame.melt() where an exception was raised if var_name was not a string (GH 55948)

• Bug in DataFrame.melt() where it would not preserve the datetime (GH 55254)

• Bug in DataFrame.pivot_table() where the row margin is incorrect when the columns have numeric names (GH 26568)

Sparse

• Bug in SparseArray.take() when using a different fill value than the array’s fill value (GH 55181)

ExtensionArray

Styler

Other

• Bug in DataFrame.describe() when formatting percentiles in the resulting percentile 99.999% is rounded to 100% (GH 55765)

• Bug in cut() and qcut() with datetime64 dtype values with non-nanosecond units incorrectly returning nanosecond-unit bins (GH 56101)

• Bug in cut() incorrectly allowing cutting of timezone-aware datetimes with timezone-naive bins (GH 54964)

• Bug in infer_freq() and DatetimeIndex.inferred_freq() with weekly frequencies and non-nanosecond resolutions (GH 55609)

• Bug in DataFrame.apply() where passing raw=True ignored args passed to the applied function (GH 55009)

• Bug in DataFrame.from_dict() which would always sort the rows of the created DataFrame. (GH 55683)

• Bug in rendering inf values inside a a DataFrame with the use_inf_as_na option enabled (GH 55483)

• Bug in rendering a Series with a MultiIndex when one of the index level’s names is 0 not having that name displayed (GH 55415)

• Bug in the error message when assigning an empty dataframe to a column (GH 55956)

Contributors