What’s new in 3.1.0 (Month XX, 2026)#

These are the changes in pandas 3.1.0. See Release notes for a full changelog including other versions of pandas.

Enhancements#

enhancement1#

enhancement2#

Other enhancements#

  • Period now supports f-string formatting via __format__, e.g. f"{period:%Y-%m}" (GH 48536)

  • Series and DataFrame with timedelta64 dtype now aligns fractional seconds in string representation for easier reading (GH 57188)

  • DataFrameGroupBy.agg() now allows for the provided func to return a NumPy array (GH 63957)

  • DataFrameGroupBy.transform() now accepts list-like and dict arguments similar to GroupBy.agg(), and supports NamedFunc (GH 58318)

  • Series.to_json() now supports serializing custom ExtensionArrays (by correctly using the _values_for_json method of an ExtensionArray) (GH 65047)

  • Timestamp.round(), Timestamp.floor(), and Timestamp.ceil() now officially accept Timedelta arguments (GH 63687)

  • Added NamedFunc, an alias to NamedAgg for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (GH 65164)

  • ExtensionArray.map() now calls ExtensionArray._cast_pointwise_result() to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through map (GH 57189, GH 62164)

  • read_csv() now supports dtype="complex64" and dtype="complex128" with the C engine, enabling round-tripping of complex-number columns written by DataFrame.to_csv() (GH 9379)

  • to_datetime() and strptime parsing now support the %N directive for matching exactly 9 digits representing nanoseconds, providing symmetry with strftime formatting (GH 65863)

  • DataFrame.select_dtypes() can now select datetime64 and timedelta64 columns by a specific resolution (e.g. "datetime64[us]", np.dtype("timedelta64[ms]")), which previously raised; a unit-specific spec matches only columns with exactly that resolution, while a unitless "datetime64"/"timedelta64" still matches every resolution (GH 40234)

  • Series.round() now works on third-party numeric ExtensionArray types via a default ExtensionArray.round() (GH 49387)

  • Timestamp.strftime() and the array-level DatetimeIndex.strftime() / Series.dt.strftime() now support a %N directive that formats nanoseconds as a 9-digit zero-padded number; the standard %f directive is unchanged and continues to format microseconds only (GH 29461)

  • Added ExtensionArray.count() (GH 64450)

  • Added ExtensionArray.sort() for in-place sorting of ExtensionArray (GH 64977)

  • Added Index.replace() method to support value replacement functionality similar to Series.replace() (GH 19495)

  • Added union_categories parameter to concat() to preserve categorical dtype by unioning categories when concatenating categoricals with different categories (GH 14177)

  • Added reduction methods as public API on pandas-implemented extension arrays where applicable (GH 63512)

  • Building from source no longer fails on a checkout with CRLF line endings, such as under WSL (GH 64272)

  • Display formatting for float sequences in DataFrame cells now respects the display.precision option (GH 60503).

  • Improved the precision of float parsing in read_csv() (GH 64395)

  • Improved the string repr of pd.core.arrays.SparseArray (GH 64547)

  • Improved type inference of comparison and arithmetic operators on Series and DataFrame for static type checkers (e.g. ser == "a" is now inferred as Series instead of Any) (GH 40762)

  • MSVC is no longer required to build on Windows, and build errors when using the MinGW compiler have been fixed (GH 63160)

  • Building from source no longer fails when compiling the windowing extensions against C++ standard libraries that provide only std::signbit (GH 50979, GH 51047)

  • Setting values with DataFrame.at() or Series.at() using a non-scalar indexer (e.g. a boolean mask, list, or array) now raises a clearer InvalidIndexError directing users to .loc (GH 51866)

Notable bug fixes#

These are bug fixes that might have notable behavior changes.

Timedelta.total_seconds now includes the nanosecond component#

Previously Timedelta.total_seconds() mirrored the stdlib datetime.timedelta.total_seconds() formula and silently dropped the sub-microsecond nanosecond residual — e.g. Timedelta(nanoseconds=999).total_seconds() returned 0.0 instead of 9.99e-7. The result now includes the nanosecond component. The vectorized TimedeltaIndex.total_seconds() and Series.dt.total_seconds() already included the nanosecond component; they now match the scalar result exactly, including at large magnitudes where floating-point rounding previously differed (GH 46819).

Timestamp no longer ignores date components and the positional tzinfo#

Timestamp takes either a value to convert (a datetime-like object, a string, or an epoch number) or the individual date components, but passing both silently dropped the components:

In [1]: pd.Timestamp(datetime(2020, 12, 31), hour=5)
Out[1]: Timestamp('2020-12-31 00:00:00')

This now raises ValueError; write pd.Timestamp(datetime(2020, 12, 31, 5)) instead. Null-like input (None, NaT, NaN, pd.NA, numpy.datetime64("NaT")) is exempt and gives NaT for any combination of the components; Timestamp(None, year=2020, month=1, day=1) and friends therefore give NaT where they used to raise TypeError.

In the positional datetime.datetime-like form the parameter names are shifted by one – the parameter named hour is datetime.datetime’s minute – so a keyword argument used to land one field early:

In [2]: pd.Timestamp(2020, 12, 31, hour=5)
Out[2]: Timestamp('2020-12-31 00:05:00')

This now raises ValueError as well, as does leaving the same gap with an explicit None (Timestamp(2020, 1, 2, None, 5)); write pd.Timestamp(year=2020, month=12, day=31, hour=5) instead. Two cases cannot be detected. An int first argument is indistinguishable from a year, so Timestamp(2020, year=1, month=1) is still read as Timestamp(2020, 1, 1); and a keyword that exactly fills the next free positional slot is indistinguishable from passing it positionally, so Timestamp(2020, 12, 31, day=5) is still read as Timestamp(2020, 12, 31, 5).

Finally, in that same positional form the eighth argument is datetime.datetime’s tzinfo and used to be dropped; it is now attached to the result just as datetime.datetime does. Attaching differs from the localizing tz and tzinfo keywords: with a pytz timezone, Timestamp(2000, 1, 2, 3, 4, 5, 6, pytz.timezone("US/Eastern")) now matches datetime(2000, 1, 2, 3, 4, 5, 6, pytz.timezone("US/Eastern")) and therefore carries that zone’s LMT offset. copy.replace() reconstructs a Timestamp through this form, so it too now keeps the timezone instead of returning a naive result; it still drops nanoseconds, and still raises ValueError for a fold=1 (DST-ambiguous) timestamp (GH 31930, GH 45307).

Backwards incompatible API changes#

Increased minimum versions for dependencies#

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package

Minimum Version

Required

Changed

numpy

2.0.2

X

X

python-dateutil

2.9.0

X

X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package

Minimum Version

Changed

adbc-driver-postgresql

1.7.0

X

adbc-driver-sqlite

1.7.0

X

beautifulsoup4

4.13.4

X

bottleneck

1.5.0

X

fsspec

2025.7.0

X

gcsfs

2025.7.0

X

Jinja2

3.1.6

X

lxml

6.0.0

X

matplotlib

3.10.5

X

numba

0.61.2

X

numexpr

2.11.0

X

PyQt5

5.15.11

X

pyiceberg

0.9.1

X

pyreadstat

1.3.0

X

pytables

3.10.2

X

python-calamine

0.4.0

X

qtpy

2.4.3

X

s3fs

2025.7.0

X

SciPy

1.16.1

X

sqlalchemy

2.0.42

X

xarray

2025.7.1

X

xlrd

2.0.2

X

xlsxwriter

3.2.5

X

See Dependencies and Optional dependencies for more.

Other API changes#

  • Index.values and Index.array now return read-only arrays for all dtypes, so an Index can no longer be silently corrupted by mutating the returned array in place (previously this left the Index displaying the new value while lookups still used the old one) (GH 38547)

  • infer_freq() on quarter-start data now reports the equivalent anchor from the first calendar quarter, e.g. QS-JAN instead of QS-OCT (GH 36939)

  • testing.assert_frame_equal() with check_freq=True now also checks the freq of DatetimeIndex and TimedeltaIndex columns; previously only the freq of the index was checked (GH 51920)

  • testing.assert_series_equal() with check_index=False no longer checks the freq attribute of a DatetimeIndex or TimedeltaIndex, as freq is an attribute of the index (GH 51920)

  • to_numeric(), and the readers built on it such as read_csv() and read_xml(), now report an unparsable value in the "Unable to parse string" message using its repr(), e.g. Unable to parse string 'apple' at position 2 rather than Unable to parse string "apple" at position 2. Previously the value was interpolated as a C string, so it was truncated at an embedded NUL byte, e.g. "abc\x00def" was reported as abc (GH 66524)

  • DataFrameGroupBy.sum() and DataFrameGroupBy.mean() on float dtypes with sorted grouping keys may differ from prior versions in the last few floating-point digits, due to a faster summation algorithm that does not use Kahan compensation (GH 65103)

  • GroupBy.sum() and GroupBy.prod() on ArrowDtype decimal columns now return the maximum decimal precision (decimal128(38, scale) or decimal256(76, scale)) instead of a precision inferred from the group results; a result needing more digits than the maximum precision still falls back to a wider inferred type (GH 63416)

  • DataFrame.select_dtypes() now raises TypeError with an informative message when "period" is passed to include or exclude, instead of a bare NotImplementedError. Pass PeriodDtype to select all period columns, or a string such as "period[D]" to select a single frequency (GH 24558)

  • DataFrame.select_dtypes() now selects only columns with exactly the given dtype when a dtype instance is passed to include or exclude. Previously an instance selected all columns with a matching dtype.type, e.g. pd.DatetimeTZDtype(tz="UTC") also selected other timezones, np.dtype("int64") also selected Int64 and Arrow int64 columns, and pd.CategoricalDtype(["a", "b"]) selected every categorical column (GH 40234)

  • DataFrame.to_hdf() no longer writes a pandas_version attribute into the HDF5 file; the value was hardcoded to "0.15.2" and never reflected the pandas version. As a consequence, format="table" files written with pandas 3.1 or later cannot be read by pandas 3.0 or earlier (GH 62792)

  • APIs that accept an engine="numba" parameter with engine_kwargs will no longer pass through a nopython argument to numba.jit. This argument has had no effect since numba 0.59.0 (GH 64483).

  • Removed the freq and freqstr attributes from DatetimeArray and TimedeltaArray. Frequency is now stored only on DatetimeIndex and TimedeltaIndex; access Series.dt.freq or wrap the array in an Index to retrieve a frequency. The check_freq keyword on testing.assert_extension_array_equal() for these array types has also been removed (GH 24566).

Build#

  • C++20 is required to build from source (GH 66735)

Deprecations#

Performance improvements#

Bug fixes#

Categorical#

  • Bug in Categorical.__repr__() where the values and categories lines could exceed display.width (GH 12066)

  • Bug in Categorical.map() and Series.map() raising NotImplementedError when the mapper returned tuples for the categories, instead of returning an Index of tuples (GH 51488)

  • Bug in Categorical.map() where mapping with a defaultdict and na_action=None would bypass the default factory by using dict.get, causing NA values to be replaced with NaN instead of the mapper’s default value (GH 62710)

  • Bug in CategoricalIndex.union() and CategoricalIndex.intersection() giving incorrect results when the two indexes have the same unordered categories in different orders (GH 55335)

  • Bug in Index.fillna() raising TypeError when filling with a tuple value (e.g. on object-dtype or CategoricalIndex with tuple categories) (GH 37681)

Datetimelike#

  • Bug in Timestamp.days_in_month, and in month-end and business-day offsets, using a wrapped year to decide whether the year is a leap year, for second-resolution datetimes with a year beyond 2**31 (GH 66549)

  • Bug in ArrowExtensionArray where adding a DateOffset to a date32[pyarrow] or date64[pyarrow] Series raised an ArrowTypeError (GH 57168)

  • Bug in DatetimeIndex constructor raising ValueError when passing equivalent but not equal frequencies (e.g. QS-FEB vs QS-MAY) (GH 61086)

  • Bug in DatetimeIndex raising AttributeError when comparing against Arrow date types (date32, date64) (GH 62051)

  • Bug in Timestamp constructor where a timezone-aware datetime near the implementation bounds raised OverflowError instead of OutOfBoundsDatetime when shifted to UTC (GH 66510)

  • Bug in Timestamp constructor where passing np.str_ objects would fail in Cython string parsing (GH 48974)

  • Bug in Timestamp constructor where strings with a negative year of fewer than 4 digits (e.g. "-111-01-01") silently dropped the leading "-" and were parsed as a positive year; BC dates with 1-4 digit years now parse correctly, matching numpy.datetime64 (GH 55954)

  • Bug in Timestamp constructor, Timedelta constructor, to_datetime(), and to_timedelta() on scalar or object-dtype input, including plain lists of NumPy scalars, with a NumPy float narrower than 64 bits (e.g. np.float32) and a unit: results gained spurious sub-second digits, np.float16 raised, a spurious RuntimeWarning was emitted, and np.float16("-inf") silently became NaT (GH 56996)

  • Bug in Timestamp constructor, Timedelta constructor, to_datetime(), and to_timedelta() with non-round float input and unit failing to raise when the value is just outside the representable bounds (GH 57366)

  • Bug in Timestamp constructor, Timestamp.replace(), to_datetime(), and read_csv() with parse_dates where a datetime landing on the NaT sentinel, either directly or after the shift to UTC, silently became NaT instead of raising OutOfBoundsDatetime (GH 66510)

  • Bug in api.types.infer_dtype() returning "date" or "mixed" instead of "datetime" / "timedelta" for lists of Timestamp/Timedelta values mixed with pd.NA (GH 53023)

  • Bug in date_range() where inclusive="left" and inclusive="right" returned a single-element result instead of empty when start equals end (GH 55293)

  • Bug in date_range() where inclusive parameter failed to filter endpoints when only start and periods or end and periods were specified (GH 46331)

  • Bug in date_range() where periods=1 with offsets that disallow n=0 (e.g. offsets.LastWeekOfMonth, offsets.FY5253) raised ValueError (GH 41563)

  • Bug in date_range() where offsets that preserve start’s time-of-day (e.g. MS, ME, QS, YS, B, C) could exclude the last offset boundary when end’s time-of-day was earlier than start’s (GH 35342)

  • Bug in date_range() where passing end off the offset together with periods and an anchored offset (e.g. W-SUN, ME, MS, QS) silently returned fewer than periods dates (GH 64834)

  • Bug in to_datetime() and Timestamp where a datetime near the implementation bounds with a fixed UTC offset silently wrapped when shifted to UTC instead of raising OutOfBoundsDatetime (GH 65353)

  • Bug in to_datetime() and to_timedelta() on ARM platforms where round float values outside the int64 domain (e.g. float(2**63)) could silently produce incorrect results instead of raising (GH 64619)

  • Bug in to_datetime() and to_timedelta() on object-dtype input, including plain lists of NumPy scalars, with a NumPy integer and a unit: values of a dtype narrower than 64 bits raised OutOfBoundsDatetime/OutOfBoundsTimedelta (or became NaT under errors="coerce"), and with unit="Y" or unit="M" any NumPy integer, np.int64 included, raised ValueError (GH 56996)

  • Bug in to_datetime() and to_timedelta() where uint64 values greater than int64 max silently overflowed instead of raising OutOfBoundsDatetime or OutOfBoundsTimedelta (GH 60677)

  • Bug in to_datetime() raising a bare AssertionError instead of ValueError when passed an invalid errors value (GH 66542)

  • Bug in to_datetime() when passed a DataFrame of date/time field columns raising for years outside the range 1000-9999 (GH 65195)

  • Bug in to_datetime() when passed a DataFrame of date/time field columns with errors="coerce" returning datetime64[s] dtype instead of datetime64[us] when all rows were invalid (GH 65195)

  • Bug in to_datetime() when using a low time resolution unit, higher resolution in origin is now preserved instead of silently dropped (e.g. unit="D" with microsecond precision origin) (GH 63419)

  • Bug in to_datetime() with string input silently ignoring dayfirst and yearfirst when unit was also passed (GH 63472)

  • Bug in DataFrame.replace() and Series.replace() raising AssertionError instead of OutOfBoundsDatetime when replacing with a datetime value outside the datetime64[ns] range (GH 61671)

  • Bug in DataFrame.to_string() and Series.to_string() where na_rep was ignored for datetime and timedelta columns, always displaying NaT (GH 55426)

  • Bug in DatetimeArray.isin() and TimedeltaArray.isin() where mismatched resolutions could silently truncate finer-resolution values, leading to false matches (GH 64545)

  • Bug in DatetimeIndex.intersection() returning incorrect results when the two indexes had matching freq but did not lie on the same grid, e.g. ranges with the same business-day frequency but different time-of-day components (GH 44025)

  • Bug in DatetimeIndex.tz_localize(), Series.dt.tz_localize(), and the Timestamp constructor with a timezone that observes DST, where a wall time whose shift to UTC lands exactly on the NaT sentinel read out of bounds and reported a spurious “nonexistent time” ValueError instead of raising OutOfBoundsDatetime (GH 66550)

  • Bug in DatetimeIndex.union() and TimedeltaIndex.union() with sort=False silently dropping values from the other index that fell after the end of self (GH 66322)

  • Bug in Series.dt.isocalendar() with a pyarrow-backed datetime or date dtype not preserving the original index, resetting it to a default RangeIndex (GH 65894)

  • Bug in Series.dt.to_pydatetime() not preserving the original index and name (GH 66443)

  • Bug in day_of_week, Timestamp.weekday() and Timestamp.day_name() returning a wrong weekday, or raising OverflowError, for second-resolution datetimes with a year beyond 2**31 (GH 66549)

  • Bug in adding a BusinessDay offset to a DatetimeIndex or Series where an out-of-bounds result silently wrapped to an unrelated date, and a result landing exactly on the NaT sentinel came back as a missing value, instead of raising OverflowError (GH 66552)

  • Bug in adding a BusinessDay offset with n of magnitude 2**31 or more to a DatetimeIndex or Series silently shifting by a wrapped number of business days, e.g. n=2**32 behaving like n=0 (GH 66549)

  • Bug in adding a DateOffset to a DatetimeIndex or Series where a result landing exactly on the NaT sentinel came back as a missing value, and where promoting the values to the offset’s finer resolution silently wrapped to an unrelated date, instead of raising as the equivalent Timedelta operations already did (GH 66552)

  • Bug in adding a DateOffset with a milliseconds component to a Timestamp returning a microsecond-resolution result, inconsistent with the millisecond-resolution result of the equivalent DatetimeIndex and Series operations (GH 64806)

  • Bug in adding a DateOffset with both a month- or year-based component and a timedelta component (e.g. DateOffset(months=1, days=-31)) to a DatetimeIndex or Series raising OutOfBoundsDatetime for a representable result, where the equivalent Timestamp operation succeeds (GH 66549)

  • Bug in adding a Day offset to a timezone-aware DatetimeIndex or Series where an out-of-bounds result silently wrapped to an unrelated date, and a result landing exactly on the NaT sentinel came back as a missing value, instead of raising OverflowError as the timezone-naive equivalent already did (GH 66552)

  • Bug in adding a Week offset to a DatetimeIndex or Series where an out-of-bounds result silently wrapped to an unrelated date, and a result landing exactly on the NaT sentinel came back as a missing value, instead of raising OverflowError (GH 66552)

  • Bug in adding a Week offset with n of magnitude 2**31 or more to a DatetimeIndex or Series silently shifting by a wrapped number of weeks, e.g. n=2**32 behaving like n=0 (GH 66549)

  • Bug in adding a month-, quarter-, half-year-, year- or semi-month-based offset (e.g. MonthEnd, QuarterEnd, YearEnd, SemiMonthEnd) with n of magnitude 2**31 or more silently shifting by a wrapped number of periods, e.g. n=2**32 behaving like n=0 (GH 66549)

  • Bug in adding a month-, quarter-, year- or semi-month-based offset (e.g. MonthEnd, QuarterEnd, YearEnd, SemiMonthEnd) to a DatetimeIndex or Series where an out-of-bounds result raised a bare OverflowError naming a pandas C source file, instead of the OutOfBoundsDatetime naming the offending date that the equivalent Timestamp operation raises (GH 66549)

  • Bug in adding non-nano DatetimeIndex with non-vectorized offsets (e.g. CustomBusinessDay, CustomBusinessMonthEnd) having a sub-unit offset parameter incorrectly truncating the result or raising AttributeError (GH 56586)

  • Bug in adding or subtracting a Timedelta and a Timestamp where a result landing exactly on the NaT sentinel, e.g. Timestamp.min + Timedelta(-1, "ns"), returned NaT instead of raising OutOfBoundsDatetime as it already did one step further out (GH 66549)

  • Bug in adding or subtracting a Timedelta and a datetime64 numpy.ndarray where the promotion to the finer of the two resolutions, or the addition itself, silently wrapped to an unrelated date, and a result landing exactly on the NaT sentinel came back as a missing value, instead of raising OutOfBoundsDatetime or OutOfBoundsTimedelta as the equivalent Timestamp operation already did (GH 66552)

  • Bug in adding or subtracting a timedelta64 NumPy array and a timezone-naive Timestamp where the promotion to the finer of the two resolutions, or the addition itself, silently wrapped to an unrelated date instead of raising OutOfBoundsDatetime, unlike the timezone-aware and scalar equivalents (GH 66552)

  • Bug in arithmetic on DatetimeIndex, TimedeltaIndex, PeriodIndex and the corresponding Series where a result landing exactly on the NaT sentinel came back as a missing value instead of raising OverflowError as it already did one step further out (GH 66549)

  • Bug in subtracting BusinessHour (or CustomBusinessHour) from a Timestamp giving incorrect results when the subtraction would land exactly on the business-hour opening time (GH 33682)

  • Bug in subtracting two Timestamp objects whose difference lands exactly on the NaT sentinel raising AssertionError, or with python -O returning a corrupt Timedelta for which isna() was False, instead of raising OutOfBoundsDatetime (GH 66552)

Timedelta#

  • Bug in TimedeltaIndex.resolution raising when the index has no frequency (GH 65186)

  • Bug in DateOffset where DateOffset(1) and DateOffset(days=1) returned different results near daylight saving time transitions (GH 61862)

  • Bug in Timedelta constructor and to_timedelta() raising a bare OverflowError instead of OutOfBoundsTimedelta for infinite or overflowing float input, which also prevented errors="coerce" from returning NaT (GH 63275)

  • Bug in Timedelta constructor and to_timedelta() where passing np.str_ objects would fail in Cython string parsing (GH 48974)

  • Bug in Timedelta constructor where keyword arguments (e.g. days=365000) that exceeded nanosecond int64 bounds raised OutOfBoundsTimedelta instead of falling back to a coarser resolution (GH 46587)

  • Bug in to_timedelta(), TimedeltaIndex, and Series/DataFrame construction or astype on object-dtype input, where a numpy.timedelta64 NaT whose unit differed from the array’s inferred resolution raised OutOfBoundsTimedelta instead of being preserved as NaT, e.g. to_timedelta([np.timedelta64("NaT", "D")]) (GH 63018)

  • Bug in to_timedelta(), TimedeltaIndex, and Series/DataFrame construction or astype on object-dtype input, where a timedelta string such as "1 days" silently became a value 1000 times too small if an earlier element carried nanosecond resolution, e.g. to_timedelta([Timedelta(1, "ns"), "1 days"]) (GH 63196)

  • Bug in Series.cumsum() and DataFrame.cumsum() on timedelta64 data where a running total that left the representable range silently wrapped to an unrelated value instead of raising OutOfBoundsTimedelta (GH 66551)

  • Bug in Series.sum() and DataFrame.sum() on timedelta64 data returning a rounded result for totals above 253 nanoseconds, and returning NaT instead of the exact total for a few representable values such as Timedelta.min (GH 66551)

  • Bug in Series.sum() and DataFrame.sum() on an overflowing timedelta64 object where Series.sum() raised a plain ValueError and DataFrame.sum() silently returned a saturated result instead of raising OutOfBoundsTimedelta (GH 43178)

  • Bug in Series.dt.seconds and Series.dt.microseconds with ArrowDtype durations returning the Series.dt.components field values (e.g. 0-59 for seconds, or 0 for microseconds on a "ms" unit) instead of the totals within each day and second respectively, inconsistent with NumPy-backed timedeltas (GH 63283, GH 63470)

  • Bug in adding or subtracting a Timedelta and a timedelta64 numpy.ndarray where an out-of-bounds result silently wrapped, and a result landing exactly on the NaT sentinel came back as a missing value, instead of raising OutOfBoundsTimedelta as the equivalent Timedelta and TimedeltaIndex operations already did (GH 66552)

  • Bug in adding or subtracting two Timedelta objects where a result landing exactly on the NaT sentinel returned NaT, e.g. Timedelta.min - Timedelta(1, "ns"), instead of raising OutOfBoundsTimedelta as it already did one step further out (GH 66552)

  • Bug in dividing a Timedelta by a float numpy.ndarray where a quotient outside the int64 range silently saturated instead of raising OutOfBoundsTimedelta as the equivalent TimedeltaIndex operation already did (GH 66552)

  • Bug in dividing a Timedelta by an integer, where the quotient was computed in float64 and so was rounded once it exceeded 2**53; dividing Timedelta.min by 1 returned NaT and dividing Timedelta.max by 1 raised OverflowError instead of returning the original value (GH 66551)

  • Bug in multiplying a Series of timedelta64 by an integer or float, or dividing it by a float, where an overflowing result silently wrapped or saturated instead of raising OutOfBoundsTimedelta; multiplying by inf now raises OutOfBoundsTimedelta instead of returning NaT (GH 43178)

  • Bug in multiplying a Timedelta by a NumPy integer or float scalar, where an overflowing product silently wrapped instead of raising and a np.float32 multiplier lost precision, unlike the equivalent Python int or float (GH 66551)

  • Bug in multiplying a Timedelta by an integer or float numpy.ndarray where an out-of-bounds product silently wrapped or saturated, and one landing exactly on the NaT sentinel came back as a missing value, instead of raising OutOfBoundsTimedelta as the equivalent TimedeltaIndex operation already did (GH 66552)

  • Bug in multiplying or dividing a Timedelta by a numeric scalar where a result landing exactly on the NaT sentinel raised a message-less AssertionError, or under python -O returned a Timedelta that was not NaT but read back as NaT once stored; these now raise OutOfBoundsTimedelta (GH 66551)

  • Bug in multiplying or dividing a Timedelta, TimedeltaIndex or timedelta64 Series by a whole-numbered float such as 1.0, where the operation was applied in float64 and so was rounded once the result exceeded 2**53; Timedelta.min * 1.0 returned NaT and Timedelta.max / 1.0 raised OverflowError instead of returning the original value (GH 66551)

  • Bug in the Series and DataFrame constructors not honoring the unit of a timedelta64[unit] dtype when constructing from floats or a mix of integers and floats, e.g. pd.Series([1.5, 90.0], dtype="timedelta64[s]") interpreted the numbers as nanoseconds and silently truncated them to zero instead of interpreting them as seconds (GH 63499)

Timezones#

Numeric#

Conversion#

  • Bug in DataFrame constructor raising TypeError when given a list whose first element is a list-like dataclass (e.g. a collections.UserList subclass); such elements are now treated as list-like rows (GH 41682)

  • Bug in DataFrame constructor where NaT in a TimedeltaIndex row was incorrectly inferred as datetime64 instead of timedelta64 (GH 23985)

  • Bug in DataFrame constructor where constructing from a list of uniform-dtype arrays (e.g. pyarrow, CategoricalDtype, nullable dtypes) lost the dtype (GH 49593)

  • Bug in pd.array() silently converting NaN to a nonsensical integer when given float data containing NaN and a NumPy integer dtype (GH 41724)

  • Bug in DataFrame.convert_dtypes() and Series.convert_dtypes() raising OverflowError for object-dtype data holding integers outside the int64/uint64 range; these now retain object dtype, and data holding both NA and integers above the int64 range now converts to UInt64 (GH 66517)

  • Bug in Series.astype() to an ArrowDtype duration with a non-nanosecond unit silently returning wrong values (often all zeros) when converting a string column that had been sliced, including the default str dtype (GH 64320)

  • Bug in dtype inference from object-dtype data (e.g. Series construction, Series.map(), DataFrame.infer_objects()) raising OverflowError instead of inferring object dtype when given an integer outside the float64 range (GH 66519)

  • Bug in dtype inference from object-dtype data (e.g. Series construction, Series.map(), DataFrame.infer_objects()) returning float64 instead of object when a None came before an integer outside the int64/uint64 range, or before a mix of signed and unsigned integers; these now infer the same dtype as the equivalent data without the None (GH 66519)

  • Fixed pandas.array() to preserve mask information when converting NumPy masked arrays, converting masked values to missing values (GH 63879)

  • Fixed bug in DataFrame constructor where mutating the result could corrupt the source Series or Index when built with dtype="str" and infer_string=False (GH 63936)

  • Fixed bug in to_numeric() where a string containing an embedded NUL byte was converted using only the bytes before the NUL, so e.g. "1.5\x00xyz" silently became 1.5; such values now raise, or become NaN with errors="coerce" (GH 66524)

  • Fixed bug in DataFrame.from_records() where exclude was ignored when data was an iterator and nrows=0 (GH 63774)

  • Fixed bug in DataFrame.replace() and Series.replace() raising TypeError when to_replace was Ellipsis (...) (GH 50373)

  • Fixed bug in DataFrame.to_dict() with orient="index" did not respect the into argument in nested mappings (GH 65778)

Strings#

Interval#

  • Bug in IntervalArray and IntervalIndex constructors allowing unsupported object dtype on the right endpoint, causing dtype.subtype to disagree with right.dtype (GH 66518)

  • Bug in IntervalArray and IntervalIndex constructors unnecessarily upcasting sub-64-bit numeric dtypes (e.g. float32, int32) to 64-bit (GH 45412)

  • Bug in cut() and other operations building an IntervalIndex engine raising TypeError on 32-bit platforms when there were more than 100 intervals (GH 44075, GH 23440)

Indexing#

  • Bug in TimedeltaIndex string indexing using the resolution of the parsed value rather than the string, so keys like "720s" (an exact multiple of a minute) matched a minute-sized window instead of a second-sized one (GH 33603)

  • Bug in DataFrame.loc() and Series.loc() replacing the index name with the key’s name when indexing with an Index (GH 17110)

  • Bug in DataFrame.loc() raising ValueError when setting a row on a DataFrame with no columns and the label is not in the index (GH 17895)

  • Bug in DataFrame.loc() returning incorrect dtype when the column key is a slice (GH 63071)

  • Bug in Index.get_indexer() where method="pad", "backfill", or "nearest" returned incorrect results when the target contained NaT or NaN instead of -1 (GH 32572)

  • Bug in Series.loc() and DataFrame.loc() setitem-with-expansion silently corrupting a value just outside the integer dtype’s range (e.g. 2**63 for int64) on some platforms instead of keeping the float64 result; inf was similarly affected, and both leaked a RuntimeWarning (GH 66394)

  • Bug in Series.loc() and DataFrame.loc() setitem-with-expansion widening a datetime64 or timedelta64 column to microsecond resolution instead of keeping a coarser unit such as s or ms, even when the new value fit in it losslessly (GH 66402)

  • Bugs in setitem-with-expansion when adding new rows failing to keep the original dtype in some cases (GH 32346, GH 15231, GH 47503, GH 6485, GH 25383, GH 52235, GH 17026, GH 56010)

  • Bug in DataFrame.__getitem__() raising InvalidIndexError when indexing with a tuple containing a slice on a DataFrame with MultiIndex columns (e.g., df[:, "t1"]) (GH 26511)

  • Bug in DataFrame.__setitem__() silently keeping only the first column when assigning a 2D array to an existing column label, instead of raising as the new-column and DataFrame.loc() cases already did (GH 46544)

  • Bug in DataFrame.__setitem__() with a boolean DataFrame mask raising a cryptic “cannot assign mismatch length to masked array”; it now raises a ValueError describing the mismatch between the number of values and the number of True entries in the mask (GH 45593)

  • Bug in DataFrame.at() raising TypeError when accessing a MultiIndex with a partial date string on a DatetimeIndex level (GH 43395)

  • Bug in DataFrame.duplicated() returning an empty Series without the DataFrame’s index when the DataFrame had no columns (GH 61191)

  • Bug in DataFrame.iloc() and Series.iloc() setitem raising ValueError (“cannot set using a slice indexer with a different length than the value”) when assigning to a negative-step slice with an open-ended start or stop (e.g. ser.iloc[::-2]) (GH 66100)

  • Bug in DataFrame.iloc() setitem raising AttributeError when assigning a Series or Index with a nullable EA dtype (e.g. Int64, Float64, boolean) into a column with a NumPy dtype (GH 47776)

  • Bug in DataFrame.loc() raising ValueError when assigning a list of tuples to an object-dtype column with a boolean mask on a mixed-dtype DataFrame (GH 37629)

  • Bug in DataFrame.loc() raising ValueError when setting a row with a list-like value on a single-column DataFrame with ExtensionArray dtype (GH 44103)

  • Bug in DataFrame.loc() setitem-with-expansion writing the truncated string "n" (or raising TypeError) into rows outside the indexer when adding a new column from a list of strings or booleans (GH 42099)

  • Bug in DataFrame.loc() with a MultiIndex returning wrong results instead of raising KeyError when passing string keys for numeric index levels (GH 60104)

  • Bug in DataFrame.mask() with inplace=True where incorrect values were produced when other was a Series with ExtensionArray values (GH 64635)

  • Bug in DataFrame.rename() and Series.rename() not preserving nullable extension dtype (e.g. Int64, Float64) when relabeling index or column labels (GH 65315)

  • Bug in DataFrame.where() and DataFrame.mask() raising TypeError when cond is a Series and axis=1 (GH 58190)

  • Bug in DataFrame.xs() where drop_level=False was ignored for fully specified MultiIndex keys when level was not explicitly provided (GH 6507)

  • Bug in Index.get_indexer_non_unique() raising ZeroDivisionError instead of returning an all-missing result when called on an empty index with a non-empty target (GH 54746)

  • Bug in Index.get_level_values() mishandling boolean, NA-like (np.nan, pd.NA, pd.NaT) and integer index names (GH 62169)

  • Bug in Index.get_loc() raising KeyError when looking up a tuple in an object-dtype Index with duplicates (GH 37800)

  • Bug in Index.insert() silently casting booleans to numeric when used with nullable numeric dtypes like Float64 or Int64 (GH 61709)

  • Bug in Index.take() where fill_value was silently ignored and integer-dtype indexes raised ValueError instead of filling with the provided value. Passing fill_value=None now fills -1 entries with the Index’s NA value (matching the ExtensionArray convention); omit fill_value to retain the previous behavior where negative indices wrap (GH 65210)

  • Bug in Index.where() and Index.putmask() preserving numpy.datetime64 / numpy.timedelta64 NaT scalars in the object-dtype result for mismatched-dtype inputs, instead of normalizing to pandas.NaT as Series.where() does (GH 55174)

  • Bug in MultiIndex.get_loc() returning a slice instead of an integer for a unique key when the MultiIndex contained duplicates elsewhere, causing .loc to return a Series instead of a scalar (GH 42102)

  • Bug in RangeIndex.get_indexer() returning spurious matches for extreme integer targets (near INT64_MIN or uint64 values above INT64_MAX) and failing to match genuine members of ranges spanning more than INT64_MAX (GH 64148)

  • Bug in RangeIndex.memory_usage() and RangeIndex.nbytes raising TypeError on PyPy (GH 46176)

  • Bug in Series.where() and Series.mask() raising ValueError when other is a tuple on object-dtype Series (GH 37681)

  • Bug in setitem (e.g. Series.iloc()) silently storing corrupted values in the untouched entries of an ArrowDtype binary or large_binary column when assigning NA to a column that had been sliced; later operations on the column, such as DataFrame.to_parquet(), could then raise ArrowInvalid (GH 64320)

  • Bug in setitem (e.g. Series.iloc()) silently storing wrong values when assigning a nullable or ArrowDtype extension array into a NumPy-dtype column that could not hold it losslessly, such as a negative value into an unsigned-integer column (-1 became 18446744073709551615) or a value overflowing a narrower float dtype (GH 47776)

  • Bug in setitem on nullable masked arrays (Int64, Float64, boolean, …) silently coercing list-like values that the equivalent scalar setitem would reject — e.g. arr[[0]] = ['1'] parsed the string into the numeric dtype, arr[[0]] = [True] on Int64/Float64 coerced the boolean to 1, and an out-of-bounds value such as arr[[0]] = [1000] on Int8 silently wrapped instead of raising (GH 45404)

  • Fixed bug in DataFrame.loc() where assigning an iterable to a single cell in an object dtype column incorrectly raised a ValueError (GH 26333, GH 57962)

  • Fixed bug in DataFrame.loc() where assigning a Series to a subset of rows of one column upcast the dtype of other columns in a single-dtype DataFrame (GH 66105)

  • Fixed bug in DataFrame.loc() where assigning with duplicate column names and new columns corrupted unrelated columns (GH 58317)

  • Fixed segfault in DataFrame.loc() when repeatedly adding new rows to an object-dtype-indexed DataFrame (GH 21968)

Missing#

MultiIndex#

  • Bug in MultiIndex where pickling a DataFrame with a datetime64[ns] level raised NotImplementedError (GH 63078)

  • Bug in DataFrame.loc() with a MultiIndex where using a tuple indexer with a scalar and a list (e.g., (scalar, list)) did not drop the scalar-indexed level (GH 18631)

  • Bug in MultiIndex.get_loc() where looking up a tuple key containing a scalar inside an IntervalIndex level with overlapping intervals raised KeyError or returned incorrect results (GH 27456)

  • Bug in MultiIndex.set_levels() and MultiIndex.set_codes() raising IndexError instead of a clear ValueError when passing an empty sequence with level=None or a list-like level (GH 16147)

  • Bug in MultiIndex.sortlevel() not raising TypeError when sorting a level with incomparable types (e.g., Timestamp and str) (GH 21136)

  • Bug in MultiIndex.union() raising InvalidIndexError when combining levels containing datetime.date and Timestamp values representing the same date (GH 61807)

  • Bug in the DataFrame constructor where passing a DataFrame with a unique integer MultiIndex together with a flat integer index raised ValueError: Buffer dtype mismatch instead of reindexing to an all-NaN result (GH 26460)

  • MultiIndex.isin() now raises an informative ValueError when the values are not tuples matching the number of levels, instead of raising a cryptic TypeError or AssertionError or silently returning incorrect results (GH 20252, GH 26622)

I/O#

  • read_csv() with engine="pyarrow" now raises EmptyDataError (matching the c and python engines) instead of a cryptic ParserError reporting Empty CSV file or block when no columns can be parsed from the input (GH 66065)

  • read_csv() with engine="pyarrow" now raises ValueError for the unsupported na_filter=False instead of silently ignoring it (GH 66053)

  • read_csv() with engine="pyarrow" now raises an informative TypeError when passed a text-mode file handle (the engine requires a binary buffer) instead of a cryptic a bytes-like object is required error (GH 66065)

  • read_csv() with engine="pyarrow" now raises an informative ValueError for a list-valued header (MultiIndex columns are not supported by this engine) instead of a cryptic TypeError (GH 66059)

  • read_csv() with memory_map=True and an in-memory buffer (e.g. BytesIO) now raises a clear ValueError instead of a cryptic UnsupportedOperation: fileno (GH 45630)

  • read_json() with engine="pyarrow" now raises ValueError when passed an option the engine cannot apply – such as typ="series", convert_dates, keep_default_dates, date_unit, precise_float, convert_axes, encoding, encoding_errors, storage_options, an explicit compression, or a non-ArrowDtype dtype – instead of silently ignoring it and returning an incorrect result (GH 66144)

  • read_sas() now raises an informative ValueError for a SAS7BDAT file whose column metadata does not fit the rows it describes – a column extending past the end of a row, or a numeric column wider than 8 bytes – instead of reading or writing past the end of the row buffer and returning unrelated memory as data or crashing (GH 47339)

  • read_sas() now raises an informative ValueError for a SAS7BDAT file whose header declares a row_length larger than the page size, instead of segfaulting (GH 66475)

  • read_sas() now recognizes more of the encodings a SAS7BDAT file can record, including the “no encoding” and “US-ASCII” markers written by most SAS sessions (GH 66470)

  • read_sql_table() now raises an informative NotImplementedError (instead of one with no message) when passed a DBAPI connection such as sqlite3, and reading from a URI string without a usable sqlalchemy install now raises a clearer ImportError (GH 41237)

  • read_sql() now raises an informative DatabaseError explaining that reading a table by name requires a SQLAlchemy connection when a bare table name is passed with a DBAPI connection (e.g. sqlite3), instead of a cryptic SQL syntax error (GH 54233)

  • DataFrame.to_json() and Series.to_json() now raise an informative ValueError instead of a cryptic OverflowError: Maximum recursion level reached when a column contains an unsupported object type such as pathlib.Path (GH 36211)

  • DataFrame.to_sql() now raises a clearer ValueError when a non-string dtype is passed for a raw DB-API (e.g. sqlite3) connection (GH 61385)

  • Fixed bug in read_csv() where a field containing an embedded NUL byte was parsed as a float from the bytes before the NUL with both the c and the python engine, so e.g. "1.5\x00xyz", "1e3\x00xyz" and "inf\x00xyz" were read as 1.5, 1000.0 and inf with the trailing bytes silently discarded; such columns are now read as strings (GH 66524)

  • Fixed bug in read_csv() with engine="pyarrow" where names was silently ignored when header was also an integer; names now replaces the header row, extra leading columns form the index, and passing too many names raises ValueError as with the other engines. usecols together with names and an integer header is not supported by this engine and now raises an informative ValueError (GH 65862)

  • Fixed bug in read_csv() with engine="pyarrow" where a defaultdict passed as dtype did not apply its default to columns not explicitly listed (GH 41574)

  • Fixed bug in read_csv() with engine="pyarrow" where an empty usecols (e.g. usecols=[]) was ignored and returned all columns instead of an empty frame, unlike the other engines (GH 66056)

  • Fixed bug in read_csv() with engine="pyarrow" where passing tuples in names produced flat columns instead of MultiIndex columns as with the other engines (GH 65862)

  • Fixed bug in read_csv() with the c engine and dtype="category" where encoding_errors was ignored, so an undecodable byte raised UnicodeDecodeError even with encoding_errors="replace" (GH 66525)

  • Fixed bug in read_csv() with the c engine where a column name containing an embedded NUL byte was truncated at the NUL, which could also collide two distinct names into one (GH 19886)

  • Fixed bug in read_csv() with the c engine where a field containing a NUL byte was compared against na_values only up to that NUL, so e.g. "NA\x00x" or any field beginning with a NUL was read as NaN instead of its actual value, unlike engine="python"; a column mixing such fields with numbers is consequently inferred as string rather than float (GH 19886)

  • Fixed bug in read_csv() with the c engine where a field containing an embedded NUL byte was parsed as an integer or boolean from the bytes before the NUL, so e.g. "1\x00xyz" was read as 1 and "True\x00xyz" as True with the trailing bytes silently discarded; such columns are now read as strings, matching engine="python", and raise when an incompatible dtype is requested (GH 66524)

  • Fixed bug in read_csv() with the c engine where a quoted field containing an embedded NUL byte was silently truncated at the NUL under the default string dtype or dtype_backend="pyarrow" (GH 66415)

  • Fixed bug in read_csv() with the c engine where a value in na_values, true_values or false_values containing an embedded NUL byte was truncated at the NUL, so it also matched unrelated fields sharing the prefix before it (GH 19886)

  • Fixed bug in read_csv() with the c engine where a value passed to a converters callable was truncated at an embedded NUL byte (GH 19886)

  • Fixed bug in read_csv() with the c engine where an embedded \r followed by a space in an unquoted field could cause an infinite re-parsing loop, producing spurious rows or a buffer overflow (GH 51141)

  • Fixed bug in read_csv() with the c engine where reading from a chunked or iterator reader after it had been closed – explicitly, by leaving its with block, or automatically after a chunk raised – crashed the interpreter instead of raising ValueError (GH 66622)

  • Fixed bug in read_csv() with the c engine where running out of memory while the parser was reporting an error crashed the interpreter instead of raising ParserError (GH 66660)

  • Fixed bug in read_csv() with the c engine where two fields differing only after an embedded NUL byte were read as the same value with dtype="S" (GH 19886)

  • Fixed bug in read_csv() with the c engine where two fields differing only after an embedded NUL byte were read as the same value with an explicit string-like dtype (object, "str", "string" or "category") (GH 66525)

  • Fixed bug in read_excel() where usage of skiprows could lead to an infinite loop (GH 64027)

  • Fixed bug in read_excel() with the openpyxl engine where reading a sheet set its max_row and max_column to None on the workbook exposed through ExcelFile.book (GH 63010)

  • Fixed bug in read_sas() where encoding="infer" raised LookupError: unknown encoding: infer instead of falling back to latin-1, for SAS7BDAT files recording an encoding pandas does not recognize and for XPORT files, which record no encoding at all (GH 66470)

  • Fixed bug where read_html() parsed nested tables incorrectly when using html5lib or bs4 flavors (GH 64524)

  • Fixed regression in read_csv() where sep=None raised TypeError: object of type 'NoneType' has no len() instead of sniffing the separator with the python engine; passing sep=None together with engine="c" or engine="pyarrow", which cannot sniff the separator, again raises an informative ValueError instead of raising TypeError or silently parsing each line as a single column (GH 66639)

  • Fixed bugs in read_csv() with engine="pyarrow" where column names were handled inconsistently with the other engines: duplicated names were not de-duplicated to "x.1"-style names, empty header fields did not get "Unnamed: {i}" placeholder names, and an unnamed index_col produced an index named "" instead of an unnamed index (GH 13017, GH 35211)

  • Fixed bug in read_csv() with engine="pyarrow" where parse_dates did not parse numeric-looking date columns or index columns, leaving them as integers or strings instead of datetimes (GH 34066)

  • Fixed bug in read_csv() with engine="pyarrow" where invalid sep, quotechar, escapechar, and decimal arguments raised cryptic errors, or in the case of quotechar=None were silently ignored, instead of raising the same informative errors as the other engines (GH 66317)

  • Fixed bug in read_csv() with engine="pyarrow" where passing a scalar (non-dict) dtype together with index_col raised AttributeError; the scalar dtype is now also applied to the index column, matching the default c engine (GH 45801)

  • Fixed bug in read_pickle() where the timezone of a Timestamp was silently dropped when reading a pickle written by pandas 1.2 or earlier, so the timestamp came back tz-naive at its UTC wall time (GH 61792, GH 31930)

  • Fixed memory leak in read_csv() (GH 19941)

  • Fixed memory leak in DataFrame.to_json() and Series.to_json() when serializing Timestamp with timezones (GH 54865)

  • Fixed segfault in DataFrame.to_json() and Series.to_json() when serializing an object-dtype datetime.date or datetime.timedelta subclass that carries a _value attribute but no _creso attribute (GH 65904)

  • Fixed segfault when instantiating the internal pandas._libs.parsers.TextReader with no arguments; it now raises TypeError (GH 53131)

  • Fixed several segfaults in DataFrame.to_json() and Series.to_json() when serializing python objects that are very abnormal, including massive int values, and strings that cannot be encoded as utf-8 (GH 66356)

  • Fixed segfaults in DataFrame.to_json() and Series.to_json() when serializing dictionary keys or labels that cannot be encoded as UTF-8 or that are integers too large to stringify under sys.get_int_max_str_digits(), and when serializing objects with a raising __dir__ or set subclasses with a raising __iter__; these now raise instead (GH 66356, GH 66489)

  • Fixed DataFrame.to_json() and Series.to_json() silently returning invalid or incorrect JSON when the error raised while serializing one element of a list, tuple, set, or dict was discarded while serializing a later element (GH 66356)

  • Fixed memory leaks in DataFrame.to_json() and Series.to_json() when serialization failed, particularly for large outputs and for index or column labels that could not be encoded (GH 66356)

  • Fixed read_json() with lines=True and chunksize to respect nrows when the requested row count is not a multiple of the chunk size (GH 64025)

  • HDFStore.put() and HDFStore.append() now support storing Series and DataFrame columns with PeriodDtype in both "fixed" and "table" formats (GH 41978)

  • Bug in DataFrame.__repr__() raising TypeError for a column with a NumPy structured dtype (e.g. produced by DataFrame.from_records() from a structured ndarray) (GH 55011)

  • Bug in DataFrame.__repr__() where horizontally truncated output could exceed the terminal width by up to 4 characters (GH 32461)

  • Bug in DataFrame.to_json() and Series.to_json() with orient="table" silently dropping the timezone of columns with a fixed-offset timezone (e.g. datetime.timezone(timedelta(hours=1))), so the offset was lost on round-trip through read_json() (GH 39537)

  • Bug in DataFrame.to_json() and Series.to_json() writing unsigned NumPy integer scalars stored in object dtype as negative numbers when they exceeded the signed int64 maximum (GH 66142)

  • Bug in DataFrame.to_stata() raising KeyError when column names require renaming and convert_dates is specified for a different column (GH 60536)

  • Bug in DataFrame.to_string() where formatters dict was applied to wrong columns when output was horizontally truncated via max_cols (GH 35410)

  • Fixed read_json() with lines=True and nrows=0 to return an empty DataFrame (GH 64025)

  • DataFrame.to_hdf() now raises a clear NotImplementedError when writing a column or Index of an unsupported extension dtype (such as IntervalDtype, SparseDtype, or the nullable integer/float/boolean dtypes), instead of a low-level AttributeError or PyTables TypeError (GH 26144, GH 38305, GH 42070)

  • read_hdf() can again read fixed-format files written by very old pandas versions (<=0.15.x) that stored a freq attribute on non-datetimelike indexes, which previously failed with a TypeError or ValueError (GH 33186)

  • DataFrame.to_hdf() and HDFStore now emit a UserWarning when complib is passed without complevel; because complevel defaults to 0 the data is written uncompressed, which previously happened silently (GH 29310)

  • DataFrame.to_hdf() with format="fixed" now compresses object dtype (e.g. string) columns when complib/complevel are given; previously the compression settings were silently ignored for these columns, producing much larger files (GH 45286)

  • HDFStore.put(), HDFStore.append(), and DataFrame.to_hdf() now emit a UserWarning instead of silently doing nothing when writing an empty DataFrame or Series with format='table' or append=True (GH 13016)

  • HDFStore.select() and read_hdf() now warn when a nested where of the form "(A & B) | (C & D)" over indexed columns may return incorrect results because of an upstream PyTables bug, suggesting writing with index=False or running the OR branches as separate queries (GH 50598)

  • HDFStore.select() now raises a clear ValueError with a workaround, instead of an opaque too many inputs error, when a where expression has too many comparisons for a query against indexed columns (GH 39752)

  • HDFStore.select() now raises an informative NotImplementedError when a where clause contains an arithmetic expression such as "(A % 3) == 0", instead of an opaque PyTables TypeError; arithmetic in where filters is not supported (GH 41100)

  • HDFStore.select(), HDFStore.select_as_coordinates(), and HDFStore.select_as_multiple() now raise an informative NotImplementedError instead of a cryptic KeyError when a column selection such as where="columns=['A']" is used with any coordinate-based read (iterator=True, chunksize, or the select_as_* methods); pass the columns argument instead (GH 12953)

  • Bug in HDFStore.select() where a where query on a categorical data column for a value that is not one of the categories incorrectly matched rows with missing (NaN) values (GH 22977)

  • Fixed MemoryError in HDFStore.select() when iterating large tables with chunksize and no where filter (GH 15937)

  • Fixed bug in read_hdf() raising on files written by older pandas versions whose freq index attribute could not be decoded; the freq is now dropped with a warning instead of corrupting the index (GH 35917)

  • Fixed bug in read_hdf() where a categorical column containing a category equal to the nan_rep string (e.g. the default "nan") raised ValueError: operands could not be broadcast together instead of reading that category back as NaN (GH 21741)

  • Fixed bug in read_hdf() where a string Index or MultiIndex level did not round-trip missing values and the literal string "nan": a missing value was read back as "nan" and a literal "nan" was read back as NaN (GH 9604)

  • Fixed bug in read_parquet() crashing the interpreter when called with to_pandas_kwargs={"self_destruct": True} (GH 66509)

  • Fixed bug in DataFrame.to_hdf() and HDFStore.put() where writing an object to a key silently deleted any nested keys stored beneath it (GH 17267)

  • Fixed bug in DataFrame.to_hdf() raising TypeError when the index had a non-tick DateOffset freq (e.g. DateOffset(years=1)) (GH 45790)

  • Fixed bug in DataFrame.to_hdf() with format="table" where a TimedeltaIndex was reconstructed as a PeriodIndex (when freq was set) or an integer Index (otherwise) on read-back (GH 21466)

  • Fixed bug in HDFStore.select() where a where combining a row condition with an ==/!= filter on a data column of more than 31 values (e.g. "index>0 & col=big_list") silently returned extra rows that did not match the filter when read with iterator=True or chunksize (GH 12953)

  • Fixed bug in HDFStore.select() where an ==/!= filter on a timezone-aware datetime data column of more than 31 values silently returned the wrong rows (typically none) on a plain read, because the timezone was dropped before the filter was applied (GH 12953)

  • Fixed DataFrame.to_hdf() and Series.to_hdf() to round-trip a CategoricalIndex in both "fixed" and "table" formats; previously raised AssertionError (GH 33909, GH 16118)

  • Bug in DataFrame.to_json() and Series.to_json() with date_format="epoch" where datetime or timedelta values held behind another dtype, such as CategoricalDtype or SparseDtype, were written in their own resolution instead of date_unit, and where SparseDtype raised AttributeError (GH 66709)

  • Bug in Series.to_json() with date_format="iso" where a timezone-aware datetime Series was serialized without the trailing Z marker, losing the timezone information that is retained for an equivalent DatetimeIndex or DataFrame column (GH 65744)

  • Fixed DataFrame.from_arrow() to be consistent with other methods (such as read_parquet()) in the conversion from PyArrow to pandas, e.g. consistently using the default string dtype regardless of the PyArrow version (GH 65696)

  • Fixed bug in read_hdf() where a store it had opened was left open if reading failed with an error other than ValueError, TypeError, or LookupError (e.g. AssertionError), so a subsequent attempt to reopen the file for writing raised “file is already opened” (GH 28430)

  • Fixed bug in DataFrame.to_parquet() (pyarrow engine) where a local file path was opened twice, once by pandas and again by pyarrow, wasting a syscall and silently truncating output to 0 bytes on filesystems that finalize a file’s contents on close (GH 65810)

  • Fixed bug in HDFStore.get_storer() where .shape reported a phantom row for a fixed-format Series or DataFrame stored with no rows (GH 37235)

  • Fixed bug in HDFStore.remove() where a where clause selecting on more than 31 values (e.g. "index in [...]") deleted the wrong rows instead of only the matching rows (GH 17567)

  • Fixed bug in HDFStore.select() where passing where as a list of conditions referencing caller-scope variables failed on Python 3.12+ due to PEP 709 inlining list comprehension stack frames (GH 64881)

  • Fixed bug in HDFStore.select() with format="table" where reading a frame with a string Index could crash with a bus error on strict-alignment platforms such as 32-bit ARM (GH 54396)

  • Storing a DataFrame or Series with a MultiIndex level named 'index' via HDFStore.put() or HDFStore.append() with format='table' now raises a clear ValueError instead of an opaque reshape error (GH 6208)

  • The PerformanceWarning emitted by DataFrame.to_hdf() for object columns now names only the columns that cannot be mapped to a c-type, instead of every object column sharing the same block (GH 28460)

  • Writing a DataFrame with format='table' and a column named 'index' as a data_columns entry (including data_columns=True) now raises a clear ValueError instead of an opaque reshape error (GH 41437)

Period#

  • Bug in Period constructor where passing np.str_ objects would fail in Cython string parsing (GH 48974)

  • Bug in DatetimeIndex.to_period() where anchored offsets YS, BYS, QS, BQS, BYE, and BQE produced incorrect period frequencies, losing the month anchor (GH 36939)

  • Bug in Period.strftime() where unknown format directives (e.g. "%Q") silently produced platform-dependent output and crashed the Python process on Windows; an Invalid format string ValueError is now raised on all platforms (GH 53562)

  • Bug in Period.to_timestamp() and PeriodIndex.to_timestamp() returning incorrect timestamps when the target frequency normalized to nanoseconds (e.g. "1ns") or when converting a nanosecond Period to a coarser target frequency (GH 63760)

  • Bug in Period.to_timestamp() and PeriodIndex.to_timestamp() with how="end" losing nanosecond precision when the target frequency normalized to nanoseconds (e.g. "1ns"); the target frequency is now also validated when how="end" (GH 63760)

  • Bug in PeriodIndex.from_fields() incorrectly rejecting quarterly freq values not anchored on December (e.g. QuarterEnd(startingMonth=2)), even though the equivalent scalar Period works, has been fixed (GH 55784)

  • Bug in adding an integer, timedelta or offset to a Period returning NaT when the result was one step past the lower bound, instead of raising OverflowError as it already did further out (GH 66552)

Plotting#

Groupby/resample/rolling#

Reshaping#

  • concat() with keys now raises an informative ValueError instead of an AssertionError when the concatenated objects do not all have the same number of index levels (GH 25413)

  • DataFrame.pivot() and pivot() now raise an informative KeyError naming the offending labels when index, columns, or values are not columns of the frame, instead of a cryptic TypeError (GH 35785)

  • Bug in concat() raising InvalidIndexError when keys or the concatenated objects’ index was an overlapping IntervalIndex (GH 64825)

  • Bug in concat() with a null[pyarrow] column incorrectly changing the dtype of the other columns, e.g. casting date32[pyarrow] to timestamp[ms][pyarrow], dropping the timezone of a tz-aware timestamp[pyarrow], or casting decimal128[pyarrow] to object (GH 62343)

  • Bug in merge() where merging on a MultiIndex containing NaN values mapped NaN keys to the last level value instead of NaN (GH 64492)

  • Bug in merge() where the join key column was not upcast to the highest datetime64 resolution, keeping the lower resolution for how="inner" and how="left" when the left frame had lower resolution, and for how="inner" and how="right" when the join key came from a lower-resolution right frame (GH 55212)

  • Bug in DataFrame.combine() raising OverflowError when the combining function returned a value too large for the columns’ common integer dtype (e.g. 2**64) instead of keeping the result (GH 66394)

  • Bug in DataFrame.melt() where var_name colliding with an id_vars column or value_name silently overwrote the affected column data instead of raising (GH 65654)

  • Bug in DataFrame.pivot_table() with margins=True raising TypeError when values has an ExtensionDtype that cannot hold NA (e.g. IntervalDtype with an integer subtype) and no columns were specified (GH 55484)

  • Bug in DataFrame.select_dtypes() where a nullable extension dtype name such as "Int64" or "boolean" also selected numpy columns sharing the same scalar type; it now selects only columns of that extension dtype (GH 40234)

  • Bug in DataFrame.select_dtypes() where an interval spec giving a subtype but no closed keyword, such as "interval[int64]" or pd.IntervalDtype("int64"), selected no columns; it now selects every interval column with that subtype, for any closed value (GH 40234)

  • Bug in DataFrame.select_dtypes() where the string form of an Arrow dtype such as "int64[pyarrow]" selected no columns and "float64[pyarrow]" selected numpy float columns; an Arrow dtype string now selects only columns of that exact dtype (GH 59888)

  • Bug in DataFrame.stack() raising a bare AssertionError or an IndexError when level contained duplicate entries, including duplicates produced by resolving level names or negative level numbers; it now raises an informative ValueError (GH 66588)

  • Bug in DataFrame.unstack() and Series.unstack() with sort=False placing values under the wrong row labels, or collapsing distinct index combinations into a single row (GH 62816)

  • Bug in Index.union() where the result could be unsorted when both inputs were monotonic increasing but disjoint, when sort was not False (GH 54646)

  • Fixed bug in Series.sort_values() where ignore_index=True had no effect on an already-sorted Series (GH 65833)

  • In pivot_table(), when values is empty, the aggregation will be computed on a Series of all NA values (GH 46475)

Sparse#

  • Bug in Series.mean() with skipna=False ignoring missing values for SparseDtype-backed Series (GH 65478)

  • Bug in Series.reindex() and alignment raising when extending a boolean SparseDtype-backed Series; missing entries now upcast to object to match the dense behavior (GH 32119)

  • Bug in Series.sum(), Series.min(), and Series.max() with skipna=False ignoring missing values for SparseDtype-backed Series (GH 65478)

  • Bug in SparseArray.astype() where converting a datetime64 SparseArray with NaT fill value to "Sparse[int64]" silently replaced the fill value with 0 instead of iNaT (GH 49631)

  • Bug in SparseArray.mean() raising a TypeError when called with the skipna argument (GH 65478)

  • Bug in SparseArray.sum() with skipna=False returning NA for a non-null fill_value and ignoring missing values under a null fill_value (GH 65478)

  • Bug in indexing a SparseArray with an out-of-bounds integer with the value of the length of the array returning the fill value instead of raising an IndexError (GH 64183).

  • Bug in logical operators (&, |, ^) between a SparseDtype-backed Series and a differently-indexed Series raising an uninformative error instead of aligning and returning the expected result (GH 32119)

ExtensionArray#

Styler#

  • Fixed bug in Styler.to_excel() where quoted strings in CSS properties were incorrectly lowercased, causing the exported Excel styling to use the wrong values; now quoted strings are left as-is (GH 63101)

Other#

  • Bug in DataFrame constructor where passing the same Index object as both index and columns shared a single object between the two axes, so mutating metadata such as names on one would also change the other (GH 42934)

  • Bug in eval() and DataFrame.eval() where passing a Series or DataFrame as expr silently parsed its (possibly truncated) repr instead of raising, producing a confusing error (GH 16289)

  • Bug in DataFrame.eval() where a duplicate column name was resolved to a single column (yielding a Series), inconsistent with pandas.eval() using resolvers=(df,) and with DataFrame.__getitem__(), which include every column with that label (GH 65588)

  • Bug in DataFrame.from_dict() where passing a dict of only scalar values raised a ValueError telling users to pass an index, even though DataFrame.from_dict() has no index parameter; the message now points to orient='index' or the DataFrame constructor (GH 25515)

  • Bug in DataFrame.replace() and Series.replace() with inplace=True and a list-like or dict to_replace where the result stopped being copy-on-write protected, so a view taken afterwards shared memory with it and writing to either one silently modified the other (GH 58966)

  • Bug in DataFrame.replace() raising IndexError instead of replacing when to_replace was list-like or dict-like, the replacement changed a column’s dtype, and two or more other columns were left unchanged (GH 61972)

  • Bug in DataFrame.select_dtypes() with an ExtensionDtype subclass such as ArrowDtype or DatetimeTZDtype raising TypeError, emitting a spurious UserWarning, or selecting the wrong columns; passing such a class now selects every column whose dtype is an instance of that class (GH 65366)

  • Bug in Series.transform() and DataFrame.transform() where passing a list of duplicate function names did not raise errors.SpecificationError (GH 54929)

  • Bug in register_option where registering an option whose name was a prefix of an existing option (e.g. "a.b" when "a.b.c" was already registered) silently overwrote the existing option’s namespace instead of raising (GH 29242)

Contributors#