What’s new in 3.2.0 (Month XX, 2027)

These are the changes in pandas 3.2.0. See Release notes for a full changelog including other versions of pandas.

Enhancements

enhancement1

enhancement2

Other enhancements

• Series, DataFrame, Index, and ExtensionArray now support the minimummaximum ufunc reduction backing np.minmax (NumPy 2.6+), returning the same results as np.minimum.reduce and np.maximum.reduce (GH 71380)

• numpy.delete() on an ExtensionArray, and numpy.concatenate() on ExtensionArray objects of the same dtype, now return an ExtensionArray of that dtype instead of a NumPy array (GH 26380)

• read_csv() and read_table() with sep=None now sniff the separator with the C engine instead of falling back to the slower python engine (GH 9645)

• read_csv() with the "c" and "python" engines now accepts a DatetimeTZDtype in dtype (GH 24542)

• DataFrameGroupBy.plot() now titles each group’s figure with its group key, unless title or ax is passed (GH 12452)

• DataFrameGroupBy.rank() and SeriesGroupBy.rank() now accept a numeric_only argument, matching DataFrame.rank() (GH 44438)

• DataFrame.plot.area() and Series.plot.area() now accept keyword arguments that only the lines or only the filled areas support, such as hatch, edgecolor and marker, instead of raising AttributeError; hatch also accepts a list with one hatch per column (GH 12354)

• DataFrame.to_csv() and other writers now accept an os.PathLike such as pathlib.Path for the archive_name entry of a zip or tar compression dict (GH 31934)

• DataFrame.to_json(), Series.to_json() and DataFrame.to_xml() now warn when a compression method is requested but no path is given, since the returned string is not compressed, and raise for an unrecognized compression as they do with a path (GH 32326)

• Series.astype() and array() with dtype="boolean" now convert values the same way as astype(bool) instead of raising for values other than booleans, 0 and 1 (e.g. "False" becomes True), with missing values becoming NA (GH 37614, GH 40566)

• DatetimeIndex.floor(), DatetimeIndex.ceil(), Series.dt.floor(), Series.dt.ceil(), Timestamp.floor(), and Timestamp.ceil() now accept a single step of a date-anchored frequency such as "MS", "QE", "W-SUN", or "B" (GH 15303)

• Series.groupby() now accepts as_index=False, returning aggregation results as a DataFrame with the group labels as columns (GH 36507)

• Added api.extensions.Accessor to the public API, for adding or overriding an accessor on a single class such as a DataFrame subclass (GH 40103)

• Added Categorical.to_dense() and Series.cat.to_dense() to get the values with the dtype of the categories, including extension dtypes (GH 26410)

• Added DatetimeIndex.set_freq() and TimedeltaIndex.set_freq(), which return a new index with the given freq instead of modifying it in place (GH 61094)

• Added Series.dt.shift() to shift datetime, timedelta, and period values by a number of frequency increments, like DatetimeIndex.shift() (GH 31705)

• Display formatting for complex numbers in object dtype and for interval dtype values with float endpoints now respects the display.precision and display.float_format options (GH 25920).

• In Jupyter notebooks, a Series is now displayed as an HTML table, with its name and dtype shown below the table as in its text repr (GH 5563)

• Printing a Styler now shows the underlying data instead of the object’s address, so the table also appears in notebooks exported to PDF with nbconvert (GH 55908)

• The ExtensionArray base tests in pandas.tests.extension.base now issue a errors.PerformanceWarning when the array uses the default, possibly slow, implementation of unique, factorize, argsort, argmin, argmax, or searchsorted (GH 24433)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

DataFrame.rank() ranks each column in its own dtype

DataFrame.rank() with axis=0 cast the whole frame to one common dtype before ranking, so its result could differ from Series.rank() on the same column. Each column is now ranked on its own and the two agree. Besides retaining extension dtypes, this changes the ranks themselves for an ordered Categorical whose categories are not in the natural order of their values, and for integers above 2**53 in a frame whose common dtype was float64 (GH 52829, GH 15420).

In [1]: cat = pd.Categorical(
   ...:     ["b", "a", "c", "a"], categories=["c", "b", "a"], ordered=True
   ...: )

In [2]: df = pd.DataFrame({"a": cat})

Previous behavior:

In [3]: df.rank()["a"].tolist()
Out[3]: [3.0, 1.5, 4.0, 1.5]

New behavior:

In [3]: df.rank()["a"].tolist()
Out[3]: [2.0, 3.5, 1.0, 3.5]

Backwards incompatible API changes

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package	Minimum Version	Required	Changed
		X	X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
tabulate	0.10.0	X

See Dependencies and Optional dependencies for more.

Other API changes

• Styler now displays missing float values as NaN rather than nan by default, matching DataFrame (GH 12710)

• ExtensionArray now implements __array_function__. Code expecting numpy.concatenate() or numpy.delete() to return a NumPy array for ExtensionArray input, including _concat_same_type and delete implementations in ExtensionArray subclasses, may need updating, as may code mixing an ExtensionArray with another library’s __array_function__ arrays (GH 26380)

• Index now displays missing float values as NaN rather than nan in its repr, matching Series and DataFrame (GH 64733)

• Timedelta and to_timedelta() now raise ValueError for a string with a comma between digits outside the seconds field, such as "1,5 days" or "1,234 days", instead of ignoring the comma (GH 71877)

• ExtensionArray._concat_same_type() now has an axis keyword (default 0); pandas only passes it for dtypes that support 2D, so existing implementations without it keep working (GH 47514)

• Styler.to_excel() now omits the rows, columns, index and header levels, and names hidden with Styler.hide(), matching the other Styler output methods; hiding the whole index with MultiIndex columns now raises NotImplementedError, as index=False does (GH 42276)

• DataFrame.copy() and Series.copy() with deep=True (the default), and ArrowExtensionArray.copy(), now copy pyarrow-backed data instead of sharing its memory, so a copy no longer keeps the original’s memory alive (GH 61930)

• DataFrame.query() now raises a ValueError when the expression does not evaluate to a boolean array, instead of silently using the result as row labels; use DataFrame.loc for label selection (GH 8560)

• range values are now displayed as range(0, 3) rather than as the tuple (0, 1, 2), matching Styler (GH 44799)

Deprecations

deprecation1

deprecation2

Other deprecations

• Deprecated merge() and DataFrame.join() producing duplicated columns when suffixed left and right column labels collide, e.g. suffixes=("_x", "_x"); this will raise a MergeError in a future version (GH 13659)

• Deprecated DatetimeIndex.snap(); for non-fixed offsets use the offset’s rollback and rollforward methods. For fixed frequencies snap returns the values unchanged; use DatetimeIndex.round() if rounding was intended (GH 27098)

• Deprecated median, min, max, idxmin, idxmax, argmin, argmax, cummin, cummax, clip, and the comparisons <, <=, >, >= on complex data; complex numbers have no natural ordering, so in a future version these will raise TypeError (GH 43770, GH 70087)

• Deprecated DataFrame.iterrows(). Use DataFrame.itertuples() instead (GH 43874)

• Deprecated pandas.to_pickle, use the DataFrame.to_pickle() or Series.to_pickle() method instead (GH 48402)

• Deprecated passing None entries in objs to concat(); filter them out before calling concat() instead (GH 57846)

• Deprecated passing a list-like name to Index.copy(); pass the label itself instead (GH 19171)

• Deprecated passing a negative min_count to DataFrame.sum(), DataFrame.prod(), Series.sum(), Series.prod(), TimedeltaIndex.sum(), the sum and prod methods of pandas arrays, and the sum, prod, min, max, first, and last methods of DataFrameGroupBy, SeriesGroupBy, and Resampler. Pass min_count=0 instead, which behaves the same (GH 50022)

• Deprecated setting an out-of-bounds numpy scalar into an array of a numpy-nullable integer dtype. The value is currently stored altered to fit (np.int64(-1) becomes 255 in UInt8); in a future version this will raise, as it already does for a Python integer. Cast the array to a dtype that can hold the value first (GH 48867)

• Deprecated the DataFrame constructor, DataFrame.from_dict() with orient="index", and DataFrame.from_records() splitting a str or bytes row into its elements when the first row is list-like; in a future version this will raise TypeError. Pass list(row) instead (GH 50461)

• Deprecated the cache_dates argument in read_csv(), read_table(), and read_fwf() (GH 68705)

• Deprecated the columns keyword in Series.drop(), which was silently ignored; use index or labels instead (GH 39509)

• Deprecated constructing a DataFrame column, Index or array() from a NumPy structured (compound) array, and casting to a compound dtype with DataFrame.astype(), Series.astype() or array(); these will raise in a future version, as the Series constructor already does. Use pd.DataFrame(arr) for one column per field or arr.astype(object) for tuples (GH 4464, GH 49388, GH 60108)

Performance improvements

• Performance improvement in DataFrame.apply() with raw=True (GH 66203)

• Performance improvement in DataFrame.explode() when exploding multiple columns (GH 53967)

• Performance improvement in read_csv() with engine="c" across the board, in tokenizing and in the machinery shared by all column types; large uncompressed local files are additionally read in parallel across multiple CPU cores by default, see above (GH 64347, GH 64515, GH 65350, GH 66271, GH 66272, GH 66273, GH 66274, GH 66276, GH 66798, GH 68314, GH 69632, GH 69748, GH 70140)

• Performance improvement and reduced memory usage in DataFrame.loc() and Series.loc() with a MultiIndex when selecting on the first level with slice(None) for the remaining levels (GH 52714)

• Performance improvement in eval(), DataFrame.eval() and DataFrame.query() with engine="numexpr" (GH 71806)

• Performance improvement in infer_freq() and DatetimeIndex.union() for values with no regular frequency (GH 70884)

• Performance improvement in read_csv() with a dict dtype on wide files or with a small chunksize (GH 70769)

• Performance improvement in DataFrameGroupBy.sum() and SeriesGroupBy.sum() with PyArrow-backed string data (GH 60861)

• Performance improvement in DataFrame.skew() and DataFrame.kurt() with types different from float64 (GH 66662)

• Performance improvement in DataFrame.to_csv() and astype(str) with timezone-aware datetime data (GH 44764)

• Performance improvement in DataFrameGroupBy.pct_change() and SeriesGroupBy.pct_change() (GH 70972)

• Performance improvement in DataFrameGroupBy.transform() and SeriesGroupBy.transform() with a user-defined function when the input rows are not already ordered by group or NA group keys are dropped (GH 71957)

• Performance improvement in DatetimeIndex.strftime(), Series.dt.strftime(), and DataFrame.to_csv() with a date_format built from %Y, %m, %d, %H, %M, %S, %f and %N (GH 44764)

• Performance improvement in MultiIndex.append() and concat() when combining many objects with a MultiIndex whose levels have few values in common (GH 70884)

• Performance improvement in PeriodIndex.strftime() with a custom date_format (GH 70106)

• Performance improvement in RangeIndex.isin(), which no longer builds the full integer array of the index when the values are integers or whole-number floats (GH 66263)

• Performance improvement in Series.var(), Series.std(), Series.sem(), and their DataFrame equivalents when bottleneck is disabled (GH 64884)

• Reduced import pandas time on Python 3.15+ by deferring rarely used subsystems with PEP 810 lazy imports (GH 65885)

• Reduced memory usage of DataFrame.loc() and DataFrame.iloc() when selecting a single row or a small slice of rows together with a list of columns (GH 69477)

• Reduced memory usage of Series.duplicated(), non-unique Index.duplicated(), and the corresponding drop_duplicates methods with keep="first" or keep="last" on data with many distinct values (GH 39799)

Bug fixes

Categorical

• Bug in DataFrame.values, DataFrame.to_numpy() and DataFrame.transpose() silently converting missing values in a Categorical column with integer or boolean categories to arbitrary integers or True (GH 38240)

• Bug in Series.clip() and DataFrame.clip() with ordered categorical data ordering lower and upper by value instead of by category position (GH 49217)

• Bug in Categorical.map(), CategoricalIndex.map() and Series.map() raising TypeError or returning incorrect results when the mapper returned unhashable values such as lists or dicts; these now give an object-dtype result (GH 40892, GH 54359)

• Bug in comparing a Categorical with an iterator raising TypeError instead of treating it as a scalar, as with a tuple (GH 31646)

• Bug in comparing unordered Categorical objects with mixed-type categories in a different order raising TypeError; Categorical.equals() and merge() were also affected (GH 51543)

Datetimelike

• Bug in Timestamp constructor, to_datetime(), and read_csv() with parse_dates where a comma used as a date separator was read as a decimal point, silently giving a wrong result (e.g. "2013,11,04" parsed as 2013-04-01); such strings now raise, and read_csv() leaves the column unparsed (GH 17265)

• Bug in infer_freq() returning None for a semi-monthly DatetimeIndex, e.g. one built with freq="SME" or freq="SMS" (GH 45819)

• Bug in DataFrame.astype(), Series.astype(), Index.astype(), DatetimeIndex, array() and to_datetime() where float dtype data outside the int64 range, such as inf, silently became a valid-looking datetime64/timedelta64 value or NaT; these now raise OutOfBoundsDatetime/OutOfBoundsTimedelta (GH 68926)

• Bug in DataFrame.to_csv() and astype(str) raising for some timezone-aware datetime data outside the range of datetime.datetime (GH 44764)

• Bug in DatetimeIndex.strftime(), Series.dt.strftime(), and DataFrame.to_csv() with date_format on Windows silently replacing values that Timestamp.strftime() cannot format (e.g. %y for years before 1900) with their default string representation; these now raise ValueError (GH 58178)

• Bug in Series.isin(), DataFrame.isin(), and Index.isin() on datetime-like data missing numpy.datetime64 or numpy.timedelta64 values (NaT, or any value on numpy < 2.2) when values also held non-datetime-like entries (GH 71186)

• Bug in Series.quantile() and DataFrame.quantile() with datetime64 or timedelta64 data and interpolation="linear" or "midpoint" losing precision for large values, so that e.g. q=0 did not return the minimum, and returning an incorrect value when the values spanned more than the int64 range (GH 49110)

• Bug in != between a Tick offset such as Hour and an array-like of timedelta64 values or pd.NA raising ValueError or TypeError (GH 71162)

• Bug in comparisons, indexing, setitem and Series.astype() to CategoricalDtype with datetime, timedelta or period values not parsing ArrowDtype string and large_string values the way StringDtype values are parsed (GH 62051)

Timedelta

• Bug in Timedelta and to_timedelta() ignoring a decimal comma in a string, so "00:00:01,500" parsed as 25 minutes instead of 1.5 seconds (GH 71877)

Timezones

• Bug in DataFrame.query() and DataFrame.eval() comparing a timezone-aware datetime column to a string without a UTC offset raising TypeError or returning incorrect results; the string is now localized to the column’s timezone, as in df["col"] < "2000-01-02" (GH 26236)

Numeric

• Bug in DataFrame.idxmax() with a nullable boolean column containing missing values returning the first False row instead of the first True row (GH 71833)

• Bug in Series.skew() and Series.kurt() (and their DataFrame counterparts) on complex data discarding the imaginary part; these now raise TypeError (GH 43770)

• Bug in Series.var() and Series.std() (and their DataFrame counterparts) returning very inaccurate results for large float32 data when bottleneck is installed (GH 70945)

• Bug in arithmetic and comparison operations between integer and float32 data on large inputs computing in float32 instead of float64 when numexpr is installed, losing precision (GH 71272)

• Bug in in-place arithmetic and logical operations (e.g. ser += other) upcasting the result or raising when other has index labels not in the left operand (GH 12755)

• Bug in integer exponentiation (**, Series.pow(), DataFrame.pow()) on large inputs returning imprecise results, 0 for negative exponents, or an upcast dtype when numexpr is installed (GH 54546)

• Fixed bug in Series.corr() with method="pearson" returning an inaccurate result for nearly-constant data, and 0.0 instead of NaN for some constant data (GH 59652)

• Fixed bug in Series.isin(), DataFrame.isin(), and Index.isin() reporting false matches when values was a list of integers above 2**53 mixed with floats, NaN, or integers outside the int64 range, or a list of bytes ending in a null byte (GH 70217)

• Fixed bug in Series.var(), Series.std(), and Series.sem() (and their DataFrame counterparts) returning inf or incorrect results for float16 dtypes when the variance or the number of values exceeded the float16 range; these now return float64 like Series.mean() (GH 41277)

Conversion

• Bug in DataFrame constructor and DataFrame.from_records() where a list of Series with extension dtypes lost those dtypes (GH 56231)

• Bug in to_numeric() raising TypeError for object values containing 0-dimensional numpy.ndarray elements instead of converting them (GH 47408)

• Bug in to_numeric() where a long value with a leading + was not correctly rounded (GH 44145)

• Bug in to_numeric() with errors="coerce" on ArrowDtype string data storing a float NaN for unparsable values instead of a missing value, and duplicating a parsed value into the position of a failed parse when the input already contained missing values (GH 67949)

• Bug in to_numeric() with a datetime-like scalar not matching the result for a Series holding the value (GH 43280)

• Bug in DataFrame.compare() and Series.compare() upcasting integer and boolean columns when the equal values were only in rows dropped from the result (GH 39899)

• Bug in DataFrame.from_records() not dropping the exclude and index columns when all of their labels were falsy, such as 0 (GH 18679)

• Bug in DataFrame.replace() with a dict or list replacing values with None casting other columns of the same dtype to object even when they contained no value to replace (GH 30512)

• Bug in DataFrame.to_dict() and Series.to_dict() sometimes returning NumPy complex scalars instead of Python complex (GH 27616)

• Bug in Series.astype() and array() raising TypeError when converting object-dtype data mixing integers and integer strings (e.g. [1, "1"]) to a nullable integer dtype (GH 34460)

• Fixed bug in to_numeric() with errors="coerce" raising TypeError for object values containing unhashable nested elements, such as a tuple containing a list, instead of coercing them to NaN (GH 69916)

Strings

• Bug in StringDtype with PyArrow storage rejecting a PyArrow string scalar in item assignment and Series.fillna(), and Series.where() with one storing a character code (GH 71075)

• Bug in DataFrame.replace() and Series.replace() silently ignoring a compiled re.Pattern for ArrowDtype string and large_string dtypes, including a Categorical of them (GH 69026)

• Bug in DataFrame.replace() and Series.replace() with regex=True on ArrowDtype string and large_string dtypes raising or silently coercing a replacement the column cannot hold instead of casting to object (GH 70458)

Interval

• Bug in IntervalIndex.mid and arrays.IntervalArray.mid returning incorrect values for numeric endpoints whose sum overflows their dtype (GH 25499)

• Bug in IntervalIndex set operations with an empty RangeIndex, and in IntervalIndex construction or astype("interval") from a RangeIndex or non-interval extension array, raising an uninformative TypeError (GH 68343)

• Bug in arrays.IntervalArray setitem with a list-like of missing values and the future.distinguish_nan_and_na option enabled storing NaN endpoints instead of a missing interval for a nullable float subtype, or raising for a nullable integer subtype (GH 71950)

• Bug in concat() and Index.append() with IntervalDtype not retaining an extension dtype subtype, either casting it to a NumPy dtype or raising TypeError (GH 64297)

• Bug in interval_range() with large integer start or end silently returning too few intervals, or empty or wrongly spaced intervals, due to float precision loss (GH 71288)

• Bug in Series.shift() and DataFrame.shift() with an IntervalDtype column not retaining the subtype, either changing it or raising TypeError (GH 69922)

Indexing

• Bug in Index where creating or converting to numpy string dtypes would raise NotImplementedError now raises TypeError (GH 50127)

• Bug in DataFrame.__setitem__() silently corrupting the DataFrame when the key matched several columns and value was a timezone-aware datetime64 or period scalar, or a one-dimensional Series of either dtype; it now broadcasts across the matched columns as the other dtypes already did (GH 71078)

• Bug in DataFrame.__setitem__() silently corrupting the DataFrame when the key matched several unique columns, e.g. a partial date string on DatetimeIndex columns; value now broadcasts across them (GH 71098)

• Bug in DataFrame.__setitem__() silently corrupting the DataFrame, or dropping columns of value, when the key matched several columns (duplicate or MultiIndex columns) and value was a two-dimensional array without one column per matched column; it now raises ValueError (GH 68445)

• Bug in DataFrame.__setitem__() with a key matching several columns and a one-dimensional extension array value (e.g. Int64) storing one array for all of them, so a later write to one column changed the others (GH 71098)

• Bug in DataFrame.isetitem() with several positions silently corrupting the DataFrame when value was a scalar or a one-dimensional extension array such as Int64; a scalar now fills every selected column and the array raises ValueError (GH 71098)

• Bug in DataFrame.isetitem() with several positions silently corrupting the DataFrame, or dropping columns of value, when an arraylike value did not have one column per position; it now raises ValueError (GH 68445)

• Bug in DataFrame.loc() and DataFrame.at() setitem with a single row label adding a new column from a list-like value raising or setting the wrong value (GH 30649)

• Bug in DataFrame.loc() and DataFrame.iloc() raising an exception when assigning values using zero-dimensional NumPy array indexers (GH 70256)

• Bug in DataFrame.loc() and DataFrame.iloc() raising or flattening a 2D value (e.g. a list of lists) set into a single object cell of a mixed-dtype DataFrame (GH 69152)

• Bug in DataFrame.loc() and DataFrame.iloc() setitem on a 1-column DataFrame with an extension dtype raising when the column key is an all-False boolean mask or an empty list-like (GH 70232)

• Bug in DataFrame.loc() and Series.loc() raising KeyError for a True or False label present in an object-dtype Index or the first level of a MultiIndex (GH 50165)

• Bug in DataFrame.loc() and Series.loc() setitem with an integer slice raising TypeError where the same .loc getitem succeeds, e.g. on an index of decimal.Decimal values (GH 70219)

• Bug in DataFrame.loc() and Series.loc() slicing with an integer or float bound that is not in the index: an object-dtype index of numbers (e.g. decimal.Decimal) raised TypeError instead of slicing by sort order, and a non-monotonic index of non-string values raised TypeError instead of KeyError (GH 70295)

• Bug in DataFrame.loc() and Series.loc() with a partial tuple key on a sorted MultiIndex ignoring the later levels of the key when a key element equals a level value of a different type, e.g. numpy.datetime64 on a level of datetime.date objects (GH 55969)

• Bug in DataFrame.loc() where setting a scalar column with a DataFrame value failed to align indices properly (GH 58482)

• Bug in Index.copy() treating a tuple name as a list of names (GH 19171)

• Bug in Index.difference() dropping elements that are not in other when other was a DatetimeIndex, TimedeltaIndex or PeriodIndex and the elements were strings that parse to its values, or when other was an IntervalIndex and the elements were scalars contained in its intervals (GH 58971)

• Bug in Series.where() and Series.mask() with a list other of nested tuples, such as [(("a", "b"), ("c", "d"))], silently returning wrong values (GH 43780)

• Bug in comparison operators between an Index and itself with a nullable or ArrowDtype dtype returning a NumPy boolean array with False or True at missing positions instead of propagating NA, inconsistent with comparing against a copy of the same Index (GH 71022)

• Bug in hash-based operations on object-dtype data (e.g. Series.isin(), factorize(), Index.get_loc() on a unique Index) inconsistently matching a key of a different type that compares equal to a value but hashes differently, such as "float64" and np.dtype("float64"); these now never match, as with a dict (GH 43581)

• Bug in selecting, setting or dropping with a list of labels that are nested tuples, such as [(("a", "b"), ("c", "d"))], raising ValueError (GH 43780)

Missing

• Bug in DataFrame.ffill() and DataFrame.bfill() with axis=1 and inplace=True not modifying the DataFrame and returning a new object instead of self (GH 71159)

• Bug in DataFrame.interpolate() and Series.interpolate() with method="index", "values", "time" or a SciPy method giving incorrect results for a nanosecond-resolution DatetimeIndex or TimedeltaIndex spanning less than about 104 days, or an integer index with values beyond 2**53 spanning less than 2**53 (GH 34601)

• Bug in DataFrame.interpolate() and Series.interpolate() with method in "linear", "index", "values" or "time" ignoring fill_value="extrapolate" and repeating the first or last valid value instead of extrapolating linearly (GH 31949)

• Bug in DataFrame.interpolate() and Series.interpolate() with a large limit raising on 32-bit platforms and being very slow on long runs of missing values (GH 34936)

• Bug in Index.equals() and Series.equals() returning False for identical float32 or complex NaN values in object-dtype data that also holds elements such as a Series (GH 55492)

• Bug in DataFrame.mask(), DataFrame.where(), Series.mask(), and Series.where() handling missing values in a nullable boolean cond inconsistently between the two methods and depending on inplace; missing values are now treated as False, as in boolean indexing (GH 35429, GH 52955, GH 56844, GH 60729)

• Bug in array(), Series.map(), Index.insert(), the DataFrame constructor and setting with enlargement with the future.distinguish_nan_and_na option enabled converting NaN to NA for nullable and PyArrow dtypes, or raising for Series.map() on nullable integer data when every result is NaN (GH 71870)

• Bug in Series.drop_duplicates(), Series.unique(), Series.value_counts(), Series.isin() and related methods on object dtype treating NumPy NaN scalars such as np.float64("nan") as distinct values (GH 16632)

• Bug in Series.map() and Index.map() with the future.distinguish_nan_and_na option enabled turning None results and values missing from a dict or Series mapper into NaN instead of NA for nullable and PyArrow dtypes (GH 71949)

MultiIndex

• Bug in MultiIndex constructor and MultiIndex.set_codes() silently converting NaN or non-integer float codes to valid codes instead of raising ValueError (GH 26210)

• Bug in DataFrame.dropna() raising KeyError when subset contained a partial key of a MultiIndex (GH 17737)

• Bug in DataFrame.sort_index(), Series.sort_index(), MultiIndex.sort_values() and MultiIndex.is_monotonic_increasing() ignoring the category order of a CategoricalIndex level (GH 47607)

• Bug in Index.map() raising when mapping to tuples with a different number of elements than a MultiIndex has levels, or when only some results are tuples (GH 24800)

• Bug in MultiIndex.append(), concat(), DataFrame.join() and merge() emitting a spurious RuntimeWarning when MultiIndex level values could not be sorted (GH 61477)

• Bug in MultiIndex.from_tuples() with an object ndarray, MultiIndex.append() and Index.map() truncating or raising IndexError on tuples of different lengths; MultiIndex.from_tuples() now raises a ValueError for an ndarray of tuples of different lengths (GH 50293)

• Bug in MultiIndex.intersection(), MultiIndex.union(), MultiIndex.difference() and MultiIndex.symmetric_difference() with a list-like other holding non-tuple entries raising TypeError or splitting a string entry into level values; such entries are now treated like an Index other (GH 39699)

• Bug in MultiIndex.rename() raising TypeError for the name keyword documented by Index.rename() (GH 33778)

• Bug in MultiIndex.symmetric_difference() raising ValueError when other holds non-tuple entries (GH 39699)

• Bug in Series.map(), Index.get_indexer(), DataFrame.reindex() and DataFrame.loc with a MultiIndex raising IndexError or matching on a prefix when looking up tuples whose length differs from the number of levels (GH 50293)

• Bug in DataFrame.loc() and Series.loc() setitem with a partial MultiIndex key (e.g. df.loc[:, "a"] = df.loc[:, "a"] * 2) setting NaN (or raising) instead of the value (GH 10440, GH 15310, GH 40186, GH 43719, GH 45751, GH 46837)

I/O

• read_csv() with engine="pyarrow" now raises ValueError for the unsupported encoding_errors with bytes or file input instead of silently ignoring it (GH 71298)

• Bug in DataFrame repr and DataFrame.to_string() where numeric column labels of differing widths were misaligned with their column’s values (GH 8300)

• Bug in json_normalize() where a meta path not following record_path raised KeyError or silently returned a same-named field from the wrong object (GH 60254)

• Bug in read_csv() silently misparsing non-ASCII thousands and decimal markers with the C engine; these now fall back to the Python engine with a errors.ParserWarning, or raise ValueError if engine="c" is explicitly specified (GH 71158)

• Bug in read_csv() where a dtype given for a parse_dates column undid the date parsing, returning strings, objects, integers or categories instead of datetimes (GH 57512)

• Bug in read_csv() with engine="c" and comment not skipping a line containing only whitespace before the comment character (GH 55916)

• Bug in read_csv() with engine="c" and skiprows treating a quote in a skipped leading line as opening a field, skipping the wrong lines or raising EmptyDataError (GH 34732)

• Bug in read_csv() with engine="c" occasionally dropping leading whitespace from the first value of a line in large inputs (GH 34085)

• Bug in read_csv() with engine="c" shifting a row’s values one column left when the row starts with the delimiter and follows a blank, comment, or skipped line in a file with \r line endings (GH 71409)

• Bug in read_csv() with engine="pyarrow" and parse_dates converting a non-date index_col (e.g. integers) to strings (GH 71873)

• Bug in read_csv() with engine="pyarrow" raising EmptyDataError for input whose only line lacks a trailing newline (GH 62635)

• Bug in read_csv() with engine="pyarrow" silently wrapping values outside the range of the requested integer dtype instead of raising (GH 38013)

• Bug in read_csv() with engine="python" and index_col reading a date like "20180101" as nanoseconds since the epoch when dtype is a pyarrow timestamp dtype (GH 24542)

• Bug in read_csv(), read_excel(), read_fwf() and read_html() returning wrong column labels and treating header rows as data when header was a list not in increasing order, such as header=[1, 0] (GH 47011)

• Bug in read_csv(), read_html() and other I/O functions raising UnicodeEncodeError for URLs containing non-ASCII characters (GH 21499)

• Bug in read_csv(), DataFrame.to_csv() and other I/O functions raising URLError for sftp:// URLs instead of reading or writing them with fsspec (GH 46765)

• Bug in read_excel() raising ValueError when header is a list and index_col consumes all columns (GH 66372)

• Bug in read_html() raising ImportError when lxml is not installed instead of falling back to bs4 + html5lib with the default flavor=None (GH 30281)

• Bug in read_html() with the bs4 or html5lib parser dropping tables that have identical content, so only the first was returned (GH 49934)

• Bug in read_json() raising ValueError on numbers whose integer part does not fit in int64 or uint64 (e.g. 100000000000000000000.5), and silently returning a wrong value for some of them (GH 20599)

• Bug in read_orc() raising whenever filesystem was passed (GH 58746)

• Bug in read_sas() exhausting memory instead of raising ValueError on a SAS7BDAT file that claims more rows than it can hold, when read from an uncompressed file, a zip archive, or a BytesIO (GH 70721)

• Bug in read_sas() returning rows that were deleted from an uncompressed SAS7BDAT file (GH 15963)

• Bug in read_sas() with format="xport" reading zeros as 5.397605e-79 (GH 50670)

• Bug in read_xml() with names and an iterparse list repeating a name, where values were shifted into the wrong column or dropped in large files or when repeated values were equal or empty (GH 71275)

• Bug in DataFrame.to_csv() with MultiIndex columns and an unnamed index or index_label=False where a first row of missing values was read back by read_csv() as the index names and dropped; an empty index-names line is now written in this case (GH 21995)

• Bug in DataFrame.to_csv(), DataFrame.to_json(), DataFrame.to_pickle() and similar writers where writing to a file:// URL raised for a new file and silently left an existing file unchanged; writing to other URLs such as http:// now raises instead of silently discarding the data (GH 55828)

• Bug in DataFrame.to_excel() where a NaN label in MultiIndex columns was written as the last value of that level instead of an empty cell (GH 62340)

• Bug in DataFrame.to_json() and Series.to_json() dropping the nanoseconds of tz-aware datetime columns and Timestamp objects, and with date_format="epoch" mis-rounding pre-1970 datetime objects or raising OverflowError for ones outside the nanosecond bounds at a coarser date_unit (GH 53473)

• Bug in DataFrame.to_parquet() with engine="pyarrow" where a misspelled keyword argument truncated an existing file at the destination or was reported as an error about the path (GH 45815)

• Bug in DataFrame.to_sql() creating MySQL and MariaDB datetime and time columns that silently dropped fractional seconds (GH 48435)

• Bug in DataFrame.to_sql() with if_exists="append" or if_exists="delete_rows" writing to an existing table hanging on SQL Server while another session had uncommitted changes to an unrelated table (GH 52601)

• Bug in DataFrame.to_sql() with if_exists="delete_rows" raising InvalidRequestError for an existing temporary table (GH 52601)

• Bug in DataFrame.to_stata() raising ValueError for an empty DataFrame with a categorical column, and read_stata() not converting the column back to a categorical (GH 71126)

• Bug in DataFrame.to_string(), Series.to_string() and DataFrame.to_html() ignoring na_rep for None, pd.NA, and NaT values in object-dtype columns and an explicit na_rep="NaN" for any missing value, and in DataFrame.to_xml() raising for na_rep with nullable, pyarrow-backed, or categorical columns containing missing values (GH 54872)

• Bug in DataFrame.to_string(), Series.to_string() and their repr where values or labels containing ANSI color codes were counted toward the column width, misaligning the other rows (GH 55122)

• Bug in displaying a Series or DataFrame in IPython with display.html.table_schema enabled printing a traceback when the data cannot be converted to JSON, such as non-UTF-8 bytes (GH 16848)

• Bug in the display of a MultiIndex (repr, DataFrame.to_string(), DataFrame.to_html(), Styler) where distinct labels that display identically (e.g. 1 and "1") were shown as a single label; DataFrame.to_excel() merged them into one cell, losing all but the first value (GH 10796)

• Bug in the repr of a Series or DataFrame where values displayed as 0 by the display.chop_threshold option still caused the column to be shown in scientific notation (GH 12374)

• Bug in the repr of a Series or DataFrame with longdouble (e.g. float128) dtype showing values outside the float64 range as 0 or inf (GH 17809)

• Fixed bug in read_csv() where callable skiprows received incorrect row numbers on 32-bit big-endian architectures (GH 69985)

• Bug in ExcelWriter with engine="openpyxl" ignoring date_format and datetime_format (GH 44284, GH 59882)

• read_json() with orient="table" now reads back non-string column labels and index names, such as the default integer columns of DataFrame([[1, 2]]), instead of returning all-NaN values or raising IntCastingNaNError; labels it cannot match to the data, such as 5 alongside "5", now raise ValueError (GH 19129)

• DataFrame.to_json() and Series.to_json() with orient="table", and build_table_schema(), no longer raise AttributeError for a MultiIndex with non-string level names (GH 19129)

• Bug in read_csv() where using engine="python" with custom converters returning unhashable types (such as lists or dicts) raised TypeError (GH 56848)

• Bug in read_excel() with dtype_backend="numpy_nullable" converting mixed-type columns and datetime.time columns to string dtype; such columns now preserve object dtype and the original Python types (GH 63830)

• Bug in read_excel() with dtype_backend="pyarrow" raising pyarrow.ArrowInvalid for columns with mixed types (e.g. integers and strings); such columns now fall back to object dtype, matching the default behaviour without dtype_backend="pyarrow" (GH 63830)

• Bug in read_json() with orient="table" raising for a tz-naive datetime field whose values carry a UTC offset, such as files written by pandas before 1.5; the values are now converted to UTC (GH 52595)

• Bug in read_json() with typ="series" returning datetime64 dtype for all-null data instead of float64, matching DataFrame columns (GH 35464)

• Bug in read_sas() raising IndexError or reading the wrong format or label for SAS7BDAT files with columns renamed or relabeled after the file was written (GH 60809)

• Bug in read_sql_query() and read_sql_table() with a SQLAlchemy connection returning column labels that require sqlalchemy to unpickle (GH 52816)

• Bug in read_sql() and read_sql_query() raising or altering a query containing an unescaped % and no params when using a SQLAlchemy connection to a database driver such as pymysql or psycopg2 (GH 49561)

• Bug in DataFrame.to_json() and Series.to_json() with orient="table" declaring complex data as a "number" field that read_json() then failed to read with TypeError; it now round-trips (GH 50782)

• Bug in DataFrame.to_json() and Series.to_json() writing floats near the largest float64 value in a form that reads back as inf (GH 21454)

• Bug in HDFStore.append() raising ValueError: Buffer dtype mismatch instead of the “different categories” error when appending a categorical whose categories have a different dtype than the existing ones (GH 71128)

• Bug in HDFStore.remove() deleting unselected rows when passing unsorted row coordinates (GH 71276)

• Bug in HDFStore.select() and HDFStore.select_as_multiple() returning the wrong rows when combining where with start or stop and chunksize or iterator (GH 71278)

• Bug in Series.to_json() with orient="table" writing a falsy Series name such as 0, False or "" as "values" (GH 19129)

• Bug where the pickled bytes (e.g. for joblib.hash) of a Series, DataFrame or Index with datetimelike or extension-dtype data depended on which operations had previously been run on it (GH 49170)

• Fixed a segfault in DataFrame.to_json() and Series.to_json() with date_format="iso" when datetime conversion raises an exception (GH 69461)

Period

• Bug in PeriodIndex and array() with a period dtype raising when given a list of nanosecond-unit np.datetime64 values, and PeriodIndex raising AttributeError instead of ValueError for datetime64 data without a freq (GH 70139)

• Bug in Period constructor where the string of a weekly Period such as "2011-12-26/2012-01-01" or "2000-01-03/2000-01-09" was silently parsed as a minute-frequency Period (GH 48000)

• Bug in Period, PeriodIndex, period_range() and casting to PeriodDtype silently dropping the timezone of timezone-aware input; these now emit a UserWarning like Timestamp.to_period() (GH 47005)

• Bug in Period.strftime() and PeriodIndex.strftime() where %Y was not zero-padded for years before 1000 on Linux (GH 48746)

Plotting

• Bug in plotting.table() ignoring rowLabels and colLabels or raising a TypeError when they were passed (GH 37811)

• Bug in DataFrame.hist() and Series.hist() raising ValueError when passing weights together with by (GH 9540)

• Bug in DataFrame.hist() with by and no column plotting the numeric grouping column alongside the other columns (GH 41188)

• Bug in DataFrame.plot.hist() with by raising TypeError when bins was a sequence of bin edges (GH 11483)

• Bug in DataFrame.plot.hist() with subplots=True and sharex=False using the same bins for every subplot, so columns on a smaller scale collapsed into a single bar (GH 13489)

• Bug in DataFrame.plot.pie() and Series.plot.pie() silently ignoring color (GH 11228)

• Bug in DataFrame.plot() and Series.plot() drawing data indexed by period-end timestamps (e.g. freq="ME" or "W") at the start of each period when combined with a higher-frequency series on the same or shared axes (GH 11574)

• Bug in DataFrame.plot() and Series.plot() ignoring style when xerr or yerr is passed (GH 7023)

• Bug in DataFrame.plot() and Series.plot() ignoring the axes.grid.axis and axes.grid.which rcParams, e.g. from a style file, and drawing major gridlines on both axes (GH 17725)

• Bug in DataFrame.plot() and Series.plot() raising ValueError when plotting a single series or column with color as a tuple of three or four colors, instead of treating it like the equivalent list (GH 18449)

• Bug in DataFrame.plot() and Series.plot() with kind="bar" or kind="barh" raising TypeError on timedelta64 data containing NaT (GH 39320)

• Bug in DataFrame.plot() and Series.plot() with style as a tuple, np.ndarray or Series plotting the styles as data instead of applying them like the equivalent list (GH 46289)

• Bug in DataFrame.plot() and Series.plot() with a TimedeltaIndex that has no regular frequency, or with x_compat=True, plotting the points evenly spaced and leaving most x-axis ticks unlabeled (GH 19965)

• Bug in DataFrame.plot() and Series.plot() with a regular-frequency DatetimeIndex or PeriodIndex cutting off data already plotted on axes sharing the x-axis (sharex=True) (GH 14074)

• Bug in DataFrame.plot() and Series.plot() with a regular-frequency DatetimeIndex or PeriodIndex mislabeling ticks set with xticks= or ax.set_xticks (GH 18881)

• Bug in Series.plot() and DataFrame.plot() drawing datetimes outside the visible range when they share the x-axis (sharex=True, twinx, or secondary_y=True) with a time-series plot (GH 14322, GH 40470)

• Bug in Series.plot.hist() and DataFrame.plot.hist() with datetime64 data drawing only empty bars (GH 17372)

• Bug in Series.plot() and DataFrame.plot() raising when a datetime index containing NaT is drawn on a time-series axes (GH 40470)

• Bug in Series.plot() and DataFrame.plot() widening the x-axis to start at 1970 when a time series is added to a time-series plot that holds an axhline (GH 71436)

• Bug in Series.plot() and DataFrame.plot() with a timezone-aware index raising ValueError when the first timestamp falls in a daylight saving time transition, e.g. the repeated hour of a fall-back day (GH 62936)

Groupby/resample/rolling

• Bug in DataFrameGroupBy.apply() and SeriesGroupBy.apply() with group_keys=False not preserving the order of the rows of the input when the index has duplicates (GH 57906)

• Bug in DataFrameGroupBy.cumsum(), DataFrameGroupBy.cummin(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.rank() (and Series variants) on datetime64, timedelta64, and PeriodDtype data treating NaT as a regular value and losing precision when the group key contains NA (GH 69923)

• Bug in DataFrameGroupBy.cumsum(), DataFrameGroupBy.cumprod(), DataFrameGroupBy.cummin(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.rank() raising NotImplementedError, often with no message, instead of TypeError on columns that do not support them, e.g. object columns and ArrowDtype date, time and binary columns (GH 69791)

• Bug in DataFrameGroupBy.max(), DataFrameGroupBy.min(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.cummin() (and Series variants) on nullable float data with the future.distinguish_nan_and_na option enabled returning inf or -inf for all-NaN groups and ignoring NaN in other groups; NaN now propagates as in DataFrameGroupBy.sum(); under the default option, ArrowDtype float data containing NaN now gives NA instead of ignoring the NaN (GH 59106)

• Bug in DataFrameGroupBy.resample() followed by Resampler.aggregate() with a list of functions aggregating the grouping and on columns, raising TypeError or returning them as extra result columns (GH 32162)

• Bug in DataFrameGroupBy.shift() and DataFrameGroupBy.pct_change() with freq including the grouping columns in the result, or raising for non-numeric grouping columns (GH 46944)

• Bug in DataFrameGroupBy.shift(), SeriesGroupBy.shift(), DataFrameGroupBy.diff(), and SeriesGroupBy.diff() with periods=0 not setting rows with an NA group key to missing (GH 70996)

• Bug in DataFrameGroupBy.sum(), SeriesGroupBy.sum() and DataFrame.sum() with axis=1 ignoring min_count for ArrowDtype string data (GH 60861)

• Bug in DataFrameGroupBy.transform() with a function returning a scalar for each column casting the results to a common dtype (GH 42617)

• Bug in GroupBy.median(), GroupBy.skew(), and GroupBy.kurt() on complex data silently discarding the imaginary part; these now raise TypeError (GH 43770)

• Bug in GroupBy.quantile() raising NotImplementedError with no message, instead of TypeError, on ArrowDtype columns it does not support, e.g. binary columns (GH 69808)

• Bug in GroupBy.std() and GroupBy.sem() on complex data silently discarding the imaginary part, and GroupBy.var() raising; these now match Series.std(), Series.sem(), and Series.var() (GH 43770)

• Bug in Rolling.aggregate() with on raising ValueError when a column is given a list of functions, e.g. agg({"B": ["min", "max"]}) (GH 43769)

• Bug in SeriesGroupBy.transform() with a callable raising for Series subclasses that cannot hold the transformed values; the result now falls back to the type the subclass chooses (geopandas/geopandas#3059, GH 71165)

• Bug in SeriesGroupBy.value_counts() returning counts attached to the wrong interval when the passed bins resulted in empty counts (GH 32471)

• Bug in DataFrame.agg(), DataFrameGroupBy.agg(), and resample/rolling agg with a dict, list, or named aggregation over more than 100 columns emitting a spurious PerformanceWarning with as_index=False or on a later DataFrame.reset_index() (GH 61628)

• Bug in DataFrame.groupby() and Series.groupby() with sort=True and dropna=False placing the NaN group in the middle of the result instead of last for object-dtype keys; Index.union() had the same misordering (GH 70216)

• Bug in DataFrame.groupby(), DataFrame.resample(), DataFrame.rolling(), DataFrame.expanding(), and DataFrame.ewm() raising TypeError when selecting MultiIndex columns with a tuple key (GH 13914)

• Bug in DataFrame.resample(), Series.resample(), and Grouper raising NotImplementedError when rule or freq is a generic DateOffset (e.g. DateOffset(months=1)) (GH 31697)

• Bug in Series.resample() and DataFrame.resample() with a PeriodIndex and closed="right" returning shifted labels or incorrect aggregations (GH 44363)

• Bug in Series.resample() and DataFrame.resample() with a timezone-aware index and a Day or anchored frequency (e.g. "D", "W-FRI") raising when a bin edge fell on a midnight that does not exist or is ambiguous due to a daylight saving time transition (GH 62601)

• Bug in time-based Rolling.cov() and Rolling.corr() raising ValueError or silently returning wrong results when other has a different index; the result is now computed on the aligned index, as with integer windows. With on, a mismatched index now raises ValueError (GH 62937)

Reshaping

• Bug in concat() and the DataFrame constructor losing the freq of a DatetimeIndex or TimedeltaIndex depending on the order of the inputs (GH 64253)

• Bug in concat() raising ArrowInvalid when casting integers exceeding 2**53 to a PyArrow float dtype, e.g. int64[pyarrow] with uint64[pyarrow]; these now round as with NumPy dtypes (GH 58819)

• Bug in concat() raising ValueError when tuple keys contain a missing value such as None (GH 57993)

• Bug in merge_asof() with right_index=True and an empty left returning the index of right instead of left (GH 41171)

• Bug in merge() raising ValueError when merging a numeric key with an object key that is not all strings, e.g. [1.5, None], [1.5, "a"], or an empty or all-missing key (GH 70218)

• Bug in DataFrame.pivot() and pivot() with a list of values casting the pivoted columns to a common dtype, e.g. integers to float64 or object (GH 43547)

• Bug in DataFrame.pivot(), DataFrame.pivot_table(), DataFrame.unstack() and Series.unstack() raising TypeError on complex data when the result has missing cells (GH 25221)

• Bug in DataFrame.stack() where missing labels in the result’s MultiIndex could not be found by lookups such as .loc; unstacking such a result now places the missing label first unless sort=False is passed (GH 40366)

Sparse

• Bug in setting integer SparseDtype values into an integer Series or Index (__setitem__, .loc, .iloc, Series.where(), Series.mask(), Series.update(), Index.putmask()) raising AttributeError: 'SparseDtype' object has no attribute 'itemsize' (GH 68457)

ExtensionArray

• Bug in read_csv() and other parsing of strings into a decimal ArrowDtype raising ArrowInvalid instead of parsing the values (GH 69838)

• Bug in ArrowExtensionArray.factorize() ignoring use_na_sentinel=True for dictionary-encoded PyArrow data that stores nulls as dictionary values, keeping the null in the returned uniques or raising ArrowInvalid for chunked data (GH 66490)

• Bug in DataFrame.apply() and DataFrame.aggregate() where a single callable reducing each column (or row) of a nullable- or PyArrow-backed frame to a scalar gave a NumPy-dtyped result, e.g. float64 rather than double[pyarrow] (GH 61812)

• Bug in Series.dt.strftime() on PyArrow-backed timestamp and date data where %S included fractional seconds and %f and %N were not supported (GH 55001)

• Bug in Series.isin(), DataFrame.isin(), and Index.isin() on PyArrow-backed dtypes raising when values held a mix of types PyArrow cannot combine (GH 70300)

• Bug in Series.mode() and DataFrame.mode() with dropna=False raising TypeError for "boolean" dtype (GH 70487)

• Bug in Series.rank() raising ArrowNotImplementedError for an ArrowDtype that PyArrow cannot rank, such as list[pyarrow]; the values are now ranked as Python objects, so lists rank lexicographically (GH 69136)

• Bug in the DataFrame constructor with a dtype such as "Float64" not copying its input when it should, so that writes to one changed the other and operations like DataFrame.unstack() could be very slow (GH 52338)

• Bug in Series.replace(), Series.where(), Series.fillna(), item assignment and related methods on a PyArrow-backed integer dtype silently truncating a fractional value, including one passed in a list (replace(2, 1.5) stored 1); these now raise TypeError, while Index.where(), Index.putmask() and Index.fillna() upcast, matching the nullable integer dtypes (GH 68638, GH 71076)

• Bug in Series.skew() and Series.kurt() (and their DataFrame counterparts) on PyArrow-backed numeric data returning an arbitrary value for constant data, losing precision on data with a large offset, and raising TypeError with pyarrow < 20; undefined results are now NA instead of NaN, matching the nullable float dtypes (GH 71829)

• Bug in Series.value_counts() with dropna=False on nullable dtypes placing missing values last (or last among equal counts) instead of in order of first appearance (GH 23074)

• Bug in arithmetic and logical operations on PyArrow-backed data raising pyarrow.ArrowInvalid or pyarrow.ArrowTypeError instead of TypeError when the other operand contains objects PyArrow cannot convert (GH 62682)

• Bug in comparisons between PyArrow-backed data and objects PyArrow cannot convert raising pyarrow.ArrowInvalid or pyarrow.ArrowTypeError; ==/!= now compare elementwise, ordered comparisons raise TypeError, and a missing value on either side propagates instead of answering False (GH 62682)

• Bug in constructing or setting values into a numeric, boolean or temporal ArrowDtype from a NumPy string array sometimes crashing the interpreter or returning incorrect values (GH 71827)

Styler

• Bug in Styler methods taking a level argument, such as Styler.hide() and Styler.format_index(), ignoring or raising on negative, out-of-range or NumPy integer levels; level is now resolved as in DataFrame.droplevel(), so an integer matching a level name refers to that level (GH 42933)

• Bug in Styler methods taking a subset argument, such as Styler.format(), Styler.map() and Styler.hide(), raising KeyError for a callable subset (GH 46685)

• Bug in Styler.background_gradient() and Styler.text_gradient() coloring missing values with the lowest colormap color instead of the missing-value color when all valid values in a column, row, or table were equal (GH 47695)

• Bug in Styler.bar() raising TypeError for data containing pd.NA when align is "mean" or a callable, or when cmap is given (GH 56425)

• Bug in Styler.map(), Styler.apply() and Styler.set_table_styles() truncating CSS values that contain a ; inside quotes or parentheses, such as url("data:image/png;base64,...") (GH 59623)

• Bug in Styler.to_excel() applying styles to the wrong columns when columns selects or reorders columns (GH 42276)

• Bug in Styler.to_html() where Chrome and other Chromium-based browsers did not apply a style shared by more than 8192 cells to the cells beyond the first 8192 (GH 40913)

Other

• Bug in api.interchange.from_dataframe() with allow_copy=False always raising RuntimeError for objects supporting the Arrow PyCapsule Interface, even when a zero-copy conversion is possible (GH 71189)

• Bug in eval() with a multi-line expression and a target raising UndefinedVariableError when a line after the first referenced a variable (GH 31952)

• Bug in eval(), DataFrame.eval() and DataFrame.query() where not performed a bitwise inversion instead of a logical negation, e.g. pd.eval("not True") returned -2 and df.query("a > 1 and not True") could return rows (GH 11367)

• Bug in testing.assert_series_equal() and testing.assert_frame_equal() with check_dtype=False and check_exact=True comparing missing values against an object column differently from check_exact=False; pd.NA and NaT no longer match np.nan or None (GH 61473)

• Bug in testing.assert_series_equal(), testing.assert_frame_equal() and testing.assert_index_equal() treating np.nan and None as equal when they appear inside NumPy arrays stored in object-dtype values (GH 55492)

• Bug in util.hash_array() and util.hash_pandas_object() raising UnicodeDecodeError on mixed-type object values containing non-ASCII bytes; bytes in mixed-type values now hash the same as in non-mixed ones (GH 27215)

• Bug in DataFrame.agg() with a list of functions raising ArrowInvalid on pyarrow-backed columns with large integer results (GH 71875)

• Bug in DataFrame.agg(), Series.agg() and the agg methods of groupby, resample and window objects silently ignoring *args and **kwargs when func is a dict, e.g. df.agg({"a": "sum"}, min_count=1); they are now passed to each function, as with a list of functions, so a function that does not accept them now raises TypeError (GH 23889)

• Bug in DataFrame.apply() with raw=True casting scalar results to the dtype of the first one, e.g. 1 followed by 2.5 gave 2; the dtype is now inferred from all results, as with raw=False (GH 61632)

• Bug in DataFrame.duplicated() and DataFrame.drop_duplicates() raising ValueError when subset is an empty list; no rows are now marked as duplicates, matching a DataFrame with no columns (GH 12869)

• Bug in DataFrame.eval() and DataFrame.query() with engine="numexpr" where a column used again in a later line of a multi-line expression, or after a subscript such as a[a > 0], was treated as a NumPy array, raising TypeError or silently giving wrong results; eval() also no longer replaces the values of a passed resolvers mapping with NumPy arrays (GH 59062)

• Bug in DataFrame.eval(), DataFrame.query(), and eval() with the default engine="numexpr" raising RecursionError for arithmetic between object or string columns (e.g. "a + b" on two string columns) and ValueError for an expression that is a single object or string column or string literal, such as "c = 'foo'" (GH 15320, GH 21088, GH 27639)

• Bug in DataFrame.eval(), DataFrame.query(), and eval() raising TypeError when adding a string to an object or string column, e.g. df.eval("a + 'x'") (GH 47734)

• Bug in DataFrame.eval(), DataFrame.query(), and eval() with the default engine="numexpr" raising ValueError for a unary operator such as not on an object or string column (GH 45876)

• Bug in DataFrame.from_dict() with orient="index" silently setting a row’s values to NaN when its tuple key contained None or NaN (GH 28390)

• Bug in DataFrame.query() and DataFrame.eval() raising ValueError or UndefinedVariableError when the expression calls Timestamp, datetime, list or tuple, e.g. df.query("a > Timestamp('2020-01-01')") (GH 50442)

• Bug in DataFrame.query() and DataFrame.eval() where == and != against a string gave different results than the same comparison outside of query, e.g. returning no rows for a datetime64, timedelta64, or PeriodDtype column, or raising NotImplementedError with parser="python" (GH 54199)

• Bug in DataFrame.query() and DataFrame.eval() where a column named inf or Inf was ignored in favor of np.inf, raising KeyError or silently returning wrong results (GH 47859)

• Bug in DataFrame.query() and DataFrame.eval() with engine="numexpr" raising AttributeError for some comparisons and in/not in checks whose operand is a math function or unary operator, e.g. abs(a) in [1, 2] (GH 31848)

• Bug in Index.get_loc(), Series.value_counts(), Series.duplicated(), Series.isin() and other object-dtype operations where a SystemExit raised by a custom __hash__ or __eq__ was silently suppressed (GH 57052)

• Bug in calling a generalized ufunc on a Series raising AttributeError when it reduced object data to a single object pandas does not recognize as a scalar, e.g. shapely.geometrycollections returning a geometry (GH 71109)

• Bug in eval(), DataFrame.eval(), and DataFrame.query() raising on some datetime64 and timedelta64 operations, e.g. df.query("time > @now - @delta"), and returning float32 for a float32 column times a np.timedelta64 (GH 30729, GH 52559)

• Bug in DataFrame.eval() and eval() where assigning to a backtick-quoted name that is not a valid Python identifier, e.g. df.eval("`a-b` = a + b"), created a column labeled BACKTICK_QUOTED_STRING_a_MINUS_b instead of "a-b" (GH 47699)

• Bug in DataFrame.replace() and Series.replace() with inplace=True and a list-like or dict to_replace using memory proportional to the number of values to replace times the length of the data, raising MemoryError for large mappings (GH 25816)

• Bug where configuration options and pandas’ C extension modules could be left uninitialized when running under Python’s global lazy imports mode (-X lazy_imports=all) (GH 71173)

Contributors

A total of 101 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

• ASWANTH R +

• Aahan G +

• Aarya +

• Aarya Prakash +

• Aarya2004 +

• Abhishek Kumar +

• Abhishek kr. Chauhan +

• Abhishek1106kr +

• Aidan Feldman

• Ammar Hassona +

• Anirudh +

• Aniruktha +

• Anushka Dudhe +

• Atik +

• Ayush Kumar +

• Ayush Srivastava +

• Breno Barreto +

• Brock

• Chaimaa Abi +

• Chetan Sahney +

• Davin Khotob +

• Deon Binny +

• DeviousCardi +

• Dr-Irv

• Enrico Lockner +

• Eugenio “Jay” Zuccarelli +

• Fangchen Li

• Filipe Rangel +

• Furqan Khan +

• GON +

• Ge Li +

• Gore Williams +

• H23 +

• Iason Krommydas +

• Igor Souza Lima +

• Ishaan

• Ishan Kumar +

• Itssubhraneelbruv +

• Johnny Kao +

• Johnny Wilson Dougherty +

• Joris Van den Bossche

• Juan Miguel Contreras +

• Julian Harbeck

• Junpei Zhang +

• Koussay-mehdouani +

• Krishn +

• Laraib Butt +

• Leo Camus +

• Likhitha J +

• Maha Rana Yadavalli +

• Maha Yadavalli +

• Mannava Daasaradhi +

• Manvi Soni +

• Matthew Roeschke

• Minh Tuan Pham +

• Mohd Maaz Khan +

• Monika Dvorackova +

• N3verm1nd

• Natalia Mokeeva

• Noah

• Odysseas Yiakoumis +

• Pablo Galindo Salgado +

• Parman Mohammadalizadeh +

• Pieter Eendebak +

• Pranav Choudhary

• Riaz1729 +

• Richard Shadrach

• Ryan George +

• S.A +

• Saihritesh

• SamuelOndrus +

• Seungtaek Oh +

• Sharathkumara +

• Shriraj Jadhav +

• Shuheng (Markus) Ma +

• Sugandh Garg +

• TensorDevLJ +

• Tharoboiz +

• Tiereonna +

• Trinh Quoc Anh

• Trupti603 +

• Vincent Ngo +

• Vividh Yadav +

• Weijian He +

• Wrethis +

• britz +

• furqankhan +

• guptaishaan

• jbrockmendel

• layeh +

• maharanay22 +

• matiaslindgren

• pei711 +

• ramchaser

• root

• sundeep8967 +

• tanveer-arch +

• ump45nose

• yuchen-xu +

• Álvaro Kothe

• “galleonli” +