What’s new in 3.2.0 (Month XX, 2027)

These are the changes in pandas 3.2.0. See Release notes for a full changelog including other versions of pandas.

Enhancements

enhancement1

enhancement2

Other enhancements

• read_csv() and read_table() with sep=None now sniff the separator with the C engine instead of falling back to the slower python engine (GH 9645)

• read_csv() with the "c" and "python" engines now accepts a DatetimeTZDtype in dtype (GH 24542)

• DataFrameGroupBy.rank() and SeriesGroupBy.rank() now accept a numeric_only argument, matching DataFrame.rank() (GH 44438)

• DataFrame.to_csv() and other writers now accept an os.PathLike such as pathlib.Path for the archive_name entry of a zip or tar compression dict (GH 31934)

• Series.groupby() now accepts as_index=False, returning aggregation results as a DataFrame with the group labels as columns (GH 36507)

• Added Categorical.to_dense() and Series.cat.to_dense() to get the values with the dtype of the categories, including extension dtypes (GH 26410)

• Added DatetimeIndex.set_freq() and TimedeltaIndex.set_freq(), which return a new index with the given freq instead of modifying it in place (GH 61094)

• Display formatting for complex numbers in object dtype and for interval dtype values with float endpoints now respects the display.precision and display.float_format options (GH 25920).

• Printing a Styler now shows the underlying data instead of the object’s address, so the table also appears in notebooks exported to PDF with nbconvert (GH 55908)

• The ExtensionArray base tests in pandas.tests.extension.base now issue a errors.PerformanceWarning when the array uses the default, possibly slow, implementation of unique, factorize, argsort, argmin, argmax, or searchsorted (GH 24433)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

notable_bug_fix1

notable_bug_fix2

Backwards incompatible API changes

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package	Minimum Version	Required	Changed
		X	X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
		X

See Dependencies and Optional dependencies for more.

Other API changes

• Styler now displays missing float values as NaN rather than nan by default, matching DataFrame (GH 12710)

• Index now displays missing float values as NaN rather than nan in its repr, matching Series and DataFrame (GH 64733)

• DataFrame.copy() and Series.copy() with deep=True (the default), and ArrowExtensionArray.copy(), now copy pyarrow-backed data instead of sharing its memory, so a copy no longer keeps the original’s memory alive (GH 61930)

• DataFrame.query() now raises a ValueError when the expression does not evaluate to a boolean array, instead of silently using the result as row labels; use DataFrame.loc for label selection (GH 8560)

Deprecations

deprecation1

deprecation2

Other deprecations

• Deprecated merge() and DataFrame.join() producing duplicated columns when suffixed left and right column labels collide, e.g. suffixes=("_x", "_x"); this will raise a MergeError in a future version (GH 13659)

• Deprecated median, min, max, idxmin, idxmax, argmin, argmax, cummin, cummax, clip, and the comparisons <, <=, >, >= on complex data; complex numbers have no natural ordering, so in a future version these will raise TypeError (GH 43770, GH 70087)

• Deprecated pandas.to_pickle, use the DataFrame.to_pickle() or Series.to_pickle() method instead (GH 48402)

• Deprecated passing None entries in objs to concat(); filter them out before calling concat() instead (GH 57846)

• Deprecated passing a negative min_count to DataFrame.sum(), DataFrame.prod(), Series.sum(), Series.prod(), TimedeltaIndex.sum(), the sum and prod methods of pandas arrays, and the sum, prod, min, max, first, and last methods of DataFrameGroupBy, SeriesGroupBy, and Resampler. Pass min_count=0 instead, which behaves the same (GH 50022)

• Deprecated setting an out-of-bounds numpy scalar into an array of a numpy-nullable integer dtype. The value is currently stored altered to fit (np.int64(-1) becomes 255 in UInt8); in a future version this will raise, as it already does for a Python integer. Cast the array to a dtype that can hold the value first (GH 48867)

• Deprecated the DataFrame constructor, DataFrame.from_dict() with orient="index", and DataFrame.from_records() splitting a str or bytes row into its elements when the first row is list-like; in a future version this will raise TypeError. Pass list(row) instead (GH 50461)

• Deprecated the cache_dates argument in read_csv(), read_table(), and read_fwf() (GH 68705)

• Deprecated the columns keyword in Series.drop(), which was silently ignored; use index or labels instead (GH 39509)

• Deprecated constructing a DataFrame column, Index or array() from a NumPy structured (compound) array, and casting to a compound dtype with DataFrame.astype(), Series.astype() or array(); these will raise in a future version, as the Series constructor already does. Use pd.DataFrame(arr) for one column per field or arr.astype(object) for tuples (GH 4464, GH 49388, GH 60108)

Performance improvements

• Performance improvement in read_csv() with engine="c" across the board, in tokenizing and in the machinery shared by all column types; large uncompressed local files are additionally read in parallel across multiple CPU cores by default, see above (GH 64347, GH 64515, GH 65350, GH 66271, GH 66272, GH 66273, GH 66274, GH 66276, GH 66798, GH 68314, GH 69632, GH 69748, GH 70140)

• Performance improvement in read_csv() with a dict dtype on wide files or with a small chunksize (GH 70769)

• Performance improvement in DataFrame.skew() and DataFrame.kurt() with types different from float64 (GH 66662)

• Performance improvement in DataFrameGroupBy.pct_change() and SeriesGroupBy.pct_change() (GH 70972)

• Performance improvement in PeriodIndex.strftime() with a custom date_format (GH 70106)

• Performance improvement in Series.var(), Series.std(), Series.sem(), and their DataFrame equivalents when bottleneck is disabled (GH 64884)

• Reduced memory usage of DataFrame.loc() and DataFrame.iloc() when selecting a single row or a small slice of rows together with a list of columns (GH 69477)

Bug fixes

Categorical

• Bug in DataFrame.values, DataFrame.to_numpy() and DataFrame.transpose() silently converting missing values in a Categorical column with integer or boolean categories to arbitrary integers or True (GH 38240)

• Bug in Series.clip() and DataFrame.clip() with ordered categorical data ordering lower and upper by value instead of by category position (GH 49217)

• Bug in comparing a Categorical with an iterator raising TypeError instead of treating it as a scalar, as with a tuple (GH 31646)

• Bug in comparing unordered Categorical objects with mixed-type categories in a different order raising TypeError; Categorical.equals() and merge() were also affected (GH 51543)

Datetimelike

• Bug in DataFrame.astype(), Series.astype(), Index.astype(), DatetimeIndex, array() and to_datetime() where float dtype data outside the int64 range, such as inf, silently became a valid-looking datetime64/timedelta64 value or NaT; these now raise OutOfBoundsDatetime/OutOfBoundsTimedelta (GH 68926)

• Bug in DatetimeIndex.strftime(), Series.dt.strftime(), and DataFrame.to_csv() with date_format on Windows silently replacing values that Timestamp.strftime() cannot format (e.g. %y for years before 1900) with their default string representation; these now raise ValueError (GH 58178)

• Bug in Series.isin(), DataFrame.isin(), and Index.isin() on datetime-like data missing numpy.datetime64 or numpy.timedelta64 values (NaT, or any value on numpy < 2.2) when values also held non-datetime-like entries (GH 71186)

• Bug in != between a Tick offset such as Hour and an array-like of timedelta64 values or pd.NA raising ValueError or TypeError (GH 71162)

• Bug in comparisons, indexing, setitem and Series.astype() to CategoricalDtype with datetime, timedelta or period values not parsing ArrowDtype string and large_string values the way StringDtype values are parsed (GH 62051)

Timedelta

Timezones

• Bug in DataFrame.query() and DataFrame.eval() comparing a timezone-aware datetime column to a string without a UTC offset raising TypeError or returning incorrect results; the string is now localized to the column’s timezone, as in df["col"] < "2000-01-02" (GH 26236)

Numeric

• Bug in Series.skew() and Series.kurt() (and their DataFrame counterparts) on complex data discarding the imaginary part; these now raise TypeError (GH 43770)

• Bug in Series.var() and Series.std() (and their DataFrame counterparts) returning very inaccurate results for large float32 data when bottleneck is installed (GH 70945)

• Bug in integer exponentiation (**, Series.pow(), DataFrame.pow()) on large inputs returning imprecise results, 0 for negative exponents, or an upcast dtype when numexpr is installed (GH 54546)

• Fixed bug in Series.corr() with method="pearson" returning an inaccurate result for nearly-constant data, and 0.0 instead of NaN for some constant data (GH 59652)

• Fixed bug in Series.isin(), DataFrame.isin(), and Index.isin() reporting false matches when values was a list of integers above 2**53 mixed with floats, NaN, or integers outside the int64 range, or a list of bytes ending in a null byte (GH 70217)

• Fixed bug in Series.var(), Series.std(), and Series.sem() (and their DataFrame counterparts) returning inf or incorrect results for float16 dtypes when the variance or the number of values exceeded the float16 range; these now return float64 like Series.mean() (GH 41277)

Conversion

• Bug in DataFrame constructor and DataFrame.from_records() where a list of Series with extension dtypes lost those dtypes (GH 56231)

• Bug in to_numeric() with a datetime-like scalar not matching the result for a Series holding the value (GH 43280)

• Bug in DataFrame.compare() and Series.compare() upcasting integer and boolean columns when the equal values were only in rows dropped from the result (GH 39899)

• Bug in DataFrame.from_records() not dropping the exclude and index columns when all of their labels were falsy, such as 0 (GH 18679)

• Bug in DataFrame.replace() with a dict or list replacing values with None casting other columns of the same dtype to object even when they contained no value to replace (GH 30512)

• Bug in DataFrame.to_dict() and Series.to_dict() sometimes returning NumPy complex scalars instead of Python complex (GH 27616)

• Bug in Series.astype() and array() raising TypeError when converting object-dtype data mixing integers and integer strings (e.g. [1, "1"]) to a nullable integer dtype (GH 34460)

• Fixed bug in to_numeric() with errors="coerce" raising TypeError for object values containing unhashable nested elements, such as a tuple containing a list, instead of coercing them to NaN (GH 69916)

Strings

• Bug in StringDtype with PyArrow storage rejecting a PyArrow string scalar in item assignment and Series.fillna(), and Series.where() with one storing a character code (GH 71075)

• Bug in DataFrame.replace() and Series.replace() silently ignoring a compiled re.Pattern for ArrowDtype string and large_string dtypes, including a Categorical of them (GH 69026)

• Bug in DataFrame.replace() and Series.replace() with regex=True on ArrowDtype string and large_string dtypes raising or silently coercing a replacement the column cannot hold instead of casting to object (GH 70458)

Interval

• Bug in IntervalIndex.mid and arrays.IntervalArray.mid returning incorrect values for numeric endpoints whose sum overflows their dtype (GH 25499)

• Bug in IntervalIndex set operations with an empty RangeIndex, and in IntervalIndex construction or astype("interval") from a RangeIndex or non-interval extension array, raising an uninformative TypeError (GH 68343)

• Bug in concat() and Index.append() with IntervalDtype not retaining an extension dtype subtype, either casting it to a NumPy dtype or raising TypeError (GH 64297)

• Bug in interval_range() with large integer start or end silently returning too few intervals, or empty or wrongly spaced intervals, due to float precision loss (GH 71288)

• Bug in Series.shift() and DataFrame.shift() with an IntervalDtype column not retaining the subtype, either changing it or raising TypeError (GH 69922)

Indexing

• Bug in DataFrame.__setitem__() silently corrupting the DataFrame when the key matched several columns and value was a timezone-aware datetime64 or period scalar, or a one-dimensional Series of either dtype; it now broadcasts across the matched columns as the other dtypes already did (GH 71078)

• Bug in DataFrame.__setitem__() silently corrupting the DataFrame, or dropping columns of value, when the key matched several columns (duplicate or MultiIndex columns) and value was a two-dimensional array without one column per matched column; it now raises ValueError (GH 68445)

• Bug in DataFrame.isetitem() with several positions silently corrupting the DataFrame, or dropping columns of value, when an arraylike value did not have one column per position; it now raises ValueError (GH 68445)

• Bug in DataFrame.loc() and DataFrame.at() setitem with a single row label adding a new column from a list-like value raising or setting the wrong value (GH 30649)

• Bug in DataFrame.loc() and DataFrame.iloc() raising or flattening a 2D value (e.g. a list of lists) set into a single object cell of a mixed-dtype DataFrame (GH 69152)

• Bug in DataFrame.loc() and DataFrame.iloc() setitem on a 1-column DataFrame with an extension dtype raising when the column key is an all-False boolean mask or an empty list-like (GH 70232)

• Bug in DataFrame.loc() and Series.loc() raising KeyError for a True or False label present in an object-dtype Index or the first level of a MultiIndex (GH 50165)

• Bug in DataFrame.loc() and Series.loc() setitem with an integer slice raising TypeError where the same .loc getitem succeeds, e.g. on an index of decimal.Decimal values (GH 70219)

• Bug in DataFrame.loc() and Series.loc() slicing with an integer or float bound that is not in the index: an object-dtype index of numbers (e.g. decimal.Decimal) raised TypeError instead of slicing by sort order, and a non-monotonic index of non-string values raised TypeError instead of KeyError (GH 70295)

• Bug in Index.difference() dropping elements that are not in other when other was a DatetimeIndex, TimedeltaIndex or PeriodIndex and the elements were strings that parse to its values, or when other was an IntervalIndex and the elements were scalars contained in its intervals (GH 58971)

Missing

• Bug in DataFrame.ffill() and DataFrame.bfill() with axis=1 and inplace=True not modifying the DataFrame and returning a new object instead of self (GH 71159)

• Bug in DataFrame.interpolate() and Series.interpolate() with method="index", "values", "time" or a SciPy method giving incorrect results for a nanosecond-resolution DatetimeIndex or TimedeltaIndex spanning less than about 104 days, or an integer index with values beyond 2**53 spanning less than 2**53 (GH 34601)

• Bug in DataFrame.interpolate() and Series.interpolate() with a large limit raising on 32-bit platforms and being very slow on long runs of missing values (GH 34936)

• Bug in DataFrame.mask(), DataFrame.where(), Series.mask(), and Series.where() handling missing values in a nullable boolean cond inconsistently between the two methods and depending on inplace; missing values are now treated as False, as in boolean indexing (GH 35429, GH 52955, GH 56844, GH 60729)

MultiIndex

• Bug in MultiIndex constructor and MultiIndex.set_codes() silently converting NaN or non-integer float codes to valid codes instead of raising ValueError (GH 26210)

• Bug in DataFrame.sort_index(), Series.sort_index(), MultiIndex.sort_values() and MultiIndex.is_monotonic_increasing() ignoring the category order of a CategoricalIndex level (GH 47607)

• Bug in MultiIndex.append(), concat(), DataFrame.join() and merge() emitting a spurious RuntimeWarning when MultiIndex level values could not be sorted (GH 61477)

• Bug in MultiIndex.from_tuples() with an object ndarray, MultiIndex.append() and Index.map() truncating or raising IndexError on tuples of different lengths; MultiIndex.from_tuples() now raises a ValueError for an ndarray of tuples of different lengths (GH 50293)

• Bug in MultiIndex.rename() raising TypeError for the name keyword documented by Index.rename() (GH 33778)

• Bug in Series.map(), Index.get_indexer(), DataFrame.reindex() and DataFrame.loc with a MultiIndex raising IndexError or matching on a prefix when looking up tuples whose length differs from the number of levels (GH 50293)

I/O

• read_csv() with engine="pyarrow" now raises ValueError for the unsupported encoding_errors with bytes or file input instead of silently ignoring it (GH 71298)

• Bug in DataFrame repr and DataFrame.to_string() where numeric column labels of differing widths were misaligned with their column’s values (GH 8300)

• Bug in read_csv() silently misparsing non-ASCII thousands and decimal markers with the C engine; these now fall back to the Python engine with a errors.ParserWarning, or raise ValueError if engine="c" is explicitly specified (GH 71158)

• Bug in read_csv() where a dtype given for a parse_dates column undid the date parsing, returning strings, objects, integers or categories instead of datetimes (GH 57512)

• Bug in read_csv() with engine="c" and comment not skipping a line containing only whitespace before the comment character (GH 55916)

• Bug in read_csv() with engine="pyarrow" silently wrapping values outside the range of the requested integer dtype instead of raising (GH 38013)

• Bug in read_csv() with engine="python" and index_col reading a date like "20180101" as nanoseconds since the epoch when dtype is a pyarrow timestamp dtype (GH 24542)

• Bug in read_csv(), read_excel(), read_fwf() and read_html() returning wrong column labels and treating header rows as data when header was a list not in increasing order, such as header=[1, 0] (GH 47011)

• Bug in read_csv(), DataFrame.to_csv() and other I/O functions raising URLError for sftp:// URLs instead of reading or writing them with fsspec (GH 46765)

• Bug in read_excel() raising ValueError when header is a list and index_col consumes all columns (GH 66372)

• Bug in read_html() raising ImportError when lxml is not installed instead of falling back to bs4 + html5lib with the default flavor=None (GH 30281)

• Bug in read_html() with the bs4 or html5lib parser dropping tables that have identical content, so only the first was returned (GH 49934)

• Bug in read_orc() raising whenever filesystem was passed (GH 58746)

• Bug in read_sas() exhausting memory instead of raising ValueError on a SAS7BDAT file that claims more rows than it can hold, when read from an uncompressed file, a zip archive, or a BytesIO (GH 70721)

• Bug in read_sas() returning rows that were deleted from an uncompressed SAS7BDAT file (GH 15963)

• Bug in read_sas() with format="xport" reading zeros as 5.397605e-79 (GH 50670)

• Bug in DataFrame.to_csv() with MultiIndex columns and an unnamed index or index_label=False where a first row of missing values was read back by read_csv() as the index names and dropped; an empty index-names line is now written in this case (GH 21995)

• Bug in DataFrame.to_csv(), DataFrame.to_json(), DataFrame.to_pickle() and similar writers where writing to a file:// URL raised for a new file and silently left an existing file unchanged; writing to other URLs such as http:// now raises instead of silently discarding the data (GH 55828)

• Bug in DataFrame.to_excel() where a NaN label in MultiIndex columns was written as the last value of that level instead of an empty cell (GH 62340)

• Bug in DataFrame.to_json() and Series.to_json() dropping the nanoseconds of tz-aware datetime columns and Timestamp objects, and with date_format="epoch" mis-rounding pre-1970 datetime objects or raising OverflowError for ones outside the nanosecond bounds at a coarser date_unit (GH 53473)

• Bug in DataFrame.to_parquet() with engine="pyarrow" where a misspelled keyword argument truncated an existing file at the destination or was reported as an error about the path (GH 45815)

• Bug in DataFrame.to_sql() creating MySQL and MariaDB datetime and time columns that silently dropped fractional seconds (GH 48435)

• Bug in DataFrame.to_sql() with if_exists="append" or if_exists="delete_rows" writing to an existing table hanging on SQL Server while another session had uncommitted changes to an unrelated table (GH 52601)

• Bug in DataFrame.to_sql() with if_exists="delete_rows" raising InvalidRequestError for an existing temporary table (GH 52601)

• Bug in DataFrame.to_stata() raising ValueError for an empty DataFrame with a categorical column, and read_stata() not converting the column back to a categorical (GH 71126)

• Bug in DataFrame.to_string(), Series.to_string() and DataFrame.to_html() ignoring na_rep for None, pd.NA, and NaT values in object-dtype columns and an explicit na_rep="NaN" for any missing value, and in DataFrame.to_xml() raising for na_rep with nullable, pyarrow-backed, or categorical columns containing missing values (GH 54872)

• Bug in DataFrame.to_string(), Series.to_string() and their repr where values or labels containing ANSI color codes were counted toward the column width, misaligning the other rows (GH 55122)

• Bug in displaying a Series or DataFrame in IPython with display.html.table_schema enabled printing a traceback when the data cannot be converted to JSON, such as non-UTF-8 bytes (GH 16848)

• Bug in the display of a MultiIndex (repr, DataFrame.to_string(), DataFrame.to_html(), Styler) where distinct labels that display identically (e.g. 1 and "1") were shown as a single label; DataFrame.to_excel() merged them into one cell, losing all but the first value (GH 10796)

• Bug in the repr of a Series or DataFrame with longdouble (e.g. float128) dtype showing values outside the float64 range as 0 or inf (GH 17809)

• Fixed bug in read_csv() where callable skiprows received incorrect row numbers on 32-bit big-endian architectures (GH 69985)

• Bug in ExcelWriter with engine="openpyxl" ignoring date_format and datetime_format (GH 44284, GH 59882)

• DataFrame.to_json() and Series.to_json() with orient="table", and build_table_schema(), no longer raise AttributeError for a MultiIndex with non-string level names (GH 19129)

• Bug in read_csv() where using engine="python" with custom converters returning unhashable types (such as lists or dicts) raised TypeError (GH 56848)

• Bug in read_json() with typ="series" returning datetime64 dtype for all-null data instead of float64, matching DataFrame columns (GH 35464)

• Bug in read_sas() raising IndexError or reading the wrong format or label for SAS7BDAT files with columns renamed or relabeled after the file was written (GH 60809)

• Bug in read_sql_query() and read_sql_table() with a SQLAlchemy connection returning column labels that require sqlalchemy to unpickle (GH 52816)

• Bug in HDFStore.append() raising ValueError: Buffer dtype mismatch instead of the “different categories” error when appending a categorical whose categories have a different dtype than the existing ones (GH 71128)

• Bug in HDFStore.remove() deleting unselected rows when passing unsorted row coordinates (GH 71276)

• Bug in Series.to_json() with orient="table" writing a falsy Series name such as 0, False or "" as "values" (GH 19129)

• Bug where the pickled bytes (e.g. for joblib.hash) of a Series, DataFrame or Index with datetimelike or extension-dtype data depended on which operations had previously been run on it (GH 49170)

• Fixed a segfault in DataFrame.to_json() and Series.to_json() with date_format="iso" when datetime conversion raises an exception (GH 69461)

Period

• Bug in PeriodIndex and array() with a period dtype raising when given a list of nanosecond-unit np.datetime64 values, and PeriodIndex raising AttributeError instead of ValueError for datetime64 data without a freq (GH 70139)

• Bug in Period constructor where the string of a weekly Period such as "2011-12-26/2012-01-01" or "2000-01-03/2000-01-09" was silently parsed as a minute-frequency Period (GH 48000)

• Bug in Period, PeriodIndex, period_range() and casting to PeriodDtype silently dropping the timezone of timezone-aware input; these now emit a UserWarning like Timestamp.to_period() (GH 47005)

• Bug in Period.strftime() and PeriodIndex.strftime() where %Y was not zero-padded for years before 1000 on Linux (GH 48746)

Plotting

• Bug in plotting.table() ignoring rowLabels and colLabels or raising a TypeError when they were passed (GH 37811)

• Bug in DataFrame.hist() with by and no column plotting the numeric grouping column alongside the other columns (GH 41188)

• Bug in DataFrame.plot() and Series.plot() with kind="bar" or kind="barh" raising TypeError on timedelta64 data containing NaT (GH 39320)

• Bug in DataFrame.plot() and Series.plot() with a TimedeltaIndex that has no regular frequency, or with x_compat=True, plotting the points evenly spaced and leaving most x-axis ticks unlabeled (GH 19965)

• Bug in DataFrame.plot() and Series.plot() with a regular-frequency DatetimeIndex or PeriodIndex mislabeling ticks set with xticks= or ax.set_xticks (GH 18881)

• Bug in Series.plot() and DataFrame.plot() with a timezone-aware index raising ValueError when the first timestamp falls in a daylight saving time transition, e.g. the repeated hour of a fall-back day (GH 62936)

Groupby/resample/rolling

• Bug in DataFrameGroupBy.cumsum(), DataFrameGroupBy.cummin(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.rank() (and Series variants) on datetime64, timedelta64, and PeriodDtype data treating NaT as a regular value and losing precision when the group key contains NA (GH 69923)

• Bug in DataFrameGroupBy.cumsum(), DataFrameGroupBy.cumprod(), DataFrameGroupBy.cummin(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.rank() raising NotImplementedError, often with no message, instead of TypeError on columns that do not support them, e.g. object columns and ArrowDtype date, time and binary columns (GH 69791)

• Bug in DataFrameGroupBy.max(), DataFrameGroupBy.min(), DataFrameGroupBy.cummax(), and DataFrameGroupBy.cummin() (and Series variants) on nullable float data with the future.distinguish_nan_and_na option enabled returning inf or -inf for all-NaN groups and ignoring NaN in other groups; NaN now propagates as in DataFrameGroupBy.sum(); under the default option, ArrowDtype float data containing NaN now gives NA instead of ignoring the NaN (GH 59106)

• Bug in DataFrameGroupBy.resample() followed by Resampler.aggregate() with a list of functions aggregating the grouping and on columns, raising TypeError or returning them as extra result columns (GH 32162)

• Bug in DataFrameGroupBy.shift(), SeriesGroupBy.shift(), DataFrameGroupBy.diff(), and SeriesGroupBy.diff() with periods=0 not setting rows with an NA group key to missing (GH 70996)

• Bug in DataFrameGroupBy.transform() with a function returning a scalar for each column casting the results to a common dtype (GH 42617)

• Bug in GroupBy.median(), GroupBy.skew(), and GroupBy.kurt() on complex data silently discarding the imaginary part; these now raise TypeError (GH 43770)

• Bug in GroupBy.quantile() raising NotImplementedError with no message, instead of TypeError, on ArrowDtype columns it does not support, e.g. binary columns (GH 69808)

• Bug in GroupBy.std() and GroupBy.sem() on complex data silently discarding the imaginary part, and GroupBy.var() raising; these now match Series.std(), Series.sem(), and Series.var() (GH 43770)

• Bug in SeriesGroupBy.transform() with a callable raising for Series subclasses that cannot hold the transformed values; the result now falls back to the type the subclass chooses (geopandas/geopandas#3059, GH 71165)

• Bug in DataFrame.agg(), DataFrameGroupBy.agg(), and resample/rolling agg with a dict, list, or named aggregation over more than 100 columns emitting a spurious PerformanceWarning with as_index=False or on a later DataFrame.reset_index() (GH 61628)

• Bug in DataFrame.groupby() and Series.groupby() with sort=True and dropna=False placing the NaN group in the middle of the result instead of last for object-dtype keys; Index.union() had the same misordering (GH 70216)

• Bug in DataFrame.groupby(), DataFrame.resample(), DataFrame.rolling(), DataFrame.expanding(), and DataFrame.ewm() raising TypeError when selecting MultiIndex columns with a tuple key (GH 13914)

• Bug in Series.resample() and DataFrame.resample() with a PeriodIndex and closed="right" returning shifted labels or incorrect aggregations (GH 44363)

• Bug in time-based Rolling.cov() and Rolling.corr() raising ValueError or silently returning wrong results when other has a different index; the result is now computed on the aligned index, as with integer windows. With on, a mismatched index now raises ValueError (GH 62937)

Reshaping

• Bug in concat() and the DataFrame constructor losing the freq of a DatetimeIndex or TimedeltaIndex depending on the order of the inputs (GH 64253)

• Bug in merge_asof() with right_index=True and an empty left returning the index of right instead of left (GH 41171)

• Bug in merge() raising ValueError when merging a numeric key with an object key that is not all strings, e.g. [1.5, None], [1.5, "a"], or an empty or all-missing key (GH 70218)

• Bug in DataFrame.pivot() and pivot() with a list of values casting the pivoted columns to a common dtype, e.g. integers to float64 or object (GH 43547)

• Bug in DataFrame.stack() where missing labels in the result’s MultiIndex could not be found by lookups such as .loc; unstacking such a result now places the missing label first unless sort=False is passed (GH 40366)

Sparse

• Bug in setting integer SparseDtype values into an integer Series or Index (__setitem__, .loc, .iloc, Series.where(), Series.mask(), Series.update(), Index.putmask()) raising AttributeError: 'SparseDtype' object has no attribute 'itemsize' (GH 68457)

ExtensionArray

• Bug in read_csv() and other parsing of strings into a decimal ArrowDtype raising ArrowInvalid instead of parsing the values (GH 69838)

• Bug in ArrowExtensionArray.factorize() ignoring use_na_sentinel=True for dictionary-encoded PyArrow data that stores nulls as dictionary values, keeping the null in the returned uniques or raising ArrowInvalid for chunked data (GH 66490)

• Bug in Series.isin(), DataFrame.isin(), and Index.isin() on PyArrow-backed dtypes raising when values held a mix of types PyArrow cannot combine (GH 70300)

• Bug in Series.mode() and DataFrame.mode() with dropna=False raising TypeError for "boolean" dtype (GH 70487)

• Bug in Series.replace(), Series.where(), Series.fillna(), item assignment and related methods on a PyArrow-backed integer dtype silently truncating a fractional value, including one passed in a list (replace(2, 1.5) stored 1); these now raise TypeError, while Index.where(), Index.putmask() and Index.fillna() upcast, matching the nullable integer dtypes (GH 68638, GH 71076)

• Bug in arithmetic and logical operations on PyArrow-backed data raising pyarrow.ArrowInvalid or pyarrow.ArrowTypeError instead of TypeError when the other operand contains objects PyArrow cannot convert (GH 62682)

• Bug in comparisons between PyArrow-backed data and objects PyArrow cannot convert raising pyarrow.ArrowInvalid or pyarrow.ArrowTypeError; ==/!= now compare elementwise, ordered comparisons raise TypeError, and a missing value on either side propagates instead of answering False (GH 62682)

Styler

• Bug in Styler methods taking a level argument, such as Styler.hide() and Styler.format_index(), ignoring or raising on negative, out-of-range or NumPy integer levels; level is now resolved as in DataFrame.droplevel(), so an integer matching a level name refers to that level (GH 42933)

• Bug in Styler.background_gradient() and Styler.text_gradient() coloring missing values with the lowest colormap color instead of the missing-value color when all valid values in a column, row, or table were equal (GH 47695)

• Bug in Styler.bar() raising TypeError for data containing pd.NA when align is "mean" or a callable, or when cmap is given (GH 56425)

• Bug in Styler.map(), Styler.apply() and Styler.set_table_styles() truncating CSS values that contain a ; inside quotes or parentheses, such as url("data:image/png;base64,...") (GH 59623)

• Bug in Styler.to_excel() applying styles to the wrong columns when columns selects or reorders columns (GH 42276)

• Bug in Styler.to_html() where Chrome and other Chromium-based browsers did not apply a style shared by more than 8192 cells to the cells beyond the first 8192 (GH 40913)

Other

• Bug in api.interchange.from_dataframe() with allow_copy=False always raising RuntimeError for objects supporting the Arrow PyCapsule Interface, even when a zero-copy conversion is possible (GH 71189)

• Bug in eval() with a multi-line expression and a target raising UndefinedVariableError when a line after the first referenced a variable (GH 31952)

• Bug in eval(), DataFrame.eval() and DataFrame.query() where not performed a bitwise inversion instead of a logical negation, e.g. pd.eval("not True") returned -2 and df.query("a > 1 and not True") could return rows (GH 11367)

• Bug in testing.assert_series_equal() and testing.assert_frame_equal() with check_dtype=False and check_exact=True comparing missing values against an object column differently from check_exact=False; pd.NA and NaT no longer match np.nan or None (GH 61473)

• Bug in util.hash_array() and util.hash_pandas_object() raising UnicodeDecodeError on mixed-type object values containing non-ASCII bytes; bytes in mixed-type values now hash the same as in non-mixed ones (GH 27215)

• Bug in DataFrame.agg(), Series.agg() and the agg methods of groupby, resample and window objects silently ignoring *args and **kwargs when func is a dict, e.g. df.agg({"a": "sum"}, min_count=1); they are now passed to each function, as with a list of functions, so a function that does not accept them now raises TypeError (GH 23889)

• Bug in DataFrame.duplicated() and DataFrame.drop_duplicates() raising ValueError when subset is an empty list; no rows are now marked as duplicates, matching a DataFrame with no columns (GH 12869)

• Bug in DataFrame.eval() and DataFrame.query() with engine="numexpr" where a column used again in a later line of a multi-line expression, or after a subscript such as a[a > 0], was treated as a NumPy array, raising TypeError or silently giving wrong results; eval() also no longer replaces the values of a passed resolvers mapping with NumPy arrays (GH 59062)

• Bug in DataFrame.eval(), DataFrame.query(), and eval() with the default engine="numexpr" raising RecursionError for arithmetic between object or string columns (e.g. "a + b" on two string columns) and ValueError for an expression that is a single object or string column (GH 21088, GH 27639)

• Bug in DataFrame.from_dict() with orient="index" silently setting a row’s values to NaN when its tuple key contained None or NaN (GH 28390)

• Bug in DataFrame.query() and DataFrame.eval() where == and != against a string gave different results than the same comparison outside of query, e.g. returning no rows for a datetime64, timedelta64, or PeriodDtype column, or raising NotImplementedError with parser="python" (GH 54199)

• Bug in DataFrame.query() and DataFrame.eval() where a column named inf or Inf was ignored in favor of np.inf, raising KeyError or silently returning wrong results (GH 47859)

• Bug in DataFrame.query() and DataFrame.eval() with engine="numexpr" raising AttributeError for some comparisons and in/not in checks whose operand is a math function or unary operator, e.g. abs(a) in [1, 2] (GH 31848)

• Bug in calling a generalized ufunc on a Series raising AttributeError when it reduced object data to a single object pandas does not recognize as a scalar, e.g. shapely.geometrycollections returning a geometry (GH 71109)

• Bug in eval(), DataFrame.eval(), and DataFrame.query() raising on some datetime64 and timedelta64 operations, e.g. df.query("time > @now - @delta"), and returning float32 for a float32 column times a np.timedelta64 (GH 30729, GH 52559)

• Bug in DataFrame.replace() and Series.replace() with inplace=True and a list-like or dict to_replace using memory proportional to the number of values to replace times the length of the data, raising MemoryError for large mappings (GH 25816)

Contributors

A total of 61 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

• ASWANTH R +

• Aahan G +

• Aarya +

• Aarya Prakash +

• Aarya2004 +

• Abhishek Kumar +

• Aidan Feldman

• Ammar Hassona +

• Aniruktha +

• Anushka Dudhe +

• Atik +

• Ayush Kumar +

• Ayush Srivastava +

• Breno Barreto +

• Chaimaa Abi +

• DeviousCardi +

• Fangchen Li

• Furqan Khan +

• GON +

• Gore Williams +

• H23 +

• Ishaan

• Itssubhraneelbruv +

• Johnny Kao +

• Johnny Wilson Dougherty +

• Joris Van den Bossche

• Juan Miguel Contreras +

• Julian Harbeck

• Leo Camus +

• Likhitha J +

• Maha Rana Yadavalli +

• Maha Yadavalli +

• Mannava Daasaradhi +

• Manvi Soni +

• Matthew Roeschke

• Minh Tuan Pham +

• Mohd Maaz Khan +

• N3verm1nd

• Natalia Mokeeva

• Odysseas Yiakoumis +

• Parman Mohammadalizadeh +

• Richard Shadrach

• Ryan George +

• S.A +

• Saihritesh

• SamuelOndrus +

• Sugandh Garg +

• TensorDevLJ +

• Tiereonna +

• Trinh Quoc Anh

• Trupti603 +

• Vincent Ngo +

• Weijian He +

• britz +

• furqankhan +

• guptaishaan

• jbrockmendel

• maharanay22 +

• ump45nose

• yuchen-xu +

• Álvaro Kothe