BUG: NaN vs NA in mixed reduction

[jbrockmendel]

df = pd.DataFrame(
    {
        "B": [1, None, 3],
        "C": pd.array([1, None, 3], dtype="Int64"),
    }
)
result = df.skew()

>>> result
B    <NA>
C    <NA>
dtype: Float64

>>> df[["B"]].skew()
B   NaN
dtype: float64

Based on test_mixed_reductions. The presence of column "C" shouldn't affect the result we get for column "B".

[rhshadrach]

Expected Behavior

NA

I would expect Float64 dtype with NaN and NA. One might also argue object, but so far it appears we coerce to the nullable dtypes.

df = pd.DataFrame(
    {
        "B": [1, 2, 3],
        "C": pd.array([1, 2, 3], dtype="Float64"),
    }
)
print(df.sum())
# B    6.0
# C    6.0
# dtype: Float64

[jbrockmendel]

Hah, in the expected behavior section i put "NA" to mean "I'm not writing anything here", not pd.NA.

[jorisvandenbossche]

I would say that the current behaviour is fine. We indeed currently coerce to the nullable dtype for the result if there are mixed nullable and non-nullable columns, and at that point converting NaN to NA seems the correct thing to do (if you would first cast the original non-nullable column to its nullable dtype, you would get the same result)

[jbrockmendel]

I would say that the current behaviour is fine

It's fine in a never-distinguish world. We're currently in a sometimes-distinguish world in which I had previously thought the rule was "distinguish when NaNs are introduced through operations". We can make that rule more complicated "distinguish when NaNs are introduced through operations but not reductions", but I do think that makes it a worse rule.

if you would first cast the original non-nullable column to its nullable dtype

In this case the user specifically didn't do that.

[rhshadrach]

We indeed currently coerce to the nullable dtype for the result if there are mixed nullable and non-nullable columns

Should coercion to Float64 change NaN to NA?

ser = pd.Series([np.nan, pd.NA], dtype="object")
print(ser.astype("Float64"))
# 0    <NA>
# 1    <NA>
# dtype: Float64

[jbrockmendel]

Should coercion to Float64 change NaN to NA?

That is expected behavior ATM, yes. In an always-distinguish world it would not (xref #62040)