41 Pull requests opened by 27 people

• [SPARK-52996][TESTS] Update brace-expansion to 1.1.12

  #51703 opened Jul 29, 2025

• [SPARK-52998][Core] Multiple variables inside declare

  #51705 opened Jul 29, 2025

• [SPARK-53015][BUILD] Upgrade log4j to 2.25.1

  #51719 opened Jul 30, 2025

• [SPARK-53019][SQL] Fix job attempt path conflicts in o.a.hadoop..FileOutputCommitter

  #51724 opened Jul 30, 2025

• [SPARK-53022][TESTS] Add MemoryConsumerBenchmark

  #51728 opened Jul 30, 2025

• [SPARK-53038][SQL][HIVE] Call initialize only once per GenericUDF instance

  #51743 opened Jul 31, 2025

• [SPARK-52844][PYTHON] Update protobuf to 5.29.5

  #51747 opened Jul 31, 2025

• [SPARK-53044] Change Declarative Pipelines import alias convention from "sdp" to "dp"

  #51752 opened Jul 31, 2025

• Wip naming sources

  #51756 opened Jul 31, 2025

• [SPARK-53030][PYTHON] Support Arrow writer for streaming Python data sources

  #51757 opened Jul 31, 2025

• [SPARK-42360][SQL] Rule to convert Left Outer Join with suitable filter to Left Anti Join

  #51762 opened Aug 1, 2025

• [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders

  #51768 opened Aug 1, 2025

• [SPARK-52844][PYTHON] Update mlflow to 3.1.0

  #51774 opened Aug 1, 2025

• [SPARK-53064][CORE] Rewrite MDC LogKey in Java

  #51775 opened Aug 1, 2025

• [SPARK-53067][BUILD] Increase GCLockerRetryAllocationCount for SBT

  #51780 opened Aug 1, 2025

• [SPARK-53066][SQL] Improve EXPLAIN output for DSv2 Join pushdown

  #51781 opened Aug 1, 2025

• [SPARK-53069][SS] Fix incorrect state store metrics with virtual column families

  #51790 opened Aug 2, 2025

• [SPARK-53084][CORE] Supplement default GC options in SparkContext initialization

  #51796 opened Aug 3, 2025

• Fix invalid exit codes and enhance CLI validation tools

  #51797 opened Aug 3, 2025

• Just test

  #51798 opened Aug 3, 2025

• [SPARK-53094][SQL] Fix cube-related data quality problem

  #51810 opened Aug 4, 2025

• [SPARK-53098][SQL] `DeduplicateRelations` shouldn't remap expressions if old `ExprId` still exists in output

  #51812 opened Aug 4, 2025

• [SPARK-53097][CONNECT] Make WriteOperationV2 in SparkConnectPlanner side effect free

  #51813 opened Aug 4, 2025

• [SPARK-53103][SS] Throw an error if state directory is not empty on batch 0

  #51817 opened Aug 4, 2025

• [SPARK-53074][SQL] Avoid partial clustering in SPJ to meet a child's required distribution

  #51818 opened Aug 4, 2025

• [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses

  #51820 opened Aug 4, 2025

• [SPARK-53104][PS] Introduce ansi_mode_context to avoid multiple config checks per API call

  #51821 opened Aug 4, 2025

• [SPARK-53106] Add schema evolution tests for TWS Scala spark connect suites

  #51822 opened Aug 4, 2025

• [SPARK-53107][SQL] Implement the time_trunc function in Scala

  #51823 opened Aug 4, 2025

• [SPARK-53113][SQL] Support the time type by try_make_timestamp()

  #51824 opened Aug 4, 2025

• [SPARK-53110][SQL][PYTHON][CONNECT] Implement the time_trunc function in PySpark

  #51825 opened Aug 4, 2025

• [SPARK-53108][SQL] Implement the time_diff function in Scala

  #51826 opened Aug 4, 2025

• [SPARK-51462][SPARK-51564][SQL][FOLLOWUP][TESTS] Add E2E SQL tests for TIME literals in 24hr and 12hr formats

  #51827 opened Aug 4, 2025

• [SPARK-53109][SQL] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in Scala

  #51828 opened Aug 4, 2025

• [SPARK-53111][SQL][PYTHON][CONNECT] Implement the time_diff function in PySpark

  #51829 opened Aug 4, 2025

• [SQL] Run scalafmt on DateTimeUtils and DateTimeUtilsSuite

  #51830 opened Aug 4, 2025

• [SPARK-53112][SQL][PYTHON][CONNECT] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark

  #51831 opened Aug 4, 2025

• [SPARK-53105][Structured Streaming] Fix tests for checkpoint v2 in RocksDBSuite

  #51834 opened Aug 4, 2025

• [SPARK-53116][CONNECT] Fix conf setter when client is built with jar containing SQLConf but not its engine counterparts

  #51835 opened Aug 4, 2025

• [SPARK-53122][CORE] Support `moveFile` in `SparkFileUtils` and `JavaUtils`

  #51841 opened Aug 5, 2025

• [SPARK-53123][CORE][SQL] Support `getRootCause` in `SparkErrorUtils`

  #51842 opened Aug 5, 2025

25 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-52989][SS] Add explicit close() API to State Store iterators

  #51701 commented on Aug 5, 2025 • 34 new comments

• [SPARK-52971] [PYTHON] Limit idle Python worker queue size

  #51684 commented on Aug 4, 2025 • 15 new comments

• [SPARK-52729][SQL] Add MetadataOnlyTable in DS v2 API

  #51419 commented on Aug 4, 2025 • 12 new comments

• [SPARK-52844][PYTHON][TESTS] Update black to 24.3.0

  #51687 commented on Aug 1, 2025 • 8 new comments

• [SPARK-52931][Core] Restrict declare variable naming

  #51669 commented on Aug 4, 2025 • 8 new comments

• [SPARK-52394][PS] Fix autocorr divide-by-zero error under ANSI mode

  #51192 commented on Aug 5, 2025 • 5 new comments

• [SPARK-52991][SQL] Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source

  #51698 commented on Jul 31, 2025 • 4 new comments

• [SPARK-51585][SQL] Oracle dialect supports pushdown datetime functions

  #50353 commented on Aug 4, 2025 • 4 new comments

• [SPARK-52582][SQL] Improve the memory usage of XML parser

  #51287 commented on Aug 4, 2025 • 2 new comments

• [SPARK-52593][PS] Avoid CAST_INVALID_INPUT of `MultiIndex.to_series`, `Series.dot` and `DataFrame.dot` in ANSI mode

  #51310 commented on Aug 5, 2025 • 2 new comments

• [SPARK-52978][SQL] Make FileFormatWriter customizable via SQL configuration

  #51690 commented on Jul 30, 2025 • 1 new comment

• [SPARK-52976][PYTHON] Fix Python UDF not accepting collated string as input param/return type

  #51688 commented on Aug 4, 2025 • 1 new comment

• [SPARK-52226] [SQL] Fix unusual equality checks in three operators

  #50949 commented on Jul 29, 2025 • 1 new comment

• [SPARK-51069][SQL] Add big-endian support to UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl

  #49773 commented on Jul 29, 2025 • 1 new comment

• [WIP][SPARK-51169] Set up a daily job for Python 3.14

  #51532 commented on Aug 1, 2025 • 0 new comments

• [SPARK-52867][SQL] Remove redundant GetTimestamp

  #51556 commented on Jul 29, 2025 • 0 new comments

• [SPARK-52819][SQL] Making KryoSerializationCodec serializable to fix java.io.NotSerializableException errors in various use cases

  #51615 commented on Aug 1, 2025 • 0 new comments

• [SPARK-52943][PYTHON] Enable arrow_cast for all pandas UDF eval types

  #51635 commented on Aug 4, 2025 • 0 new comments

• [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library`

  #51650 commented on Aug 1, 2025 • 0 new comments

• [SPARK-52807][SDP] Proto changes to support analysis inside Declarative Pipelines query functions

  #51502 commented on Jul 31, 2025 • 0 new comments

• [SPARK-52777][SQL] Enable shuffle cleanup mode configuration in Spark SQL

  #51458 commented on Aug 4, 2025 • 0 new comments

• [SPARK-52407][SQL] Add support for Theta Sketch

  #51298 commented on Aug 4, 2025 • 0 new comments

• Enable -Xsource:3 compiler flag

  #50474 commented on Jul 29, 2025 • 0 new comments

• [SPARK-52988][SQL] Fix race conditions in SessionCatalog's metastore function handling

  #51696 commented on Aug 1, 2025 • 0 new comments

• [WIP][SPARK-51348][BUILD][SQL] Upgrade Hive to 4.0

  #50213 commented on Aug 4, 2025 • 0 new comments