You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-53090][CORE][SS][TESTS] Use Java OutputStream.write instead of IOUtils.write
### What changes were proposed in this pull request?
This PR aims to use Java `OutputStream.write` instead of `IOUtils.write`.
### Why are the changes needed?
To use a better implementation for our use cases.
```scala
scala> val s = "a".repeat(400_000_000)
scala> spark.time(new java.io.FileOutputStream("/tmp/a").write(s.getBytes()))
Time taken: 270 ms
scala> spark.time(org.apache.commons.io.IOUtils.write(s, new java.io.FileOutputStream("/tmp/a")))
Time taken: 1070 ms
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#51804 from dongjoon-hyun/SPARK-53090.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
0 commit comments