Add brotli support to FoundationNetworking

[zhenchaoli]

Add brotli as a dependency to enable brotli decoding for curl.

brotli is a general purpose compression algorithm used extensively on web. (https://datatracker.ietf.org/doc/html/rfc7932)

brotli support for curl is enabled by integrating the brotli compression library (https://github.com/google/brotli). I've also added some tests for FoundationNetworking in swiftlang/swift-corelibs-foundation#5251 which exercises this feature by

• looking at the accept-encoding http header
• decoding a brotli encoded payload

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[zhenchaoli]

Seems windows ci passed. Linux failed due to test compilation error. I've added one more commit to enable test on Linux for swiftlang/swift-corelibs-foundation#5251

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[zhenchaoli]

Linux test failure should be fixed now on swiftlang/swift-corelibs-foundation#5251

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test Linux platform

[zhenchaoli]

rebased.

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[compnerd]

It would be nice to get a bit more background on what is being done and why.

[compnerd]

Do we need to install it? Can we not use it from the build tree?

[zhenchaoli]

I'm following the example of zlib here, and it feels cleaner to rely on build artifacts being in a consistent place for all dependency outputs. It seems all other dependencies already do this, so I think we should be consistent. (Maybe let's do these kind of changes in a separate PR for all dependencies if you feel strongly about it)

[compnerd]

zlib is the only one doing this because it cannot be used from the build tree (I would love to migrate to that model for zlib as well). We no longer install any library which is not being distributed as a DLL - but that requires switching to the master branch rather than a release.

[zhenchaoli]

Just to be clear, by "use it from the build tree", we'll be referencing .lib object files in brotli build tree like brotli\out\Release\brotlicommon.lib, and header files in its source tree at paths like brotli\c\include\brotli\decode.h?
I could try but I doubt it'll play well with the goal in the other comment which is using Brotli_DIR.

[compnerd]

Yes, except, the idea with Brotli_DIR is that you tell CMake the directory to look into for BrotliConfig.cmake and it will wire up the headers and libraries appropriately. You don't need to pass flags or paths to the libraries or headers.

[compnerd]

Do we need to pass in the library paths or do they generate a config file that we can use with Brotli_DIR?

[zhenchaoli]

The build output of brotli does include 3 .pc files, but unsure how well curl supports BROTLI_DIR. The input variables for curl's brotli cmake are BROTLI_INCLUDE_DIR, BROTLICOMMON_LIBRARY, BROTLIDEC_LIBRARY according to https://github.com/curl/curl/blob/master/CMake/FindBrotli.cmake

[compnerd]

The .pc files are for pkg-config, which we do not use or support as that is not something that is available on Windows by default. https://github.com/curl/curl/blob/master/CMake/FindBrotli.cmake#L62-L68 is the bit that matters - it should hopefully allow us to specify just the _DIR suffixed variable to find everything.

[zhenchaoli]

It would be nice to get a bit more background on what is being done and why.

Updated the overview for this PR. Let me know if any details are missing.

[zhenchaoli]

swiftlang/swift-corelibs-foundation#5251

@swift-ci Please clean test

[zhenchaoli]

@compnerd is https://github.com/swiftlang/swift/blob/main/docs/WindowsBuild.md the best way to go about testing the build on Windows? So far I've been relying on the CI for the toolchain side of changes.

[compnerd]

@compnerd is https://github.com/swiftlang/swift/blob/main/docs/WindowsBuild.md the best way to go about testing the build on Windows? So far I've been relying on the CI for the toolchain side of changes.

No, I would suggest that you reference https://github.com/compnerd/swift-build/blob/main/docs/WindowsQuickStart.md. It is much simpler and up-to-date. The use of repo is what keeps me from merging that into the tree.

[zhenchaoli]

@compnerd is https://github.com/swiftlang/swift/blob/main/docs/WindowsBuild.md the best way to go about testing the build on Windows? So far I've been relying on the CI for the toolchain side of changes.

No, I would suggest that you reference https://github.com/compnerd/swift-build/blob/main/docs/WindowsQuickStart.md. It is much simpler and up-to-date. The use of repo is what keeps me from merging that into the tree.

unfortunately the build did not go very far for me while following the instructions:

-- The Swift compiler identification is unknown
CMake Error at S:/SourceCache/swift/CMakeLists.txt:108 (enable_language):
  The CMAKE_Swift_COMPILER:

    S:/b/toolchains/swift-6.1.2-RELEASE-windows10/Library/Developer/Toolchains/unknown-Asserts-development.xctoolchain/usr/bin/swiftc.exe

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment
  variable "SWIFTC" or the CMake cache entry CMAKE_Swift_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.


-- Configuring incomplete, errors occurred!

I looked at the path and don't see swiftc.exe there. Instead it's a nested folders S:\b\toolchains\swift-6.1.2-RELEASE-windows10\LocalApp\Programs\Swift\Runtimes\6.1.2\usr\bin that contains nothing.

Is there anything obvious I'm missing?

[compnerd]

I think that a small optimization that matters for local development is biting you. We don't re-extract the toolchain, and I suspect that the extraction failed in the first run. Try again adding -Clean and if that run fails (i.e. the extraction fails) you might be missing the .NET runtime which is needed for the extraction.

[zhenchaoli]

Very likely to be the case. I just installed the .NET runtime and retrying. Separately when you get a chance, would you also mind taking a look at my questions above: #83441 (comment)