Unable to establish connection to syslog-ng server: SSL_get_error()=5

[ValeriiVozniuk]

Hi, I'm trying to push journald logs via systemd-netlogd to our central syslog-ng install. I was able successfully push the logs via tcp, but we got a requirement to do it via tls, and I'm not able to make this config working.
On syslog-ng side I've created a self-signed cert, and disable requirement for clients to use certs authentication.
On systemd-netlogd side I've added the certificate above on machine, added path to it to TLSServerCertificate= option, set TLSCertificateAuthMode= to warn to see just a warning if something is not right, changed Address/Protocol options to use TLS port. After start, systemd-netlogd wait for some time, and then produces error: SSL_get_error()=5
I've enabled debug output, but that didn't help much. I see that server certificate is being validated ok

Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Connected to remote server: '10.15.11.41:6514'
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: enable certificate verification with mode warn
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Verifying SSL certificates of server 10.15.11.41:6514: certificate: subject='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' issuer='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' depth=1 preverify_ok=1 error=0/ok ...
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Verified SSL certificate of server=10.15.11.41:6514 (certificate: subject='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' issuer='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' depth=1): ok
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Verifying SSL certificates of server 10.15.11.41:6514: certificate: subject='/CN=syslog-ng.domain.com' issuer='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' depth=0 preverify_ok=1 error=0/ok ...
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Verified SSL certificate of server=10.15.11.41:6514 (certificate: subject='/CN=syslog-ng.domain.com' issuer='/C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com' depth=0): ok
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: SSL Cipher Version: TLSv1.3 Name: TLS_AES_256_GCM_SHA384
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: SSL Subject: /CN=syslog-ng.domain.com
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: SSL Issuer: /C=US/ST=Some-State/O=MyCompany/CN=syslog-ng.domain.com

Then debug output captures the data to send

Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Reading from journal cursor=s=887a1b2d16434e6fb0a2625087f30210;i=17ba3155;b=6184f1db00744fd8957f1bd4c9a649be;m=16edeccc68e;t=63933ad83cad3;x=6fcdced80368d6aa
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Received from journal MESSAGE='Received from journal MESSAGE='Received from journal MESSAGE='Received SIGTERM from PID 1 (n/a).'''
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Successful TLS SSL_write: 201 bytes
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Reading from journal cursor=s=887a1b2d16434e6fb0a2625087f30210;i=17ba3156;b=6184f1db00744fd8957f1bd4c9a649be;m=16edeccc6ba;t=63933ad83caff;x=c903e07d611d0b48
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Received from journal MESSAGE='TLS: Successful TLS SSL_write: 169 bytes'
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Successful TLS SSL_write: 143 bytes
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Reading from journal cursor=s=887a1b2d16434e6fb0a2625087f30210;i=17ba3157;b=6184f1db00744fd8957f1bd4c9a649be;m=16edeccc6ed;t=63933ad83cb32;x=c8f8820f011ea381
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Received from journal MESSAGE='Reading from journal cursor=s=887a1b2d16434e6fb0a2625087f30210;i=17ba30d5;b=6184f1db00744fd8957f1bd4c9a649be;m=16edeccaeca;t=63933ad83b30f;x=fa86266005039cad'
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Successful TLS SSL_write: 260 bytes
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Reading from journal cursor=s=887a1b2d16434e6fb0a2625087f30210;i=17ba3158;b=6184f1db00744fd8957f1bd4c9a649be;m=16edeccc721;t=63933ad83cb66;x=af194eb1c4381bf4
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Received from journal MESSAGE='Received from journal MESSAGE='TLS: Successful TLS SSL_write: 137 bytes''

And then it suddenly fails

Jul 05 19:37:43 test.host systemd-netlogd[1737709]: TLS: Failed to invoke SSL_write to 10.15.11.41:6514: SSL_get_error()=5
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Failed to send via TLS, performing reconnect: Unknown error 1073741856
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Disconnecting network ...
Jul 05 19:37:43 test.host systemd-netlogd[1737709]: Closing journal input.

On syslog-ng side I see only

Jul  5 19:37:43 syslog-ng-85b7d46dd4-dchzm syslog-ng[8]: Syslog connection accepted; fd='26', client='AF_INET(10.42.3.0:65525)', local='AF_INET(0.0.0.0:6514)'
Jul  5 19:37:43 syslog-ng-85b7d46dd4-dchzm syslog-ng[8]: Invalid frame header; header=''
Jul  5 19:37:43 syslog-ng-85b7d46dd4-dchzm syslog-ng[8]: Syslog connection closed; fd='26', client='AF_INET(10.42.3.0:65525)', local='AF_INET(0.0.0.0:6514)'

Any ideas what could be wrong here? Or this setup just would not work?

Also, am I understand correctly, that systemd-netlogd doesn't support authentication with certificates for tls connection?

[ValeriiVozniuk]

The issue was on syslog-ng, their default_network_drivers() seems broken with TLS, replacing that with network() fixed connection issue, but only with peer verification disabled. Security didn't like this solution, so systemd-netlogd was dropped in favor of syslog-ng as a client too.

[derobert]

Just stumbled across this ticket. It's expecting RFC5425 framing (ran into this with a log appliance which is running syslog-ng internally); I wrote #140 to fix the appliance — it will probably fix the issue you saw, too.