Since the new JSSE version was backported to OpenJDK (JDK-8248721) to support TLSv1.3 the logic of TLS debug traces has changed a bit. If you use the old -Djavax.net.debug=all system property syntax, the trace messages will be written to stdout of the process, but not to the Java System.out (redirected) stream. This means, it will not show app in the application server logfiles (like Wildfly/JBoss).
The new debug logger will actually use a configured java.util.logger (JUL) logger when you specify the system property empty (without an value: -Djavax.net.debug). In this case the log messages will be ready for a application container to log via a LogHandler - for example an internal logging bridge. The logger name will be javax.net.ssl and all with level FINE or lower.
The mechanism works with Zulu and AdoptOpenJDK, the Oracle Java binaries seem to have some problems in this are, as reported by me on jdk8u-dev mailing list.
I managed to get JBoss to log the messages with specifying a new category javax.net.ssl with level ALL and specifying the -Djavax.ssl.debug system property, however in my case the log messages are missing the hex dump attachments. This is because the log records contain an additional string parameter without a format parameter and the JBoss logmanager does not formating those excessive (multi line) strings at the end of the log message. You would probably have to extend logmanager to do that (which might be good for malformed log format strings in the first place).