I ended up polling the latestBockhash every half a second and using the cached one for my transactions since the block hashes are valid for longer and apparently it does not need to be the latest even when the name implies that.
But there are some other options.
subscribeBlock pushed the whole block data to the client and could be used to get a hash but it contains lots of data and the rpc may not support it. Maybe it could be filtered somehow, but I did not try that.
The more complicated approach from Solana break works like this as Jordash explained me is to send the transaction directly to the current leader. Sadly with the Introduction of QUIC it will probably not be possible anymore to push transactions directly to the TPU UDP port of the leader anymore.
https://github.com/solana-labs/break/blob/c97989858498f056954649c0f8ae03862a805a4d/server/src/available_nodes.ts#L33
Polling getClusterNodes
is used to obtain https://solana-labs.github.io/solana-web3.js/types/ContactInfo.html which has the TPU address for nodes.
onSlotUpdate
is used to track recent slots
https://github.com/solana-labs/break/blob/c97989858498f056954649c0f8ae03862a805a4d/server/src/leader_tracker.ts#L20
This is combined with the leader schedule to determine the current leader
Sockets are opened to nodes
https://github.com/solana-labs/break/blob/c97989858498f056954649c0f8ae03862a805a4d/server/src/tpu_proxy.ts#L161
and raw signed transactions are sent over these sockets
A websocket from the server forwards to the TPU proxy
https://github.com/solana-labs/break/blob/c97989858498f056954649c0f8ae03862a805a4d/server/src/api.ts#L13
The client has a socket connection to the server to send transactions over