Replying since I found the cause on my phone. When you connect a keyboard there is still an "Virtual keyboard" active. On my android you can select a different virtual keyboard by tapping on the keyboard icon on the screen (probably different location for different androids) and select another virtual keyboard. For me SwiftKey keyboard was really slow and the fastest seemed to be Gboard. Hope this helps someone else
This would be an useful feature!