The outcome is this simple inherited TextBox control I decided to call TextBoxWithoutCaret :)
class TextBoxWithoutCaret : TextBox
static extern bool HideCaret(IntPtr hwnd);
static extern bool ShowCaret(IntPtr hwnd);
protected override void OnGotFocus(EventArgs e)
protected override void OnLostFocus(EventArgs e)
Every time the TextBox control is focused I hide the caret and enable it again when focus is lost. This doesn't really make much practical sense and the only reason I do this is because HideCaret() is described to perform a cumulative operation meaning ShowCaret() must be called the same number of times HideCaret() was called for the caret to be visible again.