Indeed, they have a peculiar relationship. Towards the end of the battle at Marineford, he had a fashback to some time he spent with Roger, and how he was not interested in Raftel at all even if Roger told him where it was. How come then, that earlier than that, when Shanks confronted him about stopping Ace from the pursuit of Blackbeard, he admitted that they used to fight a lot with Roger's ship.
Yes we know no pirates have any proper relation, but would someone like Doffy have a cup of tea with Crocodile at all? Not to mention the enormous secret Roger was just about to reveal so casually.