You Can Serve as a Groomsman Instead
It’s possible that you dislike the idea of becoming the “best” man. As was indicated at the outset, it’s possible that you simply don’t know the man very well. It’s also possible that he chose you over his brother or another best friend, which has caused some friction with his family. Offer to be a groomsman instead of the best man if at all possible.
What you need to do is inform him right away. Timing is the last point to consider. You don’t want to give the groom no answer at all. The sooner you go, the sooner he can replace you and you can continue acting as if the awkward situation never occurred. Offer to assist in other ways if at all possible. Even if you don’t mean it, it will sound as like you care about what happened and feel at least a little bit bad about dumping him.
There is ultimately no simple way to turn down an offer with such strong emotional undertones, but you’ll find that most guys are understanding, especially if you give them enough notice and have a good reason for refusing the title of best man.