background: I’ve built a bunch of computers in the last couple years - handful of windows, couple BSDs, lots of linux, including several high end $6-10k builds. The only compatibility problems I’ve had were between linux, an Asus x99 workstation mobo and a xeon e5-2690v4 right after v4 came out (since fixed) and…sound cards.
In my opinion, if you’re only buying one (or a couple), high-end workstations from major OEMs are a rip-off. You’re easily paying >50% markup and getting lower quality components (motherboard, power supply, cpu cooler, fans, etc…and please, for the love, do not buy a GPU from them). They achieve compatibility by avoiding cutting edge stuff, choosing “stable” distributions (mainly rhel/centos or debian/ubuntu which in turn achieve stability by using older kernel + key packages). I’m not saying that’s bad - it’s great to have stuff just work, but it’s easy to make those same choices yourself.
I’ve never bought from a custom builder, but I’ve come close and done a bunch of research. I really like www.pugetsystems.com approach (I used one of their configurations as the basis for my main machine). They publish a lot of their research on components/builds, do lots of work on compatibility, thermal management, noise, etc. What a good custom builder will be able to help you with is getting the most out of your budget and choosing the right configuration for your use case.
For me - I chose to build myself when I realized how much more capability I could afford by DIYing it. In hindsight, I wasted a lot of time worrying about micro-optimizations and trying to make absolute bleeding edge stuff work - it’s super easy to make a computer if all the parts are > 6 months old…but try and use that super-cool brand new part and you might throw your machine out the window.