You are not alone.

An interesting activity is to collect various copies of the same recording session. CD, tape, LP and compare them. For example play a UK, Japanese, and US pressing of the same album. Compare pressings from different eras. You might find that they sound quite different. When possible, include a MFSL release.

Personally, I find the noise and various distortions inherent in tape and LP unattractive. I’m not saying that digital is always “better”, due to sloppy, uninformed processing along the way, but the noise and distortion floors can be much lower with digital technology.

Considering the bragging rights available to the person with the golden ears, esoteric audio gear and perfect listening environment you’d have thought that at least one of these folks would have set up a proctored, double-blind ABX test to prove just how superior their setup was.

So far I haven’t seen anybody credible claiming they are able to reliably identify music formats that meet or exceed CD quality.

https://infogalactic.com/info/ABX_test

Down a ways he gets to bit depth testing: https://audiophilestyle.com/ca/bits-and-bytes/fun-with-digital-audio-%E2%80%93-bit-perfect-audibility-testing/

Test files: https://www.audiocheck.net/testtones_highdefinitionaudio.php

Quick fun test: https://www.npr.org/sections/therecord/2015/06/02/411473508/how-well-can-you-hear-audio-quality

It isn't just for this that there are no such tests. There also are no such tests for stereo amps of similar power delivery/distortion spec, DACs, cables that meet the thickness required for the cable run in question - regardless of the prices of the items being compared, even where price differences may be in orders of magnitude. 

But for each, there are endless subjective claims, led by the media that writes about these. Human frailties make this domain a target rich environment for both scamsters and for those that are just more prone to sincerely believing whatever they see in print and who are not familiar with the subject of psychological biases.

On music formats, I haven’t seen such tests for even CD format lossless v 256/320k lossy, where headphones have not been used for listening.

I understand the rational of ABX, but I have a little difficulty with the thought that the ABX box is perfect and the attached equipment is not. We built an ABC..X box using relays that might be used in the actual ABX boxes quoted in reviews. On day one it was apparent to all of us that the ABC...X box masked some of the differences between the devices. We are not the self proclaimed “golden ear” types, we are simply a group of engineers, musicians, and regular folk who can pay attention. And, we are not afraid to disagree with one another with respect to “best”. Yes, we are aware of and can deal with comparison complications associated with level differences.

Thinking of building an ABX box brings to mind so many possible issues that I’d have hated to think of it, even back in my electronics technician era.

Every component in the audio path is critical and any interactions with the power supply or neighboring equipment that vary by input selected would be big issues.

You can buy matched components of some types, the price is usually not pretty. Others you’d have to test and match individually.

Using a circuit-board versus direct wiring only gives you a choice of the undesired impacts of either, not a “good” option.

Relays are tough, you’d likely want magnetic latching so you could eliminate external current flows during the tests. Contact types, who knows mercury wetted, gold, vacuum sealed unobtainium?

Connectors, solder joins, vibration proofing, too much for my poor brain.

Maybe you could ABX test the ABX box by feeding each side the exact same signal?

On ABX - I have found Butterworth to be one of the level headed people writing on audio on the net, and here is a link to a relevant article by him:

https://hometheaterreview.com/why-do-audiophiles-fear-abx-testing/

Unfortunately, he has not expressed any views on any conclusion he may have reached, but one thing that he also leaves unexplained is how any ABX box does level matching of signal voltages necessary to achieve the necessary identical sound levels.

He does say this though: The problem for audiophiles is, ABX testing has, to date, rarely revealed differences in sound among audio electronics components.

Also, even a much simpler A/B switcher than the ABX box in the link can suffice for amateur tests IF the level matching can be done well enough. It ought to be easy to build or get such a simple switcher that does not add its own issues to the test.

Kumar,

Typically, there will be an input attenuator as part of the ABX box. This will be used while testing devices  that may not have input or output attenuators. For a power amplifier that has no input attenuator, the test will require two trips through the ABX box-- before and after the device(s) under test.

If you are a fan of low output impedance power amplifiers, fat wire, and high damping factor, power amplifiers with output impedance in the range of 0.00x Ohms are common. If you are not familiar with the terms, they are not very related to the recommended load impedance for the amplifier. Relay contact resistance is commonly in the 0.1 Ohm range. In an otherwise low impedance speaker wiring setup the relay contact resistance becomes a factor. It is common for power amplifiers to include a relay that keeps the amplifier disconnected from the speakers while the amplifier powers up and powers down. This protects the speakers from difficult to control transients. Very high quality amplifiers tend to omit this relay -- claiming that it injures sound quality. If this is true, why would one want to insert a relay in the test setup?

That is why I would just use a simple switch and do a A/B test and get most of the outcome reliability that adding the X will deliver but with all the complications of the ABX box. The unsolved problem will still be getting the same sound levels for A and B, and that needs to be quite precisely achieved to get a reliable result in both cases.

ABX protocol is better than AB no doubt, but for this kind of testing to satisfy just oneself, I believe that AB is adequate.