Or should you get the proper size collets? 
High quality reducers are unlikely to cause issues on a router. I can see precision being an issue on a mill, but on a router, I would expect other factors to swamp any error contribution from the reducer. There’s an extra element to get right, though, so there’s a slightly higher chance of user error when installing a bit.
I haven’t used reducers, but I have used extensions (which also typically reduce). They’ve worked well for me, though I accept worse chatter and higher wear on the router bearings.
I’ve run a 1/4" to 1/8" Amana reducer in my DeWalt router on an Avid Benchtop Standard with success. It does require replacement after some amount of use, but it outlives the tool itself. I wasn’t overly concerned with chasing thousandths of an inch so I can’t verify precision, but it didn’t show up any obvious issues.
One thing I do use as a reducer, is an ER-16 collet on a cut-off stub shaft held by the ER-25 collet as a reducer. This is used mostly for drilling where it is not putting any side loads on it. Pretty much only for holding smaller drills since many metric ER-16 sets cover all the tiny in-between sizes.
I bought an Amana 1/4" to 1/8" reducer for a Dewalt trim router. The reducer is too tight for the 1/8" endmills and fits too loosely in the router collet. Try to find the proper size collets. I believe “Think and Tinker” carries high quality collets and nuts for various routers.