You also have Burley who was the main focus of the guys who never got a chance, he gets massive credit for how good he was at welterweight yet hardly ever fought under the limit, his main fights down there being against Zivic (1-1) and a win over Cocoa Kid. Once moving in the direction of middleweight (he tended to box at around LMW for the best part of his career) he scored up wins over Billy Soose and Fritzie Zivic (to win the rivalry) but had to fight his murderers row compatriots a lot in Holman Williams (2-3), Jimmy Bivins (0-1), Archie Moore (1-0), Ezzard Charles (0-2), Cpcpa Kid (1-0-1) Aaron Wade (3-0), Jack Chase (3-0), Bert Lytell (1-1) and Lloyd Marshall (0-1).
So he has a quality record, apparently fighting above his natural weight but spent a lot of his career fighting other coloured and avoided fighters. Won the coloured world championship but never really fought many of the top fighters in and around his weight such as Robinson (famously), Gavilan, LaMotta, Yarosz, Fox, Janiro, Basora etc.
I've just got one of his books of his so I'm sure I'll learn a lot more about him then, maybes he was avoided by the plague, unwilling to make sacrifices for the mob, had a bad promoter/agent/manager or just plain wasn't exciting enough but he never got as many big fights as many in his era did so he's a hard one to rank accordingly.