Personally, I would say Jimmy's prime ended after Moore in 45. Although we know very little on the subject as Jimmy didn't like to talk about it, he was badly hurt during the war. After that beating he took, his reflexes were never the same, at least according to his sister and his mentor (who quit when jimmy refused to retire at age 26).
What we know as fact is that after his strange loss to Jersey Joe, he started losing to guys he previously walked over (Lee Q Murray for instance). A lot of critics argue that Bivins only beat Charles and Moore because of the weight advantage he held over them, and that when these fighters increased their weight he started to lose. Too simplistic for me, and it completely overlooks his beating during the war, and the feelings of his close friends