Burt, here's what I found on the Coney Island Velodrome:
Googling "Coney Island Velodrome" with "Sugar Ray Robinson" (just those words within quotation marks), pulls up Dave Anderson's 1992 forward for Ray's 1969 autobiography (third link from the top):
One morning, in the late Summer of 1950 my father
noticed in the New York newspapers that Sugar Ray was
fighting that holiday afternoon in the Coney Island
Velodrome, a little outdoor wooden arena not far from
our Brooklyn apartment.
"Let's go," he said. "You've got to see this guy fight."
Sugar Ray was between titles. He had been the world
welterweight champion from 1946 to 1949 and now he
was campaigning as a middleweight. His opponent that
Labor Day was Billy Brown, durable but not overly
talented. Sugar Ray earned a 10-round decision without
breaking much of a sweat. five months later he would
stop Jake LaMotta in Chicago to win the middleweight
title for the first of five times
This article describes something of that event, in the second half of the second paragraph, the 3,000 in attendance, "in rainy, blustery conditions.":
Everything I can find on the CIV indicates that September 4, 1950 was its final day of operations before shutting down and getting leveled.
Bobby ***** was listed as being six feet tall, Billy Brown just an inch less, if that.
Here's Brown in action the next year getting stopped by Randy Turpin:
Seeing Billy Brown in action on that brief footage, having viewed Robinson-***** a few times, then finding that every on-line resource states the Coney Island Velodrome closed down operations after September 4, 1950, suggests to me that the 5'11" Billy Brown must have been who you saw against Robinson on that date, not the 6'0" Bobby *****, who Ray took on for the television camera in Chicago two months later.
If the reported description of "rainy, blustery conditions" jives with what you recall the weather to have been, then this mystery can be resolved as a very understandably honest mistake.
Here's that complete card, as listed by Boxrec, for that Labor Day Monday in 1950:
Sugar Ray Robinson W UD 10 Billy Brown
Bobby Rosado W PTS 6 Danny Womber
Jimmy Cerello W PTS 6 Ronnie Hopp
Angel Martinez KO 4 Wayne Eckhardt
Gene Robnett W PTS 4 Abdoul Ali
Does any of this jog your memory?