Davis developed into a tough, street-smart young man, and became well known in a neighborhood that was famed as the home of Murder, Inc. His two brothers were affiliated with the notorious gang, acting as collectors. However, because of Al's toughness and fierce independence, he was able to steer clear of the temptations of the easy wealth of the gangster life by concentrating on his boxing. In fact, he was one of the few young men in the neighborhood who was unafraid to stand up to feared local hoodlums like Murder, Inc.'s Abe Reles.