Jake Forster-Caskey put the Seagulls ahead with a penalty after he had been fouled in the box by Paul Downing.
Milan Lalkovic equalised for the hosts before Brighton's Chris O'Grady blazed over the bar from the penalty spot.
Rico Henry then completed the Saddlers' comeback, as he fired in a left-footed shot from Romaine Sawyers' cross.
Brighton manager Chris Hughton told BBC Sussex:
Media playback is not supported on this device
"I felt that we had some really good periods in the game. I can't remember a game for a while where we've had so much good possession breaking into their final third.
"We had ample opportunities and got into some great areas, but of course if you're not able to get that second goal, it's always going to put pressure on.
"Until we learn to kill off games, we're always going to make life difficult for ourselves."
League One side Walsall came from behind to beat Championship opponents Brighton and earn a League Cup tie with Chelsea.