Former Derby County, Hull City and Preston boss Brown, 57, took the job at Roots Hall on 25 March 2013.
He took over with the Shrimpers in League Two, but guided them to promotion via the play-offs in 2015 and his side are now sixth in League One.
"I started off a job four years ago to prove the world of football wrong," Brown told BBC Essex.
The ex-Bolton and Hartlepool defender said he had had "to go down into the second division to get a job".
Brown had been out of management since December 2011, when he was fired by Preston, before being appointed by Southend in 2013.
He said many chairmen these days are "trigger happy", leading to "good men ending up losing their jobs".
"I'm so happy the chairman [Ron Martin] isn't influenced by the outside world as much as other chairmen are," Brown continued.
"I think if you have a good relationship with the guy up top, that is the most important thing. I firmly believe that, given time, most managers will get it right."
Brown is the 11th longest-serving manager in England's top four divisions, with under-pressure Arsenal manager Arsene Wenger top of the list having been in charge of the Gunners for more than 20 years.
"I'm hoping Arsene hangs on to his job," Brown added. "If that crumbles, unfortunately the whole mentality of the world of football will change."
Southend United manager Phil Brown will reach four years at the club when his side visit AFC Wimbledon on Saturday.