The pilot said the decision to return to Perth 90 minutes after leaving was made because of "technical issues" with the passenger plane.
"I could tell by the cabin crew's reaction that it was really bad," passenger Sophie Nicolas told ABC News.
The plane landed safely at 10:00 local time (02:00 GMT) on Sunday.
"I was crying a lot, a lot of people were crying, trying to call their mums and stuff but we couldn't really do anything just wait and trust the captain," Ms Nicolas told ABC, adding: "Everybody burst into applause when we landed."
Police said marine emergency services north of Perth were put on standby to prepare for a possible water landing, the news network reports.
A number of passengers on board the budget flight said the plane had suffered an "engine seizure" causing it to shake.
AirAsia X did not give details about the technical issue behind the incident.
In a statement, it said the safety of passengers was "our utmost priority".
Earlier this month an Airbus A330 belonging to China Eastern Airlines was forced to make an emergency landing in Sydney after a huge hole appeared in one of its engine casings.
Several terrified passengers later described hearing a very loud noise soon after leaving Sydney for Shanghai. Images were posted on social media showing the extent of the damage.
In December 2014, an AirAsia plane crashed into the Java Sea, killing all 162 people on board after the aircraft's rudder control system malfunctioned during the flight.
An AirAsia X flight bound for Kuala Lumpur was forced to turn back to Australia after an engine problem left it "shaking like a washing machine".