Though imposing, the 3.7m (12ft) cephalopod was much smaller than the 13m they can grow to.
It spent several hours in the harbour and was filmed by local divers on Christmas Eve.
Mitsuhiro Fuwa, a curator at the local Uozu Aquarium, told the BBC there had been more than a dozen giant squid sightings in the last year.
Giant squid are more often seen trapped in fishing nets than swimming near the surface.
Professional underwater cameraman Takayoshi Kojima told the BBC he rushed to the harbour from nearby Toyama city when a manager at the marina called him.
He helped guide the squid to a deeper part of the marina, near the exit to the sea, where it disappeared.
Mr Kojima said he did not know whether the squid made it to open waters but said it did not look like it would survive long.
Mr  Fuwa said that although the squid showed some signs of energy - squirting ink and trying to wrap its legs around the divers - it did not look very healthy, with its body pointing downwards and its legs up.
Mr Kojima said that with a rise in sea temperature, giant squid have moved north to waters off Toyama, from where they swim up from deeper waters during the winter season.
He added that most sightings happen between December and January.
Toyama is about 300km (186 miles) west of Tokyo.
Giant squid prefer the deep sea, so it was a rare treat for onlookers in Toyama Bay harbour when one swam in.