The Byakko sank at about 2:40 a.m. after colliding with the chemical tanker just before midnight in the Kurushima Strait, Chief Cabinet Secretary Katsunobu Kato told a news conference.
Nine of its 12 crew were rescued by the coast guard and nearby ships, he said, adding that coast guard patrol vessels were scouring the waters for the remaining three.
There were no injuries among the tanker crew, and no leakage or floating objects at the site, he said.
A Japanese coast guard spokesman identified the tanker as the Ulsan Pioneer, registered in the Marshall Islands.
The coast guard had deployed 14 patrol ships and three aircraft in the search, he said.