Taking the boat from Ferry Overlook to Bai Mansion is a quick cutscene, and then you're standing on the shores of the island.