One of the interesting quirks of water is that it expands when it goes from a liquid state (water) to a solid state (ice). Very few substances in the world exhibit this property. This means that ice floats on top of water, and it also means that when ice melts, it takes up less space than it did as a solid. If you had a glass full to the brim with ice (not ice cubes, but uniform solid ice), the water level would actually drop below the rim of the glass when the ice melted.

So you have a point here: Ice that is already underwater should not cause ocean levels to rise when it melts. The water level should drop, if anything. HOWEVER — and this is an extremely big, written-with-neon-exclamation-points however — there's a lot of ice in the Arctic and Antarctica that is not currently underwater: namely, icebergs.* You've probably heard that 90% of an iceberg is underwater, but the 10% that's above the surface can be more than 100 metres high. That's a lot of frozen water!

If the planet gets warmer, these icebergs will melt away, and suddenly all that solid ice that was suspended above the ocean will turn into liquid water in the ocean. Not only that, but glaciers and other ice stores on land could melt, trickling into streams and rivers and eventually into the ocean. A mountain of ice in Greenland could turn into two metres of standing water in Manhattan.

The ice covering Antarctica is, on average, 2 km thick. Canadian Geographic estimates that if Antarctica were to completely melt, the world's ocean levels would rise a whopping 60 metres.

[* CLARIFICATION: Icebergs that are already floating in the ocean will only cause a negligible rise in ocean levels when they melt. As they melt, the volume of water will decrease (because water is denser than ice) and the resulting volume should be approximately the same as that of the water the iceberg originally displaced. In other words, the volume of the iceberg will shrink by approximately the same percentage as the percentage of ice that was sticking up above the water.

This melting will still contribute slightly to an increase in ocean levels, because icebergs are made of freshwater, which is less dense — takes up more space — than the seawater it is melting into. But the much greater impact is caused by ice stores on land dropping these icebergs into the ocean in the first place. (And, really, you can't have one without the other. If the floating icebergs are melting, the glaciers are going to be melting too.)

Thanks to Carl S. for straightening me out.]

Sources: Canadian Geographic, CNN