The shop inside Parkway mall has a filter that can be attached to s faucet.

Just buy filtered water in the large blue bottles! there are many well established brands and they only cost Usd1 per bottle!

Every little bit helps, and in addition to the previous suggestions there's an asian company making standalone water filter / ultraviolet light treatment systems in a package that dispenses hot or cold water. As you probably already know, unless you buy a reverse osmosis filtration system you'll still be drinking any mercury, lead, arsenic, pesticides, etc. that's in the river / city water. There's a few stores on Monivong that sell the reverse osmosis type.