Agreed that it little affects the Custom House from the west, but from the east it really does look ridiculous I think.
I'd like to see renderings of the city and Custom House without it next door to make any sort of judgement on its absence.
It's difficult nowdays to genuinely despise this view all the same - in 1965 it was offensive, in 2005 it's more humourous
Overall though yes, it is an iconic structure on a number of levels, and is not what most people could describe as an 'ugly' building. Certainly I've always liked it, particularly when viewed from the west rising up over the quays, marking the bend in the river and reflecting the late afternoon sun, often dazzlingly lit while dark clouds loom behind. And the roof is still as distinctive as ever, even if it is a contrivance of sorts.
The notion of knocking it purely for a 'better' building is simply detestable.
A present crestfield it is not a protected structure.