Do you think Lord British would destroy Buccaneer's Den if he knew about it? Mixing several layouts, I altered the layout a bit from Ultima IX to make it a bit more covert.
This place is like 20 feet away from Lord British's castle, and I feel if he knew it was there, he'd gather an army and storm it, since it's a pirate den? But realistically, maybe it's just a port town that happens to have pirate elements. And you can't just sack a soverign town.
Here's the original fb UDIC post.