You may remember the problems we had with Firefox 3.0.0 and the encoding in XMLHttpRequest back in June. The Firefox team now tried to fix it ( Issue 431701 in their Bugzilla), but failed miserably, they made the problem worse (unintentially, of course :)). While it was technically correct before, now it's not even that, as it still sends ISO-8859-1 encoded content, while it claims it's UTF-8 under some circumstances. The bug reports claim, that it only happens on non-US versions of firefox, but that doesn't help us much. This new regression is tracked at #464958, is already fixed and should be in Firefox 3.0.5, which is due around mid-December. With the auto-update feature of Firefox, a lot of our users will have updated to 3.0.4 and can't really wait until mid-December, so we had to find a workaroundā¦
As it is really a bug this time and not just a different implementation of the problem, a correct and beautiful solution can't be done, so we had to fall back to some ugly trickery. When the DOM Parser can't read the XML string due to malformed UTF-8, we check, if it could be an ISO-8859-1 string and convert that to UTF-8. If it then still doesn't work, we just tell the parser to disregard all invalid UTF-8 characters as a last resort.
I currently see no other decent way to tackle this problem. āIf it's not UTF-8, even if it claims to be, assume that it could be ISO-8859-1 and try with thatā seems to be an ok-ish idea here in the Western Europe Alphabet World. Anyone else should maybe really wait with the upgrade until Firefox 3.0.5 is ready.