[OS X TeX] TeXShop 3 and text encoding on Lion - serious problems

[Ross Moore]

Hello Luis,

On 11/09/2011, at 8:39 PM, Luis Sequeira <lfsequeira at gmail.com> wrote:

> I am experiencing serious issues with TeXShop and text encodings in Lion. 
> 
> My wife sent me a tex file written using the mac roman encoding. I opened it, selecting the encoding, as usual, from the dropdown menu in the open dialog. 
> The accented characters don't appear. The file typesets correctly - using \usepackage[applemac]{inputenc} - which confirms that the file itself is not corrupted and is using the right encoding.

The file is just a byte stream, and the  \usepackage[applemac]{inputenc}  tells LaTeX how to deal with the characters outside the ASCII range.

> Making changes to the file causes havoc, as would be expected - now new accented characters that are input show up correctly in the source but not in the pdf - suggesting that TeXShop is using a different encoding than it is was supposed to.

No. Adding new accented characters to this file is a really bad idea.
I do not believe that the file encoding tells your Mac software to translate bytes to a different encoding upon input; maybe it does for UTF8 or UTF16, but certainly not for other legacy encodings. 

> 
> I then converted the file to utf8 (using Smultron) and replaced applemac by utf8 (\usepackage[utf8]{inputenc}) - and again opened the file from the open dialog, selecting utf8 encoding.

Is this the original file, or with the extra typed characters?
Personally, I would look at the converted file with a command-line editor, such as vi , to see whether the bytes have really been converted correctly. The physical file size should have increased slightly, as UTF8 requires a byte sequence for each non-ASCII character.

> In this encoding the accented characters still did not display correctly, but still the file typeset ok.

Display correctly in which software? TeXshop's editor?
What does it look like in other Mac editing software, such as TextEdit?
Maybe there is a memory of the old encoding.
Did you change the file's name for the new encoding?

> 
> Both versions of the file (using macroman or unicode) display and typeset correctly in TeXShop 2 on Leopard.
> 
> I think this is a serious bug in TeXShop in Lion (I am using 3.04, but the same problem occurs in 2.29 in Lion). 
> Smultron in Lion (although an old version from 2009) has no problem handling the files. TeXShop 2 in Leopard also has no problem. The same flle displays and typesets perfectly in my wife's iBook.

That is disturbing, that newer software is not doing the same as in older versions.
I'd guess that the change is in more recent Apple libraries, perhaps dropping some aspects of the support for really old methods.

What happens if you take your wife's file and convert to UTF8 using iconv on the command-line, before ever opening it in any window-based program?
I'd expect that now everything will look right in all programs. (touchwood)

> 
> For now this makes working with accented characters (which are abundant in my language, Portuguese) a nuisance. I am reverting to writing stuff like \'a for á for the first time in years.

Knuth's original methods show their value!

> 
> Luis Sequeira

Hope this helps,

       Ross