[OS X TeX] TeXShop 3 and text encoding on Lion - serious problems

[Luis Sequeira]

> Hello Luis,
> 
> On 11/09/2011, at 8:39 PM, Luis Sequeira <lfsequeira at gmail.com> wrote:
> 
>> I am experiencing serious issues with TeXShop and text encodings in Lion. 
>> 
>> My wife sent me a tex file written using the mac roman encoding. I opened it, selecting the encoding, as usual, from the dropdown menu in the open dialog. 
>> The accented characters don't appear. The file typesets correctly - using \usepackage[applemac]{inputenc} - which confirms that the file itself is not corrupted and is using the right encoding.
> 
> The file is just a byte stream, and the  \usepackage[applemac]{inputenc}  tells LaTeX how to deal with the characters outside the ASCII range.
> 

Yes, I am fully aware of that.

>> Making changes to the file causes havoc, as would be expected - now new accented characters that are input show up correctly in the source but not in the pdf - suggesting that TeXShop is using a different encoding than it is was supposed to.
> 
> No. Adding new accented characters to this file is a really bad idea.
> I do not believe that the file encoding tells your Mac software to translate bytes to a different encoding upon input; maybe it does for UTF8 or UTF16, but certainly not for other legacy encodings. 
> 

Any editing software - and, in this case, it is TeXShop - needs to use *some* encoding to store the text file in. When I open a file in TeXShop with, say, ISO Latin 1 encoding, and start typing accented characters into it, they must be stored using the same encoding - otherwise, accented characters could not be used at all. If, say, I open as MacRoman a file that was stored using Latin 1, and *then* start typing accented characters, all hell breaks loose.

>> 
>> I then converted the file to utf8 (using Smultron) and replaced applemac by utf8 (\usepackage[utf8]{inputenc}) - and again opened the file from the open dialog, selecting utf8 encoding.
> 
> Is this the original file, or with the extra typed characters?

The original.

> Personally, I would look at the converted file with a command-line editor, such as vi , to see whether the bytes have really been converted correctly. The physical file size should have increased slightly, as UTF8 requires a byte sequence for each non-ASCII character.
> 

Sound advice - but as will explain below, by all indications the files are ok, and TeXShop handling of encodings has become broken by some changes in the underlying Apple libraries.

>> In this encoding the accented characters still did not display correctly, but still the file typeset ok.
> 
> Display correctly in which software? TeXshop's editor?
> What does it look like in other Mac editing software, such as TextEdit?
> Maybe there is a memory of the old encoding.
> Did you change the file's name for the new encoding?
> 


Perhaps I should clarify. In trying to troubleshoot this further I have created a new, little example file - using utf-8 encoding, but I guess I could have used a different one. 
This file has the  "%! TEX encoding ... " line at the start and "\usepackage[utf8]{inputenc}" command in it *and* typesets correctly (from command line AND from TeXShop), which indicates clearly to me that it is INDEED stored as utf-8.

It displays CORRECTLY in Smultron, TextEdit, TextWrangler, even in the Finder preview.
It displays INCORRECTLY in TeXShop - even though TeXShop is supposedly opening it as utf-8  (I am using the File-Open... and specifically requesting the utf-8 encoding from the menu, even though it should be unnecessary, since the file has the usual comment line at the beginning).

I did not even try to add more accented characters to this example file. I am sure it would be a mess.


>> 
>> Both versions of the file (using macroman or unicode) display and typeset correctly in TeXShop 2 on Leopard.
>> 
>> I think this is a serious bug in TeXShop in Lion (I am using 3.04, but the same problem occurs in 2.29 in Lion). 
>> Smultron in Lion (although an old version from 2009) has no problem handling the files. TeXShop 2 in Leopard also has no problem. The same flle displays and typesets perfectly in my wife's iBook.
> 
> That is disturbing, that newer software is not doing the same as in older versions.
> I'd guess that the change is in more recent Apple libraries, perhaps dropping some aspects of the support for really old methods.
> 

I am sure it must be, since old TeXShop 2.* also shows the problem under Lion - and 3.* uses the same code as far as encodings are concerned, Richard Koch told me. 
He has been traveling but was very kind to respond to my problem. I hope he can resolve this when he has a little more time.

> What happens if you take your wife's file and convert to UTF8 using iconv on the command-line, before ever opening it in any window-based program?
> I'd expect that now everything will look right in all programs. (touchwood)
> 

I don't have that file right now, but my example above seems to imply that it would show up wrong in TeXShop.

>> 
>> For now this makes working with accented characters (which are abundant in my language, Portuguese) a nuisance. I am reverting to writing stuff like \'a for á for the first time in years.
> 
> Knuth's original methods show their value!
> 

:-)

>> 
>> Luis Sequeira
> 
> Hope this helps,
> 
>       Ross

Thanks a lot!

Luis Sequeira