additional texmf tree for all users (was Re: [OS X TeX] Beginner help with TeXshop/MacTex needed)

[Gerben Wierda]

(Down below I have a new idea what might have happened in May 2005. Please
read carefully to the end)

>> On Aug 26, 2006, at 12:56 , Rowland McDonnell wrote:
>>
>> > (I am very worried about i-installer overwriting my data - in my
previous attempt to install and use a modern Mac TeX (before MacTeX
was put together; I tried using i-installer, and abandoned the
attempt after some weeks of trying to crack it), I did find that
i-installer threw away several hundred megabytes of data I'd
>> > downloaded (mostly via my dialup line), which did make me pretty
unhappy.  It's not safe software to use if you ask me - I intend to
do a full backup before I run i-installer, *every* time I run it. I
have the disc space to do so now.)
>>
>> It is generally better for me to ignore messages/threads like this,
>
> <rolls eyes>
>
>> but for the record I want to state that the TeX i-Package does not
overwrite *any* data, except what is in /usr/local/teTeX (and even not
everything there:
>
> In other words, the TeX i-package overwrites the region where all the
texmf trees are kept.
>
> You've expressed yourself in a very misleading fashion.

Misleading is not a friendly word, why have I deserved this? Having said
that, the TeX i-Package tries very hard only to destroy data it has
written itself. Hence, it will only delete stuff in /usr/local/teTeX (and
/var/tmp/texfonts, the temporary data cache and that only when you decide
to remove TeX entirely form your system).

> Is this documented anywhere?  I'd like to be able to find out this sort
of thing myself, you see.

Try the README tab of the TeX i-Package. There it says:

On the layout of my TeX distribtion (foundation) (important when you want
to add files yourself):

I use four (five) trees where files are stored:
	1. The first is /usr/local/teTeX/share/texmf; this is where
program-related stuff is stored and it is upgraded when the TeX programs
are updated.
	2. The second is /usr/local/teTeX/share/texmf.tetex; this is the original
texmf tree maintained by Thomas Esser, and is kept as default as possible
so it can be upgraded without destroying local modifications.
	3. The third tree is in /usr/local/teTeX/share/texmf.gwtex and contains
Mac OS X specific settings and additional packages not in teTeX's
foundation distribution. This tree is maintained by me.
	4. The fourth tree is in /usr/local/teTeX/share/texmf.local and is for
the local TeX administrator (normally, you) to store system-wide
additional files. To store in that area, you need System Administrator
rights.
	5. In addition, individual users can store their own files in
~/Library/texmf.
When you reinstall the TeX i-Package, files in the first three trees are
overwritten. [[In the README this is in red]] But files in
/usr/local/teTeX/share/texmf.local and in user's home directories are
normally left left alone (the exception being setup files created there by
automated setup programs, which would break an upgrade if they remained in
place). See http://www.rna.nl/tex.html for more information on adding
files.

> And what about the other i-packages?

Same thing. They will only remove what they plan to install or have
installed. Most of them install stuff in /usr/local.

Note that by listing the content of the archive (Pkg Contents tab) you can
see what is inside.  By inspecting the preparation and removal scripts,
you can exactly see what will be removed. As far as I know, there is no
installer program more open with information like this than i-Installer.

>> unless you explicitely tell the TeX i-Package
>> when uninstalling TeX, it will leave texmf.local alone), so if you want
your data overwritten you will first personally have to move you stuff
there.
>
> I find this confusing.

If you copy your holiday photographs to /usr/local/teTeX/share/texmf.tetex
and reinstall the TeX i-Package, it will remove your holiday photographs
because it removes /usr/local/teTeX/share/texmf.tetex and all its
subdirectories before installing a new one.

> How do you `tell' the TeX i-Package to leave or not leave texmf.local
alone?

texmf.local is potentially affected when you run a uninstall of the TeX
i-Package. If you do run "uninstall" on the TeX i-Package it will ask you
what you want. You can choose what to delete. You get a panel and you
answer the question.

Also, the results of certain configuration steps (like compiling maps and
formats) live in texmf.local. These have to be removed at
(re)configuration because there inputs may have changed. So, configuration
reruns updmap, fmtutil, etc. (unless you tell it not to in expert mode) so
that your freshly updated files are actually used. This is necessary and
cannot be prevented. Otherwise, you would have new improved map files but
still use the compiled ones from before the update.

> Now, it seems at first glance that the sensible way of getting a local
texmf tree accessible to all users is to put it somewhere in:
>
> /usr/local/teTeX
>
> but since an i-installer might mess about with it, it seems to me that a
more sensible approach would be to put it in a directory that no
i-installer would think of looking it.

If you feel more comfortable that way, I would think that is a good idea.
As I wrote earlier, /Users/Shared/texmf.rowloandmcdonnell would be a nice
place. If you put your private system-wide stuff there, you can be as
certain as humanly possible that no i-Package I have ever written or
currently in existence will touch it.

> Or could one create a directory (say)
>
> /usr/local/teTeX/share/texmf.rjmm
>
> and be reasonably sure that no i-installer package would touch it?

Yes, you would be pretty sure. In fact, if you would tell the TeX
i-Package to remove TeX from your system, it will try to remove
/usr/local/teTeX and fail because the directory is not empty.

But a /Users/Shared/texmf.rowlandmcdonnell is safer still because if I
would make a mistake in the future it is very unlikely that the mistake
would remove that directory.

>> The data the TeX i-Package overwrites/removes in /usr/local/teTeX it
*has* to if you do not want to risk a nonfunctional mess of old and
updated TeX when you update your i-Package and install it again (think
about a renamed subdirectory, like 'dstroke' being renamed to
'doublestroke' and you would install doublestroke but not remove
dstroke, which file would be used by TeX? I am not able to tell you)
>
> So the TeX i-Package deletes some data and adds others.  Of course.  How
would a maintainer of a TeX installation on a Mac discover what is going
to happen on running an update?

How do you check that with an Apple .pkg? You can't (not easily and
without the command line at least).

Anyway, with i-Installer you can read the scripts and read the listing of
the i-Package's archive contents. It is not perfect, but it is something
(and more than any other GUI installer I know of)

>> What i-Installer *can* remove or overwrite is if you thin or update
your i-Package (in ~/Documents/i-Packages normally) and you *always*
get a warning panel before any of that will happen. My guess is that
this happened to you May last year,
>
> Your guess is as wrong as wrong can be - as I have explained several
times recently.  It's a shame you've not believed me, since you're the
person who's supposed to make sure i-installer is safe to use.
>
> What i-installer did was downloaded a large amount of data in multiple
packages.  I did not install it - I just grabbed the packages.  I then
tried to grab another one, and whoosh!  All the previously downloaded
data had been deleted.

There is a problem of understanding on my side here. You "just grabbed"
the i-Packages (or "another one"). But i-Installer does not have an
interface to do that unless you mean opening one from the Known i-Packages
window. It's design is about handling downloads in the background and
letting the user concentrate on what he/she wants to do (opening -
installing/configuring/removing - closing).

"Downloading a large amount of data in multiple packages" is not something
that is in i-Installer's user interface. So, either you downloaded using
other means or you do not remember exactly what happens. It is as if you
are telling a car maker that you flew your car somewhere and it crashed.
The car maker will not understand because his car is incapable of flying.

But maybe you opened a lot of i-Packages from the "Known i-Packages"
window. For any substantial download to happen you would have had to
install of fatten all of them *individually* (one by one).

So, in my case, I try to match your experience of loss of data with the
only ways i-Installer does data removal. And there are only three. In the
entire source code of i-Installer there are only thee places where
downloaded data is being removed:
1. When you tell i-Installer to "thin" an i-Package
2. When you update your i-Package (outdated contents inside the i-Package
are removed)
3. When after a download of some internal part of the i-Package the MD5
checksum or GPG signature is wrong. The downloaded data is then assumed to
be corrupted and it is immediately removed to protect you from possible
harmful content (think someone intercepting your traffic and exchanging
the configuration script you are downloading with something nasty). The
third way points to network/proxy/cache problems. The others are part of
the design of the program.

Basically, there is *no* other code statement in i-Installer that removes
*anything*, except...:

there is one other way you can lose track of your i-Packages. If you go
into i-Installer Preferences and change the default i-Package location
(the location on your disk where i-Installer creates new i-Packages from
remote repositories by default, normally ~/Documents/i-Packages),
i-Installer will try to move your i-Packages to the newly entered location
(move, not destroy). So, that is another thing that might have happened.
Also, having had experience with many users and many reports, I know that
even slightly damaged OS X systems may have strange behaviour. E.g. if
your permissions are wrong on your system disk, suddenly seemingly
unrelated stuff inside Apple Frameworks may stop working, including but
not limited to file system interaction. A large chunk of problems I have
had reports on were of this nature and fixed by repairing permissions on
your system disk.

> There was *NO WARNING AT ALL*.  There was no installation.  Just
downloads, and data loss.

Well, I have a problem with that. There are no code statements in
i-Installer capable of that. Now, here is my problem:
- Your description does not match the user interface of i-Installer - Your
description does not match what I know (and everybody can see) what is in
the source code of i-Installer
- Your experience (as others have told you) is unique.

I am not saying you are lying. You experienced data loss. I accept that
for a fact. I just do not accept your description on how i-Installer
performed that feat (without warning during downloads).

>>that you painstakingly downloaded
>> the TeX i-Package over a slow link and it got updated on the
>> reprository and you had to download it all over again when you
>> accepted the upgrade.When you accepted the upgrade, it did not tell you
that you had to download it all again. Which was bad luck, bcause those
upgrades are not that often. TeX is huge, and if it changes the
re-downloads are huge too. There is currently not much I can do about
that.
>
> I'm afraid I do not understand the above paragraph.

It is not the case, I guess, but here is the explanation:

If the repository was updated during your download your already downloaded
internals of the TeX i-Packages may have become outdated, or the newly
downloaded internals (like archive chunks) may not have matched the
checksums that were in your older i-Package table of contents. In that
circumstance, i-Installer would download megabytes and then remove the
donloaded data immediately because it does not match the checksum.

>> Having said that, it kind of irks me that this message (which could
scare people without there being a good reason for them to be
>> scared).
>
> What irks me is that you are insistent on not believing what I tell you,
and you insist that what happened was something other than what actually
happened.

My problem is that you may have exprienced data loss (again: I believe you
there) but that your conclusion that i-Installer is to blame is by and
large impossible the way you tell it happened (as far as you give details)
because that does not match (a logical NOT) what i-Installer has ever been
capable of doing.

So, I propose a truce: I accept you experienced data loss (I do anyway)
and you accept that i-Installer is incapable of doing what you describe,
just like Photoshop is incapable of performing Excel spreadsheet
functionality. Still, i-Installer may still have been responsible (there
might have been a bug), but not how you have described it.

(Here is one reason I carry on with this: I go to pretty extraordinary
lengths to check bug reports, including those where I get flamed. It has
to do with not wanting to provide buggy software ;-) So as long as there
still is a potential issue with i-Installer, I am interested in finding
out what it is.

>> None of what was removed/overwritten was your personal data
>> (but only inside the TeX i-Package) nor was any of it destructive for
your system or purpose. You just had to let i-Installer download the
updated version.
>>
>> > Since i-installer does blindly over-write things without warning - well,
>> > of *course* it'll delete data you want to keep.
>>
>> i-Installer does not blindly over-write anything without warning and
since it is open source people could check.
>
> Yes, but I have no reason to believe you.

Interestingly you maintain that I must believe you but you must not
believe me. Rowland, i-Installer has never been capable of the action
(removing i-Packages during download) as you describe. In fact, on my wish
list is an option to remove an i-Package. As it is now, you can thin an
i-Package, upgrade it and use it (install/configure/uninstall), but you
cannot remove it. If you want to remove it you have to drag it to the
trash. And you can move your default save location in i-Installer
preferences (that is *not* the install location of an i-Package
properties!) and if the new location is on another volume this will
include copying and removing a lot of i-Packages.

>> Rowland, please download
>> the sources from SourceForge and tell me where it does so
>
> Don't be ridiculous.  You know perfectly well that I cannot check your
source code.  You also know that your claims about what I did are purely
your invention - frankly, you're trying to harm my reputation.

They are invention, yes, but not intended to harm your reputation. They
are the best explanation so far I could invent for what you may have
experienced. They may be wrong and if you actually lost downloaded
i-Packages (like foo.ii2 disappearing from your local disk) they are
definitely wrong if your description of what you did is correct.

> What I'm telling you is that i-installer threw away a lot of my data.
You have chosen to disbelieve my report.  Of course I dare not use your
software - it's not safe.

I have no problem believing that you lost data. I have problems believing
your conjecture that i-Installer did it the way you told me (without
warning and doing the things you say you were doing). I can back up my
claim in various ways, some of which are even falsifiable.

>> and I'll
>> eat humble pie and thank you for helping me fix the program. Until
then, it would be friendly to refrain from making these remarks.
>
> I think it would be sensible of you to stop insisting on your guess as
to what happened, and pay some attention to my reports.

Well, Rowland, I paid an inordinate amount of attention to your reports. I
might be forgiven that they had bits like ("I tried to install TeXShop
with i-Installer" (there is no such i-Package) or "You are the author of
iTeXMac") But maybe some extra attention has helped (nobody is perfect,
certainly not me). Here is my new idea:

Do you remember that you tried (from the start) to install TeX in a
non-standard location? Might it have been that you tried to do this by
changing the i-Installer default i-Package *save* location instead of the
(TeX) i-Package *install* location? One of your reports of May 2005
contain an image of a warning panel that the move of freetype.ii2 has
failed. That might have been a bug in i-Installer (or an OS X problem)
when handling the move of all already downloaded i-Packages at that time.
And a failure with the change of default i-Package directory is also the
only way in which multiple i-Packages are affected in one operation by
i-Installer. So it is the *only* possible way i-Installer could have been
involved in the data loss of all your downloaded i-Packages.

So, maybe it was like this:
- You changed the default save location in i-Installer preferences.
Somehow that failed (either i-Installer bug or OS X problem and it might
have been triggered by your exact naming of the new location, maybe
somehow string manipulation or file manipulation failed). You may have
been trying to change the install location (which is what you were after
at the time).
- You did get a warning (so not, "NO WARNING AT ALL", Rowland and I am
quite certain of that because you mailed the image of the warning panel to
me in May 2006 "Error I cannot move
/Users/rowland/Documents/i-Packages/freetype.ii2 to (null). Cause: Already
exists."). Something there really went amiss. The new location is a null
pointer, so the Apple method of transforming what you entered into a file
system representation failed. Already exists is a very weird error
statement from the OS when the pointer is (null). It will be impossible to
find out what caused tihs to go wrong and it is clear that i-Installer did
not take into account that this string manipulation by Apple frameworks
could fail. Or the system *was* damaged in some way (still a possibility,
even if you repaired permissions).

Why did I miss this earlier (if I did, I did no search in my mail to see
if I actually mailed this in 2005)? Two reasons:
- The data loss statement was in one message (saying it was without
warning), the error panel in another (and entirely at the end without a
reference in the text)
- The message with the error panel was full of remarks at the wrong
address (how superiour OS 9 was to OS X etc) and I just had spent 4 entire
hours that day helping you (and only you). Hence, I was a bit irritated
and I might not have read your message entirely to the end. I cannot
recall exactly.

Basically what this means for me is that I am going to analyze the "move
default save directory" code once again to see if this coul dhave
originated in i-Installer and if not I must assume it was an OS problem
that made i-Installer fail. GIven the sensitivity of moving i-Packages
around I might also put in extra safeguards against OS trouble.

And lets be clear: the only data you ever lost was stuff you could
download. You never lost anything irreparable. In fact you did lose a copy
of my data, not your data ;-)

G



------------------------- Info --------------------------
Mac-TeX Website: http://www.esm.psu.edu/mac-tex/
          & FAQ: http://latex.yauh.de/faq/
TeX FAQ: http://www.tex.ac.uk/faq
List Archive: http://tug.org/pipermail/macostex-archives/