[Solved] Aptitude disaster... Noob mistake

[mdevour]

I need help to recover and/or rebuild my system. I figure if nothing else, this thread can serve as a cautionary tale for others, or at least be of some entertainment value to the competent.

I'm running Mint 17.1 Mate on a middle-aged HP desktop.

I was following a tutorial that walks you through compiling an application from source. I'd reached the step where you update your system and install some dependencies. The instructions were to run the following in a terminal:

Code: Select all

sudo aptitude update
sudo aptitude upgrade -y
sudo aptitude install -yr [a list of packages here...]

NOT THINKING... I looked at this, said to myself, "Okay, self, they want you to update your system and install these packages. Well, my system is up to date as of about a half hour ago, so I'll just run the last line and install those packages..."

Well, when it started removing some packages, I got concerned. When I saw it start removing packages I KNEW were important parts of the system, I realized I'd done something seriously wrong... After a bit of yelling that brought my wife to the top of the basement stairs asking if I was okay, I made the decision to NOT interrupt the command, for better or worse, for fear of leaving it in a totally unrecoverable state. I don't know if that was right or not, but it's what I did.

So, I'm left with a system that's more or less running. No icons on the desktop, but the panels are still there and mostly populated. It'll go into screen saver and come back. The Mint menu comes up, but I don't know yet how much of it works, since many of the packages have been uninstalled. Which includes networking, so it is out of communication with the world for the moment.

I have not powered down or re-booted yet, in case there's anything worth preserving in its current state. I cut and pasted the terminal output for the session into a text file and saved it, and that's about it.

I have the Mint 17.1 Mate install media at hand on a USB stick.

Okay, obviously aptitude knows nothing about the apt configuration if you don't update and upgrade before trying to install stuff. It didn't take me long to recognize more or less what I'd done once I engaged my brain again.

A few initial questions:

Apt's list of installed packages ought to still be there, right? If I can mount the hard disk from a Live Boot session I ought to be able to get ahold of it?

Can I reinstall a base, working system without overwriting all of my previous data and configuration files? Or restore them after reinstalling, given that they should be recoverable from the Home directory?

Please have mercy on this benighted soul. I've been using Linux for a decade now and never have I pulled such a dumb stunt before! Please help me put my system back together, if it can be done.

Thank you.

Mike

[Pjotr]

Nasty experience...

The user settings are in hidden files and folders in each home directory. You can make them visible in your file manager by the key combination Ctrl h (they all have a dot before their name) and then copy them onto a USB thumb drive (for instance). These you can put back in a new fresh install.

By the way, and rather off topic: you should use consistently either "apt-get install" or "aptitude install". So if you choose one, never use the other. Because otherwise malfunctions might occur under certain circumstances. I myself stick to apt-get, because that's simply the most commonly used method.

[Cosmo.]

I think it is the best, to make a new installation of the system. As you feared yourself, after rebooting the machine things can get even worse, inclusive the possibility that get stuck.

As in every such case: Make a backup of your complete home folder on an external medium. Than you can use this tutorial to install anew without loosing your settings.

And as Pjotr told already: Don't mix apt-get and aptidude, either the one or the other. As to my knowledge Synaptic, the software-manager and the update-manager are front-ends for apt-get, I would not use aptidude (which does not mean, that there is anything wrong with it).

[karlchen]

Hello, mdevour.

Not your main concern right now, so only a minor correction:

Okay, obviously aptitude knows nothing about the apt configuration if you don't update and upgrade before trying to install stuff.

This assumption is incorrect. Software Manager, Update Manager, Synaptic, apt, apt-get, aptitude, they all use exactly the same locally cached list of available software packages and they have the same knowledge about the software packages which have been installed on your system.

The dangerous detail in your aptitude commandline was the the option -y.
As "man aptitude" will reveal the option -y is equivalent to --assume-yes and will answer all questions which aptitude might ask you with "yes" automatically. I.e. when aptitude detected that installing the long list of software packages meant removing a lot of other packages it did not have to ask for your consent, because you had already consented.
Considering the recommended packages as dependencies, option -r = --with-recommands, may or may not have made the situation worse.
But the really careless thing to do was consenting in advance by using the commmandline option -y. (The same will apply to apt-get as well.)
Something to remember in future whenever you come across commandlines holding an option that reads -y.

Cheers,
Karl

[xfrank]

I had a similar disaster some month ago in a laptop with LM17. Trying to downgrade a key package (fontconfig) in the terminal, apt informed me about a large list of dependencies and the need to remove a lot of packages. I copied the list of packages that will be removed and saved it in a file. Then I did follow (a bit naïvely, but mostly curious) apt in the painful uninstalling process and I ended up with HALF the system removed!
Well, I was able to restore all with patience step by step, installing each packages one by another (more than 400). It took me several hours, but I did.
If you find the list of the packages that were removed, you (probably) can restore your system.

[mdevour]

... you should use consistently either "apt-get install" or "aptitude install". So if you choose one, never use the other. Because otherwise malfunctions might occur under certain circumstances. I myself stick to apt-get, because that's simply the most commonly used method.

Yep, and I KNOW this... I just didn't happen to think about the fact I was about to execute an aptitude command on my system that's routinely maintained with apt. I was too focused on the project at hand.

... use this tutorial to install anew without losing your settings.

That's a good one, Cosmo. Enough of the system files have been removed that I cannot do all of the steps, but I should be able to piece together a useful procedure and workarounds. I'm studying it.

Software Manager, Update Manager, Synaptic, apt, apt-get, aptitude, they all use exactly the same locally cached list of available software packages and they have the same knowledge about the software packages which have been installed on your system. ...the really careless thing to do was consenting in advance by using the commmandline option -y.
Something to remember in future...

So it looks like I unleashed a dependency avalanche, perhaps because one or more of the packages represented a downgrade, that brought in another, and another, and continued until over 300 packages were removed for lack of dependencies.

I will certainly be more careful next time!

If you find the list of the packages that were removed, you (probably) can restore your system.

Well, I have the terminal session saved in a text file. From that I've been able to glean an idea of what happened, and found out that about 360 packages were removed.

Using various recipes I found for scanning the logs, I've come up with a partial list of packages I installed manually. Anything that's still there and functional, at least, which seems to be most of my added programs. I can recall a couple that have been removed and aren't listed, though, so I'm not getting the whole list.

I'm working through the steps of the tutorial Cosmo cited, and will try to use that to recover my user settings after a re-install. I should be able to feed at least the partial list of my added applications to dpkg to batch-install them and save some time.

I think that also going through my home directory and identifying the source of all the "." files should let me pick up most of the rest of the programs I've missed.

So once I've gotten everything I can out of the zombie session that's still running, I'll re-boot with the USB stick, move and shrink the partitions that hold the current install so they're squashed off at the end of the drive, then do the new install. Hopefully, I'll end up with a working install with ready access to my old files for copying over and any further log analysis I need to do. Happily the drive's got enough empty space!

Thanks for the help, everyone. Any and all input is welcome. I'll be back with further questions or a progress report.

Peace,

Mike

[grizzler]

Using various recipes I found for scanning the logs, I've come up with a partial list of packages I installed manually. Anything that's still there and functional, at least, which seems to be most of my added programs. I can recall a couple that have been removed and aren't listed, though, so I'm not getting the whole list.

Did you look at /var/log/dpkg.log (and possibly /var/log/dpkg.log.1 and any gzipped older versions, if necessary)? That should list them all. Run a grep or sed with 'remove' on it to get a 'clean' list.

[mdevour]

...I've come up with a partial list of packages I installed manually. Anything that's still there and functional, at least...

Did you look at /var/log/dpkg.log (and possibly /var/log/dpkg.log.1 and any gzipped older versions, if necessary)? That should list them all. Run a grep or sed with 'remove' on it to get a 'clean' list.

Yes, that's good, Grizzler. Grepping dpkg.log gives me a list of about 320 things that were removed. Things I installed consciously at some point are fairly easy to pick out. Between this and the list of packages still installed I should be able to cobble together darned close to the original configuration. Once all of my packages are installed and a few other tweaks done, I should be ready to finish the job by replacing the new /home/user directory with the old one and get my settings and data back.

It's clear I have to be careful to install only top-level packages with these lists, so that the system will be able to track dependencies properly.

At least that's the theory! I love Linux, but command-line arcana are NOT my favorite subject!

Peace,

Mike

[mdevour]

I'd like to report on my progress in rebuilding my system:

http://forums.linuxmint.com/viewtopic.php?f=46&t=192635

... and have a question, before I attempt to finish the process.

I used gparted from the Live USB to create a copy of my main partition at the tail end of the hard drive. So, I've got a complete backup of everything that's left of the old install, including my /home/user directory and all log files, in case they're still needed to reconstruct my former package list.

Part of the system that was uninstalled included the Mint Backup utility and all the gui package management stuff, so I could not create a backup of the machine state in that fashion before shutting down the zombie session. Instead I used various command line actions to preserve the list of installed packages that remained, plus a record of those that were removed by the disastrous aptitude command.

Once I have a clean list of packages to reinstall, I hope to feed it to dpkg or apt to reinstall them all at once.

The order of steps in the tutorial that Cosmo linked to suggests that the Mint Backup tool would non-destructively restore the old packages, reusing my pre-existing configurations and data, but I don't have an "official" backup to restore, nor do I know what facility it's using as a back-end.

My question: When installing a package via apt or dpkg, will it overwrite .config files for the program that already exist in my user directory? Are their any command-line switches or other gyrations I need in order to preserve (or restore) my configurations?

Getting close, here. Thank you for all the help so far!

Mike

[grizzler]

My question: When installing a package via apt or dpkg, will it overwrite .config files for the program that already exist in my user directory?

Short answer: no.

Longer answer: a package built to (Debian/Ubuntu) standards should never touch anything in the user's home directory during installation. The application installed with the package may create user specific settings there once it's started, but if it finds any that were created previously, it should simply use those.
That's the theory. Does every package/application behave itself in that respect? I have no idea...

[Cosmo.]

Important: If you copy partitions with GParted and place them on a internal hard drive you must change the UUID of those copies. Otherwise Linux gets confused, that there are 2 partitions with the same UUID and fails to boot. Changing the UUID can be easily done with GParted, look at the context menu of the copied partitions.

Regarding restoring the software selection: Of course there must have been created a backup as described in the tutorial in step 2.a.

Regarding the user configuration:
As grizzler already wrote, mostly existing settings do not get changed. But mostly != never. A change of the settings may appear during the update (rather unlikely) or during the start of the new version for the first time. At the moment I know only one example for this: if you upgrade from LM 17 to LM 17.1 some (not all) of the settings get restored to default values.
The way to protect against that: Make regular backups. I make a daily backup of all hidden files and folders (excluding parts of them as the Trash or the cache) automatically with Back in Time and I do not even notice, that this happens, because this goes so fast.

[mdevour]

...a package built to (Debian/Ubuntu) standards should never touch anything in the user's home directory during installation. The application installed with the package may create user specific settings there once it's started, but if it finds any that were created previously, it should simply use those.
That's the theory. Does every package/application behave itself in that respect? I have no idea...

So far so good, grizzler!

Following the general outline of Cosmo's recommended tutorial, I rearranged the partitions to put /home on one of its own, then reinstalled the system again. I'd always arranged things that way before, but decided when I installed this system to accept the defaults. I then moved a backup copy of my /home/user data into the new /home/user directory.

Finally, I booted into the new install. I cannot tell you how much of a relief it was to see my familiar desktop configuration re-appear, finally!

I only have the one user, me, so I didn't need to restore any other user files. I confirmed the software sources were updated. I then demonstrated that just copying the contents of /etc/apt won't work to restore your PPA's, and then did as directed to re-install them from their sources on the internet. Then I used the Update Manager to bring everything up to current versions.

What was next was to reinstall all the rest of my packages. Without a backup of the old package list, I had to spend a good while synthesizing one from the data I had available. I had one list of the packages still installed after the disaster that were not part of the original installation of the system, that I'd generated from some clever command-line tricks I found in posts on an online help forum. Then I had the list of packages that had just been uninstalled by the fatal aptitude command, courtesy of the dpkg log -- which required cleaning up with some basic sed commands.

Once I had both lists reformatted cleanly, I manually culled out anything that was obviously a dependency of some other package; libraries, for instance. I went through both lists one at a time, putting the package name in the filter field of Synaptic and eliminating that item from the list if it was part of another package to be installed or already installed on the system. I finally concatenated the two lists, sorted them together and removed duplicates, and had my final master list of about 120 packages.

After a bunch of head-scratching and delving into the depths of arcane linux commands, I arrived at a relatively simple way to batch install the packages. I edited the list to replace newlines with spaces using the search/replace function of the text editor, then added the command I wanted to execute before the list of packages... As a preliminary test, I made a new file something like this:

Code: Select all

apt-get -s install package1 package2 

... with a couple of small packages I knew were on the list. I chmod +x'd the file, then ran it first as user, then using sudo, and the simulations appeared to run just fine.

I then swapped in the full list and ran the simulations again. A couple of packages were not found, a couple more were already installed, so I eliminated them from the script. It didn't take long to get a clean run without error messages. It would be installing over 300 packages and dependencies... an amazingly similar number to that which had been removed!

Taking a big breath, I removed the -s from the script and re-ran it with sudo. It took a long while, but I sat watching it and saw nothing amiss but a couple of warning messages that seemed harmless. It didn't fill up memory or disk space, or otherwise choke or explode.

Lo and behold! Everything I've tried so far appears to be back and functioning. I had to reinstall Google Earth, tweak Firefox to get Flash working again properly, and a couple other minor quirks, but nothing I'd object to after all I've been through.

All I've got to say is, WOW! And thanks for the help, folks...

Important: If you copy partitions with GParted and place them on a internal hard drive you must change the UUID of those copies. Otherwise Linux gets confused, that there are 2 partitions with the same UUID and fails to boot. Changing the UUID can be easily done with GParted, look at the context menu of the copied partitions.

THANKS for pointing out this detail, Cosmo! I'm sure it would have tripped me up and wasted a bunch more time. Indeed the two copies had the same UUID. The dialog box that you can bring up from the context menu gives you the choice to generate a new random UUID. Problem solved!

Regarding restoring the software selection: Of course there must have been created a backup as described in the tutorial in step 2.a.

Which wasn't possible AFTER the disaster, when among everything else, the Mint Backup tool wasn't there and the packages I would have wanted a list of had just been uninstalled anyway. OOPS!

The way to protect against that: Make regular backups. I make a daily backup of all hidden files and folders (excluding parts of them as the Trash or the cache) automatically with Back in Time and I do not even notice, that this happens, because this goes so fast.

And this experience has certainly driven home the importance of doing just that. Even if I'd just backed things up regularly on another partition of the HD I'd have been able to restore the system to its previous state with minimal grief.

So, I've still got to reinstall my Brother printer drivers and check a few more things, but all in all, I believe I'm just about back to where I was. Next, actually impliment a backup strategy!

Thank you, again.

Mike