I have also been having issues with XORG (that's the package Xserver and such is related to) breaking with 64 bit KDE Mint 16. In fact, I have had to re-do my desktop due to the XOrg suite breaking twice in the last three months.
It doesn't seem to have anything to do with graphics cards as I have had that issue nomatter what graphics card I was using: I suspect that Xorg is unstable in 64 bit Mint 16. Doing anything to update programs can cause it to glitch out with the rest of Mint 16.
Both breaks happened after adding programs to my system. The first break happened when I tried to update some programs and it decided to update Xorg along with them. I was doing things manually in terminal, so I blamed myself. The second break happened after installing Pidgin, which, again, also updated Xorg. This time I let Mint do the work by getting the program through Synaptic. Suddenly I realized it probably wasn't human error after all. I think it's because updating Xorg caused dependencey issues with the rest of my system.
After the first break, trying the upgrade and update commands helped, but weren't a cure-all. My system was riddled with errors that came up at the strangest times and over the smallest things. I had to re-install Mint 16 to fix all the issues I kept finding. Even though I recovered my desktop from the second break, I'm probably just going to
upgrade to Mint 17 now they have a stable KDE version and see if that makes things stop breaking.
The cool part about mint 17 is that it's not just a Long Term stable release: they are keeping the same program base for the next two years, so if we keep having issues like this there's a good chance it can be fixed by the developers! I'm also hoping that a more stable base (two years!) will mean less breaks.
I hope this helps. Good luck!
