How to recover files from a deleted or even overwritten partition

First of all, this is something that happened to me and, it is just a post about how I recovered myself from the mess. You are gonna slap me if I tell you how idiotic I was to loose all my data anyway. I was updating my Linux Mint 14 laptop with Linux Mint 15 RC, which has been released yesterday by a fresh reinstall. As usual I backed up all my files in the /home/napster folder, which is nothing but my home folder, to another spare PC here and since all of my critical documents are on Dropbox and Google Drive, that part was pretty safe. Before starting gone-deadly installation, my TOSHIBA-250GB HDD looked something like the following.

The partition labelled Data was the central place where I kept all my stuff like movies, music, photos and e-books and everything as such. And this drive was 200GB in size, and it was around 99% occupied. As you see this is a pretty deadly combination that, if you accidentally lose the data, there is not much space left to stand up and do the recovery from within the disc. Anyway, this is what I did in the apocalyptic re-installation. First I booted the machine with a Linux Mint 15 RC CD-ROM and while the installation process, I have chosen Replace Linux Mint 14 with Linux Mint 15. This was somewhat a very bad thing to do, although I didn’t expect all my partitions will be lost. After the installation, I have immediately noticed that all my partitions and data are lost, forever. This is somewhat the HDD looked like after the re-installation.

I was stupid enough to use the new installation for a few more minutes and install a few applications like Chrome Browser or the VLC player. Then I realized that the situation is real worse. With each apps being used, and with each app being newly installed, I’m technically overwriting one file or the other in my old partitions. I have stopped immediately, turned off the PC and looked around for solutions. I was pretty panicked and didn’t know what to do. I have headed to some IRC chat rooms and most of them were pretty sure that, I’m never going to get those data back.

I paused for a while and the next thing to do was to audit the damages. I have thoroughly tried to recall what files did I have on the machine before loosing the partitions. I have listed all those I could recall and analyzed which of them are potentially high priority files and which of them I could afford to loose. That opened little bit of a room to settle down for a while. Because most of the files where not that critical and to be honest, almost with any of them, I could live in the absence of them. So by the evening I decided not to approach any professional companies to do this recovery process, and assigned myself to give it a try.

Boring Story is over, You may read from here on.

For the sake of safety, I have gathered a 500GB external HDD. I thought I could use it in case if I need to profile the entire hard disk to an image. The most important tool to use was testdisk. This is an elegantly written open source software, which is the only tool I used in the entire process of the recovery. I have found a lot of threads discussing data loss due to the same incorrect Linux installation attempts. But none of them lead me to a proper recovery solution. Most threads where flooded by commercial data recovery companies and their sucky ads. So, I have found testdisk on the Ubuntu’s data recovery guide. But it was not at all documented there. I have setup a virtual machine quickly to get myself familiar with the interface of the testdisk tool. It is infact a 100% command line tool, with menu like interface much similar to what is provided by parted or such tools. I don’t know why I choose testdisk since the other two tools (one was gparted - I knew it was of no use) were documented there properly. Anyway I headed to the testdisk wiki and learned how to do the Deep Search process for the lost partitions and how to write the partition table back to the disk itself. The following were the major steps in the process.

Boot the machine from a Live CD. I used Linux Mint 15 RC CD-ROM itself, since it was already available, as you would guess. Please note that the entire system is in the RAM when booting live, so the recovery process depends on the available RAM you do have. If possible, extend the RAM before initiating the recovery, or move the HDD to another PC with faster/larger RAM, although I’m not sure the later one is a good idea.
Install the testdisk application. Please note that the installation goes to the RAM as well. So its better not to run any other apps from this Live CD for the sake of speed. Now, run the tool sudo testdisk
Select the correct HDD first. If you have multiple hard disks on your PC, you can identify the correct one with the name or the size of the HDD.
Select the Quick Search and proceed, and this will list you the new partitions, I mean the current available ones. In my case, the one with Linux Mint 15 RC and its 2GB swap.
Select the root partition from the result and proceed to the Deep Search. This process is going to take a while, in my case it was around 1.5 hours.
After the Deep Search, you will be getting a list of all deleted partitions that could possibly be recovered to some level whatsoever. It is not at all guaranteed to recover any files with any level of certainty, I mean, it depends on your luck as well. Anyway, from this list of partitions, correctly select the one you need to recover. Strangely, in my case I got two partitions with the same label Data. I’m not sure why 2 of such entries appeared.
I selected one of them and pressed Enter to proceed. Now, the partition is displayed with all details about the drive. I could navigate through the file system by using the ⬅ and ➡ arrow keys. You may find very old and previously deleted files on the drive in red color, which could not be recovered. Once the drive is properly identified, proceed to the next step.
Change the partition type from D to P or L. D represents deleted, P represents Primary and L represents Logical partitions. If it was a primary drive, you may not be able to change the type to L at all. Once this step is done, press enter to proceed.
Now, testdisk will ask you if you want to write back the partition table or not. Choose Write and confirm if asked again.
Reboot the PC to complete the process.
Once rebooted, the partitions will be restored and the files in it may found to have incorrect permissions or even some files or folders cannot be accessed at all. Check chmod command to see how you could fix this problem.

Ok, so that’s the story, I have executed all the steps above, and successfully recovered almost all the files without being corrupted. I faced some boot related issues after the recovery, but I’m not sure yet if it is caused by the recovery process or not. Also, the most important files were recovered and so, a bad day became a good day afterall for me! :)