18. 11. 2013 Juergen Vigna NetEye, Unified Monitoring

Remake SW RAID1 from a new HDD and an old HDD with bad blocks

Question:

If you have a SW RAID1 and I just replaced, say, /dev/sda with a new HDD as the old one failed. Now, upon trying to recreate the RAID array you discover that the “good” HDD (/dev/sdb) has bad blocks which prevents mdadm from resyncing the array. While you could make backups, replace /dev/sdb as well and re-install the server completely, one is wondering if there is any way one could “trick” mdadm into resyncing the RAID array and then replace /dev/sdb with a new HDD. The badblocks should be located in an unused area of /dev/sdb which is only used when trying to recreate the RAID array, otherwise this data is lost to you.

Answer:

You should verify whether the affected blocks and underlying bad sectors on the disk are reallocated to “spare sectors” area? The bad sector should be reallocated when write operation fails. Verify it with smartctl:

[box]smartctl -a /dev/sdb | grep -i reallocated[/box]

The last column should contain a number of total reallocated sectors. If there is zero try to read the bad sector:

[box]hdparm –-read-sector XXXXXXXX /dev/sdb[/box]

It should return an I/O error otherwise I would recommend to skip next section.

The error means the sector was not reallocated yet. So you can try to reallocate it forcibly by writing it.

IMPORTANT: Remember that any data stored in this sector will be lost after this step !!!:

[box]hdparm –-write-sector XXXXXXXX --yes-i-know-what-i-am-doing /dev/sdb[/box]

By the way, the sector number XXXXXXXX should be possible to obtain from kernel messages (dmesg command or from /var/log/messages). As you had bad blocks during resynchronization there should be some related messages similar to:

[box]... end_request: I/O error, dev sdb, sector 1261071601[/box]

Then, try to verify it with smartctl again. Does the counter increased? If so try to read it with hdparm. Now, it should read it without any error as it is supposed to be reallocated. Done.

Finally, you can continue with mdadm and with adding the disk to your degraded mirror.

Juergen Vigna

Juergen Vigna

NetEye Solution Architect at Würth Phoenix
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.

Author

Juergen Vigna

I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.

Leave a Reply

Your email address will not be published. Required fields are marked *

Archive