Foobarred zfs filesystem
When replacing our new disks there were hard errors on the disk being reslivered from. (spot the error....)
root@bs2020:~# zpool status -v
pool: devel
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: resilvered 9.95G in 0h6m with 4 errors on Fri Nov 16 14:04:48 2018
config:
NAME STATE READ WRITE CKSUM
devel DEGRADED 10 0 0
mirror-0 DEGRADED 23 0 4
scsi-35000c50047d16807 DEGRADED 40 0 12 too many errors
scsi-35000c50047d0926f ONLINE 0 0 27
errors: Permanent errors have been detected in the following files:
devel/containers/naomi7oct2018:/rootfs/home/feurig/mailstuff.tgz
devel/containers/naomi7oct2018:/rootfs/usr/lib/x86_64-linux-gnu/gconv/UTF-7.so
devel/containers/naomi7oct2018:/old.rootfs/home/feurig/var/lib/lxc/naomi/rootfs/home/feurig/mailstuff.tgz
devel/containers/naomi7oct2018:/old.rootfs/home/feurig/var/lib/lxc/naomi/rootfs/home/don/Maildir/.INBOX.arduino/cur/1441292156.M90971P8887.bernie,S=8756,W=8933:2,Sab
pool: infra
state: ONLINE
scan: scrub repaired 0B in 0h0m with 0 errors on Sun Nov 11 00:24:23 2018
config:
NAME STATE READ WRITE CKSUM
infra ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000cca00b33a264 ONLINE 0 0 0
scsi-350000395a8336d34 ONLINE 0 0 0
errors: No known data errors
root@bs2020:~#
sync [pool] ...
root@bs2020:~# zpool detach devel scsi-35000c50047d0926f
root@bs2020:~# zpool status
pool: devel
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: resilvered 9.95G in 0h6m with 4 errors on Fri Nov 16 14:04:48 2018
config:
NAME STATE READ WRITE CKSUM
devel DEGRADED 10 0 0
scsi-35000c50047d16807 DEGRADED 40 0 12 too many errors
errors: 4 data errors, use '-v' for a list
pool: infra
state: ONLINE
scan: scrub repaired 0B in 0h0m with 0 errors on Sun Nov 11 00:24:23 2018
config:
NAME STATE READ WRITE CKSUM
infra ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000cca00b33a264 ONLINE 0 0 0
scsi-350000395a8336d34 ONLINE 0 0 0
errors: No known data errors
pool: devel
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Nov 16 14:49:37 2018
1.32G scanned out of 9.95G at 16.0M/s, 0h9m to go
1.32G resilvered, 13.31% done
config:
NAME STATE READ WRITE CKSUM
devel DEGRADED 10 0 0
replacing-0 DEGRADED 0 0 0
scsi-35000c50047d16807 DEGRADED 40 0 12 too many errors
scsi-35000c50047d0926f ONLINE 0 0 0 (resilvering)
errors: 4 data errors, use '-v' for a list
pool: infra
state: ONLINE
scan: scrub repaired 0B in 0h0m with 0 errors on Sun Nov 11 00:24:23 2018
config:
NAME STATE READ WRITE CKSUM
infra ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000cca00b33a264 ONLINE 0 0 0
scsi-350000395a8336d34 ONLINE 0 0 0
errors: No known data errors
root@bs2020:~# zpool status[ 9307.615155] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9309.788008] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9312.115335] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9313.886154] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9315.603474] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9421.337403] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9424.263000] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9426.668087] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9428.468338] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9430.192036] print_req_error: critical medium error, dev sdc, sector 45059769
[ 9502.892589] print_req_error: critical medium error, dev sdc, sector 99975477
[ 9505.406331] print_req_error: critical medium error, dev sdc, sector 99975477
[ 9628.405254] print_req_error: critical medium error, dev sdc, sector 106690474
[ 9630.596015] print_req_error: critical medium error, dev sdc, sector 106690474
[ 9638.557126] print_req_error: critical medium error, dev sdc, sector 107074277
[ 9641.058573] print_req_error: critical medium error, dev sdc, sector 107074277