Conversation

Notices

1. nixCraft 🐧 (nixcraft@mastodon.social)'s status on Saturday, 02-Mar-2024 01:31:17 JST nixCraft 🐧

   Your indexes in MySQL or Postgresql have become corrupted, and as a result, your backups now contain these corrupted indexes. This is a serious issue, and the only way to fix it is to recreate the indexes while ensuring that any duplicates are removed. I had to do this a couple of weeks ago, and it was a frustrating and time-consuming process. It is frustrating that those responsible did not verify the backups properly.

   • nixCraft 🐧 (nixcraft@mastodon.social)'s status on Saturday, 02-Mar-2024 01:34:53 JST nixCraft 🐧

     in reply to

     • Schenkl 🏳️‍🌈

     @schenklklopfer They attempted to restore the database from the web option that fetches the backup stored in S3, but it failed due to corrupt indexes. I was then called in to investigate the issue. They never in 7 years verified backups. So I guess issue got developed in few weeks and they wanted to restore in time or something like that. That is when they discovered the mess.

   • Schenkl 🏳️‍🌈 (schenklklopfer@chaos.social)'s status on Saturday, 02-Mar-2024 01:34:55 JST Schenkl 🏳️‍🌈

     in reply to

     @nixCraft how to check a backup properly?

   • nixCraft 🐧 (nixcraft@mastodon.social)'s status on Saturday, 02-Mar-2024 01:37:56 JST nixCraft 🐧

     in reply to

     • Schenkl 🏳️‍🌈

     @schenklklopfer To answer your question, you can restore the database manually on a fresh system periodically. Other times, you can automate the process using checksums and file sizes while restoring the database and send those logs to someone like sysadmin/DBA or devs. The manual process is great because you can see exact errors on the screen.

   • nixCraft 🐧 (nixcraft@mastodon.social)'s status on Saturday, 02-Mar-2024 01:42:50 JST nixCraft 🐧

     in reply to

     • Schenkl 🏳️‍🌈

     @schenklklopfer it is there but not in free version of mysql/mariadb. you can find MySQL Enterprise Backup solution that has verification features including hot backups. I know it sucks for home users.

   • Schenkl 🏳️‍🌈 (schenklklopfer@chaos.social)'s status on Saturday, 02-Mar-2024 01:42:51 JST Schenkl 🏳️‍🌈

     in reply to

     @nixCraft so there is no "mysql dump --verify" or something like that?