Image a Linux HD

[Glennbo]

Are there any good utilities to make a FULL image of a functioning Linux that could be restored to say, a brand new drive so that it boots and functions with minimal fuss or geeky commands in a terminal window?

I've used an old copy of Acronis to clone my MythTV server to a backup drive, and had to use it the other day after upgrading the Myth server from 16.04 Xenial to 18.04 Bionic Beaver. After upgrading I noticed some things with MythTV acting weird, and after screwing with it, started seeing it boot into "Emergency" mode.

I pulled out my Acronis clone of that machine, swapped HDs and was back on 16.04 with everything working. After a bit of research I found that the message I was getting might be related to fstab mounting (or failing to mount) a volume, so I used a bare drive USB interface and edited the fstab on the original drive, put it back in the MythTV server, and it booted up fine, but did some final tasks from the upgrade to 18.04 Bionic. I booted it a couple more times and it all seemed OK so I double checked the UUID of the second 1TB drive which is used for recording video and it was the same, so I un-commented the original line in fstab, rebooted and it all is working great and updated to 18.04.

The problem I have with using the old copy of Acronis is that while it can indeed clone a whole Linux drive, it does NOT get the boot loader, and it took me considerable time to correct that problem. I did try using DejaDup a little bit, but I don't know that I would trust it to pull my azz out of the fire like Acronis was able to, even with me having to hand install a boot loader.

So how are you guys backing your systems up where you could put in a virgin HD, restore your image, and then boot from the new drive like there was never a problem?

[MusicMan74]

I regularly backup/restore entire disks with Clonezilla:

https://clonezilla.org/

You can perform a disk to disk clone via a text wizard in about 6 steps or so...

I can supply more detailed info if you are interested.

[Glennbo]

Originally Posted by MusicMan74

I regularly backup/restore entire disks with Clonezilla:

https://clonezilla.org/

You can perform a disk to disk clone via a text wizard in about 6 steps or so...

I can supply more detailed info if you are interested.

Thanks! I'm downloading the .iso version of Clonzilla Live right now.

I have many older HDs laying around the house so I use them to clone entire boot drives where I can just swap them out in the case of a failure. My old version of Acronis True Image can do it, but it requires extra work to get a functioning boot loader after cloning the drive.

I did pickup both 64 and 32 bit versions of "Boot-Rescue", which will automatically install a boot loader for me, but I'd prefer to not need doing extra step like that in the event of a hard disk failure.

[Glennbo]

Originally Posted by MusicMan74

I regularly backup/restore entire disks with Clonezilla:

https://clonezilla.org/

You can perform a disk to disk clone via a text wizard in about 6 steps or so...

I can supply more detailed info if you are interested.

OK, I already have a question (Clonezilla 2.5.6-22 is burning right now).

I downloaded the 64 bit version of it, but I have a couple machines running 32 bit Xubuntu. Can the 64-bit version clone and restore a 32-bit version of Linux?

[MusicMan74]

Originally Posted by Glennbo

OK, I already have a question (Clonezilla 2.5.6-22 is burning right now).

I downloaded the 64 bit version of it, but I have a couple machines running 32 bit Xubuntu. Can the 64-bit version clone and restore a 32-bit version of Linux?

It is my understanding that the live ISO architecture has to do with the bare metal hardware running the boot environment and nothing to do with the OS bits of the source disk you are cloning...

An i686 iso exists to support those with older hardware...

https://clonezilla.org/downloads/dow...?branch=stable

[Glennbo]

Originally Posted by MusicMan74

It is my understanding that the live ISO architecture has to do with the bare metal hardware running the boot environment and nothing to do with the OS bits of the source disk you are cloning...

An i686 iso exists to support those with older hardware...

https://clonezilla.org/downloads/dow...?branch=stable

Cool! The machine I haven't backed up yet is my DAW, which is 64-bit, so I plan to try cloning it tomorrow. I did boot up Clonzilla and got all the way up to the selection menu for backing up, so it looks like it's ready to rock.

I have two other machines (a MythTV server and a laptop) that are running 32-bit versions of Linux though. The MythTV server I have made two complete clones and fixed the boot loader issue, so that one is covered for now. Myth is so freeking complex, I never want to have to set it all up from scratch again. Way too many little parts, like making it see and access the 8 network HDTV tuners, making the channel guide work with a scraper, getting channel icons where they need to live and with the correct names, station IDs, modifying privileges, adding users and groups, and on and on and on.

When I pulled out one of my backup drives and put it in that machine the other day, it booted and worked and all that stuff was setup, where I would have spent a week getting everything setup if I had to do it all from scratch.

I guess I should try booting Clonezilla on my MythTV server, as it is older hardware that I re-purposed when it became obsolete for other things. It's an older 3Ghz Core 2 Duo, and I'm pretty sure it's mobo is not 64-bit capable.

[Snap]

A great alternative to clonezilla is fsarchiver, present in most distro repos.

http://www.fsarchiver.org/

IMHO it's the greatest partition/system backup and deplyoment tool. It also has a Qt front-end for those wanting a GUI.

If you don't have dual boot machine the easiest way to use fsarchiver is with a live ISO. This one is just ready to go and great for that purpose plus system recovering and rescue. A must have.

http://www.system-rescue-cd.org/

[Glennbo]

Originally Posted by Snap

A great alternative to clonezilla is fsarchiver, present in most distro repos.

http://www.fsarchiver.org/

IMHO it's the greatest partition/system backup and deplyoment tool. It also has a Qt front-end for those wanting a GUI.

If you don't have dual boot machine the easiest way to use fsarchiver is with a live ISO. This one is just ready to go and great for that purpose plus system recovering and rescue. A must have.

http://www.system-rescue-cd.org/

Thanks! I'm grabbing that one too. Can it clone an entire drive? Reason I ask is that's what I really am wanting to do. Clone a perfectly functioning and fully setup system off to another physical drive, so I could simply swap out HDs in the event of a failure, or clone the clone drive back to the original, if the system simply became corrupt but the drive is still OK.

[Tobbe]

In Linux Lite there is Timeshift - System Restore Utility. And in AVLinux they use Systemback. Haven't digged into them yet.

//Tobbe

[Glennbo]

Originally Posted by Tobbe

In Linux Lite there is Timeshift - System Restore Utility. And in AVLinux they use Systemback. Haven't digged into them yet.

//Tobbe

That is one thing I sorely miss from Windows. I would never worry about trying things in Windows, because I knew I could simply restore back to ten minutes earlier. I had a habit of making a restore point before trying any any new software, so if I didn't like it or want to keep it for any reason, I could uninstall it and restore to the point I made just before installation.

I don't see either of those in Synaptic, but I'm going to look around and see what I can find on both of these. I'd love to have some tool that is like Windows System Restore for Linux.

[cyrano]

There's only one tool that works everywhere, every time: dd

Works on Windows, OSX, BSD and every Linux I've ever come across.

https://serverfault.com/questions/49...k-cloning#4912

There are even a couple of gui frontends (gdiskdump, DD gui) for it, if you fear the command line, but I've never used those and it looks like they are not seeing much development these days.

Make a script, taylored to YOUR needs, throw it in cron and forget about it.

[Tobbe]

Originally Posted by Glennbo

That is one thing I sorely miss from Windows. I would never worry about trying things in Windows, because I knew I could simply restore back to ten minutes earlier. I had a habit of making a restore point before trying any any new software, so if I didn't like it or want to keep it for any reason, I could uninstall it and restore to the point I made just before installation.

I don't see either of those in Synaptic, but I'm going to look around and see what I can find on both of these. I'd love to have some tool that is like Windows System Restore for Linux.

Found this for Timeshift:
https://github.com/teejee2008/timeshift

[Glennbo]

Originally Posted by Tobbe

Found this for Timeshift:
https://github.com/teejee2008/timeshift

Cool! Thanks Tobbe!

[Snap]

Originally Posted by Glennbo

Thanks! I'm grabbing that one too. Can it clone an entire drive? Reason I ask is that's what I really am wanting to do. Clone a perfectly functioning and fully setup system off to another physical drive, so I could simply swap out HDs in the event of a failure, or clone the clone drive back to the original, if the system simply became corrupt but the drive is still OK.

Yes. That's the purpose, It supports NTFS too.

Here's a nice howto:

http://vsido.org/index.php?topic=217.msg2771#msg2771

[Glennbo]

Originally Posted by Snap

Yes. That's the purpose, It supports NTFS too.

Here's a nice howto:

http://vsido.org/index.php?topic=217.msg2771#msg2771

OK, I just skimmed over that page a bit, but it looks like they are not cloning a drive so that you could just remove the failing one, and replace it with one that already is an identical copy ready to boot, with no kind of restore needed.

Right now, I have two complete clones of my MythTV server, where if the current drive were to fail, I can simply remove the current HD, pop in one of my two identical clones, and I'm back up and running, without needing to run any kind of restore, because the backup is a fully functional bit for bit, non compressed or changed in any way clone of of the original drive.

I'm going back now to read more. Also, there seems to be a lot of commands being issued in a terminal window, which I am totally not a fan of.

The old copy of Acronis True Image that I used to clone my MythTV drive boots up, I select (graphically) clone disk, it asks for the source and target drives, and then it makes a clone. The only problem I have with it is that it does not pickup the boot loader, so after cloning I have to screw around in a terminal window, which I don't really like having to do, issuing commands to try and get a boot loader on the new cloned drive.

[cyrano]

Timeshift doesn't include user data.That means you can restore a previous system while keeping your data. Very useful. For instance for experimenting with other distro's without changing harddisks.

NTFS support is included, but be prepared to delve into the Windows registry before relying on it. As usual, NTFS support is experimental. Especially with media servers, there could be minor problems.

The disadvantage of dd is that it has a learning curve. You need to invest some time. Once.

After that, it just works since it gets updated with your OS. Other utilities, that are not included with the distro, might break one day. And you might notice it when things went wrong and you badly need to get your system up and running again.

[babag]

i've used system rescue cd. this is a tutorial i found useful as it's the only one i found when i started using system rescue cd that included instructions on backing up the boot sector. prior to finding this i hit a wall where i could resstore but it wouldn't boot.

backup:
https://www.youtube.com/watch?v=gh6cnP4rP7Y

restore:
https://www.youtube.com/watch?v=-WN7GZzxrP4

thanks,
BabaG

[Glennbo]

Originally Posted by cyrano

Timeshift doesn't include user data.That means you can restore a previous system while keeping your data. Very useful. For instance for experimenting with other distro's without changing harddisks.

NTFS support is included, but be prepared to delve into the Windows registry before relying on it. As usual, NTFS support is experimental. Especially with media servers, there could be minor problems.

The disadvantage of dd is that it has a learning curve. You need to invest some time. Once.

After that, it just works since it gets updated with your OS. Other utilities, that are not included with the distro, might break one day. And you might notice it when things went wrong and you badly need to get your system up and running again.

So can you do things like tell Timeshift to make a restore point, right before trying something that might corrupt the system? That was how I always used System Restore in Windows.

If I were going to edit the registry, say for instance searching and deleting every occurrence of a named product that didn't clean up very good after uninstalling. I would always make a System Restore point, then go hack the registry, and if on the next boot any problem or error messages got thrown out, I could just restore the system to the way it was ten minutes earlier.

I'd do the same thing before installing any new software that I would try out. Just before installing a new software, I would always make a restore point, install the software, and if I didn't like or need it after trying it, I would uninstall it, followed by a system restore, specifically to keep stray junk out of my registry and off my HD.

[cyrano]

I can tell where you're coming from... Preventing Windows and/or user disaster...

dd is a dumb utility. It doesn't run at intervals, like weekly. That's what's cron is for. It doesn't do selective backup easily. But there's one thing it's meant for: cloning (dumping in Linux speak) entire disks or entire partitions. It can do a lot of other things too, like modifying disk headers. But you won't need that.

There's one BIG danger lurking behind the curtain. If you tell it to restore to the wrong disk, that disk is gone forever. That's why it is dangerous for beginners. You always need to make damned certain your /dev/sda1 is the right disk and it's not /dev/sda2, fi. And, yes, been there, done that too.

As to TimeShift, Ive tried it and used it. It works very well for what it is meant for: experimenting. And in that sense, it could be useful. But it doesn't clone an entire disk out of the box. I'm not even sure it can do that. There are some other options too, like including and excluding directories. But I've never tried those since TimeShift isn't included in distro's.

Personally, I always try to find a utility in the distro. That won't be abandoned by the developers and it will be updated. It might be replaced some day and force you to learn a new one, but that doesn't happen very often.

I used to be a fan of rsync. rsync can do what you want too. But it's also a command line tool and it's first use is to copy data to network disks. It can do a lot of other stuff too. It's very well supported, but not included in every distro.

I was reminded of my personal mantra about ten months ago, when one of my Debian servers was broken into. The cleanup wasn't too bad. What was worrying was that the people who did it, left almost no trace and only uploaded a well hidden RAT (remote access tool). It took me a week to find out that there was an obscure zero day exploit for rsync. I found it on a pastebin page that had only been accessed by 40 people when I read it. A week later, the rsync devs had an update. So there was just a small timeframe and I was just unlucky. But the point is, if I had used a tool that was included in the distro, it would have been upgraded automatically. I wouldn't have missed the update...

This isn't very important for a REAPER DAW machine, as it isn't attached to the net 24/7 and it is hidden behind your NAT router. It's just my personal mantra about tools in general.

With Linux, you don't need to worry about a software install ruining your system. It can happen, but it is very, very rare and usually points to problems with a wrong setup in BIOS or malfunctioning hardware.

And most Linux software is easy to uninstall. In Debian fi, an apt-get remove will clean up everything except maybe a manually made config txt file. And that won't harm the system. And that goes for Ubuntu, Xubuntu and all other Debian based distro's. In other distro's it's another tool, but based on the same principle. What you add with the tool, you remove with the tool.

The only thing interfering, are dependencies. You should check before installing how many dependencies software has. If it comes with numerous dependencies, be wary. Check the net. But most software has none, or just a few because most dependencies are already on your system. That's another learning curve. Knowing what you have already...

[Glennbo]

Originally Posted by cyrano

I can tell where you're coming from... Preventing Windows and/or user disaster...

dd is a dumb utility. It doesn't run at intervals, like weekly. That's what's cron is for. It doesn't do selective backup easily. But there's one thing it's meant for: cloning (dumping in Linux speak) entire disks or entire partitions. It can do a lot of other things too, like modifying disk headers. But you won't need that.

There's one BIG danger lurking behind the curtain. If you tell it to restore to the wrong disk, that disk is gone forever. That's why it is dangerous for beginners. You always need to make damned certain your /dev/sda1 is the right disk and it's not /dev/sda2, fi. And, yes, been there, done that too.

On both my desktop machines running Linux I went with a philosophy of using extremely small partitions of a bout 40GB to install the OS and apps. Then on both of those machines, I have 1TB second physical drives to hold any large data, like audio on my DAW, and HDTV video on my MythTV server. My thought was I have lots of old 40 and 80GB drives laying around that I can clone the whole primary boot/OS/programs off to, so in the event of a failure, I can just swap the whole drives out, replacing one old obsolete drive with another.

That said, I do also have a 1TB NAS that could be used to store automated backups of the little OS drives, but in that scenario I would probably want the backup to be easily accessible on a file by file, or folder by folder basis, so I could also just grab a folder or file and replace a corrupted one on the otherwise still functioning boot drive.

With Windows backup I have in the past bailed myself out of the fire by mounting the Windows backup as a VHD, and then just grab from it files or folders to replace ones that developed problems on the live OS partition.

As to TimeShift, Ive tried it and used it. It works very well for what it is meant for: experimenting. And in that sense, it could be useful. But it doesn't clone an entire disk out of the box. I'm not even sure it can do that. There are some other options too, like including and excluding directories. But I've never tried those since TimeShift isn't included in distro's.

Personally, I always try to find a utility in the distro. That won't be abandoned by the developers and it will be updated. It might be replaced some day and force you to learn a new one, but that doesn't happen very often.

I used to be a fan of rsync. rsync can do what you want too. But it's also a command line tool and it's first use is to copy data to network disks. It can do a lot of other stuff too. It's very well supported, but not included in every distro.

I guess I'm separating OS from everything else in this regard. When I would use Windows system restore, I knew it would not bring back things outside of the OS, but keeping the OS in tact without orphaned references all over the place was what I used it for. Even Linux leaves lots of crap behind after installing and uninstalling, followed by an apt-get autoclean or autoremove. Just yesterday I noticed I had folders and files in multiple places that were left behind after trying and then uninstalling Solaar.

That's where in Windows I would have created a restore point, right before I searched the whole drive and nuked ever file and folder referencing Solaar.

Fortunately, my system was totally happy and a few pounds lighter afterward.

I was reminded of my personal mantra about ten months ago, when one of my Debian servers was broken into. The cleanup wasn't too bad. What was worrying was that the people who did it, left almost no trace and only uploaded a well hidden RAT (remote access tool). It took me a week to find out that there was an obscure zero day exploit for rsync. I found it on a pastebin page that had only been accessed by 40 people when I read it. A week later, the rsync devs had an update. So there was just a small timeframe and I was just unlucky. But the point is, if I had used a tool that was included in the distro, it would have been upgraded automatically. I wouldn't have missed the update...

This isn't very important for a REAPER DAW machine, as it isn't attached to the net 24/7 and it is hidden behind your NAT router. It's just my personal mantra about tools in general.

Do you have any port forwarding being used in your router? What I mean is, can this kind of exploit happen if you have nothing being forwarded at your router? For a while I used to have a few things in my house setup to be accessible from the internet, but I have since then set the router to no longer forward requests coming from the outside world to any machines or NAS on the local network.

With Linux, you don't need to worry about a software install ruining your system. It can happen, but it is very, very rare and usually points to problems with a wrong setup in BIOS or malfunctioning hardware.

The kind of stuff I get concerned over is things like I go to install something like Carla, and get a prompt asking if I want to remove a bunch of things, that I don't know if it will replace, or will function as they did before, so I decide instead that I'm not willing to take that chance and blow off trying it out.

And most Linux software is easy to uninstall. In Debian fi, an apt-get remove will clean up everything except maybe a manually made config txt file. And that won't harm the system. And that goes for Ubuntu, Xubuntu and all other Debian based distro's. In other distro's it's another tool, but based on the same principle. What you add with the tool, you remove with the tool.

The only thing interfering, are dependencies. You should check before installing how many dependencies software has. If it comes with numerous dependencies, be wary. Check the net. But most software has none, or just a few because most dependencies are already on your system. That's another learning curve. Knowing what you have already...

That sounds like the experience I had when I thought about trying Carla. I don't know Linux well enough to know what's safe to allow when prompted about a bunch of dependencies will get smoked or altered to do the install.

Thanks for all the info BTW.

[clepsydrae]

Originally Posted by cyrano

There's only one tool that works everywhere, every time: dd

Works on Windows, OSX, BSD and every Linux I've ever come across.

Strangely, and despite countless tutorials online claiming otherwise, it never worked for me for raspbian linux on SD cards. When I googled around I found graybeards saying that it was not possible to do (working) partition-level backups of raspbian using linux, for some reason. Perhaps it has changed now... been a couple years since I tried.

Originally Posted by Snap

Yes. That's the purpose, It supports NTFS too.

I like and use fsarchiver, but as mentioned by cyrano, it doesn't actually support NTFS and is not a safe choice. (At least this was true some years ago -- not sure if anything has changed. The website info doesn't seem to have changed.) Gory details here:

http://forums.fsarchiver.org/viewtopic.php?p=4268#p4268
https://forum.tuxera.com//viewtopic....1cc17461953121
https://forum.tuxera.com//viewtopic....2f2920d#p36939

...TLDR: default windows installations have lots of files that fsarchiver will not back up; sounds like a fix is possible, but not implemented; unclear if those files would prevent a windows install from running.

I use partimage to back up my windows partition, but I'm not sure that's totally safe either: does anyone know if it really and truly supports NTFS, as it claims? Clonezilla also says it supports NTFS... but supporting an NTFS data drive is different from a working OS installation.

The issue with fsarchiver not being safe for ntfs has to do with interoperatinon ntfs-3g... wouldn't partimage and clonezilla use ntfs-3g as well? Maybe they have some workaround in place...

[cyrano]

Originally Posted by clepsydrae

Strangely, and despite countless tutorials online claiming otherwise, it never worked for me for raspbian linux on SD cards. When I googled around I found graybeards saying that it was not possible to do (working) partition-level backups of raspbian using linux, for some reason. Perhaps it has changed now... been a couple years since I tried.

I use dd on OSX. Just works. I've also used some Mac specific gui tool; but can't even remember the name.

I've never used dd on Raspbian running on the Pi, tho, if that's what you mean. Unless you'd have a pair of USB harddisks attached, there isn't much to clone there. And you should NEVER clone a running system disk. Always boot from something else.

It is theoretically possible to clone a running system disk from the disk dd is running on, but you will run into problems like USB2 being to slow and hence the network port also being to slow. Something will crap out if you block the system from running normally for hours on end.

[clepsydrae]

Yeah I'm talking about pulling the micro SD card out of a raspberry pi and making a backup of the unmounted OS raspbian partition using dd. dd works fine to write the ISO to the SD card when initially setting up the pi, but trying to do backups of the OS after configuration/installation/whatever on the pi is what never worked for me (and what folks online said isn't doable, although I can not now find those links.) Given the large number of "how to back up your raspberry pi" posts, it seems like I must be wrong, but it never actually worked for me, and more experienced folks than I said it wouldn't work, so I gave up...

I'm not knocking dd for other uses, though.

I will say that file systems are always a bit more complicated than I give them credit for, and very often things that are supposed to 'just work' don't, so I tread carefully when it comes to choosing a backup system that I need to rely on.

[cyrano]

Sorry, I've done just that numerous times. I really don't see the problem.

In the beginning of the Pi era, lots of folks had problems with SD cards. And some still do. That's what you get when Raspbian was new and with a gigantic community like it is today, I guess.

[babag]

isn't the issue with dd supposed to be that it will blindly copy everything, including bad sectors, thus potentially losing odd bits now and again? i've used it a number of times without issue but isn't the above supposed to be a problem, if only in theory?

thanks,
BabaG

[clepsydrae]

This post has an interesting warning about how dd handles read errors with the conv=noerror,sync options... I wonder if that's why I had issues: maybe I had a crappy SD card causing corruption when using dd to read/write the partition. Or some other related issue as babag implies.

Anyway, I'm probably just spreading FUD. Given the ubiquity of dd as a recommended block-level backup tool I must have just gotten unlucky in some way. I should give it another look.

When you backup a partition with dd, does the empty space that it captures compress to near-nothing? Or does it read non-zero values in any of the empty space that was once used? Just wondering how space efficient it is in that way.

[cyrano]

Originally Posted by Glennbo

On both my desktop machines running Linux I went with a philosophy of using extremely small partitions of a bout 40GB to install the OS and apps. Then on both of those machines, I have 1TB second physical drives to hold any large data, like audio on my DAW, and HDTV video on my MythTV server. My thought was I have lots of old 40 and 80GB drives laying around that I can clone the whole primary boot/OS/programs off to, so in the event of a failure, I can just swap the whole drives out, replacing one old obsolete drive with another.

I used to do that too. These days, storage is cheap. So I don't bother.

But... I use Linux only for servers. No gui and backup goes to a network disk on another machine.

That said, I do also have a 1TB NAS that could be used to store automated backups of the little OS drives, but in that scenario I would probably want the backup to be easily accessible on a file by file, or folder by folder basis, so I could also just grab a folder or file and replace a corrupted one on the otherwise still functioning boot drive.

With Windows backup I have in the past bailed myself out of the fire by mounting the Windows backup as a VHD, and then just grab from it files or folders to replace ones that developed problems on the live OS partition.

You can mount dd images just as easily. I usually mount these on OSX, though, so YMMV.

I guess I'm separating OS from everything else in this regard. When I would use Windows system restore, I knew it would not bring back things outside of the OS, but keeping the OS in tact without orphaned references all over the place was what I used it for. Even Linux leaves lots of crap behind after installing and uninstalling, followed by an apt-get autoclean or autoremove. Just yesterday I noticed I had folders and files in multiple places that were left behind after trying and then uninstalling Solaar.

I've never had a problem with files or directories left over after apt-clean. But I also don't worry about small files and directories. They don't affect performance. And there's no registry to clean.

That's where in Windows I would have created a restore point, right before I searched the whole drive and nuked ever file and folder referencing Solaar.

Fortunately, my system was totally happy and a few pounds lighter afterward.

Had to look up Solaar. Never had anything to do with Logitech software on Linux, but I've seen the problem on OSX.

Do you have any port forwarding being used in your router? What I mean is, can this kind of exploit happen if you have nothing being forwarded at your router? For a while I used to have a few things in my house setup to be accessible from the internet, but I have since then set the router to no longer forward requests coming from the outside world to any machines or NAS on the local network.

NAT traversing is possible. Skype and some other software uses it to connect. But an average home user will probably never be attacked that way. The risk that your router itself will get hijacked is far greater, imho.

The kind of stuff I get concerned over is things like I go to install something like Carla, and get a prompt asking if I want to remove a bunch of things, that I don't know if it will replace, or will function as they did before, so I decide instead that I'm not willing to take that chance and blow off trying it out.

I understand now. Dependencies hell.

That sounds like the experience I had when I thought about trying Carla. I don't know Linux well enough to know what's safe to allow when prompted about a bunch of dependencies will get smoked or altered to do the install.

There's no replacement for experience. And you get experience by experimenting, hence making mistakes.

But I still think dd will do the job for you.

Thanks for all the info BTW.

You're welcome.

[cyrano]

Originally Posted by clepsydrae

This post has an interesting warning about how dd handles read errors with the conv=noerror,sync options... I wonder if that's why I had issues: maybe I had a crappy SD card causing corruption when using dd to read/write the partition. Or some other related issue as babag implies.

Obviously, if the disk is defective, you'll have a problem...

And that's not only bad sectors, but also failing controllers, badly configured RAID setups, corrupt tables. dd doesn't check and it certainly doesn't repair.

Anyway, I'm probably just spreading FUD. Given the ubiquity of dd as a recommended block-level backup tool I must have just gotten unlucky in some way. I should give it another look.

It's not fud. It's too easy to forget about gotcha's when you're used to doing things one way. I know not to clone a running system disk. Despite that, I've done it with Carbon Copy Cloner on OSX. It should handle that, but it fails in a few very specific cases. Like cloning an active system from a hybrid harddisk (one with 16 GB SSD and 750 GB spinning rust). The copy is complete, but the system on the cloned disk doesn't start up. Even "blessing" it (a util to make the disk bootable that links the startup sector to the kernel) doesn't work. And I've seen that when cloning from one SSD to a very different SSD too. When I did it with the system offline and dd, in both cases it worked. It's the power of a simple tool.

When you backup a partition with dd, does the empty space that it captures compress to near-nothing? Or does it read non-zero values in any of the empty space that was once used? Just wondering how space efficient it is in that way.

It's a block by block copy. No compression. So it's slow, but you don't have to babysit it. And copy times are pretty predictable. CCC and Apple's Disk utility can do compression. And Disk utility does encryption too, if you need it.

If checking the disk before cloning reports any errors, I use a program that's called "Testdisk". Very hard to find via Google because of the generic name.

https://www.cgsecurity.org/wiki/TestDisk

I use it to check the disk before cloning, as there isn't much sense in cloning a defective disk, is there?

It not only can check the disk, but also repair it and it does data recovery too. It's free, but donations are welcome. It supports every OS under the sun, and most disk formats. You should download and learn it if you want to play around with disks, SD cards and almost any other form of storage. Perhaps make your own Live CD with it?

And you can thank Christophe Grenier as his software is second to none. I have at least a dozen other data recovery tools. Some, like Data Rescue are nicer (with a gui) and almost equally capable, but none is as reliable as this one. There's also a forum where Christophe is present to answer questions in English, French and German. You can be pretty sure that if he doesn't have an answer, it'll be a long and hard job to resurrect a disk.

And a typical problem that I think might have been caused by using dd on OSX:

https://forum.cgsecurity.org/phpBB3/...php?f=6&t=8070

[clepsydrae]

Originally Posted by cyrano

Obviously, if the disk is defective, you'll have a problem...

I didn't read it too closely, but I think the point of that blog was that if a bad spot was reached in the disk, it wasn't just the bad data that wouldn't be copied, but that the entire block would be corrupted. In other words, a small bad section might corrupt a file that overlapped the same block that would have otherwise been fine with a file-level clone (e.g. fsarchiver):

If you are positive that your disk does not contain any errors, you could proceed using a larger block size, which will increase the speed of your copying several fold. For example, changing bs from 512 to 64K changed copying speed from 35 MB/s to 120 MB/s on a simple Celeron 2.7 GHz system. But keep in mind that read errors on the source disk will end up as block errors on the destination disk, i.e. a single 512-byte read error will mess up the whole 64 KiB output block.

...maybe I'm misinterpreting. But if I'm understanding, that might be a small argument in favor of file-level archivers?

Don't some filesystems detect and work around bad blocks? (I know disks do this at a hardware level as well.) If that was the case, maybe dd would run the risk of ignoring such workarounds and corrupting whole bs=X blocks when copying?

[Glennbo]

Originally Posted by cyrano

I used to do that too. These days, storage is cheap. So I don't bother.

But... I use Linux only for servers. No gui and backup goes to a network disk on another machine.

I just have this habit of trying to turn old useless stuff into something useful again. My MythTV server is 100% old junk parts that I threw together, but it can record multiple streams of HDTV from a farm of ethernet HDHomerun tuners connected to a 90 mile HDTV antenna in my attic. I set it all up and then cancelled my cable TV saving about $140 a month, and now I have whole house DVR functionality for the low price of $0.00 a month.

You can mount dd images just as easily. I usually mount these on OSX, though, so YMMV.

Cool. I've never once needed to restore any full backup, but I've grabbed bits and pieces from them a few times.

I've never had a problem with files or directories left over after apt-clean. But I also don't worry about small files and directories. They don't affect performance. And there's no registry to clean.

I'm sure the amount of space occupied by leftover junk was minimal, I just hate finding remnants of an app still lying around when I've uninstalled it, then followed with an apt autoclean and apt autoremove. When I find a folder for something I've removed, it makes me wonder if it didn't get removed, or if I didn't uninstall it properly, or if pieces of it are still firing up and running all the time in the background.

Had to look up Solaar. Never had anything to do with Logitech software on Linux, but I've seen the problem on OSX.

Installing Solaar was a vain attempt to get all the special keys on my Logitech keyboard to work. Most of them do, but five or six of them have keycodes that are higher than what is seen by Linux, and Logitech makes no Linux driver.

NAT traversing is possible. Skype and some other software uses it to connect. But an average home user will probably never be attacked that way. The risk that your router itself will get hijacked is far greater, imho.

Any tips on preventing that from happening? I've got it locked down as well as I know how to, and occasionally pull it's power cable just to cycle it. I've also installed the last firmware, but it's about four years old now, and probably won't be updated again.

I understand now. Dependencies hell.

There's no replacement for experience. And you get experience by experimenting, hence making mistakes.

Hehe, I've already made a few big ones. I installed DejaDup and used it to make a backup of my home folder (I tried just copying it, but got too many access errors), and after making the backup decided to try restoring it to another location. I made a folder on my NTFS 1TB drive and instructed DejaDup to restore to that folder. It started doing it and all looked OK, but then it kept telling me that the 1TB was out of space, and I knew that was not true. Finally I cancelled the restore and didn't think about again, UNTIL . . .

The next time I booted Windows 7, I got the message from windows that the 1TB needed to be scanned, so I let it do it. When Windows finally go booted up, I opened the 1TB and found . . . ONE FILE with a .dat extension, and everything else on it was gone. Fortunately, my NAS had copies of it all, and I was able to rebuild it, but what an eye opener.

THAT was what prompted me to ask the original question for this thread. I only want to copy whole drive to whole drive now, using my throw away antique drives I have laying around. My NAS is using some form of Linux, so doing automated backups to it should be safe, but I'm never using any of my NTFS drives to store backups again after that catastrophe.

[cyrano]

@ clepsydrae

Disk corruption will produce failure, both with a block copy tool and a file copy tool. It just depends on how much and where the corruption occurs.

"Normal" bad blocks aren't a problem. They get noted by the disk's controller and don't get used again. Not even when dd is copying blocks. For dd, the OS and all other software, these bad blocks no longer exist. The controller replaces them with a block from the spare blocks.

The problem only surfaces when the disk controller's list is full. But then, the disk is end of life and no software tool can do anything about it.


@ Glennbo

Maybe you should test restoring once in a while? I've seen that NTFS problem before. Even with MS' own software. There's NO documentation for NTFS, so all implementations on other OSes are experimental. Sometimes, even MS' own implementation seems experimental. That's happening to Apple too, with APFS. But that's brand new, so I still have hope it will grow up one day.

I've seen too many backups where people felt good to have one, just to feel bad that they couldn't restore them. The most famous calamity was DOS3 to DOS4 when a lot of people couldn't restore their data backup because they made it with DOS3 and the backup utility from DOS4 couldn't use that backup.

Like you, I hate throwing away working stuff. I still have a few 20 MB disks laying around somewhere. I'd be surprised if these still work, tho. And like you, I hate littered disks. But I've learned not to worry and love the bomb

One piece of that old junk is a Logitech keyboard with USB and PS/2 connectors. It has media keys and those work with Windows, Linux, BSD and OSX, without any supplementary software. But that keyboard dates back to the time Logitech was still a Swiss company...

I don't watch TV, but I have an old Apple TV1 to serve movies. Still works, despite Apple's attempts to make it obsolete. I have a Linux distro for it, ready to be installed. But it's just one of the many projects and since it's already vintage, there's no hurry.

[Glennbo]

Originally Posted by cyrano

@ Glennbo

Maybe you should test restoring once in a while?

That was exactly what I was doing with DejaDup. After making a backup of my home folder using it, I decided to test restore it to a folder on my 1TB NTFS drive. DejaDup says you can restore to other locations, so I assumed that I could restore to a folder on an NTFS partition, and it would ALL live inside that folder, but instead it corrupted the entire drive.

I've seen that NTFS problem before. Even with MS' own software. There's NO documentation for NTFS, so all implementations on other OSes are experimental. Sometimes, even MS' own implementation seems experimental. That's happening to Apple too, with APFS. But that's brand new, so I still have hope it will grow up one day.

I'd seen that happen before with drives that had bad sectors or were failing in some way at the physical level, but never with a perfectly functioning drive. I get the impression that DejaDup messed with things outside the folder I instructed it to restore my test backup into.

I've seen too many backups where people felt good to have one, just to feel bad that they couldn't restore them. The most famous calamity was DOS3 to DOS4 when a lot of people couldn't restore their data backup because they made it with DOS3 and the backup utility from DOS4 couldn't use that backup.

I totally remember the backup issue with DOS because I was the programmer/tech support for a DOS program that had on it's menu backup and restore options that called the backup and restore from DOS. I had to help many customers roll back the previous version of the restore program, and then add a "version" override so DOS4 would allow it to run, and not kick out a version mismatch error.

Like you, I hate throwing away working stuff. I still have a few 20 MB disks laying around somewhere. I'd be surprised if these still work, tho. And like you, I hate littered disks. But I've learned not to worry and love the bomb

I finally did toss out all my old drives smaller than 40GB, after smashing them pretty good with a claw-hammer.

One piece of that old junk is a Logitech keyboard with USB and PS/2 connectors. It has media keys and those work with Windows, Linux, BSD and OSX, without any supplementary software. But that keyboard dates back to the time Logitech was still a Swiss company...

All the media keys work on my Logitech keyboard, but there are zoom in, zoom out keys, plus buttons for gadgets, photos, music, and Media Center that don't function properly. Interestingly enough, the music and Media Center keys appear to be the same keycode to Linux.

I don't watch TV, but I have an old Apple TV1 to serve movies. Still works, despite Apple's attempts to make it obsolete. I have a Linux distro for it, ready to be installed. But it's just one of the many projects and since it's already vintage, there's no hurry.

I retired early in life and have lots of free time, so I record a lot of stuff that I could really live without, just to throw and run on my second monitor while doing other stuff on the main monitor. Daytime teevee is much more lame.

So no thoughts on things that might improve router security? I checked to see if any newer firmware was available, and I am already running the last one produced. Remote administration has never been enabled, no port forwarding is enabled, ipv6 is only enabled locally, Wifi has a guest network that is for things like Rokus, BluRay DVD player, and anything I feel might be a security hole. Oh, and UPnP is disabled.

Anything else I should take a look at?

[clepsydrae]

Originally Posted by Glennbo

I finally did toss out all my old drives smaller than 40GB, after smashing them pretty good with a claw-hammer.

Don't forget to get the neodymium magnets out! :-)

[Glennbo]

Originally Posted by clepsydrae

Don't forget to get the neodymium magnets out! :-)

What do I need those for? Are they worth a lot of money, or can I make my own compass or something?

[clepsydrae]

Originally Posted by Glennbo

What do I need those for? Are they worth a lot of money, or can I make my own compass or something?

I just think the disposability of electronics is a shame, and powerful magnets are handy to have around. We use HD magnets on the fridge. Unlike most fridge magnets, they don't fall off when you actually use them to stick a piece of paper on there. :-) I also use one embedded in the wall as a way to keep a sliding door shut against the wall when it's closed.

The voice coils can sometimes be useful as little actuators in DIY projects, too.

And the platters are kind of neat. I'm currently considering using the platters and maybe some spacers as percussion toys.

[Glennbo]

Originally Posted by clepsydrae

I just think the disposability of electronics is a shame,

The 8 and 10GB drives I hammered and tossed were almost 25 years old, so they were eligible.

and powerful magnets are handy to have around. We use HD magnets on the fridge. Unlike most fridge magnets, they don't fall off when you actually use them to stick a piece of paper on there. :-) I also use one embedded in the wall as a way to keep a sliding door shut against the wall when it's closed.

The voice coils can sometimes be useful as little actuators in DIY projects, too.

Those are some useful ideas. Next drive I smash, I'll try pulling the magnets out.

And the platters are kind of neat. I'm currently considering using the platters and maybe some spacers as percussion toys.

Now THAT sounds like something I would be into! I have a passive heat sink from an old CPU that has almost 2" tall fins. I kept that thing, because it's like a musical instrument. You can strum it with a flat blade screwdriver, and it sounds like a cousin to a bell tree, or if you randomly hit individual fins, it sounds like some kind of Asian kalimba or something. I've not used it in a song, but it is on my to do list.

[clepsydrae]

Originally Posted by Glennbo

I have a passive heat sink from an old CPU that has almost 2" tall fins.

Yeah! Heat sinks can be really neat sounding!

[cyrano]

Originally Posted by clepsydrae

Don't forget to get the neodymium magnets out! :-)

It will surprise you, but magnets don't harm drives. You need to make the magnetic force oscillating before it will erase even a floppy. And most HD's are encased in metal, rendering even that useless.

A tape head demagnetiser operates at 50 or 60 Hz. An erase head at 100 kHz or more...

[cyrano]

Originally Posted by Glennbo

What do I need those for? Are they worth a lot of money, or can I make my own compass or something?

Magnet fishing

See Youtube.

[clepsydrae]

Originally Posted by cyrano

It will surprise you, but magnets don't harm drives. You need to make the magnetic force oscillating before it will erase even a floppy. And most HD's are encased in metal, rendering even that useless.

A tape head demagnetiser operates at 50 or 60 Hz. An erase head at 100 kHz or more...

(in case it wasn't clear, i was only advising retrieving the magnets for their utility)