-
Notifications
You must be signed in to change notification settings - Fork 0
/
Linux-Complete-Backup-and-Recovery-HOWTO.sgml
1238 lines (1119 loc) · 123 KB
/
Linux-Complete-Backup-and-Recovery-HOWTO.sgml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype article public "-//OASIS//DTD DocBook V4.1//EN"
[ <!entity header system "header.sgml">
<!-- These allow the equivalent of #includes below. C^2 -->
<!ENTITY make.fdisk SYSTEM "cooked/make.fdisk">
<!ENTITY install SYSTEM "cooked/install">
<!ENTITY make.dev.hda SYSTEM "cooked/make.dev.hda">
<!ENTITY make.lvs SYSTEM "cooked/make.lvs">
<!ENTITY mount.dev.hda SYSTEM "cooked/mount.dev.hda">
<!ENTITY mount.lvs SYSTEM "cooked/mount.lvs">
<!ENTITY dev.hda SYSTEM "cooked/dev.hda">
<!ENTITY dev.hda.sfd SYSTEM "cooked/dev.hda.sfd">
<!ENTITY save.metadata SYSTEM "cooked/save.metadata">
<!ENTITY restore.metadata SYSTEM "cooked/restore.metadata">
<!ENTITY back.up.all SYSTEM "cooked/back.up.all">
<!ENTITY back.up.all.ssh SYSTEM "cooked/back.up.all.ssh">
<!ENTITY restore.all SYSTEM "cooked/restore.all">
<!ENTITY restore.all.ssh SYSTEM "cooked/restore.all.ssh">
<!ENTITY post.sh SYSTEM "cooked/post.sh">
<!ENTITY get.tester SYSTEM "cooked/get.tester">
<!ENTITY get.target SYSTEM "cooked/get.target">
<!ENTITY restore.tester SYSTEM "cooked/restore.tester">
<!ENTITY first.stage SYSTEM "cooked/first.stage">
<!ENTITY myemail "charlescurley at charlescurley dot com">
<!ENTITY myurl "https://charlescurley.com">
<!ENTITY % review "IGNORE">
]>
<!--
Compile with:
make all
To do:
Change notes:
-->
<!-- Allow review comments. To make them go away for release, change
"INCLUDE" to "IGNORE" -->
<!-- <!entity % review "INCLUDE"> -->
<article lang="en" id="index">
<articleinfo>
<title>Linux Complete Backup and Recovery HOWTO</title>
<date>2002 January 20</date>
<author>
<firstname>Charles</firstname>
<surname>Curley</surname>
<affiliation>
<address>
<email><ulink url="&myemail;">&myemail;</ulink></email>
</address>
</affiliation>
</author>
<revhistory>
<!-- Additional revision history entries go here -->
<!-- Can we automate this? C^2 -->
<revision>
<revnumber>2.14</revnumber>
<date>2022-04-08</date>
<authorinitials>c^2</authorinitials>
<revremark>Fdisk now returns the string gpt in either upper or lower case. Other improvements.</revremark>
</revision>
<revision>
<revnumber>2.13</revnumber>
<date>2019-07-24</date>
<authorinitials>c^2</authorinitials>
<revremark>save.metadata no longer changes all /dev/sda device names to /dev/hda names. Major changes in the README.</revremark>
</revision>
<revision>
<revnumber>2.12</revnumber>
<date>2018-06-09</date>
<authorinitials>c^2</authorinitials>
<revremark>Added checksum generation and testing to improve reliability.</revremark>
</revision>
<revision>
<revnumber>2.11</revnumber>
<date>2017-07-29</date>
<authorinitials>c^2</authorinitials>
<revremark>Several improvements after working with Debian 9.</revremark>
</revision>
<revision>
<revnumber>2.10</revnumber>
<date>2017-07-12</date>
<authorinitials>c^2</authorinitials>
<revremark>We now check to see if the Perl UUID module is installed and in Perl's module path. Also, this is our first release under git.</revremark>
</revision>
<revision>
<revnumber>2.9</revnumber>
<date>2014-01-06</date>
<authorinitials>c^2</authorinitials>
<revremark>Added a new file, which is a list of excludes for use by get.target.</revremark>
</revision>
<revision>
<revnumber>2.8</revnumber>
<date>2011-06-13</date>
<authorinitials>c^2</authorinitials>
<revremark>Maintenance release. Many edits due to changes in technology. URL fixes.</revremark>
</revision>
<revision>
<revnumber>2.7</revnumber>
<date>2010-01-12</date>
<authorinitials>c^2</authorinitials>
<revremark>More improvements in make.fdisk: Removed a bug related to UUIDs and found a better way to get them.</revremark>
</revision>
<revision>
<revnumber>2.6</revnumber>
<date>2009-12-24</date>
<authorinitials>c^2</authorinitials>
<revremark>Improvements in make.fdisk: we now have a better way to get UUIDs, and we can preserve them for swap space.</revremark>
</revision>
<revision>
<revnumber>2.5</revnumber>
<date>2008-04-24</date>
<authorinitials>c^2</authorinitials>
<revremark>Changed shebangs in the backup scripts to point to bash instead of sh because some ubunutu installations don't make sh a symlink to bash.</revremark>
</revision>
<revision>
<revnumber>2.4</revnumber>
<date>2007-11-08</date>
<authorinitials>c^2</authorinitials>
<revremark>Some Debian/Ubuntu adaptations. see the notes below on them.</revremark>
</revision>
<revision>
<revnumber>2.3</revnumber>
<date>2007-05-26</date>
<authorinitials>c^2</authorinitials>
<revremark>Changes for FHS compliance. Changes in <link linkend="save.metadata"><filename>save.metadata</filename></link> to handle the libata problem.</revremark>
</revision>
<revision>
<revnumber>2.2</revnumber>
<date>2006-07-11</date>
<authorinitials>c^2</authorinitials>
<revremark>Clarified that the ZIP disk is not required, and there are alternatives.</revremark>
</revision>
<revision>
<revnumber>2.1</revnumber>
<date>2006-03-28</date>
<authorinitials>c^2</authorinitials>
<revremark>Added notes for NTFS. Edited the To Do list. Started work on LVM and using <ulink url="http://www.finnix.org/">Finnix</ulink>.</revremark>
</revision>
<revision>
<revnumber>2.0</revnumber>
<date>2005-10-12</date>
<authorinitials>c^2</authorinitials>
<revremark>Notes for Fedora Core 4. Removed notes for older versions of FC and Red Hat. Also, changes in the writeup and scripts to reflect using <ulink url="http://www.knoppix.org/">Knoppix</ulink> instead of <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>. See the scripts for change notes. Changed some scripts so that long lines don't fall off the right side of printed pages (oops).</revremark>
</revision>
<revision>
<revnumber>1.8</revnumber>
<date>2005-02-19</date>
<authorinitials>c^2</authorinitials>
<revremark>Added notes for Fedora Core 3</revremark>
</revision>
<revision>
<revnumber>1.7</revnumber>
<date>2004-05-11</date>
<authorinitials>c^2</authorinitials>
<revremark>Adjusted copyright language.</revremark>
</revision>
<revision>
<revnumber>1.6</revnumber>
<date>2004-04-29</date>
<authorinitials>c^2</authorinitials>
<revremark>Added <ulink url="http://www.knoppix.org/">Knoppix</ulink> notes, Syslinux, PPART, QtParted, some other rescue CDs, and made some fixes.</revremark>
</revision>
<revision>
<revnumber>1.5</revnumber>
<date>2003-12-19</date>
<authorinitials>c^2</authorinitials>
<revremark>Fedora 1 and GRUB notes.</revremark>
</revision>
<revision>
<revnumber>1.4</revnumber>
<date>2003-08-17</date>
<authorinitials>c^2</authorinitials>
<revremark>Some notes on burning CD-ROMs, and more on files to exclude.</revremark>
</revision>
<revision>
<revnumber>1.3</revnumber>
<date>2003-04-24</date>
<authorinitials>c^2</authorinitials>
<revremark>Substituted new email address and URL for old.</revremark>
</revision>
<revision>
<revnumber>1.2</revnumber>
<date>2003-02-12</date>
<authorinitials>c^2</authorinitials>
<revremark>Added Red Hat 8.0 notes, support for FAT32, split the first stage restore scripts, and other minor changes. Notes on <link linkend="amanda">Amanda</link>.</revremark>
</revision>
<revision>
<revnumber>1.1</revnumber>
<date>2002-09-10</date>
<authorinitials>c^2</authorinitials>
<revremark>New code to handle ext3 partitions in <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, and a note on <link linkend="initrd"><filename>initrd</filename></link>.</revremark>
</revision>
<revision>
<revnumber>1.0</revnumber>
<date>2002-07-24</date>
<authorinitials>c^2</authorinitials>
<revremark>We now use bz2 compression in the first stage, have the run time option to check for bad blocks, and have a script that runs the entire first stage.</revremark>
</revision>
</revhistory>
<abstract>
<title>Abstract</title>
<para>Imagine your disk drive has just become a very expensive hockey puck. Imagine you have had a fire, and your computer case now looks like something Salvador Dalĩ would like to paint. Now what?</para>
<para>Total restore, sometimes called bare metal recovery, is the process of rebuilding a computer after a catastrophic failure. In order to make a total restoration, you must have complete backups, not only of your file system, but of partition information and other data. This HOWTO is a step-by-step tutorial on how to back up a Linux computer so as to be able to make a bare metal recovery, and how to make that bare metal recovery. It includes some related scripts.</para>
</abstract>
</articleinfo>
<sect1 id="intro">
<title>Introduction</title>
<para>The normal bare metal restoration process is: install the operating system from the product disks. Install the backup software, so you can restore your data. Restore your data. Then you get to restore functionality by verifying your configuration files, permissions, etc.</para>
<para>The process and scripts explained in this HOWTO will save re-installing the operating system. The process explained here will restore only files that were backed up from the production computer. Your configuration will be intact when you restore the system, which should save you hours of verifying configurations and data.</para>
<sect2 id="copyright">
<title>Copyright Information</title>
<para>Copyright © 2001 through last date of modification Charles Curley and distributed under the terms of the GNU Free Documentation License (GFDL) license, stated below. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled <link linkend="appendix1gfdl"><quote>GNU Free Documentation License</quote></link>.</para>
<!-- <para> -->
<!-- Unless otherwise stated, Linux HOWTO documents are -->
<!-- copyrighted by their respective authors. Linux HOWTO documents may -->
<!-- be reproduced and distributed in whole or in part, in any medium -->
<!-- physical or electronic, as long as this copyright notice is -->
<!-- retained on all copies. Commercial redistribution is allowed and -->
<!-- encouraged; however, the author would like to be notified of any -->
<!-- such distributions. -->
<!-- </para> -->
<!-- <para> -->
<!-- All translations, derivative works, or aggregate works -->
<!-- incorporating any Linux HOWTO documents must be covered under this -->
<!-- copyright notice. That is, you may not produce a derivative work -->
<!-- from a HOWTO and impose additional restrictions on its -->
<!-- distribution. Exceptions to these rules may be granted under -->
<!-- certain conditions; please contact the Linux HOWTO coordinator at -->
<!-- the address given below. -->
<!-- </para> -->
<!-- <para> -->
<!-- In short, we wish to promote dissemination of this -->
<!-- information through as many channels as possible. However, we do -->
<!-- wish to retain copyright on the HOWTO documents, and would like to -->
<!-- be notified of any plans to redistribute the HOWTOs. -->
<!-- </para> -->
<para>If you have any questions, please contact <email>linux-howto at metalab.unc.edu</email>.</para>
</sect2>
<sect2 id="disclaimers">
<title>Disclaimers</title>
<para> No liability for the contents of this documents can be accepted by the author, the <ulink url="http://www.tldp.org/">Linux Documentation Project</ulink> (LDP) or anyone else. Use the concepts, examples and other content at your own risk. There may be errors and inaccuracies that may damage your system. Proceed with caution, and, although errors are unlikely, the author take no responsibility for them.</para>
<para> All copyrights are held by their by their respective owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.</para>
<para>Naming of particular products or brands should not be seen as endorsements.</para>
<para>You are strongly recommended to take a backup of your system before major installation and backups at regular intervals. In addition, you are strongly recommended to use a sacrificial experimental computer when mucking with the material, especially the scripts, in this HOWTO. When I first wrote this, virtual machines were a rarity. Now they are readily available.</para>
</sect2>
<sect2 id="newversions">
<title>New Versions</title>
<para>You can find this document at its <ulink url="&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO.php">home page</ulink> or at the <ulink url="http://www.tldp.org/">Linux Documentation Project</ulink> web site in many formats. Please comment to <email>&myemail;</email></para>
<para>Depending on your browser, you may have to hold down the shift button while you click on these in order to get them to download.</para>
</sect2>
<sect2 id="credits">
<title>Credits</title>
<para>This document is derived from two articles originally published in <ulink url="http://www.linuxjournal.com/"><citetitle pubwork="journal">Linux Journal</citetitle></ulink>. My thanks to <citetitle pubwork="journal">Linux Journal</citetitle> for reverting the rights to those articles, thereby helping make this HOWTO possible.</para>
<para>Thanks to Joy Y. Goodreau for excellent HOWTO editing, and to David Palomares for correcting the spelling of Salvador Dalĩ's name.</para>
<para>Also, thanks to <ulink url="mailto:pon at iki dot fi">Pasi Oja-Nisula</ulink> for a bug fix and information on <ulink url="http://www.knoppix.org/">Knoppix</ulink>.</para>
<!-- <para> -->
<!-- In this version I have the pleasure of acknowledging: -->
<!-- </para> -->
<!-- <para> -->
<!-- <email>name (at) site.org</email> -->
<!-- </para> -->
<!-- <para> -->
<!-- <emphasis>Please scramble the addresses so email harvesters -->
<!-- cannot get addresses from your HOWTO and then spam people. That -->
<!-- has happened in the past.</emphasis> -->
<!-- </para> -->
<!-- <para> -->
<!-- <emphasis>Somecompany</emphasis> is acknowledged for sending me -->
<!-- documentation on their gizmos as well as permission to quote from -->
<!-- the material. These quotes have been approved before appearing -->
<!-- here and will be clearly labeled. -->
<!-- </para> -->
</sect2>
<sect2 id="feedback">
<title>Feedback</title>
<para>
Feedback is most certainly welcome for this document. Without your corrections, suggestions and other input, this document wouldn't exist. Please send your additions, comments and criticisms to me at: <email>&myemail;</email>.</para>
</sect2>
<sect2 id="translations">
<title>Translations</title>
<para>Not everyone speaks English. Volunteers are welcome.</para>
<!-- <para> -->
<!-- <itemizedlist> -->
<!-- <listitem> -->
<!-- <para> -->
<!-- <ulink url="http://linuxdoc.org/">German Translation</ulink> -->
<!-- by <email>someone (at) somewhere.de</email> -->
<!-- </para> -->
<!-- </listitem> -->
<!-- <listitem> -->
<!-- <para> -->
<!-- <ulink url="http://linuxdoc.org/">French Translation</ulink> -->
<!-- by <email>someone (at) somewhere.fr</email> -->
<!-- </para> -->
<!-- </listitem> -->
<!-- <listitem> -->
<!-- <para> -->
<!-- <ulink url="http://linuxdoc.org/">Italian Translation</ulink> -->
<!-- by <email>someone (at) somewhere.it</email> -->
<!-- </para> -->
<!-- </listitem> -->
<!-- </itemizedlist> -->
<!-- </para> -->
</sect2>
</sect1>
<sect1 id="Overview">
<title>Overview</title>
<para>The process shown below is not easy, and can be hazardous to your data. Practice it before you need it! Do as I did, and <emphasis>practice on a sacrificial computer</emphasis>! Virtual machines are excellent for the purpose.</para>
<para>The original target computer for this HOWTO was a Pentium computer. Originally, it had a <ulink url="http://www.redhat.com">Red Hat</ulink> 7.1 Linux server or workstation installation on one IDE hard drive. Since then, I have used a number of computers, and they have been ugraded to Red Hat 8.0 and <ulink url="http://fedora.redhat.com/">Fedora Cores 1, 3 and 4.</ulink>. The target computer does not have vast amounts of data because the computer was set up as a <quote>sacrificial</quote> test bed. That is, I did not want to test this process with a production computer and production data. Also, I did a fresh installation before I started the testing so that I could always re-install if I needed to revert to a known configuration.</para>
<note>
<title>NOTE</title><para>The sample commands will show, in most cases, what I had to type to recover the target system. You may have to use similar commands, but with different parameters. It is up to you to be sure you duplicate your setup, and not the test computer's setup.</para>
</note>
<para>The basic procedure is set out in W. Curtis Preston, <ulink url="http://www.oreilly.com/catalog/unixbr/"><citetitle pubwork="book">Unix Backup & Recovery</citetitle></ulink>, O'Reilly & Associates, 1999, which I have favorably reviewed in <ulink url="http://m.linuxjournal.com/article/3839"><citetitle pubwork="journal">Linux Journal</citetitle></ulink>. However, the book is a bit thin on specific, real-time questions. For example, exactly which files do you back up? What metadata should you preserve, and how? This document explores those questions.</para>
<para>Before beginning the process set forth in this HOWTO you will need to back up your system with a typical backup tool such as Amanda, <trademark class="trade">BRU</trademark>, tar, <trademark class="registered">Arkeia</trademark> or cpio. The question, then, is how to get from toasted hardware to the point where you can run the restoration tool that will restore your data.</para>
<para>Users of RPM based Linux distributions (<ulink url="http://fedora.redhat.com/">Fedora</ulink>, e.g.) should also save RPM metadata as part of their normal backups. The following is in one of the scripts in this HOWTO:</para>
<programlisting>bash# <command>rpm -Va | sort +2 -t ' ' | uniq > /etc/rpmVa.txt</command></programlisting>
<para>It provides a basis for comparison after a bare metal restoration.</para>
<para>For Debian systems, the optional program <command>debsums</command> provides an analogous facility.</para>
<para>To get to this point, you must have:</para>
<itemizedlist>
<listitem>
<para>Your hardware up and running again, with replacement components as needed. The BIOS should be correctly configured, including time and date, and hard drive parameters. At the moment, there is no provision for using a different hard drive.</para>
</listitem>
<listitem>
<para>When I started this project, I used a <ulink url="http://www.iomega.com/zip/products/par100_250.html"><trademark class="registered">ZIP</trademark> drive</ulink>. Now, they are rather cramped for space and can be inconvenient. You can substitute a USB flash disk, NFS mount, CD-RW or other medium. Just be sure that the Linux distribution you use for first stage restore supports your medium. For historical reasons, this document will refer to the <ulink url="http://www.iomega.com/zip/products/par100_250.html">ZIP drive</ulink>; please substitute the medium of your choice. There is more discussion of alternatives below in the section on <link linkend="themeandvariations">Theme And Variations</link>.</para>
</listitem>
<listitem>
<para>Your normal backup media: tape hard drive, etc.</para>
</listitem>
<listitem>
<para>A minimal Linux system that will allow you to run the restoration software, which we will call the restoration Linux.</para>
</listitem>
</itemizedlist>
<para>To get there, you need at least two stages of backup, and possibly three. Exactly what you back up and in which stage you back it up is determined by your restoration process. For example, if you are restoring a tape server, you may not need networking during the restoration process. So only back up networking in your regular backups.</para>
<para>You will restore in stages as well. In stage one, we build partitions, file systems, etc. and restore a minimum of files from the ZIP disk. The goal of stage one is to be able to boot to a running computer with a network connection, tape drives, restoration software, or whatever we need for stage two.</para>
<para>The second stage, if it is necessary, consists of restoring backup software and any relevant databases. For example, suppose you use Arkeia and you are building a bare metal recovery ZIP disk for your backup server. Arkeia keeps a database on the server's hard drives. You can recover the database from the tapes, if you want. Instead, why not tar and gzip the whole arkeia directory (at /usr/knox), and save that to another computer over NFS or SSH? Stage one, as we have defined it below, does not include X, so you will have some experimenting to do if you wish to back up X as well as your backup program. Some restore programs require X.</para>
<para>Of course, if you are using some other backup program, you may have some detective work to do to. You will have to find out the directories and files it needs to run. If you use tar, gzip, cpio, mt or dd for your backup and recovery tools, they will be saved to and restored from our ZIP disk as part of the stage one process describe below.</para>
<para>The last stage is a total restoration from tape or other media. After you have done that last stage, you should be able to boot to a fully restored and operational system.</para>
<sect2 id="limitations">
<title>Limitations</title>
<para>This HOWTO is restricted to making a minimal backup such that, having then restored that backup to new hardware (<quote>bare metal</quote>), you can then use your regular backups to restore a completely working system. This HOWTO does not deal with your regular backups at all.</para>
<para>Even within that narrow brief, this HOWTO is not exhaustive. You still have some research, script editing, and testing to do.</para>
<para>The scripts here restore the partition data exactly as found on the source hard drive. This is nice if you are restoring on an identical computer or at least an identical hard drive, but that is often not the case. For now, there are two remedies (which will make more sense after you've read the rest of the HOWTO):</para>
<itemizedlist>
<listitem>
<para>Edit the partition table input file. I've done that a few times. You can also do this to add new partitions or delete existing ones (but edit the scripts that use the partition table input file as well).</para>
</listitem>
<listitem>
<para>Hand build a new partition table and go from there. That is one reason why <link linkend="restore.metadata"><filename>restore.metadata</filename></link> does not call the hard drive rebuilding script. Use the <link linkend="make.dev.hda">rebuilding script</link>.</para>
</listitem>
</itemizedlist>
<para>The scripts shown here only handle ext2/3/4, FAT12, FAT16 and FAT32. Until some eager volunteer supplies code for doing so in these scripts, you will need other tools for backing up and restoring file systems we haven't covered. However, see the note below on <link linkend="ntfs">NTFS</link>. <ulink url="http://www.partimage.org/">Partition Image</ulink> looks like a useful candidate here.</para>
</sect2>
</sect1>
<sect1 id="Preparation">
<title>Preparation</title>
<note>
<title>WARNING</title>
<para>Do your normal backups on their regular schedule. This HOWTO is useless if you don't do that.</para></note>
<para>Build yourself a restoration Linux disk. For this I now use <ulink url="http://www.finnix.org/">Finnix</ulink>. I have used <ulink url="http://www.knoppix.org/">Knoppix</ulink> in the past. See the notes on <link linkend="knoppix">Knoppix</link> below. However, everything here is command line. We don't need a GUI. A GUI-less distribution (such as Finnix) will boot faster and can load itself into memory (so you can use the CD drive) even on a minimal machine.</para>
<para>In the past, I have used <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>. It is well documented and packs a lot of useful tools onto one floppy diskette. Unfortunately, the changes I've had to make in the scripts to handle more recent Linux systems cause problems for <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>. The <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> 2.0.103 tar is based on <ulink url="http://www.busybox.net/">busybox</ulink>, so remarks about it may apply to other Linux disties which use busybox. And tomsrtbt has other problems.</para>
<note>
<title>WARNING</title>
<para>The version of <command>tar</command> included in <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> does not preserve ownership when it restores. This may cause problems for applications like <link linkend="amanda">Amanda</link>. A backup and restoration tool, Amanda has several directories owned by its own user. The solution is:</para>
<itemizedlist>
<listitem>
<para>Note which directories and files are not owned by root.</para>
</listitem>
<listitem>
<para>Note their owners.</para>
</listitem>
<listitem>
<para>Arrange to set the ownership correctly as part of the restoration process. E.g:</para>
<programlisting>bash# <command>chown -R amanda:disk /var/lib/amanda</command></programlisting>
<para>You can also add that line to your scripts for second state restoration, such as <link linkend="restore"><filename>restore</filename></link>.</para>
</listitem>
</itemizedlist>
</note>
<note>
<title>WARNING</title>
<para><ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> does not support restoring owners by UID/GID. To make backups suitable for restoring with <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>, remove the tar command line option <quote>--numeric-owner</quote> from the command line options for tar in the function crunch in the script <link linkend="save.metadata"><filename>save.metadata</filename></link>.</para>
</note>
<para>Next, figure out how to do the operating system backup you will need so that you can restore your normal backup. I used to follow Preston's advice and use an Iomega parallel port ZIP drive. The drives get approximately 90 MB of useful storage to a disk. I need about 85 MB to back up my desktop, so a 100MB ZIP drive may be pushing your luck. These days I use CD-RWs, NFS or USB sticks. For more on those, see the sections on using <link linkend="cd-rom">CD-ROM</link>s and <link linkend="nfs">NFS</link>.</para>
<sect2 id="installingzipdrive">
<title>Installing the ZIP Drive</title>
<para>Installing the ZIP drive is covered in the <ulink url="http://www.tldp.org/HOWTO/mini/ZIP-Drive.html">ZIP Drive HOWTO</ulink>, available at <ulink url="http://www.tldp.org/">the Linux Documentation Project</ulink> and at its home page, <ulink url="http://www.njtcom.com/dansie/zip-drive.html">http://www.njtcom.com/dansie/zip-drive.html</ulink>. <!-- njt.com: bad hostname 2003 07 25 --></para>
</sect2>
<sect2 id="backup.server">
<title>Backup Server</title>
<para>You can set up a backup server for this process. Scripts on the backup server interact with the target machines (including itself) via SSH. They assume that your backup server user (root here, for simplicity) can log in with no password to the targets. This is necessary for unattended backups.</para>
<para>First, create a suitable directory to keep all the backups in. We'll call it <filename>backs</filename>. In backs, create a directory for each target computer. The first field in the directory should be the host name. Subsequent fields can be other useful information. For example, to preserve the last backup of a target before an installation of a new version of the distribution, I use an abbreviation for the distribution, e.g. <quote>fc5</quote>. Fields are separated with periods (<quote>.</quote>). So, for example, <filename>tester.f7</filename>. The host name is required because the scripts use that to determine which host to back up.</para>
<para>Copy the scripts <filename>get</filename> and <filename>restore</filename> into each target's directory. Then customize them for each host as needed.</para>
<para>Also create in <filename>backs</filename> a directory called <filename>scripts</filename> and put in it the script <filename>get.target</filename>. This is a library for the backup and restore scripts. It performs actions common to all backups and restorations.</para>
</sect2>
</sect1>
<sect1 id="CreatingtheStage1BackUp">
<title>Creating the Stage 1 Back Up</title>
<para>Having made your production backups, you need to preserve your partition information so that you can rebuild your partitions.</para>
<para>The script <link linkend="make.fdisk"><filename>make.fdisk</filename></link> scans a hard drive for partition information, and saves it in several files. The first is an executable script, one per hard drive, called <link linkend="make.dev.hda"><filename>make.dev.x</filename></link> (where <quote>x</quote> is the name of the device file, e.g. hda or sda). Second is <link linkend="mount.dev.hda"><filename>mount.dev.x</filename></link>, which creates mount points and mounts the newly created partitions on them. The next, <link linkend="dev.hda"><filename>dev.x</filename></link>, is the commands necessary for <command>fdisk</command> to build the partitions. Last is an input file, e.g. <filename>dev.sda.sfd</filename>, for <command>sfdisk</command> to create partions. (<command>sfdisk</command> is preferable and the scripts will used it if they find it.) You specify which hard drive you want to build scripts for (and thus the file names) by naming the associated device file as the argument to <link linkend="make.fdisk"><filename>make.fdisk</filename></link>. For example, on a typical IDE system,</para>
<programlisting>bash# <command>make.fdisk /dev/hda</command></programlisting>
<para>spits out the scripts <link linkend="make.dev.hda"><filename>make.dev.hda</filename></link>, <link linkend="mount.dev.hda"><filename>mount.dev.hda</filename></link> and the input files for <command>fdisk</command> and <command>sfdisk</command>, <link linkend="dev.hda"><filename>dev.hda</filename></link> and <link linkend="dev.hda.sfd"><filename>dev.hda.sfd</filename></link>, respectively.</para>
<para>In addition, if <link linkend="make.fdisk"><filename>make.fdisk</filename></link> encounters a FAT partition, it preserves the partition's boot sector in a file named <filename>dev.xy</filename>, where x is the drive's device name (e.g. sdc, hda) and y is the partition number. The boot sector is the first sector, 512 bytes, of the partition. This sector is restored at the same time the partitions are rebuilt, in the script <link linkend="make.dev.hda"><filename>make.dev.hda</filename></link>.</para>
<para>Fortunately, the price of hard drives is plummeting almost as fast as the public's trust in politicians after an election. So it is good that the output files are text, and allow hand editing. That's the most difficult but most flexible way to rebuild on a larger replacement drive. (See the <link linkend="todo">To Do list</link>.)</para>
<para>The script <link linkend="save.metadata"><filename>save.metadata</filename></link> preserves other metadata as needed. The script saves the partition information in the file <filename>fdisk.hda</filename> in the root of the ZIP disk. It is a good idea to print this file and your <filename>/etc/fstab</filename> so that you have hard copy should you ever have to restore the partition data manually. You can save a tree by toggling between two virtual consoles, running <command>fdisk</command> in one and catting <filename>/etc/fstab</filename> or <filename>/fdisk.hda</filename> as needed. However, doing so is error prone.</para>
<para>You will also want to preserve files relevant to your restoration method. For example, if you use NFS to save your data, you will need to preserve <filename>hosts.allow</filename>, <filename>hosts.deny</filename>, <filename>exports</filename>, etc. Also, if you are using any network-backed restoration process, such as Amanda or Quick Restore, you will need to preserve networking files like HOSTNAME, hosts, etc. and the relevant software tree. The simplest way to handle these and similar questions is to preserve the entire <filename>/etc</filename> directory.</para>
<para>There is no way a 100 MB ZIP drive is going to hold a server installation of a modern distribution of Linux. We have to be much more selective than simply preserving the whole kazoo. What files do we need?</para>
<itemizedlist>
<listitem>
<para>The boot directory.</para>
</listitem>
<listitem>
<para>The <filename>/etc</filename> directory and subdirectories.</para>
</listitem>
<listitem>
<para>Directories needed at boot time.</para>
</listitem>
<listitem>
<para>Device files in <filename>/dev</filename> unless you are using udev.</para>
</listitem>
</itemizedlist>
<para>To determine the directories needed at boot, we look at the boot initialization file <filename>/etc/rc.sysinit</filename> (or equivalent). It sets its own path like so:</para>
<programlisting><![ CDATA [PATH=/bin:/sbin:/usr/bin:/usr/sbin
]]><![ CDATA [export PATH]]></programlisting>
<para>Trial and error indicated that we needed some other directories as well.</para>
<para>In reading the script <link linkend="save.metadata"><filename>save.metadata</filename></link>, note that we aren't necessarily saving files that are called with absolute paths.</para>
<para>We may require several iterations of back up, test the bare metal restore, re-install from CD and try again, before we have a working backup script. While I worked on this HOWTO, I made five such iterations before I had a successful restoration. That is one reason why it is essential to use scripts whenever possible. Test thoroughly!</para>
<para>One thing you can do on an RPM based system is use the <command>rpm</command> program to determine which files are where. For example, to get a complete list of the files used by the openssh package, run:</para>
<programlisting>bash# <command>rpm -ql openssh</command></programlisting>
<para>Similarly of debian boxes, using <command>apt-file</command>.</para>
<programlisting>bash# <command>apt-file search openssh-server</command></programlisting>
<para>There are some things you don't need, like the man pages. You can inspect each one and decide whether to back it up or not.</para>
<note>
<title>WARNING</title>
<para>The second stage of restoration is run without overwriting previously restored files. This means that the files restored in the first stage are the ones that will be used after full restoration. So update your bare metal backups whenever you update files in these directories!</para>
</note>
<note>
<title>WARNING</title>
<para>Recent kernels have incorporated a new ATA (IDE) hard drive driver, libata. Because of this, parallel ATA drives (PATA) now show up as SCSI drives, as serial ATA (SATA) have always done. However, not all rescue distributions use this new driver. There is a line toward the bottom of <link linkend="save.metadata"><filename>save.metadata</filename></link> which very carefully replaces "/dev/sda" with "/dev/hda". Use this as a template if you have multiple IDE hard drives. Comment it out or delete it if this is not an issue for you.</para>
<para>Note that there is no guaranteed mapping! Systems with multiple hard drives may have confusing mappings. Be sure to edit this line carefully. Check it if you add or remove a hard drive of any interface type to or from your system!</para>
<para>N.B: if you have libata IDE drive issues, the grub-install line at the end of <link linkend="restore.metadata"><filename>restore.metadata</filename></link> won't work. If it doesn't, use your rescue disk to do the same. Or burn and boot to the boot image that is made as part of the first stage backup. Boot to it and do the second stage restore as usual. The second stage restore should re-run <filename>grub-install</filename> or you can run it manually.</para>
</note>
<sect2>
<title id="thearchive">The Archive</title>
<para>All of this gets stored into an archive under <filename>/var/lib/bare.metal.recovery</filename>. Each day a first stage backup is made a new directory is prepared, with the date encoded as YYYYMMDD, and the day's archive deposited therein. It is up to you to prune obsolete archives. It is a good idea to keep at least one old archive around in case the computer crashes while you are making an archive. If a second archive is made in a day, the earlier one for that day is replaced.</para>
<para>The files in the archive directory include a <filename>README.txt</filename>, which has information about the backup and the computer the backup was made on. Other files are there in case hand intervention is required.</para>
<para>Below the daily archive directory are several text files and three directories. The scripts reside in <filename>bin</filename>, the tarballs in <filename>data</filename>, and information about the system such as partitions and LVM volume backups are in <filename>metadata</filename>.</para>
<para>To create a CD, simply use a script or graphical tool to create a CD starting at the daily archive directory. It is up to you to be sure your archive will fit onto your medium, or to make other arrangements.</para>
<para>Also in the archive are checksum files for each daily archive. Use those to check the integrity of the daily archives. This is useful for checking archives when you copy them to other computers (such as a backup server) or to the target prior to resoration. If you launch the bare metal backup from the backup server using <link linkend="get"><filename>get</filename></link>, that script will copy the checksum files to the server, and check the archive copy on the server.</para>
</sect2>
<sect2 id="ThemeAndVariations">
<title>Theme And Variations</title>
<sect3>
<title>No ZIP drive</title>
<para>This backup process used to require you to have the ZIP disk drive present at each backup. It now creates the archive in a directory, which you can back up over the net. Then you only need to build a ZIP disk (with <command>cp -rp</command>) on the backup server when you need to restore.</para>
<para>The backup process will be faster than directly writing to the ZIP drive, but you should check that the resulting directory will fit on your ZIP disk (with the output of <command>du -hs $target.zip</command> in the script <link linkend="save.metadata"><filename>save.metadata</filename></link>)! See the definition of the variable <varname>zip</varname> in that script.</para>
<para>One of my laptops has problems running both a network card and a ZIP drive, so this is the process I use to back it up. I keep a backup image as well as the current one, so that I have a fallback in case the computer crashes during a backup.</para>
</sect3>
<sect3>
<title id="cd-rom">CD-ROM</title>
<para>This is similar to the no ZIP drive option above. Save your backups to a directory on your hard drive, as noted. Then use <command>mkisofs</command> to create an ISO 9660 image from that directory, and burn it. This does not work with some CD-ROM based restoration Linuxes, like <ulink url="http://www.knoppix.org/">Knoppix</ulink>, because the Linux has to have the CD-ROM drive. Unless you have two CD-ROM drives, say one in a USB clamshell. I have a DVD burner set up this way with exactly this in mind. Better, have <ulink url="http://www.finnix.org/">Finnix</ulink> load itself into memory on boot and then use the CD-ROM drive from which you booted.</para>
<para>These remarks should also apply to DVDs.</para>
<para><emphasis>Test</emphasis> your CDs on the drive you will use at restoration time. If you find you need to hack the scripts, you can copy them to <filename>/tmp</filename>, usually a RAM drive, and edit them there. The scripts will run there. As a RAM disk is volatile, be sure to save your changes before you reboot!</para>
<para>Much of rest of this section is old information. I recommend you use <ulink url="http://www.finnix.org/">Finnix</ulink>.</para>
<para>You can <ulink url="http://www.knoppix.net/wiki/Knoppix_Remastering_Howto">remastering</ulink> Knoppix or <ulink url="http://www.finnix.org/">Finnix</ulink> with your first and second stage backups on the CD-ROM. You should also be able to <ulink url="http://www.finnix.org/Remastering_Finnix">remaster Finnix</ulink>.</para>
<para>These days many computers come with a CD-ROM drive but no floppy diskette. And floppy drives do fail. So it's a good idea to burn your CD-ROM with a bootable image on it. The bad news is that the <quote>El Torito</quote> format supports 1.2 MB, 1.44 MB and 2.88 MB floppy images, and <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> uses a 1.7 MB floppy. The good news is that you can get a 2.88 MB version, <filename>tomsrtbt-2.0.103.ElTorito.288.img</filename>, from the same mirrors where you get the floppy image. Place a <emphasis>copy</emphasis>
<footnote>
<para>I emphasize copy because <command>mkisofs</command> will mung the file in the directory from which it makes the ISO image.</para>
</footnote>
in the root directory of the backup files. Then use the <command>mkisofs</command> command line option -b to specify <filename>tomsrtbt-2.0.103.ElTorito.288.img</filename> as the boot image file.</para>
<para>The only down side of this process is that many older BIOSes do not support 2.88 MB floppy images on CD-ROMs. Most of those will boot to a <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> floppy.</para>
<para>An alternative is to use <ulink url="http://syslinux.zytor.com/">Syslinux</ulink>. It is not dependent on a floppy diskette image, and you can build your own CD with a number of tools, such as <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>, on it.</para>
<para>You may have to adjust the BIOS options to allow the computer to boot to CD-ROM drive. If you can't do that, either because the BIOS won't support booting to CD-ROM, or because you can't get into the BIOS, see <ulink url="http://sourceforge.net/projects/btmgr/">Smart Boot Manager (SBM)</ulink> as described in the <link linkend="resources">Resources</link>.</para>
<para>One variant is to dispense with the tarballs in the first stage, and create a tarball of the entire system. When you build your restoration CD, put the monster tarball in the data directory of the CD. The scripts will pick that up and restore for you, combining the first and second stages. This eliminates a lot of the cruft related to permissions and ownership in <link linkend="restore.metadata"><filename>restore.metadata</filename></link> and <link linkend="save.metadata"><filename>save.metadata</filename></link></para>
</sect3>
<sect3>
<title id="nfs">NFS</title>
<para>If you back up across your network to a backup server, you will have all the files on it you need. Set up the directory where you keep all your backups as an NFS export.</para>
<para>Then, on <ulink url="http://www.finnix.org/">Finnix</ulink>, do the following (tab completion is very nice here):</para>
<programlisting># <command>mkdir /mnt/nfs</command>
# <command>/etc/init.d/portmap start</command>
# <command>mount server:/path/of/exportedfs /mnt/nfs</command>
# <command>cd /mnt/nfs/.../bin</command></programlisting>
<para>Now restore as usual.</para>
<para>There are several advantages to NFS for this job: You don't have to worry about space on a CD-ROM or <ulink url="http://www.iomega.com/zip/products/par100_250.html">ZIP drive</ulink>. You can edit scripts on the server and they are preserved when you reboot the target.</para>
</sect3>
<sect3>
<title>Multiple ZIP disks</title>
<para>By splitting up the two first stage scripts, <link linkend="restore.metadata"><filename>restore.metadata</filename></link> and <link linkend="save.metadata"><filename>save.metadata</filename></link>, you could spread the first stage metadata across multiple ZIP disks.</para> <!-- &ZIP; -->
</sect3>
<sect3>
<title>Excluding From First Stage Saving</title>
<para>There are time when you need to squeeze a few megabytes from the first stage data, especially when you are pushing the limit of your ZIP disk. The function <command>crunch</command> in the script <link linkend="save.metadata"><filename>save.metadata</filename></link> takes multiple parameters to feed to <command>tar</command>. It can also take the <command>--exclude</command> parameter. So, for example, you can exclude the <filename>samba</filename> and <filename>X11</filename> directories under <filename>/etc</filename> like so:</para>
<programlisting><![ CDATA [crunch etc --exclude etc/samba --exclude etc/X11 etc]]></programlisting>
<para>Why those two? Because they're hard drive space hogs and we don't need them when booting after the first stage.</para>
<para>If you keep multiple kernels around, you can eliminate the modules for all of the kernels you won't boot to. Check your <filename>lilo.conf</filename> or <filename>/boot/grub/menu.lst</filename> to see which kernel you will use, and then check <filename>/lib/modules</filename> for module directories you can exclude.</para>
<para>How to find more good candidates for exclusion? List the target directories with <command>ls -alSr</command> for individual files, and <command>du | sort -n</command> for directories.</para>
<para>Another, neater, way to exclude directories is to put a complete list of directories into a file, then refer to it via the tar option <filename>--exclude-from=FILENAME</filename>.</para>
</sect3>
<sect3 id="initrd">
<title>Initrd</title>
<para>If your system uses an initial RAM disk, or initrd, to boot, make sure that <link linkend="restore.metadata"><filename>restore.metadata</filename></link> creates the directory <filename>/initrd</filename>. The easiest way to do this is to ensure that it is included in the list of directories used in the directory creating loop toward the end.</para>
<para>Your system will probably use an initrd if it boots from a SCSI drive or has root on an ext3fs partition. Check <filename>/etc/lilo.conf</filename> or <filename>/boot/grub/menu.lst</filename> to see if it calls for one.</para>
<para>When in doubt, check the root directory to see if <filename>/initrd</filename> is there.</para>
</sect3>
</sect2>
</sect1>
<sect1 id="firststagerestore">
<title>First Stage Restore</title>
<sect2 id="Booting">
<title>Booting</title>
<para>The first thing to do is to verify that the hardware time is set correctly. Use the BIOS setup for this. How close to exact you have to set the time depends on your applications. For restoration, within a few minutes of exact time should be accurate enough. This will allow time-critical events to pick up where they left off when you finally launch the restored system.</para>
<sect3 id="bootingfinnix">
<title>Finnix</title>
<para>One option for booting <ulink url="http://www.finnix.org/">Finnix</ulink> is the "toram" option, which lets you move the whole kazoo into RAM. That in turn should let you load another CD, with your first stage data, into the CD drive.</para>
<para>Should you run into problems with <command>grub-install</command>, see <ulink url="https://www.finnix.org/Restoring_bootloaders">Restoring bootloaders</ulink> in the Finnix wiki.</para>
<para>If it is possible that there is a LUKS or LVM partition on the hard drive, but you don't want to access them on boot (because you are about to write over them anyway, say), add the <command>nocrypt</command> or <command>nolvm</command> options to the boot command line. There are more <ulink url="https://www.finnix.org/Boot_parameters">boot parameters</ulink> available.</para>
</sect3>
<sect3 id="Bootingknoppix">
<title>Knoppix</title>
<para>These instructions will probably work with other CD-ROM or USB pen Linuxes, but you may have to vary them to suit.</para>
<para>Before booting <ulink url="http://www.knoppix.org/">Knoppix</ulink>, make sure your ZIP drive (or substitute) is installed on a parallel port, either <filename>/dev/lp0</filename> or <filename>/dev/lp1</filename>. Knoppix does not load the parallel port ZIP drive driver for you. Instead, use the command <command>modprobe ppa</command> (as root) to install it.</para>
<para>Boot <ulink url="http://www.knoppix.org/">Knoppix</ulink> as usual. I find it faster and more useful to boot to a console. At the boot menu, use the command <quote>knoppix 2</quote>. Then become the root user, with <command>su -</command>. For the password, just hit return.</para>
</sect3>
<sect3 id="Bootingtomsrtbt">
<title>tomsrtbt</title>
<para>Before booting <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>, make sure your ZIP drive is installed on a parallel port, either <filename>/dev/lp0</filename> or <filename>/dev/lp1</filename>. The start-up software will load the parallel port ZIP drive driver for you.</para>
<!-- <para> -->
<!-- I have one of those ne2000 clone Ethernet cards in my test system. This, it turns out, gives the 3c59x driver in the <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> kernel fits. The workaround is to tell the kernel to ignore its address range. At the lilo prompt, I would type: -->
<!-- </para> -->
<!-- <programlisting><![ CDATA [lilo: zImage reserve=0x300,32]]></programlisting> -->
<para>The next step is to set the video mode. I usually like to see as much on the screen as I can. When the option to select a video mode comes, I use mode 6, 80 columns by 60 lines. Your hardware may or may not be able to handle high resolutions like that, so experiment with it.</para>
</sect3>
</sect2>
<sect2 id="restoration">
<title>Restoration</title>
<para>These instructions assume you are running <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>. If you are using a different Linux for your restore system, you may have to adjust these instructions a bit. For example, you should always run these scripts as root even if some other user gives you the requisite privileges.</para>
<para>Once the restoration Linux has booted and you have a console, mount the ZIP drive. It is probably a good idea to mount it read only. On <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>:</para>
<programlisting># <command>mount /dev/sda1 /mnt -o ro</command></programlisting>
<para>Check to be sure it is there:</para>
<programlisting># <command>ls -l /mnt</command></programlisting>
<para>On <ulink url="http://www.knoppix.org/">Knoppix</ulink> or <ulink url="http://www.finnix.org/">Finnix</ulink>, you may want to make a directory under <filename>/mnt</filename> and mount it there, like so:</para>
<programlisting># <command>mkdir /mnt/zip</command>
# <command>mount /dev/sda1 /mnt/zip -o ro</command></programlisting>
<para>Now cd into the mounted device, and into the <filename>bin</filename> directory below it. On <ulink url="http://www.finnix.org/">Finnix</ulink>, for example:</para>
<programlisting># <command>cd /mnt/zip/bin</command></programlisting>
<para>The scripts assume you are in this directory, and call data files relative to it. At this point, you can run the restoration automatically or manually. Use the automated restore if you don't need to make any changes as you go along.</para>
<para>One consideration here is whether you have multiple hard drives. If your Linux installation mounts partitions on multiple hard drives, you must mount the root partition first. This is to ensure that mount point directories are created on the partition where they belong. The script <link linkend="first.stage">first.stage</link> will run the scripts to mount the drives in the order in which they are created. If you have created them (in the script <filename>save.metadata</filename>) in the order in which they cascade from root, the mounting process should work just fine.</para>
<para>If you have multiple hard drives, and they cross-mount, you are on your own. Either combine and edit the scripts to mount them in the correct order, or do it manually.</para>
<sect3>
<title>Automated</title>
<para>The automatic process calls each of the manual scripts in proper order. It does not allow for manual intervention, say for creating file systems that this HOWTO does not support. To run the first stage restore automatically, enter the command:</para>
<programlisting># <command>first.stage</command></programlisting>
<para>If you want to check for back blocks, add the <command>-c</command> option.</para>
</sect3>
<sect3>
<title>Manually</title>
<para>Run the script(s) that will restore the partition information and create file systems. You may run them in any order, so long as they build dependencies in the correct order. You can read the script <link linkend="first.stage">first.stage</link> to get an idea of the order. e.g.:</para>
<programlisting># <command>./make.dev.hda</command></programlisting>
<para>If you want to check for back blocks, add the <command>-c</command> option.</para>
<para>This script will:</para>
<itemizedlist>
<listitem>
<para>Clean out the first 1024 bytes of the hard drive, killing off any existing partition table and master boot record (MBR).</para>
</listitem>
<listitem>
<para>Recreate the non-LVM partitions from the information gathered when you ran <link linkend="make.fdisk"><filename>make.fdisk</filename></link>.</para>
</listitem>
<listitem>
<para>Make ext3+ file systems on non-LVM partitions and Linux swap partitions as appropriate. If you provide the <command>-c</command> option to the script, it will also check for bad blocks.</para>
</listitem>
<listitem>
<para>Make some types of FAT partitions.</para>
</listitem>
</itemizedlist>
<para>Now is a good time to check the geometry of the drive. Sometimes different versions of Linux pick up different geometries, so the geometry implicit in the file <filename>dev.hdX</filename> is incorrect. To force it to be correct on <ulink url="http://www.knoppix.org/">Knoppix</ulink>, edit <link linkend="make.dev.hda"><filename>make.dev.x</filename></link>. Use the -C, -H and -S options to fdisk to specify the cylnders, heads and sectors, respectively. Those you can get from the file <filename>fdisk.hdX</filename>. Then re-run it.</para>
<note>
<title>NOTE</title><para>If you have other operating systems or file systems to restore, now is a good time to do so. When you've done that, reboot to your restoration Linux and continue your first stage restoration.</para>
</note>
<para>If you have LVM volumes to restore, now is the time to run <filename>make.lvs</filename> and <filename>mount.lvs</filename>.</para>
<para>Now run the script(s) that create mount points and mount the partitions to them.</para>
<programlisting># <command>./mount.dev.hda</command></programlisting>
<para>Once you have created all your directories and mounted partitions to them, you can run the script <link linkend="restore.metadata"><filename>restore.metadata</filename></link>.</para>
<programlisting># <command>./restore.metadata</command></programlisting>
<para>This will restore the contents of the ZIP drive to the hard drive to give you a minimal bootable system.</para>
<para>You should see a directory of the ZIP disk's root directory, then a list of the archive files as they are restored. Tar on <ulink url="http://www.toms.net/rb/">tomsrtbt</ulink> will tell you that tar's block size is 20, and that's fine. You can ignore it. Be sure that lilo prints out its results:</para>
<screen><![ CDATA [Added linux *]]></screen>
<para>That will be followed by the output from a <quote><command>df -m</command></quote> command.</para>
</sect3>
<sect3 id="finishing">
<title>Finishing Touches</title>
<para>If you normally boot directly to X, you could have some problems.</para>
<para>On System V installations, to be safe, the first stage script changes the run level in <filename>/target/etc/inittab</filename> to 3. Note: different distributions use different run level schemes. 3 works on Red Hat derived distributions; it may not on others.</para>
<para>On systemd installations (and probably others), you can avoid problems by booting into recovery mode. Or edit the <ulink url="https://fedoraproject.org/wiki/Systemd#Boot_Kernel_Command_Line">command line at boot time</ulink>. If there is a better way to do this, I'd like to hear about it.</para>
<para>You can now gracefully reboot. Remove the medium from your boot drive if you haven't already done so, and give the computer the three fingered salute, or its equivalent:</para>
<programlisting># <command>shutdown -r now</command></programlisting>
<para>or</para>
<programlisting># <command>reboot</command></programlisting>
<para>or</para>
<programlisting># <command>systemctl reboot</command></programlisting>
<para>The computer will shut down and reboot.</para>
</sect3>
</sect2>
</sect1>
<sect1 id="SecondStageRestoration">
<title>Second Stage Restoration</title>
<para>As the computer reboots, go back to the BIOS and verify that the clock is more or less correct.</para>
<para>Once you have verified the clock is correct, exit the BIOS and reboot to the hard drive. You can simply let the computer boot in its normal sequence. You will see a lot of error messages, mostly along the lines of <quote>I can't find blah! Waahhh!</quote> If you have done your homework correctly up until now, those error messages won't matter. You don't need linuxconf or apache to do what you need to do.</para>
<note><title>NOTE</title><para>As an alternative, you can boot to single user mode (at the lilo prompt, enter <command>linux single</command>), but you will have to configure your network manually and fire up sshd or whatever daemons you need to restore your system. How you do those things is very system specific.</para>
</note>
<para>You should be able to log into a root console (no X, no users, sorry). You should now be able to use the network, for example to NFS mount the backup of your system.</para>
<para>If you did the two stage backup I suggested for Arkeia, you can now restore Arkeia's database and executables. You should be able to run</para>
<programlisting>/etc/rc.d/init.d/arkeia start</programlisting>
<para>and start the server. If you have the GUI installed on another computer with X installed, you should now be able to log in to Arkeia on your tape server, and prepare your restoration.</para>
<note>
<title>NOTE</title>
<para>When you restore, read the documentation for your restoration programs carefully. For example, tar does not normally restore certain characteristics of files, like suid bits. File permissions are set by the user's umask. To restore your files exactly as you saved them, use tar's p option. Similarly, make sure your restoration software will restore everything exactly as you saved it.</para>
</note>
<para>To restore the test computer:</para>
<programlisting>bash# <command>restore.all</command></programlisting>
<para>If you used tar for your backup and restoration, and used the -k (keep old files, don't overwrite) option, you will see a lot of this:</para>
<screen><![ CDATA [tar: usr/sbin/rpcinfo: Could not create file: File exists
]]><![ CDATA [tar: usr/sbin/zdump: Could not create file: File exists
]]><![ CDATA [tar: usr/sbin/zic: Could not create file: File exists
]]><![ CDATA [tar: usr/sbin/ab: Could not create file: File exists]]></screen>
<para>This is normal, as tar is refusing to overwrite files you restored during the first stage of restoration.</para>
<para>Then reboot. On the way down, you will see a lot of error messages, such as <quote>no such pid.</quote> This is a normal part of the process. The shutdown code is using the pid files from daemons that were running when the backup was made to shut down daemons that were not started on the last boot. Of course there's no such pid.</para>
<para>Your system should come up normally, with a lot fewer errors than it had before; ideally no errors. The acid test of how well your restore works is to verify all packages. During the first stage backup, a verification was performed on the system, producing the file <filename>rpmVa.txt</filename>. Verify your system again, and compare the results to the one made earlier. E.g.:</para>
<programlisting>bash# <command>rpm -Va | sort +2 -t ' ' | uniq > ~/foo.txt
diff /mnt/zip/metadata/rpmVa.txt ~/foo.txt</command></programlisting>
<para>Prelinking error messages are normal and you can ignore them. Do not first run the command <command>/etc/cron.daily/prelink</command> to remove them. Doing so may introduce new errors in the verification results that will skew your results.</para>
<para>Some files, such as configuration and log files, will have changed in the normal course of things, and you should be able to mentally filter those out of the report. Emacs users should check out its ediff facilities.</para>
<para>Now you should be up and running. It is time to test your applications, especially those that run as daemons. The more sophisticated the application, the more testing you may need to do. If you have remote users, disable them from using the system, or make it <quote>read only</quote> while you test it. This is especially important for databases, to prevent making any corruption or data loss worse than it already might be.</para>
<para>If you normally boot to X, it was disabled as part of the first stage restoration. Test X before you re-enable it. Re-enable it by changing that one line in <filename>/etc/inittab</filename>. Find the line that looks like this:</para>
<programlisting><![ CDATA [id:3:initdefault:]]></programlisting>
<para>and change it to this:</para>
<programlisting><![ CDATA [id:5:initdefault:]]></programlisting>
<para>Or just run this on the target to change it back. Note: different distributions use different run level schemes. These values work on Red Hat derived distributions; they may not on others.</para>
<programlisting>sed -i s/id:.:initdefault:/id:5:initdefault:/g /etc/inittab</programlisting>
<para>You should now be ready for rock and roll -- and some aspirin and a couch.</para>
</sect1>
<sect1 id="DistributionSpecificNotes">
<title>Distribution Specific Notes</title>
<para>Below are distribution notes from past experiences. If you have additional notes that you would like to add for other distributions, please forward them to me.</para>
<sect2 id="ubuntu">
<title>Debian/Ubuntu</title>
<para>The largest change for Debian based systems is the lack of anything analogous to RPM's integrity checking capability, <command>rpm -Va</command>, which we use to check for correct restoration. <command>debsums</command> is close but has other problems. You must install <command>debsums</command>, as soon after you have installed the system as possible. You must then run <command>apt-get clean && debsums_init</command>. This command will force the reinstallation of packages for which checksums have not already been generated. Package reinstallation can affect configuration files. Also see the caveats section of the <command>debsums</command> man page.</para>
<para>The most practical problem is that Debian and Ubuntu use UUIDs to identify partitions, rather than device files or labels (like Red Hat systems). We can preserve the UUIDs, and assign them to ext2/3/4 and swap partitions. Actually, we can get UUIDs for any partition <command>blkid</command> supports.</para>
<para>When using a Debian live CD, create a root password with <command>sudo passwd root</command>. Then <command>su -</command> as usual. Packages you may want to install: <command>apt install cryptsetup lvm2 grub2-common</command>. If the screen saver locks you out, log in again as user/live.</para>
</sect2>
<sect2 id="fedora3">
<title>Fedora</title>
<para>The scripts now reflect Fedora 7, so you should not have to make any changes to these <link linkend="thescripts">scripts</link>.</para>
<note>
<para>I tested the above on a fresh installation of FC3. I had problems with devices after booting when I worked with a system that had been upgraded from FC2 to FC3.</para>
</note>
</sect2>
<sect2 id="knoppix">
<title>Knoppix</title>
<para>I used to use <ulink url="http://www.knoppix.org/">Knoppix</ulink>. <ulink url="mailto:pon at iki dot fi">Pasi Oja-Nisula</ulink> reports:</para>
<blockquote>
<para>For me the best thing about using Knoppix is that I don't need a specific boot medium for each machine, but I can use the same tools all the time. And hardware support in Knoppix is really great. I don't have that much experience with different platforms, but all the machines I've tried have worked fine, scsi drivers are found and so on.</para>
<para>I'm doing this recovery thing by copying the backups over the network to other machine. The restore involves booting the Knoppix cd, fetching the metadata.tar.gz from the network machine. Then make.dev, mount.dev, fetching the other tar.gz files, grub and reboot. Some typing involved but thanks to your scripts it's quite straighforward. Unless changing from ide to scsi or something, but even then it's not that difficult, since Linux is easy to restore to different hardware.</para>
</blockquote>
<para><ulink url="http://www.knoppix.org/">Knoppix</ulink> detects USB devices for you, which is really nice. They make excellent (and roomier) substitutes for the ZIP drive.</para>
<para>Also see <ulink url="http://www.ibm.com/developerworks/linux/library/l-knopx/index.html"><quote>System recovery with Knoppix</quote></ulink>.</para>
<para>Do your restore as user <quote>root</quote> rather than as user <quote>knoppix</quote>. Otherwise you may get some directories and files owned by an oddball user or group. Also, for <ulink url="http://www.knoppix.org/">Knoppix</ulink>, we tar the first stage stuff saving numeric user & group values instead of by name. The names may point to different numbers on Knoppix, so we would be restoring the files with incorrect user and group IDs.</para>
</sect2>
<sect2>
<title>Finnix</title>
<para><ulink url="http://www.finnix.org/">Finnix</ulink> has some of the same advantages of Knoppix. In addition, it runs in command line mode with mouse support, which is great for the task at hand. It's small, under 100 MB as of this writing, so you can remaster it with your first stage data on it. It boots quickly. And it has LVM support. And Zile, a subset of Emacs. It is small enough that you can load it into RAM (<command>finnix toram</command> at boot) and use the CD-ROM drive. I am pleased with <ulink url="http://www.finnix.org/">Finnix</ulink> for this use, and it is now my standard first stage restoration Linux.</para>
</sect2>
</sect1>
<sect1 id="ApplicationSpecificNotes">
<title>Application Specific Notes</title>
<para>Here are some notes about backing up particular applications.</para>
<sect2 id="lvm">
<title>Logical Volume Manager</title>
<para>Handling logical volumes turns out to be a bit of a trick: use the <ulink url="http://www.finnix.org/">Finnix</ulink> distribution's startup code to turn LVM on and off. This results in distribution specific code for the first stage of restoration. It is generated in <link linkend="make.fdisk"><filename>make.fdisk</filename></link>. To edit it, search <link linkend="make.fdisk"><filename>make.fdisk</filename></link> for <quote>Hideous</quote>.</para>
<para>LVM required the addition of two new LVM specific scripts, <link linkend="make.lvs"><command>make.lvs</command></link> and <link linkend="mount.lvs"><command>mount.lvs</command></link>. They are only generated and used if there are logical volumes present.</para>
</sect2>
<sect2 id="luks">
<title>LUKS</title>
<para><ulink url="https://gitlab.com/cryptsetup/cryptsetup">LUKS</ulink> allows a user to encrypt a partition. One can then use LVM to install multiple partitions in the encrypted partition.</para>
<warning>
<para>When rebuilding a LUKS partition, you must know the passphrase! We do not handle that!</para>
</warning>
</sect2>
<sect2 id="systemd">
<title>systemd</title>
<para><ulink url="https://fedoraproject.org/wiki/Systemd">systemd</ulink> is a replacement for the System V init scripts formerly common on Linux. The main impact is that systemd replaces System V run levels with targets. See <ulink url="https://fedoraproject.org/wiki/Systemd#Boot_Kernel_Command_Line">Boot Kernel Command Line</ulink> for how to edit the command line when you boot after <link linkend="finishing">first stage restoration</link>.</para>
</sect2>
<sect2 id="selinux">
<title>Selinux</title>
<para>Selinux is disabled on the test machines. <filename>/selinux</filename> is not backed up in any of these scripts. At a guess, you should probably disable selinux after the first stage restoration, and you will probably have some selinux specific tasks to perform before turning it back on.</para>
</sect2>
<sect2 id="grub">
<title>GRUB</title>
<para>The default bootloader in <link linkend="fedora3">Fedora</link> is the <ulink url="http://www.gnu.org/software/grub/">Grand Unified Bootloader (GRUB)</ulink>. It has to run at the end of the first stage, or you won't be able to boot thereafter. To preserve it for first stage restoration, make the following changes:</para>
<itemizedlist>
<listitem>
<para>Edit the penultimate stanza of <link linkend="restore.metadata"><filename>restore.metadata</filename></link>:</para>
<programlisting><![ CDATA [
# Now install the boot sector.
# chroot $target /sbin/lilo -C /etc/lilo.conf
chroot $target /sbin/grub-install /dev/hda
]]></programlisting>
</listitem>
<listitem>
<para>Add the following stanza to <link linkend="save.metadata"><filename>save.metadata</filename></link>:</para>
<programlisting><![ CDATA [# Grub requires these at installation time.
if [ -d usr/share/grub ] ; then # Red Hat/Fedora
crunch usr.share.grub usr/share/grub
fi
if [ -d usr/lib/grub ] ; then # SuSE
crunch usr.lib.grub usr/lib/grub
fi
]]></programlisting>
</listitem>
</itemizedlist>
</sect2>
<sect2 id="tripwire">
<title>Tripwire</title>
<para>If you run Tripwire or any other application that maintains a database of file metadata, rebuild that database immediately after restoring.</para>
</sect2>
<sect2 id="Squid">
<title>Squid</title>
<para>Squid is a HTTP proxy and cache. As such it keeps a lot of temporary data on the hard drive. There is no point in backing that up. Insert <quote>--exclude /var/spool/squid</quote> into the appropriate tar command in your second stage backup script. Then, get squid to rebuild its directory structure for you. Tack onto the tail end of the second stage restore script a command for squid to initialize itself. Here is how I did it over SSH in <link linkend="restore"><filename>restore</filename></link>:</para>
<programlisting><![ CDATA [ssh $target "mkdir /var/spool/squid ; chown squid:squid /var/spool/squid;\
/usr/sbin/squid -z;touch /var/spool/squid/.OPB_NOBACKUP"]]></programlisting>
<para>The last command creates a file of length 0 called .OPB_NOBACKUP. This is for the benefit of <link linkend="arkeia">Arkeia</link>, and tells Arkeia not to back up below this directory</para>
</sect2>
<sect2 id="Arkeia">
<title>Arkeia</title>
<para>These notes are based on testing with Arkeia 4.2.</para>
<para><ulink url="http://www.arkeia.com/">Arkeia</ulink> is a backup and restore program that runs on a wide variety of platforms. You can use Arkeia as part of a bare metal restoration scheme, but there are two caveats.</para>
<para>The first is probably the most problematic, as absent any more elegant solution you have to hand select the directories to restore in the navigator at restoration time. The reason is that, apparently, Arkeia has no mechanism for not restoring files already present on the disk, nothing analogous to <command>tar</command>'s -p option. If you simply allow a full restore, the restore will crash as Arkeia over-writes a library which is in use at restore time, e.g. <filename>lib/libc-2.1.1.so</filename>. Hand selection of directories to restore is at best dicey, so I recommend against it.</para>
<para>The second caveat is that you have to back up the Arkeia data dictionary and/or programs. To do that, modify the <filename>save.metatdata</filename> script by adding Arkeia to the list of directories to save:</para>
<programlisting><![ CDATA [# arkeia specific:
tar cf - usr/knox | gzip -c > $zip/arkeia.tar.gz]]></programlisting>
<para>You <emphasis>must</emphasis> back up the data dictionary this way because Arkeia does not back up the data dictionary. This is one of my complaints about Arkeia, and I have solved it in the past by saving the data dictionary to tape with <ulink url="http://www.estinc.com/">The TOLIS Group's BRU</ulink>.</para> <!-- bad host name: 2003 07 25 -->
<para>The data dictionary will be restored in the script <filename>restore.metadata</filename> automatically.</para>
</sect2>
<sect2 id="amanda">
<title>Amanda</title>
<para><ulink url="http://www.amanda.org/">Amanda</ulink> (The Advanced Maryland Automatic Network Disk Archiver) works quite well with this set of scripts. Use the normal Amanda back-up process, and build your first stage data as usual. Amanda stores the data on tape in GNU tar or cpio format, and you can recover from individual files to entire backup images. The nice thing about recovering entire images is that you can then use variants on the scripts in this HOWTO to restore from the images, or direct from tape. I was able to restore my test machine with the directions from W. Curtis Preston's <ulink url="http://www.oreilly.com/catalog/unixbr/"><citetitle pubwork="book">Unix Backup & Recovery</citetitle></ulink>. For more information on it, see the <link linkend="resources">Resources</link>. The Amanda chapter from the book is <ulink url="http://www.backupcentral.com/components/com_mambowiki/index.php/AMANDA">on line</ulink>.</para>
<para>I made two changes to the script <link linkend="restore"><filename>restore</filename></link>. First, I changed it to accept a file name as an argument. Then, since Amanda's <command>amrestore</command> decompresses the data as it restores it, I rewrote it to cat the file into the pipe instead of decompressing it.</para>
<para>The resulting line looks like this:</para>
<programlisting>cat $file | ssh $target "umask 000 ; cd / ; tar -xpkf - "
</programlisting>
<para>where <command>$file</command> is the script's argument, the image recovered from the tape by <command>amrestore</command>.</para>
<para>Since the command line arguments to <command>tar</command> prohibit over-writing, restore from images in the <emphasis>reverse</emphasis> of the order in which they were made. Restore most recent first.</para>
<para>Amanda may require setting ownership by hand if you back up the amanda data directory with <link linkend="save.metadata"><filename>save.metadata</filename></link>. Something like:</para>
<programlisting>bash# <command>chown -R amanda:disk /var/lib/amanda</command></programlisting>
<para>You can also add that line to your scripts for second state restoration, such as <link linkend="restore"><filename>restore</filename></link>.</para>
<para><link linkend="save.metadata">save.metadata</link>now has support for backing up enough of Amanda to use <command>amrecover</command>. However, this may mean you have to run <link linkend="save.metadata">save.metadata</link> often in order to keep the saved data in synch with amanda's backups.</para>
</sect2>
<sect2 id="ntfs">
<title>NTFS</title>
<para>OK, NTFS isn't an application. It is a file system used by Microsoft operating system Windows NT and its descendents, including Windows 2000 and Windows XP. You can back it up and restore to it from Linux with <command>ntfsclone</command>, one of the NTFS utilities in the ntfsprogs suite, available from <ulink url="http://www.linux-ntfs.org/">http://www.linux-ntfs.org/</ulink>.</para>
<para>These scripts will create NTFS partitions, but will not put a file system on them. It is not clear from the docs whether <command>ntfsclone</command> will lay down a file system on a virgin partition or not.</para>
</sect2>
</sect1>
<sect1 id="SomeAdviceforDisasterRecovery">
<title>Some Advice for Disaster Recovery</title>
<para>You should take your ZIP disk for each computer and the printouts you made, and place them in a secure location in your shop. You should store copies of these in your off-site backup storage location. The major purpose of off-site backup storage is to enable disaster recovery, and restoring each host onto replacement hardware is a part of disaster recovery.</para>
<para>You should also have several restoration Linux floppies or CD-ROMS, and possibly some ZIP drives in your off-site storage as well. Also, have copies of the rescue linux distribution on several of your computers so that they back each other up.</para>
<para>You should probably have copies of this HOWTO, with your site-specific annotations on it, with your backups and in your off-site backup storage.</para>
</sect1>
<sect1 id="WhatNow">
<title>What Now?</title>
<para>This HOWTO results from experiments on one computer. No doubt you will find some directories or files you need to back up in your first stage backup. I have not dealt with saving and restoring X on the first stage, nor have I touched at all on processors other than AMD or Intel.</para>
<para>I would appreciate your feedback as you test and improve these scripts on your own computers. I also encourage vendors of backup software to document how to do a minimal backup of their products. I'd like to see the whole Linux community sleep just a little better at night.</para>
<sect2 id="todo">
<title>To Do</title>
<para>Volunteers are most welcome. Check with me before you start on one of these in case someone else is working on it already.</para>
<itemizedlist>
<listitem>
<para>Add support for <ulink url="https://gitlab.com/cryptsetup/cryptsetup">LUKS</ulink> and the DMCrypt encryption behind it. Currently one must rebuild a LUKS partition manually.</para>
</listitem>
<listitem>
<para>For those systems with <filename>/etc/mke2fs.conf</filename>, get that onto the first stage data so that it is used when partitions are built. We have to copy it into <filename>/etc</filename> before first stage restore.</para>
</listitem>
<listitem>
<para>A partition editor to adjust partition boundaries in the <filename>dev.hdx</filename> file. This will let users adjust partitions for a different hard drive, or the same one with different geometry, or to adjust partition sizes within the same hard drive. A GUI would probably be a good idea here. On the other tentacle, the FSF's <ulink url="http://www.gnu.org/software/parted"><filename>parted</filename></ulink> looks like it will fill part of the bill. It does re-size existing partitions, but with restrictions.</para>
</listitem>
<listitem>
<para><link linkend="make.fdisk"><filename>make.fdisk</filename></link> currently only recognizes some FAT partitions, not all. Add code to <link linkend="make.fdisk"><filename>make.fdisk</filename></link> to recognize others and make appropriate instructions to rebuild them in the output files.</para>
</listitem>
<listitem>
<para>For FAT12 or FAT16 partitions we do not format, write zeros into the partition so that Mess-DOS 6.x does not get confused. See the notes on <command>fdisk</command> for an explanation of the problem.</para>
</listitem>
<listitem>
<para>Translations into other (human) languages.</para>
</listitem>
<listitem>
<para>I've referred to Red Hat Package Manager (rpm) from time to time. What are the equivalent deb commands?</para>
</listitem>
<listitem>
<para>Modify the first stage backup code to only save the current kernel.</para>
</listitem>
</itemizedlist>
</sect2>
</sect1>
<sect1 id="TheScripts">
<title>The Scripts</title>
<para>See the notes in the beginning of each script for a summary of what it does.</para>
<sect2 id="FirstStage">
<title>First Stage</title>
<sect3 id="make.fdisk">
<title><filename>make.fdisk</filename></title>
<para>This script, run at backup time, creates scripts similar to <link linkend="make.dev.hda"><filename>make.dev.hda</filename></link> and <link linkend="mount.dev.hda"><filename>mount.dev.x</filename></link>, below, for you to run at restore time. It also produces data files similar to <link linkend="dev.hda"><filename>dev.hda</filename></link> and <link linkend="dev.hda.sfd"><filename>dev.hda.sfd</filename></link>, below. The names of the scripts and data files produced depend on the device given this script as a parameter. Those script, run at restore time, build and mount the partitions on the hard drive. <filename>make.fdisk</filename> is called from <link linkend="save.metadata"><filename>save.metadata</filename></link>, below.</para>
<!-- #include program listings to make updates easier. C^2 -->
<programlisting>&make.fdisk;</programlisting>
</sect3>
<sect3 id="make.dev.hda">
<title><filename>make.dev.hda</filename></title>
<para>This script is a sample of the sort produced by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, above. It uses data files like <link linkend="dev.hda"><filename>dev.hda</filename></link>, below. It builds partitions and puts file systems on some of them. This is the first script run at restore time.</para>
<para>If you are brave enough to edit <link linkend="dev.hda"><filename>dev.hda</filename></link> or <link linkend="dev.hda.sfd"><filename>dev.hda.sfd</filename></link> (q.v.), say, to add a new partition, you may need to edit this script as well.</para>
<para>If you want make.dev.hda to check for bad blocks when it puts a file system on the partitions, use a "-c" command line option.</para>
<programlisting>&make.dev.hda;</programlisting>
</sect3>
<sect3 id="make.lvs">
<title><filename>make.lvs</filename></title>
<para><filename>make.lvs</filename> is generated by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, but only if logical volumes are present. As the name suggests, it builds the logical volumes and makes file systems on them.</para>
<programlisting>&make.lvs;</programlisting>
</sect3>
<sect3 id="mount.dev.hda">
<title><filename>mount.dev.hda</filename></title>
<para>This script is a sample of the sort produced by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, above. It builds mount points and mounts partitions on them, making the target file system ready for restoring files. This is the second script run at restore time.</para>
<para>If you are brave enough to edit <link linkend="dev.hda"><filename>dev.hda</filename></link> (q.v.), say, to add a new partition, you may need to edit this script as well.</para>
<programlisting>&mount.dev.hda;</programlisting>
</sect3>
<sect3 id="mount.lvs">
<title><filename>mount.lvs</filename></title>
<para><filename>mount.lvs</filename> is generated by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, but only if logical volumes are present. As the name suggests, it mounts the logical volumes ready for restoration.</para>
<programlisting>&mount.lvs;</programlisting>
</sect3>
<sect3 id="dev.hda">
<title><filename>dev.hda</filename></title>
<para>This data file is used at restore time if <command>sfdisk</command> is not present on the restoration Linux. It is fed to <command>fdisk</command> by the script <link linkend="make.dev.hda"><filename>make.dev.hda</filename></link>. It is produced at backup time by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>. Those familiar with <command>fdisk</command> will recognize that each line is an <command>fdisk</command> command or value, such as a cylinder number. Thus, it is possible to change the partition sizes and add new partitions by editing this file. That's why the penultimate command is <command>v</command>, to verify the partition table before it is written.</para>
<programlisting>&dev.hda;</programlisting>
</sect3>
<sect3 id="dev.hda.sfd">
<title><filename>dev.hda.sfd</filename></title>
<para>This data file is used at restore time if <command>sfdisk</command> is present on the restoration Linux system. It is fed to <command>sfdisk</command> by the script <link linkend="make.dev.hda"><filename>make.dev.hda</filename></link>. It is produced at backup time by <link linkend="make.fdisk"><filename>make.fdisk</filename></link>. Each line represents a partition. Thus, it is possible to change the partition sizes and add new partitions by editing this file.</para>
<programlisting>&dev.hda.sfd;</programlisting>
</sect3>
<sect3 id="save.metadata">
<title><filename>save.metadata</filename></title>
<para>This is the first script to run as part of the backup process. It calls <link linkend="make.fdisk"><filename>make.fdisk</filename></link>, above. If you have a SCSI hard drive or multiple hard drives to back up, edit the call to <link linkend="make.fdisk"><filename>make.fdisk</filename></link> appropriately.</para>
<note>
<title>WARNING</title>
<para>Recent kernels have incorporated a new ATA (IDE) hard drive driver, libata. Because of this, parallel ATA (PATA) drives now show up as SCSI drives, as serial ATA (SATA) have always done. However, not all rescue distributions (e.g. Finnix) use this new driver. There is a line toward the bottom of <link linkend="save.metadata"><filename>save.metadata</filename></link> wich very carefully replaces "/dev/sda" with "/dev/hda". Use this as a template if you have multiple IDE hard drives. Comment it out or delete it if this is not an issue for you.</para>
<para>Note that there is no guaranteed mapping! Systems with multiple hard drives may have confusing mappings. Be sure to edit this line carefully. Check it if you add or remove a hard drive of any interface type to or from your system!</para>
<para>N.B: if you have libata IDE drive issues, the grub-install line at the end of <link linkend="restore.metadata"><filename>restore.metadata</filename></link> won't work. If it doesn't, use your rescue disk to do the same. Or burn and boot to the boot image that is made as part of this script. Boot to it and do the second state restore as usual. The second state restore should re-run <filename>grub-install</filename>.</para>
</note>
<programlisting>&save.metadata;</programlisting>
</sect3>
<sect3 id="restore.metadata">
<title><filename>restore.metadata</filename></title>
<para>This script restores metadata from the ZIP disk as a first stage restore.</para>
<para> N.B: if you have libata IDE drive issues, the grub-install line at the end of this script won't work. If it doesn't, use your rescue disk to do the same.</para>
<para>This script runs <command>grub-install</command> toward the end. It may fail. In that case, <ulink url="https://sourceforge.net/p/boot-repair/home/Home/">Boot-Repair</ulink> may repair it for you. Be sure to check options before letting it do its thing.</para>
<programlisting>&restore.metadata;</programlisting>
</sect3>
<sect3 id="first.stage">
<title><filename>first.stage</filename></title>
<para>This script runs the entire first stage restore with no operator intervention.</para>
<para>If you want to check for bad blocks when it puts a file system on the partitions, use a "-c" command line option.</para>
<programlisting>&first.stage;</programlisting>
</sect3>
</sect2>
<sect2 id="SecondStage">
<title>Second Stage</title>
<para>These scripts run on the computer being backed up or restored.</para>
<sect3 id="back.up.all">
<title><filename>back.up.all</filename></title>
<para>This script saves to another computer via an NFS mount. You can adapt it to save to tape drives or other media.</para>
<programlisting>&back.up.all;</programlisting>
</sect3>
<sect3 id="back.up.all.ssh">
<title><filename>back.up.all.ssh</filename></title>
<para>This script does exactly what <link linkend="back.up.all"><filename>back.up.all</filename></link> does, but it uses SSH instead of NFS.</para>
<programlisting>&back.up.all.ssh;</programlisting>
</sect3>
<sect3 id="restore.all">
<title><filename>restore.all</filename></title>
<para>This is the restore script to use if you backed up using <link linkend="back.up.all"><filename>back.up.all</filename></link>.</para>
<programlisting>&restore.all;</programlisting>
</sect3>
<sect3 id="restore.all.ssh">
<title><filename>restore.all.ssh</filename></title>
<para>This is the restoration script to use if you used <link linkend="back.up.all.ssh"><filename>back.up.all.ssh</filename></link> to back up.</para>
<programlisting>&restore.all.ssh;</programlisting>
</sect3>
</sect2>
<sect2 id="BackupServerScripts">
<title>Backup Server Scripts</title>
<para>The SSH scripts above have a possible security problem. If you run them on a firewall, the firewall has to have access via SSH to the backup server. In that case, a clever cracker might also be able to crack the backup server. It would be more secure to run backup and restore scripts on the backup server, and let the backup server have access to the firewall. That is what these scripts are for.</para>
<para>These scripts back up and restore the target completely, not just the stage one backup and restore. <filename>get</filename> backs up the bare metal archive separately so that you can make a CD-ROM or NFS mount from it.</para>
<para>I use these scripts routinely.</para>
<sect3 id="get">
<title><filename>get</filename></title>
<programlisting>&get.tester;</programlisting>
</sect3>
<sect3 id="post.sh">
<title><filename>post.sh</filename></title>
<para>This is called at the end of scripts to get target backups, such as <link linkend="get"><filename>get</filename></link>. It does some post-processing to be sure we have transferred the data from the target to the server correctly.</para>
<para>Note that it expects and tests copies of the checksums made on the target. It also creates new checksum files, which include the archive tarball which <link linkend="get"><filename>get</filename></link> creates.</para>
<programlisting>&post.sh;</programlisting>
</sect3>
<sect3 id="get.target">
<title><filename>get.target</filename></title>
<programlisting>&get.target;</programlisting>
</sect3>
<sect3 id="restore">
<title><filename>restore</filename></title>
<programlisting>&restore.tester;</programlisting>
</sect3>
</sect2>
<sect2 id="misc.files">
<title>Miscellaneous Files</title>
<sect3 id="install">
<title><filename>install</filename></title>
<para>This little script just installs things on target systems and sets up a few directories. It tests for the presence of the perl UUID module.</para>
<para>It would be a useful basis for an RPM or deb package. The placement of files is based on the <emphasis><ulink url="http://www.pathname.com/fhs/">Filesystem Hierarchy Standard</ulink></emphasis>, version 2.3, announced on January 29, 2004.</para>
<programlisting>&install;</programlisting>
</sect3>
</sect2>
</sect1>
<sect1 id="Resources">
<title>Resources</title>
<para>In no particular order. These are things you might want to investigate for yourself. A listing here should not be taken as an endorsement. In fact, in many case I have not used the product and cannot comment on it.</para>
<itemizedlist>
<listitem>
<para><ulink url="http://osdev.berlios.de/netboot.html">Network-booting Your Operating System</ulink> describes several techniques for booting across a network, using <ulink url="http://www.gnu.org/software/grub/">grub</ulink> and some other tricks. I haven't tried it, but I have a sneaky suspicion that with an especially trained floppy diskette, you could get your entire first stage image onto the computer to be restored.</para>
</listitem>
<listitem>
<para><quote><ulink url="http://btmgr.webframe.org/">Smart Boot Manager (SBM)</ulink> is an OS independent and full-featured boot manager with an easy-to-use user interface. There are some screen shots available.</quote> It is essential if your BIOS will not allow you to boot to CD-ROM and you want to use a CD-ROM based Linux for Stage 1 recovery.</para>
</listitem>
<listitem>
<para><ulink url="http://www.oreilly.com/catalog/unixbr/author.html">W. Curtis Preston</ulink>'s excellent <ulink url="http://www.oreilly.com/catalog/unixbr/"><citetitle pubwork="book">Unix Backup & Recovery</citetitle></ulink>. This is the book that got me started on this bare metal recovery stuff. I highly recommend it; <ulink url="http://www2.linuxjournal.com/lj-issues/issue78/3839.html">read my review</ulink>. However, you should probably get the latest edition.</para>
</listitem>
<listitem>
<para><ulink url="http://www.toms.net/rb/">tomsrtbt</ulink>, <quote>The most Linux on 1 floppy disk.</quote> Tom also has links to other small disties.</para>
</listitem>
<listitem>
<para>The <ulink url="http://www.tldp.org/">Linux Documentation Project</ulink>. See particularly the <quote><citetitle pubwork="article">LILO, Linux Crash Rescue HOW-TO</citetitle>.</quote></para>
</listitem>
<listitem>
<para>The Free Software Foundation's <ulink url="http://www.gnu.org/software/parted"><filename>parted</filename></ulink> for editing (enlarging, shrinking, moving) partitions.</para>