In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".
Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy. Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.
Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown. Murphy's Law states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.
Causes of data loss edit
Data may be lost as a result of:
- Device failure
- Software bugs and poorly designed user interfaces
- Software lock-in
- Human error
- Overreliance on online services
For a more detailed explanation, read the sub page Backup/Causes of data loss.
Types of data storage edit
Magnetic and optical edit
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.
Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.
Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).
Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing vendor lock-in.
Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.
In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as QpxTool and Nero DiscSpeed, before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.
Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation. However, humidity should be avoided during hard drives' operation, as it increases the likelihood of failure. ard drive motors' lubrication slowly loses effectiveness as well. Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.
On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists. If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.
Humidity in an enclosed storage location can be reduced using silica gel baglets.
Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm, as it is water-resistant and has no internal electrical components vulnerable to single effect events.
There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.
Flash storage edit
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time. This is also referred to as "bit fading".
Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.
The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.
While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.
Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.
Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.
As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.
Memory cards edit
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.
This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through USB On-The-Go would compromise necessary ergonomy.
Online services edit
Online services like cloud storage are just an extension and not a substitute to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies, and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well. However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.
Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little file management knowledge.
For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.
Privacy on cloud storage services edit
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers. It does not necessarily mean they will, but that they technically are able to.
Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.
If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.
If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.
Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in § Flash storage, it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.
For dedicated cameras and camcorders, memory cards can be cycled through.
File system structure image edit
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being orphaned and only detectable by forensic software through file headers and footers. Fragmented files would need to be puzzled together.
The file system structure (or header) is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.
While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (LBAs) shortly after the file system header itself), it is a fallback solution which is better than nothing.
Verifying backups edit
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.
The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.
For archive file formats such as ZIP and 7z and GZip, archive managers such as 7-Zip on Windows and File Roller on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the
badblocks command on Linux and the freemium shareware "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.
Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in § Magnetic and optical.
Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data. However, as stated earlier (§ Flash storage), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the
dd command-line tool. Special caution should be taken not to select the wrong target device.
If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.
Other tips edit
- Lost backups: If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
- The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create logical structures of directories. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
- If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
- Distinct backup media: Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
- Tracking manual backups: Manual backups of individual directories can be tracked using a text file such as
backupinfo.txt, containing the time of the last backup and optionally additional information such as the name of the backup device.
- Reputable brands: Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
- Proprietary formats: Software that stores backups in proprietary formats such as that of Acronis (
.tib) should be used with caution due to the possibility of vendor lock-in. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
- Encryption: In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.
Compressed archives edit
Compression ratio edit
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.
Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.
Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("air") of the image and text-based files. Performance matters on disk image files due to their typically large size.
Magnification of damage edit
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where solid compression causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.
If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.
To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or hex editor), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.
This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and digital stains on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the Commons: category: Bit-blending experiment.
Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see Gzip § Damage recovery.
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.
On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through cluster overhead, whereas archive formats handle many small files efficiently, even with compression deactivated.
However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as Zip, RAR, and 7z would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.
Note that a program stream video (most mobile video) with an end-of-file moov atom depends on completeness or else be unplayable. This is not the case for transport stream video, frequently used by dedicated camcorders such as those of Panasonic and Sony (AVCHD).
Preventing data lock-in edit
Wherever you store your data, verify in advance that proper export functionality exists.
For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.
On the Android mobile operating system, applications might store user data inside the
/data/ folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, root access must be unlocked.
Format obsolescence edit
There is a slight likelihood that formats used today will become unsupported in future.
A past example is the HD-DVD, an optical disc format which did not succeed in the market. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.
Other areas edit
Web browsing edit
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.
Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.
Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.
Web forms edit
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.
Browser extensions such as Textarea Cache prevent loss of form data by backing it up automatically.
Sites for which this is not wanted can be added to an exclusion list in the extension's settings.
Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.
The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.
Browsing history edit
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.
If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named
places.sqlite and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.
Note that on mobile phones, restrictions by operating system could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as PaperBak (open-source), which can encode digital data into printable machine-readable matrix bar code.
Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.
Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.
Commercial Backup products edit
See Also edit
- Wikipedia: Backup
- Wikipedia: Backup
- How to estimate the lifespan of LTO tapes – "30 years" (source)
- HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru
- Military warns EMP attack could wipe out America
- Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections – Kurzgesagt - in a nutshell (Video, 09m43s)
- Could an EMP attack be a part of the end times?
- "Verbatim Gold" (silver and gold) and "M-Disc" (rock-like)
- "Understanding Life Expectancy of Flash Storage". www.ni.com. 2022-06-01. Retrieved 2022-10-18.
- "Understanding Life Expectancy of Flash Storage". www.ni.com. 2020-07-23. Retrieved 2020-12-19.
- For example, Dropbox retains unused accounts for up to one year.
- "Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user @ValdikSS, April 10, 2019 (multi-tweet)
- You Can't Control Your Data in the Cloud – Karl Voit
- Web.de mail terms of service with six-month retention policy: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."
- 3 Things I Wish I Knew when I First Started on YouTube (at 01m02s) by Video Creators TV
- Janet Jackson will have your Instagram permanently deleted if you post tour videos (FactMag, October 21, 2015)
- John Gruber (2014-09-03). "Security Trade-Offs". Retrieved 2023-01-21.
- Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: Daniel AJ Sokolov (2022-01-26). "Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein" (in German). Retrieved 2023-01-21.
- How to Pack and Store DVDs and Discs Long-Term – Mini Mall Storage
- [https://www.clir.org/pubs/reports/pub121/sec5/ – 5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]
- Robin Harris (2016-03-08). "Heat doesn't kill hard drives. Here's what does". ZDNet. Retrieved 2022-11-02.
- Brian Beach (2014-05-12). "Hard Drive Temperature—Does It Matter?". BackBlaze. Retrieved 2022-11-02.
- "Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community". discussions.apple.com. 2014-06-14. Retrieved 2023-02-08.
In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity.
- Building data liberation infrastructure – Beepb00p.xyz
- Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to this guide by Joey Adams.
- Textarea Cache for Firefox
- Benson, Ryan. "Archived History files removed from Chrome v37". Obsidian Forensics. Archived from the original on 2014-10-10.
- Export History/Bookmarks to JSON/CSV*/XLS* extension from the Google Chrome web store
- Features - Kiwi Browser