These open source tools can help storage administrators or home users backup, compress, manage and secure their stored files.
It's no secret that digital storage needs are increasing rapidly. Experts at IDC predict that by 2020, there will be 40 zettabytes (ZB) of data in the digital universe, or 5,247 GB per person alive at that time. That's the equivalent of 57 times the number of all the grains of sand on all the beaches on earth.
The open source community has developed a number of tools to help people store, secure and manage all that data. This month, we're spotlighting 58 of the best of these projects for home users, businesses and enterprises. They include software for setting up a NAS or SAN appliance on standard hardware, backup and synchronization applications, RAID tools and more.
If you know of additional open source storage projects that you feel should be on the list, please make note in the comments section below.
Ceph describes itself as a "distributed object store and file system designed to provide excellent performance, reliability and scalability." It's built on the Reliable Autonomic Distributed Object Store (RADOS) and allows enterprises to build their own economical storage devices using commodity hardware. Operating System: Linux
The FreeNAS website touts itself as "the most potent and rock-solid open source NAS software," and it counts the United Nations, The Salvation Army, The University of Florida, the Department of Homeland Security, Dr. Phil, Reuters, Michigan State University and Disney among its users. You can use it to turn standard hardware into a BSD-based NAS device, or you can purchase supported, pre-configured TrueNAS appliances based on the same software. Operating System: FreeBSD
Red Hat-owned Gluster is a distributed file system that can handle big data—up to 72 brontobytes. It promises high availability and performance, an elastic hash algortithm, an elastic volume manager and more. Operating System: Linux
Now owned by Xyratex, Lustre is a highly-scalable file system that provides fast performance. It can handle "tens of thousands of nodes and petabytes of storage." Note that although the name is similar to "Gluster," the two are completely independent projects. Operating System: Linux
Like FreeNAS, NAS4Free makes it easy to create your own BSD-based storage solution from standard hardware. It promises a low-cost, powerful network storage appliance that users can customize to their own needs. Operating System: FreeBSD
Openfiler aims to combine NAS features (NFS, CIFS and HTTP) with SAN features (iSCSI, FC) in a very easy-to-setup storage appliance. It's been downloaded more than a million times, and its users include Motorola, Pratt & Whitney, BillMeLater and the London Metropolitan Police. Paid support is available. Operating System: Linux
Similar to Openfile, OpenSMT can be used to create storage devices with some SAN and some NAS features. It uses the ZFS file system. Operating System: OpenSolaris
8. Open Media Vault
This NAS solution boasts really easy-to-use storage management, fast setup, uninterruptible power supply support and statistics reporting. It's based on Debian Linux and offers a number of plug-ins to extend its capabilities. Operating System: Linux
9. Turnkey Linux
Turnkey offers more than 100 different software appliances based on open source software. Among them is a file server that offers simple network attached storage. Operating System: Linux
This popular file system is incorporated into many other open source storage projects. It offers excellent scalability and data integrity, and it's available for most Linux distributions. Operating System: Solaris, OpenSolaris, Linux, OS X, FreeBSD
DRBD provides high availability for storage by applying RAID-1 to the network. Support and training are available through the project owner, LinBit. Operating System: Linux
This piece of the Linux kernel makes it possible to set up and manage your own RAID array using standard hardware. It also provides reporting capabilities. Operating System: Linux
Raider applies RAID 1, 4, 5, 6 or 10 to hard drives. It relies heavily on Linux's mdadm. Operating System: Linux
Monitor your OS X-based RAID arrays with this helpful tool. It runs in the background, and if it detects an error or failure, it will send you an email. Operating System: OS X
This project aims to make it easy for anyone to use RAID on their Linux systems. The website explains the project name by noting, "Salamanders are the only vertebrates that can regenerate limbs. In the same way, a system installed with Salamander can regenerate after a hard-drive failure." Operating System: Linux
SnapRAID backs up disk arrays, primarily those used to store home media collections. It allows recovery even if two separate disks in the array fail. Operating System: Windows, Linux, OS X
Backup and Synchronization
The "Advanced Maryland Automatic Network Disk Archiver," a.k.a. AMANDA, backs up networks to tape or hard drives. Paid support and a cloud-based version are available through Zmanda, which is owned by Carbonite. Operating System: Windows, Linux, OS X.
18. Areca Backup
A good option for backing up a single system, Areca aims to be simple and versatile. Key features include compression, encryption, filters and support for delta backup. Operating System: Windows, Linux
Designed for enterprise users, BackupPC claims to be "highly configurable and easy to install and maintain." It backs up to disk only (not tape) and offers features that reduce the amount of storage space necessary, as well as the number of I/Os. Operating System: Windows, Linux
Another enterprise-grade open source back solution, Bacula offers a number of advanced features for backup and recovery, as well as a fairly easy-to-use interface. Commercial support, training and services are available throughBacula Systems. Operating System: Windows, Linux, OS X
Conceived as a replacement for True Image or Norton Ghost, Clonezilla is a disk imaging application that can do system deployments, as well as bare metal backup and recovery. Operating System: Linux
22. Create Synchronicity
Create Synchronicity's claim to fame is its lightweight size—just 220KB. It's also very fast, and it offers an intuitive interface for backing up standalone systems. Operating System: Windows
Another imaging and cloning solution, FOG makes it easy for administrators to backup networks of all sizes. It also includes features like virus scanning, memory testing, disk wiping, testdisk, and file recovery. Operating System: Linux, Windows.
This file and folder synchronization tool can be very useful for backup purposes. It can save a lot of time and receives rave reviews from users. Operating System: Linux, Windows, OS X
Built for developers, FullSync offers synchronization capabilities suitable for backup purposes or for publishing Web pages. Features include multiple modes, flexible tools, support for multiple file transfer protocols and more. Operating System: Linux, Windows, OS X
Grsync provides a graphical interface for rsync, a popular command line synchronization and backup tool. It's useful for backup, mirroring, replication of partitions, etc. Operating System: Linux, Windows, OS X
Award-winning LuckyBackup offers simple, fast backup. Note that while it is available in a Windows version, it was first developed for Linux, and the Windows version is not as stable. Operating System: Linux, Windows
This imaging software is helpful for creating recovery partitions on your system or for deploying multiple systems with identical images. It's extremely fast, and it supports backup to another system on the network, as well as to a different partition on the disk. Operating System: Linux
Downloaded more than 750,0000 times, Redo claims to be “so simple that anyone can use it.” In addition to backups, it can do bare metal restores in as little as ten minutes. Operating System: Windows, Linux
As you might guess from the name, this is a data synchronization tool. Key features include blacklisting, analysis and restore. Operating System: Windows, Linux, OS X
A client-server system, UrBackup does both file and image backups. It requires very little configuration and does incremental backups while you work without slowing your system. Operating System: Windows, Linux
Online Data Storage
This app allows users to set up cloud-based storage services on their own servers. It supports FTP, SFTP or FTPS file syncing. Operating System: Windows
Formerly known as AjaXplorer, this app helps enterprises set a file-sharing service on their own servers. It's very easy to install and offers an attractive, intuitive interface. Operating System: Windows, Linux (Android and iOS clients available)
Another self-hosted cloud storage solution, SparkleShare is a good storage option for files that change often and are accessed by a lot of people. (It's not as good for complete backups.) Because it was built for developers, it also includes Git. Operating System: Windows, Linux, OS X.
Syncany is similar to Dropbox, but you can use it with your own server or one of the popular public cloud services like Amazon, Google or Rackspace. It encrypts files locally, adding security for sensitive files. Operating System: Windows, Linux, OS X
36. Libvirt Storage Management
Libvirt is a storage virtualization API that allows users to set up storage pools and volumes. Drivers are available for all of the popular hypervisors. Operating System: Linux
The Online Hierarchical Storage Manager, or OHSM, helps control storage costs by moving data to high- or low-cost media in accordance with pre-determined policies. It allows administrators to set up rules for allocation (for the first time a file is saved) and relocation (for files that haven't been accessed in a while). Operating System: Linux
This tool makes it easier to manage virtualized environments, including virtualized storage. It's highly scalable and includes some advanced security features. Operating System: Linux
This helpful compression utility supports multiple formats, including .zip files and .7z files, which compresses up to 70 percent smaller than .zip. It also includes AES-256 encryption capabilities. Operating System: Windows, Linux, OS X
Multiple award-winning ArcConvert can convert archive files among dozens of different formats. It automatically detects passwords, and it's available in several different languages. Operating System: Windows
ArcThemALL! is very useful for compressing or extracting batches of archive files. It can compress to three different file formats and extract 33 different formats. Operating System: Windows
This Mac-only tool provides 7-zip compression for OS X. Like the original 7-zip, it also extracts more than a dozen other archive formats. Operating System: OS X
This app works with an amazing total of more than 150 archived file formats. It can compress, convert, split, encrypt and securely delete stored files. Operating System: Windows, Linux, OS X.
For storage managers, BleachBit's secure deletion features can come in handy. If you're using it on a single PC, it can also improve performance and protect privacy by freeing cache, deleting cookies, clearing Internet history, shredding temporary files, deleting logs, and discarding other junk. Operating System: Windows, Linux
45. Darik's Boot And Nuke
This app can securely wipe an entire disk so that the data cannot be recovered. The owner of the app, Blancco, offers related paid products, including some that support RAID. Operating System: OS Independent
This utility securely deletes individual files. It includes a helpful scheduler. Operating System: Windows
Another option for secure file deletion, this app allows the user to determine how many times deleted data is overwritten depending on the sensitivity of the data being deleted. It offers fast performance and can handle large files. Operating System: Windows
Document Management Systems (DMS)
48. bitfarm-Archiv Document Management
This professional-caliber tool aims to make it easy to archive and manage documents for a large organization. A paid enterprise version and paid services are available. Operating System: Windows
Highly rated DSpace describes itself as "the software of choice for academic, non-profit, and commercial organizations building open digital repositories." It offers a Web-based interface and very easy installation. Operating System: Windows, Linux, OS X
Epiware offers customizable, Web-based document capture, management, storage, and sharing. Paid support is also available. Operating System: Windows, Linux
This document management system for mid-size to large enterprises boasts that it requires nearly no end user training. Paid enterprise and cloud versions are also available. Operating System: OS Independent
Standards-based OpenKM includes features like version control and file history, metadata, scanning, workflow and search. It also offers some collaboration and social business capabilities. Operating System: OS Independent
Downloaded nearly 3 million times, AxCrypt is the "leading open source file encryption software for Windows." It works with the Windows file manager and with cloud-based storage services like Dropbox, Live Mesh, SkyDrive and Box.net. Operating System: Windows
Extremely lightweight, the 44KB Crypt promises very fast encryption and decryption. You don't need to install it, and it can run from a thumb drive. However, users should note that it doesn't have a GUI. Operating System: Windows
55. Gnu Privacy Guard (GPG)
Gnu's implementation of the OpenPGP standard allows users to encrypt and sign data and communication. A very mature project, it's been under active development for sixteen years. Operating System: Linux
The original GPG runs only on Linux, but this version ports it for Windows. It's easy to install and includes plug-ins for Outlook and Windows Explorer. Operating System: Windows
57. GPG Tools
Lest OS X users feel left out, this project ports GPG to the Mac. The latest version supports Mavericks, as well as older versions of OS X. Operating System: OS X.
Extremely popular, this utility has been downloaded nearly 30 million times. It can encrypt both single files or entire drives or partitions. Operating System: Windows, Linux, OS X.
Photo courtesy of Shutterstock.