| Data deduplication is a storage technique that keeps one copy of each repeated block of data and replaces the rest with pointers back to it. It gets judged on two numbers an owner can read off an invoice: what the business pays to store data each month, and how long a restore takes when somebody asks for a file back. |
A 40-person firm in Christchurch gets a bill for its offsite backup. The line says 9 TB, and the finance manager reads it twice, because the business has 40 staff and nobody can picture nine thousand gigabytes of anything.
Then a partner asks for a file from a project that closed in 2023. It arrives on Thursday.
The bill and the Thursday have different causes and the same habit behind them. The firm is storing the same data over and over, so there is more of it to pay for and more of it to search.
The figures below are illustrative. Take that firm at 9 TB of backup storage, billed at $30 per terabyte per month, which comes to $3,240 a year. When the volume was measured, 55 per cent of it turned out to be repeated blocks.
The same email attachment sitting inside forty mailboxes. Fourteen nightly copies of a virtual machine that changes by a few hundred megabytes a day. A project folder duplicated when somebody left and never reconciled against the original.
That percentage came from a tool already installed on the server, run in an afternoon, at no cost. Most firms have never run it.
The Thursday had its own bill. Reading back through fourteen months of backup sets to find one version took an engineer most of a Saturday, and the firm was charged about $2,400.
This post is about the volume of what gets stored and the time it takes to read back. How often to back up, how many copies to hold and where to hold them belong to our guide to a backup strategy. What follows deals with the layer underneath, meaning how much space that plan consumes and why.
Why Does a Storage Bill Keep Growing When the Business Has Not?
Because stored volume grows with activity, not with headcount. A firm of 40 people can hold ten times the data it held five years ago without hiring anybody, and four ordinary habits do most of it.
- Attachments that live in every mailbox they were sent to, and again in every backup of every mailbox
- Nightly copies of the same virtual machine, where each night holds a full picture of a disk that barely changed
- Folders duplicated during a migration, a departure or a restore that nobody deleted afterwards
- Working files kept in a personal drive as well as the shared one, because somebody wanted a version they controlled
None of that is carelessness. Storage has been cheap enough to add for twenty years, and no job description has ever included subtracting it.
The bill records the result without explaining it. A line reading 9 TB tells an owner nothing about how many of those terabytes are the same October board pack, saved by several people into four folders and then captured by fourteen consecutive backups.
Would deleting more solve it?
Partly, and the two work on different things. Deleting is a decision about what the business still needs, and that ground belongs to a retention policy. business.govt.nz gives the same instruction in its guidance for New Zealand businesses: do not store data longer than you need to.
Deduplication works on what you have decided to keep. It removes repetition without anybody choosing what to lose, so it is the safer of the two to start with.
What Is Data Deduplication?
Data deduplication is a process that finds repeated blocks of data across a storage volume, keeps one copy of each, and replaces the rest with a pointer. The industry shortens it to dedup, and the Windows Server documentation uses that abbreviation too.
The mechanism matters, because it explains every limit further down. Microsoft’s documentation for Windows Server describes a post-processing model: data is written to disk unoptimised, then a scheduled job breaks files into variable-size chunks, identifies the unique ones, and moves them into a chunk store.
What is left in place of the original file is a reparse point, a small tag telling the file system to fetch the content from the chunk store instead. Staff opening the file see no difference at all.
Is deduplication the same thing as compression?
No. Compression shrinks each file on its own by encoding it more efficiently, and deduplication removes repetition between files across a whole volume. The documentation draws the same line: NTFS compression optimises a file at write time, where deduplication finds patterns shared between files that compression cannot see.
The two stack. Windows can compress the chunks after deduplicating them, so a measured saving often comes out ahead of what either gives on its own.
An older feature called Single Instance Store, or SIS, did a cruder version of this and only caught files that were byte-for-byte identical. It was removed in Windows Server 2016, and the name still turns up in documentation written before then.
How Much Space Does Deduplication Save?
It depends entirely on how much repetition the data holds, which is why any vendor quoting a fixed percentage before looking at your volume is guessing. The pattern across workloads is consistent enough to plan against.
Files that are already compressed or encrypted deduplicate poorly. Photographs, video, and zipped archives carry few repeated blocks, so a volume full of JPEGs will show almost no saving however it is configured.
| Where the data sits | How well it deduplicates | Why |
|---|---|---|
| Backup repository | Best | Each nightly set repeats almost all of the one before it |
| Virtual machine library | Very well | Machines built from one image share nearly every block |
| Shared file server | Well | Teams keep their own copies of the same documents |
| Mailbox archive | Moderately | Attachments repeat across recipients, message bodies do not |
| Photo, video and design files | Poorly | Already compressed, so there is little repetition left to find |
| Encrypted volumes | Not at all | Encryption removes the patterns deduplication looks for |
What do firms find when they measure?
More than they expected on the backup target and less than they hoped on the file share. The backup number is high because the repetition is structural, and the file share number depends on how the team works.
A firm that keeps one master copy of everything will see a modest saving. A firm where four people each hold their own version of the price list will see a large one, and will also learn something about how it works.
That second finding is worth as much as the storage. The measurement names the folders where duplication collects, and that list is the fastest way into a conversation about how the team files things.
What does a deduplication ratio mean?
A deduplication ratio compares the size of the data before optimisation against the space it occupies after, so 2:1 means the volume holds twice what it appears to. Ratios sound impressive, and they describe the data more than the technology that processed it.
The number that matters to an owner is the terabytes on the invoice, so ask for the saving in gigabytes and dollars instead of a ratio.
Can You Measure the Saving Before Buying Anything?
Yes, and the tool is already on the server. Microsoft ships a savings evaluation tool called DDPEval, found at C:\Windows\System32\DDPEval.exe once the deduplication role is installed, and it reports what a folder or volume would save before anything is changed.
It runs against local drives, cluster shared volumes and network shares. The output is eight lines of text: processed size, optimised size, space saved and the percentage, with the compression saving reported separately so a vendor cannot claim credit for both.
Run it on the backup repository first, because that volume holds the most repetition. Nothing else has to be gathered before somebody can decide.
What should you do with the result?
Take the percentage, apply it to the terabytes on your current invoice, and multiply by the price per terabyte. If the answer is smaller than the cost of the work, stop there and spend the money elsewhere.
If the volume is mostly photographs there may be nothing worth doing, and the afternoon spent finding that out still beats guessing.
Which Systems Should Deduplication Run On?
Three workloads are safe to switch on without testing, and the documentation names them: general purpose file servers holding team shares and home folders, virtual desktop infrastructure servers, and virtualised backup applications. All three combine heavy repetition with the idle time the optimisation job needs.
Three more need evaluating first. General purpose Hyper-V hosts, SQL servers and line-of-business servers can benefit, and they can also end up slower, because deduplication adds latency to random reads and a database reads randomly by nature.
The rule underneath both lists is idle time. A file server with nothing running after seven at night is an easy candidate. A system busy around the clock is harder, because the optimisation job then competes with the work people are doing.
Does it need a lot of memory?
Less than people expect. Microsoft’s stated minimum is 300 MB plus 50 MB for every terabyte of logical data, so a 10 TB volume needs 800 MB to function, and 10 GB to run at full speed.
Where a virtual environment is involved the sizing question belongs with the host, and our post on server virtualisation covers that.
Is there a licence cost?
No, on Windows Server. Deduplication is a role included in the licence the business already holds, so the only money involved is the work to switch it on and keep an eye on it.
Does Deduplication Make a Restore Slower or Faster?
Faster in most real restores, and slower in one specific case. A smaller repository means less data to read and less to move across a connection, so a full restore of a deduplicated volume usually finishes sooner than the same data stored in full.
The exception is a single-file restore from a heavily fragmented chunk store on spinning disks, where the system may read from scattered locations. On solid-state storage that penalty largely disappears, because access time is the same wherever the data sits.
The Thursday file in the opening was a search problem before it was a storage problem. A smaller repository shortens the search, because there is less to read through to reach the version somebody wants.
What has to be tested before it is trusted?
A restore. Deduplication changes how data is stored, so the only proof that it has not broken anything is pulling files back and opening them, which business.govt.nz names as a step in its own backup guidance.
Test a full-volume restore and a single-file restore, and time both. Write both down. They are the figures you compare against next year, and the ones that outlast a change of provider.
Do the test before the volume is optimised as well, if there is time. Two sets of timings turn an argument about whether deduplication helped into a table.
Can the chunk store itself be damaged?
It can, and Windows runs a weekly job against exactly that risk. An integrity scrubbing job looks for corruption from disk faults, and the system keeps spare copies of chunks referenced more than a hundred times so that popular data can be rebuilt.
Dedup is not a substitute for having more than one copy of the data, and our post on BCDR and backup draws that line.
What Does Deduplication Not Fix?
It does not give the business a second copy of anything, and this is the misunderstanding worth clearing up before anybody signs off on it. Deduplication makes one copy smaller, and a smaller single copy is still a single copy.
It also does not decide what should be kept. A volume full of files nobody needs will deduplicate neatly and still be full of files nobody needs, which is why measuring usually comes after somebody has read the retention schedule.
Does it protect against ransomware?
No, and it can make the wrong assumption easier to hold. Ransomware encrypts data, and encrypted data does not deduplicate, so an infected volume may show its saving collapsing before anybody notices the files are unreadable.
A falling deduplication ratio can work as an early warning. It stops nothing on its own, and the controls that do help belong to a different discipline.
Does it replace archiving old data?
No. Archiving moves data somewhere cheaper and slower, and deduplication shrinks it wherever it happens to be held. A firm holding fifteen years of project files benefits from both, in that order.
How Much Does Data Deduplication Cost?
For the illustrative firm above, the 55 per cent measured saving takes 9 TB down to about 4 TB, so the $3,240 a year of storage becomes $1,440. Against that, monitoring the deduplication jobs as part of managed support runs about $600 a year, leaving a net recurring saving near $1,800.
The one-off number is separate and should stay separate. Installing the role, choosing the usage type, scheduling the jobs around the backup window and running a tested restore comes to about $1,900, set against the $2,400 that one Saturday file search already cost.
Both figures move with the shape of the data. The recurring saving covers the recurring cost several times over, and the one-off spend is about what one bad restore weekend costs.
The saving also compounds in a direction owners rarely price. Every terabyte removed is a terabyte that does not have to be copied offsite each night, so the connection carries less and the backup window closes earlier.
When is it not worth doing?
When the data does not repeat, when the volume is small enough that the saving is measured in tens of dollars, or when every server runs flat out and there is no window for the optimisation job.
DDPEval answers the first two in an afternoon and costs nothing, so the question rarely has to stay open.
How Do You Enable Data Deduplication on Windows Server?
Windows Server data deduplication is a role and not a product, so there is nothing to buy and nothing to install beyond the feature itself. Measure first, then work through five steps:
- Run DDPEval against the backup repository and the main file share, and write down the reported saving for each.
- Multiply that percentage by the terabytes on your current storage invoice to get the annual figure.
- Install the deduplication role on the volumes that scored well, and choose the usage type that matches the workload.
- Move the optimisation and garbage collection jobs to a window when the server is idle, clear of the backup.
- Restore a full volume and a single file, time both, and keep the two numbers for comparison next year.
None of the five needs new hardware or a new licence. Install-WindowsFeature adds the role, Enable-DedupVolume switches a volume on with the usage type you pick, and the rest is a schedule and a test.
Which usage type should you choose?
Default for a general purpose file server, Hyper-V for virtual desktop hosts, and Backup for a virtualised backup application. The usage type sets the defaults, including how old a file has to be before it is optimised, which is three days on Default and zero on Backup.
Picking the wrong one is not fatal and it is worth getting right, because Backup optimises files the moment they land and Default leaves recent work alone.
The firm in the opening did the first step on a Tuesday afternoon. The number it produced was larger than the quote to do the rest.
Find Out What Half Your Storage Bill Is Buying
Exodesk has supported South Island businesses since 1989 and works with clients across Canterbury, Otago and Southland from offices in Christchurch and Dunedin. Our cloud and storage team measures what data deduplication would save on your file servers and backup repository, configures it around your backup window, and tests the restore afterwards.
Contact us today to discuss how we can help your business or connect with us on LinkedIn to stay updated with more insights.
Frequently Asked Questions
What is data deduplication in simple terms?
Deduplication stores one instance of anything repeated and leaves a pointer everywhere else it used to sit. A volume then holds more information than the room it takes up, and staff opening a file notice no difference, because the content is fetched behind the scenes.
How much storage does deduplication save?
It depends on how much the data repeats, so the honest answer comes from measuring, and any fixed percentage quoted in advance is a guess. Backup repositories and virtual machine libraries save the most, because each copy repeats the last. Volumes of photographs or video save close to nothing.
Is dedup the same as compression?
No. Compression shrinks each file individually, and deduplication removes repetition between files across a whole volume. They work well together, and Windows can compress chunks after deduplicating them.
Does deduplication cost extra on Windows Server?
No. It is a role included in the Windows Server licence a business already holds, so the only cost is the work to set it up and keep an eye on it.
Will deduplication slow our systems down?
It can, on the wrong workload. The optimisation job runs after data is written and needs idle time, so file servers and backup targets suit it and database servers running flat out often do not. Random reads from spinning disks are where the latency shows.
What is DDPEval and how do we use it?
DDPEval is a free evaluation tool Microsoft ships with the deduplication role, and it reports what a folder or volume would save before anything is changed. Run it from a command prompt against the backup repository first, because that volume usually holds the most repetition.
Is deduplicated data still safe to restore?
Yes, provided the restore has been tested. Deduplication changes the way data is stored while leaving the data itself intact, and Windows runs a weekly integrity job that keeps spare copies of frequently referenced chunks. Test a full restore and a single-file restore before relying on it.
Does deduplication replace having backups?
No. It reduces the size of what you store, and it does not give you a second copy in a second place. Those are two different jobs and a business needs both. Deduplication makes the copy you have cheaper to keep, and a backup strategy decides how many copies exist and where they are held.
Where should deduplication never be turned on?
On volumes holding mostly compressed or encrypted files, where there is nothing repeated to find, and on servers with no idle period for the optimisation job to use. Microsoft also lists workloads that need evaluating first, including SQL and line-of-business servers.
How long does it take to set up?
Installing the role and enabling a volume takes minutes. The first optimisation pass on a large volume can run for hours or overnight, and it does not need supervising. The part worth planning is the job schedule, so optimisation does not compete with the nightly backup. After that it runs on its own.
Which deduplication usage type should we choose on Windows Server?
Default suits a general purpose file server, Hyper-V suits virtual desktop hosts, and Backup suits a virtualised backup application. The usage type sets the defaults behind the scenes, including the minimum age a file must reach before it is optimised. That is three days on Default and zero on Backup.
Does Exodesk configure data deduplication for New Zealand businesses?
Yes. Exodesk works with businesses across Canterbury, Otago and Southland from offices in Christchurch and Dunedin, and has supported South Island businesses since 1989. That covers measuring the likely saving, configuring the volumes and testing the restore afterwards.
How many of your terabytes are the same file?
The measurement is free, it runs in an afternoon, and it either produces a number worth acting on or tells you to spend the money elsewhere. An IT assessment covers your file servers and backup repository alongside the rest of your setup.
Or read more about our managed IT services.
