| Data deduplication is a storage technique that keeps one copy of each repeated block of data and replaces the rest with pointers back to it. How much it saves depends on the data, the storage platform and any optimisation already in place. This guide covers Windows Server file storage and compatible backup repositories. |
A 40-person firm in Christchurch gets a bill for its offsite backup. The line says 9 TB, and the finance manager reads it twice, because the business has 40 staff and nobody can picture nine thousand gigabytes of anything.
A good deal of that is the same data over and over. The same email attachment sitting inside forty mailboxes. Fourteen nightly copies of a virtual machine that changes by a few hundred megabytes a day. A project folder duplicated when somebody left and never reconciled against the original.
The figures used through this post are illustrative. Take that firm at 9 TB of backup storage. When the volume was measured, 55 per cent of it turned out to be repeated blocks, which is the sort of number a backup target tends to produce.
Whether that reduces the bill is a separate question, and it is the one this post spends the most time on. A smaller volume and a smaller invoice are not automatically the same thing.
This post is about the volume of what gets stored and what that does to the cost of holding it. How often to back up, how many copies to hold and where to hold them belong to our guide to a backup strategy. What follows deals with the layer underneath, meaning how much space that plan consumes and why.
Why Does a Storage Bill Keep Growing When the Business Has Not?
Because stored volume grows with activity, not with headcount. A firm of 40 people can hold ten times the data it held five years ago without hiring anybody, and four ordinary habits do most of it.
- Attachments that live in every mailbox they were sent to, and again in every backup of every mailbox
- Nightly copies of the same virtual machine, where each night holds a full picture of a disk that barely changed
- Folders duplicated during a migration, a departure or a restore that nobody deleted afterwards
- Working files kept in a personal drive as well as the shared one, because somebody wanted a version they controlled
None of that is carelessness. Storage has been cheap enough to add for twenty years, and no job description has ever included subtracting it.
The bill records the result without explaining it. A line reading 9 TB tells an owner nothing about how many of those terabytes are the same October board pack, saved by several people into four folders and then captured by fourteen consecutive backups.
Would deleting more solve it?
Partly, and the two work on different things. Deleting is a decision about what the business still needs, and that ground belongs to a retention policy. business.govt.nz gives the same instruction in its guidance for New Zealand businesses: do not store data longer than you need to.
Deduplication works on what you have decided to keep. It removes repetition without anybody choosing what to lose, so it is the safer of the two to start with.
What Is Data Deduplication?
Data deduplication is a process that finds repeated blocks of data across a storage volume, keeps one copy of each, and replaces the rest with a pointer. The industry shortens it to dedup, and the Windows Server documentation uses that abbreviation too.
The mechanism matters, because it explains every limit further down. Microsoft’s documentation for Windows Server describes a post-processing model: data is written to disk unoptimised, then a scheduled job breaks files into variable-size chunks, identifies the unique ones, and moves them into a chunk store.
What is left in place of the original file is a reparse point, a small tag telling the file system to fetch the content from the chunk store instead. Staff opening the file see no difference at all.
The technique itself is general, and it also lives inside backup products and storage platforms with their own behaviour and controls. Everything below about configuration refers to Windows Server.
It also leaves your filing exactly as it was. Staff still see the same files, folders and versions, so deduplication does not tidy duplicate filenames, choose the right version, or decide what should have been deleted years ago.
Is deduplication the same thing as compression?
They are different techniques. Compression encodes data more compactly, and Windows deduplication reuses repeated chunks both within a file and between files across a whole volume. The documentation draws the same line: NTFS compression optimises a file at write time, where deduplication finds patterns shared between files that compression cannot see.
The two stack. Windows can compress the chunks after deduplicating them, so a measured saving often comes out ahead of what either gives on its own, which is why it is worth knowing whether a reported figure includes both.
How Much Space Does Deduplication Save?
It depends entirely on how much repetition the data holds, which is why any vendor quoting a fixed percentage before looking at your volume is guessing. The pattern across workloads is consistent enough to know where to look first.
Already-compressed formats usually offer less. Photographs, video and zipped archives carry little repetition inside themselves, though identical copies of the same JPEG or archive still share their data exactly, so a volume of media is worth measuring rather than writing off.
| Where the data sits | What to check first |
|---|---|
| Usually a good opportunity | |
| Backup repositoryOften the largest, because each set repeats most of the one before | Whether the backup product already compresses, deduplicates or encrypts, in which case a second layer may add little |
| Virtual machine libraryOften large, since machines built from one image share nearly every block | Whether the host is running VDI or general purpose Hyper-V, which Microsoft treats differently |
| Shared file serverUsually worthwhile, because teams keep their own copies of the same documents | Little, as this is one of the workloads Microsoft recommends |
| Mixed or limited: measure first | |
| Mailbox archiveMixed, since attachments repeat across recipients and message bodies do not | Whether the mail is on your server at all, because a server setting does nothing to hosted mailbox storage |
| Photo, video and design filesUsually limited, as the formats are already compressed | Identical copies still share their data exactly, so measure rather than assume. Design files span many formats |
| Encrypted dataDepends where the encryption happens | Data encrypted before deduplication examines it will not deduplicate. Volume encryption such as BitLocker is a separate layer and can coexist with it |
What do firms find when they measure?
More than they expected on the backup target and less than they hoped on the file share. The backup number is high because the repetition is structural, and the file share number depends on how the team works.
A firm that keeps one master copy of everything will see a modest saving. A firm where four people each hold their own version of the price list will see a large one, and will also learn something about how it works.
That second finding is worth as much as the storage. The measurement names the folders where duplication collects, and that list is the fastest way into a conversation about how the team files things.
What does a deduplication ratio mean?
A deduplication ratio compares the size of the data before optimisation against the space it occupies after. A 2:1 ratio means 2 TB of logical data sits in roughly 1 TB of physical space, a 50 per cent reduction on that basis. Ask whether the figure includes compression as well, because Windows reports the two separately and a combined number flatters both.
A 50 per cent reduction in stored space is not a 50 per cent reduction in a bill. What the invoice measures decides that, and it is covered below.
Can You Measure the Saving Before Buying Anything?
Yes, with one prerequisite people miss. Microsoft ships a savings evaluation tool called DDPEval, which appears at C:\Windows\System32\DDPEval.exe only after the deduplication feature is installed on the server. It is not sitting there waiting on a server that has never had the feature added. Installing the feature does not switch deduplication on for any volume, so this step changes nothing on its own.
It runs against local drives, cluster shared volumes and network shares. The output is eight lines of text: processed size, optimised size, space saved and the percentage, with the compression saving reported separately so a vendor cannot claim credit for both.
Run it on the backup repository first, because that volume holds the most repetition. Nothing else has to be gathered before somebody can decide.
What should you do with the result?
Convert the percentage into terabytes first, then find out whether those terabytes are what you are billed for. The second step is the one people skip, and it is where the business case is won or lost.
Keep in mind what DDPEval is telling you. It estimates the storage saving on eligible data. It is not evidence that your applications support deduplication, that restores will be faster, or that the invoice will move.
Will a smaller volume reduce the bill?
Not always, and it turns on what the provider charges for. Find out whether your invoice measures logical source data, physical stored data, provisioned capacity or a fixed package, because a measured reduction only shows up if it moves the thing being billed.
On fixed-capacity storage the benefit is usually headroom and a later upgrade rather than a smaller invoice. On usage-based storage it can reduce the charge, where the provider bills for space actually consumed. Ask before assuming, and ask whether the destination already optimises what arrives.
Which Systems Should Deduplication Run On?
Microsoft names three recommended workloads: general purpose file servers holding team shares and home folders, virtual desktop infrastructure servers, and virtualised backup applications. All three combine heavy repetition with the idle time the optimisation job needs, and Microsoft does not ask for a pilot on them.
That is not the same as needing no checks. Confirm the applications on the volume are supported, confirm your backup product handles optimised volumes, and confirm the server has resources to spare. Then test a restore, which is a separate requirement covered further down.
Three more need evaluating first. General purpose Hyper-V hosts, SQL servers and line-of-business servers can benefit, and they can also end up slower, because deduplication adds latency to random reads and a database reads randomly by nature.
The rule underneath both lists is spare capacity. A file server with nothing running after seven at night is an easy candidate. A system busy around the clock is harder, because the optimisation job competes with the work people are doing, though busy is not the same as having nothing spare. Size it against the other workloads rather than ruling it out on the absence of a quiet period.
Does it need a lot of memory?
Less than people expect, though the figures describe what the deduplication jobs need rather than the whole server. Microsoft’s stated minimum is 300 MB plus 50 MB for every terabyte of logical data, so a 10 TB volume needs about 800 MB available to it, with roughly 1 GB per terabyte recommended for the jobs to run at speed. Size that alongside everything else the server is doing.
Where a virtual environment is involved the sizing question belongs with the host, and our post on server virtualisation covers that.
Is there a licence cost?
No separate licence on Windows Server. Deduplication is a feature included in the Windows Server licence a business already holds, so the cost is the work to enable it and keep an eye on it. That assumes a supported and suitably licensed version, and the environment still needs the capacity and resources to do the work.
Does Deduplication Make a Restore Slower or Faster?
It can help some paths and slow others, and the answer depends on your backup method, storage and network rather than on deduplication alone. A smaller stored volume can mean less data to move, while restoring an ordinary file can require reconstructing it from chunks shared with other files.
The slow case is a single-file restore from a heavily fragmented chunk store on spinning disks, where the system reads from scattered locations. Solid-state storage reduces that penalty because access time is the same wherever the data sits, though it does not remove every bottleneck in the chain.
Test both paths on your own system, a full recovery and an individual file, and keep the timings. That is the only answer that applies to your environment.
One thing deduplication does not do is help anybody find the right file. It preserves your files and versions without improving a backup catalogue, an index or a badly defined search. Microsoft is explicit that Windows Search does not support deduplication: it cannot index the reparse points deduplication leaves behind, so deduplicated files are skipped and results on those volumes may be incomplete. Check that before enabling it where staff rely on Windows Search.
What has to be tested before it is trusted?
A restore. Deduplication changes how data is stored, so the only proof that it has not broken anything is pulling files back and opening them, which business.govt.nz names as a step in its own backup guidance.
Test a full-volume restore and a single-file restore, and time both. Write both down. They are the figures you compare against next year, and the ones that outlast a change of provider.
Do the test before the volume is optimised as well, if there is time. Two sets of timings turn an argument about whether deduplication helped into a table.
Can the chunk store itself be damaged?
It can, and Windows runs a weekly job against exactly that risk. An integrity scrubbing job looks for corruption from disk faults, and the system keeps spare copies of chunks referenced more than a hundred times so that popular data can be rebuilt.
Dedup is not a substitute for having more than one copy of the data, and our post on BCDR and backup draws that line.
What Does Deduplication Not Fix?
It does not give the business a second copy of anything, and this is the misunderstanding worth clearing up before anybody signs off on it. Deduplication makes one copy smaller, and a smaller single copy is still a single copy.
It also does not decide what should be kept. A volume full of files nobody needs will deduplicate neatly and still be full of files nobody needs, which is why measuring usually comes after somebody has read the retention schedule.
Does it protect against ransomware?
No. Ransomware encrypts data and encrypted data does not deduplicate, so an infected volume may show its saving falling.
That is not an early warning you can rely on. Ratios move for ordinary reasons, and because optimisation runs after data is written, the change tends to appear only once the damage is done. Treat it as an observation rather than a control. The measures that do help sit in cyber security and in a backup you have tested.
Does it replace archiving old data?
No. Archiving moves data somewhere cheaper and slower, and deduplication shrinks it wherever it happens to be held. A firm holding fifteen years of project files benefits from both, in that order.
How Much Does Data Deduplication Cost?
Three numbers decide it, and they have to stay separate. The gross storage saving, which is what the measured reduction is worth only if it changes what you are billed. The ongoing cost of monitoring the deduplication jobs as part of managed support. And the one-off cost of enabling it, which covers installing the feature, choosing the usage type, scheduling the jobs around the backup window and running a tested restore.
For the illustrative firm, a 55 per cent measured saving takes 9 TB to about 4 TB. Subtract the ongoing cost from the gross storage saving to get the recurring benefit, then set the one-off cost against that recurring benefit to get a payback period. The order matters: comparing a gross saving against a setup cost makes almost any project look like it pays for itself immediately.
What moves the numbers is how much the data actually repeats, whether your storage is billed on space consumed or capacity provisioned, how many volumes are involved, and whether the backup product or the destination already optimises the data before you add another layer.
One frequently claimed benefit needs qualifying. A smaller stored volume does not automatically mean less sent offsite each night, because incremental transfer and backup-side optimisation may already avoid resending unchanged data. Some copy paths reconstruct the logical data and others preserve the optimisation, and Microsoft notes that a full garbage collection can change the size of an incremental backup. Check what your own chain actually does.
When is it not worth doing?
When the data does not repeat, when the volume is small enough that the saving would not change anything, when the storage is billed in a way the saving does not touch, or when the application or the backup product does not support it.
DDPEval answers the first of those before anything is changed. The rest need a look at the invoice and a compatibility check.
How Do You Enable Data Deduplication on Windows Server?
Windows Server data deduplication is a feature rather than a product, so there is nothing to buy beyond the licence you already hold. Measure first, then work through five steps.
Install the feature, then measure
Add the deduplication feature to the server, which is what puts DDPEval on it, then run the tool against the backup repository and the main file share and write down the reported saving for each. Installing the feature does not enable deduplication on anything.
Work out whether the saving reaches the invoice
Convert the percentage into terabytes, then check what your storage provider actually bills for. On provisioned capacity the benefit is often headroom and a deferred upgrade rather than a smaller charge, and a saving that never reaches the invoice is not a saving.
Enable it on the volumes that scored well
Enable-DedupVolume switches on a volume you choose, with the usage type that matches the workload. Confirm first that the applications on that volume and your backup product both support optimised volumes.
Schedule the jobs around the backup
Move optimisation, garbage collection and integrity scrubbing to a window when the server is quiet and clear of the nightly backup, so the two are not competing for the same disks.
Restore a full volume and a single file
Time both and keep the two numbers for comparison next year. That test is the only thing that proves nothing has broken, and it is worth doing before optimisation as well so you have a baseline.
Install-WindowsFeature adds the feature and Enable-DedupVolume enables a volume, and Microsoft documents the commands in full. A supported and suitably licensed Windows Server already includes the feature, though the environment still needs the capacity and resources to run the jobs.
None of it is unattended afterwards. Check for failed jobs, keep an eye on free space, and watch performance on the volume for the first few weeks.
Which usage type should you choose?
Default for a general purpose file server, Hyper-V for virtual desktop hosts, and Backup for a virtualised backup application. The usage type sets the defaults, including the minimum age a file must reach before it becomes eligible for optimisation, which is three days on Default and zero on Backup.
Zero does not mean immediate. Windows deduplication is post-processing, so a zero minimum age makes new files eligible for the next optimisation job rather than optimising them as they arrive. Picking the wrong usage type is not fatal, and it is worth getting right.
Find Out How Much of Your Storage Bill Is Repeated Data
Exodesk has operated since 1989, with offices in Christchurch and Dunedin serving businesses across New Zealand. Our cloud and storage team measures what data deduplication would save on your file servers and backup repository, works out whether that saving would reach your invoice, configures it around your backup window and tests the restore afterwards.
Contact us today to discuss how we can help your business or connect with us on LinkedIn to stay updated with more insights.
Frequently Asked Questions
What is data deduplication in simple terms?
Deduplication stores one instance of anything repeated and leaves a pointer everywhere else it used to sit. A volume then holds more information than the room it takes up, and staff opening a file notice no difference, because the content is fetched behind the scenes.
How much storage does deduplication save?
It depends on how much the data repeats, so the honest answer comes from measuring, and any fixed percentage quoted in advance is a guess. Backup repositories and virtual machine libraries usually save the most, because each copy repeats the last. Volumes of photographs or video normally save least, since the formats are already compressed, though identical copies of the same file still share their data exactly. A saving in stored space is also not automatically a saving on the invoice.
Is dedup the same as compression?
They are different techniques. Compression encodes data more compactly, and Windows deduplication reuses repeated chunks both within a file and between files across a whole volume. They work together, and Windows can compress chunks after deduplicating them, so ask whether a reported figure includes both.
Does deduplication cost extra on Windows Server?
No separate licence. It is a feature included in the Windows Server licence a business already holds, so the cost is the work to enable it and keep an eye on it. That assumes a supported and suitably licensed version, and the server still needs the capacity and resources to run the jobs.
Will deduplication slow our systems down?
It can, on the wrong workload. The optimisation job runs after data is written and needs idle time, so file servers and backup targets suit it and database servers running flat out often do not. Random reads from spinning disks are where the latency shows.
What is DDPEval and how do we use it?
DDPEval is an evaluation tool Microsoft ships with the deduplication feature, and it reports what a folder or volume would save before anything is changed. It only appears on the server once that feature is installed, so it is not already waiting on a server that has never had it added. Run it from a command prompt against the backup repository first, because that volume usually holds the most repetition. It estimates storage savings, not application compatibility or restore performance.
Is deduplicated data still safe to restore?
Yes, provided the restore has been tested. Deduplication changes the way data is stored while leaving the data itself intact, and Windows runs a weekly integrity job that keeps spare copies of frequently referenced chunks. Test a full restore and a single-file restore before relying on it.
Does deduplication replace having backups?
No. It reduces the size of what you store, and it does not give you a second copy in a second place. Those are two different jobs and a business needs both. Deduplication makes the copy you have cheaper to keep, and a backup strategy decides how many copies exist and where they are held.
Which workloads are unsuitable or unsupported?
Three different reasons get mixed together here, so separate them. Poor savings: volumes holding mostly compressed data, or data encrypted before deduplication examines it. Performance: servers with little spare capacity for the optimisation job, and workloads dominated by random reads. Support: Microsoft lists SQL and line-of-business servers as needing evaluation first, and your backup product and applications have to support optimised volumes. A workload can be a poor saving and still perfectly supported, or the reverse.
How long does it take to set up?
Installing the feature and enabling a volume takes minutes. The first optimisation pass on a large volume can run for hours or overnight and does not need supervising. The part worth planning is the job schedule, so optimisation, garbage collection and integrity scrubbing do not compete with the nightly backup. After that it needs occasional attention rather than none: check for failed jobs, free space and volume performance.
Which deduplication usage type should we choose on Windows Server?
Default suits a general purpose file server, Hyper-V suits virtual desktop hosts, and Backup suits a virtualised backup application. The usage type sets the defaults behind the scenes, including the minimum age a file must reach before it becomes eligible for optimisation, which is three days on Default and zero on Backup. Zero makes new files eligible for the next optimisation job rather than optimising them on arrival, because Windows deduplication runs after data is written.
Will deduplication reduce our offsite backup bill?
Only if it changes what the provider bills for. Find out whether your invoice measures logical source data, physical stored data, provisioned capacity or a fixed package. On usage-based storage a real reduction can lower the charge; on provisioned capacity the benefit is usually headroom and a later upgrade instead. Also check whether the backup product or the destination already deduplicates or compresses, because a second layer may add little.
Does Exodesk configure data deduplication for New Zealand businesses?
Yes. Exodesk has operated since 1989, with offices in Christchurch and Dunedin serving businesses across New Zealand. That covers measuring the likely saving, checking whether it would reach your invoice, configuring the volumes and testing the restore afterwards.
How many of your terabytes are the same file?
The measurement tells you how much of your stored data is repeated, and whether reducing it would change what you pay. An IT assessment covers your file servers and backup repository alongside the rest of your setup.
Or read more about our managed IT services.
