Posts Tagged ‘SmartDedupe’

EMC Isilon InsightIQ 3.1 is now available!

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

The latest release of EMC® Isilon® InsightIQ includes new and enhanced reports that help you become a rock star at managing space on your cluster.

New file system reports

The following new reports are available to help you manage cluster capacity, deduplication, and quotas in OneFS. For useful tips about these reports, refer to the InsightIQ 3.1 User Guide.

Usable capacity reporting
Do you often wonder how much free space is on your cluster when accounting for the space that is being used to protect your data? The usable capacity report is an excellent resource that helps you prevent your cluster from reaching capacity. The report anticipates how much protection overhead you might need in addition to capacity that is already reserved for snapshots and virtual hot spares. Essentially, the report breaks down an estimate of how much capacity can be used for storing data and how much capacity can be reserved for protecting your data. Keep in mind that this report only provides estimates.

Usable capacity report in InsightIQ 3.1

Deduplication reporting

Running a deduplication job in OneFS by using SmartDedupe® software module creates free space on your cluster. In OneFS, you can assess the amount of disk space you’ll save before you start a deduplication job. You can also do this in InsightIQ 3.1. However, InsightIQ also lets you view historical and current information about how much space is saved by deduplication over a specific range of time.

Two sections from the deduplication report in InsightIQ 3.1. Historical deduplication job information is not cumulative.

Two sections from the deduplication report in InsightIQ 3.1. Deduplication job information is not cumulative.

Quota reporting
The quota report enables you to simplify quota management in OneFS. This report displays information about quotas created through the SmartQuotas® software module. You can view quotas that are assigned to specific directories, the limits defined by those quotas, and the amount of data stored in the directories that those quotas are applied to. This information can help you compare the data usage of a directory to the quota limits over time, and predict when a directory is likely to reach its quota limit.

A section from the quota report in InsightIQ 3.1. Historical data is generated by quota reports in OneFS.

Two sections from the quota report in InsightIQ 3.1. Click on a directory to view quota limit usage over time. Historical data is generated by quota reports in OneFS.

Enhanced reporting

InsightIQ 3.1 includes enhancements to cache reporting and exporting capabilities on file system analytics reports. For example, you can now view information about L3 cache usage in performance reports and download the data from file system analytic reports to CSV files.

Upgrading or installing InsightIQ 3.1

If you want to install this release, review the InsightIQ 3.1 Installation Guide for requirements and procedures. If you want to upgrade to this release, first explore your upgrade options covered in the Isilon Supportability and Compatibility Guide, and then perform the procedure provided in the InsightIQ 3.1 Installation Guide.

For more information about all the features, fixes, and changes in functionality in this release, refer to the InsightIQ 3.1 Release Notes. For information about using InsightIQ to monitor your cluster, refer to the InsightIQ 3.1 User Guide.

Start a conversation about Isilon content

Have a question or feedback about Isilon content? Visit the online EMC Isilon Community to start a discussion. If you have questions or feedback about this blog, contact us at To provide documentation feedback or request new content, contact


Get your hands on EMC Isilon OneFS 7.1 at EMC World 2014

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein


EMC World 2014 is around the corner. If you plan to be in Las Vegas, Nevada on May 5-8 for this event, you have the opportunity to try out the EMC® Isilon® OneFS operating system in person.

There will be three labs hosted by EMC Isilon that are available throughout the conference, where you can test drive new features and functionality in OneFS using real data.

  • Isilon Cluster Setup, Configuration, and Management (HOL 29)
    An introductory lab that demonstrates how to create a storage cluster, join the cluster to an Active Directory domain, navigate the OneFS web administration interface, and create and manage directories or shares.
  • Isilon OneFS 7.1 Enhancements (HOL 30)
    An intermediate lab that explores the enterprise-ready enhancements built into OneFS 7.1.
  • Deploying Hadoop with EMC Isilon and VMware (HOL 28)
    An advanced lab that walks you through the process of deploying and using your first Hadoop cluster. Learn how to use VMware Big Data Extensions to deploy a small Hadoop cluster with an EMC Isilon NAS storage cluster.

Anyone can sign up for the labs and attend at any time. All labs are self-paced and Isilon representatives will be available to answer any questions you might have. For lab hours and information about how to register, visit the EMC World vPass website.

Take a test drive with OneFS 7.1

This blog has covered several of the enhancements and features included in OneFS 7.1. If you’re curious about OneFS 7.1 and want to take it for a test drive, visit the OneFS 7.1 Enhancements (HOL 30) lab. Here’s a closer look at the following features will be covered in this lab session:

  • Role based access control
  • EMC Isilon SmartDedupe™
  • EMC Isilon SyncIQ™
  • Audting

Role Based Access Control

Role based access control (RBAC) in OneFS 7.1 enables you to control configuration-level access of your Isilon cluster through roles and privileges. OneFS 7.1 comes with built-in administrator roles: SecurityAdmin, SystemAdmin, AuditAdmin, and VMwareAdmin. You can also create custom roles with assigned privileges and add users and groups to those roles.

In this lab, you will learn how to:

  • View built-in roles
  • Create a custom role
  • Add privileges to a role
  • Add a user to a role

If you are unable to attend EMC World, but would like an RBAC demonstration, watch the following video, “Technical Demo: Role Based Access Control.”

EMC Isilon SmartDedupe™

When you want to save space on your EMC Isilon cluster, use EMC Isilon SmartDedupe™ to remove, or deduplicate, redundant data. SmartDedupe deduplicates data by scanning an Isilon cluster for identical data blocks. When it finds redundant data blocks, it moves one data block to a shadow store. It then deletes the duplicate block from the original file and replaces it with a pointer to the shadow store. For more information, watch the video, “Enterprise Features of EMC Isilon OneFS 7.1: SmartDedupe.”

dedupe assessment report

Figure 1: A DedupeAssessment report. Space that can be recovered after deduplication is circled in red.

The deduplication process is performed through jobs that are managed in the same way you manage other cluster maintenance jobs. It is recommended that you run deduplication jobs when clients are not modifying data on the cluster. This maximizes the amount of space you can save. It is also recommended that you run a deduplication job every ten days.

To begin the deduplication process, first determine how much space you can save on specified directories by running a DedupeAssessment job and viewing a DedupeAssessment report (Figure 1). You can then run a Dedupe job on those directories to then remove redundant data and place it in the shadow store.

In the OneFS 7.1 Enhancements (HOL 30) lab, you will learn how to:

  1. Start a DedupeAssessment job
  2. View active jobs
  3. View the deduplication assessment report
  4. Activate the SmartDedupe license
  5. Start a Dedupe job
  6. View the deduplication report

EMC Isilon SyncIQ™

For data protection and disaster recovery, EMC Isilon SyncIQ™ replicates data from one Isilon cluster to another. In the event of disaster scenario where your original cluster goes down, you can retrieve replicated data stored on your backup cluster.


Figure 2: A new option (circled in red) for SyncIQ policies, which is available in OneFS 7.1

To replicate data using SyncIQ, first create a SyncIQ policy in OneFS. The policy specifies the source directory and backup/target cluster, and when to run the replication job. In OneFS 7.1, there is a new policy option available that enables OneFS to replicate data whenever the source directory is modified (Figure 2). This enhancement ensures that data is replicated as soon as a change occurs, independent of the replication job schedule.

In the OneFS 7.1 Enhancements (HOL 30) lab, you will learn how to:

  1. Activate a SyncIQ license
  2. Configure a SyncIQ policy
  3. Verify that the SyncIQ policy successfully synchronized between a source and target cluster


OneFS 7.1 can audit system configuration and SMB protocol access events on your Isilon cluster. To start collecting auditing information, simply enable configuration change auditing or SMB protocol access auditing in either the OneFS web administration interface or the OneFS command-line interface (Figure 3). System configuration changes and changes performed on files and folders through the SMB protocol are recorded in an auditing log. Protocol auditing logs can be exported to Varonis DatAdvantage® or other third-party vendors that support the EMC Common Event Enabler (CEE) framework. For more information, watch the video, “Enterprise Features of EMC Isilon OneFS 7.1: Auditing”.

Figure 3: How to enable auditing (circled in red) in OneFS 7.1 web administration interface.

Figure 3: How to enable auditing (circled in red) in OneFS 7.1 web administration interface.

In the OneFS 7.1 Enhancements (HOL 30) lab, you will learn how to:

  1. Enable auditing
  2. Make an access zone into an audited zone
  3. Add an audit event, which will modify the audited zone to audit different events
  4. Generate an event
  5. View and locate audit logs
  6. View event forwarding
  7. View the AuditAdmin role
  8. Open DatAdvantage and view user statistics and event details

For more information

For more details about these features, refer to OneFS 7.1 release notes, OneFS 7.1 Web Administration Guide, and the OneFS 7.1 CLI Administration Guide.

For more information about Isilon sessions and labs at EMC World, visit the EMC World 2014 vPass website to browse the EMC World Session Catalog for more information.

A closer look at EMC Isilon SmartDedupe and the Isilon OneFS 7.1 Job Engine

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

We recently learned that our blog readers are most interested in the new EMC Isilon SmartDedupe software offering and the Job Engine enhancements incorporated into in the recently released EMC Isilon OneFS 7.1.

In this video, we take a closer look at these features. First, we cover basic concepts about SmartDedupe and Job Engine performance enhancements in OneFS 7.1. Next, we provide brief demonstrations of how to use these features in the OneFS web administration interface. This video also highlights details about data at rest encryption in OneFS 7.1.

Download the EMC Isilon OneFS 7.1 release notes for more information. You can also review the video transcript below.

Video Transcript

Hello, I’m André Morrissen, a Senior Technical Writer with the Information Development team.

Version 7.1 of OneFS contains numerous enhancements that will improve the performance of your Isilon cluster.

In this video, we’ll take a look at a SmartDedupe, job engine improvements, and data at rest encryption and find out how they can improve your workflow.

SmartDedupe is a new licensed feature of OneFS which enables you to save storage space on your cluster by reducing redundant data—in other words, by deduplicating that data.

SmartDedupe is most beneficial for workflows that incorporate large amounts of duplicate data, such as archiving or when a large amount of virtual machines are stored on a cluster.

As you write files to the cluster, some of those files or blocks of data in the files might be duplicates. You can run a deduplication job that scans the file system to see if that data already exists. If it does, OneFS moves that data to a hidden file called a shadow store and replaces the duplicate data in the files with a pointer to the shadow store.

Deduplication is applied at the subdirectory level and targets all files and directories underneath one or more root directories.

The deduplication job is set to run at low priority by default, so impact to your workflow should be minimal. However, it’s a good idea to wait until users have finished modifying their files on the cluster before you run the job.

You can perform the following deduplication tasks from the OneFS web administration interface.

Assign specific subdirectories for deduplication.

Run an assessment job to determine how much space you might save in a given directory.

And view detailed reports of deduplication jobs.

OneFS 7.1 includes major improvements to the job engine, the system that helps you schedule and manage maintenance jobs on your cluster.

As with previous versions of OneFS, the job engine can adjust jobs based on the amount of cluster resources available. For example, if clients require more system resources, threads allocated to the job engine are decreased.

However, now you can run up to three jobs simultaneously, with a few exceptions that keep similar types of jobs from colliding. For example, you can run an AutoBalance, IntegrityScan, and DedupeAssessment job all at the same time.

OneFS 7.1 also introduces support for Data at Rest Encryption.  With this feature, you’ll be able to create a cluster of nodes that contain self-encrypting drives or SEDs. Data at Rest Encryption provides data security that meets specific regulatory requirements for financial and governmental workflows.

Isilon’s use of hardware-based encryption provides the following benefits:

Less consumption of system resources.

Removed drives remain encrypted, which prevents data theft.

The data encryption is performed at the drive level using special processors on each SED that provide 256-bit AES encryption protection. The encryption has less than 1% impact on the performance of the drives themselves.

If you’re interested in creating a cluster with SEDs and using Data at Rest Encryption, contact your account representative.

For more information about the features in this videos, see the OneFS Web Administration.

For a full list of new features, see the OneFS 7.1 Release Notes.

Thanks for watching.