Posts Tagged ‘job engine’

No fooling: A new issue of the EMC Isilon Uptime Bulletin

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein
Download the latest issue from EMC Online Support site:

Download the latest issue from the EMC Online Support site:

A new issue of the EMC® Isilon® Uptime Bulletin is now available on the EMC Online Support site (requires login).

Each quarter, the EMC Isilon Uptime Bulletin describes some of the best practices to help you achieve the highest level of operational reliability for your Isilon environment. The latest issue of the Uptime Bulletin covers the following topics:

  • Benefits of the Isilon OneFS Job Engine
    The Job Engine is the OneFS work distribution system. Learn how the Job Engine can make routine cluster maintenance easy and efficient.
  • Best practices for working with snapshots in OneFS
    Snapshots are used to protect data against accidental deletion. Learn best practices that can help improve snapshot management in OneFS.
  • Recommended releases of OneFS
    Get links to the release notes for the latest recommended release versions of OneFS 7.0, 6.5, and 6.0.
  • The latest firmware package releases
    Review the following firmware package release information:

    • Supported drives in drive firmware package 1.6
    • Supported storage platforms in node firmware package 9.0.1

The EMC Isilon Uptime Bulletin is published at the end of each quarter. If you have any suggestions or requests for the next issue, email

7 best practices for using the EMC Isilon OneFS Job Engine

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein
The OneFS Job Engine primary functions.

OneFS Job Engine primary functions

In the EMC Isilon OneFS® operating system, the Job Engine manages the scheduling, reporting, and the initiation of cluster protection and maintenance tasks. A significant benefit of the Job Engine is that it can adaptively manage maintenance jobs based on the amount of cluster resources available.

To ensure that your cluster is performing at optimal levels, follow these Job Engine best practices:

  • Schedule jobs to run during your cluster’s low usage hours.
  • When possible, use the default priority, impact, and scheduling settings for each job.
  • Ensure that the cluster, including individual node pools, is less than 90 percent full.
  • If the EMC Isilon SmartPools™ software module for resource management is licensed, ensure that default spillover setting is enabled.
  • Set up notification rules to alert you when the cluster begins to reach capacity thresholds and enable virtual hot spares (VHS) to reserve free space that’s needed in case you need to rebuild data if a disk failure occurs.
  • Do not disable the Snapshot Delete job.
  • Run the FSAnalyze job only if you are using InsightIQ. FSAnalyze creates data for the InsightIQ file system analytics tools and provide details about data properties and space usage with the OneFS file system.

Additional best practices can be found in the white paper, “EMC Isilon OneFS Job Engine.”

In the latest release of OneFS, several enhancements were made to the Job Engine. These enhancements were highlighted in our video, “Key New Features in EMC Isilon OneFS 7.1.” For more information about the enhanced Job Engine, read the white paper, “EMC Isilon OneFS Job Engine.”

If you have questions or feedback about this blog, email To provide documentation feedback or request new content, email

A closer look at EMC Isilon SmartDedupe and the Isilon OneFS 7.1 Job Engine

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

We recently learned that our blog readers are most interested in the new EMC Isilon SmartDedupe software offering and the Job Engine enhancements incorporated into in the recently released EMC Isilon OneFS 7.1.

In this video, we take a closer look at these features. First, we cover basic concepts about SmartDedupe and Job Engine performance enhancements in OneFS 7.1. Next, we provide brief demonstrations of how to use these features in the OneFS web administration interface. This video also highlights details about data at rest encryption in OneFS 7.1.

Download the EMC Isilon OneFS 7.1 release notes for more information. You can also review the video transcript below.

Video Transcript

Hello, I’m André Morrissen, a Senior Technical Writer with the Information Development team.

Version 7.1 of OneFS contains numerous enhancements that will improve the performance of your Isilon cluster.

In this video, we’ll take a look at a SmartDedupe, job engine improvements, and data at rest encryption and find out how they can improve your workflow.

SmartDedupe is a new licensed feature of OneFS which enables you to save storage space on your cluster by reducing redundant data—in other words, by deduplicating that data.

SmartDedupe is most beneficial for workflows that incorporate large amounts of duplicate data, such as archiving or when a large amount of virtual machines are stored on a cluster.

As you write files to the cluster, some of those files or blocks of data in the files might be duplicates. You can run a deduplication job that scans the file system to see if that data already exists. If it does, OneFS moves that data to a hidden file called a shadow store and replaces the duplicate data in the files with a pointer to the shadow store.

Deduplication is applied at the subdirectory level and targets all files and directories underneath one or more root directories.

The deduplication job is set to run at low priority by default, so impact to your workflow should be minimal. However, it’s a good idea to wait until users have finished modifying their files on the cluster before you run the job.

You can perform the following deduplication tasks from the OneFS web administration interface.

Assign specific subdirectories for deduplication.

Run an assessment job to determine how much space you might save in a given directory.

And view detailed reports of deduplication jobs.

OneFS 7.1 includes major improvements to the job engine, the system that helps you schedule and manage maintenance jobs on your cluster.

As with previous versions of OneFS, the job engine can adjust jobs based on the amount of cluster resources available. For example, if clients require more system resources, threads allocated to the job engine are decreased.

However, now you can run up to three jobs simultaneously, with a few exceptions that keep similar types of jobs from colliding. For example, you can run an AutoBalance, IntegrityScan, and DedupeAssessment job all at the same time.

OneFS 7.1 also introduces support for Data at Rest Encryption.  With this feature, you’ll be able to create a cluster of nodes that contain self-encrypting drives or SEDs. Data at Rest Encryption provides data security that meets specific regulatory requirements for financial and governmental workflows.

Isilon’s use of hardware-based encryption provides the following benefits:

Less consumption of system resources.

Removed drives remain encrypted, which prevents data theft.

The data encryption is performed at the drive level using special processors on each SED that provide 256-bit AES encryption protection. The encryption has less than 1% impact on the performance of the drives themselves.

If you’re interested in creating a cluster with SEDs and using Data at Rest Encryption, contact your account representative.

For more information about the features in this videos, see the OneFS Web Administration.

For a full list of new features, see the OneFS 7.1 Release Notes.

Thanks for watching.

How to start a job in the EMC Isilon OneFS Job Engine

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

Optimizing free space on your Isilon cluster.

Scanning the file system to ensure that data is protected.

Removing duplicate data.

These are examples of cluster maintenance tasks that are managed by the Job Engine, a work distribution service in the EMC Isilon OneFS operating system.

The Job Engine plays an important role in maintaining the reliability and integrity of your cluster, and is well documented on the EMC Online Support site. It’s also a popular feature with customers, as indicated by our recent poll results.

This post provides a brief overview of the Job Engine and a video demonstration of how to start a job.

Job Engine basics

The Job Engine prioritizes, schedules, and balances the amount of resources dedicated to a specific set of tasks, or jobs, that run in the background. Specifically, the Job Engine distributes job segments—broken into phases and tasks—across the nodes in your Isilon cluster. For a list of cluster maintenance jobs that are managed by the Job Engine, see the OneFS administration guides or the knowledgebase article titled OneFS 5.0 – 7.0: Complete list of jobs by OneFS version. (Log in to the EMC Online Support site to view this article and OneFS administration guides.)

You can review the schedule of jobs and monitor the progress of running jobs in the OneFS web administration interface or command-line interface. Because some jobs can take many hours to complete or consume a significant amount of resources while running, you can also manually start, pause, or stop jobs. If a job is paused, you can restart the job at any time from the point at which it was interrupted.

In OneFS versions 6.0.x through 7.0.x, only one job can run at a time in the background. In OneFS 7.1, up to three jobs can run at the same time (with a few exceptions to keep similar types of jobs from interfering with each other). Each job has a default impact policy and priority to determine when it will run. You can also modify these default settings when you start a job.

How to start a job

If you want to start a job before it is scheduled to run automatically, you can start it manually. This basic procedure for OneFS 6.0.x through 7.0.x is described in the following video.

To start a job in OneFS 7.1, go to Cluster Management > Job Operations > Job Types click More (circled in red below), and then click Start Job.

How to start a job in Isilon OneFS 7.1

How to start a job in OneFS 7.1.

Additional details about the Job Engine in OneFS 7.1 can be found in the recently updated white paper titled EMC Isilon OneFS: A Technical Overview.

Are you looking for more specific information about job management in OneFS? Let us know. Send an email to, or post a comment on this blog.