Archive for the ‘Advanced Topics’ Category

Creating SMB shares with expansion variables in EMC Isilon OneFS

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

To make it easy for users in your organization to connect to a home directory through a Windows client, you can create an SMB share in EMC® Isilon® OneFS®. The share specifies configurable permissions, performance, and security settings for each individual user. Managing SMB shares in OneFS 6.5 through 7.1 can be done manually for each user, or dynamically for a large number of users. To create an SMB share or home directory, you can take advantage of these approaches:

  • Create unique SMB shares for user home directories
    • Dynamically create a unique share for each user home directory
    • Manually create a unique share for each user home directory
  • Create a common SMB share for user home directories
    • Dynamically create user home directories in a common share
    • Manually create user home directories in a common share

Each one of these approaches is highlighted in the new white paper, “Managing SMB shares and user home directories in OneFS 6.5 and later.”

How to dynamically create an SMB share using expansion variables

One of the approaches, as described in “Managing SMB shares and user home directories in OneFS 6.5 and later,” is to dynamically create SMB shares and home directories for new users. Instead of creating per-user SMB shares, you can create a single share that includes expansion variables, such as %U for the user name. For example, when a new user logs in through Active Directory, OneFS automatically creates a unique SMB share and directory for that user.

To dynamically create unique SMB shares using name expansion variables, follow these steps:

In OneFS 7.0 and OneFS 7.1

To take full advantage of expansion variables in SMB shares, you should be running OneFS 7.0.2.9 and later, or OneFS 7.1.0.2 and later.

  1. Log in to the OneFS web administration interface.
  2. Click Protocols > Windows Sharing (SMB) > SMB Shares > Add a Share.
  3. Type a share name (for example, Home) and optional description (for example, User Home Directories).
  4. In the Directory to Be Shared box, type /ifs/home/%U. If you store home directories in another location, specify that location instead.
  5. Click Apply Windows Default ACLs.
  6. Select the Allow Variable Expansion check box.
  7. Select the Auto-Create Directories check box.
  8. Click Create.

Dynamically create SMB share and home directories using expansion variables.

In OneFS 6.5

  1. Log in to the OneFS web administration interface.
  2. Click File Sharing > SMB > Add Share.
  3. Type a share name (for example, Home) and description (for example, User Home Directories).
  4. In the Directory to share box, type /ifs/home/%U. If you store home directories in another location, specify that location instead.
  5. Click Apply Windows default ACLs.
  6. Select the Allow Username Expansion check box.
  7. Selectthe Automatically Create User Directory check box.
  8. Click Submit.

More information about SMB and home directories in OneFS

For more information about expansion variables, see the “Create an SMB share” and “Home directory creation in a mixed environment” sections in the OneFS web administration guides. The administration guide also provides configuration information for accessing home directories through FTP or SSH.

[display_rating_result]

Pick your protocol: Multiprotocol file access in EMC Isilon OneFS

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

You’re rushing to meet a project deadline, and you need to update some related files that are stored on an EMC® Isilon® cluster. You’re working on a Linux computer, and you’re connected to the cluster over a Network File System (NFS) protocol. You need to access files in a directory that your coworker, who uses a Windows computer, created when they were connected to the same cluster over a Server Message Block (SMB) protocol. Thanks to the Isilon OneFS® operating system, you can seamlessly access your coworker’s files even though you are doing so through a very different protocol.

Multiple protocol support is a necessity in today’s IT organizations, which comprise a mix of Windows and UNIX/Linux operating environments. OneFS is designed to provide users with unified access to data on an Isilon cluster using a mix of common protocols, such as SMB, NFS, HTTP, and Hadoop Distributed File System (HDFS). For a full list of supported protocols, see the OneFS administration guides or “EMC Isilon Multiprotocol Data Access with a Unified Security Model”.

So how does OneFS support a multiprotocol environment? What are the steps a system administrator needs to take to set up multiprotocol access in OneFS?

We have two videos that cover the basics and provide recommendations for setting up multiprotocol access in OneFS. The first video, “File Access Basics in an Isilon OneFS Multi-Protocol Environments,” provides a whiteboard overview of this topic. The second video, “Technical Demo: Multi-Protocol File Access Using EMC Isilon OneFS,” provide a demonstration of common multiprotocol commands and tasks.

File access basics and AIMA in OneFS

Supporting a mix of protocols requires supporting a mix of user identities and file permissions. This requirement can leave system administrators with several considerations when configuring OneFS.

Before discussing how OneFS handles multiprotocol file access, let’s first review how two operating environments, Windows and UNIX/Linux, authorize access to files. In a Windows environment, users are identified based on unique security identifiers (SIDs). Files or directories are secured through an Access Control List (ACL). In an UNIX environment, users and groups are identified through user identifiers (UIDs) and group identifiers (GIDs), respectively. Files are secured using POSIX mode bits.

OneFS uses Authentication, Identity Management, and Authorization (AIMA) to assign the right permissions and identifiers to users (and groups) no matter which protocols they use to connect to the cluster. To securely support NFS and SMB clients, OneFS does three things:

  • Connects to directory services, such as Microsoft Active Directory (AD) and Lightweight Directory Access Protocol (LDAP), which provides a security database of user and group accounts along with their information
  • Authenticates users and groups
  • Controls access to directories and files

When a user connects to an Isilon cluster, OneFS scans Active Directory and LDAP for the user’s identifiers. Once the user is authenticated, OneFS creates an access token for the user. OneFS then maps the user’s account (known as “user mapping” in OneFS) in one directory service to another. This single access token is the key to authorizing the user so they can access files that are stored and created on the cluster using different protocols.

For example, if a user, Mike, accesses a file share through SMB, OneFS will scan Active Directory and find an SID for him. If OneFS does not find any UIDs or GIDs associated with Mike via LDAP, OneFS will generate a UID and GID for him and save them to Mike’s access token, so he can access files created by NFS users.

The same type of mapping occurs for file permissions. If a file was created through SMB, it will be assigned an ACL to control who can access the file. OneFS will create equivalent POSIX mode bits for this file. File permissions can be saved to the Isilon cluster on disk in one of three modes: native, UNIX, or SID. For more information about each mode, and about AIMA and user mapping, read the “Identities, Access Tokens, and the Isilon OneFS User Mapping Service” white paper.

This is a brief summary of how multiprotocol file access works in OneFS. Watch the following video, “File Access Basics in an Isilon OneFS Multi-Protocol Environments,” for more information and recommendations for configuring multiprotocol access in OneFS. In this video, Principal Solutions Architect Amol Choukekar answers the following frequently asked questions:

  • What are multiprotocol basics?
  • How do Window and UNIX clients differ when they access files on OneFS?
  • How does OneFS handle user and group identities?
  • How does OneFS store file permissions in a multiprotocol environment?
  • How do clients access files that were created using a different protocol?
  • How does OneFS manage file permissions?
  • What if user names are not similar across authentication providers?

How to configure multiprotocol support in OneFS

You can manage user identity mapping and file permissions using the OneFS command-line interface and OneFS web administration interface. Watch the following video, “Technical Demo: Multi-Protocol File Access Using EMC Isilon OneFS” for demonstrations of the following tasks:

  • Review configured authentication providers
  • Review an access token for a user
  • Review existing identity mappings stored on the cluster
  • Delete existing identity mappings
  • Review ACL policies on the cluster
  • Create a user mapping rule for joining different user names

This video also offers the following demonstrations:

  • File access between Windows and UNIX
  • Creation of a synthetic ACL, which dynamically maps UNIX permissions to Windows rights
  • File permissions management

 

For more information about implementing multiprotocol in OneFS, contact your account representative. If you have feedback about this blog or these videos, send an email to isi-knowledge@emc.com. If you have a request for new documentation, send an email to isicontent@emc.com.

7 best practices for setting up Hadoop on an EMC Isilon cluster

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein

If you’re considering adding an Apache™ Hadoop® workflow to your EMC® Isilon® cluster, you’re probably wondering how to set it up. The new white paper “EMC Isilon Best Practices for Hadoop Data Storage” provides useful information for deploying Hadoop in your Isilon cluster environment.

The white paper also introduces the unique approach that Isilon took to Hadoop deployments. In a typical Hadoop deployment, large unstructured data sets are ingested from storage repositories to a Hadoop cluster based on the Hadoop distributed file system (HDFS). Data is mapped to the Hadoop DataNodes of the cluster and a single NameNode controls the metadata. The MapReduce software framework manages jobs for data analysis. MapReduce and HDFS use the same hardware resources for both data analysis and storage. Analysis results are then stored in HDFS or exported to other infrastructures.

Traditionl Hadoop Deployment

In an EMC Isilon Hadoop deployment, the HDFS is integrated as a protocol into the Isilon distributed OneFS® operating system. This approach gives users direct access through the HDFS to data stored on the Isilon cluster using standard protocols such as SMB, NFS, HTTP, and FTP. MapReduce processing and data storage are separated, allowing you to independently scale compute and data storage resources as needed.

EMC Isilon Hadoop Deployment

Every node in the Isilon cluster acts as the NameNode and DataNode. Compute clients running MapReduce jobs can connect to any node in the cluster. Data analysis results can be accessed by Hadoop users through standard protocols without the need to export results.

To learn more about the benefits of Hadoop on Isilon scale-out network attached storage (NAS), read “Hadoop on EMC Isilon Scale-Out NAS” and “EMC Isilon Scale-Out NAS for In-Place Hadoop Data Analytics.”

Best practices for deploying Hadoop to your Isilon cluster

You can connect Apache Hadoop or an enterprise-friendly Hadoop distribution, such as Pivotal HD or Cloudera, to your Isilon cluster.

First, you’ll need to turn on the HDFS protocol in OneFS. Contact your account representative to complete this step. Next, follow these best practices:

  1. Review the EMC Hadoop Start Kit 2.0. Visit the EMC Hadoop Starter Kit (HSK) 2.0 for step-by-step guides on how to connect a Hadoop distribution to your Isilon cluster. HSK guides are available for Apache Hadoop, Pivotal HD, Cloudera, and Hortonworks. A video demonstration for Pivotal HD is also available.
  2. Find your Isilon cluster’s optimal point to help determine the number of nodes that will best serve your Hadoop workflow and compute grid. The optimal point is the point at which it scales in processing MapReduce jobs and reduces run times in relation to other systems for the same workload. Contact your account representative to help you determine this information.
  3. Create directories and set permissions. OneFS controls access to directories and files with POSIX mode bits and access control lists (ACLs). Make sure directories and files are set up with the correct permissions to ensure that your Hadoop users can access their files.
  4. Don’t run NameNode and DataNode services on clients. Because the Isilon cluster acts as the NameNode and DataNodes for the HDFS, these services should only run on the cluster and not on compute clients. On compute clients, you should only run MapReduce processes.
  5. Increase the HDFS block size from the default 64 MB to 128 MB to optimize performance. Boosting the block size lets Isilon nodes read and write HDFS data in larger blocks. The result is an increase in performance of MapReduce jobs.
  6. Store intermediate jobs on an Isilon cluster. A Hadoop client typically stores its intermediate map results locally. The amount of local storage available on a client affects its ability to run jobs. Storing map results on the cluster can help performance and scalability.
  7. Consult the Isilon best practices white paper for additional tips. You can find more details about some of these best practices in “EMC Isilon Best Practices for Hadoop Data Storage.” You can also find additional tips for tuning OneFS for HDFS operations, using EMC Isilon SmartConnect™ for HDFS, aligning datasets with storage pools, and securing HDFS connections with Kerberos.

 

If you have questions related to Hadoop and your Isilon environment, contact your account representative. If you have documentation feedback or want to request new content, email isicontent@emc.com.

[display_rating_result]

7 best practices for using the EMC Isilon OneFS Job Engine

Kirsten Gantenbein

Kirsten Gantenbein

Principal Content Strategist at EMC Isilon Storage Division
Kirsten Gantenbein
Kirsten Gantenbein
The OneFS Job Engine primary functions.

OneFS Job Engine primary functions

In the EMC Isilon OneFS® operating system, the Job Engine manages the scheduling, reporting, and the initiation of cluster protection and maintenance tasks. A significant benefit of the Job Engine is that it can adaptively manage maintenance jobs based on the amount of cluster resources available.

To ensure that your cluster is performing at optimal levels, follow these Job Engine best practices:

  • Schedule jobs to run during your cluster’s low usage hours.
  • When possible, use the default priority, impact, and scheduling settings for each job.
  • Ensure that the cluster, including individual node pools, is less than 90 percent full.
  • If the EMC Isilon SmartPools™ software module for resource management is licensed, ensure that default spillover setting is enabled.
  • Set up notification rules to alert you when the cluster begins to reach capacity thresholds and enable virtual hot spares (VHS) to reserve free space that’s needed in case you need to rebuild data if a disk failure occurs.
  • Do not disable the Snapshot Delete job.
  • Run the FSAnalyze job only if you are using InsightIQ. FSAnalyze creates data for the InsightIQ file system analytics tools and provide details about data properties and space usage with the OneFS file system.

Additional best practices can be found in the white paper, “EMC Isilon OneFS Job Engine.”

In the latest release of OneFS, several enhancements were made to the Job Engine. These enhancements were highlighted in our video, “Key New Features in EMC Isilon OneFS 7.1.” For more information about the enhanced Job Engine, read the white paper, “EMC Isilon OneFS Job Engine.”

If you have questions or feedback about this blog, email isi.knowledge@emc.com. To provide documentation feedback or request new content, email isicontent@emc.com.