As part of Metal Toad’s Managed Services Team, I have worked extensively in custom cloud based solutions. There is no doubt Amazon Web Services (AWS) leads the pack in Cloud hosting, but when it comes to a Network Attached Storage (NAS) solution, I have found a shortfall...a fast solution.
Benchmark NAS on AWS by mounting EFS or NFS on EC2 and running IOzone or fio. Measure throughput, IOPS, and latency across file sizes and record lengths to compare vendors and configurations.
Simple Storage Service (S3) is AWS’s file solution to NAS, and it is a great system, but without refactoring our code to serve the images directly from S3, we are left with using the S3 Fuse Driver which responds slowly when serving content.
One could also argue for the use of Elastic Block Storage (EBS), and it solves the issue of slowness when using S3. But EBS is unable to be mounted on multiple EC2 machines, essentially defeating the purpose (NAS needs to be shared).
When searching for a solution, many “How To’s” recommend a program called GlusterFS as a NAS storage solution on AWS. It is a software package that is fast and replicates data between machines. However, I am never satisfied taking other people’s word for “the best solution”. So, I decided to test Gluster against some of the other NAS options offered for AWS.
Description:
Pros:
Cons:
Setup:
Description:
Pros:
Cons:
Setup:
Description:
Pros:
Cons:
Setup:
The tool I decided to used for the test is called IOZone. It is an older tool with many positive reviews on multiple many forums. IOZone allows you to test different file sizes, record lengths, and io options. The IO options range from read, write, random read/ random write, backwards read, and lots more.
I wanted to see the whole gambit of tests so I ran iozone -A. The ‘-A’ is a flag that gives a wide range of record lengths and files sizes. We ran the test for files up to 512MB but the data in the graphs is capped at 8MB because 99.9% of our files are smaller than 8MB.
Write: This test measures the performance of writing a new file. When a new file is written not only does the data need to be stored but also the overhead information for keeping track of where the data is located on the storage media. This overhead is called the “metadata” It consists of the directory information, the space allocation and any other data associated with a file that is not part of the data contained in the file. It is normal for the initial write performance to be lower than the performance of re-writing a file due to this overhead information.
Re-write: This test measures the performance of writing a file that already exists. When a file is written that already exists the work required is less as the metadata already exists. It is normal for the rewrite performance to be higher than the performance of writing a new file.
Read: This test measures the performance of reading an existing file.
Re-Read: This test measures the performance of reading a file that was recently read. It is normal for the performance to be higher as the operating system generally maintains a cache of the data for files that were recently read. This cache can be used to satisfy reads and improves the performance.
Random Read: This test measures the performance of reading a file with accesses being made to random locations within the file. The performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.
Random Write: This test measures the performance of writing a file with accesses being made to random locations within the file. Again the performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.
Below are the straight up comparisons of speeds for the tests we’re interested in.
Blue: Gluster
Red: SoftNAS
Yellow: Zadara
Here we see that Zadara is better at Write Speeds on smaller files, though they even- off as they get larger. Winner: Zadara.
For Random Writes we see all of the solutions perform better as the record length gets bigger but Gluster clearly outperforms all of them. Winner Gluster.
Rewrite is a much closer write then the random or straight write test. I would call this a tie between Gluster and Zadara.
Read tests: Zadara performed much slower with small record lengths and once the file got larger then a 1MB Gluster started slowing down. Winner: SoftNAS.
Random read: all clients performed equally well. Winner: Tie.
Reread is very close once as the record length grows, but the for smaller records lengths SoftNAS has better record lengths.
Since SoftNASperformed marginally better at reads and Zadara was better at writes I wanted to get a better look to see the differences between them.
The above graph shows the difference between Zadara and SoftNAS. Using the Zadara speed, we subtract the SoftNAS speed, allowing us to see the difference between the speeds. The results will then show a negative number when SoftNAS is faster and positive when Zadara is faster.
As you can see, the write is slightly better towards Zadara but the reads are extremely fast for Soft NAS. Our primary use for the NAS would be to server static files to various web servers, so read speed is a big plus.
Selecting the optimal NAS solution requires careful analysis of your specific workload patterns and performance requirements. The test results reveal distinct performance profiles that align with different use case scenarios, making workload classification the critical first step in your decision process.
Read-Heavy Workloads (Web Assets, CDN Origins, Static Content) For applications primarily serving static content like images, CSS, JavaScript files, or document repositories, SoftNAS emerges as the clear winner. Its exceptional read performance, particularly with smaller files under 1MB, makes it ideal for web applications where users frequently request assets. The solution's ability to handle concurrent read operations efficiently translates directly to improved page load times and user experience. Consider SoftNAS when your read-to-write ratio exceeds 10:1 or when serving high-traffic websites with substantial static content libraries.
Write-Heavy Workloads (Log Aggregation, Data Processing, Backup Systems) Applications with frequent write operations, particularly those involving random write patterns, should prioritize GlusterFS. Its superior random write performance across all record lengths makes it suitable for log aggregation systems, real-time data processing pipelines, and applications with frequent file updates. The distributed nature of GlusterFS also provides natural load distribution for write operations across multiple nodes. However, for sequential write operations with smaller files, Zadara's dedicated hardware approach may provide better throughput, especially for applications requiring consistent, predictable write performance.
Mixed Workloads (CMS Platforms, Development Environments, Collaboration Tools) Content management systems, development environments, and collaboration platforms typically exhibit mixed I/O patterns with both significant read and write operations. For these scenarios, evaluate the specific read-to-write ratio and file size distribution. GlusterFS often provides the best balance for mixed workloads due to its strong random write performance and acceptable read speeds. However, if read operations dominate even in mixed environments (common in CMS scenarios where content is written once but read frequently), SoftNAS may still be the optimal choice despite slightly lower write performance.
Performance Scaling Considerations Remember that performance characteristics change significantly with file sizes. While our testing focused on files under 8MB, larger file operations (128MB+) showed Zadara's clear advantage due to its dedicated hardware and optimized storage controllers. Applications handling media files, database backups, or large document repositories should factor this scaling behavior into their decision matrix.
Implementing robust high availability for NAS solutions on AWS requires understanding each platform's native redundancy capabilities and designing appropriate failover mechanisms. The choice of NAS solution significantly impacts your HA architecture complexity and recovery time objectives.
GlusterFS Distributed Redundancy GlusterFS provides inherent high availability through its distributed architecture. By configuring replica volumes across multiple availability zones, you create automatic failover capabilities with no single point of failure. The recommended configuration involves at least four nodes distributed across two AZs, with each file replicated to nodes in both zones. This setup ensures continued operation even if an entire availability zone becomes unavailable.
Monitor GlusterFS cluster health using native volume status commands and integrate with AWS CloudWatch for automated alerting. Implement automated node replacement using Auto Scaling Groups with custom AMIs pre-configured with GlusterFS. The self-healing capabilities of GlusterFS automatically synchronize data when failed nodes rejoin the cluster, though this process can be resource-intensive during large-scale recovery operations.
SoftNAS Enterprise HA Configurations SoftNAS offers built-in high availability features through its SNAP HA™ technology, which provides automated failover between primary and secondary instances. This active-passive configuration typically achieves recovery times under 60 seconds for most workloads. The solution monitors both instance health and storage connectivity, automatically promoting the secondary instance when failures are detected.
Configure SoftNAS HA pairs across availability zones with cross-zone replication of EBS volumes. Implement AWS Route 53 health checks to automatically update DNS records during failover events. The enterprise features include automated EBS snapshot management and point-in-time recovery capabilities, crucial for maintaining data consistency during failover scenarios.
Zadara Managed Redundancy Zadara's approach to high availability leverages their dedicated hardware infrastructure with built-in redundancy at the storage controller level. Their Virtual Private Storage Arrays (VPSA) include automatic failover between redundant controllers and the ability to configure RAID levels that survive multiple drive failures.
Implement application-level failover by deploying multiple Zadara arrays across availability zones with application logic handling array selection. While this requires more sophisticated application design, it provides the highest performance during normal operations since both arrays can serve read traffic simultaneously. Configure automated health monitoring using Zadara's REST APIs integrated with AWS Lambda functions for custom failover logic.
Network and DNS Failover Strategies Regardless of your chosen NAS solution, implement DNS-based failover using Route 53 health checks and weighted routing policies. Configure health checks that verify both NFS mount availability and actual file system responsiveness, not just instance health. Use short TTL values (30-60 seconds) to minimize failover detection time while balancing DNS query overhead.
Consider implementing client-side failover logic for applications that can handle temporary storage unavailability. This approach provides faster recovery than DNS-based solutions and allows for more sophisticated retry logic tailored to your specific application requirements.
Understanding the total cost of ownership for AWS NAS solutions requires analyzing multiple cost components that vary significantly between approaches. Hidden costs and scaling factors can dramatically impact long-term financial planning, making comprehensive cost modeling essential for informed decision-making.
Instance Costs and Scaling Patterns GlusterFS typically requires the most compute resources since storage and processing functions run on the same instances. Budget for m5.large or m5.xlarge instances as minimum configurations, with memory requirements scaling based on the number of files rather than total storage capacity. A GlusterFS cluster serving 1 million small files may require 16GB+ of RAM per node, while the same storage capacity with fewer, larger files might operate efficiently with 8GB.
SoftNAS leverages dedicated appliance instances that separate storage logic from application workloads. The vendor recommends m5.large instances for most implementations, with costs predictable based on the number of concurrent connections rather than total throughput. This predictability simplifies capacity planning but may result in over-provisioning during low-utilization periods.
Zadara's hybrid model includes both AWS instance costs for NFS clients and Zadara's own infrastructure fees. While EC2 instance requirements are minimal (typically m5.medium suffices), the Zadara infrastructure fees often represent 60-70% of total costs. However, these fees include enterprise features like 24/7 support and hardware management that would otherwise require additional staffing costs.
EBS IOPS and Storage Optimization EBS costs vary dramatically based on IOPS requirements and storage type selection. GlusterFS installations benefit from gp3 volumes with provisioned IOPS for metadata storage and st1 volumes for bulk data storage in distributed configurations. Calculate IOPS requirements based on concurrent client connections multiplied by average operations per connection, then add 40% overhead for metadata operations.
SoftNAS can leverage EBS IOPS reservations more efficiently due to its optimized storage stack. Provisioned IOPS SSD (io2) volumes often provide better cost-per-performance ratios than gp3 for SoftNAS implementations exceeding 10,000 IOPS. However, monitor actual IOPS utilization closely since reserved IOPS costs apply regardless of actual usage.
Zadara eliminates EBS IOPS costs entirely, replacing them with predictable per-GB storage fees that include all necessary IOPS. This model simplifies budgeting but may be cost-prohibitive for applications with low IOPS requirements relative to storage capacity.
Data Transfer and Network Costs Inter-AZ data transfer costs can accumulate quickly with distributed storage solutions. GlusterFS replica configurations generate significant cross-AZ traffic for both replication and client access patterns. Budget approximately $0.02 per GB for cross-AZ transfers, which can add $200-500 monthly for moderate-usage scenarios.
SoftNAS HA configurations generate data transfer costs during synchronization operations but minimize ongoing transfer fees since clients typically access a single instance. However, the automated failover process may cause temporary spikes in data transfer costs during recovery operations.
Zadara's co-location within AWS facilities minimizes data transfer costs for most operations. However, their pricing model includes bandwidth charges that may exceed standard AWS data transfer rates for high-throughput applications. Compare Zadara's bandwidth fees against projected AWS data transfer costs using your application's specific traffic patterns.
Hidden Costs and Scaling Surprises Factor in operational costs including backup storage, monitoring systems, and administrative overhead. GlusterFS requires the most operational expertise, potentially requiring specialized staff or consulting services. SoftNAS reduces operational complexity through its management interface but includes ongoing licensing fees that scale with storage capacity.
Consider cost implications of performance tuning and scaling. GlusterFS scaling requires careful rebalancing operations that may necessitate temporary performance degradation. SoftNAS scaling typically requires storage migration procedures with associated downtime and data transfer costs. Zadara scaling involves provisioning requests that may include setup fees and minimum commitment periods.
Optimizing NFS performance on AWS requires understanding the interaction between network configuration, protocol tuning, and underlying storage characteristics. Default NFS settings rarely provide optimal performance for cloud environments, making systematic tuning essential for achieving the performance levels demonstrated in our benchmarks.
NFS Protocol Optimization Configure NFS version 4.1 or higher to leverage parallel data transfers and improved caching mechanisms. Enable pNFS (parallel NFS) when supported by your NAS solution, as it allows clients to access storage directly rather than routing all traffic through a single NFS server. This configuration particularly benefits read-heavy workloads by distributing network load across multiple paths.
Optimize rsize and wsize parameters based on your network configuration and typical file sizes. For AWS Enhanced Networking instances, set rsize and wsize to 1048576 (1MB) to maximize throughput for larger files while maintaining efficiency for smaller files. However, reduce these values to 65536 (64KB) if your workload primarily involves small files under 1MB to minimize network overhead and improve latency.
Tune the NFS read-ahead parameters aggressively for sequential read workloads. Set the read_ahead_kb value to 15MB or higher on client systems to improve throughput for applications that read files sequentially. For random access patterns, disable read-ahead entirely by setting the value to 0 to prevent unnecessary network traffic and reduce memory usage.
AWS Network Configuration Enable Enhanced Networking on all instances participating in NFS operations to achieve single root I/O virtualization (SR-IOV) performance. This feature provides higher packet-per-second performance, lower latency, and reduced jitter compared to standard networking. For NFS server instances, consider using placement groups to ensure optimal network connectivity between cluster members.
Configure Jumbo Frames (9000 MTU) across your entire NFS infrastructure when all components support this configuration. Jumbo frames reduce per-packet overhead and CPU utilization for large file transfers, potentially improving throughput by 15-25% for sequential operations. However, ensure your entire network path supports jumbo frames, including load balancers and security groups, to prevent fragmentation issues.
Implement multiple network interfaces on NFS servers to separate management traffic from data traffic. Configure dedicated subnets for NFS traffic with optimized routing tables that minimize network hops. This separation improves both performance and security by isolating storage traffic from general application communication.
Client-Side Performance Tuning Optimize the NFS client cache settings to match your access patterns. Increase the number of RPC slots (max_slot_table_entries) to 128 or higher for applications with high concurrency requirements. This setting controls how many simultaneous NFS operations can be outstanding, directly impacting throughput for multi-threaded applications.
Configure appropriate timeout values to balance responsiveness with stability. Set soft mount options with reasonable timeout values (timeo=600, retrans=2) for non-critical data to prevent applications from hanging indefinitely during network issues. Use hard mounts with longer timeouts for critical data where data integrity takes precedence over availability.
Implement local caching strategies using tools like CacheFS or application-level caching to reduce NFS traffic for frequently accessed files. This approach particularly benefits read-heavy workloads by serving repeated requests from local storage rather than generating network traffic for every access.
Monitoring and Performance Analysis Deploy comprehensive monitoring to identify network bottlenecks and optimize configuration iteratively. Use nfsiostat to monitor NFS operation latency and throughput at the client level, identifying whether performance limitations originate from network, server, or storage subsystems. Monitor network utilization using CloudWatch Enhanced Monitoring to ensure you're not approaching bandwidth limitations.
Implement application-level performance monitoring to understand how NFS performance impacts end-user experience. Many applications can mask NFS latency through asynchronous operations or caching, while others may be severely impacted by even minor performance degradations. Understanding these application-specific requirements guides optimization priorities and helps validate the effectiveness of tuning efforts.
Configure automated alerting for performance degradation using CloudWatch metrics combined with custom monitoring scripts. Set thresholds based on baseline performance measurements rather than arbitrary values, ensuring alerts indicate meaningful performance changes rather than normal operational variations.
The IOZone ran files sizes up to 512MB but I couldn’t get GlusterFS to finish without the NFS mount hanging.
As I mentioned above, the graphs stopped at 8MB because of what we used the servers for, but when the file size made it to 128MB the Write speed for Zadara greatly out- performed both SoftNAS and GlLuster.
Gluster is best for quick setup and reduced cost since it can run on the same instance of its server, though it can be bogged down if the server load gets too high.
Zadara is great if you need the added security of having dedicated drives, or if you have a right intensive application.
SoftNAS works best at serving lots of small files quickly.
Pick the one thats right for you.