SNIA, the storage industry association, notes that most, if not all, storage vendors implement S3 as their primary object storage API, and that real-world interoperability issues still persist. The “S3-compatible” label tells you which API a platform speaks. It doesn’t tell you how much of that API works with your backup, archive, or analytics applications.
An object storage solution is only as compatible as the features your applications actually use, so the safest way to choose is to test your own workload rather than compare feature lists. This guide is for IT and infrastructure leaders deciding where to keep backups, archives, and large unstructured data. It explains S3 feature coverage, security, performance and scale, data protection, integrity, cost, and platform fit, and ends with a proof-of-concept plan.
What is S3-compatible object storage, and why does compatibility vary?
S3-compatible object storage is a storage system that accepts requests written for the Amazon S3 API, so applications read and write data as objects in buckets over HTTP. Because S3 is a large, evolving API, compatibility can vary based on which features have been implemented by your storage vendor.
If S3 API compatibility falls short of what an application expects, it can look like a network problem at first. A backup job passes basic tests, but fails intermittently on large files. After the network team spends hours investigating, the storage logs show that multipart uploads were handled differently from what the application expected.
Three layers have to match between an application and the storage. The first is the wire format (REST calls over HTTP), the second is authentication (Signature Version 4), and the third is the data model (buckets, objects, keys, and metadata). The harder problems sit in less common features and behaviors, such as multipart upload, versioning, and Object Lock.
Compatibility is a spectrum. SNIA notes that Amazon S3 serves as the de facto standard for object storage, and each vendor makes a decision on their own how much of it to implement in their solution. In a poll of attendees at a 2024 SNIA webinar, 72% reported incompatibility issues between client applications and object storage implementations.
Which S3 features should I test before choosing a platform?
Test the S3 operations your own applications call, using a real client against a test bucket. The areas that most often diverge are multipart upload, versioning, Object Lock, lifecycle rules, and listing behavior.
Start by asking each vendor for its list of supported S3 operations, and compare it with what your application calls. Then check these areas.
- Multipart upload. Large files go up in many parts. A system that handles part sizes or abandoned uploads differently can fail jobs intermittently. Maximum object size also differs by platform.
- Versioning. Check that listing versions, deleting, and overwriting all work fully, not only that the setting can be switched on.
- Object Lock. Object Lock has two retention modes. Governance mode lets accounts with a special permission override a lock, by design. Compliance mode blocks every account, including administrators, until retention expires. Confirm the platform enforces compliance mode in storage, and not only in its API responses.
- Lifecycle rules. Check which rule types run, such as expiring old versions and cleaning up incomplete multipart uploads. The risk is a rule the platform accepts but never runs, because capacity then grows without anyone noticing.
- Listing at scale. Behavior can change when a bucket holds millions of keys and the application pages through them.
- Error codes and consistency. Clients branch on specific error codes, and many assume a read right after a write returns the new data.
- Newer S3 features. Capabilities added to S3 recently tend to be the least consistently implemented, such as conditional writes and newer checksum headers that current AWS SDKs send by default. Compare the dates on any claims of “100% API Coverage” provided by vendors.
A demo that uploads and downloads one file proves very little. SNIA’s multi-vendor interoperability plugfests turned up missing or incorrect response headers, unsupported API calls, and unexpected behavior across implementations.
What security controls should an S3 platform have?
Check access control, encryption, and audit logging, and test them with your own identity setup. These differ across platforms as much as the data features do, and they often determine whether new storage passes your security review.
- Access control. Confirm how access keys map to users and groups, whether bucket policies are supported, and which policy conditions actually work. On multi-protocol systems, ask how S3 permissions interact with file permissions on the same data.
- Encryption in transit. Require TLS on the S3 endpoint, check how certificates are issued and renewed, and restrict which networks can reach the endpoint.
- Encryption at rest. Ask where keys live, and what happens to data access if the key manager is unreachable.
- Server-side encryption requests. Some applications ask for S3 server-side encryption on upload. Check whether the platform honors those requests, ignores them, or rejects them, since each one plays out differently in your application.
- Audit logging. Confirm that S3 requests are logged with the user, action, and object, and that logs can be sent to your SIEM. On TrueNAS, S3 auditing requires TrueNAS Enterprise.
- Protection of the management plane. Ransomware often goes after administrator credentials. Ensure that multi-factor authentication can be enabled on the management interface, and leverage role-based access controls to limit permissions.
How is the data protected, and which failures does it survive?
Every platform protects data with full copies, parity, or both, and the method matters less than the failures it covers. Scope is the key question: does protection span disks inside one system, or nodes and sites?
- Replication keeps full copies on separate devices, nodes, or sites. It’s simple and quick to recover from, though every copy consumes full capacity.
- Erasure coding splits data into fragments plus parity, so it can rebuild after several failures with less overhead than full copies. TrueNAS uses OpenZFS RAIDZ to deliver single, double, or triple parity, and divides larger groups of disks into multiple protection groups to deliver robust fault tolerance.
Protection inside one system covers disk failures. Surviving the loss of a controller, node, or site takes more, such as redundant controllers, a multi-node design that spreads fragments across nodes, or replication to a second system. Ask every vendor which failures each layer covers and how long a rebuild takes at your capacity, since a second failure during a long rebuild is the real risk.
Parity and replication protect against hardware failure, not deletion or ransomware. A real-time replica copies a delete as faithfully as a write. Protecting against these types of attacks falls to versioning, Object Lock, and even additional layers of protection beyond the S3 implementation.
How do I verify data integrity and self-healing?
Data integrity is the guarantee that what you read back is what you wrote. To verify it, ask each vendor three questions. Is every block checksummed and checked on read? Is idle data scanned on a schedule? Can the system repair bad data automatically from redundant copies?
Silent corruption, often called bit rot, changes stored data without raising an error. A checksum only helps if it’s verified. Data that is rarely read stays unchecked until a scheduled scan reads it, and repair needs a second good copy or parity to restore from. Without protection through redundancy measures, a system can detect corruption but cannot fix it.
OpenZFS is one documented example. As the TrueNAS ZFS primer describes, it writes a checksum for each block, validates it on every read, and corrects errors using redundancy, while scheduled scrubs read the blocks that normal use rarely touches. Other platforms reach the same goal in different ways, so ask each vendor for the mechanism and scan schedule, and what alerts it sends.
In a proof of concept, ask the vendor how to cause a controlled failure in the lab, such as pulling a disk, and watch the system detect and repair it.
How do I evaluate performance and scale?
Measure performance with your own object sizes, client counts, and access patterns, because published benchmarks rarely match a real workload. Object storage behaves very differently with many small objects than with a few large ones.
- Throughput for large objects. Test sustained read and write rates using the part sizes and number of parallel streams your application uses. Backup and media jobs depend on this.
- Request rate for small objects. Test operations per second and latency for small objects, metadata calls, and listings. Analytics and AI pipelines often stress this more than raw throughput.
- Behavior under load. Run S3 traffic while other workloads are busy, including file and block traffic on multi-protocol systems, and again during a disk rebuild.
- The network path. Measure from the clients that will actually connect, so a network bottleneck isn’t mistaken for a storage limit.
For scale, ask for documented limits rather than headline maximums. That means objects per bucket, buckets per system, maximum object size, and usable capacity per system. Ask how capacity is added (new disks, new shelves, or new nodes), whether growth needs downtime or data migration, and how performance and rebuild times change as the system fills.
How does on-premises S3 cost compare with cloud object storage?
Compare total cost per usable terabyte over the system’s life, at the same level of protection. Cloud object storage bills capacity, requests, and data transfer separately. On-premises cost is hardware, software and support, power, space, and staff time.
Start by matching the protection levels. Cloud redundancy options vary, and some providers may differentiate between offering redundancy within a single datacenter, or redundancy across multiple datacenters, with costs scaling to match. A single on-premises system keeps data in one place, under your control. A fair comparison adds a second copy to the on-premises side, at another site, in the cloud, or on another form of media such as tape.
Then model the cloud side the way the bill is built. Take stored capacity times the storage rate, add requests times the request rate, and add data read back out times the egress rate. For data that is read back often, egress costs can add a surprising amount to the bottom line of a traditional price-per-TB comparison. At $0.09 per GB, a common first-tier list rate for internet egress, reading back 50 TB in one month comes to over $4,500, and volume discounts bring that down only slightly. Rates change often, so use each provider’s current price list.
On the on-premises side, model these inputs over a realistic hardware lifecycle.
- Hardware, software licenses, and support contracts
- The second copy needed to match cloud redundancy, if offered
- Power, cooling, and rack space
- Staff time for upgrades and monitoring
- Capacity you expect to add each year
- Hardware refresh and data migration at end of life
The access pattern usually decides the comparison. Data that is written once and rarely read favors cloud pricing at small scale. Data that is read often, or that grows steadily, tends to shift the math toward owned hardware. Price the worst case too. Restoring everything after a ransomware incident or a provider exit can trigger a large one-time egress bill.
Should I choose a dedicated object platform or one that also serves file and block?
A dedicated object platform fits when scale, multiple sites, or high request rates drive the design, and you’re certain that the requirements will never expand to include other protocols on the same data. A multi-protocol platform fits when the same data needs to be accessed via both file and object protocols, or the same hardware has to play multiple roles within your organization.
Here’s how the main designs line up.
- Dedicated object platforms are built around object storage alone. They tend to scale out across many nodes and sites, and they usually treat file access as an add-on or leave it out entirely.
- Multi-protocol platforms serve file, block, and object from one system. Confirm that their Object capacity and scaling methods are sufficient for your workloads.
TrueNAS is an example of a multi-protocol platform. It has offered S3-compatible object storage since TrueNAS 11.0 in 2017, and the upcoming TrueNAS 27 introduces a refreshed S3-compatible service mapping Object buckets directly to ZFS datasets. The updated service is labeled Early Access in the currently-available Release Candidate, with full production status targeted for early 2027.
If you’re considering a multi-protocol platform, three more questions are worth asking.
- When the same data is reached over file and S3, which permission model applies?
- Does a failure of one protocol service affect the others?
- What happens to S3 performance when file workloads are busy?
Evaluation checklist
Use this table as a scorecard for each platform on your shortlist.
| Criterion | What to check | Why it matters |
|---|---|---|
| S3 operation coverage | The vendor’s supported-operation list against the calls your application makes | Missing calls show up as failed jobs, not clear errors |
| Multipart and large objects | Part sizes, cleanup of abandoned uploads, maximum object size | Backups and media files upload in many parts |
| Versioning | Full version listing, delete behavior, overwrite behavior | Partial support can look like missing history |
| Object Lock | Whether compliance mode blocks every account, whether governance overrides are limited to permitted accounts | Immutable backups depend on real enforcement |
| Lifecycle rules | Which rule types run, including version expiry and incomplete-upload cleanup | Rules that never run let capacity grow unnoticed |
| Listing and consistency | Paging through millions of keys, reads right after writes | Applications assume both |
| Error codes | Whether failures return the codes your client expects | Clients branch on error codes |
| Security | Access policies, TLS, key management, server-side encryption handling, audit logs, admin MFA | Determines whether the platform passes security review |
| Performance | Large-object throughput, small-object request rate, behavior under load and during rebuilds | Results shift sharply with object size and concurrency |
| Scale limits | Documented limits, how capacity is added, whether growth needs downtime | Limits are easy to miss until capacity grows |
| Protection scope | Behavior after disk, controller, node, and site failures, and rebuild time | Disk-level protection alone does not survive a controller or site loss |
| Integrity checks | Block checksums, scheduled scans, automatic repair | Silent corruption goes unnoticed without them |
| Cost model | Capacity, requests, egress, support, power, staff time, and a second copy over the lifecycle | List price per gigabyte hides the real bill |
| Licensing and support | License terms, potential limitations, and available support tiers and SLAs | Capacity-based licensing may impact scalability, and free solutions may offer minimal or no formal support |
How do I run a proof of concept?
Run your real application against a test bucket, on realistic data volumes, with retention locks switched on, and read the storage logs afterward. A short test with the production application tells you more than a long feature matrix.
- List the S3 calls your application makes, from its logs or documentation.
- Point the application at the test endpoint and run a full cycle, such as a backup and a restore, or a complete pipeline run.
- Enable versioning and Object Lock in compliance mode, then try to delete or overwrite a locked object from an administrator account. The attempt should fail. Repeat in governance mode and confirm that only accounts with bypass permission can override. Use retention periods of a day or two, since a compliance-mode lock cannot be shortened, and a test bucket set to years will hold its capacity for years.
- Upload your largest files and a large number of small ones, then page through a bucket listing at realistic scale.
- Measure throughput and request rates with your real client count, then again during a disk rebuild.
- Test access policies with your own identity setup, and confirm each action appears in the audit log.
- In the lab, fail a disk or a node, and record how the system detects the failure, repairs the data, and alerts you.
- Capture the cost inputs from the cost section using real measurements.
- Review the logs for retries, unexpected error codes, and failed multipart uploads.
Score each platform against the checklist and record what you couldn’t test, so those gaps stay visible as open risks.
Frequently asked questions
Do I need object storage for a data lake? Not always. Object storage suits large, read-heavy datasets that many tools reach over one standard API, which is why analytics and machine learning frameworks often speak S3 directly. If your tools read from file shares and the dataset is modest, a file system can work with fewer changes.
Can I run S3-compatible storage on premises? Yes. S3 is an API, not a location, so vendors and open-source projects implement it on hardware you own. In exchange for predictable cost and control over where data lives, you take on capacity planning, upgrades, and often support.
What is Object Lock? Object Lock is an S3 feature that blocks deletion or overwrite of an object version for a set retention period. In governance mode, accounts with a special permission can override the lock. In compliance mode, no one can. It requires S3 versioning and usually cannot be turned off for a bucket once enabled, as TrueNAS documents for its own S3 service. Backup software makes use of Object Lock as a layer to make its data immutable.
How do I migrate from cloud S3? Start by pointing a test copy of your application at the new S3 endpoint, since most S3 clients accept a configurable endpoint. Once you’ve validated initial connectivity, run a proof-of-concept exercise to verify performance, scalability, and functionality of your on-premises solution. Identify and create a migration plan, accounting for egress charges on your current provider, and verify that your sync or copy tool will preserve version history, retention settings, and metadata tags where applicable. If using Object Lock in your cloud-based S3 provider, ensure that Compliance mode data stored will expire properly, or can be removed if Governance mode was used.