Ecs RN 3803
Ecs RN 3803
Ecs RN 3803
2
May 2024
These release notes describe new features, resolved issues, and known issues that are included in this version of ECS.
Current Release Version: 3.8.0.3
Release Type: Patch (P)
Topics:
• Revision history
• Product description
• New features
• Changed features
• Resolved issues
• Known issues
• Environment and system requirements
• Installation and upgrade considerations
• Where to get help
Revision history
This section provides a description of document changes.
Product description
This section provides a high-level overview of the ECS Appliance and ECS Software products.
For additional ECS documentation resources, refer to www.dell.com/support.
NOTE: Throughout this document ECS refers to the ECS Appliance and ECS Software products.
1
About ECS
ECS has been purpose-built to store unstructured data at public cloud scale with the reliability and control of a private cloud.
Capable of scaling to exabytes and beyond, ECS empowers organizations to manage a globally distributed storage infrastructure
under a single global namespace with anywhere access to content.
Features:
● Supports mobile, cloud, big data, and social networking applications
● Can achieve higher performance using ECS flash-based storage appliances to fuel the next wave of data-hungry AI, machine
learning, IoT, analytics, and cloud-native applications
● Can do more with data at a lower total cost of ownership than public cloud solutions
● Features rich S3 compatibility and multiprotocol interoperability
● Offers universal access to Object and HDFS protocols on a single platform
● Unmatched combination of storage efficiency and data access
● Anywhere read/write access with strong consistency simplifies application development
● No single points of failure increases availability and performance
● Universal accessibility eliminates storage silos and inefficient ETL/data movement processes
● Support for ViPR SRM. ViPR SRM is multi-vendor storage resource reporting software which offers a reporting solution for
ECS
● This Dell product utilizes an embedded Linux distribution which is licensed to Dell. This embedded Linux is only authorized to
be used to operate the Dell product and only the Dell product.
ECS hardware
ECS offers multiple appliance configurations supporting many applications and certification of third-party hardware for use with
ECS Software.
The Dell EMC Gen3 hardware is the next generation of ECS hardware. In previous releases, the ECS Gen 2 hardware which
consisted of D- and U-series Intel-based unstructured servers with separate disk array enclosures (DAEs) and Cisco/Arista
private 1 GbE switches and public Arista public 10 GbE switches.
2
Hardware generations
ECS appliances are characterized by hardware generation.
Gen3
● EX5000 Gen3 models offered in a one-node chassis configuration, the EX5000S; and offered in a dual-node chassis
configuration, the EX5000D supported on ECS 3.7 and later. This solution supports 25 to 100 drives, at 25-drive increments
and density of 14 PB per rack. ECS 3.7 and later support 20 TB data drives.
● EXF900 Gen3 models featuring 3.84 TB NVMe SSD. ECS 3.6 and later support this hardware configuration and 7.86 TB and
15.36 TB NVMe SSD are available.
● EX500 Gen3 models featuring 8 TB or 12 TB disks (12 or 24 x HDD per node). ECS 3.5 and later supports 16 TB data drives
and optional Read Cache SSD upgrade. ECS 3.7 supports 2 TB, 4 TB, 12 TB, 16 TB, and 20 TB disks.
● EX300 Gen3 models featuring 1 TB, 2 TB, 4 TB, or 8 TB disks (12 HDD per 2U node). ECS 3.5 and later supports 16 TB data
drives and optional Read Cache SSD upgrade.
● EX3000 Gen3 models are offered in a one-node chassis configuration, the EX3000S, and a dual-node chassis configuration,
the EX3000D. This solution supports 30 to 90 drives at 15-drive increments and a density of 11.52 PB per rack. ECS 3.5 and
later support 16 TB data drives and an optional Read Cache SSD upgrade.
Gen2
For more information about Gen2 hardware, see the Dell EMC ECS D- and U-Series Hardware Guide.
● U-Series Gen2 models featuring 8 TB and 12 TB disks.
● The D-Series was introduced featuring 8 TB disks. D-Series models featuring 10 TB disks.
● The original U-Series appliance (Gen1) was replaced with second-generation hardware (Gen2).
New features
This section summarizes the new features in ECS 3.8.
vNest ECS 3.8 provides enhancements for a more stabilized vNest, including the automatic recovery of stuck
enhancements invisible keys, vNest writer performance enhancements, and improved memory management in vNest.
Data movement ECS 3.8 provides copy-to-cloud functionality where locally stored S3 object data can be copied to an
(Copy-to-Cloud) external S3 target. This feature supports AWS and external ECS. See ECS Version 3.8 Administration
Guide for more details.
IAM STS ECS 3.8 provides the GetFederationToken API, which is part of the Security Token Service (STS) along
GetFederationTok with AssumeRole and AssumeRoleWithSAML, used to provide temporary security credentials for IAM
en users. See ECS Version 3.8 Administration Guide, Data Access Guide for more details.
Extended key ECS 3.8 supports the new EKM cluster type, Thales CipherTrust. KMIP is an extensible communication
management protocol that defines message formats for the manipulation of cryptographic keys on a key management
interoperability server. See ECS Version 3.8 Administration Guide for more details.
protocol (KMIP)
Support
Object lock ECS 3.8 supports Object Lock with read-only ADO and adds new flags to allow enabling Object Lock with
enhancements for writable ADO on buckets. See ECS Version 3.8 Data Access Guide for more details.
ADO
Memory upgrade ECS 3.8 supports memory upgrade expansion to 192 GB on EX300, EX500, EX3000, and Gen2 platforms,
expansion with the support of professional services. Contact your ECS Customer Service representative for more
information.
SSH key rotation ECS 3.8 supports new key rotation in an ECS cluster.
improvements
Firmware upgrade ECS 3.8 supports automating the firmware upgrade procedure on all Gen 3 clusters, except for the
using service EX3000, using the SC DSU script. See ECS version 3.8 Administration Guide for more details.
console
Simplified bucket ECS 3.8 provides a better deletion option that empties the bucket on your behalf. It runs asynchronously
delete and is available through S3 and management APIs. See ECS Version 3.8 Administration Guide for more
details.
3
Priority task The priority task coordinator is a new backend scheduler that runs tasks by priority and controls how
coordinator many jobs can run in parallel. A new limited management interface is available to monitor tasks such as
those created by the new Simplified bucket delete feature. See ECS version 3.8 Administration Guide for
more details.
Metadata search ECS 3.8 supports the Metadata search acceleration feature, a search query with the logical operation
acceleration AND joining two or more conditions, improving the performance of queries. See ECS Version 3.8
Administration Guide for more details.
Partial listings ECS 3.8 provides an option where a partial listing can be provided instead of waiting for the full set of
listing results.
It helps in limiting timeouts from certain listing operations where possible. See ECS Version Data Access
Guide for more details.
HDFS protocol ECS HDFS is a Hadoop Compatible File System (HCFS) that enables you to run Hadoop 3.1.1 applications
support on top of the ECS storage infrastructure. See ECS version 3.8 Data Access Guide for more details.
Support for new ECS 3.8 supports:
capacity HDDs on ● 16 TB and 20 TB
EX5000
Changed features
This section provides information on features that are changed or deprecated in ECS 3.8.
Dashboard APIs
Learn about the APIs that are changed or deprecated.
4
transactionReadTransactionsPerSec, transactionWriteTransactionsPerSec,
transactionErrors.*
diskReadBandwidthTotal, diskWriteBandwidthTotal, diskReadBandwidthEc,
diskWriteBandwidthEc
diskReadBandwidthCc, diskWriteBandwidthCc, diskReadBandwidthRecovery,
diskWriteBandwidthRecovery
diskReadBandwidthGeo, diskWriteBandwidthGeo, diskReadBandwidthUser
diskWriteBandwidthUser, diskReadBandwidthXor, diskWriteBandwidthXor
Where replacement can be found See Monitoring list of metrics: Non-Performance > Database monitoring_op >
Node system level statistics.
Measurements cpu, mem, net
Where replacement can be found See Monitoring list of metrics: Non-Performance > Database monitoring_vdc.
Measurement cq_node_rebalancing_summary
Where replacement can be found See Monitoring list of metrics: Non-Performance > Database monitoring_last
> Export of configuration framework values.
Measurement dtquery_cmf
Where replacement can be found For VDC metrics, see Monitoring list of metrics: Non-Performance > Database
monitoring_vdc.
For Node metrics, see Monitoring list of metrics: Non-Performance > Database
monitoring_main.
5
Table 2. Alternative places to find removed data (continued)
Data removed
diskReadBandwidthTotal, diskWriteBandwidthTotal,
diskReadBandwidthEc, diskWriteBandwidthEc,
diskReadBandwidthCc, diskWriteBandwidthCc,
diskReadBandwidthRecovery, diskWriteBandwidthRecovery,
diskReadBandwidthGeo, diskWriteBandwidthGeo,
diskReadBandwidthUser, diskWriteBandwidthUser,
diskReadBandwidthXor, diskWriteBandwidthXor
Where replacement can be found For VDC metrics, see Monitoring list of metrics: Non-Performance > Database
monitoring_vdc.
For Node metrics, see Monitoring list of metrics: Non-Performance > Data for ECS
Service I/O Statistics.
Resolved issues
The section lists issues that are resolved in this release:
6
Table 3. Resolved issues (continued)
Issue Description of issues fixed in ECS 3.8.0.3
STORAGE-33987 Improved Temporary Site Outage handling in case of network partition.
STORAGE-33989 Fixed a reset bootstrap issue to skip the journal chunk shipping of the adopted tree.
STORAGE-33995 Fixed an issue where 'Unable to get current IP of node' error might have been returned for Dashboard
API query while node evacuation was in progress.
STORAGE-34030 Fixed an issue with geo task sticking when there were too many unsealed copy failed on copy chunk.
STORAGE-34263 Fixed an issue in AFA environments with separate dump enabled, so that a btree is always re-
initialized in dtsm after blobsvc does a local dump.
STORAGE-34376 Fixed an issue where inconsistent ordering of MI_ENTRYs with INTEGER_TYPE and value null might
have lead to dump failure and caused data to be unavailable.
Table 3. Resolved issues
Issue Description of issues fixed in ECS 3.8.0.2
OBSDEF-24368 Fixed an issue that would cause an error in objectscale-nvmf-engine pods and then restart the pod
after a node down.
OBSDEF-23555 Fixed an issue that would cause GC verification scanner in objectsvc to stop running.
OS-3419 Fixed an issue with netsnmp.Varbind that would cause memory leakage in rackServiceMgr.
STORAGE-33813 Fixed an issue that would prevent directory tables from initializing.
STORAGE-33741 Fixed an issue that would cause listing versions with the "+" symbol in the key name to fail.
STORAGE-33620 Fixed an issue that would prevent directory tables to update its lastRegionKey, resulting in infinite
replays of the first batch of journal replays until the timeout.
STORAGE-33591 Fixed an issue that would cause NVMe engine to fail from creating partition after ECS is redeployed.
STORAGE-33476 Fixed an issue that would allow a namespace root user to view the Manage > Geo Replication view
when no replication groups span multiple VDCs.
STORAGE-33374 Reduced the minimum retention day for vnest recycleBin.
STORAGE-33307 Improved the efficiency in partial GC flow by using the result from DirectoryTableListResult.
STORAGE-33305 Fixed an issue to avoid premature journal chunk reclamation.
STORAGE-33303 Fixed an issue that would prevent vnest service from starting without .out log file.
STORAGE-33302 Fixed an issue that would cause CHUNK_REFERENCE_SCAN_PROGRESS to be deleted to avoid
exceeding the 2.13 MB batch size limit by using separated batch to avoid exceeding the limit.
STORAGE-33300 Fixed an issue that would cause reading failure due to duplicate system metadata.
STORAGE-33298 Fixed an issue that would cause the etag to be returned as an empty string during object listing when
copying an object on to itself.
STORAGE-33297 Fixed an issue that would cause bottleneck in the DJS process as the objects had only dmarkers
versions.
STORAGE-33245 Fixed an issue that would cause the epoll instance of the RDMA server to get stuck causing a crash
due to race condition.
STORAGE-33241 Fixed the connection pool to use the same RDMA connection for request and response to avoid
duplicate connections.
STORAGE-33223 Fix an issue in the recoverLocalChunkFromRemote dtquery API that requires many retires to
recover successfully.
STORAGE-33191 Fixed an issue that would cause BTREE chunks of old trees to be deleted prematurely on AFA.
STORAGE-33124 Fixed an issue that would cause two positive RepoReferences for the same physicalOffset with
different objectids.
7
Table 3. Resolved issues
Issue Description of issues fixed in ECS 3.8.0.1
STORAGE-33148 There is a possibility of data becoming unavailable and unrecoverable if the following conditions occur
simultaneously:
● The shallow copy object is deleted.
● A virtual negative repo response is mistakenly added for the shallow copy object.
● The source object still resides on the affected chunk.
● The journal parser falls far behind the schedule and fails to add the positive repo response for the
source object.
● All other objects on the same chunk are deleted.
Table 3. Resolved issues
Issue Description of issues fixed in ECS 3.8
FABRIC-9613 Set the memory limits for Fabric LM container memory limits to be dynamic, based on cluster node
RAM.
FABRIC-9626 Added support for vfat disk format.
FABRIC-9766 Added support for disk which filters based on slot numbers.
FABRIC-9786 Added a prefix with enclosure path to slot names.
OBSDEF-17191 Fixed an issue where the default response for S3 Bucket Location is a null value.
OS-2516 Fixed an issue where the update Dell script does not support special characters of password for Dell
switch's OS image version below version 10.5.
OS-3288 Increased the file descriptor maximum limit to the kernel default of approximately 100k per 1 GB
physical memory.
OS-3246 Increased the task limit for fabric agent tasks 512–2048 as false alerts were triggered when the tasks
were close to the limit.
OS-3325 Updated journalctl to exclude rpmva file from nlcwatch.
STORAGE-26670 Fixed an issue that replicates a chunk to only one of the remote sites but the replication checker on
the other sites falsely treats this chunk as replicated to other sites as well.
STORAGE-27747 If any nonstandard PSO operations were performed prior to 3.4, contact ECS Customer Support
before upgrading.
STORAGE-28190 Fixed an issue that caused some keys to be not persisted in time leading to the recovery initiated in
preparation for disk replacement to be stuck.
STORAGE-28301 Fixed an issue to control the unexpected growth of .out (ssm, rm, dtquery, and blobsvc) files growing
greater than 30G on some ECS services.
STORAGE-29435 Added a throttler on the geo-replication logic to raise an alert when the source vdc detects the target
vdc capacity is full. The target vdc is also added to the blacklist for 15 minutes and all geo-request to
the target vdc in blacklist is paused.
STORAGE-29716 Fixed an issue that changed the secure JCE configuration on the ECS nodes when upgrading ECS.
STORAGE-29737 Fixed an issue that caused complete but unused MPU parts to cause chunk hole.
STORAGE-29842 Removed the reset operation when retrieving CHECK_JOURNAL tasks to ensure that the old journal
chunks can still be deleted.
STORAGE-30259 Fixed an issue that caused ECS to display negative alerts counter in the UI.
STORAGE-30602 Added a DtQuery API to remove the PARTIAL_GC_CLEANUP_JOBS using the API.
STORAGE-30420 Fixed an issue that caused the exported stats file to not show the tags created for buckets.
STORAGE-31067 Fixed an issue where reconciliation phase of native Centera to ECS migration does not report
mismatched clips.
STORAGE-31079 Fixed an issue that caused btree corruption during btree-gc by triggering reconfig for vnest nodes in
corrupt state.
8
Table 3. Resolved issues (continued)
Issue Description of issues fixed in ECS 3.8
STORAGE-31085 Fixed an issue in data corruption circumstance where frequent requests result in more corruption
reports and an unstable chunk management service.
STORAGE-31095 Added an error logger for vnest btree corruption that is found in vnest btree GC.
STORAGE-31100 Fixed an issue that caused a temporary failure of the remove version operation.
STORAGE-31126 Reduced the read latency of objects that have been updated multiple times and are missing bucketId
system metadata to improve the read performance of such objects.
STORAGE-31189 Fixed an issue where if an object is deleted and re-created on a different zone, and then a TSO of
that zone could cause the object to be not writable during that TSO.
STORAGE-31184 Fixed an issue where DT status Alert is not triggered when btree dump fails repeatedly.
STORAGE-31181 Fixed an issue that caused comma separated values in domain attributes to result in login failure for
AD users.
STORAGE-31205 Fixed an issue where requests to modify a bucket whose replication group is in the failover TSO state
times out and returns an error 500 instead of immediately returning error 403.
STORAGE-31210 Fixed an issue which caused the CAS Space Reclamation sensors to be nonfunctional.
STORAGE-31211 Fixed an issue that caused the dataheadsvc to stop working when disabling unsecure http port for S3.
STORAGE-31214 Fixed an issue where the rejoin tasks does not detect a new TSO and continues to perform
unsuccessfully.
STORAGE-31215 Fixed an issue with /etc/cron.hourly/cleanup_logs to clean up frequently on Gen1 ECS with
root File System.
STORAGE-31262 Fixed an issue where the rebuilding process is stuck at IN PROGRESS, when VDC has several storage
pools.
STORAGE-31264 Fixed an issue that would block metering reconstruction after only one attempt.
STORAGE-31266 Improved the performance of the MIRjoinTaskScanner process during the temporary site outage.
STORAGE-31291 Improved MI rejoin task scanner to retry cleaning the tasks when the task scanner stops.
STORAGE-31293 Fixed an issue that blocks CAS object processing bootstrap after a permanent site outage.
STORAGE-31297 Fixed an issue that causes VNest timeout.
STORAGE-31314 Fixed an issue that caused objects with missing size system metadata to block space reclamation.
STORAGE-31319 Fixed an issue where ECS S3 ListBucket API returns incorrect NextContinuationToken when the path
has Chinese characters.
STORAGE-31326 Increased the timeout to perform CAS objects processing orders.
STORAGE-31348 Improved the performance of rejoin operations during temporary site outages.
STORAGE-31350 Improved the performance and reduced the risks of data unavailability for cases when one object is
requested in parallel during a temporary site outage.
STORAGE-31351 Removed unwanted listing in LSRejoinTaskScanner.
STORAGE-31460 Fixed an issue that caused
com.emc.ecs.metering.reconstruction.reconstruct_metering_data_run_paused to reset the metering
reconstruction.
STORAGE-31517 Fixed an issue to prevent fall back to previous ownership ranges when reading null version.
STORAGE-31523 Improved the performance of multiple object deletion operations from the S3 API.
STORAGE-31560 Fixed an issue that would block the data migration due to the legacy ip format device id, when 3.1 or
earlier is installed.
STORAGE-31621 Fixed an issue where the ECS S3 operations would fail when the request is encoded with URL safe
characters.
9
Table 3. Resolved issues (continued)
Issue Description of issues fixed in ECS 3.8
STORAGE-31650 Added code to verify whether a bucket exists while processing creates a bucket PUT request.
STORAGE-31655 Fixed an issue to set a threshold to let L2 borrow space from L1, when L2 capacity is full.
STORAGE-31682 Fixed an issue when the MGMT API POST method fails with an NPE for deleted namespaces or invalid
namespaces.
STORAGE-31691 Changed the ECS interface(datahead) service GC algorithm to G1GC for non-AFA cluster.
STORAGE-31721 Fixed an issue where you are unable to write data onto a bucket and namespace.
STORAGE-31747 Fixed an issue that caused the response of a partial ECS S3 object that is read to exclude the HTTP
reason phase.
STORAGE-31774 Fixed an issue with the stage Atmos listable tasks that occupy the background thread pool and block
background operations.
STORAGE-31792 Fixed an issue where nussubg replication of page table journal chunks occurs due to sharing of the
same PAGE_TABLE_JOURNAL_REGION_BOOTSTRAP_JOURNAL_MARKER.
STORAGE-31857 Fixed an issue where expired entries would appear in listing object or version responses when a
bucket's life-cycle rule contained an object name prefix filter.
STORAGE-31899 Fixed an issue that caused the local dump to get blocked when the remote journal region uses the
chunked sealed length greater than the journal region sealed length, causing an infinite loop.
STORAGE-31915 Fixed an issue where the ECS storage class field was not returning a list bucket response.
STORAGE-31920 Added a parameter to skip the SOURCE_PENDING_RECLAIMATION stage to speed up partial GC.
STORAGE-31934 Increased the default value of
com.emc.ecs.chunk.suicide_prevention.threshold.merge_chunk_for_partial_gc to fix the issue of
clusters becoming read-only.
STORAGE-31941 Fixed an issue where Tech Refresh: Data Migration would still show data to migrate after completion.
STORAGE-31987 Resolved an issue where spill-over repo references cannot be counteracted and corresponding chunks
cannot be reclaimed in a multizone federation.
STORAGE-32022 Fixed an issue to handle insert storage tiers to VNest for two storage pool.
STORAGE-32048 Fixed an issue that caused memory pressure issue when copying and reading large objects.
STORAGE-32056 Removed partsCount from non-MPU S3 response.
STORAGE-32095 Fixed an issue where complete MPU could fail in a bucket with versioning set to OFF with a
noncurrent version expiration life-cycle rule.
STORAGE-32099 Removed certain operations that were blocking the fiber dump to increase the dump performance.
STORAGE-32128 Fixed an issue that prevents services in the unupgraded node to receiving response from services
under upgraded nodes, if the services in the unupgraded nodes are restarted.
STORAGE-32133 Fixed an issue to prevent SSDr performance from reducing when changing the ownership of multiple
DTs.
STORAGE-32179 Fixed an issue where the ECS S3 get bucket life-cycle configuration API error response did not
include the correct message.
STORAGE-32225 Fixed an issue to prevent S3 Read object request to succeed with management auth token x-sds-
auth-token.
STORAGE-32259 Fixed an issue that caused the ECS S3 OPTIONS to stop working.
STORAGE-32339 Fixed an issue when domain user belonging to the group with capital letters cannot log in.
STORAGE-32362 Added an exception handling for addMaxMinorRemoteRegionKV to prevent missing minor issue.
STORAGE-32408 Fixed an issue that prevented adding or removing of Trusted Certificates in ECS version 3.8.
STORAGE-32416 Fixed an issue that caused MGMT API to fail with JSON payload.
10
Table 3. Resolved issues (continued)
Issue Description of issues fixed in ECS 3.8
STORAGE-32440 The frequency of heartbeat for Non-Vnest members is reduced.
STORAGE-32441 Fixed an issue that helps reduce false-positive BlobsvcMemoryCacheSize alerts.
STORAGE-32472 Fixed the ATMOS API to accept an expiration date of 25 days or more.
STORAGE-32494 Fixed an issue that prevents ECS IAM to detach the group policy from the system-managed policy.
STORAGE-32537 Fixed an issue that caused an error when calling AWS API listNexBatchOfVersions on a nonexistent
list.
STORAGE-32748 Fixed an issue where the DT ownership changes from node to node when the background thread pool
is occupied by MU cleaning tasks.
STORAGE-32857 Fixed an issue that blocks page table journal GC when trying to get RG zones.
STORAGE-32928 Fixed an issue where no result is returned when a user creates a bucket with a metadata search index
containing one of the invalid characters (@, (), =,<>).
STORAGE-31123 Prevented an error that would prevent the directory table (DT) from serving for up to two hours
when a rare temporary issue occurs.
STORAGE-33021 Fixed partial upgrade issues with vdc1 (3.7.0.3) and vdc2 (>= 3.7.0.3), when vdc2 is the bucket owner
(B1) and the bucket has CAS or metadata search enabled, which fails internally to make remote calls
to the remote zone (VDC 2) to insert MI entries.
STORAGE-32995 Fixed an issue which is related to server firmware when DTs are not ready after MM exit operation.
Known issues
The section lists the known issues for this release of ECS.
11
Table 4. Known issues (continued)
Issue Description Workaround
STORAGE-32198 The ownership transfer request that is sent by DT Identify unavailable keys from the journal and
gets timed out when processed by VNest. This reinsert them into DT.
timeout resulted in dual owners of DT and some keys
in DT became unavailable.
Contact OE or DEV for any data unavailable issues.
STORAGE-31984 When a socket connection is initiated, the counter Restart the service.
party is closed simultaneously, resulting in socket
closure and communication request timeout.
STORAGE-31318 When a user's and the group policy name are the Create a policy with a name different from the
same, the S3 requests fail. user with the same permissions as the first
policy. Attach the newly created policy to the
user. Remove the user from the first policy that
has the same name.
Optional: Validate if the first policy is attached
to any other user. If no other user is attached,
delete the policy.
STORAGE-31292 When the root directory of the FS bucket and the KB article provides more information.
bucket for ACL are not in synchronization, the owner
changes are not reflected in the root directory ACL.
STORAGE-31219 System performance may degrade during excessive Contact ECS Remote Support for assistance.
bucket listing operations.
STORAGE-31115 Writes to read-only buckets may occupy space in the -
system until the garbage is collected.
STORAGE-30396 After ECS 3.6, dashboard page does not function if Enable monitoring containers to restore the
monitoring containers are disabled. dashboard page.
STORAGE-30617 When non-ASCII characters are not displayed in the Add- Dfile.encoding=UTF-8 to objcontrolsvc
UI, it cannot be deleted.
STORAGE-30015 Journal SR is blocked due to missing MJPM. Disable blob side metering.
STORAGE-29786 In a geo-replicated setup, if all segments or copies Follow the manual procedure that OE provide to
are corrupted, follow the manual procedure that OE fix the chunk.
provide to fix the chunk.
STORAGE-29814 ECS does not handle NFS health check messages in Disable the health check in the F5 load balancer.
plain text, which causes a memory leak leading to an
unresponsive system.
STORAGE-27509 When a metadata modification of an object happens OE has tools to recover correct DEK.
between two creations of the same object, the DEK
is stored in the system metadata, resulting in data
unavailability of the object.
STORAGE-27007 If the L2 capacity is full, L1 DT might not be able to Contact ECS Remote Support for assistance.
initialize which cause DU.
STORAGE-26938 If RPO (geo-replication) is not updated before zone Use the DTQuery API to recover the partial GC
PSO, partial GC tasks can be blocked. task.
STORAGE-26049 During the TSO, if the bucket has many updates in Contact ECS Remote Support for assistance.
a short time, CRUD operations on objects fail for the
specific bucket.
STORAGE-31565 MIGRATION_STATUS is stuck at Restart Storage Server Manager(SSM).
DATA_MIGRATION_ONGOING when returning to
data migration.
12
Table 4. Known issues (continued)
Issue Description Workaround
STORAGE-33044 In single-zone RG, if one node is down, there could be When inserting a new zone into RG, ensure that
failure in inserting a new zone into the RG. all nodes are up and running.
STORAGE-31582 When ECS STORAGE_TIER_UNAVAILABLE_INFO Contact ECS Remote Support for assistance.
(13007) is observed repeatedly.
OS-2957 Stale replication secondary interfaces require cleanup. Contact ECS Remote Support for assistance.
OS-2500 When the drive stops responding and causes NICs to Reboot node
be inaccessible.
FABRIC-9805 When EX5000 was released with slot 2 Pravega Perform an OE workaround for ECS versions less
drives, it could not be used in ECS classic, but fabric than 3.8 before upgrading, extending new nodes,
is incorrectly allocating SSDr to the pravega disk in and node replacements.
slot 2 rather than the SSDr disk in slot 1.
STORAGE-33069 In a multisite setup, if a user upgrades only one site Upgrade all the sites and add a new EKM.
and tries to add a new EKM of type Thales Cipher
Trust, the EKM cannot be activated on a partially
upgraded site.
STORAGE-32094 On long-running systems, Java GC logs begin to This is expected behavior and can be overlooked.
consume up to 25 GB of root drive space.
FABRIC-9811 When DTs are in an unready state, the alert is not Logs contain message that is related to Secure
sent to the Secure Remote Services server. It is sent Remote Services which can be documented.
to other alert. New alert type added to ECS 3.8
channels such as UI, SNMP, and syslog.
ECS Software
ECS Software is a software-only solution for users seeking to deploy on an ECS appliance.
To install ECS Software on custom hardware, see https://www.dell.com/support/home/en-us/product-support/product/ecs-
appliance-software-without-encryption/drivers.
13
Minimum requirements for triggering Permanent Site Outage, Temporary Site
Outage, and Remove VDC from a Replication Group
For the minimum ECS code requirements to run the operations Permanent Site Outage (PSO), Temporary Site Outage (TSO),
and Remove a VDC from a Replication Group, see https://www.dell.com/support/kbdoc/en-us/000180322.
Gen 2 - U-series with 8 TB or U-series DAE drawers hold up One 1 GbE switch (Turtle) Two 10 GbE switches (Hare
12 TB disks to 60 3.5-inch disk drives. ● Arista 7010T-48 and Rabbit)
● Arista 7048T ● Arista 7050SX-64
● Cisco 3048 48-P or
This switch is available when ● Arista 7050S-52
customers supply their own
14
Table 6. ECS 3.8 and later hardware compatibility matrix (continued)
Servers Disk Array Enclosures Switches
(DAEs)
Private Public
public Cisco switches through
an RPQ.
Gen 2 - D-series with 8 TB or D-series DAE drawers hold up One 1 GbE switch (Turtle) Two 10 GbE Arista
10 TB disks to 98 3.5-inch disk drives. ● Arista 7010T-48 7050SX-64 switches (Hare
● Cisco 3048 48-P and Rabbit)
This switch is available when
customers supply their own
public Cisco switches through
an RPQ.
Gen 2 - C-series with 8 TB N/A One or two 1 GbE switches Two or four 10 GbE switches
disks (Turtle). The second switch (Hare and Rabbit). The third
is required for configurations and fourth switches are
with more than six servers. required for configurations
● Arista 7010T-48 with more than six servers.
● Arista 7048T ● Arista 7050SX-64
or
● Arista 7050S-52
Gen 1 - U-series with 6 TB U-series DAE drawers hold up One 1 GbE switch (Turtle) Two 10 GbE switches (Hare
disks to 60 3.5-inch disk drives ● Arista 7010T-48 and Rabbit)
● Arista 7048T ● Arista 7150S-24
or
● Arista 7124SX
Gen 1 - C-series with 6 TB N/A One or two 1 GbE switches Two or four 10 GbE switches
disks (Turtle). The second switch (Hare and Rabbit). The third
is required for configurations and fourth switches are
with more than six servers. required for configurations
● Arista 7010T-48 with more than six servers.
● Arista 7048T ● Arista 7050SX-64
or
● Arista 7050S-52
NOTE: SSH connection to Rack1 Node 1 and Node 2 is required for all access methods.
Service Console and xDoctor tools are used in the upgrade procedures.
15
Points to remember during upgrade
During the rolling upgrade, expect I/O timeouts on the upgraded node. When a node that is being upgraded enters Maintenance
Mode, all the other nodes in the cluster experience I/O timeout for approximately 30 s–90 s. In GEO Virtual Data Centers (VDC)
configuration I/O interruption is mitigated by redirecting traffic to the other VDC.
The total upgrade time depends on factors such as:
● Total number of VDCs: Upgrade is not complete until all VDCs are upgraded. If multiple upgrade steps are required, each
VDC needs to be upgraded to the interim version before proceeding to the target version.
● Number of Nodes in a cluster: There is varying increase in time with the greater number of nodes depending on the type of
the upgrade.
● ECS source and target versions: If direct upgrade to the target version is not supported, the total time increases due to an
upgrade to an interim version.
● Type of the upgrade used: ECS operating system together with ECS Services upgrade takes longer than ECS Service only
upgrade as reboot is not required for patch upgrade.
Time estimate per upgrade type:
● Rolling Upgrade takes around 1 hour x number of nodes in the VDC x number of VDC.
● ECS Service Only upgrade takes around 0.5 hour x number of nodes in the VDC x number of VDC.
For nonencrypted: ECS ecs-3.8.0.3-8416.5a924dd1- Nonencrypted ECS installation files for full
3.8.0.3 Fabric and production-no-encryption.tgz installation of new systems. Available for
Object Installation installations in China and Russia.
Operating ECS - operating system ecs-os- ECS operating system installation files.
Environment Setup 3.8.0.3 setup.x86_64-3.8.0.3-2206.44cc40a.
62.install.iso
Update ECS - OS Update 3.8.0.3 ecs-os- ECS operating system update files.
update-3.8.0.3-2206.44cc40a.62.zip
Add-Ons GeoDrive 2.2.3 (64-bit) GeoDrive.2.2.P3_x64.exe Local GeoDrive file system installation files.
GeoDrive-Remote 2.0 GeoDrive-Remote.2.0_x64.exe Remote GeoDrive file system installation
(64-bit) files.
ECS HDFS Client 3.8.0.3 hdfsclient-3.8.0.3.138685.3a0a9b6.zi HDFS for ECS installation files. Cloudera
p Data Platform is the only Hadoop version
supported for HDFS.
ECS HDFS Client Library hdfsclientrpm-3.8.0.3.138685.3a0a9b HDFS for ECS installation files for Linux.
RPM 3.8.0.3 6.3.1.rpm
16
Table 7. ECS files for installation, upgrade, and support (continued)
Category Title File name for ECS 3.8.0.3 Contains
Veritas Enterprise Vault ECSStreamer.2.0.6.21.exe Veritas Enterprise Vault (EV) Streamer
Streamer Driver for ECS Driver for ECS installation files.
2.0.6.21
ECS - MIB ECS-MIB-v2.mib ECS provides support for monitoring SNMP
servers during installation and SNMP traps
during the ECS Fabric life cycle. SNMP
support requires that you download the
latest ECS MIB file.
Product Tool Service Console service-console-<version>.tar The Service Console is periodically updated.
- service-console- Download the latest version of the Service
<version> Console when installing or upgrading the
ECS Software. Service Console files are
located in the Production Tool folder
that is associated with the ECS Software
version you are installing.
Container versions
The following container versions are installed with both the ECS 3.8.0.3 encrypted and non-encrypted installation files.
package: 3.8.0.3-8416.5a924dd1
fabric: 3.8.0.3-4354.47090e4
jdk: 11.0.19
fabric-registry: 3.8.0.3-88.dd9eaaf
fabric-zookeeper: 3.8.0.3-125.63fdda3
object: 3.8.0.3-138685.3a0a9b6bf3a
hal: 3.8.0.3-1993.660bfed
utilities: 3.8.0.3-1171.d96416d
emc-lsi-storelibir-2: 17.01-657.1b62e78.1
mhash: 0.9.9.9-38.22
storcli: 1.15.05-1
object-telegraf: 3.8.0.2-1551.f3ade7de
object-influxdb: 3.8.0.2-1551.f3ade7de
object-fluxd: 3.8.0.2-1551.f3ade7de
object-throttler: 3.8.0.2-1551.f3ade7de
object-grafana: 3.8.0.2-1551.f3ade7de
vdckeyupgrade.tar: 3.6.0.0-123247.6b4b7d43b53
After installation, you can see the versions and status of the containers for all nodes in the current rack by running the following
command:
17
See Software Installation Guide for EX-Series (EX300/EX500/EX3000/EX5000) Hardware, Software Installation Guide for D
and U series (Gen 1 and Gen 2) Hardware, and Software Installation Guide for EXF900 Hardware for installation details.
3.6.0.1 No No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
3.6.1.1 No No No Yes Yes Yes Yes Yes Yes Yes Yes Yes
3.6.1.2 No No No No Yes Yes Yes Yes Yes Yes Yes Yes
3.6.1.3 No No No No No Yes Yes Yes Yes Yes Yes Yes
3.6.1 No No No No No No Yes Yes Yes Yes Yes Yes
3.6.2 No No No No No No No Yes Yes Yes Yes Yes
3.6.2.1 No No No No No No No No Yes Yes Yes Yes
3.6.2.2 No No No No No No No No No Yes Yes Yes
3.6.2.3 No No No No No No No No No No Yes Yes
3.6.2.4 No No No No No No No No No No No Yes
3.6.2.5 No No No No No No No No No No No No
3.7 No No No No No No No No No No No No
3.7.0.1 No No No No No No No No No No No No
3.7.0.2 No No No No No No No No No No No No
18
Table 8. Supported upgrade paths from 3.6 to 3.8.0.3 (continued)
ECS To 3.6 3.6.0.1 3.6.1.1 3.6.1.2 3.6.1.3 3.6.1 3.6.2 3.6.2.1 3.6.2.2 3.6.2.3 3.6.2.4 3.6.2.5
Version
3.7.0.3 No No No No No No No No No No No No
3.7.0.4 No No No No No No No No No No No No
3.7.0.5 No No No No No No No No No No No No
3.7.0.6 No No No No No No No No No No No No
3.8.0.1 No No No No No No No No No No No No
3.8.0.2 No No No No No No No No No No No No
3.8.0.3 No No No No No No No No No No No No
19
Table 9. Supported upgrade paths from 3.6 to 3.8.0.3 (continued)
ECS To 3.7 3.7.0.1 3.7.0.2 3.7.0.3 3.7.0.4 3.7.0.5 3 3.8.0.1 3.8.0.2 3.8.0.3
Version .
7
.
0
.
6
3.6.2.5 No No No Yes Yes Yes Y Yes Yes Yes
e
s
3.7 No Yes*** Yes*** Yes*** Yes*** Yes*** Y Yes Yes Yes
e
s
*
*
*
3.7.0.1 No No Yes*** Yes*** Yes*** Yes*** Y Yes Yes Yes
* * * * e
s
*
*
*
*
3.7.0.2 No No No Yes Yes Yes Y Yes Yes Yes
e
s
3.7.0.3 No No No No Yes Yes Y Yes Yes Yes
e
s
3.7.0.4 No No No No No Yes Y Yes Yes Yes
e
s
3.7.0.5 No No No No No No Y No Yes Yes
e
s
3.7.0.6 No No No No No No N Yes Yes Yes
o
3.8.0.1 No No No No No No N No Yes Yes
o
3.8.0.2 No No No No No No N No No Yes
o
3.8.0.3 No No No No No No N No No No
o
* Per STORAGE-31675, ECS 3.6 cannot be upgraded to version 3.7.0.0 or 3.7.0.1 when the CMF parameter is changed. This is
expected behavior. You must upgrade directly from 3.6 to 3.8.0.3.
** Upgrades from 3.6.2.3 to either 3.7.0.0 or 3.7.0.1 are not supported.
*** For upgrades from either 3.7 or 3.7.0.x to 3.7.0.5, if you are not upgrading the operating system, you can use the patch
upgrade procedure.
**** For upgrades from 3.7.0.x to 3.7.0.4, upgrade the operating system, then use the OS Quick Fit procedure, then the patch
procedures.
***** Per SCONSOLE - 3602, ECS 3.6.2.4 cannot be upgraded to version 3.7.0.0, 3.7.0.1, or 3.7.0.2.
20
Service Console
If upgrading from a 3.0.x version of ECS, the Service Console can be used to perform health checks and upgrade however, the
Service Console will not collect logs prior to 3.1.
NOTE: The upgrade procedure requires SC 8.0.0.0 and above. We recommend using the latest available version of Service
Console.
21
Understanding when to run the different ECS health checks
The following table describes the three required health checks.
xDoctor Collects system-level and node-level xDoctor health checks can be activated and
diagnostics with data analysis. run when ECS software is installed. xDoctor
supports scheduled automated execution of
predefined diagnostics and manual diagnostics
execution based on the support activity such as,
upgrade, pre, and post problem resolution and
general health validation.
Product information
For documentation, release notes, software updates, or for information about ECS, licensing, and services, go to the online
support website (registration required) at:
www.dell.com/support
Technical support
For technical support, go to Customer Service on the Online Support site. To open a service request through Online Support,
you must have a valid support agreement. Contact your ECS sales representative for details about obtaining a valid support
agreement or to answer any questions about your account.
22
2. Type the product name in the Find a Product field and click the arrow button. The Service Life information, if available for
a given product, is displayed on the left side of the associated product page, and states when the product was Generally
Available (GA), when it is out of Service, and if it has Extended Service coverage.
3. Click the Release and End of Life Dates link available on the Service Life information section to view existing Service Life
information documents. The Dell EMC Software Release and End of Service Life Notifications document provides (End of
Standard Support) EOSS dates for Dell EMC software products.
For information about how to use Dell Online Support, click the Online Support FAQ link available at: https://www.dell.com/
support/.
Documentation
Customer documentation can be accessed from www.dell.com/support, which is available to registered customers, and also
includes access to software downloads, service requests, support knowledge base, and other Customer Support tools.
Internal resources are provided for ECS customer support representatives and partners through SolVe.
The following documents are updated or added in ECS 3.8 release:
23
Table 11. ECS 3.8 documentation list
Documents to support EX5000, EXF900, Documents not tied to hardware Other documents
EX300, EX500, and EX3000 hardware release
(internal resource)
● ECS EX5000D Network Daughter Card
Replacement Guide
● ECS EX5000S System Board Replacement
Guide
● ECS EX5000D System Board Replacement
Guide
● ECS EX5000S Sled Replacement Guide
● ECS EX5000D Sled Replacement Guide
● Dell EMC ECS EX5000 Expander Module
and HBA Card Replacement Guide
● ECS EX5000D Cable M2 Riser Board
Replacement Guide
● ECS EX5000S Cable M2 Riser Board
Replacement Guide
● ECS EX5000 Power Supply Replacement
Guide
● ECS EX5000S Expansion Card
Replacement Guide
● ECS EX5000D Expansion Card
Replacement Guide
● ECS EX5000S Full Height, Full Length
Interposer Board Replacement Guide
● ECS EX5000S Full Height Riser Board
Replacement Guide
● ECS EX5000D System Battery
Replacement Guide
● ECS EX5000S System Battery
Replacement Guide
● Dell EMC ECS S5248F-ON Switch
Replacement Guide
● Dell S5248F-ON Switch Power Switch
Unit Replacement
● Dell S5248F-ON Switch Fan Replacement
Guide
● ECS EX5000 General Field Replacement
Guide
● ECS EX5000D Motherboard Bridge
Module Replacement Guide
● ECS EX5000S Motherboard Bridge
Module Replacement Guide
● ECS EX5000S Sled Cables Kit
Replacement Guide
● ECS EX5000D Sled Cables Kit
Replacement Guide
● ECS EX5000S Trusted Platform Module
Replacement Guide
● ECS EX5000D Trusted Platform Module
Replacement Guide
● Dell EMC ECS EX5000 Backplanes
Replacement Guide for the Dell
EX7100Chassis
● ECS 3.7 Software Installation for EXF900
● ECS EXF900 Installation Guide
(Hardware).
● ECS EXF900 Third Party Rack Guide.
24
Table 11. ECS 3.8 documentation list
Documents to support EX5000, EXF900, Documents not tied to hardware Other documents
EX300, EX500, and EX3000 hardware release
(internal resource)
● ECS EXF900 Control Panel Replacement
● ECS EXF900 Disk Addition Guide
● ECS EXF900 Backplane Replacement
Guide
● ECS EXF900 DIMM Replacement Guide
● ECS EXF900 Battery Replacement Guide
● ECS EXF900 Node Replacement Guide
● ECS EXF900 PSU Replacement Guide
● ECS EXF900 M2 OS Replacement Guide
● ECS EXF900 Fans Replacement Guide
● ECS EXF900 BOSS Card Replacement
Guide
● ECS 3.7 Software Installation EX Series.
● ECS Dell S5148F-ON S5248F-ON S5232F-
ON Switch PSU Replacement Guide
● ECS Dell S5148F-ON S5248F-ON S5232F-
ON Switch Fan Replacement Guide
Document feedback
If you have any feedback or suggestions regarding this document, mailto:ecs.docfeedback@dell.com.
25
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the
problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 2023 Dell Inc. or its subsidiaries. All rights reserved. Dell Technologies, Dell, and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners.