906278712.doc (650.

00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Node Replacement Guide

Isilon
994-0015-01 Rev K

Replace a node

About this guide

This guide provides instructions on how to replace a failed node. If a node is unresponsive and cannot be
repaired by replacing individual components, a replacement node is required.
This guide includes the procedures for:
 Shutting down the failed node.
 Transferring drives, DIMMs, boot drives, and PCIe cards from the failed node to the replacement
chassis.
 Installing the replacement node.
 Verifying the successful installation of the new node.

Replacing a node
You can replace a failed node using a replacement chassis. This procedure allows you to avoid
smartfailing the node, while maintaining data integrity.
Contact Isilon Technical Support to obtain a replacement chassis before beginning this procedure.
Be aware that there may be down time associated with this procedure. Discuss the timing and duration of
the down time with your field engineer, and schedule accordingly.
This procedure requires a work area large enough to place two Isilon nodes side by side.

CAUTION: Your work area must also have an outlet available to charge the replacement node.
Adequate power must be available to the IB/NVRAM card throughout this procedure. When a node
is unplugged from power, the IB/NVRAM card is relying on the NVRAM batteries for power.
Because of that, it is important to plug the replacement node in to power immediately and allow
the batteries to charge for a minimum of 30 minutes.

Working with clusters in SmartLock compliance mode

Clusters running in SmartLock compliance mode require a sudo prefix to run root commands.
If a cluster is running in SmartLock compliance mode, root access is disabled on the cluster. Because of
this, you can run some commands only through the sudo program. Prefixing a command with sudo
enables you to run commands that require root access. For example, if you do not have root access, the
following command fails:

1
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

isi drivefirmware status

However, if you are on the sudoers list, the following command succeeds:
sudo isi drivefirmware status
Compliance mode commands that require changes beyond the sudo prefix are noted in the procedure
steps.
For more information on the sudo program and compliance mode commands, see the OneFS CLI
Administration Guide.

Task 1: Gather logs

Before you begin any maintenance on a cluster, gather cluster logs.
About this task
You must collect cluster logs before all maintenance procedures. Cluster logs provide snapshots of the
cluster, which you can review to make sure that maintenance is successful.
Procedure
1. [ ] Open a secure shell (SSH) connection to any node in the cluster and log in.
2. [ ] Gather cluster logs by running the following command:
isi_gather_info

Task 2: Download a Field Replacement Unit (FRU) package

Before you replace a component in a configure-to-order (CTO) node, obtain a Field Replacement Unit
(FRU) package from the EMC FTP site. The FRU package updates the CTO and as-built information on
the node, then forwards the updated information to Isilon Technical Support.
About this task
Procedure
1. [ ] Download the latest FRU package from ftp://ftp.emc.com/outgoing/Fru_Package/.
2. [ ] Note the name of the FRU package. You will use the name for other commands.
Package names follow this convention:
IsiFru_Package_ <date-time-stamp> .tgz
For example: IsiFru_Package_201507072125.tgz
3. [ ] Place the FRU package on the cluster through a network drop, or by asking someone at
the cluster site to place the package for you. If neither of these options is available to you, contact
Isilon Technical Support for assistance.

Replace the failed node

Perform the following tasks in the order presented to replace the failed node.

Task 3: Unpack the replacement node

Upon arrival from Isilon, replacement nodes must be unpacked.
About this task

2
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

CAUTION: To avoid personal injury or damage to equipment, always use two people to lift and
move nodes.

Procedure
1. [ ] Remove the replacement node from the shipping package and inspect it for any sign of
damage. Notify EMC Isilon Technical Support if the node appears damaged in any way. Do not install
a damaged node.
Do not discard the shipping container and packaging. You will use the container and packaging to
return the failed node to Isilon.
2. [ ] Connect the replacement node to power.
3. [ ] Power on the replacement node.
4. [ ] Before proceeding, allow the node to charge with power on for 30 minutes. Charging the
node in this way will ensure that the batteries have enough charge to provide adequate power to the
IB/NVRAM card.

CAUTION: Before shutting down the replacement node after thirty minutes, look at the
batteries on the back panel. If either battery is showing red after charging, it has failed and
must be removed from the node and set aside. If both batteries are showing red, contact Isilon
Technical Support.

Task 4: Power down the failed node

You must power down the failed node before you disconnect it.
About this task

CAUTION: Before powering down the node, look at the back panel and confirm that the LED on
both batteries are showing green. If either battery is showing red, it has failed and must be
removed from the node and set aside. If both batteries are showing red, contact Isilon Product
Support.

Procedure
1. [ ] Using a serial cable, or a network drop provided by the customer, connect to a node in
the cluster that is not the node you are replacing.
2. [ ] From that node, open a secure shell (SSH) connection to the node that you want to shut
down. Run the following command:
ssh <cluster name>-<node number>
3. [ ] Shut down the node. Run the following command:
shutdown -p now

3
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

If the node does not respond to the shutdown command, press the Power button on the failed node
three times, and then wait five minutes. If the node still does not shut down, you are at risk for losing
data. Do not proceed. Contact EMC Isilon Technical support for assistance.

CAUTION: A forced power down should be attempted only if the failed node is unresponsive.
Forcing the power down of a healthy node can result in data loss.

4. [ ] Verify that the failed node has been taken down. Run the following command:
isi status
Confirm that the node has a D (Down) status. See the DASR column for node 3 in the example
below.
ID |IP Address |DASR| In Out Total| Used / Size |Used / Size
---+---------------+----+-----+-----+-----+------------------+-----------------
1|10.53.217.201 | OK | 48M| 0| 48M| 19G/ 6.2T(< 1%)|(No SSDs)
2|10.53.217.202 | OK | 46M| 0| 46M| 23G/ 6.2T(< 1%)|(No SSDs)
3|10.53.217.203 |D---| n/a| n/a| n/a| n/a/ n/a( n/a)|n/a/n/a( n/a)
Do not attempt any hardware operations until the power down process is complete. The process is
complete when the node LEDs are no longer lit.

Task 5: Disconnect cables from the failed node

You must disconnect all cables before a node can be removed from a rack.
Procedure
1. [ ] Label InfiniBand and ethernet cables to ensure that they are plugged into the replacement
node correctly.
2. [ ] Unplug all cables from the back of the node, including power.

CAUTION: Once a node has been disconnected from power, the IB/NVRAM card is supplied
power by the NVRAM battery. It is vital that the IB/NVRAM card is attached to a battery while
moved to a replacement node, and the replacement node is plugged back in to power, before
the battery expires (approximately 30 minutes).

Note: If there are transceivers connected to the end of your IB or ethernet cables, make sure to
remove them with the cables. If you are using fibre ethernet cables, you will need to disconnect the
cable from the transceiver, then remove the transceiver from the node.

Task 6: Remove drives from the failed node

Remove all drives from the failed node before moving it.
About this task
Note: Be sure to note the bay number from which each drive is removed. The drive must be transferred
to that same bay number in the replacement node. As they are removed, drives can be labeled or stacked
in order to ensure that they are placed into the correct by in the replacement node.

4
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Figure 1 Front drive bays

Figure 2 Back drive bays

Procedure
1. [ ] To access drives at the front of the node, remove the front panel.
There are two release buttons on either side of the front panel. To remove the panel, press in the
panel release buttons while pulling the panel away from the node.

1. Front panel

2. [ ] To remove a drive, pull the locking handle on the drive toward you.
The drive will release from the node.

5
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

1. Locking handle 2. Drive bay

3. [ ] Repeat step two until all drives are removed from the front of the node.
4. [ ] To access the drives in the back of the node, remove the rear EMI shield.
There are two vertical handles at either end of the shield. To remove the shield, press those handles
toward the center of the shield while pulling away from the node.

1. Rear EMI shield

5. [ ] Repeat step two until all drives are removed from the back of the node.

6
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Task 7: Remove the failed node from the rack cabinet

Move the failed node to a clear work space next to the replacement node.
Before you begin
At this point, the node should be shut down, with all cables disconnected from the back panel and all hard
drives removed.
Procedure
1. [ ] Remove the retaining screws that secure the node to the rack cabinet.
2. [ ] Slide the node out from the rack cabinet to fully extend the slide rails and provide clear
access to the node.

DANGER: Slide the node out from the rack slowly. Do not extend the rails completely until
you confirm that the node is latched and safely secured to the rails.

3. [ ] Press the release latch on the inner slide rails and slide the node forward to remove the
node from the outer slide rails. The inner slide rails remain attached to the node. Do not remove the
inner slide rails.

CAUTION: To minimize the chance of personal injury, Isilon recommends using two people to
lift and move the node.

7
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

1. Inner slide rail 2. Outer slide rail

Task 8: Disconnect power from the replacement node

Prior to working with the replacement node, check to ensure the batteries are charged, then disconnect
the node from power.
About this task
At this point, the replacement node should have been connected to power and charging for at least thirty
minutes.

CAUTION: Before you unplug the replacement node, look at the batteries on the back panel. If
either battery is showing red after charging, it has failed and must be removed from the node and
set aside. If both batteries are showing red, contact Isilon Technical Support.

Procedure
1. [ ] If neither battery is showing red, unplug the power cables from the back of the
replacement node.

Task 9: Remove the top panels from the failed and replacement nodes
In order to transfer components to the replacement node, the top panel of both nodes must be removed.
About this task
If either the replacement or failed node is connected to power, disconnect the power cords at this time.

8
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

If you are not already grounded, properly ground yourself to prevent electrostatic discharge from
damaging the node components (for example, use an ESD wrist strap, attached to the node chassis for
grounding).
Procedure
1. [ ] Loosen the captive screw securing the top panel of the failed node.
2. [ ] Slide the top panel toward the rear of the failed node, and lift the top panel off to access
the node interior.
3. [ ] Repeat steps one and two to remove the top panel from the replacement node.

Task 10: Remove the cross bracket from the failed and replacement nodes
To provide clear access to the inside of the failed and replacement nodes, remove the cross bracket from
each.
Procedure
1. [ ] Locate the cross bracket within the failed node.

1. Cross bracket

9
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

2. [ ] To remove the cross bracket, press in on the side of the node chassis where the cross
bracket is connected. Unhook the cross bracket from the chassis and lift that end straight up until you
can unhook the other side of the bracket.
3. [ ] Repeat steps one and two to remove the cross bracket from the replacement node.

Task 11: Remove the air baffles from the failed and replacement nodes
In order to gain full access to internal components of both the failed and replacement nodes, remove the
air baffles from each.
Procedure
1. [ ] Locate the air baffle within the failed node.

1. Air baffle

2. [ ] Lift the air baffle straight up out of the failed node.

3. [ ] Repeat steps one and two to remove the air baffle from the replacement node.

Task 12: Transfer the memory from the failed node to the replacement node
All DIMMS must be removed from the failed node and installed in the replacement node.
Before you begin

10
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Make note of the slots from which the DIMMs are removed and install them into the same slots in the
replacement node.

CAUTION: Always follow electrostatic discharge (ESD) prevention procedures when removing
and replacing hardware components. Use a wrist strap, attached to the node chassis, for
grounding.

Procedure
1. [ ] Locate a DIMM module to transfer.
Make note of the slot from which the DIMM is removed. Transfer it into the same slot in the
destination node.
2. [ ] Press down on the two DIMM locking arms on either side of the DIMM to release the
DIMM from the slot.
3. [ ] Gently lift the DIMM out of the slot and remove the DIMM from the failed node.

1. DIMM

4. [ ] Locate the corresponding memory slot in the replacement node. Align the notch in the
DIMM with the tab of the open slot and gently press down on both ends of the DIMM until the two
arms lock into place and secure the DIMM to the motherboard.
5. [ ] Repeat this process until all DIMMs are transferred into the replacement node.

11
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Task 13: Transfer the NIC from the failed node to the replacement node
The Network Interface Card (NIC) must be removed from the failed node and installed in the replacement
node.
About this task
Make note of the PCIe slot from which the NIC is removed and install it into the same slot in the
replacement node.
If the replacement node arrived with PCIe cards installed, remove those cards and install them in the
failed node for return to Isilon.
Procedure
1. [ ] Remove the mounting screw securing the NIC to the node.
2. [ ] Remove the NIC from the node.

1. Mounting screw 2. NIC

3. [ ] If necessary, remove the mounting screw and metal guard from the bay where you will be
placing the card.

12
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

4. [ ] In that bay, gently press the card into the PCIe slot on the motherboard until it is fully
seated in the motherboard connector.
5. [ ] Replace the mounting screw to secure the card to the back panel of the node.

Task 14: Transfer the boot drives from the failed node to the replacement node
Both SSD boot drives must be removed from the failed node and installed in the replacement node.
About this task

CAUTION: The boot drive slots are labeled J3 and J4. Each SSD boot drive must be reinstalled
into its corresponding bay in the replacement chassis. Do not swap drive locations within the
chassis during transfer.

Procedure
1. [ ] Grasp both sides of one of the boot drives and lift the boot drive from its slot in the failed
node.

2. [ ] Insert the boot drive into the corresponding boot drive slot (J3 or J4) in the replacement
node. Gently press down to secure the drive.
3. [ ] Repeat the above steps with the second boot drive.

Task 15: Transfer the IB/NVRAM card to the replacement node

The file system journal, contained in the IB/NVRAM card, must be removed from the failed node and
installed in the replacement node.

13
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

About this task

Note: Before proceeding with the transfer, allow the batteries in both the failed node and the replacement
node to charge for at least 30 minutes. In order to retain the journal information through the transfer
process, the IB/NVRAM card must be continuously supplied with power.

The replacement node may arrive from Isilon with an IB/NVRAM card already installed. You'll need to
remove the card to make way for the card being transferred from the failed node.
The IB/NVRAM card you remove from the replacement node must be returned to Isilon with the failed
node.

Task 16: Remove the NVRAM battery assembly

One of the two NVRAM battery assemblies can be removed from the failed node, as long as neither of the
battery assembly status LED lights are red.
About this task
Note: Do not remove both NVRAM batteries at the same time. Removing both NVRAM batteries puts
data at risk during the transfer procedure. If you removed a failed battery assembly from the failed node at
the beginning of the procedure, you must not remove the remaining battery assembly or you will risk
losing data.

Procedure
1. [ ] Confirm that there are two healthy batteries in the failed node.
 If neither battery assembly has been removed from this node, remove one of those assemblies.
 If a failed battery assembly has been removed from the failed node, and both batteries are healthy
in the replacement node, use a battery assembly from the replacement node.
 If failed battery assemblies have been removed from both nodes, contact Isilon Product Support
for assistance. Only remove a battery assembly if both batteries are healthy.
2. [ ] To remove a battery assembly, lift up the locking tab at the bottom of the assembly.

14
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

1. Locking tab 2. NVRAM battery

3. [ ] Remove the battery assembly from the node, using the handle.

Task 17: Remove the IB/NVRAM card using a transfer battery

The IB/NVRAM card can be safely transferred from one node to another as long as a healthy battery is
attached to the card before it is removed from the failed node.
Before you begin
Before you proceed, confirm the following:
 The node the IB/NVRAM is being removed from, and the node the card is going into, are next to one
another with top panels removed.
 The IB/NVRAM card in the destination node should be removed to clear space for the card being
transferred.
 A healthy battery assembly has been removed from a node and is available.

CAUTION: The IB/NVRAM card must be connected to a power source throughout the transfer.
Never disconnect an IB/NVRAM cable without a battery already attached to the card, and never
disconnect a battery without an IB/NVRAM cable already connected to the card.

Once you connect a transfer battery to an IB/NVRAM card, you must complete the journal transfer
before the transfer battery fully discharges (approximately 30 minutes) or the journal data will be
lost.

Procedure

15
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

1. [ ] Separate the battery from the battery assembly.

a. Release the hook-and-loop fastener securing the battery to the assembly.

b. Disconnect the battery cable from the assembly.
For the rest of the procedure, the separated battery is referred to as the transfer battery.
2. [ ] Without disconnecting the IB/NVRAM cable from the IB/NVRAM card, connect the
transfer battery cable to the J4 connector on the IB/NVRAM card to be transferred.
The LED on the back side of the IB/NVRAM card should light green.
If the LED on the IB/NVRAM card does not light green, the battery is not healthy. Return to the
previous step to identify a healthy battery.

1. J4 Connector 2. Transfer Battery

3. IB/NVRAM Cable Connector 4. IB/NVRAM Cable

3. [ ] With the transfer battery connected and the LED light showing green, disconnect the
IB/NVRAM cable from the card and remove the card from the node.
4. [ ] With the transfer battery connected, install the IB/NVRAM card in the replacement
chassis within 30 minutes.

Task 18: Connect the IB/NVRAM card

Once an IB/NVRAM card has been removed from a failed node using a transfer battery, you must
connect it immediately to the replacement node.
About this task

16
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

CAUTION: The IB/NVRAM card contains journal data and must remain attached to a transfer
battery until it is connected to the replacement node. The card must be installed in the
replacement node before the battery discharges (approximately 30 minutes) or journal data will be
lost.

Procedure
1. [ ] With the transfer battery still attached, connect the IB/NVRAM cable in the replacement
chassis to the cable connector at the back of the IB/NVRAM card.

Task 19: Install the IB/NVRAM card in the replacement node

The IB/NVRAM card must be installed in a PCIe slot in the motherboard of the replacement node.
Before you begin
Before inserting the IB/NVRAM card into a node, the node's IB/NVRAM cable must be connected to the
IB/NVRAM card and the transfer battery must be disconnected.
Procedure
1. [ ] Insert the IB/NVRAM card into the PCIe slot in the motherboard, ensuring that the card is
fully seated in the motherboard connector.

17
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

1. Captive screw 2. IB/NVRAM card

2. [ ] Tighten the captive screw on the rear I/O panel.

Note: If you previously removed an IB/NVRAM card from the replacement node to clear room for the
card you just installed, follow these same steps to install that card in the failed node for return to
Isilon.

Task 20: Replace the battery assembly

Return the battery to the battery assembly and insert it back into the node.
Procedure
1. [ ] Disconnect the battery from the NVRAM card.
2. [ ] Replace the battery in the battery assembly.
3. [ ] Insert the battery assembly into the empty battery bay.
The locking tab clicks into place when the battery assembly is fully inserted.

18
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Task 21: Replace the air baffles in the failed and replacement nodes
After all components have been moved from the failed node to the new node, replace the air baffles in
each.
Procedure
1. [ ] Lower the air baffle back into its original position within the node.
Perform this step on both the failed and replacement nodes.

Task 22: Replace the cross bracket in the failed and replacement nodes
After the air baffles have been inserted in the failed and new nodes, replace the cross brackets in each.
Procedure
1. [ ] Replace the cross bracket by hooking it to the bracket holes on the interior of the node,
then snapping the other end in place on the chassis wall.
Perform this step on both the failed and replacement nodes.

Task 23: Replace the top panels on the failed and replacement nodes
After the cross brackets have been inserted in the failed and new nodes, replace the top panels on each.
Procedure
1. [ ] Place the top panel on the node so that the front edge of the top panel is about one inch
behind the drive bays and then slide the top panel forward into place.

CAUTION: Placing the top panel too far back on the node before sliding the top panel forward
can damage the chassis intrusion switch.

2. [ ] Tighten the captive top panel screw to secure the top panel to the node.
Perform these steps on both the failed and replacement nodes.

Task 24: Install the replacement node in the rack cabinet

After the top panel is secured on the replacement node, install the node in the rack cabinet.
Procedure
1. [ ] Ensure that the left and right intermediate slide rails are in the fully open and locked
position.
2. [ ] Keeping the node chassis level with the slide rails, align the ends of the inner rails with
the outer rails.

CAUTION: To minimize the chance of personal injury during node installation, install Isilon
nodes in the rack cabinet without the hard drives installed. Isilon recommends using two
people to lift and move the node.

19
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

3. [ ] Insert the inner rails into the outer rails, and then continue to close the slide rails until the
node is fully retracted into the rack.
4. [ ] Secure the node in the rack cabinet with the chassis retaining screws.

DANGER: Do not continue with this procedure until you have confirmed the following:
Both rails are secured to the rack and all mounting screws are in place and tightened.
The inner slide rails attached to the node are inserted correctly, and firmly secured, in the
intermediate slide rails that are attached to the rack.
If you fail to attach the node to the rails correctly it can lead to severe injury when the node is
pulled for future maintenance.

5. [ ] Reconnect all cables to the back of the node.

Task 25: Install drives in the replacement node

After the replacement node has been secured in the rack, install all the drives that were removed from the
failed node.
About this task
Note: Pay attention to the bay number from which the drive was removed. The drive must be transferred
to that same bay number in the replacement node.

Procedure
1. [ ] With the locking handle on the drive open, insert a drive into an empty drive bay by sliding
the drive along the bay rails until it stops.
Do not force the drive into the drive bay. Forcing the drive into the drive bay could result in damage to
both the drive and the drive bay.
2. [ ] Hold the drive in place and gently push the drive locking handle down against the end of
the drive to secure the drive in the node.
3. [ ] Repeat the previous steps until the front bays of the node are full.

Note: Drives that are not fully seated will not be recognized when the node is booted and a red light
will appear above the drive. To avoid this error, run your finger across all the installed drives to
ensure that they are all seated evenly.

4. [ ] Replace the front panel to cover the drives.

5. [ ] Repeat steps one and two until the back bays of the node are full.
6. [ ] Replace the rear EMI shield to cover the drives.

Task 26: Power up and boot replacement node

After all the drives have been installed in the replacement node, connect all cables to the back of the
node, then power up and boot the node.
Procedure
1. [ ] Connect all cables, including power, to the back of the node.

20
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

2. [ ] Power up the replacement node.

3. [ ] Confirm that the node is back in the cluster. Type the command:
isi status
Confirm that the node has an OK status, like all three nodes in the example below:
ID |IP Address |DASR| In Out Total| Used / Size |Used / Size
---+---------------+----+-----+-----+-----+------------------+-----------------
1|10.53.217.201 | OK | 48M| 0| 48M| 19G/ 6.2T(< 1%)|(No SSDs)
2|10.53.217.202 | OK | 46M| 0| 46M| 23G/ 6.2T(< 1%)|(No SSDs)
3|10.53.217.203 | OK | n/a| n/a| n/a| n/a/ n/a( n/a)|n/a/n/a( n/a)

Task 27: Retrieve and correct the NVRAM data

After a replacement node has been powered on for the first time, the node's Vital Product Data (VPD)
must be updated to reflect the current NVRAM card in the node.
About this task
To update the node's VPD, you will need to first retrieve the NVRAM data, then change the node's VPD to
match the NVRAM data.
Procedure
1. [ ] Retrieve the NVRAM data. Type the command:
/usr/bin/isi_hwtools/isi_ib_fw -d mthca0 -byte_mode query | grep VSD
The command will display the NVRAM data as follows:
VSD: PN: [Config Number] RN: [Revision] DN: [Deviation] SN: [Serial Number]
2. [ ] Correct the NVRAM configuration number in the node's VPD. Type this command using
the configuration number value displayed in step one:
/usr/bin/isi_hwtools/isi_fcb_vpd_tool write nvram_config_number "[Config
Number]"
3. [ ] Correct the NVRAM serial number in the node's VPD. Type the same command as step
2, substituting "config number" with "serial number" information displayed in step one:
/usr/bin/isi_hwtools/isi_fcb_vpd_tool write nvram_serial_number "[Serial
Number]"
4. [ ] Display the node's VPD to confirm the change. Type the command:
/usr/bin/isi_hwtools/isi_fcb_vpd_tool dump
The updated NVRAM configuration and serial number should be displayed.

Install the FRU package and run scripts

Update the configure-to-order (CTO) and as-built information on the node by installing a FRU package.

Note: If your cluster is running in SmartLock compliance mode with OneFS 7.0.2.10 or later, 7.0.1.4 or
later, or 7.1.1.0 or later you will need to enter the provided compliance mode commands to run the FRU
scripts. If your cluster is running in compliance mode but is not running one of these versions, you will
need to upgrade your OneFS version to support the compliance mode commands. Contact Isilon
Technical Support.

21
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Task 28: Install the FRU package on the node

Unpack and install the FRU package on the node.
Procedure
1. [ ] Place the FRU package on the node.
2. [ ] Unpack the FRU package by running the following command:
tar -zxvf IsiFru_Package_<date-time-stamp>.tgz
3. [ ] Type cd to change to the directory containing the FRU tar.
4. [ ] Install the package. Depending on your version of OneFS, run one of the following
commands:
OneFS 8.0 or later
isi upgrade patches install IsiFru_Package_<date-time-stamp>_.tar
Earlier than OneFS 8.0
isi pkg install IsiFru_Package_<date-time-stamp>.tar
As the package installs, the following message appears:
Preparing to install the package...
Checking the package for installation...
Installing the package
Committing the installation...
Package is committed.

Task 29: Run the update script

After the FRU package is installed on the node, run the update script.
Procedure
1. [ ] Move to the FRU package location by running the following command:
cd /var/crash/cto/fruPackages/IsiFru_Package_<date-time-stamp>
2. [ ] Perform the update script by running the following command:
./isi_fru_update_cluster
The system displays confirmation of the following items:
 CTO capability
 Current node hardware configuration

Task 30: Run the ABR script

Run the As Built Record (ABR) script to report the updated hardware to Isilon Technical Support.
Procedure
1. [ ] Verify installation of the updated hardware by running the following command:
./isi_cto_update --abr
The update is verified and a series of status messages confirm the node configuration, and if an FTP
connection is available, an updated ABR is sent to Isilon Technical Support.
2. [ ] If an external connection is not available, manually collect and deliver to Isilon Technical
Support the updated ABR.

22
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

3. [ ] If the cluster is running in SmartLock compliance mode, verify installation of the updated
hardware by running the following command:
sudo /usr/bin/isi_hwtools/isi_cto_update --abr --filepath .

Note: You must include the period at the end of the command.

Sending an ABR to Isilon with no connectivity

If no external connectivity is available, the As Built Record on a Configure to Order (CTO) node cannot be
automatically delivered to Isilon Technical Support.
If external connectivity is available, the ABR is automatically generated and delivered to Isilon Technical
Support. If there is no external connectivity available, you must generate and copy the ABR from the
node, and then send the ABR to Isilon Technical Support through an alternate connection.

Task 31: Generate an ABR

You can manually send an As Built Record (ABR) by copying an XML file from the node and emailing the
file to Isilon Technical Support. You need network access to the node, or you can request that the
customer provide the file to you.
Procedure
1. [ ] Generate an ABR by running the following command:
isi_make_abr
The command generates a temporary file named asbuilt_ <serial-number>_<date-time-
stamp> .xml.
2. [ ] Identify the full name of the ABR file by running the following command:
isi_inventory_tool --display --itemType asbuilt | grep asbuiltFileName=
The system output contains information about the ABR file.
3. [ ] Place the ABR file where you can copy it by running the following command:
isi_inventory_tool --display --itemType asbuilt > /ifs/asbuilt_ <serial-
number>_<date-time-stamp> .xml
4. [ ] Copy the generated asbuilt_ <serial-number>_<date-time-stamp> .xml file.
5. [ ] If an FTP connection is not available, contact Isilon Technical Support for an alternate
delivery method.

Task 32: Remove the FRU package from the node

After all scripts are run, remove the FRU package from the node.
Procedure
1. [ ] Change out of the FRU package directory by running the following command:
cd /
2. [ ] Delete the FRU package from the node. Depending on your version of OneFS, run one of
the following commands:
OneFS 8.0 or later
isi upgrade patches uninstall IsiFru_Package_ <date-time-stamp>

23
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Earlier than OneFS 8.0

isi pkg delete IsiFru_Package_ <date-time-stamp>

Complete the replacement procedure

Complete the replacement procedure by verifying the replacement node, gathering information about the
cluster, and returning the failed node to Isilon.

Task 33: Verify the replacement node is operating correctly

One of the final steps when replacing a failed node is to ensure that the replacement node is operating
correctly.
Procedure
1. [ ] Run the following command:
mount
Confirm that /ifs is mounted. The command should return a line similar to:
OneFS on /ifs (efs, NFS exported, local, noatime, noexec)
2. [ ] Run the following command:
isi alerts
Confirm that there are no unexpected alerts.
3. [ ] Run the following command:
isi status
to confirm that the node status shows as good.

Task 34: Transfer the serial number tag

The serial number tag on the back of the failed node must be removed and installed on the replacement
node.
Procedure
1. [ ] Cut the zip tie to remove the serial number tag from the back of the failed node.
If the node does not have a serial number tag attached to the back panel, locate the blank serial
number tag and pen that are included in the replacement node packaging and write the node serial
number on the tag.
Make sure that the serial number is accurate and easily read.
You can determine the node serial number by running the following command on the replacement
node:
isi_hw_status -i
Write the SerNo value on the blank tag.
2. [ ] Locate the new zip tie that is included in the replacement node packaging.
3. [ ] Attach the serial number tag to the back of the replacement node with the new zip tie.

Update node firmware

It is recommended that you update the firmware on replacement nodes.

24
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

The latest node firmware package and release notes are available for download from EMC online
support.
Follow the instructions in the release notes to update firmware on the replacement node only.

CAUTION: Node firmware updates require a node reboot. To reduce cluster down time, do not
update node firmware on the entire cluster. Only update firmware on the replacement node.

Task 35: Gather logs

After you complete maintenance on a cluster, gather cluster logs.
About this task
You must collect cluster logs after all maintenance. Cluster logs provide snapshots of the cluster that you
can review to make sure that maintenance is successful.
Procedure
1. [ ] Gather cluster logs by typing the command:
isi_gather_info

Task 36: Return the failed node to Isilon

After replacing a failed node, return the failed node to Isilon Technical Support for analysis.
Procedure
1. [ ] Contact Isilon Technical Support to notify them that you will be returning a failed node.
2. [ ] Package the failed node for return shipment using the box and packaging materials
provided with the replacement node.
3. [ ] Attach the return label, provided with the replacement node, to the package.
4. [ ] Complete the return shipping label as directed by Isilon Technical Support. Use the
support case number provided by Isilon Technical Support as the RMA number.
5. [ ] Ship the failed node to the address specified on the return label.

Task 37: Update the install database

After all work is complete, update the install database.
Procedure
1. [ ] Browse to the EMC Product Registration and Install base Maintenance service portal,
at: http://emc.force.com/createPSCcase.
2. [ ] Select the Product Registration and Install Base Maintenance option.
3. [ ] To open the form, select the IB Status Change option.
4. [ ] Complete the form with the applicable information.
5. [ ] To submit the form, click Submit.

25
 906278712.doc (650.00 KB)
6/30/2025 1:29 PM
Last saved by EMC

Where to go for support

Contact EMC Isilon Technical Support for any questions about EMC Isilon products.

Online Support Live Chat

Create a Service Request
Telephone Support United States: 1-800-SVC-4EMC (800-782-4362)
Canada: 800-543-4782
Worldwide: +1-508-497-7901
For local phone numbers for a specific country, see
EMC Customer Support Centers.
Help with Online Support For questions specific to EMC Online Support
registration or access, email [email protected].
Isilon Info Hubs For the list of Isilon info hubs, see the Isilon Info Hubs
page on the EMC Isilon Community Network. Isilon info
hubs organize Isilon documentation, videos, blogs, and
user-contributed content into topic areas, making it easy
to find content about subjects that interest you.

Support for IsilonSD Edge

If you are running a free version of IsilonSD Edge, community support is available through the EMC Isilon
Community Network. However, if you have purchased one or more licenses of IsilonSD Edge, you can
contact EMC Isilon Technical Support for assistance, provided you have a valid support contract for the
product.

26