If there is an outage between the SANs, then the mirrors may break and go into either System Fractured or Administrativly Fractured state.
This can be viewed in Unisphere by logging into CX4-DC1, and then browse to Replicas -> Mirrors.
Click in the Mirror of interest in the top window to see the primary and secondary LUNs in the bottom window. The condition should be “Normal” or “Updating” (updating means its synchronising). If it says “System Fractured” or “Administrativly Fractured”, then you have problems. What you want to do is manually force a synchonisation. Do this by highlighting the broken mirror LUN and click on Synchronise. If it works, hurrah. But if it does not, you will get a vague error. In my case, the the error hinted that there was a connectivity issue between the SP’s between the SANs. Remember, Mirrorview works on a SAN1 SPA <-> SAN2 SPA path. (Repeat for SPB). So we need to check connectivity. Considering a cut in connectivity is the most likely reason for a break in the mirror, this makes sense. So, let’s check connectivity.
You can use the Unisphere GUI to test, and we will in later steps but it is helpful to have a Unix/Linux shell at your disposal on the the same VLAN as your iSCSI traffic network. You need to use ping to test connectivity. To add to this mix, if you are using Jumbo frames (which we are), you need to test that as well.
Test ping with Jumbo packets and no frag: ping -s 8972 -M do 10.3.9.3 -I eth1 (This will ping 10.3.9.3 from eth1 (on GTO) with a packet size of 8972 (+28 bytes of header to equal 9000 MTU) and with no defragment of packent ( -M do). This needs to work. If not, please follow up with your friendly Netops staff. They will need to check connectivity and MTU setting for EACH interface along the way. This is working now so it would only break if Netops made a change.
To test with Unisphere, open the SAN you want in Unisphere drop down window. Click on System. Under SPA Tasks and SPB Tasks there are ping command. Use the interface, choose a source and destination IP and test. These should be successful.
Once you are confident the connectivity is there, confirm the SAN to SAN relationship from the System window. Click on “Storage System Connectivity status”. Then view “MirrorView Initiators”. There should be two initiators, one for SPA and one for SPB. They should both be registered and logged in. Once confirmed, check “Connections Between Storage Systems” inder iSCSI Management (Still in the System view). This allows you to view and test the connectivity between the SANs. Do so before moving on.
The last thing to test is under Replicas -> Mirrors. Click on “Manage Mirror Connections” on the left hand side menu under “Configuration and Settings”. The left hand side is for iSCSI and the right is for Fibre Channel. We only need to worry about iSCSI. Make sure its says Enabled. If not then disable/Enable the connection.
Even though it said Enabled, I had to disable and re-enable the connection here to regain connectivity.
Go back to the fractured mirror LUN. Go into its properties and on the Secondary Image tab, change the Recovery method to manual. Try to synchronise. It should now work. If it does not check to see if it says “System Fractured” or “Administratively Fractured”. If “System Fractured”, then turn of the connection as stated in the line above, and then manually fracture the mirror. This should change “System Fractured” to “Administratively Fractured”. You may need to refresh to see the change. Then, restart the Mirror connection (as stated in the line above), and then manually synchronise.
Change the recovery method back to automatic.