OK this is really important, when Update one for ESXi 5.5 was released, customers connecting to datastores via the NFS protocol were experiencing intermittent connectivity and in particular All Paths Down or APD.
What is ‘All Paths Down’?
For those who are unfamiliar with this “condition”, APD occurs on an ESXi host when a storage device is removed in an uncontrolled manner from the host, or the device simply fails, and the VMkernel essentially panics. This results in the datastore not accepting any I/O from the virtual machines for the duration of the APD condition. The result is that Windows virtual machines begin BlueScreening and filesystems becoming read only for Linux VMS. This can be permanently or temporarily, either way bad stuff happens.
but alas, a patch has just been released for this.
This patch resolves the following issues:
PR1242103: When you run ESXi 5.5 Update 1, the ESXi host intermittently loses connectivity to NFS storage and an All Paths Down (APD) condition to NFS volumes is observed. During the duration of the APD condition and after, the array still responds to ping and the netcat tests are also successful. There is no evidence to indicate a physical network or a NFS storage array issue.
Entries similar to the following are logged in the vobd.log file for volume named 12345678-abcdefg0 as an example:
YYYY-04-01T14:35:08.075Z: [APDCorrelator] 9414268686us: [esx.problem.storage.apd.start] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down state.
YYYY-04-01T14:36:55.274Z: No correlator for vob.vmfs.nfs.server.disconnect
YYYY-04-01T14:36:55.274Z: [vmfsCorrelator] 9521467867us: [esx.problem.vmfs.nfs.server.disconnect] 192.168.1.1/NFS-DS1 12345678-abcdefg0-0000-000000000000 NFS-DS1
YYYY-04-01T14:37:28.081Z: [APDCorrelator] 9553899639us: [vob.storage.apd.timeout] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
Where can I get it?
Earlier this year, I tweeted that 2014 is a great year to re-baseline and reinvent yourself. What I didn’t know then that I realise now was how much I was going to do this. So what has been reinvented and created change for me in my personal life and professional life?
A new wife – On the 15th February, I married my long-term girlfriend Elise in a small ceremony in Victoria, Australia. As any wedding is, it was a special day as we were joined by friends and family from around the globe to celebrate.
A new degree – Last week I flew back to my homeland of New Zealand to graduate with the degree of Master of Management from the Massey University School of Business, this is special for me as it was approximately six years ago I was forced to withdraw from the very last paper due to relocating across to Australia.
Last year I finally completed that paper and submitted just before I went overseas to the UK. I remember being nervous and unsure at the time as one minute I felt I have done enough and my report was good enough and the next I began to question whether or not I should review once more and add more. I am glad I didn’t as I would of never submitted.
A new role – Last Monday (2nd), I started at Veeam Software as their Alliances Manager for Asia-Pacific and Japan looking after the business for some of our top alliances partners. I have always admired Veeam not for the incredible software capabilities they offer, but for the people and culture I have come to know during my time in my presales roles over the years. As part of the APAC team, I am tasked with driving business and fostering healthy business relationships right across the region with these alliances which I am truly excited about. Furthermore, Veeam has always been on the bleeding edge of virtualisation and well respected within the industry, this is appealing as it will still keep me close to the virtualisation community and outstanding people I have met through this community
A new pet – Elise and I adopted our first pet – a kitten named Miller after seeing a friend post on Facebook that he had found a group of them in a box beside a very busy road in Melbourne. This has created havoc in our household as there is no longer peace and quiet but a welcome addition to our household
We are now halfway through 2014, for me change has come in many forms as you have read. I look forward to the remainder of the year and hopefully more exciting changes.