- aktuell - lists.lrz.de

Upd:Garching: Sonderwartung Mi 20.3. um 7 Uhr
by LRZ aktuell March 20, 2013

March 20, 2013

Aenderungen an der Mitteilung: Update Der Neustart des Routers von 07:00 bis 07:05 war erfolgreich. Bitte wenden Sie sich an den Servicedesk, falls Sie noch Netzprobleme im Bereich Campus Garching feststellen. Wegen eines Softwareproblems muss der Router csr2-kw5 morgen, Mittwoch um 7:00 Uhr gebootet werden. Im Campus Garching wird es dadurch zu einigen Minuten Netzausfall kommen. Diese Information finden Sie im WWW unter http://www.lrz-muenchen.de/services/netz/aktuell/ali4546/ Wolfgang Beyer

1 0

Garching: Sonderwartung Mi 20.3. um 7 Uhr
by LRZ aktuell March 19, 2013

March 19, 2013

Wegen eines Softwareproblems muss der Router csr2-kw5 morgen, Mittwoch um 7:00 Uhr gebootet werden. Im Campus Garching wird es dadurch zu einigen Minuten Netzausfall kommen. Diese Information finden Sie im WWW unter http://www.lrz-muenchen.de/services/netz/aktuell/ali4546/ Wolfgang Beyer

1 0

Netzwartung am 19.3.
by LRZ aktuell March 18, 2013

March 18, 2013

1. Campus Garching Durch eine Konfigurationsanpassung am Router csr2-kw5 kann es zwischen 7:30 Uhr und 8:00 Uhr fuer einige Anschluesse zu ca. 10 Sekunden Unterbrechung kommen. 2. Campus Grosshadern/Martinsried Durch eine Konfigurationsanpassung am Router csr1-kic kann es zwischen 8:00 Uhr und 8:30 Uhr fuer einige Anschluesse zu ca. 10 Sekunden Unterbrechung kommen. 3. Max-Planck-Institut fuer Plasmaphysik Garching Zwischen 8:00 Uhr und 8:30 Uhr kommt es durch den Reboot des Switches swz1-kai zu einigen Minuten Ausfall der Anbindung an das MWN. Diese Information finden Sie im WWW unter http://www.lrz-muenchen.de/services/netz/aktuell/ali4545/ Wolfgang Beyer

1 0

SuperMUC: Explanation for recent system problems
by LRZ aktuell March 15, 2013

March 15, 2013

Dear users of SuperMUC, After a phase of stable operation since end of January, problems with the stability of the I/O subsystems have resurfaced after the last maintenance on March 11 and 12. This document intends to explain what has happened based on the root cause analysis performed by IBM and DDN throughout the last days. During the maintenance, a mechanism on the storage systems that is intended to contribute to data integrity was temporarily switched off; when returning this mechanism to operation, a number of data blocks was erroneously marked as bad due to a bug in the controller firmware. As a consequence, a total of three files was damaged, leading to GPFS outage if any of these three files was read. The resolution process consisted in identifying the bug, the list of bad blocks, and the three files associated with the bad blocks. The files have been rendered inaccessible, and their owners have been notified. The firmware bug is triggered if a combination of exactly three conditions is fulfilled (one of them being the switching-on process mentioned above); unfortunately with a storage system of the size installed at our site, the probability for this occurring is rather high. By changing a setting on the controllers one of the conditions that trigger the bug is eliminated; for this reason we believe that the system can now again be safely operated. The changed setting will be reverted to its old value once a new firmware release is installed that fixes the bug. Please remember to archive important data stored on the WORK or SCRATCH file systems to tape via TSM (see http://www.lrz.de/services/compute/ backup/ and use a login to supermuc-tsm.lrz.de; the file system policies are described in http://www.lrz.de/services/compute/supermuc/ filesystems/). Even for rapidly changing datasets (like restart files), it may be advisable to do this at least occasionally in order to prevent accumulated loss of CPU budget. Corrupted data can occur not only through system-related bugs, but also may be caused by external influences, e.g., a power failure. Once more, apologies for the delay in processing your jobs and thanks for your patience. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4544/ Reinhold Bader

1 0

Upd:SuperMUC: Returned to user operation
by LRZ aktuell March 15, 2013

March 15, 2013

Changes to this message: Returned to user operation Status update (March 15, 9:00) The system was returned to user operation yesterday evening at 21:30. We intend to publish a separate announcement on the details of the observed problems soon. Status update (March 13, 17:00) Unfortunately the data corruption problem has recurred. We have again suspended user operation. Status update (March 13, 16:15) IBM in collaboration with DDN and Mellanox has isolated and repaired two technical issues. One of these issues was located in the I/O subsystem and has caused data corruption which was noticed by GPFS but could not be automatically repaired, leading to unavailability of the file system. It is believed that the scope of this data corruption was limited to a single file, which was identified and removed from the GPFS file system. After successful internal test runs the system has been returned to regular user operation. ----------------------------------------------------------------------- Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0

Upd:SuperMUC: Returned to user operation
by LRZ aktuell March 15, 2013

March 15, 2013

Changes to this message: Returned to user operation Status update (March 15, 9:00) The system was returned to user operation yesterday evening at 21:30. We intend to publish a separate announcement on the details of the observed problems soon. Status update (March 13, 17:00) Unfortunately the data corruption problem has recurred. We have again suspended user operation. Status update (March 13, 16:15) IBM in collaboration with DDN and Mellanox has isolated and repaired two technical issues. One of these issues was located in the I/O subsystem and has caused data corruption which was noticed by GPFS but could not be automatically repaired, leading to unavailability of the file system. It is believed that the scope of this data corruption was limited to a single file, which was identified and removed from the GPFS file system. After successful internal test runs the system has been returned to regular user operation. ----------------------------------------------------------------------- Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0

Upd:SuperMUC: User operation suspended
by LRZ aktuell March 13, 2013

March 13, 2013

Changes to this message: User operation suspended again Status update (March 13, 17:00) Unfortunately the data corruption problem has recurred. We have again suspended user operation. Status update (March 13, 16:15) IBM in collaboration with DDN and Mellanox has isolated and repaired two technical issues. One of these issues was located in the I/O subsystem and has caused data corruption which was noticed by GPFS but could not be automatically repaired, leading to unavailability of the file system. It is believed that the scope of this data corruption was limited to a single file, which was identified and removed from the GPFS file system. After successful internal test runs the system has been returned to regular user operation. ----------------------------------------------------------------------- Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0

Upd:SuperMUC: Returned to user operation
by LRZ aktuell March 13, 2013

March 13, 2013

Changes to this message: Returned to user operation Status update (March 13, 16:15) IBM in collaboration with DDN and Mellanox has isolated and repaired two technical issues. One of these issues was located in the I/O subsystem and has caused data corruption which was noticed by GPFS but could not be automatically repaired, leading to unavailability of the file system. It is believed that the scope of this data corruption was limited to a single file, which was identified and removed from the GPFS file system. After successful internal test runs the system has been returned to regular user operation. ----------------------------------------------------------------------- Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0

Upd:SuperMUC: Access to GPFS is disrupted
by LRZ aktuell March 12, 2013

March 12, 2013

Changes to this message: Status update Status update (March 12, 18:50) The root cause for the problem has unfortunately not been found up to now, so the machine is effectively offline while further investigations are pursued. ----------------------------------------------------------------------- Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0

SuperMUC: Access to GPFS is disrupted
by LRZ aktuell March 12, 2013

March 12, 2013

Dear users of SuperMUC, Due to a problem with the GPFS services access to the file systems WORK and SCRATCH is presently disrupted. IBM is attending to the problem, and we'll keep you up to date on the status via this document. Apologies for any delays in job processing. This information is also available on our web server http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4543/ Reinhold Bader

1 0