Team OcUK Site News

Welcome to Team OcUK

Site News Archive

A possible cause for the recent disk errors has been found. Tests are ongoing to verify the reliability of the system hardware. This will involve some extensive database tests on the SETI@home stats (as SETI@home uses the most disk space). If the results are positive, the data server will be taken down for a few hours on Wednesday, and then the stats updates for the last two weeks will be applied to get the site back up-to-date sometime Thursday.

Once again, thankyou for your patience.
Tue, 6th Nov 2007 00:03 by Mark

Having checked the logs, 36 disk errors have appeared over the last three weeks. Most of these have not resulted in serious issues, however there have been enough issues to warrant concern - including corruption of the SETI@home stats database which contains over 7GB worth of data.

While the test update this morning didn't cause corruption (even though it did trigger another error) I can't guarantee that further updates won't cause more damage. For this reason, I have taken the tough decision to suspend all updates for the time being. A second server will continue to collect stats so that once the cause of these problems has been identified and fixed, the missing stats can be reinstated.

My sincere apologies for this extended and ongoing downtime, but it is unfortunately beyond my control.

Please check back regularly - news updates will be posted as and when available. Thank you for your support over the three years this site has been running, and I hope that this will prove to be only a temporary setback.
Mon, 5th Nov 2007 11:48 by Mark

Following on from a second data corruption, this time affecting the databases, all stats data has been restored from a recent backup. Data updates are ongoing and a gradual catch-up should happen over the next 24-48 hours.

The most likely culprit for the issues is suspected to be a recent Linux kernel upgrade from 2.6.22.4 to 2.6.22.9. This upgrade has been backed out and no further errors have been noticed so far.
Mon, 5th Nov 2007 10:42 by Mark

Following from the earlier disk corruption, disk checks are complete. Upload of backups of the corrupted data is likely to take several days. However, the site can be operated during this time.

The site is now playing 'catch up' with several days worth of missed statistics. As this is being done with a 'fast update' process, this will cause projects to become slow or unresponsive while updates take place (update status is shown on the project's main page). SETI@home is first on the list as it is also the biggest.
Sat, 3rd Nov 2007 19:42 by Mark

The observant among you will know that this site has undergone several periods of maintenance and very slow response times over recent days. The original issue was due to power supply maintenance at the datacentre, but subsequent to this, some long-outstanding server maintenance was attempted. Unfortunately this in turn caused some significant disk corruption.

The disk corruption has been fixed, but many files were damaged. None of these files are critical to site operation, but they do need to be repaired and any obvious cause of the corruption resolved.

For this reason, updates are being done manually on a when-possible basis and careful attention is being paid to the system. Further downtime is expected while software updates are installed.

Hopefully all this work should be completed and the site back to normal this weekend.

Apologies for any inconvenience.
Wed, 31st Oct 2007 21:03 by Mark

My apologies for the site downtime over the past five days. Unfortunately, as I don't monitor the site much at the moment, I was unaware of the downtime until a site user contacted me. Many thanks for the report.
Sat, 24th Mar 2007 20:02 by Mark

The work to move Team OcUK to the new data server has been completed and everything is running as it should. Automatic updates have been turned on and at least one project has successfully completed an update. Thankyou for your patience over the weekend.
Mon, 5th Mar 2007 22:43 by Mark

All statistics have been successfully loaded. However, some work is needed to get the update process set up. This is planned for tonight but may take 24-48 hours.
Mon, 5th Mar 2007 11:54 by Mark

Statistics are currently being loaded on the new server, and will appear here over the next 24 hours. Thank you for your patience.
Mon, 5th Mar 2007 03:20 by Mark

The main server that operates the Team OcUK stats site will be replaced with new, faster, and hopefully more reliable hardware during this weekend (3rd-4th March).

To allow this to take place, all stats updates will be turned off on Friday evening. Manual catch-up should take place Sunday evening, and normal updates should resume Monday morning.

While downtime of the site will be kept to the minimum possible, it is anticipated that some outages are likely. Holding pages will be in place where possible.
Thu, 1st Mar 2007 20:53 by Mark

more...