UPDATE: 5:44 pm EST 26th September 2011

Summary of issue and current status:

Visitors utilising the Telstra network for internet access to websites hosted on the Zeus Dynamic web cluster (websites resolving to 202.124.241.200 IP) are currently experiencing Timeout errors.

One of the IP addresses used for our shared hosting infrastructure (zeus-dynamic, 202.124.241.200) has been the subject of a DDOS attack. We have had to block this IP with one of our upstream providers (Telstra), in order to allow other client-facing services to be able to be kept online.

This means that some clients who are accessing their websites via Optus/PIPE connections will be able to access their services, whereas Telstra network traffic will not be able to get through to this webserver cluster. For example, TPG customers will be able to see the websites, while Bigpond customers will not.

Our engineers are working on solutions that will allow us to bring zeus-dynamic services back online, without affecting other client facing services. This work has been going on all day, and will continue overnight. At this point in time we cannot give an accurate estimate of when the Zeus Dynamic services will be brought back online via the Telstra internet service provider connections.

UPDATE: 4:18 pm EST 26th September 2011

Telstra link to Zeus Dynamic (202.124.241.200) has been shut down again temporarily. This means customers using Telstra internet connections will not be able to access websites hosted on Zeus Dynamic shared hosting infrastructure.

Further update will be posted when confirmed as resolved.

Unfortunately we cannot supply an ETA on a DDOS attack mitigation process.

UPDATE: 2:40 pm EST 26th September 2011

Engineers have confirmed the network traffic to have stabilised, and all services should operate as normal. Please note, mail sent from web applications during these attacks will be delayed in arrival to the recipients. No data was lost.

UPDATE: 2:20 pm EST 26th September 2011

There appears to be a secondary attack taking place at the moment. Services affected appear to be only the Shared infrastructure Zeus Dynamic and Apache sites, currently with no impact on shared mail, nor IIS or Zeus Static services.

TheConsole admin panel access has been reported as slow.

Further updates will be posted as information becomes available.

UPDATE: 1:23 pm EST 26th September 2011

Telstra have confirmed traffic has stabilised and HTTP access has been re-enabled on Zeus Dynamic websites as of 1:20pm. All services should now be back to normal and fully functional.

UPDATE: 12:17pm EST 26th September 2011

HTTP access to Zeus Dynamic (202.124.241.200) cluster was completely blocked temporarily to stop offending traffic. Access to the network will be re-enabled shortly following confirmation from Telstra that traffic has stabilised. Currently no ETA is available due to the nature of the event.

As mentioned previously, DDOS attacks only affect the speed/performance of service, no data will be lost.

UPDATE: 11:37am EST 26th September 2011

Please note, one service is still intermittently unavailable. Websites hosted on the shared Zeus Dynamic cluster (202.124.241.200) are still experiencing intermittent outages due to our operations surrounding blocking of the attacker. Update will be posted when fully resolved.

UPDATE: 11:10am EST 26th September 2011

Network connectivity problems have now been resolved. Connectivity is being restored across all services, and no data was lost. We apologise again for this issue, the damage from this attack was mitigated within a record time by our engineers.

UPDATE: 10:50am EST 26th September 2011

It has been confirmed by the engineers that we are currently experiencing a distributed denial of service attack (DDOS) on our network.

Unfortunately given the nature of the attack, it is difficult to confirm ETA on completion of work associated with re-diverting network bandwidth abuse. Further details will be posted as soon as available.

Please note, DDOS attacks only affect the speed/performance of service, no data will be lost.

Original notice: 10:30am EST GMT +1000

TPP Internet Technical Services team would like to acknowledge and advise of a current network connectivity problem within our client facing data centre facilities.

Date and time: connectivity problems commenced at 10:15AM EST (GMT +1000)

Impact: Customers on shared and VPS hosting will experience intermittent connectivity problems to services. Websites and email connections will throw Timeout errors.

Services affected: All services reliant on network connectivity. No data will be lost.

We apologise for inconvenience  caused and will post an update as soon as permanently resolved.

TPP Internet wishes to inform all clients that all TPP staff will be unavailable between 11.15 and 11.45am approx due to a mandatory building fire drill.

We apologise for the short notice however the building managers have only just informed us of this requirement.

Calls will still be queued during this period and we will endeavour to clear the queue as promptly as possible.

Thank you for you patience.

TPP Internet

This notification is part of our commitment to update you on the status of the Windows Plesk service issue.

Status Update – Issue Resolved
We are pleased to inform you that at an overall level, the PLESK and Webmail issues experienced this week have now been resolved. No further server-wide issues have been identified.

It is possible that your site may still be experiencing intermittent issues as a result of this outage, and we wish to assure you that we will be here to fix these issues to your satisfaction, as quickly as possible.

Further, we would like to assure you that we will continue to monitor the situation closely and if any new issues do arise, we will proactively notify you as soon as possible.

Technical Queries
If you identify technical issues resulting from the restoration of your services, please contact TPP Internet technical support:

• By email on techsupport@tppinternet.com
• By phone on 1300 665 491

Please supply as much detail as possible. We suggest that you;

• Summarise the issue(s) with bullet points
• Attach screenshots where appropriate
• Include any other information or details to aid us in locating and correcting the problem.

We’re here to help
For other questions, feedback or concerns of a non-technical nature, please contact feedback@onlinegrowthsolutions.com.au. All emails to this address will receive a personal response.

Finally, TPP Internet apologises for any inconvenience that this outage (and our restoration of old data) may have caused you and your business.

A new server is in the process of being tested and provisioned. When testing is completed, data and changes from the existing server will be synchronized to the new platform. Further updates will be provided as this process continues.

Our engineers are currently investigating the issue as their highest priority. As it currently stands, a new server has been built and is in the process of configuration and restoration of customer data that will replace the current system.

Significant progress has been made however a final ETA of full service restoration cannot be provided at this stage. We will keep customers updated as matters progress.

We’re here to help
If you have any questions or concerns about this scheduled outage, just let us know. Our customer service team will be more than happy to answer any questions you may have.

We are currently experiencing network difficulties as a result of a new Windows server installation.

We apologise to our customers who may be affected by this issue and hope to have this problem resolved as soon as possible.

Kind regards,
TPP Internet Team

Unfortunately, on Saturday we experienced an outage on the TPP network infrastructure. The network started losing some connections at 2pm and by 2.30pm was almost completely offline. Our team of engineers worked hard to get the problem resolved as quickly as possible and the network was back online around 5.15pm.

Although we are still investigating the exact cause, the problem occurred when network interfaces on both the primary and fail-over gateways failed. To get the network back online as quickly as possible we removed the redundant network configurations and routed all traffic through a standalone gateway.

We are still yet to determine if this cascading failure was caused by a software or hardware issue, but are investigating this to discover the root of the problem.

We have scheduled an emergency 3-hour maintenance window for tonight starting at 10pm, where we will be installing additional hardware and conducting testing in an effort to replicate the conditions leading to the failure. This will help us to rectify any issues and eradicate them to avoid reoccurring problems.

We sincerely apologise for the outage. We understand how important a consistent Internet presence is to all our customers and we will always do our utmost to provide you with the best service possible.

After speaking with some of our partners and customers, it is clear that we need to provide better out of hours support. We are looking into this right now, along with a more proactive notification process.

If you have any specific questions for us on this issue, please do not hesitate to contact us directly on 1300 665 491. Alternatively, if you require any technical support, you can send a support ticket via our eSupport System here.

Chris Collinge

Managing Director
TPP Internet

As part of our Plesk Upgrade, we will be carrying out further work tomorrow evening, Wednesday 5th May between 9.30pm – 12.00am. Unfortunately, customers on Control 5 services will experience intermittent outages for control panel access, web & email hosting and ftp services between these hours, while we carry out this work. Customers that are not using Control 5 will not be affected.

Why an upgrade?
This upgrade will provide improved performance for our customers using the Plesk Control Panel and is part of our ongoing commitment to the continuous improvement of our services.

Minimum disruption
The upgrade has been timed to cause minimum disruption. We are sorry if this causes you any inconvenience and we will do our very best to keep the outage as short as possible.

Questions?
As always, if you have any concerns or questions about this upgrade, please don’t hesitate to call our awesome technical support team on 1300 665 491 or send a support ticket via our eSupport System here.

The TPP team

We’re upgrading our Plesk Control Panel on the night of Monday 3^rd May, which will affect customers on control5.tppinternet.net.au. Unfortunately, customers on Control 5 services will experience intermittent outages for control panel access, web & email hosting and ftp services between the hours of 9.30pm and 3.30am AET on 3^rd/4^th, while we carry out this work.

Why an upgrade?
This upgrade will provide improved performance for our customers using the Plesk Control Panel and is part of our ongoing commitment to the continuous improvement of our services.

Minimum disruption
The upgrade has been timed to cause minimum disruption. We are sorry if this causes you any inconvenience and we will do our very best to keep the outage as short as possible.

Questions?
As always, if you have any concerns or questions about this upgrade, please don’t hesitate to call our awesome technical support team on 1300 665 491 or send a support ticket via our eSupport System here.

The TPP team

We’re upgrading our Windows Web and Email servers on the night of Thursday 22nd April and wanted to keep you all updated. Unfortunately, as a necessary result, there will be intermittent outages for web and email hosting between the hours of 9pm and 3am AET on 22nd/23rd while we carry out this work.

Your email is safe
Don’t worry, there will be no mail lost during the upgrade as it will be queued to our backup servers. Access to the servers will be intermittent during this outage.

Why an upgrade?
This upgrade will provide improved performance for our Windows Hosted customers in our shared and footprint hosted environments, and is part of our ongoing commitment to the continuous improvement of our services.

Minimum disruption
The upgrade has been timed to cause minimum disruption. We are sorry if this causes you any inconvenience and we will do our very best to keep the outage as short as possible.

Questions?
As always, if you have any concerns or questions about this upgrade, please don’t hesitate to call our awesome technical support team on 1300 665 491 or send a support ticket via our eSupport System here.

The TPP team