Author Topic: Downtime (July 2011) Statement  (Read 5424 times)

Offline Suzy Scott

  • Suzy, Nuisance of the Coach Park
  • Administrator
  • Hero Member
  • ******
  • Thank You
  • -Given: 69
  • -Receive: 584
  • Posts: 6412
    • ScotBus
Downtime (July 2011) Statement
« on: July 13, 2011, 08:23:08 PM »
Anglia & Thames Valley Bus Forum
Downtime 7th to 11th July 2011
A Statement From Our Founder/Administrator


Good evening Forum members

I know some of you have been deeply disappointed, upset and angered by the recent huge glitch with our Forum. Hopefully I can now explain in more detail as to what went wrong.

As you will recall from the 1st of July newsletter, we were off on holiday to the west coast of Scotland (catch up at http://angliaandthamesvalleybusforum.com/index.php?topic=5085.0 if you missed it!) This was the first time away from the house that I’d had, aside from medical etc. Less said about the holiday – well, from a company who replace their coaches every four years, we defiantly did not expect mould on the bathroom walls, falling curtains and Caroline being unwell TWICE in the five day spell! Anyway, that’s for another night. In fact, if we’d not been moved, we’d never have had the Wi-Fi, which is a lot more use when the area has no GPRS/GSM at times, let alone 3G. Indeed, while moving at one deeply rural point, a text message took almost 90 minutes to send out!

I have looked back on the Anglia & Thames Valley Bus Forum (“the Forum) stats and logs. It is apparent that there was a server problem after the morning of Thursday 7th July 2011. Heart Internet (“the host”) had a problem on the server we use, whereby the server had been running slowly. I was still in England at this point (and still within 3G coverage!) when Andy James alerted the Admins to this fact, along with the rest of the Twitter & Facebook followers. I read this as a temporary glitch (i.e. another user of the same server misbehaving) and left it at that. At that point, attempts to get the page up were just timing out – so this was right, given the previous warning. We left it at that for the night.

At that point, I should stress we had limited kit – namely an iPad 3G each, running iOS4. I had my Blackberry mobile phone handset, which also picks up messages to the majority of my accounts – but not the BT Connect one, which I enter via webmail.

On Friday 8th July 2011, I was initially getting a blank white page. I presumed initially that there was still “hangover” from the previous day, but it was a change. I did try to view via the Blackberry during the day whilst out, but submitting a ticket via WAP and my Blackberry failed.

Once we got back to the hotel, I tried again, and did wonder what was going on. So, at 18.50, I submitted a ticket, querying what happened, and trying to ensure that anything that had perhaps gone wrong (i.e. domain losing records or something!). Now, when you read the posts below from me, do bear in mind that the iPad has a habit of auto-correcting – so any odd words from me probably meant something else i.e. Holstein rather than holiday!) Now, from looking below, I know that is probably not a lot of help, BUT I felt to keep it brief – purely as there is a high case that there might have still been overhanging issues that they did not want to broadcast, or be aware of.

“Since the downtime yesterday, angliaandthamesvalleybusforum.com is loading a blank page, but other
domains working fine.
Not a lot of testing I van do as weare on Holstein & this is on the iPad.
Please help :-)

Within 8 minutes (18.58) a response was sent…
“Hi Suzy,

Thanks for contacting us. A blank page like this would normally indicated a PHP error. Looking at the logs that does seem to be the case as the page is generating these errors:

[Fri Jul 08 18:57:00 2011] [error] [client 79.170.40.31] PHP Notice: Undefined variable: sourcedir in /home/sites/angliaandthamesvalleybusforum.com/public_html/index.php on line 54
[Fri Jul 08 18:57:00 2011] [error] [client 79.170.40.31] PHP Warning: require_once(/QueryString.php) [<a href='function.require-once'>function.require-once</a>]: failed to open stream: No such file or directory in /home/sites/angliaandthamesvalleybusforum.com/public_html/index.php on line 54
[Fri Jul 08 18:57:00 2011] [error] [client 79.170.40.31] PHP Fatal error: require_once() [<a href='function.require'>function.require</a>]: Failed opening required '/QueryString.php' (include_path='.:/usr/share/pear5') in /home/sites/angliaandthamesvalleybusforum.com/public_html/index.php on line 54
Best Wishes,
Richard Lewis”
Now, at this point, I had no way of getting into the Forum control, or indeed FTP, as I did not bring a laptop computer with me on my most recent holiday. I did not think it was necessary, but, nevertheless, in hindsight, it probably sounds much better.

At 19.35, I replied again…
“Hello there,

On Wednesday, the website was working fine,
.
Thursday, server problems were showing on the website, so I did no chase this (excuse autocorrect!)

Friday today, server has been showing a blank white page.

I am on holiday in a remote part of Scotland with only an iPad I.e. No laptop, so I cannot FTP in to it.

All I know is that it was working fine wednesday evening, and I've been 450 miles away from the pc.

Can you please investigate?

Thanks,
Suzy”

Response at 19.54, denied responsibility!
“Hi Suzy,

Thanks for your reply. Any problems on the server have since been resolved, we're not having any problems with web39 right now.

This is the line the error logs refer to:

54 require_once($sourcedir . '/QueryString.php');

The file this refers to just doesn't exist. I'm afraid this is not something I will be able to resolve for you, I would recommend restoring a backup when you can.
Best Wishes,
Richard Lewis”

Right, so who took it? Upon checking, none of us had been logged in via FTP! So, that night at 22.18, I replied again. I had initially minded to give up, but I decided to plod on to try to resolve the issue for the benefit of the Forum members…
“Richard,

Sorry but I take exception to this!

The service status page yesterday made reference to investigating customer behaviour, not mentioning
whether this was a DDOS attack or a customer maxing out on the server CPU. There has been no response or
apology to your customersv- I did check, and all your direct debits have been paid.

This particular site is running Simple Machines Forum on mySQL, hosting a very lively Forum of 400 members
- and I had planned to transfer my other webpages to you later this year.

Can you tell me if the server where this was got hacked/DDOS/cust maxing out CPU?
None of the files have been removed or modified by anyone with permission to access the servers.

I am on holiday,,,, and am now tearing my hair out over this one!!! Through no fault of my own, the server went
down yesterday. Now I find the files pinched, but you are telling me that you cannot helpmeet with this? If we'd
been pre warned to take a backup then we would have made arrangements - otherwise, we rely on the backup
system of the host. Am I also being told these dont happen either?

If you feel unable to resolve this matter then please escalate or seek further help! Otherwise, I'm very unhappy
that files cAn go missing after downtime, but also that you are denying responsibility. Why am I at fault here?

Yours stressed and very upset,
Suzy”

Now, at 22.24, a manager that I’d dealt with before replied. Success, I was thinking… although C was unwell that particular evening, I carried on!
“Hi Suzy,

Thanks for your reply. I am looking into this for you now.
Best Wishes,
Sam Howson”
   

Success, I thought – but that feeling of having someone who could help was blown away 25 minutes later at 22.49… so a user losing data is not a “disaster”? Hmm…

Hi Suzy,

Thanks for your reply. I have discussed this matter with our system administrators, and we are not aware of any trouble with this server yesterday. The issues you were experiencing with down-time to your site may be related to this, but we did not receive any reports from any other customers on this server regarding adverse performance.

I should also say that we recommend that our customer take regular backups of their websites to their computers. We only take backups in case of disaster recovery of our own systems. Taking regular backups ensures that no data is lost in cases such as this.

Our system administrators can look into restoring a backup of your website for you, but as these are kept in case of disaster recovery and restoring them is a manual process for our system administrators, we would need to charge a fee of £25+VAT for this.
Best Wishes,
Sam Howson


Bedtime for the night after that point – very unhappy, not just at the holiday (I might shut up about that next year!) and the way I felt used by the hosting company. I think a very strongly worded set of messages went around the Admin team at that point – I should stress I was not taking this out on anyone else, merely attempting to show them what was up. I could not copy & paste the entire ticket (or even forward to someone else), so was unable to share it with the others.


So, Saturday. C’s got a stinking cold, so I went out for a few items in the village before going into Glasgow to meet a friend.

09.36 – from me – as you can see, the money was not an issue, I was trying to get the Forum back up and running, feeling a little – what’s the word – exploited, perhaps? Inciodentlly, Evo had done it a few times for me over the years, within half an hour (5-15 MIN JOB!) without charge. Maybe I was spoilt!
“Sam,

Are you saying the problem on the server status page did not happen?

Go ahead and do what you need to restore it. yes I will pay, but under protest. Tell me how you would like the
money to get to you.”

As you can see, I effectively gave them carte blanche to get the Forum back!

10.05 REPLY
Hi Suzy,

Thanks for your reply. Sorry about the confusion regarding the problems we were experiencing on the server on Thursday. There was a period of unusually high load on the server web39 on Thursday morning which did cause downtime on the server, but it did not cause data loss. This seems to be an isolated problem, and not a problem with the server.

I will add a payment link onto the ticket for you now, your patience is appreciated at this time.
Best Wishes,
Richard Lewis
(Normally, everything, including domain renewals, goes via the Direct Debit, Not a problem but rather unusual, sending out even more alarm bells in my mind!)

10.43 REPLY

Hi Suzy,

Thanks for your patience. The payment link has now been added. We will process the restoration when the payment has been made.
Best Wishes,
Richard Lewis
So, I was out that day, but when I was on the coach from Glasgow, I logged into the members area to get a feel of what was happening – sadly, not a lot. Paid the amount requested (£30 including VAT - £25 without it) on the last 926 back from Glasgow – leaves BBS 18.00, and I would have been somewhere between Anniesland and Balloch when it was paid i.e. 18.30. At this point, I knew it was a 5-10-15 minute job on a medium day, hour tops.

Now, from my perspective, I thought (maybe naievely) that once the money had been paid, the Forum would be back. No teapot-ing chance.

19.23 update from the host – sorry, but they are seriously extracting the urine here!
“Hi Suzy,

Just as an update, we'll be able to process this backup restore later tonight/early tomorrow morning.
Your patience is appreciated.
Best Wishes,
David Gouldby”
FRO! IT’S AN HOUR, EVEN IF YOU SPENT 50 MINUTES ON MESSENGER AND THEN FIRED UP A SHELL TO DO THE JOB THAT I PAID YOU FOR!!!! I didn’t say, but thought.

I’d gone back to bed by the time they got the finger out. 23.33 update from the host – five hours after I paid!
“Hi Suzy,

Thanks for your patience.

This has now been restored for you, to the restore directory in the root of your webspace.
Best Wishes,
David Gouldby”
I kept trying it throughout the next morning, wondering what the hell I’d done wrong0! Now, upon our return from Oban, Caroline was physically sick after lunch, so not feeling so good. Once I’d dealt with all that, fired up the iPad. As C had been unwell, we were effectively confined to our room, but at least we had WiFi! Went to look at the Forum (as I had been doing with the Blackberry throughout the day) and I was highly irate at the situation developing. I then clicked that I realised that this would need to be in public_html directory to work properly – so the root directory was not what was why no-one could still log in – and even if we changed the address to the actual directory, the Forum was setup with the known one, so would keep looping back to it. My reply at 18.53…
“Erm....

Page still blank. I do not have access to FTP. As I paid for the restore, can you please place it under public-HTML in the
rrot pls.”


19.07 – suspiciously quick reply, given what happened the previous day!
Hi Suzy,

Thanks for the reply.

There are many files in the public_html folder which I may overwrite if I move the contents of the restore folder over.

FTP access should be working fine, what problems are you having with that?
Best Wishes,
David Gouldby


At this point, I start to hit the roof.  I am no mySQL/PHP data geek, but I knew this was wrong! Anyway, when I asked him to move the restore over the original copy, of course there would be “old” files there, but not enough to get it to work – hence why I asked for this to be done in the first place. So, at 20.16 (and yes, I Know capitals is wrong, BUT I Felt so badly upset it was only a 1/100th of what I felt!) At this point, my mind exploded – and the rate was based on what they charged me for, earlier in that ticket.

“For the last time...

I AM ON HOLIDAY WITHOUT A COMPUTER! SO NO FTP ACCESS? THIS MESSAGE IS BEING SENT OUT
ON AN IPAD!!!!!!

I PAID YOU TO DO THE RESTORE! SO WHY IS IT STILL SHOWING A BLANK PAGE?

YOU HAVE TURNED THIS TIME AWAY UPSIDE DOWN!!!

THIS MATTER HAS FRUSTRATED ME GREATLY! THIS MATTER WILL BE TAKEN FURTHER, AND AS
YOU'VVE CHARGED ME 30 QUID FOR WHAT I KNOW IS A 10-15 MNUTE COMMAND LINE JOB!

If the Forum is not back working and fully Accessibile by 6am tomorrow, I WILL BE CANCELLNG THE
SERVICE! A COMPLAINT WILL BE LODGED AS PER YOUR PROCEDURE. I will also be invoicing you for
my time at the rate you did to me I.e. 120 per he Inc. Vat, albeit at least double tiime and a half for holiday
work done, and substantial weekend premium too.

And I know capitals is shouting. Lets just say it's about A hundredth of what I am feeling tonight
Very unhappy, very unimpressed.”


At 20.34…
Hi Suzy,

Thanks for the reply.

Sorry, I had to be sure before overwriting any existing files.

To be sure, I simply renamed the public_html folder to public_html_backup.
And then I renamed the restore folder to public_html, so that should be live and working fine now.

You can do this through the File Manager in the eXtend control panel without FTP access.
eXtend control panel access is possible on both iPhones and iPads.

Best Wishes
David Gouldby


Now, call me selfish if you like. I’d explained what I needed, when by, and even gave them a generous (IMHO) timeslot – i.e. 12 hours – to fix. However, at this point, the fault had changed – it was now giving a 403 Forbidden error. In other words, permissions were not altered to match. I had the last few hours on holiday with an uncooperative webhost, a sick girlfriend and a (Literally) rotten hotel, but no computer! I started to relax & pack up – after all, cases had to be out for 06.15 the next morning.

So, by 06.00 on Monday 11th July, it’s still not up. Deadline passed – and I even held back before replying to the last ticket. It was not just access to the control panel/FTP that I would need, as I’d also need into the Forum – but that was clearly lost on them!

So, at 07.10 on Monday,  my reply…

“David.

It's now after 6am, and the Forum is still completely inaccessible.
Please reread everything.

To have a holiday where I wanted to do no work was the idea, for five days.

Very, very unhappy.

Leaving the hotel in 35mins so may not have mobile access for most of the day.”1

Reply from the host at 08.00;
“Hi Suzy,

Thank you for your reply.

I am currently investigating this and will update you again shortly.
Best Wishes,
Alistair White”
Now, we were well on the way by this point, but I was asleep. When I woke up in an area with coverage, I tried to find out why the Forum still did not work…
… I very nearly hit the roof of the coach with this response! In other words, NO DATABASE PROBLEM, AND NO MENTION OF CHANGING PERMISSIONS!

Hi Suzy,

I've traced the source of the problem - the problem isn't with QueryString.php, but Settings.php which is the file of course with all the setup information for Simple Machines Forum such as the directory of where to load QueryString.php and other necessary files.

The problem is that "Settings.php" (in public_html) is completely empty and has no data, I have also checked on the backups and this is the same thing so we have no copy of this.

There is however I note a file called "Settings_bak.php" that looks like it was created on the 22nd May, and this does contain data. Hopefully it has all the necessary data to get the site up online, so we can probably rename this to Settings.php however I will need your permission to do this as I will be of course changing files on your site and I don't know if this could be potentially harmful to your installation of this software.
Best Wishes,
Alistair White
So, as soon as I read this at 11.28, replied that “Permission given to rename the file.” – did not say anything else.
Within a few minutes, this was working, and the host ticket replied later that hour.
WHY WAS THIS NOT DONE IN THE BEGINNING?


A fax of complaint is to go into Heart Internet regarding the inconsistent service, and also seeking recompense. As to whether they even give me the £30 that they fleeced off me, then who knows? Currently investigating a new host to move to ASAP, but this will not be this week. I need to ensure that I won’t be held to ransom, or incorrectly dealt with, again!

Now, I think that, in the circumstances, I was doing what I could to try and get the Forum back up and running. I do feel strongly that this was a bad time for anything to go wrong, and I do feel a little guilty now. If someone else wants to take over running and do a better job of the Forum, then PM me and we will talk. A word of warning – I’m not going to hand over four years of blood, sweat, tears (and money!) to someone who wants to replace it with a dreaded Yahoo group! I’ve not broken even, maybe never will. But I certainly won’t be giving it away.

In the meantime, please feel free to reply to this thread with your views and thoughts.

Suzy Scott
Forum Admin & Founder

Adam Dowling

  • Guest
Re: Downtime (July 2011) Statement
« Reply #1 on: July 13, 2011, 08:55:11 PM »
Suzy, considering the circumstances you did a fine job. It just seems to have been an unfortunate coincidence that the forum went down just as you happened to go on holiday....and the host did NOT help at all with the problem, by denying responsibility over a server they operate and that they are responsible for maintaining! Hopefully they will reimburse you....I think some if not most of us would gladly make a small donation towards a change of host, if any extra funds are needed and if it does happen.   :)

Offline Derrick Pancel

  • Hero Member
  • *****
  • Thank You
  • -Given: 8
  • -Receive: 5
  • Posts: 1658
Re: Downtime (July 2011) Statement
« Reply #2 on: July 13, 2011, 10:40:23 PM »
It sounds like a very bad case of Sod's Law, that what could go wrong did go wrong - the last thing you and Caroline needed whilst attempting to relax on holiday.

I'm sure that a number of Forum members will agree that your hard work behind the scenes over the last four years (is it really that long ??) is very much appreciated, and long may it continue  :)

Offline centralsimon

  • Hero Member
  • *****
  • Thank You
  • -Given: 4
  • -Receive: 16
  • Posts: 137
    • Station Staff forum
Re: Downtime (July 2011) Statement
« Reply #3 on: July 13, 2011, 10:59:12 PM »
I know I am not one of the most active members but I appreciate everything that Suzy does for the forum. I think you went over & above to get the forum back up and I am grateful for the massive effort made especially as you were on holiday. I know if it was me the wife would be furious. Not that I think you need it, but i have total faith in Suzy as the head honcho and see no reason for that to change. Thank you for the everything.

Offline busman

  • Hero Member
  • *****
  • Thank You
  • -Given: 16
  • -Receive: 203
  • Posts: 1743
    • The Wakefield Files
Re: Downtime (July 2011) Statement
« Reply #4 on: July 13, 2011, 11:02:24 PM »
It sounds like a very bad case of Sod's Law, that what could go wrong did go wrong - the last thing you and Caroline needed whilst attempting to relax on holiday.

I'm sure that a number of Forum members will agree that your hard work behind the scenes over the last four years (is it really that long ??) is very much appreciated, and long may it continue  :)
Many thanks for all your efforts Suzy.
Never ask a dustman to talk shop.
He will only talk a load of rubbish!

Offline J621BVG

  • Hero Member
  • *****
  • Thank You
  • -Given: 6
  • -Receive: 4
  • Posts: 1309
  • J621BVG
Re: Downtime (July 2011) Statement
« Reply #5 on: July 13, 2011, 11:13:24 PM »
It sounds like a very bad case of Sod's Law, that what could go wrong did go wrong - the last thing you and Caroline needed whilst attempting to relax on holiday.

I'm sure that a number of Forum members will agree that your hard work behind the scenes over the last four years (is it really that long ??) is very much appreciated, and long may it continue  :)
Many thanks for all your efforts Suzy.
Same here. Thanks for running one of the best forums on the net. Much appreciated.
East Londoner

andrewbus42

  • Guest
Re: Downtime (July 2011) Statement
« Reply #6 on: July 13, 2011, 11:35:35 PM »
Suzy, as a forum user and a member of your admin team I would like to say how much I appreciate what you have done to the forum over the years - you have put a lot of time and effort in running and expanding the forum into what is by far the best transport related forum/group in the area we serve!

It is a shame that this unfortunate and untimely incident occured, especially when you should have been relaxing on holiday with Caroline - which you really deserved.

I think I can both personally and on behalf of the other members say that we don't want you to give up, the forum wouldn't be the same without you and almost certainly wouldn't be as good with anyone else in charge.

I hope that with the support of your admin team and of the forum users you can get through these difficalt times - you only have to look at the messages above to know you are appreciated :)

Andy
« Last Edit: July 13, 2011, 11:37:54 PM by Andy James »

stagecoach106

  • Guest
Re: Downtime (July 2011) Statement
« Reply #7 on: July 13, 2011, 11:55:12 PM »
Suzy you do a fab job on here this the best forum in the uk and probally the best ......in the world you think it not just news you bring togever it friend ships been made on ive found a best mate on here . And know one judges you on here now not even for bad spelingggs .
Three chears for suzy

Offline FF17355

  • Bouncing
  • Hero Member
  • *
  • Thank You
  • -Given: 11
  • -Receive: 3
  • Posts: 254
Re: Downtime (July 2011) Statement
« Reply #8 on: July 14, 2011, 05:24:30 AM »
  Suzy, I was so lost over here in sunny North Carolina that I even started on the Honey-Do list,thankfully you took up most of your holiday to get things back on track or would have had to finish some them. Seriously thank you for every thing you do and did to get things back and running
             David

Offline Tony

  • Hero Member
  • *****
  • Thank You
  • -Given: 22
  • -Receive: 42
  • Posts: 317
Re: Downtime (July 2011) Statement
« Reply #9 on: July 14, 2011, 01:52:08 PM »
I would also like to thank suzy for all the work she does on this forum as many people have said its very much appreciated.

Offline grandad

  • Hero Member
  • *****
  • Thank You
  • -Given: 2
  • -Receive: 5
  • Posts: 164
Re: Downtime (July 2011) Statement
« Reply #10 on: July 14, 2011, 01:56:33 PM »
Suzy
You have done nothing wrong,if it was not for you and the other admin staff we would not have a forum,thanks for all your hard work
Flog

Offline SURREYMAN

  • Hero Member
  • *****
  • Thank You
  • -Given: 16
  • -Receive: 35
  • Posts: 272
Re: Downtime (July 2011) Statement
« Reply #11 on: July 14, 2011, 03:57:49 PM »
Suzy, would just like to add that I really appreciate the site but I don't expect it to be always up and running, sorry to hear your holiday was ruined.
 If it happens again I would be more than happy for you to just leave it and enjoy your break until you return.
 Best wishes - Surreyman

Offline dennisdart

  • Hero Member
  • *****
  • Thank You
  • -Given: 31
  • -Receive: 605
  • Posts: 773
Re: Downtime (July 2011) Statement
« Reply #12 on: July 14, 2011, 04:36:05 PM »

Suzy,

Just to reinforce what everyone has said, I think you and your team do a wonderful job in keeping
the forum going and I think as someone has already said it is sods law that something would happen
when you were away.

Keep up the good work and if in the future you decide to change the host then I will support in the same
way that "will I be on the busway soon" has suggested.


Offline Colin T

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 0
  • Posts: 42
  • My bus when it was new
    • DW317.co.uk
Re: Downtime (July 2011) Statement
« Reply #13 on: July 15, 2011, 12:43:42 PM »
So sorry to read of the hassle you had whilst on your holiday.

Heart always struck me as a bit tin pot - I use streamline.net for all my internet hosting and I can honestly say I've not experienced any down time at all in over 3 years. All support issues are dealt with very promptly, and they are associated with Nominet which deal with registering all .uk domains.

I cannot recommend them highly enough and would suggest you consider them when your hosting comes up for renewal.