Starting yesterday afternoon, and continuing into today, some of Intuit’s servers went offline to the outside world.

That means that businesses using QuickBooks Online, Intuit Merchant Services, or QuickBooks Online Payroll were left twiddling their thumbs. Lots of them were not too happy. I read these tweets on Tuesday afternoon:

* * *

“How can QuickBooks Online Edition be down all day? Very frustrating.”

“Quickbooks Online has been down for hours now. Anyone else feeling like “the cloud” is more like “the smoke”?”

“Freaking out a bit that QuickBooks is unavailable and has been all day. This is when “in the cloud” stuff scares me to death.”

* * *

This is one of the disadvantages of the cloud — it’s completely out of your hands. That’s also it’s biggest advantage (when it’s working right).

The pros and cons here apply to all cloud-based applications. There were the same mumblings and grumblings a week or so ago when a chunk of Gmail’s users lost access to their accounts because of a problem on Google’s servers.

What brings me down, though, is that this is not a rare event for Intuit customers. This happened back in January, a couple of times in 2010, and you can keep going back.

Honestly, Intuit makes great software. QuickBooks rocks. Quicken rocks. TurboTax rocks. ProSeries rocks.

But Intuit’s deliverable online IT doesn’t rock, and this is not new.

Why fight it? Why doesn’t Intuit farm out their online services IT to somebody who has a fantastic track record and can deliver 99.95% uptime? IBM, maybe? Sun?

Please follow and like us:

8 thoughts on “It’s Hard to Feel Up When QuickBooks’ Servers Are Down

  1. Hello, this is Rich Walker from Intuit.

    QuickBooks Merchant Services, QuickBooks Online, QuickBooks Online Payroll and other services are intermittently available, as engineers bring back servers online. Unfortunately, the entire restoration process might take several hours .

    As a result, customers may be able to access these services, although service may be interrupted during the restoration process.

    Our first priority is restoring service and we will work with customers to resolve further concerns when service is restored.

    Engineers are “swarming” the problem, as we’re fully committed to restoring service as soon as possible. We apologize for this outage and will provide a further update by 5 p.m. Pacific time.

    Reply
  2. Hi Rich, thanks for the update. I can only imagine such a “swarm”! Do you have any comment on the bigger picture — any ideas on what Intuit might do to address uptime issues from here forward?

    Reply
  3. We’ve restored service for QuickBooks Merchant Services, QuickBooks Online, QuickBooks Online Payroll and other online offerings. As our systems come online, customers may experience a slight delay when logging in. If you cannot log in, please wait a few minutes and try again. We appreciate your patience and again apologize for this disruption. [Updated: March 22, 2011, 4:08pm PT]

    Reply
  4. I’ve written about this problem in my newsletter – and my perspective on it is colored by the fact that my consulting expertise is in Windows networks (QB is my “secondary” expertise). Most large enterprises have a failover strategy. Data is streamed constantly to a backup server farm at another location (the recommended distance is 1000 miles in case of a massive power failure, tornado, earthquake or terrorist attack at the primary site). Software is available to monitor the primary site and switch to the secondary site when there’s a problem with the primary site, or when the company wants to do maintenance. The software (which is surprisingly inexpensive) tracks the transaction file to see (and later deliver) transactions that weren’t written to the streamed data. It takes only nanoseconds to make the secondary site the active site (change the IP address of the secondary server/server farm to that of the primary site so the Internet sends requests to the changed IP address).
    Intuit’s explanations (especially during last year’s debacle) usually say that the blackout/problem/whatever it was affected both the primary and secondary servers. That means the secondary server is in the same location (possibly the same room) as the primary server – which gives me chills.

    Reply
  5. Kathy, I appreciate your comments based on your IT background.

    In this case, “the root cause of the outage was a problem during a scheduled maintenance” according to an Intuit employee on their forum (see http://bit.ly/fRUhny ). That’s kind of vague, but it points to an internal problem rather than an external one, like last year’s downtime associated with power grid problems in southern California. Scheduled maintenance by definition is supposed to be routine, and this was anything but.

    Reply
  6. Shannon, I saw the response that this was a scheduled maintenance that apparently went awry. During scheduled maintenance you offload the tasks to another server. The assumption “we’ll take the server down for several minutes and users who can’t log in will try again soon and be successful” is wildly optimistic. I have a vpn and it would be easy to tell everyone (only three users) “don’t try to get on between 6-6:15 AM because I’m doing maintenance work on the server” – but I don’t do that because it’s not “good practice” (and admittedly it keeps me up on the proper use and methods for running a server). Using software designed to help me, I move the vpn access point to another server. I cannot tell you how many times what appears to be a simple maintenance task (hardware, drivers, software, or operating system upgrades) causes problems and what should have taken 15-20 minutes can take several hours. In fact, I probably don’t have to tell you; I’ll bet you and many others have experienced the same thing.

    Reply
  7. We discovered 25+ invoices that were not delivered because of this problem. What kind of “engineers” are they using? I would have not gone over to “the Dark Side” using this online service if I had know of this. Can I get a refund for this “service”? (so-called)

    Reply

Leave a reply

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> 

required