Letbox hack/outage

tetech · January 6

From https://my.letbox.com/serverstatus.php

Outage / High Loads - Virtualization (VPS) Platform. Node Specific.

Many VPS nodes down for base O/S re-installation to clean up from prior DDOS/hacker issues and secure systems. Service will be restored per node as each process completed.

UPDATE:

We are currently working to restore service to nodes affected from prior ddos/hack issues.

While working smoothly one server at a time to clean up the prior issue, the hacker got angry and decided to corrupt the O/S on all systems he/she/they still could get to that were not re-installed/patched yet.

We are now scrambling to re-install all those nodes, and have been at it non-stop since last night.

The good news is, there has been no breach to any re-install nodes with the most recent version/patch we have been using for weeks now, so it is just a matter of working thru the platform to re-install/patch each and every node.

We do not have an ETA for full resolution yet, but all services should be restored today at some point, with nodes coming on line one by one as completed.

arirang · January 6

Up/Down repeatedly for a long time and a while ago but there's no notification such as above one.
Recently they sent out email few minute ago before Host node reinstall without a detail reason.
Very unprofessional and I don't think they do care their servers and even monitor their server.
I'm not review any vms at Letbox, Smarthost

FrankZ · January 6

@tetech said: From https://my.letbox.com/serverstatus.php

That does not look good at all. Thanks a lot for posting this.

Otus9051 · January 6

@arirang said:
Up/Down repeatedly for a long time and a while ago but there's no notification such as above one.
Recently they sent out email few minute ago before Host node reinstall without a detail reason.
Very unprofessional and I don't think they do care their servers and even monitor their server.
I'm not review any vms at Letbox, Smarthost

treesmokah · January 6

It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

yoursunny · January 6

The end is nigh.

skorous · January 6

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

It's definitely not a good luck, for sure.

Calypso · January 6

I'm pretty pissed at this provider. Hours of downtime and issues during the last couple of weeks. The mentioned downtime was already on it's way for hours and I created a ticket for it (first hours website wasn't even reachable), but never got an answer. At that time, literally zero mention of this on the website or service status - according to the latter everything was up and running. Maybe because they restored the website from backup or something. But on the other hand: haven't seen anything on mail from them about it.

Yes, I can imagine that you have a priority to get the hacker out, and repair stuff, but on the other side of things: communicate with your users. I haven't seen anything. If a hacker has gained access to your systems, you should always keep your customers informed - that's even mandatory in a lot of countries.

yusra · January 6

Is this going to be another drama that can be watched with popcorn and snacks?!

_MS_ · January 6

@yusra said:
Is this going to be another drama that can be watched with popcorn and snacks?!

SMARTHOST · January 16

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

Wrong.
That issue in our network status posting has absolutely nothing to do with the issue plaguing the the industry currently whatsoever.
You are making assumptions, and are quire incorrect.
At no time have we paid any ransom whatsoever.
This is basically libel.

~ SMARTHOST

skorous · January 16

@SMARTHOST said:

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

Wrong.
That issue in our network status posting has absolutely nothing to do with the issue plaguing the the industry currently whatsoever.
You are making assumptions, and are quire incorrect.
At no time have we paid any ransom whatsoever.
This is basically libel.

~ SMARTHOST

Was any notification sent out about the breach? I don't think I got one.

treesmokah · January 16

@skorous said:

@SMARTHOST said:

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

Wrong.
That issue in our network status posting has absolutely nothing to do with the issue plaguing the the industry currently whatsoever.
You are making assumptions, and are quire incorrect.
At no time have we paid any ransom whatsoever.
This is basically libel.

~ SMARTHOST

Was any notification sent out about the breach? I don't think I got one.

Nobody got one, I talked to their customers about it.

skorous · January 16

@treesmokah said:

Nobody got one, I talked to their customers about it.

Presumably a small sunset of customers. I was hoping for an official statement.

SMARTHOST · January 16

@Calypso said:
Yes, I can imagine that you have a priority to get the hacker out, and repair stuff, but on the other side of things: communicate with your users. I haven't seen anything. If a hacker has gained access to your systems, you should always keep your customers informed - that's even mandatory in a lot of countries.

So...I suppose the network status incident that we kept updated non-stop, and quoted by the OP even in this thread, didn't exist?
What is with people these days...making assumptions without any knowledge, and not even capable of reading.

Let me help you out:
The OP posted info from our network status information that was consistently updated with what status and steps were being taken regarding a ddos/hack attack our VPS platform was having. We literally posted tons of updates there as it was underway. At this point most of it consolidated and the update by update listing consolidated and/or removed. Clients were fully and non-stop informed here, and pointed to such with any support tickets as well.

This issue has absolutely nothing to do with the current WHMCS module/theme hack issue plaguing the industry.
This was an attack against our VPS platform specifically, and VPS nodes, and did not involve/affect client data.

~ SMARTHOST

treesmokah · January 16

@skorous said:

@treesmokah said:

Nobody got one, I talked to their customers about it.

Presumably a small sunset of customers. I was hoping for an official statement.

Entire database of Letbox and SmartHost were dumped. All clients are affected.

SMARTHOST · January 16

@treesmokah said:
Entire database of Letbox and SmartHost were dumped. All clients are affected.

Please explain how an attack on VPS nodes/platform can result in a dump of client database.
The OP posted regarding the ddos/hack issue on our VPS platform.
This ddos/hack issue had nothing to do whatsoever with the current WHMCS module/theme hack issue plaguing the industry, and hosting company billing systems.

~ SMARTHOST

treesmokah · January 16

@SMARTHOST said:

@treesmokah said:
Entire database of Letbox and SmartHost were dumped. All clients are affected.

Please explain how an attack on VPS nodes/platform can result in a dump of client database.
The OP posted regarding the ddos/hack issue on our VPS platform.
This ddos/hack issue had nothing to do whatsoever with the current WHMCS module/theme hack issue plaguing the industry, and hosting company billing systems.

~ SMARTHOST

Your "ddos" issue started right after attacker claimed to have your database and made demand to have vps reinstalled. There are literally logs of that.

Was your WHMCS compromised? (I know it was but I'd like to hear "official" story).

SMARTHOST · January 16

@treesmokah said:
Your "ddos" issue started right after attacker claimed to have your database and made demand to have vps reinstalled. There are literally logs of that.
Was your WHMCS compromised? (I know it was but I'd like to hear "official" story).\

Completely separate issue, and NOT the subject the OP posted about in this thread.
Once you stop confusing/intermingling the two separate issues, maybe you'd understand better.
At no time have we paid any hacker a ransom.
At no time have we re-installed client VPS from such issues.
There was no client data/breach from the ddos/hacks against our VPS platform.
In fact, as far as the issue in this thread the OP was discussing, we did the opposite, and brought in a team of external security consultants to handle the issue.

The WHMCS issue posted in the thread below, is entirely separate, and has ABSOLUTELY nothing to do with the VPS platform issues the OP was posting about in this thread:
https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
That was related to completely separate systems, and did NOT involve the VPS platform issues above.
The timing of such actually does NOT correlate between the two issues, just public visibility overlaps to make it appear so.
This issue, atleast for us, involved a 3rd party module issue, but seems to differ with some of many other hosting providers affected by this.
We consulted legal representation regarding this, and specifically acted based on their recommendations/advice.
For safety purposes all client accounts and server passwords were changed en'masse, rendering any potential data obtained worthless.
internal systems were re-installed/cleaned and keys/access changed and locked down to the extreme.
We do not store any highly sensitive data, such as credit card numbers, otherwise that would have changed our course of action, and impacted proactive notification requirements.
Any client that inquired regarding such, was indeed informed passwords were changed for safety due to a potential threat/breach.
From what we understand, the hacker considered this valid action, and therefore left us alone considering it responsible and secure action having been taken, and their threats not ignored.
At no time have we paid any hacker a ransom.

~ SMARTHOST

treesmokah · January 16

So you still didn't inform your clients of their personal information and billing details being in hands of unknown criminal actor. Nice.

@SMARTHOST said: Any client that inquired regarding such, was indeed informed passwords were changed for safety due to a potential threat/breach.

What a joke. Client has to ask a provider if they were breached just to get a statement.

treesmokah · January 16

@FrankZ @bikegremlin @mikho I will leave it up to staff to decide whether such providers are welcome here or not. Not informing clients of their data being in hands of a known extortionist is a fucking joke.

Even @Cloudie did better than that.

tetech · January 16

@SMARTHOST said:

@Calypso said:
Yes, I can imagine that you have a priority to get the hacker out, and repair stuff, but on the other side of things: communicate with your users. I haven't seen anything. If a hacker has gained access to your systems, you should always keep your customers informed - that's even mandatory in a lot of countries.

So...I suppose the network status incident that we kept updated non-stop, and quoted by the OP even in this thread, didn't exist?
What is with people these days...making assumptions without any knowledge, and not even capable of reading.

Let me help you out:
The OP posted info from our network status information that was consistently updated with what status and steps were being taken regarding a ddos/hack attack our VPS platform was having. We literally posted tons of updates there as it was underway. At this point most of it consolidated and the update by update listing consolidated and/or removed. Clients were fully and non-stop informed here, and pointed to such with any support tickets as well.

This issue has absolutely nothing to do with the current WHMCS module/theme hack issue plaguing the industry.
This was an attack against our VPS platform specifically, and VPS nodes, and did not involve/affect client data.

~ SMARTHOST

Hello, I am the OP here. To be clear, my VPS was never reinstalled, and I cannot speak of anything other than the Letbox VPS outage.

What is written above makes it sound like customers were kept well-informed throughout. At least for the first half day, there was complete radio silence and nothing posted on the status page (or anywhere else that I could find). My support ticket was not responded to until several hours after the service was restored. It was only because I checked the status page again after 48 hours that I learned what was going on.

I didn't necessarily expect anything better, but what you have written does not tally with my personal experience.

SMARTHOST · January 16

@treesmokah said:
Not informing clients of their data being in hands of a known extortionist is a fucking joke.

Not the case...there we go with assumptions again.
At no time have we stated we will not be proactively informing clientele, only that we have not done so yet.
We have every intention of doing so, once we have all details we need to do so, as it is the right thing to do, regardless of legal representation informing us it was not absolutely required due to the specifics.
There is no reasonable expectation of client harm here, and no sensitive information (credit card numbers, IDs, social security numbers, etc..) was obtained.
Combined with the security actions we have taken in the matter, the notification urgency is limited.
We do expect that we will be sending out notifications shortly, now that we have the information we need to do so, know what was involved and that issues were not related, and have systems patched and service all back to normal.
We have just had our hands full dealing with both issues, and only finally coming up for air, after many sleepless nights dealing with it all.
I don't wish having to deal with something like this on our biggest competitors, this was not pleasant and as stressful as it gets in this industry.

~ SMARTHOST

SMARTHOST · January 16

@treesmokah said:
@FrankZ @bikegremlin @mikho I will leave it up to staff to decide whether such providers are welcome here or not.

Take note you have not seen us on here or other similar forums lately, because we have had our hands full dealing with these two issues. Security and client services being operational has been our only concerns since that time period. Heck, we had minimal Black Friday and year end sales presence, because we were dealing with the VPS platform issues since late November...and that hurts. That was one of the most talented hacker(s) I have seen in my 30+ years in the industry. Hate them, but still impressed by their skills. The WHMCS breach I don't think was as much skill as it was luck or happening across 3rd party software flaw(s). We are finally just this weekend able to come up for air, and start working on next steps beyond security and re-install work.

~ SMARTHOST

SMARTHOST · January 16

@tetech said:
Hello, I am the OP here. To be clear, my VPS was never reinstalled, and I cannot speak of anything other than the Letbox VPS outage.
What is written above makes it sound like customers were kept well-informed throughout. At least for the first half day, there was complete radio silence and nothing posted on the status page (or anywhere else that I could find). My support ticket was not responded to until several hours after the service was restored. It was only because I checked the status page again after 48 hours that I learned what was going on.
I didn't necessarily expect anything better, but what you have written does not tally with my personal experience.

We kept it updated as best we could with constant updates, but there were long periods of time where there were no updates to be given. We literally were spending 16-18 hours workdays reinstalling hardware nodes, only to have issues again on same nodes within 24 hours. A combination of trial/error, external security consultant assistance, and software vendor assistance got us to a point of clearing up the issue, but it took alot of time to get there. I've been in this business longer than anyone, and this is the most persistent and talented hacker/attacker I have ever seen. My team had their hands full dealing with it, and we did the best we could keeping clients informed and responding to tickets. We literally had over 1,000 tickets between our brands to respond to about the matter, and were beyond overwhelmed...but we got thru it.

~ SMARTHOST

FrankZ · January 16

@treesmokah said:
@FrankZ @bikegremlin @mikho I will leave it up to staff to decide whether such providers are welcome here or not. Not informing clients of their data being in hands of a known extortionist is a fucking joke.

Even @Cloudie did better than that.

My opinion regarding this as a staff member:

The way I see this is that LES is a forum where people talk about things hosting related. Some of the key statements made in this thread by both the members and the provider are not verifiable by LES staff. Both have had their say, and aside from some understandable emotional overtones, everybody has acted reasonably well in their discourse with each other. I do think it was beneficial that the subject was discussed so that current and future members can make their own decisions on the subject. I I do not see the need for any comments to be moderated, members to be warned, or any provider tags to be removed.

Personal opinion based on the statements provided:

I think we all knew once the Cloudie hack became public that something big was going on, and the data we provided to hosting providers was most likely in the wild from one source or another. Working together to mitigate the damage and taking personal responsibility for self protection was the best anyone could do. It's a crap situation for everybody involved both provider and client.

@SMARTHOST has said that this hack was unrelated, and based on my past dealings with him, I have no reason to doubt he is telling the truth. IMO we are not talking about someone with questionable ethics. Hindsight is always 20/20 and I am sure we all can relate to wishing we had acted or said something a little differently in hindsight.

Thai is my four cents.

cold · January 16

@treesmokah said:
@FrankZ @bikegremlin @mikho I will leave it up to staff to decide whether such providers are welcome here or not. Not informing clients of their data being in hands of a known extortionist is a fucking joke.

Even @Cloudie did better than that.

If VirTrash is welcome here and tolerated everyone else will be.

bikegremlin · January 16

@cold said:

@treesmokah said:
@FrankZ @bikegremlin @mikho I will leave it up to staff to decide whether such providers are welcome here or not. Not informing clients of their data being in hands of a known extortionist is a fucking joke.

Even @Cloudie did better than that.

If VirTrash is welcome here and tolerated everyone else will be.

Some people like to be heavily policed.
Others won't accept any rules.

No one can please everyone.

We are trying to stay free, open, but without allowing for clear frauds and similar.
Sort of a middle-ground.

Not perfect, never will be, but it is what makes this place - with all its pros and cons.

Relja

skorous · January 16

@SMARTHOST said: There is no reasonable expectation of client harm here, and no sensitive information (credit card numbers, IDs, social security numbers, etc..) was obtained.

Is that 100% true though? For an indeterminate length of time an unknown actor had access to my account and through that the consoles of my VMs, right?

jarland · January 16

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

I’m a bit late to the thread and my brain is scattered a little on the subject. But I wanted to add what seems to me to also be plausible:

That the hacker in question never compromised them at all and simply made a false claim. He doesn’t seem very reliable.

skorous · January 16

@jarland said:

@treesmokah said:
It is related to https://lowendspirit.com/discussion/7088/cloudie-networks-llc-data-leakage
Attacker claimed he compromised SmartHost/LetBox and their other brands, however he stated he wont release the data because they "acted right" which could mean informing clients of the breach, which they didn't do.

It leads me to believe they have paid the ransom and are trying to downplay the severity of this whole mess.
Or they are fucking insane and are "fighting" with the attacker while not telling their customers anything.

Either way, I would never do business with them. They are absolutely incompetent and do not care about their customers security.

I’m a bit late to the thread and my brain is scattered a little on the subject. But I wanted to add what seems to me to also be plausible:

That the hacker in question never compromised them at all and simply made a false claim. He doesn’t seem very reliable.

Well there's that pesky customer database dump that was being passed around, right? Or are you thinking that came from a different source?

Letbox hack/outage

Comments

My opinion regarding this as a staff member:

Personal opinion based on the statements provided: