Hosthatch down?

bjobjo OG
edited September 2023 in Outages

I wanted to reboot a Stockholm VM into an ISO. "Booting up" -> "Offline". Wanted to create a ticket. Now Cloudflare tells me cloud.hosthatch.com is down.

Tagged:
«1

Comments

  • JabJab

    Isn't console the old panel that was supposed to be dead?
    https://cloud.hosthatch.com/login

    My server in Amsterdam is online.

    Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
    https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png

  • MallocVoidstarMallocVoidstar

    cloud.hosthatch.com is up and so is my VPS (Netherlands)

  • TheDPTheDP OG

    Both cloud.hosthatch.com and my VPS in Stockholm are accessible.

  • bjobjo OG

    At leat the console is up again, so I can create a ticket regarding the boot issue.

  • beaglebeagle OG

    My storage server in NL is down since 15:00 GMT.
    Waiting for their response to my ticket.

  • Shot²Shot²
    edited September 2023

    Storage server in Stockholm: pings over ipv4/v6, but otherwise unconnectable/unresponsive. The online console reveals a ton of error messages over the last few hours. Reboot/poweroff through the website fails. Host/hardware crash?

    YAY DRAMA, BILLIONS LOST

    edit: things are back

  • nyamenknyamenk

    HH storage STO Confirmed downtime 18min according to hetrix, now back again

    Sultan Muda - Amazon Store

  • dosaidosai OG

    Hetrixtools show 8 minutes downtime.

  • dosaidosai OG

    Seems to be an ipv6 issue again.

  • MumblyMumbly

    I don't monitor IPv6 uptime here (my bad, I should...) but right this moment when I saw your post and checked, IPv6 works for me.

  • bjobjo OG

    Support replied they had a hardware failure in Stockholm.

  • rattlecattlerattlecattle

    Same here the StockHolm VM was unbootable from the panel yesterday. Seems to be fixed now but IPv6 is now broken.

  • MumblyMumbly

    I am wondering if we're on the different nodes. No outage here and IPv6 is up & running. Last time was right the opposite. IPv6 was up for everyone except for me.

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
#              Yet-Another-Bench-Script              #
#                     v2023-09-06                    #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #

Wed 20 Sep 10:44:00 CEST 2023

Basic System Information:
---------------------------------
Uptime     : 14 days, 21 hours, 45 minutes
Processor  : AMD EPYC 7443P 24-Core Processor
CPU cores  : 2 @ 2844.656 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM        : 7.8 GiB
Swap       : 975.0 MiB
Disk       : 53.8 GiB
Distro     : Debian GNU/Linux 12 (bookworm)
Kernel     : 6.1.0-11-amd64
VM Type    : KVM
IPv4/IPv6  : ✔ Online / ✔ Online

IPv6 Network Information:
---------------------------------
ISP        : HostHatch
ASN        : AS63473 HostHatch, LLC
Host       : HostHatch, LLC
Location   : Stockholm, Stockholm County (AB)
Country    : Sweden

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 278.28 MB/s  (69.5k) | 3.34 GB/s    (52.2k)
Write      | 279.02 MB/s  (69.7k) | 3.35 GB/s    (52.4k)
Total      | 557.30 MB/s (139.3k) | 6.70 GB/s   (104.6k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 4.67 GB/s     (9.1k) | 4.89 GB/s     (4.7k)
Write      | 4.92 GB/s     (9.6k) | 5.21 GB/s     (5.0k)
Total      | 9.59 GB/s    (18.7k) | 10.10 GB/s    (9.8k)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
-----           | -----                     | ----            | ----            | ----           
Clouvider       | London, UK (10G)          | 153 Mbits/sec   | 359 Mbits/sec   | 26.4 ms        
Scaleway        | Paris, FR (10G)           | busy            | busy            | 38.6 ms        
NovoServe       | North Holland, NL (40G)   | busy            | busy            | 20.7 ms        
Uztelecom       | Tashkent, UZ (10G)        | busy            | 1.28 Gbits/sec  | 112 ms         
Clouvider       | NYC, NY, US (10G)         | 41.7 Mbits/sec  | 59.8 Mbits/sec  | 96.2 ms        
Clouvider       | Dallas, TX, US (10G)      | 13.7 Mbits/sec  | 77.0 Mbits/sec  | 124 ms         
Clouvider       | Los Angeles, CA, US (10G) | 25.2 Mbits/sec  | 35.8 Mbits/sec  | 153 ms         

iperf3 Network Speed Tests (IPv6):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
-----           | -----                     | ----            | ----            | ----           
Clouvider       | London, UK (10G)          | 153 Mbits/sec   | 382 Mbits/sec   | 26.2 ms        
Scaleway        | Paris, FR (10G)           | 5.03 Gbits/sec  | 3.11 Gbits/sec  | 35.5 ms        
NovoServe       | North Holland, NL (40G)   | busy            | busy            | 20.8 ms        
Uztelecom       | Tashkent, UZ (10G)        | busy            | 1.20 Gbits/sec  | 112 ms         
Clouvider       | NYC, NY, US (10G)         | 40.8 Mbits/sec  | 84.8 Mbits/sec  | 96.1 ms        
Clouvider       | Dallas, TX, US (10G)      | 31.0 Mbits/sec  | 77.2 Mbits/sec  | 125 ms         
Clouvider       | Los Angeles, CA, US (10G) | 25.4 Mbits/sec  | 41.7 Mbits/sec  | 153 ms         

Geekbench 6 Benchmark Test:
---------------------------------
Test            | Value                         
                |                               
Single Core     | 1824                          
Multi Core      | 3339
  • TheDPTheDP OG

    @Mumbly said: I am wondering if we're on the different nodes. No outage here and IPv6 is up & running.

    Could have been related to the storage servers.

    My E5 NVMe server has been up.

    Thanked by (1)Mumbly
  • MumblyMumbly

    @TheDP said: Could have been related to the storage servers.

    Ah, this would explain it. I don't have storage server there.

  • rattlecattlerattlecattle

    Indeed mine is a storage server in Stockholm.

    Thanked by (1)Mumbly
  • beaglebeagle OG

    @beagle said:
    My storage server in NL is down since 15:00 GMT.
    Waiting for their response to my ticket.

    Their response:

    The host node your VM is being hosted on suffered from a hardware failure and we had to dispatch a technician to replace the RAID controller. It is now back online and we are continue to monitoring it closely. We are sorry for the inconvenience caused.

  • Shot²Shot²

    Storage server in SE back up, but IPv6 is now broken :p

    Thanked by (1)bjo
  • bjobjo OG

    Same here:

    PING fe80::1(fe80::1) 56 data bytes
^C
--- fe80::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms
  • beaglebeagle OG

    Update from HH LA storage server:

    Dear Customer,

    This is an important message regarding your server(s);
    server (x.x.x.x) - Los Angeles

    Due to a multi disk failure in the RAID50 array, we were not able to restore your Storage VM (mentioned above) to its previous state. Please login to your account at cloud.hosthatch.com and use the reinstall feature to start using it again.

    We have moved your server to a node with RAID10 HDD storage at no additional cost, for better redundancy and performance of your VM in the future. We will also be issuing 30 days of credit to your account later this week.

    Thank you for your understanding and we apologize for the inconvenience caused.

  • cybertechcybertech OGBenchmark King
    edited September 2023

    @beagle said:
    Update from HH LA storage server:

    Dear Customer,

    This is an important message regarding your server(s);
    server (x.x.x.x) - Los Angeles

    Due to a multi disk failure in the RAID50 array, we were not able to restore your Storage VM (mentioned above) to its previous state. Please login to your account at cloud.hosthatch.com and use the reinstall feature to start using it again.

    We have moved your server to a node with RAID10 HDD storage at no additional cost, for better redundancy and performance of your VM in the future. We will also be issuing 30 days of credit to your account later this week.

    Thank you for your understanding and we apologize for the inconvenience caused.

    RAID10 ftw and they should have done this from the get go.

    I bench YABS 24/7/365 unless it's a leap year.

  • bjobjo OG

    Seems like bandwidth accounting does not work any more. Stockholm storage is stuck at 300mb and Amsterdam has not used anything yet.

  • MumblyMumbly
    edited September 2023

    @bjo said: Seems like bandwidth accounting does not work any more. Stockholm storage is stuck at 300mb and Amsterdam has not used anything yet.

    It sounds like 80% of my VPSes :P

  • bjobjo OG

    @Mumbly said:
    It sounds like 80% of my VPSes :P

    But I moved around 2 TB :D

  • beaglebeagle OG

    IPv6 not working on the LA replacement server.
    Opened a ticket on 27th and I'm still waiting for a response.

  • Jiffy3252Jiffy3252

    @beagle said:
    IPv6 not working on the LA replacement server.
    Opened a ticket on 27th and I'm still waiting for a response.

    same too, that's too bad

  • msattmsatt OG

    Looks like my Stockholm VM is off again (same problem as reported in this thread).
    Reboots fail and console just 'Waiting' - panel shows as @ offline.
    Power off and boot, panel showing VM offline
    Also getting cloudflare timeouts when accessing https://cloud.hosthatch.com/

  • MumblyMumbly

    Storage or NVMe package?
    My NVMe VPS is up.

  • dosaidosai OG

    My Stockholm storage vm is also down.

  • bjobjo OG
    edited October 2023

    Same here with storage, but seems up again.

Sign In or Register to comment.

© LowEndSpirit 2024

ipv6 ready