Inception UK ipv6 down

kramer · April 2020

Been down for my vps for 6 hours. Any other Inception UK customers seeing this? (Can't report to Inception due to covid-19 no support)

InceptionHosting · April 2020

You can open tickets you just need to fill in a box to confirm, thats the same box that asks you to read the announcements before opening a ticket and there has been an announcement up about it for about 3 hours now regarding ipv6, it will be back up by the end of the day, a number of hosts in Clouvider DC are impacted.

https://clients.inceptionhosting.com/announcements.php

Also displayed on the first page after logging in to the client area to open a ticket.

foxone · April 2020

Are you fucking kidding me, I've been banging my head against IPv6 not working for two hours lmao

InceptionHosting · April 2020

@foxone said:
Are you fucking kidding me, I've been banging my head against IPv6 not working for two hours lmao

Fixed.

Brueggus · April 2020

@AnthonySmith said:

@foxone said:
Are you fucking kidding me, I've been banging my head against IPv6 not working for two hours lmao

Fixed.

[vpsadmin ~]$ ssh -6 vps15.bxxxxx ssh: connect to host vps15.bxxxxxx port 22: Permission denied

Is incoming traffic blocked now?

Or... is tcp/udp traffic blocked?

[root@vps15 ~]# curl -6 ifconfig.co
curl: (7) couldn't connect to host
[root@vps15 ~]# ping6 !$
ping6 ifconfig.co
PING ifconfig.co(2606:4700:3039::681f:ef0a) 56 data bytes
64 bytes from 2606:4700:3039::681f:ef0a: icmp_seq=1 ttl=56 time=1.83 ms
64 bytes from 2606:4700:3039::681f:ef0a: icmp_seq=2 ttl=56 time=1.93 ms
^C
--- ifconfig.co ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1320ms
rtt min/avg/max/mdev = 1.837/1.888/1.939/0.051 ms

InceptionHosting · April 2020

@Brueggus said:

@AnthonySmith said:

@foxone said:
Are you fucking kidding me, I've been banging my head against IPv6 not working for two hours lmao

Fixed.

[vpsadmin ~]$ ssh -6 vps15.bxxxxx ssh: connect to host vps15.bxxxxxx port 22: Permission denied

Is incoming traffic blocked now?

Or... is tcp/udp traffic blocked?
[root@vps15 ~]# curl -6 ifconfig.co
curl: (7) couldn't connect to host
[root@vps15 ~]# ping6 !$
ping6 ifconfig.co
PING ifconfig.co(2606:4700:3039::681f:ef0a) 56 data bytes
64 bytes from 2606:4700:3039::681f:ef0a: icmp_seq=1 ttl=56 time=1.83 ms
64 bytes from 2606:4700:3039::681f:ef0a: icmp_seq=2 ttl=56 time=1.93 ms
^C
--- ifconfig.co ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1320ms
rtt min/avg/max/mdev = 1.837/1.888/1.939/0.051 ms

Working now?

# curl -6 ifconfig.co
2a04:92c7:e:ca3::6d42

Brueggus · April 2020

@AnthonySmith said: Working now?

Yep, thanks, Anthony!

Brueggus · May 2020

Since IPv6 connectivity was restore, I'm having significant packet loss on my VPS (but probably all are affected...)

Any idea?

outgoing:

 Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2a04:92c7:e:ca3:x:xxx                                                                                                                          40.5%   612    0.8   1.6   0.5  46.2   3.0
 2. 2a04:92c5:0:64::1                                                                                                                              18.8%   612   14.0  11.8   1.8 667.0  48.8
 3. 2a04:92c0::4                                                                                                                                    8.0%   612    6.0  13.7   1.4 563.6  44.8
 4. 2a04:92c0::9                                                                                                                                    6.0%   612   16.0  11.4   0.8 951.9  51.8
 5. lonap.thn2.peering.clouvider.net                                                                                                                6.4%   612    5.0   8.9   1.6 850.7  42.3
 6. e0-35.core3.lon1.he.net                                                                                                                         5.9%   612   11.0  18.1   7.1 861.8  57.1
 7. 100ge7-2.core1.lon2.he.net                                                                                                                     14.1%   611    3.9  12.2   1.3 891.9  63.5
 8. 100ge13-2.core1.nyc4.he.net                                                                                                                     6.4%   611   91.1  78.2  69.5 771.1  46.2
 9. tserv1.nyc4.he.net                                                                                                                              6.2%   611   74.5  84.0  71.7 1241.  70.0
10. xxxxxxxxxxxxxxxxxxx                                                                                                   6.4%   611   95.7  91.1  69.5 898.1  60.0


Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2a04:92c7:e:ca3:x:xxxx                                                                                                                          49.1%   553    1.3   1.4   0.5  12.1   1.4
 2. 2a04:92c5:0:64::1                                                                                                                              16.3%   553    2.8   9.6   2.0 767.6  44.3
 3. 2a04:92c0::4                                                                                                                                    8.1%   553    2.2  12.9   1.6 706.4  42.8
 4. 2a04:92c0::9                                                                                                                                    7.4%   553    3.9  13.9   1.0 968.1  65.9
 5. 2001:418:16::103                                                                                                                                7.4%   553    2.2  11.8   1.8 914.5  60.0
 6. ldn-b1-link.telia.net                                                                                                                          57.9%   553    2.7   9.7   1.4 860.8  58.5
 7. ldn-bb4-v6.telia.net                                                                                                                           97.3%   553   19.0  20.9  18.2  42.2   6.1
 8. prs-bb4-v6.telia.net                                                                                                                           96.6%   553   19.8  26.2  18.2  86.0  16.1
 9. ffm-bb4-v6.telia.net                                                                                                                            6.9%   553   18.1  28.4  17.6 924.9  71.4
10. ffm-b1-v6.telia.net                                                                                                                             7.1%   553   20.0  31.3  18.7 870.5  66.3
11. as33891-ic-331888-ffm-b1.c.telia.net                                                                                                            7.1%   553   19.3  26.8  15.6 815.6  65.5
12. 2a01:4a0:0:2021::3                                                                                                                              7.2%   553   16.8  27.2  15.8 760.9  59.2
13. 2a01:4a0:0:2071::33                                                                                                                             7.4%   553   18.4  25.7  15.6 706.4  54.3
14. 2a01:4a0:1338:6a::2                                                                                                                             7.4%   553   20.1  26.5  18.4 651.9  47.7
15. po8.core1-dus3.bb.as24961.net                                                                                                                   7.2%   553   20.3  26.4  18.7 598.2  44.2
16. po10.agr1-dus3.bb.as24961.net                                                                                                                   7.4%   553   20.6  32.1  20.1 1635.  85.3
17. po2.edge1-dus6.bb.as24961.net                                                                                                                   7.4%   553   21.8  34.6  20.1 601.5  43.6
18. 2001:4ba0:92f3:4a::2                                                                                                                            7.4%   553   19.6  26.2  18.7 537.0  36.1
19. 2a0f:5707:aa81:ffff:ffff:xxxx:xxxx:xxxxx                                                                                                         7.2%   553   19.8  26.1  19.3 483.1  31.4

incoming

 Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. tunnel379778.tunnel.tserv4.nyc4.ipv6.he.net                                                                                                     0.0%   520    6.5  14.5   3.7 158.7  20.0
 2. ve422.core1.nyc4.he.net                                                                                                                         0.0%   520    6.8  11.5   1.4 153.5  22.6
 3. 100ge16-2.core1.lon2.he.net                                                                                                                     0.6%   520   69.5  84.3  68.3 257.4  29.0
 4. linx-lon1.thn2.peering6.clouvider.net                                                                                                           0.0%   520   68.1  83.4  67.4 251.3  30.4
 5. ???
 6. 2a04:92c1:1:7::11                                                                                                                               0.0%   520   69.9  83.8  67.9 343.2  30.6
 7. 2a04:92c0::a                                                                                                                                    0.0%   519   96.6  83.1  68.5 257.7  25.3
 8. 2a04:92c0::5                                                                                                                                   11.2%   519   70.4  84.6  69.8 246.4  28.4
 9. 2a04:92c7:e::1                                                                                                                                  9.4%   519   70.2  82.0  67.7 244.6  29.0
10. 2a04:92c7:e:ca3::6d42                                                                                                                           6.9%   519   70.3  96.7  69.5 1849. 113.9
11. 2a04:92c7:e:6be:xx:xx                                                                                                                           6.9%   519  118.7  93.9  69.3 1755. 103.7

foxone · May 2020

I'm affected too

InceptionHosting · May 2020

@Brueggus said:
Since IPv6 connectivity was restore, I'm having significant packet loss on my VPS (but probably all are affected...)

Any idea?

outgoing:

 Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2a04:92c7:e:ca3:x:xxx                                                                                                                          40.5%   612    0.8   1.6   0.5  46.2   3.0
 2. 2a04:92c5:0:64::1                                                                                                                              18.8%   612   14.0  11.8   1.8 667.0  48.8
 3. 2a04:92c0::4                                                                                                                                    8.0%   612    6.0  13.7   1.4 563.6  44.8
 4. 2a04:92c0::9                                                                                                                                    6.0%   612   16.0  11.4   0.8 951.9  51.8
 5. lonap.thn2.peering.clouvider.net                                                                                                                6.4%   612    5.0   8.9   1.6 850.7  42.3
 6. e0-35.core3.lon1.he.net                                                                                                                         5.9%   612   11.0  18.1   7.1 861.8  57.1
 7. 100ge7-2.core1.lon2.he.net                                                                                                                     14.1%   611    3.9  12.2   1.3 891.9  63.5
 8. 100ge13-2.core1.nyc4.he.net                                                                                                                     6.4%   611   91.1  78.2  69.5 771.1  46.2
 9. tserv1.nyc4.he.net                                                                                                                              6.2%   611   74.5  84.0  71.7 1241.  70.0
10. xxxxxxxxxxxxxxxxxxx                                                                                                   6.4%   611   95.7  91.1  69.5 898.1  60.0


Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2a04:92c7:e:ca3:x:xxxx                                                                                                                          49.1%   553    1.3   1.4   0.5  12.1   1.4
 2. 2a04:92c5:0:64::1                                                                                                                              16.3%   553    2.8   9.6   2.0 767.6  44.3
 3. 2a04:92c0::4                                                                                                                                    8.1%   553    2.2  12.9   1.6 706.4  42.8
 4. 2a04:92c0::9                                                                                                                                    7.4%   553    3.9  13.9   1.0 968.1  65.9
 5. 2001:418:16::103                                                                                                                                7.4%   553    2.2  11.8   1.8 914.5  60.0
 6. ldn-b1-link.telia.net                                                                                                                          57.9%   553    2.7   9.7   1.4 860.8  58.5
 7. ldn-bb4-v6.telia.net                                                                                                                           97.3%   553   19.0  20.9  18.2  42.2   6.1
 8. prs-bb4-v6.telia.net                                                                                                                           96.6%   553   19.8  26.2  18.2  86.0  16.1
 9. ffm-bb4-v6.telia.net                                                                                                                            6.9%   553   18.1  28.4  17.6 924.9  71.4
10. ffm-b1-v6.telia.net                                                                                                                             7.1%   553   20.0  31.3  18.7 870.5  66.3
11. as33891-ic-331888-ffm-b1.c.telia.net                                                                                                            7.1%   553   19.3  26.8  15.6 815.6  65.5
12. 2a01:4a0:0:2021::3                                                                                                                              7.2%   553   16.8  27.2  15.8 760.9  59.2
13. 2a01:4a0:0:2071::33                                                                                                                             7.4%   553   18.4  25.7  15.6 706.4  54.3
14. 2a01:4a0:1338:6a::2                                                                                                                             7.4%   553   20.1  26.5  18.4 651.9  47.7
15. po8.core1-dus3.bb.as24961.net                                                                                                                   7.2%   553   20.3  26.4  18.7 598.2  44.2
16. po10.agr1-dus3.bb.as24961.net                                                                                                                   7.4%   553   20.6  32.1  20.1 1635.  85.3
17. po2.edge1-dus6.bb.as24961.net                                                                                                                   7.4%   553   21.8  34.6  20.1 601.5  43.6
18. 2001:4ba0:92f3:4a::2                                                                                                                            7.4%   553   19.6  26.2  18.7 537.0  36.1
19. 2a0f:5707:aa81:ffff:ffff:xxxx:xxxx:xxxxx                                                                                                         7.2%   553   19.8  26.1  19.3 483.1  31.4

incoming

 Host                                                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. tunnel379778.tunnel.tserv4.nyc4.ipv6.he.net                                                                                                     0.0%   520    6.5  14.5   3.7 158.7  20.0
 2. ve422.core1.nyc4.he.net                                                                                                                         0.0%   520    6.8  11.5   1.4 153.5  22.6
 3. 100ge16-2.core1.lon2.he.net                                                                                                                     0.6%   520   69.5  84.3  68.3 257.4  29.0
 4. linx-lon1.thn2.peering6.clouvider.net                                                                                                           0.0%   520   68.1  83.4  67.4 251.3  30.4
 5. ???
 6. 2a04:92c1:1:7::11                                                                                                                               0.0%   520   69.9  83.8  67.9 343.2  30.6
 7. 2a04:92c0::a                                                                                                                                    0.0%   519   96.6  83.1  68.5 257.7  25.3
 8. 2a04:92c0::5                                                                                                                                   11.2%   519   70.4  84.6  69.8 246.4  28.4
 9. 2a04:92c7:e::1                                                                                                                                  9.4%   519   70.2  82.0  67.7 244.6  29.0
10. 2a04:92c7:e:ca3::6d42                                                                                                                           6.9%   519   70.3  96.7  69.5 1849. 113.9
11. 2a04:92c7:e:6be:xx:xx                                                                                                                           6.9%   519  118.7  93.9  69.3 1755. 103.7

Pop a ticket in.

InceptionHosting · May 2020

@foxone said:
I'm affected too

I am checking now, span up a test VM, what I am 99% sure I am seeing is just ICMP queuing, I cannot see any issues with TCP, are you monitoring over ICMP only as well?

Brueggus · May 2020

@AnthonySmith said: I cannot see any issues with TCP, are you monitoring over ICMP only as well?

I used to monitor over UDP and switched to TCP yesterday around noon. I'm still having the same amount of packet loss.

InceptionHosting · May 2020

@Brueggus said: I used to monitor over UDP and switched to TCP yesterday around noon. I'm still having the same amount of packet loss.

Let me take a look in to this further then.

InceptionHosting · May 2020

@Brueggus some settings tweaked, can you check now please although this really only should have been impacting ping/icmp.

foxone · May 2020

@AnthonySmith said:

@foxone said:
I'm affected too

are you monitoring over ICMP only as well?

Yes, i guess i shouldn't be worried then?

InceptionHosting · May 2020

@foxone said: Yes, i guess i shouldn't be worried then?

tweaks have been tweaked, see if thats better for you.

Brueggus · May 2020

@AnthonySmith said:
@Brueggus some settings tweaked, can you check now please although this really only should have been impacting ping/icmp.

I'd say it looks much better now, there are no more gaps in the graphs. I'll check again in an hour or two.

Thanks a bunch!

sureiam · May 2020

@kramer said:
(Can't report to Inception due to covid-19 no support)

No matter what a provider says, down time is always a good reason to open a ticket (after reading announcements).

InceptionHosting · May 2020

@sureiam said: No matter what a provider says, down time is always a good reason to open a ticket (after reading announcements).

yep.

Brueggus · May 2020

IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

My internal IP: 10.10.100.80. mtrs below:

outgoing:

  1. 2a04:92c7:x:xxx::6d42  0.0%    10    2.8   1.6   0.9   2.8   0.7
  2. ???                   100.0    10    0.0   0.0   0.0   0.0   0.0

incoming:

  1.|-- 2a00:c70:1:213:246:xx:xxx:1            0.0%    10    0.3   0.3   0.3   0.4   0.0
  2.|-- ikoula.demarc.cogentco.com             0.0%    10    1.6   3.0   1.3  12.2   3.5
  3.|-- 2a00:c70::1:16                         0.0%    10    1.3  86.3   1.2 434.5 178.6
  4.|-- 2001:7f8:1::a502:1409:1                0.0%    10    3.2  37.4   3.1 337.5 105.5
  5.|-- 10gigabitethernet-2-2.par2.he.net      0.0%    10    2.7  11.1   2.6  25.1   9.7
  6.|-- 100ge11-1.core1.lon2.he.net            0.0%    10   29.9  14.4  10.7  29.9   5.9
  7.|-- linx-lon1.thn2.peering6.clouvider.net  0.0%    10   11.0  11.7  11.0  15.8   1.4
  8.|-- ???                                   100.0    10    0.0   0.0   0.0   0.0   0.0
  9.|-- 2a04:92c1:1:7::11                      0.0%    10   11.6  11.9  11.3  16.0   1.4
 10.|-- 2a04:92c0::a                           0.0%    10   29.1  17.3  11.9  29.1   5.8
 11.|-- 2a04:92c0::5                           0.0%    10   12.4  15.9  12.0  30.3   5.6
 12.|-- 2a04:92c7:e::1                         0.0%    10   11.5  11.4  11.2  12.0   0.0
 13.|-- ???                                   100.0    10    0.0   0.0   0.0   0.0   0.0

InceptionHosting · May 2020

@Brueggus said:
IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

My internal IP: 10.10.100.80. mtrs below:

outgoing:

  1. 2a04:92c7:x:xxx::6d42  0.0%    10    2.8   1.6   0.9   2.8   0.7
  2. ???                   100.0    10    0.0   0.0   0.0   0.0   0.0

incoming:

  1.|-- 2a00:c70:1:213:246:xx:xxx:1            0.0%    10    0.3   0.3   0.3   0.4   0.0
  2.|-- ikoula.demarc.cogentco.com             0.0%    10    1.6   3.0   1.3  12.2   3.5
  3.|-- 2a00:c70::1:16                         0.0%    10    1.3  86.3   1.2 434.5 178.6
  4.|-- 2001:7f8:1::a502:1409:1                0.0%    10    3.2  37.4   3.1 337.5 105.5
  5.|-- 10gigabitethernet-2-2.par2.he.net      0.0%    10    2.7  11.1   2.6  25.1   9.7
  6.|-- 100ge11-1.core1.lon2.he.net            0.0%    10   29.9  14.4  10.7  29.9   5.9
  7.|-- linx-lon1.thn2.peering6.clouvider.net  0.0%    10   11.0  11.7  11.0  15.8   1.4
  8.|-- ???                                   100.0    10    0.0   0.0   0.0   0.0   0.0
  9.|-- 2a04:92c1:1:7::11                      0.0%    10   11.6  11.9  11.3  16.0   1.4
 10.|-- 2a04:92c0::a                           0.0%    10   29.1  17.3  11.9  29.1   5.8
 11.|-- 2a04:92c0::5                           0.0%    10   12.4  15.9  12.0  30.3   5.6
 12.|-- 2a04:92c7:e::1                         0.0%    10   11.5  11.4  11.2  12.0   0.0
 13.|-- ???                                   100.0    10    0.0   0.0   0.0   0.0   0.0

yep, just recovering from an attack from China, most nodes have IPv6 again, the NAT service is not playing ball, looks like the whole node has got itself in a twist, rebooting the whole node atm.

InceptionHosting · May 2020

@Brueggus said: IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

Finally sorted, I can go to bed now for a few hours.

WSS · May 2020

@AnthonySmith said:

@Brueggus said: IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

Finally sorted, I can go to bed now for a few hours.

BUT MY 32MB OVZ6 REEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!

InceptionHosting · May 2020

@WSS said:

@AnthonySmith said:

@Brueggus said: IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

Finally sorted, I can go to bed now for a few hours.

BUT MY 32MB OVZ6 REEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!

You missed an E, totally changed the whole meme

WSS · May 2020

Sorry,

My bandwidth failed. I'm using Acorn with two serial inputs and sometimes charactrs get dropd.

Brueggus · May 2020

@AnthonySmith said:

@Brueggus said: IPv6 is down again for me at the moment. I'm still a bit unsure about ticket vs. forum, but since I do not consider this an "emergency", I'll leave it here for now.

Finally sorted, I can go to bed now for a few hours.

Anthony, you are amazing. Thanks for the effort you put into LES.

Inception UK ipv6 down

Comments

BUT MY 32MB OVZ6 REEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!

BUT MY 32MB OVZ6 REEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!