Help with HostHatch private network, can't see the other side

tarasistarasis OG
in Help

Struggling to get this running, and was hoping someone might be able to point me in the right direction.

I have enabled Private networking on old and new storage servers, and power cycled as instructed.

Old server is running Arch Linux, and uses netctl for networking.
New server currently Ubuntu. (will probably swap to Arch, want to get this working first).

Running ip -o link | grep 00:22 on both devices shows an ens4 device.

3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff\    altname enp0s4

On the Arch box, I have added an internal-network file with

Interface=ens4
Connection=ethernet
IP=static
Address=('192.168.10.3/24')
MACAddress=00:22:snip
// tried with and without the MACAddress line

On Ubuntu I have /etc/netplan/90-private.yaml per the docs https://docs.hosthatch.com/networking/#private-networking

network:
    ethernets:
        ens4:
            addresses:
            - 192.168.10.2/24
            match:
                macaddress: 00:22<skip>
    version: 2

After enabling respective files, ip link returns showing ens4 is up.

Arch:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s4

Ubuntu:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:20:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s4

And the routing

Arch:

default via 31.snip dev ens3 
31.snip/24 dev ens3 proto kernel scope link src 31.snip 
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

Ubuntu

default via 31.snip dev ens3 
31.snip/24 dev ens3 proto kernel scope link src 31.snip 
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

But when I try and ping from one machine to the other I get destination unreachable.

What am I missing? Coz I'm baffled right now

«1

Comments

  • skorousskorous OG

    Dumb question but since you don't say, they're at the same location, right?

    tcpdump the interface on the destination side and see if you see the traffic coming in?

    Thanked by (1)tarasis
  • tarasistarasis OG

    @skorous said:
    Dumb question but since you don't say, they're at the same location, right?

    Good point, I forgot to specify they are both in Amsterdam. Although one is marked "Legacy".

    tcpdump the interface on the destination side and see if you see the traffic coming in?

    Nothing appears when pinging from the Arch side to the Ubuntu side. The lines below are from me starting a ping on the Ubuntu side

    root@backup2tb:~# tcpdump -i 2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:35:51.568578 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28
01:35:52.580830 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28
  • AuroraZeroAuroraZero ModeratorHosting Provider

    Routing try ens4 not ens3

    Thanked by (1)skorous

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • FrankZFrankZ Moderator

    I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.
    Ubuntu
    ip route add 192.168.10.0/24 via 192.168.10.2 dev ens4

    Arch:
    ip route add 192.168.10.0/24 via 192.168.10.3 dev ens4

    Thanked by (2)AuroraZero tarasis

    For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add

  • skorousskorous OG

    @AuroraZero said:
    Routing try ens4 not ens3

    But the private addresses are on ens4 ... won't that just break his external network?

    Thanked by (1)AuroraZero
  • skorousskorous OG

    @FrankZ said:
    I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

    I have a feeling I'm missing something obvious but isn't that the highlighted lines?

    @tarasis said: Arch:

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

    Ubuntu

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

    Thanked by (3)yoursunny AuroraZero tarasis
  • skorousskorous OG

    @tarasis said:

    tcpdump the interface on the destination side and see if you see the traffic coming in?

    Nothing appears when pinging from the Arch side to the Ubuntu side. The lines below are from me starting a ping on the Ubuntu side

    root@backup2tb:~# tcpdump -i 2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:35:51.568578 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28
01:35:52.580830 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28

    When you say 'starting a ping on the Ubuntu side' you mean you have the tcpdump running on Ubuntu side and have Ubuntu try to ping Arch, right? So the Ubuntu machine is seeing it's own traffic ( guessing so since it's knows its name )? What happens if you tcpdump the Arch side and try the same?

    Shouldn't it be tcpdump -i ens4? I've never used a number like that...

    Thanked by (2)AuroraZero tarasis
  • FrankZFrankZ Moderator
    edited December 2022

    @skorous said:

    @FrankZ said:
    I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

    I have a feeling I'm missing something obvious but isn't that the highlighted lines?

    @tarasis said: Arch:

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

    Ubuntu

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

    Although I am not any kind of routing expert, in my opinion, after the ip route commands I showed above are entered there should be additional route lines
    192.168.10.0/24 via 192.168.10.2 dev ens4 - on Ubuntu
    192.168.10.0/24 via 192.168.10.3 dev ens4 - on Arch

    Because right now I see everything going over the default route.
    IIAC The lines you highlighted above show that the related IPs are UP and attached to dev ens4 the on the local machine, but not any routing.

    Thanked by (1)AuroraZero

    For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add

  • skorousskorous OG
    edited December 2022

    @FrankZ said:

    @skorous said:

    @FrankZ said:
    I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

    I have a feeling I'm missing something obvious but isn't that the highlighted lines?

    @tarasis said: Arch:

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

    Ubuntu

    default via 31.snip dev ens3
    31.snip/24 dev ens3 proto kernel scope link src 31.snip
    192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

    Although I am not any kind of routing expert, in my opinion, after the ip route commands I showed above are entered there should be additional route lines
    192.168.10.0/24 via 192.168.10.2 dev ens4 - on Ubuntu
    192.168.10.0/24 via 192.168.10.3 dev ens4 - on Arch

    Because right now I see everything going over the default route.
    IIAC The lines you highlighted above show that the related IPs are UP and attached to dev ens4 the on the local machine, but not any routing.

    They don't need any routing because the kernel knows what network it's on. You only need that if you've got access to something else via that network.

    [root@cerebro ~]# ip addr show ens4
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:22:da:df:41:cf brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 10.174.2.24/8 brd 10.255.255.255 scope global noprefixroute eth1
    valid_lft forever preferred_lft forever

    [root@cerebro ~]# ip route
    default via 193.29.62.1 dev eth0 proto static metric 100

    10.0.0.0/8 dev eth1 proto kernel scope link src 10.174.2.24 metric 101

    10.89.0.0/24 dev cni-podman1 proto kernel scope link src 10.89.0.1

    192.168.0.0/16 dev nebula1 proto kernel scope link src 192.168.2.157 mtu 1400

    192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

    193.29.62.0/24 dev eth0 proto kernel scope link src metric 100

    Sorry about the formatting. Stupid thing.

    Thanked by (2)AuroraZero tarasis
  • FrankZFrankZ Moderator

    I'll just crawl back in my hole then.

    Thanked by (2)AuroraZero tarasis

    For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add

  • skorousskorous OG

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    Thanked by (2)AuroraZero tarasis
  • AuroraZeroAuroraZero ModeratorHosting Provider
    edited December 2022

    Edit: told you it was stupid

    Try putting in the real Mac addresses

    Thanked by (2)skorous tarasis

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • skorousskorous OG

    @AuroraZero said:
    Stupid question did you restart networking on both machines?

    Ah, good question.

    My best guess is that they each accidentally got their own private network so they don't see each other. Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

    Thanked by (2)AuroraZero tarasis
  • FrankZFrankZ Moderator

    @skorous said:

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

    Thanked by (3)skorous AuroraZero tarasis

    For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add

  • AuroraZeroAuroraZero ModeratorHosting Provider

    @skorous said:

    @AuroraZero said:
    Stupid question did you restart networking on both machines?

    Ah, good question.

    My best guess is that they each accidentally got their own private network so they don't see each other. Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

    Normally it would but I don't know Hosthatch's setup on these. I was reading their article on it and it hit me but now I think he might not have the right MACs.

    Thanked by (2)skorous tarasis

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • skorousskorous OG

    @FrankZ said:

    @skorous said:

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

    phew Okay good. I feel better. Yeah, I'm a RHEL guy too. Was actually certified back on 6.

    The one that has me curious is that he has a mix of old and new names on the Ubuntu box. I was under the impression it was all or nothing although I guess maybe netplan is mapping the name somehow.

    Thanked by (3)AuroraZero FrankZ tarasis
  • AuroraZeroAuroraZero ModeratorHosting Provider

    @FrankZ said:

    @skorous said:

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

    Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

    Thanked by (3)FrankZ skorous tarasis

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • skorousskorous OG
    edited December 2022

    @AuroraZero said:

    @FrankZ said:

    @skorous said:

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

    Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

    Started on Slackware back in .... 93'ish? Couldn't have been much later. Dropped out by 95 I think... Still have my 0.99 3.5 inch floppies though I'm sure they've got rampant bitrot by now ...

    EDIT: rats, time for bed. Was just having fun too. >:(

    Thanked by (2)AuroraZero tarasis
  • AuroraZeroAuroraZero ModeratorHosting Provider

    @skorous said:

    @AuroraZero said:

    @FrankZ said:

    @skorous said:

    @FrankZ said:
    I'll just crawl back in my hole then.

    No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

    I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

    Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

    Started on Slackware back in .... 93'ish? Couldn't have been much later. Dropped out by 95 I think... Still have my 0.99 3.5 inch floppies though I'm sure they've got rampant bitrot by now ...

    Those were the days. No GUI command line interface by default. 8 hours of downloading to get the OS. Man we were so cool back then. Now we have become.... Ehhhhhh mainstream almost.

    Thanked by (1)tarasis

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • FrankZFrankZ Moderator

    OP seems gone from the conversation. @tarasis where are you? How are things going now ?

    Thanked by (2)AuroraZero tarasis

    For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add

  • AuroraZeroAuroraZero ModeratorHosting Provider

    @FrankZ said:
    OP seems gone from the conversation. @tarasis where are you? How are things going now ?

    Probably tried to hard taking a dump and passed out, but seriously let us know what is going on! This tension is killing me.

    Thanked by (1)tarasis

    Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?

  • tetechtetech OG

    Did you check the firewall?

    Thanked by (3)FrankZ ehab tarasis
  • arirangarirang
    edited December 2022

    I have a same problem in Amsterdam and On OGF I saw the same problem which he said was on their side.

    Thanked by (1)tarasis
  • tarasistarasis OG

    Apologies everyone for leaving the conversation. It was 2am and I crashed out, so you were kinda right @AuroraZero. Will have a read of all you have said and let you know status.

    yes I have tried rebooting after making changes.
    No firewall (yet) on Ubuntu, so it should receive pings without issue.
    Yes firewall on Arch, but its open for ping from anywhere.

    @skorous said:
    When you say 'starting a ping on the Ubuntu side' you mean you have the tcpdump running on Ubuntu side and have Ubuntu try to ping Arch, right? So the Ubuntu machine is seeing it's own traffic ( guessing so since it's knows its name )? What happens if you tcpdump the Arch side and try the same?

    Nothing appears in the output from the Ubuntu box

    but there is data from other sources

    ```[root@backup ~]# tcpdump -i ens4
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    14:06:40.156922 ARP, Request who-has 5. tell backup, length 28
    14:06:40.182060 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
    14:06:41.163887 ARP, Request who-has 5. tell backup, length 28
    14:06:41.190542 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
    14:06:42.177211 ARP, Request who-has 5. tell backup, length 28
    14:06:42.203886 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
    14:06:42.244691 ARP, Request who-has wine. tell backup, length 28


    However, tcpdump on the Ubuntu side the output is

    root@backup2tb:~# tcpdump -i ens4
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    13:14:29.756928 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
    13:17:38.343637 IP6 fe80::fc22:1ff:fec5:7af > ip6-allrouters: ICMP6, router solicitation, length 16
    13:18:11.136796 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
    13:25:51.605649 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
    13:26:18.537889 IP6 fe80::fc22:1ff:fec5:7af > ip6-allrouters: ICMP6, router solicitation, length 16


    So I think that is something for me to look deeper at

> Shouldn't it be ```tcpdump -i ens4```? I've never used a number like that...

It can be, the 2 came from when I did tcpdump -D

    [root@backup ~]# tcpdump -D
    1.ens3 [Up, Running, Connected]
    2.ens4 [Up, Running, Connected]
    3.any (Pseudo-device that captures on all interfaces) [Up, Running]
    4.lo [Up, Running, Loopback]
    5.bluetooth-monitor (Bluetooth Linux Monitor) [Wireless]
    6.nflog (Linux netfilter log (NFLOG) interface) [none]
    7.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
    8.dbus-system (D-Bus system bus) [none]
    9.dbus-session (D-Bus session bus) [none]
    ```

    Seemingly -i understands the 2 to match the output of -D.

    Slackware ... gosh that takes me back. I think I tried it and didn't get on and switched to RedHat around 1995 (we had solaris at uni, I wanted something unix esque as home so I could run some code at home in something "similar" to Uni). Then years later Suse, Ubuntu, Gentoo, Arch.

    @AuroraZero said:
    Edit: told you it was stupid

    Try putting in the real Mac addresses

    I was using the Mac reported by the ip -o link | grep 00:22 command. What I'm about to try is dropping the MAC address from the Ubuntu (Netplan) config.

    @arirang said:
    I have a same problem in Amsterdam and On OGF I saw the same problem which he said was on their side.

    Interesting, what OS's are you going between?

  • tarasistarasis OG

    Also thanks everyone for chipping in so far, greatly appreciated.

  • tarasistarasis OG

    @skorous said:

    @AuroraZero said:
    Stupid question did you restart networking on both machines?

    Ah, good question.

    My best guess is that they each accidentally got their own private network so they don't see each other.

    Thats my worry, and given their setup is basically a black box I will need to contact them I suspect. But want to try and rule out everything I can first.

    Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

    So given what you said, its interesting that the Ubuntu side is seeing absolutely nothing on that interface but Arch does.

    Going to eliminate part of the equation now and set up the Ubuntu box as Arch as well.

  • cyberpunkcyberpunk

    Yes, there seems to be an issue with private network in NL location (at least for BF sale storage VPSes). I'm not sure if anyone has an open ticket with HostHatch regarding this, so I've made one.

    Thanked by (1)tarasis
  • tarasistarasis OG

    Yeah I hadn't yet, cheers @cyberpunk. Should I hold off also opening a ticket? Maybe just mention me in yours?

    Just finished installing Arch on new box, and configured everything as per original Arch storage server and still no dice. The new storage server doesn't see anything at all on ens4

    [root@backup2TB ~]# tcpdump -i ens4
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes

    Where the old box starts seeing things within a few seconds.

  • cyberpunkcyberpunk
    edited December 2022

    @tarasis said:
    Yeah I hadn't yet, cheers @cyberpunk. Should I hold off also opening a ticket? Maybe just mention me in yours?

    I'm not really sure about internal HH policies and if extra tickets from affected clients actually help or slow down the process instead. Also, there is a good chance that the issue is already known to them, but very low on priority list for fixes, since it doesn't affect much provided service :)

    Thanked by (1)tarasis
  • tarasistarasis OG

    @cyberpunk I'll wait then, I'd rather not poke more than I have to. Cheers.

Sign In or Register to comment.

© LowEndSpirit 2024

ipv6 ready